🔍 Read the full analysis: 53 Takeaways On ChatGPT And Consultants’ Data Science Skills on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using GPT-4 with brief training scored higher on coding, statistics and prediction tasks than a group using Google and online resources. After ChatGPT was removed, researchers found no evidence that the users performed better on a technical test, suggesting the measured gains depended on access to the tool.
A randomized trial at Boston Consulting Group found that consultants given ChatGPT and 15 to 20 minutes of training scored higher on coding, statistics and prediction tasks than colleagues using Google and other online resources. But when both groups later answered technical questions without ChatGPT, the researchers found no evidence that the ChatGPT group performed better, pointing to a difference between completing work with AI and retaining skills independently.
The study, published in Nature Human Behaviour on October 26, 2026, examined consultants assigned to data-science tasks outside their stated skill set. The treatment group used GPT-4 after brief instruction in prompting for technical problems; the control group received instruction on using Google, Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks: cleaning and combining datasets with Python, evaluating a statistical model, or predicting the outcome patterns of international football matches.
On a scoring scale benchmarked against 44 BCG data scientists working without AI, the ChatGPT group scored 49 percentage points higher than the control group on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers said the AI group still averaged below the data scientists on all three tasks. For statistics and prediction, they reported that the results ruled out, at 95% confidence, performance equal to the expert benchmark.
The experiment allocated 986 consultants to treatment and control groups. Of those, 573 began the survey and 487 submitted work on both assigned tasks, forming the main analysis sample. After the work, participants answered five questions about topics including probability, machine learning and coding without ChatGPT. The authors reported no evidence of an advantage for the group that had used the chatbot.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
AI Gains May Depend on Access
The results bear on how employers describe and evaluate AI-assisted work. In this trial, employees completed tasks more effectively with a chatbot, but the follow-up test did not show that the experience left them better able to answer related questions on their own. That distinction matters when firms decide whether AI use counts as training, productivity support or both.
The study does not establish that AI cannot teach skills. It tests a particular short training session, a specific group of consultants and a limited set of data-science tasks. Still, it cautions against treating improved output while a tool is available as proof of durable learning. Businesses assessing AI-enabled work may need to distinguish the employee’s independent capability from what the employee can produce with the system, and retain checks on accuracy and understanding.
The researchers also reported an exploratory side effect: ChatGPT users became more confident that the tool could solve problems it could not actually solve. Because the authors label that analysis exploratory, it should be treated as a finding to investigate further, not as a settled conclusion about AI users generally.
Products Worth Considering
Ledger Nano X Family Pack – 3 Crypto Hardware Wallets with Bluetooth
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
Apple MacBook Pro 14.2-inch with Tandem OLED Display, Space Black
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The researchers preregistered the trial on March 13, 2024, before data collection, which took place in late March and early April that year. Participants had at least a bachelor’s degree, and the authors said most reported no coding experience. Each participant completed two of three tasks, each limited to 90 minutes.
BCG data scientists developed the tasks with OpenAI’s Economic Impacts Research team. The tasks were designed so ChatGPT could help but could not answer every question correctly on its own; simply copying its output would not be enough to complete them. Researchers compared participants’ scores with the average of 44 BCG data scientists who completed the same tasks without AI. That expert group served as a benchmark, not as a randomized comparison group.
The paper’s title describes the reported pattern as “temporary upskilling”: workers could take on more complex tasks with generative AI, while the study did not find a corresponding advantage on the later test without it. The experiment’s timing and design matter: the trial occurred in 2024, while the paper was published in 2026, and it tested a short course rather than a long-term training program.
““find no evidence” that the ChatGPT group did better on the follow-up technical questions without the tool.”
— The study authors
Limits of the Skill Test
The findings do not show whether longer or more intensive training could produce skills that persist after ChatGPT is removed. The follow-up consisted of five technical questions, not another full data-science assignment, and the source material does not provide evidence about participants’ performance weeks or months later.
The trial involved consultants at one company and a particular set of tasks and model conditions. It remains unclear how well the results apply to other occupations, training programs, newer AI systems or day-to-day work. The study also does not establish whether workers would retain skills after repeated use over time. The reported increase in confidence about ChatGPT’s abilities was described by the authors as exploratory, so its scope and reliability need further study.
Testing Longer-Term Learning
The study reports completed trial results; the source material does not identify a scheduled follow-up experiment or a next research milestone. Further studies could test whether longer instruction, repeated practice or deliberate explanations from an AI system lead to lasting independent performance, and whether the results hold across different jobs and tools.
For employers, the immediate open question is how to measure both the quality of AI-assisted output and workers’ ability to complete comparable tasks without assistance. Until longer-term evidence is available, the trial supports a narrower conclusion: briefly trained consultants performed better on these tasks with ChatGPT, but researchers found no evidence of a gain on the test taken after the chatbot was removed.
Key Questions
What did the study find about ChatGPT and data-science tasks?
Consultants using GPT-4 after brief training scored higher than a control group using Google and other online resources: 49 percentage points higher on coding, 20 on statistics and 17 on prediction. The ChatGPT group still averaged below the data-scientist benchmark.
Did participants retain the skills after ChatGPT was removed?
The researchers found no evidence that the ChatGPT group did better than the control group on five technical questions answered without the tool. The study does not establish what would happen after longer training or more extensive practice.
Who took part in the trial?
The trial allocated 986 BCG consultants to treatment and control groups. A total of 487 submitted work on both tasks and were included in the main analysis sample. The participants had at least a bachelor’s degree, and most reportedly had no coding experience.
Does this prove AI cannot teach people new skills?
No. The trial examined a specific short training session, a particular group of consultants and a five-question follow-up test. It found no evidence of a lasting advantage under those conditions, but it did not test every training method or measure long-term learning.
Source: ThorstenMeyerAI.com
