Education study praising ChatGPT retracted amid concerns
▼ Summary
– A study claiming ChatGPT positively impacts student learning has been retracted by Springer Nature due to “discrepancies” in analysis and lack of confidence in conclusions.
– The retracted paper analyzed 51 previous studies to measure ChatGPT’s effect on learning performance, perception, and higher-order thinking, finding large positive impacts.
– Experts criticized the paper for synthesizing poor-quality studies and mixing incomparable findings, suggesting it should not have been published.
– The paper was published just two and a half years after ChatGPT’s release, which experts say is too short for dozens of high-quality studies to exist.
– Despite retraction, the study has been cited 504 times, attracted nearly half a million readers, and ranked in the 99th percentile for online attention.
A study that once seemed to offer strong evidence that OpenAI’s ChatGPT could boost student learning has been pulled from publication nearly a year after it first appeared. The retraction, issued by journal publisher Springer Nature, was prompted by what the company described as “discrepancies” in the research analysis, along with a lack of confidence in the findings. By the time the paper was withdrawn, it had already accumulated hundreds of citations and generated significant buzz across social media platforms.
“The paper’s authors made some very attention-grabbing claims about the benefits of ChatGPT on learning outcomes,” said Ben Williamson, a senior lecturer at the Centre for Research in Digital Education and the Edinburgh Futures Institute at the University of Edinburgh in Scotland, in an email to Ars. “It was treated by many on social media as one of the first pieces of hard, gold standard evidence that ChatGPT, and generative AI more broadly, benefits learners.”
The now-retracted study aimed to measure “the effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking” by combining data from 51 earlier research efforts. Through a meta-analysis, the researchers calculated the effect size between experimental groups that used ChatGPT in educational settings and control groups that did not rely on the AI chatbot.
Those calculations supposedly revealed that “ChatGPT has a large positive impact on improving learning performance,” along with a “moderately positive impact on enhancing learning perception” and “fostering higher-order thinking,” according to the original authors. The findings were first published in the journal Humanities & Social Sciences Communications by Springer Nature on May 6, 2025.
“In some cases it appears it was synthesizing very poor quality studies, or mixing together findings from studies that simply cannot be accurately compared due to very different methods, populations and samples,” Williamson told Ars. “It really seemed like a paper that should not have been published in the first place.”
Williamson also raised concerns about the paper’s timing. With ChatGPT released by OpenAI in November 2022, the study appeared just two and a half years later. “It is not feasible that dozens of high-quality studies about ChatGPT and learning performance could have been conducted, reviewed, and published in that time,” Williamson said.
Despite the retraction, the paper’s influence may persist. Since its release, the study has been cited 262 times in other peer-reviewed articles within Springer Nature’s journals and received a total of 504 citations from both peer-reviewed and non-peer-reviewed sources. It also drew nearly half a million readers and earned enough online attention to rank in the 99th percentile for journal articles by attention score.
(Source: Ars Technica)
