Major study: AI chatbots struggle with news accuracy

A major international study reveals that popular AI chatbots frequently distort news content and struggle to differentiate factual reporting from opinion pieces. Research conducted by 22 public service media organizations, including prominent broadcasters, indicates these digital assistants misrepresent news stories nearly half the time. The investigation evaluated four widely used platforms, ChatGPT, Microsoft’s Copilot, Google’s Gemini, and Perplexity AI, uncovering systemic problems with accuracy and reliability across different languages and regions.

Journalists from respected outlets like the BBC and NPR assessed thousands of responses against professional standards for accuracy, sourcing, context provision, and appropriate editorial judgment. Their analysis found that 45% of all answers contained at least one significant flaw, with 31% showing serious sourcing deficiencies and 20% including major factual inaccuracies. The German broadcaster DW reported even higher error rates, with 53% of responses to their queries displaying substantial problems and 29% specifically failing accuracy checks.

The study documented concerning examples of misinformation. One response incorrectly identified Olaf Scholz as Germany’s current chancellor despite Friedrich Merz having assumed the position a month earlier. Another mistakenly named Jens Stoltenberg as NATO secretary general after Mark Rutte had already taken over the role. These errors occurred despite AI chatbots becoming increasingly popular news sources, with recent data showing 7% of online news consumers regularly using them for information, a figure that jumps to 15% among users under 25.

Jean Philip De Tender of the European Broadcasting Union, which coordinated the research, emphasized that these shortcomings represent systemic failures rather than isolated incidents. He warned that such widespread inaccuracy “endangers public trust” and could ultimately discourage democratic participation when people cannot distinguish reliable information from distorted content.

This comprehensive research project represents one of the largest examinations of AI news accuracy to date, building upon a BBC study from earlier this year that found similar issues. Media organizations from 18 countries applied consistent methodology to evaluate 3,000 AI responses to common news questions such as “What is the Ukraine minerals deal?” and “Can Trump run for a third term?” Journalists reviewed answers against their expertise and verified sources without knowing which AI platform generated each response.

While the results show minor improvement compared to the BBC’s February findings, error rates remain alarmingly high. Google’s Gemini performed poorest among the four platforms, with 72% of its responses exhibiting significant sourcing problems. Across both studies, all assistants demonstrated persistent issues with news accuracy despite their growing adoption.

BBC program director Peter Archer acknowledged the potential benefits of AI technology while stressing that trust remains paramount. “People must be able to trust what they read, watch and see,” he stated. “Despite some improvements, it’s clear that there are still significant issues with these assistants.”

In response to these findings, participating organizations are urging governments to enforce existing regulations concerning information integrity and digital services. They’re also advocating for independent monitoring systems to track AI performance as new models rapidly emerge. Additionally, the EBU has partnered with international media groups to launch the “Facts In: Facts Out” campaign, pressing AI companies to ensure their systems maintain news integrity rather than distorting or decontextualizing trusted journalism. Campaign organizers summarized their position clearly: “If facts go in, facts must come out. AI tools must not compromise the integrity of the news they use.”

(Source: DW)