BBC Research Reveals Major Issues with AI-Powered News Summaries

The BBC recently conducted extensive research into how AI-powered chatbots summarize news stories, focusing on popular tools like Microsoft Copilot, OpenAI’s ChatGPT, Google’s Gemini, and Perplexity. While generative AI continues to evolve, concerns around safety, security, and accuracy remain key barriers. One of the most puzzling challenges? These tools often generate inaccurate or misleading responses to queries.

The research involved asking chatbots to summarize 100 BBC-published news stories and then answering questions based on those summaries. The findings? Significant inaccuracies and distortions in the AI-generated answers. According to the BBC News and Current Affairs CEO Deborah Turness, major issues emerged in over half of the responses. Around 19% of the answers introduced factual errors, including incorrect figures, statements, and dates. Even worse, over 10% of the “quotations” attributed to BBC articles were either altered or completely fabricated.

Turness explained, “The team found ‘significant issues’ with just over half of the answers generated by the assistants. The AI assistants introduced clear factual errors into around a fifth of answers they said had come from BBC material. And where AI assistants included ‘quotations’ from BBC articles, more than one in ten had either been altered, or didn’t exist in the article.”

One of the biggest problems is that AI tools struggle to differentiate between fact and opinion. They often mix current and archived material, inject opinions, and fail to provide essential context. As Turness describes it, “The results they deliver can be a confused cocktail of all of these – a world away from the verified facts and clarity that we know consumers crave and deserve.”

For context, the BBC enlisted its experienced reporters and journalists to evaluate the accuracy of the chatbot-generated summaries. The study found that tools like Copilot and Gemini struggled more significantly than ChatGPT and Perplexity, frequently editorializing content and leaving out critical information.

Apple has also faced challenges with AI-generated news. The tech giant recently paused its Apple Intelligence notifications after the feature began sharing misleading headlines, sparking backlash from news organizations and advocacy groups.

An OpenAI spokesperson responded to the findings, stating, “We support publishers and creators by helping 300 million weekly ChatGPT users discover quality content through summaries, quotes, clear links, and attribution.”

However, the BBC is urging a slowdown on AI news summaries. Turness emphasized the need for collaboration with AI providers to address these challenges, saying, “We can work together in partnership to find solutions.”

This research raises an important question: Can we trust AI-generated news summaries? While the technology offers convenience, the risks of misinformation remain a serious concern. As Turness warns, “We live in troubled times, and how long will it be before an AI-distorted headline causes significant real-world harm?”