Google pulls AI health summaries over “dangerous” flaw findings

Google has disabled certain AI-generated health summaries in its search results following concerns that the system was providing dangerously inaccurate medical information. An investigation highlighted that the feature, known as AI Overviews, was delivering misleading advice at the top of search pages, which could lead seriously ill individuals to incorrectly assume they are in good health. This action underscores the significant challenges and potential risks involved in deploying generative AI for sensitive topics like healthcare, where factual precision and proper context are critical for user safety.

The company specifically removed summaries for queries like “what is the normal range for liver blood tests” after medical experts reviewed the outputs and deemed them hazardous. A particularly alarming error involved pancreatic cancer, where the AI incorrectly advised patients to avoid high-fat foods. This recommendation directly contradicts established medical guidance, which emphasizes maintaining weight and calorie intake for patients fighting this disease. Despite these documented flaws, Google only deactivated the feature for the liver test inquiries, leaving other potentially problematic health answers still available to users.

The investigation found that searches for liver test norms produced raw data tables listing enzymes such as ALT, AST, and alkaline phosphatase. However, these tables came without the crucial context needed for proper interpretation. The AI system failed to adjust the provided figures for essential patient demographics, including age, sex, and ethnicity. Specialists warned that because the AI’s definition of “normal” often deviated from accepted clinical standards, individuals with serious liver conditions might receive false reassurance and consequently delay or forgo necessary medical follow-up.

Vanessa Hebditch, Director of Communications and Policy at the British Liver Trust, emphasized the complexity of interpreting liver function tests. She explained that these tests involve a collection of different blood analyses and that understanding the results requires far more than a simple numerical comparison. The AI Overviews failed to include vital warnings that a person can receive normal test results even while suffering from a serious liver disease, necessitating further professional evaluation. This omission of context creates a risk of false reassurance that could lead to severe harm for patients.

Google chose not to comment on the specific removals when contacted. A company spokesperson stated that Google is committed to investing in the quality of its AI Overviews, especially for health-related topics, and maintained that the vast majority of these summaries provide accurate information. The spokesperson added that an internal team of clinicians reviewed the cited examples and concluded that in many instances, the information was not inaccurate and was supported by citations from high-quality websites. This response highlights the ongoing tension between automated AI systems and the nuanced, individualized expertise required in medical practice.

(Source: Ars Technica)