The “Official” Trap: Why AI is More Likely to Spread Medical Falsehoods When They Look Professional

Artificial intelligence is increasingly finding a home in our clinics and on our smartphones, promising to revolutionize how we manage our health. However, a landmark study published in The Lancet Digital Health in January 2026 warns of a sophisticated new vulnerability: AI systems are significantly more likely to repeat dangerous medical misinformation when it is dressed up in professional, clinical language rather than casual social media chatter.

Researchers from the Icahn School of Medicine at Mount Sinai in New York conducted a massive “stress test” on 20 of the world’s most widely used large language models (LLMs). They discovered that when fabricated medical advice was embedded into realistic-looking hospital discharge notes, the AI models accepted and repeated the lies nearly half the time. This finding raises urgent questions for a healthcare industry racing to integrate AI into patient portals, chatbots, and diagnostic tools.

When “Professional” Becomes a Cloak for Error

The Mount Sinai team, led by Dr. Eyal Klang, Chief of Generative AI at Mount Sinai, didn’t just ask AI models simple medical questions. Instead, they designed a cross-sectional benchmarking analysis using more than 1 million prompts to see how AI handles misinformation in context.

The researchers fed the models three types of content:

1. Real hospital discharge summaries with one fabricated, dangerous recommendation added.

2. Common health myths gathered from social media platforms like Reddit.

3. Clinical vignettes written by physicians, some containing logical fallacies.

The results revealed a startling “authority bias” within the software. While the AI models repeated misinformation from social media posts only about 9% of the time, that error rate skyrocketed to 47% when the same false information was presented within a formal hospital discharge note.

Essentially, the more “official” the misinformation looked, the more the AI believed it.

“A fabricated recommendation in a discharge note can slip through because these systems tend to treat confident, fluent medical language as correct by default,” explained Dr. Klang. “From the model’s perspective, how something is written often matters more than whether it is true.”

A High Stakes “Safety Gap”

The study also highlighted a wide gap in performance between different AI products. OpenAI’s GPT family of models proved to be among the most resilient against false claims. However, other unnamed models tested in the study accepted and propagated fabricated statements as much as 63.6% of the time.

Dr. Girish Nadkarni, Chief AI Officer of Mount Sinai Health System and co-senior author of the study, noted that the models were noticeably more skeptical of information that looked like a Reddit post. This suggests that while AI developers have been successful at training models to doubt “internet rumors,” they have not yet taught them to apply that same level of scrutiny to clinical documentation.

This is particularly concerning as hospitals begin to use AI to summarize patient records or generate follow-up instructions. If a model encounters a previous error in a patient’s history and accepts it as “clinical truth,” it could perpetuate a cycle of incorrect treatment.

The Illusion of Accuracy

This research arrives alongside other studies suggesting that AI’s ability to “talk the talk” does not always mean it can “walk the walk.” A recent study in Nature Medicine led by the University of Oxford’s Internet Institute found that while AI models could correctly identify a medical condition in 95% of test scenarios, they recommended the correct action—such as seeking urgent care—only about half the time.

Furthermore, when real people used these tools, they fared no better than those using a standard Google search. The “persuasive” nature of AI—its ability to provide clear, confident-sounding answers—can give users a false sense of security, leading them to delay necessary medical care or follow incorrect advice.

Navigating the AI Health Era: Tips for Patients

For the general public, the takeaway isn’t that AI is “bad,” but rather that it is currently a “hallucinating expert.” It can provide helpful definitions or help you organize your thoughts for a doctor’s visit, but it should never have the final word.

• Treat AI as a Librarian, Not a Doctor: Use AI to find information, but rely on your healthcare provider for the final interpretation and treatment plan.

• Verify the Source: If an AI summarizes a “hospital note” or “specialist letter” for you, ask to see the original document. Do not assume the AI’s summary is a 1:1 reflection of the facts.

• Cross-Check Recommendations: Always compare AI-generated advice against reputable sources like the CDC, Mayo Clinic, or the NHS.

• The “Vibe” Check: Just because an answer sounds professional and uses complex terminology does not mean it is accurate.

Moving Forward: Stress-Testing the Future

Dr. Mahmud Omar, the study’s first author, argues that the healthcare industry needs a new way to measure AI safety. Instead of just testing if an AI can pass a medical licensing exam, developers must measure “how often it passes on a lie.”

He and other experts are calling for “misinformation stress tests” before any AI tool is deployed in a hospital. This would involve intentionally feeding the AI “trap” documents—like the ones used in the Mount Sinai study—to ensure the system can flag inconsistencies rather than mindlessly repeating them.

“The solution isn’t to abandon AI in medicine,” the researchers noted, “but to engineer tools that can spot dubious input, respond with caution, and ensure human oversight remains central.”

Medical Disclaimer

This article is for informational purposes only and should not be considered medical advice. Always consult with qualified healthcare professionals before making any health-related decisions or changes to your treatment plan. The information presented here is based on current research and expert opinions, which may evolve as new evidence emerges.

References

https://health.economictimes.indiatimes.com/news/industry/medical-misinformation-more-likely-to-fool-ai-if-source-appears-legitimate-study-shows/128135414?utm_source=latest_news&utm_medium=homepage