Study Finds AI Chatbots Unreliable for Health Advice

- A study in *Nature Medicine* highlighted a significant gap between diagnostic and action accuracy; while chatbots identified potential conditions in 94.9% of test cases, they only recommended the correct course of action 56.3% of the time. - The unreliability often stems from the "black box" nature of large language models (LLMs), which generate statistically probable responses rather than demonstrating true clinical reasoning, leading to risks of misinformation and automation bias. - In contrast, a separate *JAMA Internal Medicine* study found that a panel of healthcare professionals preferred chatbot responses to physician responses 79% of the time, rating them as higher quality and more empathetic for answering patient questions. - Large-scale EHR vendors are focusing on more controlled applications; Epic Systems, for instance, is developing over 100 AI features designed to augment clinical workflows, such as summarizing patient data from its Cosmos database or drafting responses for clinicians to review. - In critical care, AI-driven clinical decision support (AI-CDS) is being leveraged for predictive analytics to identify early indicators of conditions like sepsis and to help optimize ventilator settings and medication dosing. - The U.S. Food and Drug Administration (FDA) is actively developing a regulatory framework for these tools, classifying many as Software as a Medical Device (SaMD) and outlining action plans to ensure their safety and effectiveness throughout their lifecycle. - A key failure mode is the inability of LLMs to handle unstructured, real-world clinical data, with performance dropping significantly when moving from curated case summaries to original medical reports. - The risk of "hallucination," where the AI confidently provides incorrect information, remains a primary safety concern; one analysis by the ECRI Institute found a chatbot incorrectly approved the placement of an electrosurgical device in a way that would risk patient burns.