Report Details Hard Lessons for AI in Healthcare

- A significant real-world example of algorithmic bias occurred when a widely used U.S. health system algorithm was found to systematically disadvantage Black patients. The model used healthcare cost as a proxy for health needs, leading to healthier white patients being ranked equally with Black patients who had more chronic conditions. - "Agentic AI" refers to systems that can proactively and autonomously perform tasks and make decisions, moving beyond simple predictions to coordinate complex workflows like patient intake or claims processing. The key design shift is from AI advising humans to AI carrying out actions under human supervision. - Data provenance, or the ability to trace the origin and lifecycle of data, is critical for building trust and ensuring auditability in healthcare AI systems. Regulations like the EU's GDPR mandate that computer-assisted decisions be explainable, which tackles the "black box" problem and engenders trust. - Automation bias is the human tendency to over-trust outputs from automated systems, even when they contradict a clinician's own judgment. This can lead to errors of omission (not taking a necessary action) or commission (taking an incorrect action) and is a growing area of focus for malpractice liability. - The concept of "clinician-in-the-loop" is a core principle for mitigating AI risks, ensuring that human expertise is integrated for validation, correction, and oversight. However, some experts now argue for a "human-on-the-loop" model, where clinicians provide oversight from a distance rather than controlling every final decision to avoid cognitive deskilling and inefficiency. - To ensure safety and efficacy, the U.S. Food and Drug Administration (FDA) regulates AI and machine learning-enabled software as medical devices, requiring robust documentation and validation. - A major challenge in deploying clinical AI is "performance drift," where a model's accuracy degrades over time as it encounters new patient populations, diseases, or even different imaging equipment. This necessitates continuous monitoring and retraining of models to maintain their reliability. - Ambient listening technology, which uses AI to automatically document clinical conversations, has seen rapid adoption, with about one-third of large health systems using it in 2025. The primary benefits are reduced clinician burnout and increased productivity.