Experts Urge Caution on AI in Healthcare
The integration of large language models in clinical environments requires critical oversight and transparency, AI ethicist Dr. Lisa Choi cautioned on *Freakonomics Radio*. While AI can augment diagnoses, challenges related to bias and patient data privacy remain unsolved, necessitating a "human in the loop" approach for safety.
The global AI in healthcare market is projected to reach $194.4 billion by 2030, a significant increase from $8.23 billion in 2020. This growth is mirrored in clinical practice, where physician adoption of AI tools jumped by 78% from 2023 to 2024, with nearly two-thirds of physicians now using AI in their work. The primary uses include documentation support for billing and medical charts, as well as the creation of discharge instructions and care plans. A significant driver of AI adoption is its potential to alleviate administrative burdens, which 57% of physicians identify as its greatest benefit. Beyond administrative tasks, AI is increasingly used in diagnostics. In a Swedish trial involving over 80,000 women, an AI-assisted approach to reading mammograms identified 20% more breast cancers and reduced the radiologists' workload by nearly half. Studies have shown AI's diagnostic accuracy to be comparable to human radiologists, with some research indicating slightly higher sensitivity in detecting abnormalities. However, the rapid integration of AI into clinical workflows has raised significant concerns about algorithmic bias. A widely used commercial algorithm in U.S. hospitals was found to be racially biased, underestimating the health needs of Black patients by using healthcare costs as a proxy for illness severity. This bias resulted in a more than 50% reduction in the number of Black patients identified for additional care. Similar biases have been identified in other areas of medicine. An algorithm used to predict the success of vaginal birth after a cesarean (VBAC) erroneously led to more unnecessary C-sections for minority patients. In dermatology, a review of 21 datasets used to train AI for skin cancer detection found a severe underrepresentation of darker skin tones, with only 11 out of over 100,000 images representing brown or black skin. In response to these challenges, regulatory bodies are beginning to establish frameworks for the safe and ethical use of AI in healthcare. In the U.S., the National Institute of Standards and Technology (NIST) released its AI Risk Management Framework in January 2023 to provide guidance on designing and deploying trustworthy AI. As of mid-2025, the FDA has cleared over 1,200 AI-enabled medical devices, with the vast majority being in radiology. Legislative efforts are also underway to address the use of AI in healthcare. The European Union's AI Act, which began to go into effect in February 2025, classifies most medical AI tools as "high-risk" and imposes strict requirements for risk management and human oversight. In the U.S., the HEALTH AI Act (H.R. 5045) was introduced in August 2025 to establish a grant program for research into the use of generative AI to reduce administrative burdens and address health disparities.
