AI Symptom Checkers Compared in Emergency Department Study
A randomized, head-to-head clinical study in an emergency department compared the diagnostic accuracy of two leading AI symptom checkers, Ada and Symptoma. The research evaluated how well each tool performed in a real-world, high-acuity setting. The findings highlight the importance of validating AI decision support tools before widespread clinical adoption.
- In the head-to-head comparison, Ada demonstrated higher diagnostic accuracy, providing the identical top diagnosis in 14% of cases and within the top 5 diagnoses in 27% of cases. Symptoma's performance was lower, identifying the identical top diagnosis in 4% of cases and within the top 5 in 13% of cases. - When considering both identical and plausible diagnoses, Ada provided a top-5 suggestion in 75% of patient cases, compared to 64% for Symptoma. This was based on a study of 450 adult patients presenting to the University Hospital Erlangen emergency department. - Regarding triage, Ada correctly triaged 34% of patients, but undertriaged 13% and overtriaged 53%. Both symptom checkers failed to suggest potentially life-threatening diagnoses in a notable number of cases: 13% for Ada and 14% for Symptoma. - The most frequent chief complaint from patients in the study was chest pain, accounting for 37% of cases. - User experience was also evaluated, with 88% of participants rating Ada as "very easy" or "easy to use," compared to 78% for Symptoma. - A key challenge in evaluating these AI tools is the methodology; studies using clinical vignettes often don't reflect real-world patient interactions where symptoms can be clarified. Researchers suggest that testing against real patient-physician interactions is necessary to truly validate performance. - The broader context for these tools is their potential to act as a "digital front door" for health systems, aiming to guide patients to the appropriate level of care and reduce the 30% of emergency department visits that are considered unnecessary. - From a health IT perspective, integrating such AI tools requires robust interoperability with existing EHRs, like Epic, using standards such as HL7 FHIR to ensure that patient-entered data can be seamlessly incorporated into the clinical workflow without creating data silos.
