Parabolabam cites safety jump to 94.1%
- A new arXiv paper from researchers at FAU Erlangen-Nürnberg and RWTH Aachen says clinical LLM safety improved sharply when models got clinician-curated “clean evidence.” - Across 34 locally deployed models, mean accuracy rose from 73.5% to 94.1%, while dangerous overconfidence fell from 8.0% to 1.6%. - The bigger point is that safer medical AI may depend more on evidence quality than on bigger models or fancier retrieval.
Clinical LLMs keep running into the same problem: a model can look smart on average and still make a few very dangerous mistakes. In medicine, those few mistakes matter more than a nice benchmark score. That is the gap this new paper is trying to close. And the headline result is blunt — giving models better evidence helped a lot more than just giving them more context, more retrieval machinery, or more inference-time compute. ### What actually came out? The paper is called *Safety and accuracy follow different scaling laws in clinical large language models*. It was posted to arXiv on May 5, 2026 by researchers from Friedrich-Alexander-Universität Erlangen-Nürnberg, University Hospital Erlangen, RWTH Aachen, and collaborators. The team built a framework called SaFE-Scale and a radiology benchmark called RadSaFE-200 to determine whether a wrong answer was high-risk, unsafe, or flatly contradicted the evidence. ### What is “clean evidence” here? Basically, it means the model was given clinician-defined supporting evidence that cleanly matched the question instead of being left closed-book or fed a noisier retrieval pipeline. That matters because medical failures are often not random hallucinations — they come from mixing weak evidence, irrelevant context, and confident wording. The benchmark was designed to separate those failure modes out. ### How big was the jump? It was big enough to be the whole story. Across 34 locally deployed LLMs and six deployment conditions, the clean-evidence setup raised mean accuracy from 73.5% to 94.1%. High-risk error dropped from 12.0% to 2.6%. Evidence contradiction fell from 12.7% to 2.3%. Dangerous overconfidence dropped from 8.0% to 1.6%. ### Why a wrong answer that sounds tentative is one kind of problem, but a wrong answer that sounds certain is much worse in a clinical workflow. That is the version more likely to slip past a rushed human reviewer. So the 8.0% to 1.6% drop is not just a calibration footnote — it suggests the system became less likely to be confidently unsafe when it had to handle distributional shift if you care about deployment, not just leaderboard performance. ### Didn’t RAG solve this already? Not in this paper. Standard RAG and agentic RAG did not reproduce the same safety profile. Agentic RAG beat standard RAG on accuracy and reduced contradiction, but high-risk errors and dangerous overconfidence stayed elevated relative to the clean-evidence
