Healthcare AI needs hard guardrails

A clinical decision support tool is supposed to work like a second set of eyes: it reads a patient’s chart, labs, and history, then helps a clinician decide what to do next. A new Journal of Medical Internet Research viewpoint says generative artificial intelligence could extend that help to patients and caregivers too, but only if the system is built with hard safety rules from the start. (jmir.org) Generative artificial intelligence is the kind that writes answers in plain language instead of just flagging a number in red. In health care, that means it could turn a dense discharge summary into instructions a family can actually follow, or help a patient prepare questions before an appointment. (jmir.org) The problem is that a fluent answer can sound correct even when it is wrong. The April 1, 2026 viewpoint says these tools need independent testing, periodic reassessment for performance drift, and clear policies for when generative artificial intelligence should and should not be used. (jmir.org) The paper lays out six concrete guardrails, and the first one starts before any model is deployed. Patients and caregivers need to be represented in design and development, because a tool built only for engineers or only for clinicians will miss the way real people read instructions, weigh tradeoffs, and make decisions at home. (jmir.org) Another guardrail is transparency and consent. The authors say health systems need policies that tell patients when generative artificial intelligence is being used, what role it played in the advice they received, and what data it relied on. (jmir.org) That lines up with how regulators already draw the boundary. The Food and Drug Administration says some clinical decision support software is excluded from device regulation only when clinicians can independently review the basis for the recommendation, which is another way of saying a black box is harder to trust in a clinic. (fda.gov) Trust is the bottleneck even before patient-facing tools become common. A 2025 systematic review in the same journal found 27 studies on health care workers’ trust in artificial intelligence clinical decision support systems, showing that adoption depends not just on accuracy but on whether staff understand the system and can fit it into practice. (jmir.org) Workflow is where a lot of health technology quietly fails. A 2025 interview study with patients, physicians, caregivers, developers, insurers, researchers, and regulators found that artificial intelligence decision support is still in its infancy in many care settings, and that integration barriers matter as much as the model itself. (jmir.org) So the real argument is not “put a chatbot in the hospital” or “ban it from care.” It is closer to “treat it like a junior assistant”: useful for drafting, translating, and organizing, but with named humans, escalation paths, and repeated checks whenever the output could change treatment, medication use, or follow-up. (jmir.org; fda.gov) If those guardrails are missing, generative artificial intelligence turns bedside advice into guesswork with a polished tone. If they are present, the technology starts to look less like an oracle and more like infrastructure: visible, limited, tested, and easier for patients and clinicians to question before they act. (jmir.org; jmir.org)