Human Oversight of Clinical AI: The Error-Interception Test

“Human in the loop” is often treated as the safety answer for clinical AI: the software recommends, and a clinician retains the final decision. But the handoff is protective only when the clinician notices a wrong answer, understands why it is wrong, and feels able to override it. A new npj Digital Medicine systematic review and meta-analysis examines what happens when automated clinical advice is erroneous.

Across four heterogeneous real-world studies of computerized ECG interpretation, the pooled estimate was that 26.6% of erroneous interpretations were not corrected or recognized. The uncertainty around that estimate was large (95% confidence interval 6.4–65.6%), and results varied substantially between studies. It is a warning signal, not a universal failure rate for today's clinical AI.

Another analysis pooled three adjusted cohorts and found clinicians had approximately 10 times greater odds of being correct when automated advice was correct than when it was incorrect (odds ratio 10.19; 95% confidence interval 7.43–13.98). This pattern captures the central concern of automation bias: assistance can improve decisions when it is right, while also making wrong advice harder to resist.

My takeaway

Human oversight should be designed and tested as an active clinical task. Can users detect errors in realistic cases? Do they challenge a confident but incorrect recommendation? Can they find the underlying evidence, override the system, and escalate safely? These are measurable capabilities. Merely assigning a physician to “supervise” a system tells us little about whether mistakes will actually be intercepted.

That changes what health systems should ask of a deployment. Evaluation should include deliberately erroneous outputs, time pressure, uncertainty and the context in which clinicians will use the tool. Training should help users calibrate trust rather than simply encourage adoption. Post-deployment monitoring should capture missed errors and overrides, then feed those lessons back into the workflow and the model.

The evidence has limits. The review brings together older computerized decision support and newer AI studies with differing technologies, tasks and settings. Contemporary AI and large-language-model studies showed harmful reliance but were too heterogeneous to pool. These results do not establish the safety performance of any particular modern system. They do, however, show why human supervision cannot be assumed to close a safety gap by itself.

For clinical AI governance, the next question is practical: not only who has final authority, but whether that person can reliably exercise it when the machine is wrong.