Most high-stakes AI products in legal, healthcare, and criminal justice are built around one quietly broken contract: the AI decides, the human signs off. That is not decision support. A 2023 JAMA study of 457 clinicians across 13 states found that biased AI predictions reduced diagnostic accuracy, and that showing clinicians visual explanations of the AI's reasoning did not protect them from agreeing with wrong answers. A separate 2023 PLOS ONE study found something worse: adding a human reviewer increased AI recommendation uptake, because the human presence signaled the decision had already been vetted. The human was not catching errors. The human was providing cover.
The failure mode has a name and a documented mechanism. Kate Goddard's 2012 clinical decision support research identified automation bias, the tendency to accept system output rather than interrogate it, and it has replicated across domains for over a decade. Harvard researchers Zana Bucinca and Krzysztof Gajos found the one design intervention that worked: forcing users to form their own judgment before seeing the AI's answer reduced blind compliance. Participants hated those interfaces and rated them worse even as they made better decisions with them. The real-world cost of ignoring this is not abstract. Attorney Steven Schwartz was sanctioned $5,000 for submitting ChatGPT-fabricated case citations in 2023. By late 2025, researcher Damien Charlotin had tracked 1,356 documented hallucination incidents in legal filings. IBM Watson for Oncology consumed $62 million at MD Anderson before auditors found it was running on outdated protocols. Epic's sepsis tool, deployed across hundreds of hospitals, missed 67% of sepsis cases at its recommended threshold.
The article is worth reading in full for how it maps the exact point where design decisions transfer accountability without transferring judgment, and what building around the human's actual evaluative capacity would require instead. The question it forces is not whether to include a human in the loop, but whether the product is designed to make that human's presence real.
[READ ORIGINAL →]