Season · AI, Trust & Human Oversight · 6 of 16
An accept button offers a clear next step; a disagreement needs one too.
Concept of the Week · Make room for a different assessment
When a clinician doesn't agree with an artificial intelligence (AI) recommendation, keep both assessments visible before anyone acts on it. Automation bias can lead clinicians to trust AI too readily; confirmation bias can lead them to favour evidence that supports their own view. Saving the reasons behind both assessments helps the next reviewer understand the difference.
Decision rule: Keep unresolved disagreement visible, with both evidence trails and a named next reviewer.
On a hospital ward, a physician checks an AI dose recommendation before signing a medication order. A new kidney-function result has changed their assessment, but the AI recommendation still uses yesterday's result. The screen lets them accept the recommendation or type a reason for rejecting it, so they open the patient record again to explain the difference.
A calmer design would let them mark the recommendation as needing review and place their assessment beside it. Each would show the kidney result it used and when that result was recorded. The screen would keep the disagreement open and show the prescriber responsible for the decision, with pharmacist input where needed.
If the concern affects safety, the team would follow its established clinical escalation process, including urgent care when needed. At handoff, the next reviewer would see the concern, both sources, and the next action without searching separate notes.
Keeping “review required” separate from “order signed” helps the next clinician see whether the decision is still open.
Ship this by designing, evaluating, and refining the disagreement workflow with representative prescribers and pharmacists across realistic cases, including handoffs, until evidence comparison and review meet the agreed usability and safety criteria for release.
From a physician seeing an AI recommendation alone to comparing its evidence with a newer clinical result, while the decision remains open for review.
Check either side
AI can help gather records and show differences, but a clinician still needs to check the comparison against the original sources. A clear explanation can still use the wrong information. Include cases where the AI is wrong, the clinician is wrong, and the available evidence cannot settle the disagreement. Save the final decision and its reason alongside the original assessments.
Asset & Resource
The AI Disagreement Checklist gives clinicians five actions when their assessment conflicts with an AI recommendation.
Name the disagreement: State where your assessment differs from the AI recommendation and which decision needs review.
Compare the sources: Open the evidence behind both assessments; check its date and relevance to this patient.
Challenge either assessment: Look for evidence that could change your view or the AI recommendation, including findings neither assessment explains.
Record what remains open: Save the decision and its reasons, or state what the evidence cannot yet settle.
Assign the next review: If disagreement remains unresolved, name the next reviewer and action; keep the concern visible at handoff and follow the established escalation process when needed.
Design and human factors teams can place these prompts beside the disputed recommendation, with direct source access, a reason field, and a route to review. Include the prompts in the iterative evaluation above; revisit them when the model, interface, or workflow changes.
The Agency for Healthcare Research and Quality's Human–AI Interaction brief explains how automation bias and confirmation bias can affect clinical AI review. It informs these proposed prompts; it does not establish that the checklist reduces bias.
Light Wisdom & Reflection
“A fair result can be obtained only by fully stating and balancing the facts and arguments on both sides of each question […]”
Which do you check more closely: evidence that challenges your view or evidence that agrees with it?
Mindful Practice
Listen to a short passage of music with someone who hears it differently.
Which sound catches their attention?
What do you notice when you listen for it too?
Next time, listen for one detail before explaining what you hear.
Best,
Andreas Walden
Share this with someone designing how clinical teams question AI recommendations.