Season · AI, Trust & Human Oversight · 2 of 16
An uncertainty communication test reveals the gap between how precise a model looks and how much its evidence supports.
Concept of the Week · Precision is not certainty
A precise risk score can look more certain than its evidence. The framing effect makes the number feel like the whole story, while the overconfidence effect makes the next check feel less necessary. Showing uncertainty, input quality, and change over time lets clinicians weigh the signal instead of merely accepting it.
Decision rule: Let the certainty of the display match the certainty of the evidence.
An intensive care unit (ICU) nurse sees a patient's blood pressure drift down while the oxygen requirement rises. The deterioration dashboard moves the patient into a high-risk category and displays one precise score.
It does not show that the latest laboratory result is six hours old, one respiratory value is missing, or the uncertainty range crosses the escalation threshold. To judge how much weight to give the signal, the nurse opens the chart, checks timestamps, and compares the score with bedside trends. The model has reduced several inputs to one number but left the verification work outside the display.
A calmer design does not add another warning; it keeps the estimate, uncertainty, input quality, and change over time together, with the source values available. The nurse can see the patient's deterioration and the model's incomplete evidence without confusing one for the other. Clinical judgment remains in charge; the score supports prioritization without pretending to settle the decision.
Keeping uncertainty visible helps ICU nurses judge what the score supports, what still needs checking, and when to escalate concern about the patient's changing condition.
Ship this by testing stable, threshold-crossing, and data-poor ICU cases, then recording whether nurses can state what each score supports and what they would check next.
The three panels compare alternative ICU interface designs: a precise score leaves the nurse uncertain, fragmented evidence increases verification effort, and visible uncertainty supports a calmer judgment.
Uncertainty as a control point
Artificial intelligence (AI) can retrieve source observations, compare recent values, and surface stale or missing inputs. It cannot invent trustworthy confidence or replace calibrated evidence, clinical judgment, representative-user testing, risk review, or an accountable decision. Treat its output as a hypothesis: preserve sources and outliers, then check important interpretations with clinicians, session recordings, and representative cases.
Asset & Resource
Use the Uncertainty Decision Record before a score enters a consequential workflow, and revisit it when the model version, calibration, input coverage, threshold, or user interpretation changes.
Decision and consequence: Name the decision the estimate informs, who could be affected, and what happens if the signal is wrong.
Evidence and uncertainty: Record the estimate, uncertainty measure, calibration population, source inputs, freshness, missing values, and known coverage limits.
Authority and control point: State who may accept, challenge, defer, or stop the recommendation, when review occurs, and what evidence each action requires.
Failure and recovery: Define what happens when evidence is weak, the reviewer disagrees, or the estimate later proves wrong, including the correction path.
Revalidation trigger: Recheck after a model, data, threshold, workflow, or population change, or when observed interpretation departs from the intended use.
Uncertainty of risk estimates from clinical prediction models explains why a point estimate can overstate what the available data support. It also makes the practical boundary clear: uncertainty communication must be tailored to the decision and tested with the people expected to use it.
Light Wisdom & Reflection
“Ignorance more frequently begets confidence than does knowledge.”
Charles Darwin
Where does your product make a precise number easier to see than the reasons it might be wrong?
Mindful Practice
Think of the last forecast, arrival time, or delivery window you checked.
What did the estimate help you decide?
What remained uncertain, even though the number looked precise?
Next time, notice whether your confidence comes from the evidence or from the way the estimate is displayed.
Best,
Andreas Walden
Share this with someone designing a consequential score.
Previous in the season: CALM: Make Verification Easier Than Acceptance