Season · AI, Trust & Human Oversight · 5 of 16
A clear “Risk recommendation unavailable” can be more useful than a confident category built on incomplete evidence.
Concept of the Week · A useful system knows when to stop
A hallucination invents or overstates what the evidence supports, but even a factually grounded artificial intelligence (AI) system can be too uncertain to answer. The overconfidence effect can make a weak recommendation appear more capable than it is, while satisficing can turn the first plausible answer into the end of the search. Test when the system must stop, show what is missing, and hand the decision to a named reviewer.
Trust rule: Decline to answer when required evidence is missing.
In the emergency department, a nurse conducting the initial triage reviews an AI-generated risk recommendation for a patient with chest discomfort. The current electrocardiogram trace is too noisy for reliable interpretation, the latest blood-pressure reading has not transferred from the vital-signs monitor, and the patient's previous electrocardiogram cannot be retrieved for comparison.
The system still returns a definitive risk category, and the interface presents it without showing that the required evidence is incomplete. Unless the nurse already knows which inputs the system requires and notices the gap herself, nothing prompts her to question the result. A confident-looking category can therefore be accepted before its evidential basis is checked, making satisficing on the first plausible answer more likely.
A calmer interface withholds the category, names the unavailable inputs, and keeps the current patient state visible. It routes review to the responsible clinician without converting lack of evidence into false reassurance. The nurse can see why the system stopped, which evidence could make a recommendation possible, and that the patient now requires assessment by the emergency physician.
A clear no-recommendation state changes what the nurse recognizes as usable evidence, which checks remain open, when the emergency physician must assess the patient, and how the workflow continues when better data arrives.
Ship this by implementing a no-recommendation state that names the unavailable evidence, routes the case to the emergency physician, and preserves the patient context—then test the handoff with nurses conducting the initial triage and the physicians receiving the case before release.
When stopping is the safer action
AI can reduce retrieval, comparison, and signal-detection work before a review. It does not replace clinical judgment, representative-user testing, risk review, or the accountable decision about what happens when evidence is insufficient. Treat the stopping rule as a hypothesis: preserve the sources and unresolved inputs, then test the message, handoff, and return path with the people responsible for the work.
Asset & Resource
Use the No-Recommendation and Handoff Review Card before releasing a risk recommendation; repeat it after any change to the inputs, handoff, or return path.
Output: Name the risk category and the clinical action it could influence.
Evidence: Set the required electrocardiogram quality, blood-pressure freshness, and record availability.
Stop: Hide the category when a required input is noisy, stale, missing, or conflicting.
Handoff: Send the available evidence and unresolved gaps to the emergency physician, and confirm receipt to the nurse.
Return: Reopen the recommendation only when the required evidence arrives; retest after a model, input, interface, or workflow change.
The AAAI paper Role of Human-AI Interaction in Selective Prediction is useful because it shows that the way a system communicates that it has withheld a prediction can change human judgment. The message and handoff therefore need to be tested with people, not only as a rule inside the model.
Light Wisdom & Reflection
Whereof one cannot speak, thereof one must be silent.
Where does your system keep speaking after its evidence runs out?
Mindful Practice
While choosing a route for an evening walk, pause when the map cannot show a closed path or temporary obstacle.
What does the map know, and what remains outside its view?
What would help you choose without pretending the uncertainty is gone?
Next time a route looks definite, notice the moment when a quiet “not enough information” would be more useful.
Best,
Andreas Walden
Share this with someone designing the moment when a system should stop.
Previous in the season: CALM: Show What the Answer Relied On