Season · AI, Trust & Human Oversight · 4 of 16
A clear answer can hide the trail a careful decision still needs.
Concept of the Week · Show the evidence path
A generated answer may sound institutional even when its supporting trail is hidden. Authority bias can lend it borrowed credibility, while confirmation bias can make a reviewer search for support after accepting familiar wording. A Provenance Trace keeps each consequential claim connected to its observations, authors, and time windows so checking does not become a separate investigation.
Trust rule: Make every consequential claim directly traceable to its supporting sources.
On a medical ward before the morning round, a bedside nurse reads an artificial intelligence (AI)-generated overnight summary stating that a patient's pain remained controlled. The wording sounds complete, but the claim does not show which observations, medication entries, or notes support it.
To verify the statement, she opens the flowsheet, medication administration record, and progress notes separately, then reconstructs the timeline. A later note records breakthrough pain, but it sits outside the simplified summary.
In a source-linked version, the claim opens with its supporting observations, authors, and time windows kept together. The contradictory note remains visible rather than disappearing behind the smoother interpretation. The nurse can correct the handoff in place without treating the model's phrasing as the clinical record.
Claim-level provenance changes what the nurse can verify, where contradictions appear, how corrections return to the shared record, and which transformations require traceability tests.
Test this in a usability study by asking representative reviewers to inspect ten consequential summary claims and checking whether they can reach each supporting source, author, and time window in one action while contradictory evidence remains visible.
The two views show the bedside nurse moving from an opaque summary with disconnected evidence to a source-linked claim with its author, time window, and contradictory evidence kept visible.
Evidence behind the claim
AI can reduce retrieval, comparison, and drafting work by assembling relevant records around a proposed summary. It does not replace clinical judgment, representative-user validation, risk review, or accountability for the handoff. Treat each generated claim as a hypothesis, preserve its sources and outliers, and check consequential interpretations against the clinical record and the people responsible for the decision.
Asset & Resource
Use the Provenance Trace Card before a generated summary enters a handoff, then review it whenever the source set, data structure, model, prompt, or workflow changes.
Decision and consequence: Name the claim, the decision it may influence, and the consequence of accepting it incorrectly.
Evidence and coverage: Link the supporting observations and show which relevant sources or time periods were not included.
Authority and control point: Record who can accept, revise, reject, or stop the claim from moving forward.
Failure and recovery: Preserve contradictions and define how corrections reach the shared record and downstream users.
Revalidation trigger: Name the changes that require the trace and its interaction to be tested again.
The FDA discussion paper on generative AI-enabled medical devices asks whether the risk framework should include traceability to primary source materials. The paper is exploratory rather than regulatory guidance, but its question is immediately useful for design: when an answer may shape clinical action, can the responsible person independently inspect what supports it?
Light Wisdom & Reflection
“A wise man, therefore, proportions his belief to the evidence.”
Which claim in your product feels trustworthy mainly because no one can see how it was assembled?
Mindful Practice
While preparing a familiar meal, notice one ingredient you reach for without looking closely.
What tells you it is the one you want?
Where did that certainty come from: the label, the place, or habit?
Next time, notice which small source turns assumption into recognition.
Best,
Andreas Walden
Share this with someone designing an answer people must verify.