Season · AI, Trust & Human Oversight · 8 of 16
A review can be mandatory and still come too late.
Concept of the Week · A finished draft is not permission to send
Artificial intelligence (AI) can draft patient instructions, while a separate automated process delivers them. If delivery starts as soon as the text is ready, the patient may receive advice the doctor has not checked.
Review rule: Require the doctor's approval before the system sends the instructions to the patient.
An emergency physician prepares instructions explaining what a patient should do after going home. In a proposed automated workflow, AI drafts the text from the care plan, and a delivery service sends it to the patient's online portal. Connecting delivery directly to draft completion would let the instructions reach the patient while the doctor's review is still pending.
Instead, keep the completed draft waiting for the treating emergency physician to compare it with the care plan and correct anything wrong or unclear. Every delivery attempt should require approval of the version being sent, including automatic retries. If the doctor is interrupted, keep the draft unsent and visible in their unfinished work. If the doctor cannot finish the review, assign it to another authorized physician rather than leaving the patient waiting with no one responsible.
Making the delivery service wait for approval keeps the doctor’s review ahead of the message, even when sending happens automatically.
Ship this by verifying that every delivery path requires approval, then evaluating the review interface with representative emergency physicians in realistic tasks and iterating until defined usability and safety criteria are met.
From an emergency physician reviewing instructions that are already being sent to the patient to the same physician reviewing a draft that cannot be sent until they approve it.
Asset & Resource
Use Failure Mode and Effects Analysis (FMEA) to examine where an automated process could act before human review. Design, engineering, and human factors teams can work through these five fields with the people responsible for the task.
Step: Mark where the system sends information or carries out an action.
Failure: Trace how unfinished review, a handoff, or an automatic retry could let the process proceed without approval.
Effect: Identify who could be affected and what recovery would require.
Control: Specify who can approve the result, how every action route requires that approval, and who takes over unfinished work.
Action: Assign improvements, verify the software controls, and evaluate the review task with representative users; revise and retest.
These fields are a focused excerpt, not a complete FMEA. Add them to the team's existing process analysis and revisit them when the model, delivery process, or review responsibilities change.
The AHRQ FMEA guide describes the full method, including causes, existing controls, risk assessment, and follow-up actions.
Review quality matters too: a 2026 study of AI-generated discharge instructions found that physician edits improved accuracy but reduced measured readability. That finding supports checking the wording as well as its clinical meaning; it does not validate the automated delivery workflow proposed here.
Light Wisdom & Reflection
“Look before you leap.”
Where could your system move ahead while a required human review is still unfinished?
Mindful Practice
Before sending a personal message to someone you care about, read it once more.
What might they understand differently from what you intended?
What would you like to change while the message is still yours to edit?
Make that small change before tapping send.
Best,
Andreas Walden
Share this with someone designing what happens between an AI draft and its delivery.
Continue with CALM: Show When a Response Suppresses Alerts—on making an action's consequences visible before someone chooses it.