Season · AI, Trust & Human Oversight · 11 of 16
A system can be ready for an update before the people using it are.
Concept of the Week · Prepare the people, too
Preparing people for an AI update means explaining what changes in their work and giving them a chance to practice and ask questions. Manufacturer teams, local technical teams, and the people using the system need to agree on who explains the change and who handles concerns afterward. This helps people judge what they can rely on, rather than discovering the differences while doing the work.
Trust rule: Before introducing an AI update, explain what changes in people’s work, let them try the changed workflow, and name who will help afterward.
On a general inpatient ward, a nurse uses an AI-based risk prediction tool to help identify patients who may need review for possible deterioration. An advisory appears when the estimate from patient observations and results crosses a review threshold; a planned software update lowers that threshold while keeping the screen familiar.
More patients could now trigger chart checks and bedside reviews, competing with other nursing tasks. Without an explanation, the nurse has to work out why familiar alerts are asking for more review.
Before rollout, the manufacturer's service team, the hospital's biomedical engineering team, and clinical representatives could agree on what changes to make, who will implement them, and what ward staff need to know. A handover with practice cases could let nurses ask questions, practice the revised response, and confirm that normal clinical assessment and escalation remain clear. After introduction, those teams could review staff feedback and alert patterns together, with a named contact for unexpected behavior. When a nurse reviews an advisory, the interface could explain the change and show its supporting observations, making clear that it prompts review rather than diagnosis or treatment.
Service design shapes how these people and steps work together: who informs ward staff, who prepares the handover, and who responds to questions after the introduction. The on-screen explanation is one part of that support.
When an update changes which patients trigger an advisory, teams must evaluate whether nurses understand it, can complete the follow-up, and know where to get help.
Ship this by defining responsibilities and acceptance criteria, developing the revised workflow and handover, evaluating them with ward nurses through realistic tasks, revising and retesting, then introducing the update with support and follow-up once the required performance, usability, and safety criteria are met.
From a ward nurse encountering an AI update alone to planning its introduction together with manufacturer service and hospital biomedical engineering.
Asset & Resource
Service design, design, and human factors teams can use this AI Update Review with manufacturer service, hospital technical, and clinical colleagues to plan how an update will be communicated, introduced, and supported.
Share what changes: Explain the affected task, the reason for the update, its limits, and what the evidence supports.
Agree on responsibilities: Identify who prepares, introduces, explains, and supports the update across the teams involved.
Prepare people: Plan the handover, practice cases, questions, and the interface explanation before introduction.
Check understanding: Evaluate the changed work and communication with representative users, revise based on findings, and retest.
Stay in touch: Identify a support contact and agree on how staff feedback and monitoring results will be reviewed afterward.
Map one update from the first notice to the follow-up after the introduction, with a row for each role. Record what people need to know, who provides it, and where questions go unanswered. This applies the service blueprint approach, which connects people, processes, and service encounters. Evaluate and revise the proposed support; a completed map does not establish readiness.
The U.S. Food and Drug Administration (FDA) provides a framework for planned changes to AI devices in its guidance on predetermined change control plans. Before making the planned changes, manufacturers describe what will change, how they will verify and validate it, the acceptance criteria, and the expected benefits and risks.
The guidance also covers how updates are implemented, how changes are communicated to users, appropriate training, and post-introduction monitoring.
For changes covered by an FDA-authorized plan, this can avoid a separate submission for each update. The planned checks still have to be completed, and any reduced testing needs justification. The AI Update Review helps a design team prepare its questions; it is not the regulatory plan.
Light Wisdom & Reflection
“Truth HAPPENS to an idea.”
Who needs to understand the update before it reaches them, and how will your team learn what remains unclear?
Mindful Practice
Before changing a small shared plan, such as the time or place of a meal with a friend, pause before sending the message.
What does the other person still expect?
What do they need to know before the change reaches them?
Explain the change clearly and leave room for a question.
Best,
Andreas Walden
Share this with someone helping people prepare for a system update.
Continue with CALM: Keep People Able to Work Independently.