Some doctors may give an agent another chance when the failure is understandable, its consequences are addressed and the affected workflow can be made dependable. Others may reasonably suspend it. For Heidi Work, the useful objective would be calibrated reliance, not persuading clinicians to maintain trust regardless of what happened.
This is a hypothetical trust-and-recovery analysis, not a report of an incident involving Heidi. It assumes substantial authorised task execution, consistent with the direction described in Heidi's September 2026 strategy statement, checked on 28 September 2026. The preview excluded the UK and Europe when checked on that date.
An appropriate refusal is not the same as an unauthorised action
Different events should produce different questions. An agent may refuse a task because required information is missing. It may make a recoverable administrative mistake within an authorised workflow. It may act beyond the authority it was given. Or it may contribute to a clinically consequential error.
Calling all of these events an "AI failure" conceals what the clinician needs to understand. A refusal may demonstrate that a limit is functioning. A wrong appointment location may expose a context or communication problem. Sending material that was meant only to be drafted would raise a different concern about control over external action.
A clinically consequential event requires attention to patient impact and appropriate clinical and organisational responses, not merely a product-usability discussion. The taxonomy is useful precisely because it avoids treating every problem as equivalent or assuming that every problem can be resolved by a more reassuring interface.
An overall error count, without task definitions and consequences, would not establish how reasonable it is to resume any particular workflow.
A fictional appointment error reveals the recovery problem
Imagine an agent authorised to confirm an appointment after the relevant arrangement has been agreed. In this fictional exercise, it sends the correct date but the wrong clinic location. The patient reads the message before the discrepancy is discovered.
The immediate product question is not whether the agent apologises convincingly. The team needs to establish what was sent, where the incorrect information came from, whether the appointment record itself is wrong and whether any subsequent actions depend on that information.
The practice also needs to know what the patient has received and what remains to be corrected. Updating an internal record alone would not resolve a misleading message already read by the recipient. Likewise, sending another message without acknowledging the conflict could leave the patient uncertain about which instruction to follow.
This exercise deliberately avoids assuming a clinical injury or attributing a real incident to Heidi. Its purpose is to show how a small administrative discrepancy can create several distinct recovery tasks that a generic "resolved" status might obscure.
An intelligible account is more useful than reassurance
A proposed recovery view should distinguish the original instruction, the information used, the action taken and the work still outstanding. It should also show whether other tasks continued after the problem arose.
The clinician should not have to infer the state of the workflow from a confident natural-language summary alone. Where available, relevant records of transmission, destination-system changes and subsequent corrections would help establish what actually happened. An agent's statement that it completed an action is not independent confirmation that the external system reached the intended state.
The explanation should be proportionate. A recoverable formatting issue does not require the same investigation as an action beyond authority. But the system should avoid using a familiar minor-error template for a problem whose consequences are not yet understood.
Uncertainty belongs in that account. "We do not yet know whether the recipient acted on the message" may be more useful than a premature declaration that no harm occurred. The aim is to support the next appropriate decision, not to minimise the apparent seriousness of the event.
Trust can become narrower instead of disappearing entirely
A clinician need not make a single permanent judgement about every feature of a product. They might continue delegating one well-understood administrative workflow while suspending another whose failure remains unexplained.
That narrower response could be appropriate when the problem is genuinely contained. It would not be enough when the event suggests a shared issue affecting multiple workflows, such as unreliable control over external actions. The decision depends on what is known about the failure, not on a desire to preserve adoption figures.
A useful recovery discussion would ask which assumption has been challenged. Was the wrong context supplied? Did the agent misinterpret a valid instruction? Was the permitted scope unclear? Did the destination system behave unexpectedly? Different answers imply different changes and different reasons for confidence or caution.
This is a more precise model than treating trust as a general emotional score. A person can be appropriately confident about one task and appropriately unwilling to delegate another.
What algorithm-aversion research does not prove
In research first published online in November 2016 and appearing in Management Science's March 2018 issue, Dietvorst, Simmons and Massey found that allowing participants some control over an imperfect algorithm's forecasts increased willingness to use it. Even limited modification could influence participants' choices in those experiments.
The research involved non-clinical forecasting, not autonomous healthcare workflows. It does not establish what degree of control would be adequate for clinical use, nor does it show that rejecting an unsafe system is irrational. Its relevance is narrower: the ability to influence or correct an automated process can affect people's willingness to rely on it.
For Heidi Work, the question would be whether control corresponds to the action's consequences. Editing a draft before transmission is different from correcting a communication after a patient has already received it. A nominal override button cannot make every external action reversible.
Recovery is part of the workflow, not an afterthought
A credible proposed recovery process would support stopping further action, identifying affected work, correcting what can be corrected and escalating what needs human attention. It would make clear when an action cannot simply be undone.
In the fictional appointment example, the record, the patient's understanding and any downstream arrangements need separate consideration. The practice should be able to establish whether the correction has reached the relevant destination rather than assume that generating a new message completes recovery.
Before resuming the affected workflow, the team would need a reason to believe that the particular problem has been addressed. That might involve correcting a local configuration, clarifying a boundary or testing the revised process in a controlled setting. These are possible responses, not a universal checklist or evidence that any one step guarantees safety.
A second chance should therefore be attached to a defined workflow and a stated basis for resuming it. "The next task worked" is much weaker evidence than understanding the failure and checking the relevant change.
Study what people do after an incident
An adoption study could follow whether clinicians stop delegating, restrict the task types they use, increase relevant oversight or return to their previous pattern. It should also examine whether they continue unchanged despite information that reasonably calls the workflow into question.
Neither maximum retention nor maximum confidence is the right objective. Withdrawal may be appropriate. More checking may be useful or may become unnecessary duplication. A narrower task set may represent improved calibration rather than a disappointing reduction in use.
The assessment should connect behaviour to the event and its resolution. A clinician who resumes because they understand a verified correction is in a different position from one who resumes because they feel pressured to use the purchased system. Interviews can help distinguish those explanations, while work records show what actually changed.
For a bounded administrative error with a clear correction, cautious resumption may be reasonable within the agreed process. For an unexplained action beyond authority or a clinically consequential concern, suspension and wider investigation may be more appropriate. The valuable outcome is not a second chance at any cost. It is a decision whose basis the clinician and organisation can explain.
Frequently asked questions
Does this article describe a real Heidi Work error?
No, the appointment example and recovery discussion are hypothetical. They examine how a clinician might respond to different kinds of failure without alleging that Heidi experienced the incident described.
Is a clinician irrational for abandoning an AI tool after an error?
Not necessarily, because the significance of the failure, its consequences and the evidence supporting recovery matter. Non-clinical algorithm-aversion experiments do not establish that continued use is appropriate in healthcare.
What would make a second chance more defensible?
An understandable account of the event, attention to its consequences and evidence that the relevant workflow problem has been addressed would provide a stronger basis for resumption. The decision should remain specific to the affected task and its controls rather than become blanket trust in the product.
Explore iatroX insights on clinical AI reliability and human judgement →