An approval click does not establish independent clinical judgement. It shows that a user authorised something, but not whether they understood its assumptions or recognised an important omission. For Heidi II, meaningful oversight must therefore be assessed through the clinician's ability to question and change the proposed work, not merely the presence of a review step.
The distinction is especially relevant after Heidi's September 2026 launch. Its 29 September trust article describes Manual and Auto modes and explicitly recognises that repetitive approvals can become reflexive. That is a constructive design acknowledgement. Whether the implemented workflow preserves effective challenge remains a question for evaluation.
What a signature can and cannot show
Approval can represent several different activities. A clinician might independently assess the patient's situation and then authorise a matching task. They might recognise familiar wording without checking the source. They might confirm the recipient while overlooking why the action was proposed. Each produces a similar visible event in an activity log.
An audit trail should therefore not be treated as a direct measure of understanding. It is useful evidence of what the software recorded, including who authorised an action and when. It does not reveal whether the reviewer had access to the contradictory information that would have changed their decision.
The relevant question is not whether doctors should be trusted. It is whether the workflow gives a responsible doctor a realistic opportunity to perform the judgement being attributed to them.
What misleading AI advice has done in a different setting
A randomised clinical-vignette study published in JAMA on 19 December 2023 involved 457 clinicians. Systematically biased AI predictions reduced diagnostic accuracy by 11.3 percentage points compared with baseline when provided without explanations. Image-based explanations did not significantly mitigate the adverse effect of the biased predictions. Standard, non-biased AI support improved performance in the study. These were simulated diagnostic tasks, not Heidi II agents, live patient outcomes or a test of its approval interface. The relevant source is the original JAMA study.
The finding supports a narrower conclusion than "clinicians cannot supervise AI". It shows that assistance can influence decisions in the wrong direction, and that adding an explanation is not automatically sufficient protection. The study does not establish how frequently Heidi II will make errors or how its users will respond.
For an agentic workflow, this suggests a testable design question: does the review stage help clinicians identify a consequential mismatch, or mainly make the proposed action easier to accept?
A fictional approval that looks reasonable
Imagine a clinician reviewing a prepared request for information from a specialist service. The correspondence is clear, the destination is correct and the patient's identity matches. However, the task assumes that the specialist review is still outstanding. A newly received letter states that it has already happened and that the current unresolved issue is different.
A reviewer who sees only the polished request may authorise unnecessary work. A reviewer shown the conflicting documents and their dates can make a different decision. The visible action, clicking approve, is identical until the surrounding information changes.
This is not a report of a Heidi incident. It is a proposed test case for any system that prepares work from changing clinical records. The failure mechanism is incomplete situational understanding, not necessarily an invented fact or an obviously poor answer.
Build the review around consequential assumptions
A useful review view should make the proposed action understandable without requiring the clinician to reconstruct every technical step. We would test a presentation that shows the material facts, their sources, unresolved uncertainty, intended destination and consequence of approval.
The reviewer should be able to distinguish an instruction they supplied from an inference the system made. A statement such as "request the report" carries a different status when explicitly directed by the clinician than when inferred from an old note mentioning a planned appointment.
The interface should also make disagreement operationally meaningful. Editing the final text is insufficient if a related task continues using the rejected assumption. A meaningful rejection should change the relevant pending work, or make clear which parts still require attention. This is a proposed requirement, not a claim about a feature verified in Heidi II.
Why explanations can become another layer of persuasion
A plausible explanation can help a reviewer understand a proposal. It can also repeat the same incomplete premise in more convincing language. The important distinction is between an account of why a system reached its answer and evidence that allows someone to establish whether the answer fits the case.
In the fictional example, a paragraph explaining that outstanding reports should be followed up would not resolve whether this particular report is outstanding. The reviewer needs the current correspondence and the connection between that evidence and the task.
We would therefore assess explanations by what they make challengeable. Can the clinician inspect the originating information? Can they see where uncertainty entered? Does the display reveal a disagreement between sources rather than selecting whichever source makes the proposed action look coherent?
Measure effective challenge without creating unsafe clinical experiments
A suitable evaluation would use approved simulated records containing both correct proposals and deliberately flawed ones. Some errors should be obvious, but others should involve plausible assumptions, changed circumstances or incomplete context. Reviewers should not be told that every case contains a problem.
Assessors could measure detection of consequential errors, rejection of correct work, time needed for review and whether corrections reach dependent tasks. They should distinguish harmless wording changes from interventions that prevent an inappropriate action. The study should also report whether participants had relevant clinical experience and adequate time to review.
These are proposed methods. They have not been run for this article, and no results are being claimed. Findings should be published from an actual documented evaluation, with the case mix and scoring criteria available for scrutiny.
Oversight should focus attention, not reproduce the whole task
The answer is not to make clinicians approve every minor interface movement indefinitely. That could remove the practical benefit of delegation while encouraging superficial checking. A proportionate approach would retain review where interpretation, authority or external consequence changes, and reduce unnecessary interruption elsewhere.
For a practice evaluating Heidi II, the question is whether it can explain why its chosen supervision arrangement is appropriate for a particular workflow. For an educator, it is whether a trainee can justify agreement and disagreement. For a supplier, it is whether meaningful challenge remains possible when the interface becomes familiar.
Heidi's new capabilities were not available in the UK or EU in its launch materials checked on 29 September 2026. The evaluation principles nevertheless apply to any locally available system that asks clinicians to authorise AI-prepared work. They also apply to iatroX outputs: source inspection and independent assessment should not be replaced by confidence in the brand presenting the answer.
Frequently asked questions
Does an approval button make an AI agent safe?
An approval button can provide a control point, but its protective value depends on the information, time and authority available to the reviewer. It is not evidence that every approved action was independently understood.
Did the JAMA study evaluate Heidi II?
No, the December 2023 study examined AI predictions in simulated diagnostic vignettes. It provides evidence about susceptibility to misleading assistance, not a Heidi II error rate.
Should clinicians manually repeat all work completed by an agent?
Not necessarily, because complete duplication may defeat the purpose of delegation. The better objective is proportionate review of consequential assumptions and a demonstrated ability to challenge, correct and recover.
Practise explaining clinical decisions with the Socratic Tutor →