A wrong AI answer becomes a different operational problem when software can act on it. A misleading paragraph may influence a reader; a connected agent may also send a message, alter a record or initiate a workflow before the mistake is recognised. The risk depends on both the information and the authority attached to it.
This does not make reference tools harmless or agents inherently unsuitable. It means evaluating the transition from language to action explicitly. The important questions are what can happen, who authorised it, how the system knows it happened and how care continues if the action was wrong.
Four levels that should not share one approval label
A useful editorial framework separates reference, recommendation, preparation and execution. It is a practical permission ladder, not a regulatory classification.
| Level | Proposed role | Question before increasing authority |
|---|---|---|
| Reference | Explain or retrieve relevant information | Can the reader inspect the supporting source and limits? |
| Recommendation | Suggest a patient-specific next step | Is sufficient reliable context available for that suggestion? |
| Preparation | Create a draft action for review | Is the intended recipient, patient and consequence clear? |
| Execution | Commit the authorised action in another system | Are permissions enforced and completion independently confirmed? |
A product can occupy more than one level. That is why a brand-wide statement such as "clinician supervised" is insufficient. Supervision may mean reviewing a recommendation, approving a draft, auditing an executed action or being available if a problem occurs. Those are different safety arrangements.
A fictional referral shows where the risk changes
Imagine a fictional outpatient workflow. An assistant summarises a history, proposes a referral and prepares a letter. The clinician approves the clinical content, but the software selects a destination from an outdated service list.
At the preparation stage, the error may still be corrected before anything leaves the organisation. Once the letter is sent, recovery includes identifying the recipient, determining whether the intended service received anything and ensuring the patient's care is not delayed. Editing the local draft does not undo the external communication.
Now change the example. The letter reaches the intended service, but the referral is rejected because required information is absent. The agent reports "sent", which is technically accurate, while the clinical task remains unresolved.
These are constructed scenarios, not allegations about a particular supplier. They illustrate three separate tests: was the action authorised, was it appropriate and did the intended pathway actually progress? Success on one does not establish success on the others.
Reversibility should shape the permission boundary
Some actions can be changed locally before they affect anyone. Others create commitments or disclosures that cannot be fully reversed. A draft can be deleted; a patient cannot unread a message already delivered.
For each proposed action, assess its consequence, the opportunity to intervene and the practical recovery process. A low-friction approval may be suitable for a bounded administrative step but inadequate for a change whose clinical implications remain unresolved.
Do not rely solely on the model to decide whether it has permission. The system surrounding it should enforce which records it can access, which tools it can call and which actions require approval. A confident recommendation should not expand those permissions.
The NCSC's agentic-AI guidance, published on 15 May 2026, recommends limiting access and scope, managing dependencies and planning for incidents. Those are security-design principles, not a claim that meeting them establishes clinical appropriateness.
External content should not become an instruction channel
An agent may read letters, messages or web pages as evidence. That material can contain language which attempts to direct the system rather than merely inform it. The consequence is especially important when the same system has access to operational tools.
The NCSC's prompt-injection discussion, published on 8 December 2025, emphasises managing residual risk and limiting consequences rather than assuming a stronger instruction will eliminate the problem.
For healthcare, a defensive design should keep trusted permissions separate from the content being processed. A document should not be able to grant itself authority to change a record or contact an external recipient. This principle can be explained without publishing an operational attack recipe.
The same boundary helps with accidental ambiguity. A quoted historical instruction in a letter should not silently become a current instruction to the agent. The system needs to preserve who said it, when and in what context.
Clinical scope still needs its own assessment
The MHRA's clarification of 29 July 2026 distinguishes documentation-only ambient voice functions from functions supporting diagnosis, treatment or certain autonomous actions. It concerns Great Britain and does not mean that every feature marketed under a scribe brand has the same regulatory status. See the MHRA announcement.
A local implementation should therefore review the exact function when authority expands. An earlier assessment of note drafting does not automatically cover sending communications, initiating orders or altering a plan.
Regulatory status is only one part of the decision. The organisation also needs to understand its local workflow, staff responsibilities, patient population and fallback arrangements. A permitted function can still be deployed badly; a non-device administrative function can still affect care.
This is a governance analysis checked on 10 October 2026, not a legal classification of any named product.
Completion needs evidence from outside the conversation
An agent saying "done" is a statement to verify. A completed external action should be supported by the relevant system status, receipt or acknowledgement, with a clear distinction between submission and acceptance.
A proposed action record would show the original objective, authorised scope, action attempted, destination, confirmation received and unresolved work. It should also identify the responsible person or team for exceptions.
Retries require particular care. If the first attempt may have succeeded but confirmation was lost, simply repeating the action can create duplicates. The safe response depends on the external system and task, not on the model's desire to finish the conversation neatly.
A useful test environment would include rejected requests, interrupted connections, changed patient instructions and ambiguous statuses. Correctly pausing and handing back should count as successful boundary management when further action would be inappropriate.
Reference tools occupy a different category, not a risk-free one
As described on 10 October 2026, Ask-iatroX supports referenced clinical questions rather than issuing prescriptions or executing record-connected clinical workflows. That distinguishes its intended role from an operational agent.
It does not establish that an incorrect reference answer cannot influence care. A clinician can still act on misunderstood, incomplete or inapplicable information. The iatroX methodology should be read as a description of safeguards and limitations, not immunity from error.
For learning, a useful exercise is to identify the point where an explanation becomes a proposed decision, and the separate point where that decision becomes an authorised action. Those transitions are where information quality, clinical judgement and software permissions need to meet.
From AI advice to action: the additional evidence question
The action-permission ladder above also changes what a manufacturer should demonstrate. This section, added on 13 October 2026, focuses on that evidence rather than introducing another general autonomy classification.
Scarlet's agentic medical-device discussion, published on 5 March 2026, considers certification questions raised when software moves beyond advice into connected actions. It is a certifier's published analysis, not permission for every autonomous workflow or proof that any named product is safe.
For a fictional referral assistant, producing an appropriate recommendation is one test. Preparing the correct referral, obtaining the required approval, transmitting the approved version and accurately reporting its status are additional tests. Success at the first stage cannot stand in for evidence about the later stages.
| Proposed action | Plausible failure | Proposed control | Verification exercise |
|---|---|---|---|
| Prepare a referral | Wrong patient or superseded information | Bind the draft to the encounter and source version | Change the selected patient in a synthetic test and inspect the result |
| Request approval | Reviewer sees a different version from the one released | Bind approval to the exact content and destination | Edit after approval and test whether fresh review is required |
| Transmit the referral | A timeout leaves completion uncertain | Preserve an explicit unresolved state | Interrupt a test connection and inspect both systems |
| Retry transmission | The first attempt succeeded but the retry duplicates it | Reconcile status before repeating a consequential action | Simulate a delayed acknowledgement |
| Report completion | The agent says the task is done without external confirmation | Use verified system status rather than generated wording | Return a rejection and assess what the user is told |
The worksheet specifies tests to conduct; it contains no observed results. A real evidence pack should record the environment, configuration, expected behaviour, actual result and any unresolved discrepancy.
Cancellation and recovery are part of the evaluated function
Suppose the fictional clinician changes the plan while a referral is queued. A useful evaluation should establish whether the queued action is stopped, whether the user can see its status and what happens if transmission already occurred. A cancellation button is not evidence that every external consequence can be reversed.
The same applies to escalation. If an agent hands a task back, test whether the receiving person can identify what was completed and what remains uncertain. The good machine learning practice principles support evaluating the human-AI team in relevant conditions rather than treating a human approval step as self-validating.
A final acceptance test should examine the chain as a whole. Individually correct components can still fail through mismatched assumptions about identity, timing or status. The manufacturer's evidence and the adopter's local tests should identify those boundaries explicitly, consistent with the division of clinical-risk responsibilities in DCB0129 and DCB0160.
As described on 13 October 2026, Insights by iatroX offers clinical safety and readiness support. That work can help map hazards to controls and verification, but it is not a claim that iatroX has tested or certified the fictional workflow or any commercial agent discussed elsewhere in the Journal.
When an integrated agent makes an error, responsibility has several layers
The shift towards AI-native platforms makes ownership of a failure more important, not less. This section, added on 15 October 2026, distinguishes professional accountability from the legal allocation of liability. The latter depends on the facts, relationships and applicable law; it cannot be settled by saying either that the doctor is always solely responsible or that the supplier takes over responsibility whenever AI is involved.
The GMC's guidance on AI and innovative technologies says clinicians remain responsible for their decisions, while those involved in creating, testing and updating technology retain responsibility for those activities. That distinction should shape the operating model. A clinician can assess a proposed clinical plan, but may not control an integration defect, a hidden permission change or the supplier's release process.
Consider a fictional hospital investigation request. A clinician approves the appropriate request for the intended patient, but an integration sends it to the wrong destination. The investigation is never completed, while the local task list shows the request as submitted. Assessing the incident requires examining clinical judgement, interface design, transmission status, local monitoring and the response to unresolved work. The clinician's approval is relevant evidence, not the whole explanation.
Preserve the chain between recommendation, approval and outcome
An incident record should make it possible to distinguish the recommendation shown, the information available, the version approved and the action actually executed. If the content or destination changed after approval, that matters. If the receiving service rejected the action, the response to that rejection matters too.
The record should not become an indiscriminate archive of patient information in personal accounts. Relevant evidence needs to be preserved through the organisation's authorised processes, with information-governance and legal input where appropriate. Traceability and data minimisation are design questions to resolve together.
Integrated platforms can simplify this work when they provide coherent status and version information. They can make it harder when several functions appear behind one interface and the user cannot tell which component supplied a fact or executed a step. More integration is therefore not automatically more or less accountable; the implementation determines what can be examined.
Allocate ownership before an incident, not after it
A proposed deployment record should identify who owns the clinical task, who maintains the software, who monitors unresolved actions and who can suspend the relevant function. It should distinguish a clinician's authority to decline a recommendation from an organisation's authority to disable a faulty integration.
For applicable English NHS deployments, DCB0129 and DCB0160 provide the manufacturer and adopter distinction. A local clinical safety case should connect supplier assumptions with actual staffing, review arrangements, escalation and contingency plans. It should not simply attach a supplier document to a local approval form.
The MHRA's intended-purpose guidance, reviewed on 15 October 2026, also helps frame the function being assessed. A reference feature, documentation function and action-taking component should not be assigned one undifferentiated status merely because they share a brand.
A defensible response to increasing autonomy is consequently bounded rather than absolute. Limit what the system may do, preserve meaningful review where required, verify completion and ensure that people have the information and authority to intervene. An independent clinical reference can help examine the medical proposition, but it cannot confirm that a remote system executed the right action or provide automatic legal protection.
Frequently asked questions
Why are healthcare agents different from medical chatbots?
An agent may execute actions through connected systems, so an information error can become an operational event. The distinction depends on the actual permissions and functions, not the product name.
Does human approval make an automated action safe?
Approval is one control, but its value depends on what the reviewer can inspect and understand. Authorisation, clinical appropriateness and confirmed completion require separate checks.
Can a clinical-reference answer still cause harm without tool access?
Yes: a reader may act on an inaccurate or inapplicable answer. Limited operational authority reduces some consequences but does not remove the need to verify clinical information.