Being better with assistance does not prove that independent skills have deteriorated, and retaining skills does not prove that a clinical agent is useful. Heidi II needs evaluation of both the clinician-system partnership and the competence clinicians retain. The comparison should reflect real practice, including ordinary reference tools, rather than an artificial choice between automation and unsupported memory.
The product context is Heidi's September 2026 launch announcement. Its new capabilities are not available in the UK or EU. The framework below is a proposed assessment, not a report of outcomes from a Heidi II deployment.
Start with the actual alternative
A clinical service considering an agent already has a way of completing the work. It may involve people, existing software, ordinary reference resources and informal coordination. The relevant comparison is with that workflow, including its errors and unfinished tasks.
A control condition that removes normal support can make an intervention appear more impressive without answering the purchasing question. Conversely, comparing an early implementation with an established, highly supported process may obscure the effort needed to make the new system work well.
The evaluation should describe both conditions clearly. It should establish what each participant could access, what training they received and which tasks were considered suitable for delegation. Otherwise a result attributed to the agent might reflect differences in the surrounding support.
Evidence already shows that collaboration is not uniform
A UK study published on 29 April 2026 involved 22 physicians completing clinical-vignette tasks with and without language-model assistance. It found improved assisted performance alongside substantial variation between participants. This was a study of clinical reasoning with a different system, not Heidi II's agents or long-term skill retention. See Healy and colleagues' original study.
The relevant implication is that an average result should not be assumed to describe every user. How clinicians interact with assistance, what they ask it to do and which parts of its output they incorporate deserve investigation.
It would also be inappropriate to interpret variable benefit as proof of deskilling. Immediate collaboration and change in independent competence require different study designs and outcomes.
Define the workflow before selecting the metrics
Consider a fictional practice evaluating assistance with administrative follow-up after consultations. The task begins with an agreed instruction and ends when the intended action is confirmed or an unresolved issue is handed back appropriately.
The evaluation should specify those boundaries. Does drafting count as completion? What evidence establishes receipt by another system? Which exceptions require a person? Which cases are unsuitable and should remain outside the automated process?
Without those definitions, a count of completed tasks may combine different achievements. A prepared document, a submitted request and a confirmed external result should not be reported as equivalent units. The measure must match the work the practice hoped to delegate.
Measure immediate benefit across the team
Relevant immediate outcomes include task quality, incomplete work, consequential errors, appropriate escalation and the effort required to review or rescue the process. The evaluation should also examine whether work moves from the clinician to reception, nursing or administrative colleagues.
Patient outcomes require measures suited to the workflow. A reduction in unfinished correspondence may be useful, but it should not automatically be presented as improved clinical outcomes. Where the evaluation is not designed to assess patient benefit, its claims should remain at the process level.
Time also needs careful interpretation. A shorter initial interaction can coexist with more later checking. An apparently slower process could produce a more complete result. We would therefore measure the full task cycle and the work remaining at its endpoint rather than only time spent inside the agent interface.
Assess retained competence separately and safely
Baseline and delayed simulated assessments could examine case interpretation, recognition of contradictions, explanation of alternatives and recovery from agent failure. The cases should be comparable in difficulty but not identical, so improved recall of a particular example is not mistaken for transferable learning.
The assessment should preserve ordinary support appropriate to the capability being tested. If the question concerns clinical interpretation, removing every reference source may introduce an irrelevant memory burden. If it concerns recovery, participants need access to the same operational information they would have in the intended fallback process.
Clinically necessary support should not be withheld from real patients to create an unaided comparison. Educational or simulated settings can assess retained competence without imposing that risk on live care.
These proposed assessments have not been performed for this article. There are no Heidi II retention scores being claimed.
Account for who uses the system and when
Selective use can change the apparent result. Clinicians may initially delegate only straightforward cases, or they may seek assistance mainly when a task is difficult. Either pattern can distort a simple comparison between assisted and unassisted work.
The study should therefore record eligibility, actual use and the reason a task was retained by a person where feasible. Experience, workload, case mix and changes in staffing should also be considered. A before-and-after improvement or decline should not automatically be attributed to AI.
Learning effects matter too. Participants may become better at the underlying task, more familiar with the assessment or more skilled at using the agent. These are different explanations and should be distinguished where the design allows it. Repeated observations and an appropriate comparison group can make the interpretation more informative.
Include failures that require the clinician to change course
A capable partnership is not demonstrated only by completing straightforward tasks. An approved simulation should include an incorrect assumption, a changed patient circumstance and an interrupted workflow. The clinician should be able to identify when to challenge, stop or take over.
Scoring should distinguish appropriate escalation from avoidable failure. It should also count incorrect acceptance and unnecessary rejection of correct work. A system or user that refuses every case may avoid some errors while providing little practical benefit.
Confidence can be measured alongside performance to assess calibration. The question is whether clinicians know when the task is well supported and when uncertainty remains, not whether they report uniformly high confidence after training.
Publish enough detail to make the result interpretable
A useful report would describe the product version, connected systems, permissions, supervision arrangement and training. It would provide task denominators, exclusions, follow-up duration and the handling of partial completion. It should also make clear who performed the review and whether they were independent of the supplier.
Headline claims should stay within the study's scope. A simulated reasoning score is not diagnostic accuracy in routine care. A model benchmark is not a clinician-system outcome. An internal test is not automatically an independent validation, although it can still provide useful development evidence.
Heidi's 29 September 2026 trust article describes clinical testing and continued monitoring. Those commitments are relevant, but buyers still need results that answer their particular workflow and competence questions.
Decide what the combined result means
A favourable assessment would show useful assisted performance without an unacceptable loss of the competence needed to supervise and recover. A mixed result might justify a narrower workflow, different training or a revised supervision arrangement rather than a verdict on the whole product.
The same standards should apply to iatroX. Adaptive questions, tutoring and simulation are educational designs whose benefits should be assessed, not presumed from their labels. For any clinical AI product, the strongest evidence explains both what the system helps people achieve and what those people remain able to understand when assistance stops.
Frequently asked questions
Does better performance with Heidi II prove it improves independent competence?
No, assisted performance and retained competence are different outcomes. They require separate assessments, including appropriate baseline and delayed measures.
Should an evaluation remove normal clinical reference resources?
Not unless that matches the specific capability being assessed. For many questions, the realistic comparison includes ordinary references and existing workflow support.
Can a before-and-after study establish that AI caused deskilling?
Not by itself, because workload, case mix, selective use and other changes may explain the result. A stronger design needs comparable assessments and appropriate controls for alternative explanations.