A clinical platform can contain several functions without having one universal measure of correctness. A note should represent an encounter faithfully. A code should express the appropriate concept under the relevant coding rules. A recommendation should be justified by evidence and apply to the situation. These requirements overlap, but they are not interchangeable.
Tandem's published evaluation accompanying Sovereign Health concerns note generation, as explained in the benchmark guide. Tandem also markets a Coding Assistant and Clinical Decision Support. Their presence in the same product makes task-specific evidence more important, not less.
This article proposes an evaluation matrix for clinical teams. It is not a report of comparative testing or a regulatory classification of those products.
Begin with the job each output must perform
Documentation asks whether the record accurately represents what was said, observed and agreed. A well-written summary should not add an examination that did not occur or turn a future possibility into a completed action.
Coding asks whether the selected concept, specificity and combinations are justified. The correct answer depends on the coding system and its current rules. A technically valid code can still be wrong for the documented encounter.
Evidence retrieval asks whether the system found material relevant to the question. Answer synthesis asks whether that material supports the generated explanation. A genuine reference can be irrelevant to the population or fail to support a particular sentence.
Decision support adds application: does the proposed action fit the patient's circumstances, available information, professional role and local setting? A reference answer about a condition is not automatically a complete assessment of a person with that condition.
Choose a reference standard appropriate to the task
A transcript can help evaluate whether a note faithfully reflects spoken information. It cannot independently establish that the care described was appropriate. The transcript may itself contain errors or omit information available through examination or the existing record.
For coding, the evaluation needs appropriately qualified coding judgement and a specified code set. For recommendations, it needs clinically appropriate assessment against the relevant evidence and context. Some cases will permit more than one defensible answer.
This means the reference standard may be an adjudicated assessment rather than a single pre-existing string of text. Reviewers should record disagreement and how it was resolved. Treating a disputed clinical judgement as unquestionable ground truth can hide uncertainty within the evaluation itself.
The criterion should be written before reviewing the system's answer. Otherwise an attractive output can quietly reshape what the evaluator decides the task was meant to achieve.
An evaluation matrix that preserves the differences
| Function | Reference for assessment | Important failure to examine | Useful outcome |
|---|---|---|---|
| Speech capture | The appropriately available source recording | Negation, number or speaker errors | Clinically meaningful capture errors |
| Note generation | Source material plus expected relevant facts | Fabrication, omission and changed certainty | Fidelity, consequential errors and editing work |
| Code suggestion | Documentation and specified coding rules | Unsupported specificity or wrong concept | Correct selection and correction burden |
| Evidence retrieval | A scoped question and relevant source set | Missing or inapplicable evidence | Retrieval of material supporting the task |
| Recommendation | Clinical context and justified expert assessment | Inappropriate action or missing prerequisite | Appropriate decisions and boundary handling |
| Transfer or execution | Authorisation and destination-system state | Wrong patient, duplicate or uncompleted action | Verified appropriate completion |
The table is a proposed design aid. It should be adapted to the actual function, rather than treated as a checklist that certifies an entire platform once each row contains a number.
Follow one fictional error across the platform
Imagine a synthetic consultation in which a patient says that a relative has a particular diagnosis. The generated note places the diagnosis in the patient's own history. A coding component then selects a corresponding code, and a recommendation component uses that coded history as context.
The coding component might have correctly mapped the words it received. The recommendation might be internally coherent given the wrong history. Yet the complete workflow is still wrong because an early attribution error has acquired authority downstream.
Testing only each downstream component against its supplied input would miss the original failure. An end-to-end test needs the original encounter and a record of the transformations between stages.
A second version of the same exercise can begin with a correctly qualified note: "diagnosis considered but not established". The question is whether later functions preserve that status or convert it into certainty. This tests a boundary between components rather than another isolated medical knowledge question.
Measure error propagation and containment
An evaluation should ask where a mistake is detected and what happens next. Does the clinician see the source that would reveal the attribution problem? Does correcting the note update pending codes or suggestions? Does an already transferred entry require a separate correction?
These are operational questions with clinical consequences. A successful edit in one window is not proof that every dependent output has changed.
A useful test record would distinguish the error introduced, the control that detected it and the remaining correction work. It should also include cases in which the system appropriately stops because the input is insufficient. Refusal to produce an unsupported code or recommendation can be correct behaviour, not a failed completion.
Regulatory scope follows the function, not the benchmark label
The MHRA's clarification of 29 July 2026 explains how existing medical-device law applies to ambient voice products in Great Britain. It distinguishes documentation-only functions, including code suggestions for clinician review, from functions intended to support diagnosis, treatment or prevention, or specified automated actions without review.
That distinction must be applied to the actual intended purpose. It does not mean that every product marketed as a scribe is outside device regulation, nor that every code-related feature has the same status. Adding a new function can require reconsideration.
The clarification also leaves clinician review and local governance important. A function outside medical-device regulation can still introduce a clinically significant documentation error. Regulatory category and error consequence are different enquiries.
The clinician-plus-system is another unit of evaluation
A model can produce an accurate answer while the interface encourages the user to apply it to the wrong problem. A less polished output may be safer in a particular interaction if it makes uncertainty visible and prompts the necessary check.
The DECIDE-AI guideline addresses reporting of early live evaluations, including human factors. Its relevance is the need to examine how the system is used, not merely how an isolated answer is scored.
A local evaluation could compare the final reviewed record, important missed errors, total staff effort and unresolved tasks against the existing workflow. This avoids comparing assisted work with an imaginary manual process that never contains errors.
The same distinction applies to educational platforms
A reference answer, a tutoring exchange and a simulation assessment should not share one undifferentiated claim of accuracy. iatroX's methodology distinguishes its reference and learning approaches, but the described design still needs task-appropriate evaluation.
For a tutor, useful questions include whether feedback identifies the learner's misconception and whether the learner can apply the distinction in a new case. For a simulation, assess the scenario, interaction and feedback against the intended educational purpose. Neither should be advertised as proving competence at the bedside solely because an automated score was produced.
What a buyer should ask for next
Request a function-by-function evidence map. For each enabled feature, identify its purpose, configuration, evaluation population, reference standard, known limitations and the human action expected afterwards.
Then examine the connections between features. The most consequential problem may sit where a documented possibility becomes a coded fact, or where a reviewed draft becomes an external action.
One benchmark can make a valuable contribution to that map. It cannot reasonably stand in for every row. The stronger supplier is not necessarily the one publishing the largest single percentage, but the one that can explain which evidence supports each promise and where additional work remains necessary.
Frequently asked questions
Can a correct note still lead to an incorrect code?
Yes. Mapping, specificity, coding rules and the interpretation of uncertainty create additional opportunities for error after the note has been generated.
Does clinician review remove the need for product evaluation?
No. The evaluation should examine whether the reviewer can detect and correct important problems within the actual workflow.
Should reference, tutoring and simulation share one accuracy score?
No. They perform different tasks and require different outcomes and reference standards. A strong result in one area cannot validate the others automatically.