A displayed accuracy of 100.0% deserves careful interpretation, not automatic dismissal and not an assumption of error-free clinical care. The first question is what was counted. The next questions concern what the test could miss and whether the result applies to the workflow a clinician will actually use.
Tandem's Sovereign Health evaluation, reviewed on 19 October 2026, describes 44 models tested on 50 consultation transcripts using its harness for medical-note generation. It reports clinician preference and clinical-claim accuracy, including hallucinations and a predefined set of clinically important omissions. Several displayed accuracy scores are 100.0%.
That is a concrete evaluation worth discussing. It is not a published demonstration that every consultation, note or downstream decision will be correct.
Start with the input and the output
The stated input is a consultation transcript. The output being assessed is a medical note. This distinction places an important boundary around the experiment.
A transcript-to-note test can examine summarisation of the information supplied. It does not automatically test the accuracy of capturing speech from a noisy consulting room, distinguishing speakers or recognising a medicine name in an unfamiliar accent. Those problems occur before note generation when the real product starts from audio.
It also does not test whether the clinician asked the necessary questions during the encounter. A note can faithfully reproduce an incomplete consultation. Documentation fidelity and adequacy of the clinical assessment are different outcomes.
The comparison therefore belongs to a specified part of the workflow. Its relevance should be extended through further evidence, not through the breadth of the product's feature list.
The accuracy unit is a clinical claim
The public description identifies clinical claims as the scoring unit. The remaining methodological questions concern how that inventory is constructed, how omitted facts enter the calculation and how results are aggregated across notes.
A hallucination introduces content unsupported by the transcript. An omission leaves out information that the scoring process expects the note to retain. Including selected important omissions is valuable because an entirely truthful but incomplete note can still be inadequate.
However, an omitted fact is not literally a sentence present in the output. Reconstructing the calculation requires understanding how expected-but-missing claims are added to the scoring inventory, how claims are segmented and how results are combined across notes. The retrievable summary does not disclose enough to reproduce every one of those choices.
The appropriate request is for the rubric and aggregation method, not an allegation that the published measure has no definition.
A worked example shows why the unit matters
Consider a hypothetical evaluation, unrelated to Tandem's actual results. Ten notes contain a total of 200 assessed claims. Two claims are wrong, both in the same note.
A simple claim-level calculation would give 198/200, or 99% accuracy. A note-level measure asking whether every assessed claim was correct would give nine out of ten notes, or 90% completely correct notes. Both calculations can be arithmetically valid while answering different questions.
Now add an important omitted instruction. Whether it changes the first denominator, the numerator or a separate completeness measure depends on the predefined rubric. That is why the rubric matters more than choosing the most impressive percentage.
A clinically useful report could show several views together: claim-level errors, notes containing any error, notes containing a consequential error and the time needed to correct them. No single measure has to carry the entire assessment.
A perfect displayed score has several limits
First, a displayed result may be rounded. A one-decimal display does not establish that the unrounded value was mathematically exact. This is a possibility to clarify, not a claim that any particular Tandem score conceals an error.
Second, zero detected errors in a finite test does not prove a zero error rate outside it. For illustration, if zero events occurred in 50 independent opportunities, the one-sided 95% binomial upper bound would be approximately 5.8%, calculated as 1 - 0.05^(1/50). That is not a confidence interval for Tandem: transcripts and claims are not interchangeable, independence cannot be assumed, and the published score may use a different aggregation.
Third, severity matters. A minor wording difference and an incorrect medication instruction should not disappear into an average without a separate account of their consequences.
Finally, the evaluation cannot reveal failures absent from its cases or rubric. A carefully selected test can still underrepresent unusual multi-speaker consultations, fragmented histories or a particular specialty's documentation needs.
Clinician preference answers another useful question
Preference measures which note a reviewer would choose under the comparison conditions. It can capture readability, organisation and practical usefulness that a narrow factual checklist misses.
It does not follow that the preferred note is always the more faithful one. A shorter note may be easier to scan while excluding a qualification. A more confident account may sound clearer while overstating uncertainty. These are hypothetical tensions that the evaluation should investigate, not reported findings about Tandem's reviewers.
Useful methodological details include whether reviewers were blinded to the model, how pairs were selected, whether ties were allowed and whether every reviewer assessed the same material. Reviewers' specialties and familiarity with local documentation conventions also matter.
Preference and accuracy can therefore be complementary. Neither should be substituted for the other, and neither alone demonstrates improved patient outcomes. The Heidi and Tandem model-strategy comparison applies this distinction to evaluations of different clinical tasks.
Fifty transcripts can support discovery without settling generalisation
A small, carefully assembled set can identify substantial differences between systems and expose obvious failure modes. It can help a developer decide which configurations deserve more testing.
The limitation is the breadth of the conclusion. A statement that transcripts represent European languages and care settings does not disclose how many examples came from each language, specialty or clinical environment. A subgroup represented by very few encounters cannot support a precise subgroup-performance claim.
Repeated generations are another issue. If model outputs vary, a single generation per transcript can miss variability. If there were repeated runs, the analysis needs to explain how their dependence was handled.
Model selection should also be separated from final assessment. When many candidates are compared, a separate held-out evaluation can help establish that a selected configuration's apparent advantage was not peculiar to the selection set. The public summary does not settle the complete development and testing sequence.
What additional disclosure would strengthen the result?
An informative methods appendix would provide the transcript-selection process, case distribution, de-identification or synthetic-data arrangements, model versions and generation settings. It would explain claim segmentation, the expected-fact inventory, error severity and reviewer disagreement.
The results would ideally include uncertainty, per-case distributions and examples of failures as well as successful notes. Where human and automated assessment are combined, their roles and validation should be explicit. The full set of available results matters more than an isolated leaderboard position.
None of these requests requires public release of identifiable consultations. A supplier can disclose methodology and suitably protected examples without exposing patient information.
The DECIDE-AI reporting guideline addresses a later, different evidential step: early live clinical evaluation and the human workflow. It is useful when moving beyond a benchmark, not a certificate that can be attached retrospectively to one.
The next evaluation should follow the note into work
A prospective assessment could examine whether clinicians detect important errors, how long review takes and whether the final record preserves the encounter. It should include integration failures, interrupted sessions and corrections that need to propagate to later outputs.
That is not the same as demanding that every product immediately prove a mortality benefit. The outcome should match the claim. A claim about reduced documentation effort needs a whole-documentation-workflow assessment. A claim about improved decisions needs an appropriate decision or patient-outcome evaluation.
The task-specific validation article explains why successful note generation cannot stand in for evidence about coding or recommendations.
Apply the same standard to iatroX
iatroX's methodology describes retrieval, source linking and checks. Those design choices should not be translated into a claim of perfect accuracy either. A method description, usage count, satisfaction survey and controlled performance evaluation establish different things.
A fair conclusion is that Tandem has published a useful, bounded comparison that raises worthwhile follow-up questions. The displayed score should be read as a result under that assessment, not as a guarantee about the next patient. Clinicians can take the work seriously while remaining precise about what it proves.
Frequently asked questions
Does 100.0% mean every generated note was completely correct?
That cannot be inferred from a clinical-claim accuracy display. Note-level completeness, rounding and the aggregation method need separate examination.
Did this benchmark test the entire audio-to-record workflow?
The stated task uses consultation transcripts to generate notes. Speech capture, record transfer and live clinician review require additional evaluation.
Is a 50-transcript evaluation worthless?
No. It can support development and identify differences under defined conditions. Its limits concern the breadth and certainty of conclusions extended beyond those conditions.