A 1% error rate is not interpretable until the unit and error definition are known. It could mean one incorrect sentence in a hundred, one affected consultation in a hundred or one harmful action in a hundred. Those are different claims with different consequences, and none should be silently substituted for another.
The first response to a reassuring percentage should be a denominator question. What was counted, what qualified as an error, which cases were included and at what point in the workflow was the measurement made?
One percentage can describe very different events
An evaluation might count words, sentences, notes, consultations, recommendations or executed actions. It might examine raw output, clinician-edited output or subsequent patient outcomes. Those choices change the meaning of the result before any statistical analysis begins.
For example, an incorrect sentence can contain several propositions, and one incorrect proposition can affect more than one decision. Conversely, a stylistic error may have no clinical consequence. The denominator should match the claim being made about the product.
A note-generation study cannot automatically establish the safety of an agent that later acts on the note. A diagnostic benchmark cannot establish whether the proposed management was appropriate. The evaluation needs to follow the function and its consequences.
The DECIDE-AI reporting guideline, published in 2022, addresses early live clinical evaluation rather than treating a technical score as the whole evidence base. It is a reporting framework, not an approval process.
A worked example: sentences are not consultations
The following numbers are a hypothetical illustration constructed on 10 October 2026. They are not results from iatroX or any other product.
Suppose a service produces 1,000 notes, each containing 20 assessed sentences. That creates 20,000 assessed sentences. If 200 sentences are incorrect, the sentence-level error rate is 1%.
Those 200 errors could be distributed across 200 different notes, giving 20% of notes at least one error. Alternatively, they could all occur in 10 notes containing 20 erroneous sentences each, giving 1% of notes affected. Both distributions have exactly the same sentence-level error percentage.
Neither distribution tells us the clinical severity of the errors. The example simply shows why a sentence-level statistic cannot be relabelled as a consultation-level statistic.
| Hypothetical distribution | Incorrect sentences | Notes with an error | What remains unknown |
|---|---|---|---|
| One error in each of 200 notes | 200 of 20,000 | 200 of 1,000 | Clinical consequence and whether review corrected it |
| Twenty errors in each of 10 notes | 200 of 20,000 | 10 of 1,000 | Clinical consequence and whether review corrected it |
A useful report would publish the relevant distributions instead of encouraging the reader to infer them from an attractive headline.
Repeated exposure changes the question
A small per-item error probability can lead to a much larger probability of encountering at least one error across many items. However, calculating that probability requires assumptions about how errors relate to one another.
As another hypothetical mathematical illustration, if 20 independent decisions each had a 1% probability of error, the probability of at least one error would be approximately 18.2%, calculated as 1 minus 0.99 raised to the twentieth power. This is not an estimate of real clinical AI performance.
The independence assumption is particularly important. Errors may cluster around a document type, patient group, software version or misunderstood rule. Repeating the same flawed assumption can produce a pattern very different from independent random mistakes.
The practical lesson is not that all repeated use is unacceptable. It is that deployment scale and error concentration belong in the evaluation. A percentage without exposure context does not describe the burden a service will experience.
Severity should not disappear into the average
An evaluation should distinguish harmless wording problems from mistakes that could change interpretation, action or follow-up. The categories should be defined before reviewing outputs, with appropriate clinical adjudication and transparent handling of disagreement.
Consider a fictional pair of errors: an awkward heading in one note and an unassessed symptom recorded as absent in another. Counting both as one error may be legitimate for a narrow technical measure, but it would be inadequate for explaining clinical risk.
The medical-summarisation framework published in May 2025 illustrates the value of separating error type from potential clinical significance. Its particular study setting and commercially affiliated authorship remain relevant when interpreting its findings.
A severity measure should also avoid pretending to know an outcome that was not observed. "Could affect management if uncorrected" is not the same as "caused patient harm". Equally, a lack of observed harm in a small evaluation does not make a potentially consequential error irrelevant.
Which outputs and patients were excluded?
A result based only on completed, accepted outputs may omit difficult cases that were rejected, abandoned or escalated. Those cases are part of the service's workload and should remain visible, even when refusal was the correct response.
Ask whether the evaluation included poor-quality inputs, interrupted encounters, conflicting records and people who could not complete the expected interaction. Examine whether the denominator represents all eligible requests, all attempted requests or only the subset retained for analysis.
Selective handling can be a sensible design. The problem is not that a system declines difficult work; it is reporting accuracy on accepted cases without also reporting how many cases it accepted and what happened to the remainder.
For procurement, the review queue matters as well as the retained accuracy. A system that performs well on a narrow subset may be useful, but its staffing implications differ from those of a tool capable of supporting a broader workload.
Measure the final workflow as well as the raw model
A raw draft's error rate does not establish the rate of errors remaining after professional review. Review may remove mistakes, miss them or introduce new ones. The complete process needs evaluation.
Likewise, an action that was technically completed may still have been clinically inappropriate. A message delivered to the intended inbox is not proof that the recipient understood it or that required follow-up occurred.
A useful report therefore separates generation quality, reviewer performance, final-record quality and downstream outcomes. It states the tested version, setting, period and comparator. Changes to the software or workflow may require further assessment rather than automatic reuse of the old percentage.
The same standard applies to educational products. As of 10 October 2026, iatroX's published methodology describes checking processes; those descriptions should not be converted into an invented platform-wide accuracy percentage or an examination pass prediction.
Questions a buyer can put beside any percentage
Ask for the numerator and denominator in plain language, the severity categories, the inclusion rules and the point in the workflow being assessed. Then ask for the distribution across relevant settings and users, the uncertainty around the estimate and the number of cases that required hand-back.
Finally, compare the result with realistic current care. A new system does not need to be perfect to be useful, but its advantages and added risks must be measured in compatible units. A reassuring number becomes decision-useful only when the reader can explain exactly what it counts.
Frequently asked questions
Does a 1% sentence error rate mean 1% of consultations contain errors?
No: the same sentence-level rate can arise from very different distributions across consultations. The consultation-level result must be measured or explicitly derived under justified assumptions.
Does zero observed harm prove that an AI system is safe?
No: the finding depends on the sample, follow-up, detection method and setting. It can be encouraging without excluding rare or unmeasured harm.
What should a clinical AI accuracy claim report?
At minimum, it should identify the task, population, version, unit of analysis, error definition and evaluation method. Review burden, coverage and clinical consequences are also important when judging a deployment.
Strengthen your evidence-appraisal skills through question practice →