A clinical AI benchmark should make the market easier to understand. In July 2026, NOHARM did something more entertaining: it supplied three companies with three different victory stories.
Doximity promoted a performance lead. OpenEvidence promoted physician choice. AMBOSS promoted its overall benchmark result. A reader following company announcements could reasonably conclude that each had emerged as the obvious winner of the same contest.
The explanation is not that every number must be wrong. It is that the word "best" can travel much further than the measurement underneath it. The task for clinicians is to identify which competition each company is actually describing.
Three announcements, three questions
The public claims can be separated without guessing about any company's internal intentions:
| Announcement | Result selected for emphasis | Question that result addresses |
|---|---|---|
| Doximity, 15 July 2026 | First on its description of the study's real-world clinical sample | Performance within the sample being referenced |
| OpenEvidence, 20 July 2026 | More frequent selection than other external AI chatbots combined | Which resources participants reported using |
| AMBOSS, 28 July 2026 | Highest severity-weighted F1 and lowest severe-error point estimate | Overall scored performance and observed error frequency |
The original statements appear in the Doximity release, the OpenEvidence release and the AMBOSS release. Comparing them is more informative than repeating whichever headline appeared first in a social feed.
The most important distinction is between endpoints
A benchmark can produce several valid measurements. Correctness asks whether recommendations are appropriate. Completeness asks whether important actions were omitted. A harm-weighted composite combines aspects of both, while a severe-error measure focuses on particularly consequential failures.
These outcomes need not produce identical rankings. Imagine two hypothetical assistants. One gives consistently comprehensive answers but occasionally includes a dangerous recommendation. The other avoids those dangerous additions but regularly misses less severe yet useful steps. Which is better depends partly on what is measured and how different mistakes are weighted.
The answer cannot be settled by selecting the metric after looking at who wins. Clinicians need to understand the purpose of each endpoint and the consequences it captures. A broad performance measure and a severe-error measure can both be valuable without being interchangeable.
The NOHARM paper reports no statistically significant differences between the four clinical RAG systems on severe-error rates. That finding concerns that outcome. It does not erase differences in F1 or establish that the products are clinically equivalent.
A leading score is not automatically a proven safety advantage
AMBOSS reported an overall severity-weighted F1 of 86.15% and a severe-error point estimate of 2.9%. It is entitled to describe a highest observed score accurately. The additional question is how much confidence the study supports in the ordering, and for which outcome.
Suppose two hypothetical systems record severe errors in five and seven cases. Calling the first observed rate lower is straightforward. Calling the first system definitively safer in future clinical practice requires more: uncertainty around the estimates, enough relevant cases, an appropriate analysis and a reason to expect the test conditions to transfer.
Equally, the absence of a statistically significant difference does not prove the systems are identical. A study may lack enough information to distinguish them. There is a material difference between "we did not establish a difference" and "we established equivalence".
These distinctions are familiar in clinical research. They should not disappear when a software company turns a results figure into a product announcement.
A sample label needs a definition
Doximity's selected phrase, "real-world clinical sample", sounds intuitively meaningful. It also invites a specific question: which cases and which test conditions does it identify?
Different subsets can rank tools differently without anyone changing the raw observations. A tool may perform particularly well in a specialty, on previously unseen cases, or under a particular instruction. Those findings can be useful if the subset has a clear clinical rationale and its selection is explained.
The danger is allowing a subset label to imply that every other part of the study is somehow less real. A reader needs the sample membership, evaluation procedure and prespecified purpose, rather than an adjective that makes the preferred result sound more representative.
An illustrative procurement comparison makes the point. A system might lead on 40 routine cases while another leads after 60 difficult cases are included. Neither ranking tells a hospital what to buy until the hospital understands which case mix resembles its own work.
Being chosen is a different kind of result
OpenEvidence's selected result concerns resource use. That can reveal accessibility, familiarity and spontaneous choice. It does not constitute a performance ranking on the benchmark's clinical endpoints.
In an unrestricted exercise, participants bring prior experience, existing accounts and established habits. A familiar tool can be selected because it is useful. It can also be selected because it is immediately available while an alternative requires learning a new interface. A study of spontaneous choice deliberately preserves some of that context.
The finding remains interesting without being promoted into a universal statement about accuracy or national market share. The correct response is to keep the unit of observation visible and then examine what happened when the selected tool was used.
This is the same discipline required when reading medical AI adoption figures: people, activities and performance scores cannot be substituted for one another.
Real clinical material does not turn a benchmark into patient outcomes
Clinical cases can be realistic, carefully curated and rooted in practice while remaining a test of proposed answers. A benchmark can identify recommendations judged capable of causing harm. That is different from observing harm occur after a clinician makes and implements a decision for a patient.
This distinction cuts both ways. Potentially harmful recommendations deserve attention even if no patient was exposed during testing. Conversely, a low benchmark error rate should not be presented as a measured rate of patient safety across deployment.
The intervening steps matter: whether a clinician notices an error, changes the answer, follows local policy, seeks specialist advice or ignores a useful recommendation. Evaluating the standalone system and evaluating the clinician using it are related projects with different endpoints.
What a useful journal club would ask
The most productive discussion would put the paper beside all three announcements and ask what survived the journey from methods to marketing.
Start by matching each headline to a table, figure or clearly described analysis. Identify whether the claim concerns the whole dataset, a subset or the separate human-use component. Then inspect whether uncertainty is preserved and whether the comparison describes a point estimate or a supported difference.
Finally, ask what decision the result can inform. It might justify a local evaluation, identify a failure mode worth testing or show that clinical tools deserve comparison with general-purpose models. It may not identify a single best product for every specialty, jurisdiction and workflow.
For education, this is an excellent reasoning exercise. A group could discuss the claims before consulting the results, then use a resource such as the iatroX Tutor to explore methodological questions. The learning objective is the ability to connect a conclusion with the observation that actually supports it.
Let the study remain more interesting than the sales pitch
NOHARM contains questions about omissions, clinical recommendations and how physicians interact with assistance. Reducing that work to a trophy contest makes the science less useful to its intended audience.
The sensible position is neither to dismiss the favourable results nor to grant every vendor its preferred interpretation. Describe each result accurately, keep the outcome measure attached and resist the temptation to nominate a universal winner.
Healthcare AI can compete on evidence. It should compete on the evidence that was measured. Clinicians assessing iatroX should apply the same standard to its methodology and intended approach. A leaderboard is useful only while readers can still see the rules of the competition.
Frequently asked questions
Did all three companies claim exactly the same result?
No. Their announcements selected different observations: performance in a specified sample, physician resource use, and overall benchmark measures. That is why apparently competing victory stories can emerge from one study.
Does a non-significant safety comparison mean the products are equally safe?
No. It means that analysis did not establish a difference on the specified measure. Demonstrating equivalence requires a design and analysis intended to answer an equivalence question.
Should clinicians ignore benchmarks because companies promote them selectively?
No. Benchmarks can reveal important strengths and failures. Read the primary methods and results, preserve the tested context, and use findings to inform a relevant comparison rather than accepting a universal ranking.