There is something irresistible about a company claiming to beat every competitor combined. It compresses a messy market into a single image: one product on one side, everybody else on the other. For a busy clinician, that can sound like a shortcut through an exhausting choice of tools.
The shortcut deserves inspection. When the underlying observation concerns resources selected during a study, the denominator determines whether the finding describes those study tasks or the habits of an entire profession.
OpenEvidence's promotion of NOHARM offers a useful example of a real result carrying more rhetorical weight than its design can bear.
What was actually counted?
The 20 July 2026 OpenEvidence announcement reported that physicians selected its platform in 22.3% of responses in an unrestricted study arm, compared with 19.8% for other external AI chatbots combined.
The NOHARM methods describe 101 physicians completing two tasks in that arm, yielding 202 response records. Resources were recorded per response, and participants could report several resources.
That is an interesting finding about behaviour under the study conditions. It is not a census of American physicians. The percentage describes task responses, not the proportion of the US medical workforce using a product.
The distinction should survive every subsequent headline, chart and investor presentation.
A response is not a unique physician
Imagine a hypothetical study involving 20 doctors, each completing ten tasks. A product appears in 60 of the resulting 200 task records. That is 30% of responses. It might have been used by six doctors in all ten tasks, by 20 doctors in three tasks each, or by many other combinations of people and frequency.
The same response share therefore accommodates different levels of individual reach. Counting appearances of a resource measures how often it features in the task records. Counting distinct doctors measures how many participants used it at least once. Neither can be substituted for the other.
This also affects comparisons. A small group of enthusiastic users can create a high response share without a large share of unique users. Conversely, a resource tried once by many participants may have broad reach but a modest share of total task records.
The general lesson extends beyond this study. Whenever a sentence begins with "physicians chose", ask whether the dataset counts physicians, choices or responses. These nouns sound similar in publicity and behave differently in arithmetic.
What does combined include?
A combined comparator is only meaningful when its contents are explicit. Does it contain every clinical AI service, a list of named general-purpose chatbots, or whatever alternatives participants happened to report? Does it include the assistant supplied by researchers? Are conventional reference websites excluded?
These decisions can be sensible. A study may have good reasons to distinguish general-purpose systems from specialised products or supplied resources from external ones. However, the resulting category remains a study category. It should not silently become a complete map of a changing commercial market.
There is also a counting question. If a doctor consults two alternatives on one task, a combined total might count that task once or count two resource appearances. Those calculations answer different questions. A transparent report must say which has been used.
The researchers' public analysis repository provides an important route into the study's reporting structure. Open methods are valuable because they let readers inspect the unit of analysis, instead of relying on the broadest promotional interpretation.
Free choice is informative, but familiarity travels into the study
Letting doctors choose their own resources offers a useful view of spontaneous behaviour. It removes the artificial requirement to use a particular product merely because a researcher supplied it.
It does not remove what participants bring with them. Someone who already has an account, knows the interface and remembers useful answers has a practical advantage when choosing under time pressure. An unfamiliar product must overcome registration, navigation and uncertainty before it can compete for the same task.
That familiarity can represent real product achievement. Building a tool people remember and can use easily matters. It does not establish that every selected answer is more accurate than every unselected alternative.
Consider an ordinary clinical analogy. Repeatedly opening a familiar reference textbook tells us something about accessibility and professional habit. It does not establish that the relevant chapter is the best available treatment of every clinical problem. Resource selection and answer evaluation remain separate exercises.
Popularity and preference are not interchangeable either
To demonstrate a preference between products, it helps to know whether the chooser has meaningfully experienced both. Otherwise, the observation may be closer to availability: the participant selected what they knew and could access.
A comparative preference study might give participants sufficient exposure to several tools, conceal branding where feasible and ask about specific tasks. A naturalistic study might deliberately preserve branding and prior familiarity to capture real behaviour. Both designs can be useful, but they estimate different things.
The first asks how people judge alternatives under structured comparison. The second asks what people reach for within their existing habits. Calling either result definitive market preference without describing the design erases the very information that makes it interpretable.
The same problem appears in published medical AI adoption comparisons. Registration, stated preference and observed activity cannot be arranged on one scale simply because they all involve users.
A numerical lead still needs the right conclusion
A higher observed count can be reported as a higher observed count. It need not be inflated into universal superiority to be interesting.
Researchers and readers also need to consider uncertainty. Repeated responses from the same participant are related observations, not necessarily independent votes. A clinician's habits can influence several tasks. The structure of the dataset matters when estimating whether an apparent difference is stable or might change in another sample.
The honest next question is therefore more demanding than who has the tallest bar. Would the pattern persist across specialties, experience levels, settings and tasks? Would different recruitment methods change it? Would the same participants make similar choices after gaining experience with other tools?
These questions turn a striking count into a research programme. They also prevent a small study from being asked to certify an entire market.
What clinicians should do with this information
For a clinician selecting a reference tool, popularity can justify taking a look. It cannot finish the assessment. The relevant question is whether the product helps with the kinds of uncertainty encountered in that clinician's work.
A practical comparison might use several recurring task types: locating a recommendation, identifying the evidence behind it, checking its jurisdiction and recognising where the evidence is weak. Assess the effort required to reach an answer that can be verified. Do not award extra clinical credit because a competitor's marketing team selected a different denominator.
For iatroX, the methodology and explanation of how the platform works provide starting points for that assessment. They should be considered alongside actual use, rather than as an exemption from the same questions applied to other platforms.
OpenEvidence's reported selection in the study is worth acknowledging. It is also worth describing accurately. A response-level observation becomes less informative when it is stretched into a statement about every doctor and every competing AI. The denominator is not a footnote to the headline. It is the thing that gives the headline meaning.
Frequently asked questions
Did the NOHARM study find that most US doctors preferred OpenEvidence?
The reported result concerns resources selected in an unrestricted task arm involving study participants. It does not estimate the preferences of the entire US physician population.
Is choosing a tool evidence that it is clinically useful?
It can support a finding that participants find it accessible or worth consulting in that setting. Establishing clinical usefulness requires examining the resulting work, including accuracy, completeness, verification effort and consequences for decisions.
Can a company accurately say its count exceeded competitors combined?
Yes, when the counts support that statement and the comparator category, unit and population are preserved. The problem is extending the result beyond the category and setting that were actually measured.