AI judges, a 100-question sample and reused June answers make this a useful evaluation to examine carefully, rather than a current physician-tested league table.
Vera Health's October comparison asks a better question than whether a system can select the right option in a medical examination: how well does it answer the questions clinicians bring to a reference tool? That shift is worth attention. Reading the results still requires separating the question set, the answer versions and the people or models that assigned the scores.
This article was updated on 28 October 2026. iatroX publishes it as a provider of clinical reference and learning tools. We have not conducted a new product trial, and iatroX was not evaluated in the study discussed here.
Start by separating two different evaluations
The original Real-POCQi preprint, submitted in June 2026, describes 620 physician questions spanning 30 specialties, alongside a separate HealthBench set. Its blinded panel of 149 physicians compared OpenEvidence with then-current general-purpose models. Vera was not included. The paper also reports differences between physician and AI judges.
Vera's 26 October follow-on analysis used 100 questions and AI judges, including its own clinician-informed judge. It reused June OpenEvidence and GPT-5.5 answers while generating newer answers for Vera and other systems. Vera reports leading on four of six measures across the field, while Claude led on clinical utility and completeness. The company explicitly says these verdicts were not delivered by the original physician panel.
Those are separate pieces of evidence. Combining them into a claim that 149 doctors preferred Vera would be incorrect. The follow-on work is a company-run automated comparison using a public resource, not a new physician-panel assessment of Vera.
Why real clinical questions are a useful starting point
A question asked during work is often incomplete. It may assume a country, omit a relevant characteristic or use shorthand. An answer needs to decide what can reasonably be answered and which missing information matters. An examination question usually supplies more of the necessary structure.
This makes real questions useful for evaluation, but it does not make any particular question set representative of every practice setting. A specialty with unusual follow-up questions may need different tests from a general outpatient service. A UK clinician may need to know how a tool handles national guidance and local pathways. A set drawn from one platform's users is informative about those questions, without automatically representing everyone else's workload.
Before comparing percentages, ask what the sample contains. Does it include ambiguous questions, changed guidance, requests for a primary source and cases where more context should be requested? An aggregate score becomes more meaningful when readers can see the types of work it covers and the situations it leaves untested.
Historical answers make version dates essential
A comparison can be accurately reported and still answer a narrower question than its headline suggests. Reusing an earlier answer makes the analysis reproducible against a fixed reference. It also means that the result concerns that saved answer, rather than a fresh response from the competitor's current service.
For rapidly changing systems, the relevant comparison unit is not simply the brand name. It is the product configuration, date, enabled tools and prompt conditions. A reader cannot assume that two outputs bearing the same brand months apart would be identical. Nor should a later run automatically be considered better simply because it is newer.
A fair presentation therefore retains an answer-date column. Someone evaluating the current products should perform a fresh, documented comparison under consistent conditions. Someone studying historical development may deliberately retain old outputs. Both are legitimate projects, but they should be labelled differently.
An AI judge is a measurement instrument
Automated judging can make a large comparison practical. It can also reward properties that only partly overlap with clinical usefulness. A clear answer may be easier to rate than a qualified one. A long answer may include more relevant material while making the central point harder to find. A response with many citations may look well supported until the cited passages are opened.
Blinding and changing answer order are sensible design choices. They address particular sources of bias; they do not make every verdict correct. A judge can still miss the same unsupported assumption that appears in the answer it is grading. Agreement across several automated judges is reassuring only to the extent that their errors are meaningfully different.
Treat a judge's explanation as an inspectable output. If it prefers one answer because of a source, open that source. If it criticises an omission, establish whether the missing point was material to the actual question. The verdict becomes more useful when it leads to a traceable reason, rather than ending the discussion.
A win rate is not a probability that an answer is correct
A pairwise preference is a comparison between two particular answers. It is not the same thing as an independently established correctness rate. Both responses could be acceptable, both could share a limitation or one could be preferred for presentation despite a subtle source problem.
There is a second distinction between overall averages and serious failures. A system may perform well on routine questions yet have a small group of important misses. Clinicians and adoption teams should inspect those misses directly. Giving every question equal weight in a summary does not mean every error has equal consequences in practice.
This is why a useful results page should show examples of losses and ties as well as wins. It should explain disputed cases and how conclusions change when the judge or scoring method changes. A transparent limitation is more helpful than stretching a score into a claim of universal superiority.
Build a small comparison that answers your own question
The following worksheet is an original evaluation aid, not a validated clinical assessment instrument. Use approved, synthetic or non-identifying reference questions and have the appropriate reviewers decide what good performance means before opening the outputs.
| Record before testing | Why it matters | What to preserve afterwards |
|---|---|---|
| Intended task and setting | Defines what useful means | The exact question and expected context |
| Relevant authority and version | Establishes the reference point | Links and supporting passages |
| Run date and product conditions | Makes version differences visible | Output text and enabled features |
| Essential answer elements | Reduces preference based on style | Which elements were found or missed |
| Uncertainty and missing context | Tests the limits of the answer | Whether clarification was appropriate |
| Checking time and navigation | Captures practical usability | Steps required to verify the key claim |
Select questions before seeing which product performs best. Include an ordinary lookup, a source whose guidance has changed, an ambiguous query and a request where country matters. Keep the set small enough that someone can inspect every key citation. A carefully reviewed handful of questions is useful for identifying workflow problems, although it cannot establish a general clinical performance rate.
Score answer content separately from readability. Record ties honestly. If reviewers disagree, preserve their reasons instead of forcing a winner. The disagreement may reveal that the task definition needs work or that different clinicians need different levels of detail.
Apply the same standard to iatroX
iatroX describes a workflow combining referenced clinical lookup, structured reasoning and learning routes. A UK clinician who wants to carry a recurring knowledge gap into question-led study has a concrete reason to assess that workflow alongside Vera and OpenEvidence.
That product fit does not supply an absent benchmark result. iatroX should face the same questions about supporting passages, current guidance and useful uncertainty. Its favourable position is most credible when a doctor values its UK reference and learning connection, rather than when an unrelated competitor score is used to imply that iatroX must perform better.
Keep two records: what published evidence establishes and what your own approved assessment reveals. The first helps you understand a provider's claims. The second helps you decide whether its workflow serves your actual work. Neither should silently take the place of the other.
FAQs
Did the original 149-physician panel evaluate Vera Health?
No. Vera was absent from the original study. Its later October analysis used AI judges, rather than a new assessment by that physician panel.
Does the October comparison show how current OpenEvidence answers today?
It reused OpenEvidence answers generated for the June study. A current-version comparison would require new runs with documented dates and conditions.
Was iatroX included in the comparison?
No. It cannot claim a performance result from this analysis. Its documented UK reference and learning workflow can be assessed separately against a clinician's needs.
Review iatroX's methodology and bring the same source-checking standards to every clinical AI tool on your shortlist.