Vera Health's 6 August 2025 benchmark report reported 97.5% overall on USMLE-style evaluation, 84.9% on NEJM-AI and 62.2% on MedXpertQA. These are historical company-reported results, rather than a current ranking of every clinical AI product. The useful question is which capabilities those tests measure and which important tasks they leave untested.
Labelling the results accurately
These figures come from a company-run evaluation. They should be attributed and dated, with the tested systems and datasets kept visible. The cited report is not an independent clinical validation of the current product. That is a statement about the evidence source and scope, rather than a reason to assume the scores are false.
Updated 28 October 2026: newer evidence now needs to be read alongside that historical report. Vera's 26 October point-of-care comparison uses 100 questions and AI judges; its reused OpenEvidence answers date from June. Vera was not assessed by the original study's physician panel. This is a different evaluation from the examination benchmarks, and our separate analysis explains the version and judge limitations.
Vera's current homepage also displays a company-reported NOHARM v0.2 result. A potential-harm evaluation addresses a different construct from exam accuracy. Do not merge scores from different datasets or NOHARM versions into a single ranking, or imply that a result on one build proves current clinical performance.
What benchmarks of this kind can measure
Standardised examination-style benchmarks are a real and useful measure of certain capabilities: medical factual knowledge, the breadth and accuracy of a system's underlying medical fact base; multiple-choice reasoning, the ability to correctly discriminate among several presented options; and performance on structured, well-defined clinical questions of the kind these examinations are specifically built around.
What they do not necessarily measure
Several important capabilities for real clinical use are not directly tested by this kind of benchmark. Reliability in actual, messy, real-world consultations, where information arrives incompletely and in an unstructured order, differs considerably from a clean, well-specified examination question. Citation faithfulness, whether a system's cited sources say what the system claims they say, is a distinct capability examination-style benchmarks do not directly test. UK-guideline concordance, whether an answer aligns with NICE, CKS or SIGN specifically, is entirely outside the scope of a US-oriented examination benchmark. Patient-specific applicability, adapting a general answer to a specific patient's actual circumstances, goes beyond what a multiple-choice format can assess. Safety under incomplete information, a common real-world condition examination questions rarely simulate, is a distinct and clinically vital capability. And calibration and appropriate abstention, a system's ability to recognise when it does not have enough information to answer confidently and to say so rather than guessing, is arguably one of the most clinically important capabilities of all, and one standard accuracy benchmarks do not directly capture.
The benchmark contamination question worth raising directly
A further, more technical concern applies to any AI system evaluated against publicly available examination questions: those same questions, or very similar ones, may have appeared within the system's own training data, meaning strong performance can partly reflect familiarity with the specific question set rather than underlying clinical reasoning ability applied fresh. This is a known, general risk across the AI benchmarking field, not a claim specific to Vera Health, and it applies to any vendor citing performance on a public examination-style benchmark.
What stronger evaluation would actually look like
A more rigorous evaluation of any clinical AI system's real-world value would draw on prospectively collected clinician questions, queries clinicians actually asked in practice rather than a fixed, publicly known question bank; blinded specialist review, where expert clinicians assess answer quality without knowing which system produced which answer; citation checking, independently verifying that cited sources support the specific claims attributed to them; adversarial cases, deliberately difficult or ambiguous scenarios designed to probe where a system is likely to fail; and country-specific guideline testing, assessing concordance with a given healthcare system's actual national guidance rather than general medical knowledge alone.
The UK benchmark iatroX should be judged against
In the same spirit of holding every platform to a consistent, meaningful standard, iatroX's own performance should reasonably be evaluated against NICE concordance, how reliably its answers align with current NICE guidance; CKS and medicines-source fidelity, accuracy against Clinical Knowledge Summaries and current SmPC information; referral and prescribing accuracy, specifically within the UK healthcare system's actual structure and conventions; and appropriate safety-netting, whether answers correctly flag when urgent escalation or specialist referral is needed.
A fair conclusion
The newer point-of-care work is relevant because real questions and omission-sensitive assessment can expose weaknesses that a multiple-choice examination misses. Its value depends on transparent methods, contemporary comparator answers and the relationship between automated judgments and clinician assessment. Vera's judge-methodology account is helpful process information, but company-reported judge agreement is not independent validation of all generated answers.
Benchmarks remain useful signals for product development and for choosing what to test next. They should not become unqualified clinical-superiority claims. Apply that principle equally to iatroX: its UK-reference and learning workflows need assessment against the task being proposed, and it was not evaluated in Vera's October comparison.