A Stanford-Harvard evaluation, referred to as the NOHARM study, tested 31 AI systems against 100 real clinical consultation cases spanning ten specialties, using 29 board-certified physician evaluators who produced 12,747 expert annotations across the full evaluation. AMBOSS AI Mode, identified in the study as LiSA 1.0, ranked first overall and recorded the lowest severe-error rate among all systems tested. This is a genuinely more informative kind of evaluation than a standard examination benchmark, and it still deserves careful, precise reading rather than an uncritical headline takeaway.
Why this design is more useful than an examination benchmark
Several features of the NOHARM study's design address real weaknesses in simpler, examination-style evaluations, covered elsewhere in this content series with respect to other platforms' benchmark claims. It preserves incomplete clinical information, reflecting how real cases actually present, rather than the clean, fully specified scenario a multiple-choice question typically offers. It measures potential benefit and harm directly, rather than only factual accuracy against a single correct answer. It includes errors of omission, capturing cases where a system failed to mention something genuinely important, a failure mode a simple accuracy score can miss entirely. And it uses open-ended management recommendations rather than a fixed set of multiple-choice options, testing genuine clinical judgement rather than recognition among presented alternatives.
The wider finding worth taking seriously
Beyond AMBOSS's own strong result, the study's broader finding deserves attention in its own right: safety, as measured through this real-case, expert-annotated methodology, correlated only moderately with performance on established medical-knowledge benchmarks of the more traditional, examination-style kind. In plain terms, a system can score well on standard medical knowledge tests while still producing genuinely harmful clinical omissions in more realistic, open-ended scenarios. This finding matters well beyond AMBOSS specifically, and is worth bearing in mind whenever any clinical AI platform, including iatroX, cites a strong examination-style benchmark score as evidence of real-world safety.
What this result genuinely supports
Taken at face value, this is a genuinely strong result for AMBOSS, and it supports two specific conclusions worth stating plainly. AMBOSS's curated, clinically structured approach to sourcing, covered in detail elsewhere in this content series, merits serious attention as a design philosophy, given that it appears to have translated into measurably lower severe-error rates in this specific, rigorously designed evaluation. And safety genuinely requires separate, dedicated evaluation from factual accuracy; the two are related but are not the same thing, and a platform strong on one should not be assumed automatically strong on the other without direct testing.
What this result does not prove
It is equally important to be precise about the limits of what a single benchmark, however well designed, can establish. It does not establish safety in every specialty or every jurisdiction; the study's ten specialties and its case source do not exhaustively cover the full breadth of clinical practice. It does not establish effectiveness in live, ongoing patient care, since the evaluation itself was retrospective and expert-annotated rather than assessed within an actual clinical workflow over time. It does not establish correct use of UK guidance specifically, given the study's evident US clinical orientation. It does not establish safe EHR write-back, a capability covered elsewhere in this series that remains in development rather than evaluated here. And it does not establish freedom from all hallucination or omission going forward, only a comparatively strong showing within this specific, bounded evaluation.
What a comparable UK evaluation would need to look like
In the same spirit this content series has applied consistently to every platform it examines, iatroX's own performance deserves evaluation against a genuinely UK-specific benchmark: NICE concordance, how reliably its answers align with current NICE guidance; prescribing fidelity, accuracy against current UK medicines information; safety-netting, whether answers correctly flag when escalation or referral is genuinely needed; and referral accuracy specifically within NHS structures. Relying solely on any platform's own internal testing, AMBOSS's or iatroX's, is not a substitute for this kind of independent, methodologically rigorous evaluation.
A conclusion that stays both positive and appropriately cautious
The NOHARM result is a genuinely credible, well-designed piece of evidence in AMBOSS's favour, and deserves to be treated as such rather than dismissed. It is not, on its own, a complete safety certification across every clinical context AMBOSS AI Mode might be used in, and the same standard of appropriately cautious reading should be applied to any single benchmark result from any clinical AI platform.