Automated reviewers can help find problems at scale. Their clinical value depends on what they miss, how disagreement is handled and what happens after an alert.
A clinical AI answer can contain accurate sentences and still fail the question. It may leave out an important qualification, use a superseded source or assume the wrong healthcare setting. An automated reviewer therefore needs to do more than identify obvious falsehoods. It needs a way to notice what a well-founded answer should have included.
Vera Health's new explanation of its AI judges is useful because it makes part of that evaluation process visible. This article was updated on 28 October 2026. It is published by iatroX, a provider of clinical reference and learning tools, and examines the described method rather than claiming an independent audit of Vera's service.
Evidence grading and answer grading are different jobs
Source evidence grading concerns the material behind an answer. Answer grading concerns what the system did with it. A strong source does not guarantee that the generated explanation correctly describes its population, limitations or relevance. Conversely, a clear statement of uncertainty can be appropriate when the available evidence is limited.
Keeping those jobs separate makes evaluation easier. First ask whether the cited material is relevant and current. Then ask whether the answer accurately represents it, addresses the question and handles missing context. A coloured evidence label cannot answer all of those questions at once.
Consider a fictional reference query about a newly revised pathway. A tool could retrieve an older, authoritative document and summarise it faithfully. A citation existence check would pass. An update check would fail. A reviewer who knows only the older document might also accept it. The important issue is access to the relevant current reference, rather than the fluency of either model's explanation.
What Vera says its judges do
In its 25 October methodology post, Vera describes prompt-engineered judges informed by practising clinicians' blinded comparisons and written reasoning. On 50 held-out comparisons it reports 82% agreement with clinician verdicts for clinical accuracy and 86% for references. It also describes reversing answer order, checking proposed releases and sampling live answers daily, with flagged cases sent to its team.
These are company-reported process and agreement measures. They do not mean that 82% of all live clinical answers are correct, that every response is reviewed by a clinician or that future releases cannot regress. The relevant denominator is a set of comparisons used to assess the judge, rather than every question a clinician might ask.
Why automated review is worth developing
A repeatable review process can look for the same classes of error across many outputs. It can help a team notice changes after an update, group recurring problems and direct human attention towards cases worth investigating. This is particularly useful when a product changes more frequently than a one-off external evaluation can be repeated.
The potential benefit comes from consistency and coverage. A fixed question can be rerun when the retrieval process changes. A previously missed qualification can become part of a permanent regression set. A review team can ask whether a correction addressed a wider pattern rather than one conspicuous example.
Automated review is also useful when it records reasons. An unexplained score tells a developer little about what to fix. A report that points to an outdated document, a missing qualification or an unsupported statement creates something an appropriate reviewer can examine. The workflow matters as much as the number at the top of the report.
Agreement is a starting point for questions
An agreement percentage summarises how often two assessment processes reached the same verdict on the chosen sample. It needs context. Were the examples straightforward or difficult? How were ties handled? Which specialties and countries were represented? What happened in the disagreements?
A small held-out set is valuable for an initial check, but a few cases can change its percentage noticeably. Publishing an uncertainty interval and the distribution of case types would help readers understand how much precision to attach to a point estimate. It would also help to know whether the system performs differently on omissions, citation problems and contextual errors.
The severity of disagreements matters too. Missing a minor formatting preference and missing a consequential qualification should not be treated as interchangeable. A sensible review asks whether the judge's misses cluster around a particular kind of problem. That gives the team a clearer target than simply trying to increase an overall score.
Agreement with clinicians is not agreement with an infallible reference. Expert reviewers can reasonably disagree when the question is underspecified. Their written reasons are therefore part of the evidence. A process that preserves disagreement can teach more than one that hides it behind a majority verdict.
Several judges can still share the same blind spot
Using different models may reduce dependence on one judge's habits. It does not guarantee independent errors. Reviewers may draw on similar source material, interpret the same ambiguous instruction in the same way or fail to recognise the same missing context.
Imagine an answer and all its judges assume that a recommendation applies in the same country. They might agree while overlooking that the user works elsewhere. A separate jurisdiction check would add information that another vote on the same wording might not supply.
This is a reason to design different checks for different failure modes. A source-version check, a structured review of missing context and an expert assessment of applicability address different questions. The goal is a set of complementary safeguards, with clear responsibility for unresolved cases, rather than a larger number of similar scores.
Ask what happens after the alert
Detection is only the beginning of a quality process. Someone must decide whether the flag is valid, identify the affected material and choose an appropriate correction. The team must then establish whether the change solves the problem without causing another one.
For a provider discussion, ask for a worked example. It need not contain patient information. A synthetic case can show the original output, the automated concern, the human disposition, the correction and the later test. That sequence is easier to assess than a broad promise to monitor quality continuously.
| Question for the provider | Why it helps |
|---|---|
| Which errors trigger escalation? | Shows whether seriousness affects priority |
| Who resolves disputed findings? | Establishes responsibility for difficult cases |
| How are source updates detected? | Separates current retrieval from plausible old answers |
| What is retained in the regression set? | Shows whether a correction has a lasting check |
| How are local-policy conflicts handled? | Tests applicability beyond general factual accuracy |
| How can a user report a problem? | Connects routine use to the improvement process |
An adoption team should request evidence appropriate to its intended use. A general reference pilot and a service connected to local documents create different responsibilities. Neither should assume that an automated quality report supplies all the governance it needs.
A practical answer-inspection exercise
Use a non-identifying reference question with a known source and version. Before generating an answer, write down the necessary elements and any context that would change its interpretation. Keep this list short enough to inspect carefully.
After the answer appears, map each important claim to a supporting passage. Mark unsupported claims, missing elements and unclear assumptions separately. Then change one contextual feature, such as the applicable country or the question's scope, and see whether the answer acknowledges it. This is a structured learning exercise, not a validated performance study or permission to experiment with live patient decisions.
The exercise can reveal why an apparently good response is difficult to verify. It also gives a clinician a concrete issue to report. Saying that an answer feels wrong is less useful than identifying the unsupported phrase or superseded document.
What this means for iatroX users
iatroX's methodology describes source-grounded clinical reference. Its learning workflow also offers a way to turn uncertainty into deliberate study. For a UK clinician, that connection can make checking an answer and understanding the underlying reasoning part of the same routine.
It does not remove the need to inspect sources, and iatroX was not evaluated by Vera's judge study. A favourable comparison rests on the workflow a clinician wants: UK reference alongside structured learning. The same questions about currency, applicability and correction should be asked of iatroX and every competing service.
FAQs
Does Vera's reported 82% agreement mean 82% clinical accuracy?
No. It describes agreement between a judge and clinician verdicts on held-out answer comparisons. It is not an accuracy estimate for every live response.
Does a clinician review every Vera answer?
The company describes automated checks and sampled monitoring informed by clinician reviews. That should not be presented as a clinician approving each live answer.
Can several AI judges replace a clinical evaluation?
They can support review and monitoring, but agreement among models does not establish the absence of shared errors. Evaluation still needs appropriate tasks, reference standards and investigation of failures.
Explore iatroX's clinical reference and learning tools, and make source inspection a regular part of evaluating any medical AI answer.