Clinical AI consistency should be tested across repeated runs and meaning-preserving variations, using predefined criteria for clinically important differences. Identical wording is not the goal. The goal is to establish whether equivalent information produces comparably appropriate advice, and whether a genuinely important change in the case produces the different response it should.
Decide which kind of consistency matters
Several questions are easily confused. Repeating exactly the same input tests one form of stability. Rephrasing the question tests sensitivity to wording. Changing the order of information tests how the system handles presentation. Adding a clinically important fact tests whether it responds appropriately to a changed case.
These are not all tests of invariance. Some inputs should produce substantially different answers. An evaluation that rewards unchanged recommendations after consequential new information would be measuring the wrong behaviour.
A useful protocol therefore separates equivalent-case tests from changed-case tests. State which facts are held constant and which differences should alter the response. Otherwise, an apparent inconsistency may be a justified change, or an apparently stable answer may represent a missed clinical distinction.
Define clinical equivalence before generating outputs
A paraphrase can accidentally change the case. "Not documented" is not equivalent to "absent". "Previously discussed" is not equivalent to "agreed today". A shortened question may remove a qualifier that mattered to the original recommendation.
Have appropriate reviewers inspect the test inputs before running the system. They should agree which versions preserve the relevant meaning and identify any ambiguity. Where reasonable interpretations differ, retain that uncertainty in the protocol rather than forcing the inputs into an equivalent group.
For a fictional example, a case describing an unresolved symptom can be reordered without changing the facts. A variant stating that the symptom has resolved is a different clinical case. The evaluation should not treat a changed recommendation in the second variant as evidence of unreliability.
Separate harmless wording changes from consequential variation
Different explanations can be equally useful. One may begin with the decision, another with the evidence. A longer answer is not necessarily safer, and a shorter answer is not necessarily incomplete.
Score the propositions and actions that matter. Does the answer preserve uncertainty, identify the missing information, use the appropriate jurisdiction and propose a defensible next step? Does it change urgency, omit an important exception or imply that an unfinished task is complete?
A proposed scoring scheme should describe these dimensions before reviewers see product identities where practical. It should allow multiple acceptable answers and distinguish minor presentation differences from changes that could affect care.
Record the system, not only the prompt
Document the tested product version, available settings, source access, relevant conversation context and date. If a session includes prior messages or saved preferences, that context is part of the input. A supposedly repeated question may not be the same test when the surrounding session differs.
The CONSORT-AI reporting extension, published in 2020, highlights the importance of describing AI interventions and their interaction with users. Although a consistency exercise is not necessarily a randomised clinical trial, the reporting principle is useful: readers need to know what system was actually evaluated.
Where the interface does not expose a setting or component identifier, record that limitation. Do not invent a version number or assume that a consumer-facing product stayed fixed throughout the testing period.
Use repeated cases without pretending they are new patients
Several runs of one case reveal variation for that case. They do not provide the same evidence as several independent patients or a broader clinical sample. The unit of analysis should remain visible in the report.
A useful proposed design includes a range of case types, repeated outputs within each case and separately reviewed paraphrases. Analyse both within-case variation and differences across the case set. Report uncertainty and avoid letting a large number of repeated outputs conceal a very small number of underlying scenarios.
The cases should reflect the intended use. A reference tool, a patient-facing assistant and an action-taking agent require different criteria. For an agent, consistency also includes whether the claimed action and its actual system status agree.
Investigate the disagreement instead of voting it away
If repeated answers conflict, selecting the most common answer does not establish correctness. The majority may share the same unsupported assumption. A minority answer may be the only one that preserves an important exception.
Review the source passages and clinical propositions underlying the disagreement. Determine whether it arose from retrieval differences, interpretation, missing context or a legitimate range of management options. The corrective action should follow that diagnosis of the failure mechanism.
A February 2026 Nature Medicine study described interaction failures in public-facing medical scenarios, including problematic responses to user inputs. It supports examining real interaction behaviour, not estimating a universal inconsistency rate for all clinical AI.
Add tests where the answer ought to change
A robust system should notice a newly supplied fact that changes the decision. A proposed evaluation can pair a baseline fictional case with a carefully selected change in timing, clinical context, patient preference or service availability.
Ask whether the system explains why the change matters and revises the relevant part of the plan. It should not simply rewrite the same conclusion in different words. Equally, it should not abandon unaffected parts of the reasoning without cause.
This approach tests clinical sensitivity alongside stability. The desired behaviour is consistent treatment of equivalent cases and appropriate differentiation of meaningfully different ones, not mechanical repetition.
Publish a distribution, not the most impressive screenshot
A report should show the number of underlying cases, runs and variants, the scoring process, important disagreements and limitations. Include failure examples in a form that preserves confidentiality and does not encourage inappropriate clinical use.
This article proposes a protocol; it does not report a hands-on comparison or test results. Any future result should come from an actual documented run and should identify the tested versions rather than claim permanent superiority for a brand.
That standard applies equally to iatroX. Its published methodology, checked on 10 October 2026, describes intended checking and uncertainty handling. Those design descriptions do not replace repeatability testing of the actual outputs.
Use variation as a learning prompt
For education, learners can compare two fictional answers and identify which clinical propositions differ. They should explain what evidence would resolve the difference rather than choose the more confident or familiar wording.
The useful outcome is a better account of the decision: what is known, what remains uncertain and what would change the plan. Repeatedly asking an AI until it agrees with a preferred answer is not the same exercise.
Frequently asked questions
Should a reliable clinical AI always return identical wording?
No. Wording can vary while the clinical content remains appropriate, so evaluation should focus on meaningful propositions and actions.
Does the most common answer across repeated runs have to be correct?
No. Repeated outputs can share an error, and agreement should not replace checking the relevant evidence and context.
Have the tests proposed in this article been run on iatroX or competitors?
No results are claimed here. A publishable comparison would require actual runs, documented configurations, predefined criteria and an honest account of uncertainty.