Clinical AI can be unsafe by reassuring someone who needs further assessment or by prompting disproportionate investigation and escalation. Mentioning every serious possibility does not solve the problem. A useful evaluation asks whether the recommended action is appropriate, understandable and achievable, then follows the consequences rather than rewarding caution alone.
Two mistakes can share the same underlying weakness
An answer that overlooks uncertainty may confidently reassure. Another may respond to the same uncertainty by recommending every available test. Neither necessarily demonstrates an adequate assessment of what is known, what is missing and which next step would resolve the important question.
The World Health Organization's large multimodal model guidance, announced on 18 January 2024, identifies risks from inaccurate or incomplete outputs and inappropriate reliance. It does not establish that one universal escalation threshold makes every healthcare AI safe.
The practical challenge is to distinguish clinically justified caution from activity generated mainly because the system cannot manage uncertainty. That distinction requires the intended task, patient population and available care pathway to be explicit.
A hypothetical comparison shows the trade-off
The following is a hypothetical teaching example constructed in October 2026, not a test of a commercial product. Imagine a set of 100 fictional encounters, with an agreed reference assessment identifying 20 as requiring urgent escalation and 80 as not requiring that escalation.
| Hypothetical strategy | Urgent cases appropriately escalated | Urgent cases missed | Other cases unnecessarily escalated | Total escalations |
|---|---|---|---|---|
| Strategy A | 18 of 20 | 2 of 20 | 12 of 80 | 30 of 100 |
| Strategy B | 19 of 20 | 1 of 20 | 41 of 80 | 60 of 100 |
Strategy B misses fewer urgent cases but creates substantially more unnecessary urgent escalations in this invented dataset. The table does not tell us which strategy should be deployed. That requires the severity of the missed cases, the consequences of unnecessary escalation, uncertainty around the estimates and the performance of realistic alternatives.
It does show why reporting only "urgent cases detected" can conceal important information. Conversely, reporting only reduced referrals could conceal dangerous under-triage. The units and both directions of error need to remain visible.
Not every unnecessary test is a harmless extra
An investigation has a purpose within a decision pathway. Ordering it without considering how its result would change management can create an unresolved branch rather than a completed assessment.
As a hypothetical mechanism, an unexpected but unimportant result may lead to further tests, patient concern and additional consultations. A test that is appropriate for one population may also be difficult to interpret in another. These possibilities should be evaluated rather than assumed to occur with every AI recommendation.
The relevant buyer question is therefore not "does the system suggest enough investigations?" It is "does each suggestion have an appropriate indication, an interpretation plan and an accountable route for follow-up?"
A proposed review should distinguish a justified investigation, a reasonable option for discussion and an action that is disproportionate to the available information. Combining them as "recommended tests" loses the clinical judgement that the evaluation is supposed to examine.
A safety-net paragraph does not repair a contradictory answer
Consider a fictional patient receiving a reassuring explanation followed by a long list of situations in which they should seek help. The final paragraph may contain sensible general information, while the preceding message has already conveyed that the problem is settled.
The evaluation should assess the overall meaning of the interaction. Does the patient understand the uncertainty and the action required, or mainly remember the confident reassurance? Does the escalation advice relate to the actual question, or is it a standard paragraph attached to every response?
The opposite design also needs testing. A list of serious diagnoses without context may leave the user unable to distinguish a possibility worth considering from an immediate instruction. More alarming wording is not automatically better safety communication.
These are proposed communication tests, not claims that a particular product has caused either outcome. They require assessment with appropriate users, not only a clinician marking whether a warning phrase appeared.
The reference assessment must allow defensible variation
A triage benchmark can look precise while its reference labels conceal disagreement. Some cases have a clear required action; others permit more than one reasonable approach depending on information, setting and patient preferences.
A proposed evaluation should therefore document how the reference assessment was formed and how uncertain cases were handled. Reviewers should distinguish clinically meaningful disagreement from differences in wording or defensible alternatives. Otherwise, a rigid label can reward conformity rather than appropriate care.
The evaluation should also include cases with missing information. A system that appropriately asks for clarification should not automatically be marked worse than one that guesses the expected category. However, endless questioning should not be counted as success when timely action is needed.
This makes the endpoint more demanding but more useful: an appropriate next step, with uncertainty handled in a way that supports the patient's actual journey.
Measure downstream work and completed escalation
A message saying "contact a clinician" is not evidence that the patient obtained the necessary assessment. Likewise, a test suggestion is not a completed investigation or a reviewed result.
For an approved prospective evaluation, follow whether escalation was accessible, whether advice was understood, whether additional contacts were required and whether responsibility remained clear. Count repeat work and unresolved tasks alongside immediate recommendations.
DECIDE-AI, published on 18 May 2022, provides a reporting framework for early live evaluation of AI decision support, including its clinical and human context. It is useful for distinguishing a model result from a functioning service, but it does not supply a universal threshold for acceptable under- or over-triage.
Any comparison should use realistic current care, including its limitations. An imaginary clinician who never misses a problem and never orders an unnecessary test is not a meaningful benchmark.
Practise proportionality, not maximum activity
A useful learning exercise presents a fictional AI plan and asks the clinician to justify every proposed action. What uncertainty would it resolve? What would change afterwards? What important issue remains unaddressed? Which step is premature?
As described in iatroX's October 2026 product information, question-based learning and Socratic Tutor can support discussion of reasoning after an attempted question. That is an educational format, not evidence that iatroX has been validated to optimise triage or prevent unnecessary investigation.
The constructive objective is not to make every answer more reassuring or more cautious. It is to make the relationship between evidence, uncertainty and action clearer. A system should be judged by appropriate care and its consequences, not by how forcefully it recommends doing more.
Frequently asked questions
Is an AI safer if it always recommends urgent assessment?
Not necessarily. Excessive escalation can create unnecessary burden, so safety evaluation should examine both missed urgent needs and unjustified escalation.
Does mentioning every serious diagnosis count as good safety-netting?
No. The user needs an understandable account of what to do and why, not an uncontextualised list of possibilities.
What should an AI triage study report besides accuracy?
It should specify the unit of analysis, severity and direction of errors, uncertainty, completion of escalation and downstream consequences. Appropriate alternatives and the actual care setting also matter.
Practise explaining why a clinical action is proportionate →