Simulation can reveal whether an AI asks relevant questions, handles a controlled uncertainty or responds appropriately to a designed challenge. It cannot reproduce every feature of real clinical care. The useful distinction is not valuable research versus worthless role-play, but which conclusions a simulated encounter supports and which require evaluation with the intended users and patients.
Simulation gives investigators control for a reason
A controlled case can expose several systems to the same information and test a particular failure repeatedly. Investigators can introduce a contradiction, an interruption or a difficult handover without putting a patient at risk. They can also examine rare but important scenarios that would be difficult to collect prospectively.
Those advantages depend on design. The simulated person has to provide information in a way that meaningfully tests the capability under study. If the interaction supplies every important fact whenever the AI asks a vague question, it may reward behaviour that would not work as reliably in practice.
A simulation result should therefore describe the patient model, available information, scoring rules and boundaries. "Tested on simulated patients" is not enough to establish what challenge the system actually faced.
Actors and simulated language models have different limitations
A trained actor can portray a case consistently, but knows the scenario and may respond within its planned boundaries. A language-model patient can generate varied dialogue, but its behaviour is itself produced by another model and its instructions. Neither should be treated as a neutral window into all real patient behaviour.
A Nature Medicine study published in February 2026 compared simulated interactions with how members of the public used medical assistants in fictional scenarios. The simulated evaluations did not adequately predict the human interaction results. That finding concerns the tested setup, not every simulation method or clinical use.
The lesson is to examine the simulator as part of the experiment. Its willingness to disclose information, interpretation of questions and response to uncertainty can materially influence the result.
A fictional case that becomes too cooperative
Imagine a fictional consultation exercise in which the patient has a symptom, a relevant recent change and a practical barrier to follow-up. The AI asks a broad opening question. The simulated patient immediately supplies all three details in a neatly organised paragraph.
The same system may then produce an excellent plan. However, the exercise has not tested whether it can identify the need to ask about the recent change or recognise that the patient may struggle with the proposed follow-up. The information arrived without the system having to elicit it.
A revised exercise could vary when and how those details become available. The patient might mention the barrier only after the plan is discussed, or require clarification before answering a question. The goal is not to make the simulation arbitrarily difficult. It is to test the intended information-gathering work.
This is a proposed teaching and evaluation scenario, not an observed failure of a named AI product.
Real encounters include purposes beyond the case script
A patient may arrive with several concerns, an expectation shaped by earlier care or uncertainty about what the clinician needs to know. They may revise their account as the conversation develops. The eventual priority may differ from the problem initially selected for assessment.
An evaluation should ask whether the simulated interaction permits those changes or forces the patient to remain inside a single predefined diagnostic puzzle. Both designs can be useful, but they support different claims.
The same applies to emotion and communication. A simulation can test responses to a designed expression of distress, but a favourable rating does not establish how the interaction affects a person's understanding, confidence or subsequent behaviour outside the exercise.
Inspect selection before interpreting performance
Which cases entered the simulation library? Which patients and clinicians were represented? Were difficult communication needs, incomplete records or multiple concerns excluded? Did investigators select tasks for which the system was already known to work well?
Selection does not automatically invalidate a study. A narrow evaluation can be appropriate for a narrow intended use. The problem is extending its conclusion to a wider service without acknowledging the change in population or task.
The good machine learning practice principles, published in October 2021, emphasise relevant testing and representative data. Simulation design should therefore be interpreted alongside the population and conditions the product is intended to serve.
A simulated success still leaves operational questions
The AI may produce an appropriate plan in a test environment where appointment availability, record access and follow-up are guaranteed. Real implementation may involve missing records, delayed responses, interruptions and external services with their own constraints.
A proposed next-stage evaluation should follow the complete workflow under supervision. Measure which information clinicians repeat, which outputs they correct, how exceptions are handled and whether patients understand what happens next. Do not treat an impressive transcript as proof that the surrounding service works.
The DECIDE-AI guideline, published in 2022, addresses reporting of early live clinical evaluation and human factors. It helps articulate the transition from controlled testing to actual use, without functioning as a universal certification scheme.
Keep educational simulation valuable without overstating it
For learners, simulation provides opportunities to rehearse reasoning and communication, make mistakes in a controlled setting and revisit a specific weakness. The educational question is whether the practice helps the learner perform better on a later, relevant task.
According to iatroX's September 2026 launch specification, its simulations include practice and uninterrupted exam modes, with transcript-linked feedback and Tutor-led remediation. These are educational design features, not proof of clinical competence, examining-body endorsement or predicted pass rates. They also do not replace bedside examination or procedural practice.
A useful training sequence combines independent performance, feedback and a new case requiring the same underlying skill. Human observation remains important where the assessment concerns interpersonal behaviour, physical examination or real-world professional judgement. Improvement within a familiar platform should not be treated as the only evidence of learning.
Read the conclusion at the same level as the experiment
A strong simulation paper can establish that a defined system performed well under specified conditions and justify further investigation. It need not prove everything to be valuable. The responsible interpretation is to preserve those conditions when describing the result.
The question for the next study is what uncertainty remains: the simulator's realism, the users' behaviour, the service's operation or the effect on patient outcomes. Naming that uncertainty makes the evidence programme stronger rather than diminishing the work already completed.
Frequently asked questions
Does simulation provide useful evidence about clinical AI?
Yes. It can test defined capabilities and failure scenarios under controlled conditions, provided the case design, simulator and scoring limitations are reported.
Does success with simulated patients validate real-world autonomous care?
No. The actual user interaction, patient population, operational pathway and relevant outcomes require further evaluation.
Can educational simulations replace bedside practice?
No. They can supplement reasoning and communication practice, but they do not establish the physical examination, procedural and interpersonal capabilities that require appropriate real-world supervision and assessment.