A healthcare AI can produce supportive language without establishing that its clinical interpretation is correct. Warmth may make an interaction easier, but it should not become evidence of competence. The design challenge is to help people feel heard while preserving their understanding of uncertainty, limitations and the need for appropriate human care.
What an empathy rating actually measures
A study published in JAMA Internal Medicine on 28 April 2023 compared physician and chatbot responses to 195 questions from a public online forum. Evaluators rated the chatbot responses more favourably for quality and empathy. This was an assessment of written responses, not a trial demonstrating better patient outcomes or a superior therapeutic relationship. See the original study.
That distinction does not make the result uninteresting. Helpful wording, acknowledgement of concern and clear explanations deserve attention. It does mean that an attractive communication score should not be used to validate diagnosis, prescribing, escalation or long-term support.
Nor does the comparison establish that clinicians are inherently less caring. The responses were produced in a particular setting, under different practical conditions from an ordinary consultation. A reader should ask what was evaluated before converting an empathy headline into a claim about the quality of care.
Caring language and justified reassurance are different
Consider a fictional patient describing an unresolved symptom. One response says: "That sounds worrying, but you can relax because this is almost certainly harmless." Another says: "I can understand why this is concerning. The information here is not enough to establish the cause, so the next step is to clarify what assessment is needed."
Both use supportive language. Only the second preserves the uncertainty built into the example. The problem with the first is not that it is kind; it is that kindness is attached to an unsupported conclusion.
This is an invented communication exercise, not an observed output from a named product. It illustrates a useful review question: if the reassuring phrases were removed, would the underlying clinical claim still be justified?
A system can also overcorrect by responding coldly or listing every frightening possibility. Neither emotional distance nor maximal alarm is a reliable proxy for safety. The aim is proportionate support connected to a defensible next step.
The interface can encourage a particular interpretation of the relationship
A product's name, first-person language, conversational memory and presentation may suggest different roles. A patient may interpret the same sentence differently when it appears to come from an information tool, a care service or a familiar personal companion.
That is a design hypothesis to evaluate, not proof that any one interface produces dependency. The practical question is what the user believes the system can do. Does the person think a clinician has reviewed the answer? Do they believe someone is monitoring deterioration? Do they assume that a friendly follow-up message means clinical responsibility has been accepted?
A service should test those beliefs directly. Asking whether users liked the interaction is not enough. A person can enjoy an experience while misunderstanding its boundaries.
The WHO guidance released on 18 January 2024 identifies automation bias as a concern for patients and professionals. It is a governance warning, not an estimate of how often a specific caring chatbot will be trusted incorrectly.
Appropriate reliance is a better target than maximum trust
Maximum trust would mean persuading the user to accept the system consistently. Appropriate reliance means accepting support where it is useful and recognising when the information, task or circumstances require something else.
For a hypothetical symptom-information service, successful behaviour might include helping a user organise questions, identify uncertainty and obtain suitable assessment. Ending the conversation to seek care could be the correct outcome, even though it reduces engagement with the product.
Conversely, repeated return visits to the same unresolved concern should not automatically be celebrated as loyalty. They could represent useful continuing support, uncertainty that has not been resolved, or a mismatch between what the user needs and what the service can provide. The interpretation requires evidence.
A proposed evaluation would therefore measure understanding of role, recognition of limitations, appropriate escalation and subsequent action alongside satisfaction. It would not treat longer conversations or stronger emotional attachment as clinical success in themselves.
Test the person-plus-tool interaction
The distinction between a model's capability and its practical use has empirical support. A randomised study published in Nature Medicine on 9 February 2026 involved 1,298 participants using medical scenarios. Strong model-only performance did not translate into better participant decisions with the tested assistance. This was not a trial of a therapeutic relationship or a deployed treatment service. See the research team's account and paper.
For caring language, the corresponding research question is whether supportive presentation helps users understand and act on appropriate advice, or makes unsupported advice harder to question. A controlled study could compare otherwise identical explanations with different communication styles while keeping the underlying clinical content fixed.
It should also test uncertainty. Do users understand the difference between "one possibility" and "the established cause"? Can they explain what information remains missing? Do they know whether a human is expected to review the conversation?
These are proposed endpoints, not reported results of the cited trial. They would help distinguish a pleasant experience from a reliably understood one.
Preserve the patient's ability to disagree
A helpful system should not make disagreement feel like breaking a relationship. Users should be able to reject an interpretation, correct remembered information or request a different form of help without encountering guilt-inducing or possessive language.
For clinical services, the same principle applies to handover. A patient should be able to tell a clinician that an AI answer did not fit, or that its reassurance made them uncertain about seeking help. The clinician can investigate the content without ridiculing the patient for finding the interaction supportive.
The GMC's decision-making and consent guidance, checked on 10 October 2026, supports a dialogue responsive to what matters to the individual. An editorial application is to examine how the AI interaction influenced the person's understanding, not simply whether its conclusion agrees with the clinician's.
Keep educational encouragement separate from clinical assurance
A tutor can encourage a learner to continue working through a difficult concept without assuring them that an unverified clinical answer is safe. As described on 10 October 2026, iatroX Tutor uses the learner's attempted question and reasoning as the starting point for discussion. That is a learning design, not proof that its feedback is always correct or that it prevents overreliance.
The same review principle applies: praise should identify something the learner actually did, and an explanation should remain open to challenge. "That is a sensible distinction to examine" is different from declaring a learner ready for unsupervised practice.
The wider objective is not to remove warmth from healthcare AI. It is to stop warmth doing work that belongs to evidence, appropriate assessment and an accountable care pathway.
Frequently asked questions
Does an empathetic AI response show that the medical advice is correct?
No. Communication quality and clinical correctness are separate dimensions, and each requires its own assessment.
Should healthcare AI avoid supportive language?
No. Supportive language can be useful, provided it does not conceal uncertainty, overstate capability or discourage appropriate human help.
What should a trust evaluation measure besides satisfaction?
It should examine users' understanding of the system's role, recognition of limitations and the appropriateness of subsequent action. Engagement alone cannot establish those outcomes.