Imagine a clinical reference tool becoming so useful that its users ask fewer questions. It answers the initial question clearly, identifies the relevant source and makes its limits intelligible. The clinician finishes the task and closes the window.
On a dashboard obsessed with engagement, that could look disappointing. Fewer messages. Shorter sessions. Less time inside the product. In the clinic, it could be excellent news.
Now imagine the opposite. An answer is vague, so the clinician asks again. A citation does not support the wording, so they challenge it. The revised answer changes direction, prompting another check. The conversation grows. So does the query count.
The second product has generated more activity. Whether it has generated more value is a different question. Healthcare AI should be embarrassed by any measurement system unable to tell these experiences apart.
The click is an event, not an achievement
A query proves that an interaction occurred under whatever definition the service applies. It does not reveal whether the question was answered, whether the clinician understood the result or whether the answer helped complete useful work.
That makes query volume a reasonable operational measure. It can help describe demand, capacity requirements and changes in usage. Its weakness appears when it is promoted into a universal measure of product quality.
Suppose, in a hypothetical comparison, two tools help clinicians complete the same kind of task to the same independently assessed standard. Tool A typically requires two exchanges. Tool B requires eight because users repeatedly have to correct omissions. B should not receive four times the credit for being four times as conversational.
The outcome must actually be checked, of course. A confident two-message exchange ending in a wrong answer is no triumph of efficiency. Nor is a user leaving because they cannot find the feature. Shorter interaction is favourable only when the work has been completed well.
This is the missing question behind many large consultation counts: what happened at the end of the conversation?
A useful product gives attention back
Clinical time has competing uses. A product that requires attention has to justify the demand against the job it helps accomplish. Keeping someone talking is an expensive achievement when they have a patient waiting or a record to finish.
The HEART research framework describes a process for mapping product goals to user-centred metrics. The underlying measurement question is straightforward: decide what success means for the person before choosing what to count.
For a reference task, success might mean locating applicable evidence, understanding its relevance and reaching an appropriate stopping point. For documentation, it might mean a completed, accurate record after review. Neither goal necessarily benefits from another minute inside the interface.
A product team could therefore report useful completion alongside the amount of effort required. It could examine how often users repeat a question, return to resolve the same uncertainty or leave to repair the answer elsewhere. Those would be proposed measures of the work, not proof that any existing vendor already collects them reliably.
Commercial growth remains legitimate. The mistake is assuming that every extra interaction represents more growth in usefulness.
Sometimes the right answer ends the conversation
An effective system should be able to say that the available material does not resolve the question, that a missing fact changes the interpretation or that the task requires a different source of information. Those responses can shorten a session by making its limits clear.
That is not the same as abandoning the user. A useful stopping point explains what remains uncertain and what would resolve it. Endless conversational reformulation can be less helpful than one precise account of the missing evidence.
Consider a hypothetical literature question whose answer depends on whether a study population matches the patient group under discussion. A system might keep producing broader summaries. Or it might identify the relevant eligibility restriction immediately, allowing the clinician to examine the source and decide applicability. The second route could generate fewer prompts while supporting more thoughtful work.
Necessary verification still belongs in that work. The aim is to reduce avoidable repair, not reward skipping the checks that make a conclusion defensible. Counting the entire task helps keep that distinction visible.
Measure the work beyond the window
The AHRQ workflow toolkit treats workflow as physical and mental tasks spanning people, organisations and different stages of a visit. That wider view is particularly useful when the apparent efficiency is confined to one screen.
A short AI interaction may create a long downstream job. Someone has to open the cited material, reconcile conflicting suggestions, enter the decision into the record and communicate it. Moving work beyond the product's analytics does not remove it.
Conversely, a slightly longer interaction may prevent later duplication. A clear explanation shared with a colleague might avoid a second search. A well-structured answer may make documentation easier. The correct unit of evaluation is the completed task and its relevant consequences, not the portion visible to one supplier.
A practical evaluation could observe a sample of ordinary tasks, record total completion time and assess whether the result meets a predefined standard. It should also record unresolved tasks and work transferred to others. Otherwise, a product can appear efficient merely by exporting inconvenience.
Education has a different rhythm
Learning creates a useful complication. A good tutor may deliberately ask another question instead of supplying the answer immediately. The additional interaction can be the point: the learner has to retrieve knowledge, explain a choice or identify why an attractive alternative is wrong.
In Roediger and Karpicke's experiments, students who practised recalling prose showed better delayed retention than those who restudied it, despite the latter's greater confidence. That finding supports examining retained learning rather than equating a smooth study session with success.
It does not establish that any particular AI tutor improves clinical performance. It suggests a better question for evaluation: what can the learner do later, when the help is absent?
The iatroX Tutor and Q-bank offer different ways to engage with questions. A meaningful assessment of educational usefulness would look beyond the number attempted. Can learners explain the reasoning, apply it to a changed scenario and recognise when their knowledge is incomplete?
A longer teaching exchange can be worthwhile. Repeatedly asking the same basic question without developing independent understanding may be less so. The number of messages cannot settle that distinction.
Lower use deserves an explanation
A fall in queries could mean successful task completion, retained learning, declining trust, seasonal workload, a broken interface or a move to another product. The title of this article is a possibility, not a licence to celebrate every downward chart.
The answer is to connect activity with evidence about experience and outcomes. Ask why users stopped. Check whether they completed the task. Compare like tasks and account for changes in difficulty. Watch for users who disappear after an unresolved problem as carefully as those who keep returning.
The ambition should be a product that is used as much as the work warrants. Sometimes that means frequent use because the tool reliably helps with many tasks. Sometimes it means a brief, decisive visit. Sometimes a successful learning session should reduce the need to ask the same question tomorrow.
Clinical AI does not need to win a contest for the longest conversation. It needs to make the clinician's work better. A company confident in that contribution should be willing to measure the moment its user has enough and can get on with the job.
Frequently asked questions
Does this mean lower query volume is always better?
No. Lower use can reflect successful resolution or a product people have abandoned. Interpret query volume alongside task completion, quality, user experience and the reasons people return or leave.
Can more follow-up questions be a positive sign?
Yes. They may reflect a complex task, useful exploration or productive teaching. They may also reflect corrections and unresolved confusion. Evaluating the purpose and result of the follow-ups is more informative than counting them alone.
How should an AI tutor's usefulness be assessed?
Look for learning that survives beyond the assisted session: later recall, sound reasoning on new examples and appropriate recognition of uncertainty. Engagement can support learning, but it cannot substitute for evidence that learning occurred.