It can do so when licences, logins or generated notes stand in for improved care. That is a measurement risk, not evidence that particular NHS organisations are manipulating targets. A useful evaluation asks whether the actual assisted service performs better than its realistic alternative, including the work required to check and correct it.
This analysis is dated 10 October 2026 and focuses on NHS services in England. It proposes a measurement framework rather than reporting a new evaluation. NHS England's ambient-scribing registry explicitly distinguishes supplier listing from endorsement; similarly, procuring a listed product should not be confused with demonstrating benefit from its local use.
A licence, a login and a completed episode answer different questions
A licence count describes potential access. A login shows that someone entered a system. A generated note shows that a process ran. None, alone, establishes that a clinician finished sooner, a patient understood their plan or an administrative backlog decreased.
These measures are not useless. They can reveal whether implementation has started, whether training is reaching users and whether access problems are preventing a fair evaluation. The mistake is promoting an implementation measure into an outcome claim without showing the connection.
Consider a fictional respiratory clinic. Nearly every clinician opens the new documentation tool, but some continue writing their usual note because the AI draft needs substantial revision. High adoption would accurately describe account use. It would not establish that the original work had been replaced. Conversely, selective use in complex consultations might produce meaningful value despite a lower overall utilisation rate.
The first dashboard question should therefore be: what counts as an eligible opportunity to use this tool, and what happened in those opportunities?
Define the benefit before counting activity
Suppose a service wants to reduce unfinished documentation without shortening patient conversations. Its primary outcome should relate to that objective, not to the number of notes generated. A plausible assessment would examine documentation completed within the intended workflow, after-hours work, clinically important corrections and patient experience.
A different service might be trying to reduce missed follow-up. There, the relevant unit is an appropriately completed follow-up episode, not an outgoing reminder. Successful contact, understood instructions, responsibility for non-response and justified decisions not to proceed all belong in the measurement.
Write the anticipated mechanism in ordinary language: "The tool drafts material that staff currently type, and the time needed to check it is less than the work removed." That sentence identifies an assumption to test. It is more useful than a general ambition to become an AI-enabled organisation.
Build a balanced board dashboard
The following is a proposed dashboard, not a mandated NHS reporting template. Each measure needs a definition, collection method and interpretation agreed before results are presented.
| Domain | Decision-useful measure | Misleading shortcut to avoid |
|---|---|---|
| Appropriate use | Use among suitable encounters, with reasons for non-use | All consultations treated as equally suitable |
| Clinical quality | Material omissions, corrections and resulting actions | Counting only invented facts |
| Workload | Total staff effort across the episode | Generation time presented as time saved |
| Patient experience | Understanding, objections and access to an alternative | Satisfaction among completers alone |
| Operational completion | Unresolved tasks and accepted handovers | Messages sent or drafts produced |
| Sustainability | Continued benefit after initial support reduces | Launch-week enthusiasm |
The board should also see uncertainty. Small samples cannot support precise claims about uncommon harms. An apparent improvement may reflect changing case mix, staffing, appointment length or selective participation. Report those explanations rather than allowing a clean chart to imply causation that the evaluation cannot establish.
Count the checking, including work done by other staff
A fictional clinic may reduce doctors' typing while increasing administrative correction of names, destinations and attachments. That could still be worthwhile, but it is not equivalent to eliminating the work. The evaluation should identify who gains time, who inherits responsibility and whether the receiving team has sufficient capacity and authority.
The GMC's February 2025 account of qualitative research with doctors using AI included verification as an additional consideration in the workflow. Those interviews do not establish a universal time penalty or saving. They support asking users what work remains after generation, rather than assuming that output speed answers the question.
For a local study, distinguish reading, checking, editing, resolving uncertainty and correcting downstream material. Observe some sessions rather than relying entirely on recollection. Record abandoned attempts too: a failed draft still consumed time even when it never became an approved document.
Include people who refuse, struggle or disappear
A patient who declines recording should not disappear from the evaluation. Neither should a clinician whose consultation type makes the tool less useful. Excluding these encounters may leave a favourable picture of the subset easiest to automate while concealing the burden of providing the whole service.
A proposed patient-experience review should examine how alternatives operate in practice. Does declining the tool lead to inconvenience, delay or a different route? Are translated explanations available where needed? Can a patient correct an error without understanding the underlying technology?
Earlier digital access research supplies a useful caution about implementation. HSSIB's July 2024 investigation of online consultation tools examined patient-safety and workforce issues around those services. It was not an evaluation of current generative systems. It nevertheless illustrates why access arrangements and workflow consequences deserve measurement alongside the software's immediate function.
Make the dashboard capable of changing a decision
Before a pilot begins, specify which findings would support expansion, modification or restriction. Better documentation with an unmanageable review queue might justify redesign rather than scale. Lower workload with more consequential omissions might require a different control. No measurable benefit may justify retaining a simpler workflow.
Avoid a single composite score that allows high activity to cancel out a serious unresolved safety signal. Equally, an isolated error should prompt investigation proportionate to its nature rather than an automatic conclusion that any use is unacceptable. The relevant question is whether risks are understood, controlled and justified by demonstrated value.
For professional learning, the same principle applies. This Journal is published by iatroX: opening a reference page or completing a question is an activity, not proof of retained understanding or better patient outcomes. A useful personal review asks what uncertainty was resolved and whether that understanding transfers to another case. That standard is more demanding, and more informative, than celebrating clicks.
Frequently asked questions
Are adoption figures useless when evaluating NHS AI?
No; they help explain implementation and access. They should be reported alongside, rather than substituted for, measures of quality, workload, patient experience and completed care.
What is the best measure of AI time savings?
There is no universal measure, but total effort across the relevant workflow is more informative than generation speed. Include checking, correction, failed attempts and work transferred to other staff.
Should low usage automatically end an AI pilot?
No; low usage may reflect appropriate selection, poor implementation or limited value. Investigate which explanation fits before expanding, redesigning or stopping the service.
Turn an AI evaluation question into focused professional learning →