A clinical AI update can arrive without changing the button a clinician presses. The note may look familiar while the system producing it has acquired different instructions, a new model or a different way of selecting information. Familiarity with the interface is therefore not evidence that the underlying behaviour is unchanged.
The answer to who checks an update is not simply "the doctor" or "the supplier". The manufacturer needs to assess the changed product. The adopting organisation needs to understand the consequences for its particular use. Clinicians need enough information to recognise relevant differences and report concerns.
Sovereign Health's deployment options make this especially pertinent: greater control over models creates choices about maintenance as well as initial selection. This article proposes a practical change-review process, not a claim that Tandem has disclosed or adopted every step below.
Start by identifying the component that changed
A model replacement is only one possible change. A new instruction can alter what the model treats as important. A retrieval update can change the guidance available. A template can promote a previously optional field into a required entry. An integration can change where the resulting text goes.
These differences suggest different tests. An alteration to a referral destination list does not primarily require another general medical-knowledge benchmark. A changed summarisation instruction requires examining what the note now includes, omits or implies.
Ask the supplier to describe the affected function, the previous and new configuration, the reason for the change and the evidence supporting release. A release note stating only "improved intelligence" is not enough to connect a modification to the work clinicians perform.
Maintenance can be beneficial and still need assessment
Updates may correct errors, close security vulnerabilities or improve performance. Freezing every component indefinitely is not a credible safety strategy.
The relevant question is whether the change alters clinical meaning, information handling, permissions or the work expected of the user. A visibly small change can be consequential, while a substantial infrastructure change may leave some clinical functions unaffected.
The MHRA's July 2026 AVT clarification asks organisations to identify and review functional changes that may alter regulatory status. That does not mean every software patch requires an identical process or a completely new approval.
A proportionate review records why a particular level of testing is sufficient. The reason should follow the change, not a supplier's preference to call it maintenance.
A fictional update that improves readability but loses a distinction
Consider an invented release that makes clinical notes shorter. In testing, most reviewers prefer the concise output. One consultation, however, contains a plan to consider an investigation after obtaining an earlier report. The new note presents the investigation as already requested.
The problem is not brevity itself. It is the loss of a distinction between a conditional plan and a completed action.
A suitable regression case would preserve that distinction as an explicit requirement. The test would examine the final note and any downstream task, rather than award success because the output contains the investigation's name.
This is a constructed example, not a reported Tandem incident. It demonstrates why an improvement on one measure can coexist with a failure elsewhere.
Build a regression set around consequential behaviour
A regression set is a collection of previously specified tasks used to check whether an update has damaged behaviour the service already needs. It should include ordinary work and important exceptions.
| Test case | What should remain intact | What would trigger investigation |
|---|---|---|
| A diagnosis discussed as a possibility | The uncertainty and who expressed it | A confirmed diagnosis appears without support |
| A relative describes their own illness | Correct person attribution | The illness becomes the patient's history |
| An investigation is conditional | The condition and action status | The record implies it has already happened |
| Relevant evidence is unavailable | The limitation is visible | The system invents a supporting source |
| A local pathway is superseded | Current status is respected | An obsolete instruction remains authoritative |
| A field is genuinely unknown | Missingness remains explicit | The template supplies a reassuring default |
| Record transfer is interrupted | Completion remains unresolved | The interface reports unconfirmed success |
| A feature is disabled locally | The approved boundary is preserved | The new release re-enables it silently |
This is an original test-design aid, not a validated universal checklist. The organisation should add examples reflecting its own specialty, languages, records and dependencies.
Record the expected result before running the update. Otherwise, the team may rationalise a changed answer after seeing it instead of identifying a regression.
Check variation, not only one successful run
Some outputs can vary even with unchanged clinical information. A test that succeeds once may therefore give an incomplete account of a configuration.
For tasks where variation matters, repeat selected cases and inspect consequential differences. The aim is not identical wording. It is consistent handling of the facts, limitations and action boundaries relevant to the task.
Also vary how the information is expressed. A clear positive statement, a negated statement and an uncertain recollection should not collapse into the same clinical entry.
A benchmark result from another model configuration cannot answer these questions automatically. The task-specific evaluation guide explains why documentation, recommendation and action require different endpoints.
Divide supplier and local responsibilities explicitly
In the applicable English NHS context, DCB0129 and DCB0160 distinguish manufacturer-side clinical-risk management from deployment and use.
A practical local arrangement can name who receives supplier notifications, who reviews relevant evidence, who assesses the effect on local workflows and who communicates changes to users. Those responsibilities may involve clinical safety, information governance, technical support and the service owner.
The individual clinician should not be expected to discover an undisclosed model substitution by noticing that the prose feels different. Conversely, local staff should report a new pattern of errors rather than assume the manufacturer's testing covered their particular scenario.
For medical-device functions, the MHRA intended-purpose guidance provides a useful scope reference. It is not a complete change-management manual; the applicable regulatory route still depends on the product and modification.
Make the release understandable to its users
A useful notice tells clinicians what changed, when it applies, whether their workflow needs to change and how to raise a concern. It should identify any temporary limitation that affects the work they are expected to perform.
Not every user needs a catalogue of engineering dependencies. A clinician reviewing correspondence does need to know when a new template changes which sections are generated or which fields require explicit confirmation.
The service owner should also know which users and encounters were exposed to the changed configuration. That supports a focused investigation if an issue emerges, without retaining unnecessary patient information merely because an update occurred.
Plan containment and recovery before trouble appears
A previous model version may be available, but rollback should not be assumed. It may have been retired, or returning to it may reintroduce a security or clinical problem.
Agree an alternative response: restrict the affected function, use an approved manual workflow, pause the integration or move to another already assessed configuration. The choice should match the problem and the authority available locally.
After correction, repeat the relevant tests and decide what additional evidence is needed before wider use resumes. A supplier saying that the bug is fixed is important information, but the organisation still needs to understand what that means for its service.
The purpose of change control is not to prevent improvement. It is to ensure that improvement reaches clinical practice without quietly changing the assumptions on which safe use depends.
Frequently asked questions
Does every clinical AI update need a completely new evaluation?
Not necessarily. The assessment should be proportionate to what changed and which functions, risks or intended uses are affected. The basis for that decision should be documented.
Is a better general benchmark score enough to approve a replacement model?
No. The replacement must also be assessed within the relevant product configuration and workflow, including important behaviours that the general benchmark does not measure.
Can we always return to the previous version if an update performs badly?
No. Availability, support and security constraints may prevent rollback. Establish a workable alternative, such as restricting the function or using an approved manual process, before deployment.