A useful clinical AI safety case connects a defined workflow to its hazards, the controls intended to address them and the evidence showing whether those controls work. A statement that clinicians will check the output is not enough unless the workflow makes that checking possible and its effectiveness has been examined.
This article develops a fictional referral-drafting example. It is an educational design exercise, not a complete compliance template, an assessment of an actual product or a report of completed testing. The NHS clinical-risk standards, checked on 13 October 2026, provide the relevant distinction between manufacturer work under DCB0129 and deployment work under DCB0160 in the applicable English NHS context.
Define the workflow before identifying hazards
The fictional product prepares a referral letter from a clinician-selected encounter and supporting records. It does not choose whether the patient needs referral. A clinician reviews the complete draft, confirms urgency and destination, and approves the final version before an authorised transmission process begins.
The proposed deployment is a defined outpatient service. Patient self-referral, autonomous triage and prescribing are outside this example. These boundaries are assumptions for the exercise, not a declaration that the fictional product falls outside medical-device regulation.
The MHRA's intended-purpose guidance explains why the function, users, population and environment need clear specification. Without those details, the safety argument could drift between a drafting aid and a decision-making service without acknowledging the change.
Write a claim that can be challenged
The proposed top-level safety claim is deliberately conditional: the defined workflow may be suitable for restricted use when the identified controls are implemented, verified and accepted through the relevant organisational process.
That is different from declaring the system safe because it contains a review button. The supporting argument must establish that the right information reaches the right reviewer, that approval applies to the released document and that incomplete actions remain visible.
The example also needs to identify its dependencies. It relies on correct encounter selection, access to the necessary records, a functioning review interface, suitable staffing and a transmission service that reports status. A failure in any dependency could invalidate part of the argument even if the generated prose is accurate.
A small illustrative hazard log
The identifiers below are fictional. Risk evaluation and acceptance criteria would need to follow the organisation's applicable method; numerical risk scores have not been invented to make the example look complete.
| Hazard | Potential consequence | Proposed control | Evidence required |
|---|---|---|---|
| H1: Wrong patient attribution | Another person's information enters the referral | Bind draft and approval to the selected encounter; display identity during review | Synthetic patient-switching and session-resumption tests |
| H2: Material information omitted | Receiving service lacks information needed for assessment | Display source context and require complete review | Clinically adjudicated omission cases and observed user review |
| H3: Urgency altered | Referral timing no longer reflects the clinician's decision | Preserve and explicitly confirm the clinician-selected urgency | Tests with provisional, changed and conflicting instructions |
| H4: Unapproved version transmitted | A changed draft bypasses meaningful approval | Tie approval to the exact content and destination | Post-approval editing and destination-change tests |
| H5: Incomplete transmission hidden | Staff believe the referral has been completed | Show unresolved status and route exceptions to an owner | Rejection, timeout and delayed-acknowledgement tests |
These controls are proposals. None is marked effective merely because it is listed. The evidence column is the work required to support the argument, not a record that testing has passed.
Make each control testable
For H1, the test should do more than check whether the patient's name appears at the top of the screen. It should examine what happens when the user changes encounters, resumes an interrupted session or opens a second record. The required behaviour should be specified before the test runs.
For H2, a generic accuracy score is insufficient. The test set should contain information whose omission would materially alter the receiving service's understanding. The evaluator should record whether the omission appears, whether the reviewer detects it and what remains in the approved document.
For H5, a successful demonstration is not enough. Deliberately simulated connection failures can reveal whether the interface distinguishes sent, acknowledged, rejected and unresolved states. The test must examine both the system's behaviour and what staff believe has happened.
The good machine learning practice principles support relevant testing and attention to the human-AI team. They do not provide results for this fictional example; those would require an actual evaluation.
Separate supplier evidence from local verification
The manufacturer may demonstrate that approval is invalidated after a draft changes. The hospital must still establish that its integration preserves that behaviour and that local users cannot bypass it through a separate release route.
Similarly, a supplier can explain how an exception is raised, but the adopter needs to identify the team receiving it and the process during absence or service disruption. A notification without an accountable recipient does not resolve the operational hazard.
This is the practical manufacturer-adopter connection in DCB0129 and DCB0160. It should avoid both extremes: assuming the supplier has proved the local workflow safe, or requiring the adopter to duplicate every engineering test irrespective of relevance.
Reach a conditional release decision without inventing results
For this exercise, the honest current decision is that release evidence remains outstanding. The proposed restricted-use decision would become available only after the relevant tests, clinical review and organisational acceptance have been completed and documented.
The decision record should specify the permitted service, users, functions and configuration. It should exclude autonomous urgency selection and unreviewed transmission. It should identify unresolved issues, owners and the conditions that would prevent release even if most tests were favourable.
A clinically important failure in identity binding, for example, should not disappear into an average performance score. The team would need to investigate, revise the control and reassess the relevant evidence before relying on it.
The point of the worked example is to show how evidence changes a decision, not to create the appearance of a completed safety case where none exists.
Define the triggers for reopening the argument
A new model, changed source selection, a shorter review screen or expanded transmission permissions could affect the argument. So could a change in staff roles or the closure of the team previously handling exceptions.
The joint change-control principles emphasise bounded modifications and lifecycle evidence. A real change process should identify which claims and controls are affected, what further verification is needed and whether additional conformity assessment is required under the applicable route.
Incident response should also return to the hazard log. A newly discovered failure may expose a missing hazard or show that an assumed control is less effective than expected. The safety case should remain a decision-supporting argument rather than a document frozen at launch.
Where clinical safety support contributes
As described on 13 October 2026, Insights by iatroX offers CSO services, hazard-log development and safety-case support. The relevant expertise is in connecting workflow, risk, evidence and ownership, not promising that a particular template guarantees compliance.
A strong safety case makes its assumptions inspectable and its unresolved questions actionable. Its value is clearest when it prevents an unjustified release, supports a defensible restriction or shows precisely what evidence would permit the next step.
Frequently asked questions
Is this a complete DCB0129 or DCB0160 compliance template?
No. It is an illustrative safety argument for a fictional workflow, and a real project needs the full applicable process, documentation and competent review.
Have the controls in the example been tested?
No actual tests are reported. The table specifies evidence that would need to be generated before a real release decision could rely on those controls.
Does every hazard need an AI-based safeguard?
No. Identity binding, permissions, version control and transaction checks may be better enforced through ordinary software and workflow controls than through another generative model.