An AI in your pocket: a powerful co-pilot, but who checks the flight plan?
The allure is undeniable. A query typed into a clean interface on your phone, and a comprehensive, well-written answer appears in seconds. General-purpose AI tools like ChatGPT and Perplexity are technological marvels, capable of drafting emails, summarising articles, and even debugging code with breathtaking speed. It's tempting to see them as an all-knowing co-pilot for every aspect of our professional lives, including clinical practice.
But while these tools are powerful, they are not designed with the principles of clinical governance in mind. For a UK clinician, relying on them for medical information introduces a host of hidden risks. This isn't a question of which AI is "cooler," but a crucial discussion about which AI is safer for you and your patients. Before you ask a general AI for clinical advice, it's essential to understand the flight plan – and who is ultimately responsible if it goes wrong.
Risk 1: the confident "hallucination"
One of the most significant dangers of large language models (LLMs) is their tendency to "hallucinate." This doesn't mean the AI is seeing things; it means it confidently invents plausible but entirely false information to fill a gap in its knowledge. Because the output is so well-written and self-assured, these fabrications can be incredibly difficult to spot.
Consider this hypothetical but realistic scenario, which highlights the serious AI medical diagnosis risk:
A junior doctor, faced with a distressed child with otitis media and a penicillin allergy, asks a general AI for an alternative antibiotic suspension. The AI confidently suggests "Azithroclarin 150mg/5ml suspension, once daily for three days," and even cites a plausible-sounding but entirely fake study, "the PIONEER Paediatric Trial (2022)."
In reality, "Azithroclarin" doesn't exist. The AI has blended parts of real drug names (Azithromycin, Clarithromycin) into a dangerously convincing fiction. This is a prime example of ChatGPT clinical errors – it's not just incorrect, it's inventively and confidently incorrect in a way that could lead to a serious patient safety incident.
Risk 2: the US-centric default
General AI models are trained on vast datasets scraped from the public internet. This data is overwhelmingly dominated by US-based content. For a UK clinician, this presents a constant, insidious risk. An AI like Perplexity, when asked about a medication or treatment protocol, will almost certainly default to the US healthcare system.
This results in critical discrepancies:
- Drug Names: It will refer to acetaminophen, not paracetamol.
- Regulatory Bodies: It will cite FDA approvals, not MHRA guidance.
- Units of Measurement: It may use lbs for weight or mg/dL for lab results.
- Clinical Guidelines: Most importantly, it will reference guidelines from American bodies like the American Heart Association (AHA) or the American Academy of Pediatrics (AAP), which can differ significantly from NICE, CKS, or SIGN guidelines used in the UK.
Relying on this information isn't just inefficient; it's a direct challenge to the principles of UK-based, evidence-led practice.
Risk 3: the black box problem & medico-legal responsibility
This is the most critical question for any practising clinician: If the AI gives you advice that leads to a patient safety incident, who is responsible?
The answer, unequivocally, is you. Your duty of care, as outlined by the GMC, is to the patient. You cannot delegate your medico-legal AI use responsibility to a "black box" algorithm. General AIs are often black boxes because you cannot see precisely how they generated a specific answer. There is no audit trail. You can't ask ChatGPT for its sources and be 100% certain they are real and have been interpreted correctly.
This poses a massive problem for clinical governance AI integration. How can you justify a clinical decision if you can't verify the source of the information it was based on? If something goes wrong, "the AI told me" is not, and will never be, a valid defence.
How iatroX is designed for safety
The risks outlined above are precisely why iatroX was built differently. Our design philosophy prioritises medical AI safety UK standards and accountability above all else. We mitigate these risks through a fundamentally different approach.
- The "Walled Garden" Approach: iatroX's AI does not roam the open internet. It operates within a "walled garden" – a curated, closed-loop dataset comprising only trusted, verified UK clinical sources like NICE, CKS, BNF, SIGN, and more. This eliminates the risk of US-centric bias and dramatically reduces the potential for dangerous hallucinations.
- Traceability and Verifiability: Every piece of information iatroX provides is directly linked and referenced back to the specific source document and paragraph it came from. This transparency is vital. It means you can always verify the information for yourself, creating a clear and defensible audit trail for your clinical decisions.
- Focus, Not Breadth: We have deliberately sacrificed the ability for our AI to "write a poem about diabetes" for the ability to give you a reliable, evidence-based answer about its management. Our focus is narrow but deep, concentrating exclusively on providing safe AI for clinicians to use for UK-specific medical information retrieval.
Conclusion: your medical indemnity will thank you
Experimenting with general AI tools like ChatGPT and Perplexity for non-clinical tasks is part of adapting to a new technological landscape. They are undeniably useful.
However, when it comes to patient care, the standards are different. Professional-grade tasks demand professional-grade, safety-conscious tools. The hidden risks of hallucinations, data bias, and the immense medico-legal burden associated with using unverifiable general AI for clinical queries are too significant to ignore. Choosing a tool designed with a "safety-first" ethos isn't just about better technology; it's about upholding your professional duty and safeguarding your patients and your career.