Genuine disagreement between reasonably well-conducted studies is one of the harder situations for any clinical AI system to handle well, and it is worth setting out directly what a genuinely honest response looks like, as distinct from a falsely confident one.
Why disagreement cannot be resolved by counting papers
Five smaller studies pointing one direction do not automatically outweigh one large, methodologically robust trial pointing the other way. Vote-counting across studies of different quality and different size is not a legitimate way to resolve genuine scientific disagreement, however tempting it is as a simple heuristic.
The possible causes of genuine disagreement
Different patient populations studied, different doses or specific interventions compared, different endpoints measured, different lengths of follow-up, ordinary random variation that a single study, however well conducted, cannot rule out on its own, differing risk of bias between studies, and publication bias systematically favouring positive results over null or negative ones, can each independently produce apparent disagreement between studies that may or may not reflect a genuinely different underlying truth.
What a proper systematic review does with heterogeneity
A well-conducted systematic review investigates heterogeneity explicitly, asking why studies disagree rather than simply averaging over the disagreement and presenting a single number as though consensus existed where it does not.
Why a pooled average can be inappropriate
Where the underlying treatment effect genuinely differs meaningfully between study populations, for reasons that can sometimes be identified and sometimes cannot, a single pooled average can misrepresent every individual population it was drawn from, technically correct as a statistical summary while being clinically misleading for any specific patient.
What the ideal AI response actually looks like
Genuinely useful handling of disagreement states plainly that the evidence is conflicting, identifies where genuine agreement does exist among the studies, explains the plausible reasons results may differ, distinguishes higher- from lower-quality studies within the disagreement rather than treating them as equally weighted, presents the current guideline interpretation where one exists, and explicitly avoids manufacturing a falsely decisive conclusion the evidence does not actually support.
Where iatroX positions itself on this
iatroX favours the strongest appropriate evidence and authoritative UK recommendations as a starting point, while being willing to explicitly acknowledge unresolved scientific uncertainty rather than smoothing it into a single confident-sounding answer where the underlying literature genuinely does not agree.
Where this shows up most often in practice
Screening recommendations, where different studies and different national bodies have sometimes reached genuinely different conclusions about the same intervention, are a recurring example. Newly approved medicines, where early trial evidence can be limited and sometimes contradictory before a larger evidence base accumulates, are another. And genuinely controversial interventions, where legitimate clinical disagreement persists even among experts working from the same underlying literature, are a third recurring category worth naming honestly rather than papering over.