AI agents can help generate research questions, draft analysis plans and produce executable code. They cannot be assumed to have implemented the intended analysis correctly. A UCL and Moorfields evaluation published on 8 September 2026 demonstrates that gap: convincing plans and readable results sometimes concealed clinically meaningful analytical errors. The JMIR evaluation
The distinction is particularly relevant to a clinician completing a dissertation or evaluating a service. Getting a script to finish is an engineering milestone. Establishing that it answers the research question is a statistical one. A fluent summary cannot perform the second check on behalf of the researcher.
The experiment covered more than code generation
The researchers evaluated Claude Sonnet 4.6 across Chat, Code and Cowork, with experiments conducted between February and July 2026. Three task types, repeated across those modes, produced 27 runs: generating questions, developing and executing a plan, and implementing an investigator-specified plan. The published evaluation found two clinically meaningful errors among 17 narrative summaries; only eight summaries were satisfactory across all assessed criteria. Study methods and findings, 8 September 2026
The underlying material was an existing, openly released Moorfields ophthalmology dataset and reference analysis. The original dataset describes 7,802 eligible eyes, not a newly recruited cohort for testing AI. Its repository includes the data, analysis script and documentation, released in November 2020. Original Dryad dataset
That setup made it possible to inspect stages separately. A reasonable question, a suitable statistical family, faithful code, correct numerical output and a justified clinical interpretation are related achievements. They are not the same achievement.
Why a good plan can produce a bad analysis
The evaluation reported errors involving cohort boundaries, model composition and concordance calculations, including repeated errors across separate runs. It also identified omissions in statistical diagnostics and problems in the reference analysis itself. The study therefore warns against assuming either generated code or an existing script is correct merely because its output looks familiar. Detailed results
A practical way to understand the issue is to distinguish a statement of intent from an executable rule. "Include eligible adults" is a statement of intent. The code must specify exactly how age is represented, what happens at the boundary and how missing values are handled. A short phrase can conceal several decisions.
The same problem occurs when a model formula contains an additional variable. The script may run and return plausible coefficients, yet answer a different question from the agreed plan. A sophisticated description in the methods section does not repair that mismatch.
These are reasons to make important analytical decisions visible before running the full dataset. The test is not whether an agent can explain the intended method when asked; it is whether the actual implementation follows it.
An original boundary test a novice can understand
Consider this deliberately simplified, fictional teaching dataset. It is not a reconstruction of the Moorfields study and has not been submitted to an AI system.
| Record | Age at enrolment | Rule: include adults aged at least 18 |
|---|---|---|
| A | 18 | Include |
| B | 17 | Exclude |
| C | 44 | Include |
| D | Missing | Apply the pre-specified missing-data rule |
The correct handling of record A depends on "at least", not "older than". Replacing one condition with the other can exclude eligible observations without producing an error message. Record D exposes a separate ambiguity: a missing value is not evidence that someone is underage.
Before any large analysis, the researcher can construct similarly small examples with known expected results. Include observations exactly on an eligibility boundary, repeated measurements and an event occurring on the final observed date. The particular examples should reflect the actual study design.
These tests do not validate an entire analysis. They make a few important decisions inspectable. An expert still needs to judge whether the rules themselves are appropriate.
A correct sentence can faithfully describe the wrong calculation
Researchers often focus on whether AI invented a result. That is only one possible failure. A narrative can accurately reproduce a number from the execution log while the code that produced the number is wrong.
A useful audit therefore moves in both directions. Starting from a sentence, locate its source table, model output, script and cohort definition. Starting from the approved protocol, check that each important definition survives into the script and then into the reported result.
A sentence such as "the association was stronger in the exposed group" needs more than a coefficient pasted accurately. The reviewer must know which group was the reference, how the outcome was coded, which adjustment variables were included and whether the model supports the interpretation.
The result may be numerically genuine but scientifically misleading. That is why checking prose alone is not an adequate review of an AI-assisted statistical analysis.
Repeated agreement is not independent replication
Asking the same system to repeat an analysis can reveal instability. It cannot, by itself, establish correctness. A mistaken rule can be applied consistently, producing reassuring agreement.
For an important result, the most useful second check may be a different representation of the problem. A researcher might calculate a simple cohort count manually, implement an eligibility condition separately, or compare a model's assumptions against the protocol without consulting the generated explanation first.
Independence is a property of the checking process, not simply the number of times a prompt was submitted. Several similar outputs can still share the same overlooked assumption.
The JMIR study's repeated-error finding makes this concern concrete, but it does not supply a universal error rate for every Claude version, every agent or every medical dataset. It was a defined evaluation of one model generation and one research setting. Study scope and limitations, September 2026
Measure time to a verified result
The authors discuss a verification burden, but the paper should not be turned into a quantified claim that AI either saves or wastes a particular number of researcher hours. It is not a controlled staff-time evaluation.
A useful local study would record the full sequence: drafting, execution, debugging, expert review, correction and final approval. The comparator should be a conventional analysis completed to the same standard. Comparing a quick unverified draft with a checked human analysis would favour the wrong endpoint.
The expertise of the user also belongs in that evaluation. A statistician may recognise a mistaken model formula immediately. A novice may spend considerable time checking superficial details while missing the consequential error. The same output can therefore impose very different verification costs.
This is a proposed measurement approach. No productivity test or independent replication of the JMIR experiments was performed for this article.
A workable division of labour
For an academic clinician, a reasonable starting arrangement is to let AI help make the work inspectable. Ask it to describe variables, draft alternative formulations of the research question, document code and propose edge cases. Keep the approved protocol and the actual implementation under expert review.
For analysis, maintain a short decision record covering the unit of observation, population, outcome, missing-data approach, adjustment strategy and interpretation limits. The point is not to generate another long document. It is to prevent a consequential choice from existing only inside a chat history.
For reporting an observational study, the relevant STROBE checklist provides an established reporting framework. Completing a checklist does not validate an analysis, but it can make omissions easier for the research team and readers to identify.
Use synthetic or appropriately governed research data while testing workflows. A demonstration using an openly released dataset does not establish permission to upload an identifiable local dataset to an external service.
What medical students should learn from this
The educational opportunity is not simply to learn better prompts. It is to learn how a research question becomes a population definition, a model and an interpretable result.
A useful tutorial could show a correct plan beside an intentionally flawed toy implementation. The learner identifies the mismatch, predicts its consequences and explains what would need checking in the full analysis. That is a more meaningful exercise than asking whether the final paragraph sounds scholarly.
iatroX publishes this analysis as part of its clinical education coverage, not as a claim to provide an autonomous statistics service. Its Socratic Tutor, described in the September 2026 product information, starts from an attempted question and explores the learner's misconception. That question-first approach can support relevant learning; it does not certify a dissertation's code or replace a statistician.
For someone beginning a project, the appropriate outcome is a clear question and an agreed analysis plan. For someone using an agent to implement that plan, it is a traceable, checked analysis. For someone reading the final paper, it is enough detail to judge what the results mean. AI can assist at all three stages, but none should be mistaken for completed verification of the next.
Frequently asked questions
Did the study show that Claude cannot perform clinical data analysis?
No: the evaluation published on 8 September 2026 found useful capabilities alongside important implementation and interpretation errors. Its findings support supervised use and targeted verification, not an all-purpose verdict on every Claude task.
Were only two of the AI analyses problematic?
No: two of the 17 narrative summaries contained clinically meaningful errors, but that is not a count of every analytical problem. The paper also reported coding, modelling and diagnostic-checking issues.
Can an AI-generated analysis be trusted when the code runs successfully?
Successful execution only establishes that the software completed that run. The research team must still verify the cohort, model, assumptions, outputs and interpretation against the intended question.
Practise question-first reasoning with the iatroX Socratic Tutor →