Ask whether a medical AI product works and somebody will eventually say it has been independently tested. Ask what independent means and the conversation can become much less fluent.
An external researcher may have written the paper. A university may appear in the affiliations. A journal may have reviewed the manuscript. Those details matter, but they leave several practical questions unanswered. Who selected the cases? Who decided which product version competed? Who marked the answers? Who could prevent an inconvenient result from being published?
The strongest evidence makes those relationships visible. The weakest sales pitch asks one reassuring adjective to cover all of them.
Three labels, three different questions
Peer review describes an assessment process before publication. It does not mean that journal reviewers repeated every experiment, inspected every private dataset or tested the current product themselves. A peer-reviewed study can be informative and still leave substantial uncertainty about a different clinical setting.
Vendor funding describes where resources came from. Industry money can support a well-designed, useful evaluation. An unfunded project can be poorly designed. Funding is relevant because it can create interests and influence, which must be understood alongside the actual methods.
Independent testing should describe who controlled consequential decisions. An external name on a title page is an incomplete answer if the vendor selected every favourable case or retained a veto over publication.
These properties should be reported separately. Collapsing them into a single hierarchy encourages lazy reading: university beats company, journal beats preprint, independent beats funded. The real task is to find out what happened and whether the result answers the clinical question being asked.
Follow control through the study
The following questions form an editorial assessment framework. They are intended to make a study's independence inspectable, rather than assign a decorative badge.
| Decision | What a reader should be able to establish |
|---|---|
| Selecting the cases | Who assembled the sample and which cases were excluded? |
| Configuring the products | Who chose versions, prompts, settings and permitted tools? |
| Defining success | Were the main outcomes and analysis decisions specified before results were examined? |
| Assessing the answers | Who judged performance, using which criteria and access to product identities? |
| Accessing the data | Which investigators could examine the complete results and underlying records? |
| Publishing the findings | Could investigators publish an unfavourable result without the sponsor's permission? |
The questions are connected. Independent marking cannot rescue a sample containing only easy cases. A representative sample does not rescue a comparison in which one product receives extensive optimisation and another receives an unsuitable prompt. Access to raw results has limited value if a contract prevents publication of the conclusion.
A reader does not need to assume bad faith to ask about these decisions. They determine what the study can show.
Disclosures should explain roles
The ICMJE recommendations on relationships and activities ask for disclosure of support and sponsors' roles in study design, data work, writing and publication decisions. They also address investigators' access to study data. Those details are more informative than a bare statement that an organisation provided funding.
Imagine two explicitly hypothetical evaluations. In the first, a vendor pays the costs, but an external team controls the protocol, receives all outputs, conducts the analysis and retains publication rights. In the second, university researchers assess answers from a demonstration set selected by the company, which can approve the final manuscript.
The university affiliation alone would not tell a reader which evaluation offers stronger protection against selective reporting. The funded study could provide the more convincing evidence of performance on its defined task.
That conclusion is uncomfortable only if labels have been doing the appraisal for us. The purpose of disclosure is to let readers understand the relationship, not to automatically disqualify anybody who has received support.
The product tested must be identifiable
A clinical AI result belongs to a particular system under particular conditions. A model update, changed retrieval collection or different user interface can alter the product clinicians encounter. A recognisable brand name is therefore insufficient to reproduce a comparison.
The CONSORT-AI reporting extension identifies information needed when reporting clinical trials involving AI, including the intervention's version, input and output handling, human interaction and analysis of errors. These reporting requirements help readers understand what was actually tested.
For a buyer, the practical follow-up is straightforward. Does the evaluated version correspond to the product being offered? If changes occurred, what evidence supports carrying the earlier result forward? A vendor need not rerun every possible study after every minor edit, but it should explain why a cited result remains relevant.
Publication age alone is a crude substitute for this reasoning. An older evaluation of an unchanged, narrowly defined feature may remain useful. A recent paper on a substantially different configuration may be less applicable than its date suggests.
Read the losing answers
An average score is an efficient summary, but it can conceal the failures a clinician most needs to recognise. The useful question is what happens when the system misunderstands the task, lacks relevant context or retrieves a source that does not support its recommendation.
Assessors should explain how they judged those cases. Were errors categorised by clinical consequence? Could the product ask for missing information? Were refusals assessed as appropriate responses or automatically scored as failures? Were incomplete answers distinguished from confidently incorrect ones?
These choices can change the interpretation of a result without changing a single model output. A system that appropriately requests clarification may deserve a different assessment from one that invents the missing detail and delivers a polished answer.
Access to representative failure examples also helps clinicians decide whether they can recognise the relevant problem in practice. An error that is obvious to an expert reviewing a transcript may be harder to detect during a busy consultation.
Publication is not the end of scrutiny
A published evaluation becomes more useful when others can challenge its conclusions. Depending on the material, that may involve sharing prompts, assessment criteria, aggregate results, code or carefully de-identified examples. Patient-level data require appropriate handling; transparency does not require publishing identifiable clinical records.
Where unrestricted access is impossible, the paper can explain the restriction and a workable route to independent assessment. The important distinction is whether another qualified team has a realistic way to examine the claim, rather than whether a data-availability sentence happens to appear.
Clinicians can apply the same discipline to a local pilot. Record the task, product version, selection process, assessment method and correction burden. Decide in advance what would count as useful performance. Then preserve the failures alongside the successes.
For iatroX, readers can begin with the published methodology and product explanation. Those pages provide material to inspect; their existence does not confer immunity from the questions in this article.
The phrase independently tested should open a conversation about control, access and reproducibility. If it closes that conversation, it is functioning as marketing before it is functioning as evidence.
Frequently asked questions
Should clinicians reject vendor-funded medical AI studies?
No. Assess the study design, sponsor's role, investigators' access to results and freedom to publish. Funding creates a relationship that should be disclosed and examined. It does not, by itself, determine whether the methods or conclusions are sound.
Does peer review establish that the current product is clinically validated?
It establishes that a manuscript underwent a review process. The study's findings still apply to its evaluated version, population, task and outcomes. Readers need to check how closely those conditions match the product and clinical use now being proposed.
What is the quickest useful question about independent testing?
Ask who controlled the cases, assessment and publication of the complete results. A clear answer often reveals more than a list of affiliations. Then inspect whether the resulting study addresses your clinical task and makes its methods sufficiently clear to challenge.