A product can acquire more authoritative content without improving every task performed with it. The Doximity-UpToDate partnership therefore creates a testable proposition: does the combined experience help eligible clinicians reach better-supported decisions, with acceptable effort, compared with what they would otherwise use?
The answer requires more than a favourable demonstration or a benchmark for an earlier system. It requires defining the comparator, the clinical task and the outcome before looking at the results.
The framework below is a proposed evaluation design. It does not report a trial of the new integration or claim that iatroX has outperformed it.
Start with the intended improvement
The combined experience could create value in several ways. It might expose an important qualification, make a source easier to inspect, reduce unnecessary searching or help a clinician recognise unresolved uncertainty.
Those benefits are not equivalent. A shorter search can be useful even when the final answer is unchanged. A longer interaction can be worthwhile when it corrects a consequential omission. A pleasing interface can still fail to improve either.
The evaluation should specify which benefit is primary. Otherwise, a weak result on accuracy can be repackaged as a success on satisfaction, or a slower workflow can be defended using an unmeasured claim about confidence.
Choose comparators that answer real decisions
One comparison is the combined experience against the previous Ask configuration. That tests the incremental contribution of the integration. Another is the combined experience against direct access to the relevant reference product. That tests whether embedding it improves the task.
A third comparison is the clinician's usual workflow, including the sources they genuinely use. This matters for adoption: a product can outperform a stripped-down control while adding little to an experienced clinician's existing routine.
Each comparison should specify product versions, entitlements and available features. Otherwise, a change in access tier or source availability can be mistaken for a difference in the underlying approach.
Evaluate completed tasks, not isolated paragraphs
A useful endpoint is a justified response to an original clinical-reference scenario, with the consequential claims checked. Record the time taken to reach that point, including source opening, follow-up questions and reconciliation of disagreement.
Time to the first generated sentence is an engineering measure. It is not the same as time to a conclusion the clinician is prepared to defend. Similarly, answer length should not be treated as a proxy for completeness.
The task should also permit an appropriate unresolved result. Recognising that the supplied information is insufficient may be better than producing a specific recommendation that depends on an invented assumption.
Distinguish errors, omissions and unsupported certainty
An assessment rubric should identify the key requirements for each scenario before outputs are reviewed. These might include the appropriate population, a decisive exception, a necessary clarification or a qualification concerning uncertain evidence.
A statement can be individually correct while the answer remains unsafe or misleading through omission. Conversely, a concise answer can be adequate when it includes the points relevant to the bounded question.
The NOHARM research illustrates why harmful recommendations and omissions deserve explicit attention. Its results should not be transferred wholesale to a newly configured product, but its focus helps define more meaningful evaluation questions than fluency alone.
Measure what the clinician does with two answers
The distinctive human-factors question is whether a second response improves the user's judgement. Does the clinician notice a disagreement? Do they inspect the relevant source? Do they choose the better-supported recommendation or simply follow the more confident wording?
A study should distinguish the quality of the generated material from the quality of the final human-AI decision. A useful qualification that the clinician never sees cannot improve the decision. A confusing disagreement can increase workload without producing a better result.
This is also why confidence should be assessed alongside correctness. Increasing confidence in an unchanged error would be a poor outcome, even if users report that the experience feels more reassuring.
Design the comparison to avoid easy sources of bias
Random assignment can help separate the tool's contribution from differences between clinicians. A crossover design may be useful, but repeating the identical scenario creates a learning effect: participants already know what mattered on the second attempt.
An alternative is to use matched scenarios and counterbalance their order. Reviewers should, where feasible, assess answers without knowing which product produced them. Disagreements between reviewers need an explicit resolution method.
There is no universal sample size that makes the study credible. The number of clinicians and tasks should follow the primary endpoint, expected variability and intended precision. A handful of selected examples can inform design, but it should not be presented as a conclusive comparative trial.
Include the failures that disappear from demonstrations
Test a missing entitlement, an unavailable source, an ambiguous question and a contradiction between two apparently relevant passages. Observe whether the interface explains the problem or silently substitutes a different answer.
Also examine repeated use. A demonstration may benefit from a carefully phrased prompt that ordinary users do not reproduce. The comparison should include realistic variation without deliberately sabotaging one product.
A useful institutional pilot records the proportion of eligible clinicians who actually use the feature. Excellent performance among a small enthusiastic group does not establish that the deployment benefits the whole organisation.
Publish enough detail for the result to remain interpretable
The report should identify the evaluated configuration and date, disclose the selection process for cases and show uncertainty around the main estimates. Negative and mixed results should remain visible.
When the product changes substantially, the claim should be reconsidered. A result belongs to a tested configuration and use case, not permanently to a company name. That principle should apply equally to Doximity, UpToDate, OpenEvidence and iatroX.
Learning products need an additional endpoint
For iatroX Tutor or a question-bank workflow, immediate answer quality is only part of the purpose. A useful evaluation would also examine later independent performance on unfamiliar examples and whether the learner can explain the relevant distinction.
That does not exempt educational tools from checking their content. It adds another responsibility: demonstrating that the interaction helps users learn rather than merely supplying answers during practice.
The strongest product claim is therefore specific: a defined group, using a defined configuration, improved a defined task. That is less spectacular than declaring a universal winner, but substantially more useful to clinicians choosing a tool.
Frequently asked questions
Does an existing Doximity benchmark validate the new integration?
Not automatically. A result applies to the system configuration and tasks evaluated. The combined experience introduces new content, interaction patterns and possible failure modes that deserve assessment.
Is user satisfaction a sufficient measure of clinical AI quality?
No. Satisfaction can reveal usability, but it should be considered alongside correctness, omissions, verification effort and the clinician's final decision. A confident or convenient answer can still be wrong.
How should iatroX be evaluated fairly against a larger platform?
Use the same relevant reference-task standards, then evaluate its educational purposes separately. Do not penalise a focused learning tool for lacking an unrelated enterprise feature, or excuse inaccurate content because the interface is educational.
Explore Ask-iatroX with a focused question and inspect the evidence supporting the answer.