The question arrives loaded, and unloading it is the answer: AI and human examiners are not competing for the same job. One is a feedback machine, consistent, immediate, infinitely repeatable, cheap enough to use nightly; the other is a judgement instrument, calibrated by years of real candidates, able to weigh context, catch the non-verbal, and recognise the competent performance that violates the checklist. Students asking which to trust should instead ask which to trust for what, and the practical model this article defends is triangulation: AI for repeated formative practice, humans at meaningful intervals for calibration, with each covering the other's documented blind spots.
What each does well, stated fairly
The AI examiner's strengths are structural. Consistency: the same performance scores the same on Tuesday and at midnight, which human examiners, honestly, do not achieve across a long assessment day. Immediacy: feedback while the performance is still in working memory, when correction encodes best. Repeatability and cost: the tenth practice consultation of the week is economically impossible with human examiners and trivial with machines. And granularity: behaviour counts, questions asked, structure followed, phrases used, tallied without fatigue. The human examiner's strengths are exactly what those miss. Calibration: knowing what a safe candidate actually looks like against thousands seen, the standard no rubric fully encodes. Context: recognising that this candidate's unusual approach worked for this patient, the judgement that separates checklist compliance from clinical competence. Non-verbal bandwidth: posture, timing, the quality of attention, the channel current AI scoring reads thinly. And educational nuance: feedback pitched to the learner in front of them, which is teaching, not measurement.
The study any school could run, and what to separate
The comparison worth having is empirical and cheap: take recorded performances, score each through an AI examiner and several independent human examiners, and analyse two things separately, because they answer different questions. Checklist agreement: how closely the machine's behaviour-counts track the humans' checklist marks, which tests the AI as a measurement instrument, and where agreement is high, formative practice can lean on it confidently. Feedback usefulness: blinded learner and educator ratings of which feedback actually improved the next performance, which tests the AI as a teacher, and where the machine's specific, immediate notes may win on volume while human comments win on depth. Publishing both numbers, rather than one blended verdict, is the honest design, and schools running it would give the whole field what vendor demonstrations cannot: local, comparative, outcome-adjacent evidence. Until then, students should hold the two properties apart themselves: an AI score is a behaviour tally with excellent reliability and unproven validity against the examination that matters, and the ten-standards benchmark's rubric-transparency test applies in full: /blog/ten-standards-realistic-ai-patient-simulation.
The triangulation protocol for students
Four rules that extract both values now. Volume with the machine: nightly or alternate-day virtual consultations with AI feedback, treated as behaviour-shaping, structure, coverage, phrasing fluency, where consistency and repetition are the whole point. Calibration with humans: at meaningful intervals, a real observed performance, peer, tutor, simulated patient with educator, treated as the validity check, and its feedback weighted above the machine's wherever the two disagree. Divergence as diagnosis: when your AI scores plateau high while human feedback stays lukewarm, the gap itself is the finding, you are optimising the rubric rather than the consultation, the formulaic-empathy failure mode at /blog/can-ai-virtual-patients-teach-empathy, and the fix is human-directed. And loop closure regardless of examiner: every session's misses convert to knowledge gaps and spaced retesting through the workflow at /blog/virtual-patient-to-qbank-close-the-loop-osce, because feedback of any provenance that changes nothing downstream was entertainment. Trusting AI feedback is not the risk; trusting it alone, for the thing it has not been validated to measure, is.
Frequently asked questions
Will AI examiners be used in real OSCEs?
Assessment bodies move carefully and should: formative use is uncontroversial now, summative use awaits exactly the validity evidence the recorded-performance study generates, and students should expect hybrid models before replacement anywhere.
How many human calibration points does a term need?
Two or three meaningful ones beat weekly shallow ones: a fully observed performance with proper debrief per rotation phase is a defensible floor, with machine practice filling every gap between.
Does examiner disagreement mean the human was right?
Not automatically, humans disagree with each other too; it means the disagreement is information, and the resolution is another observed performance with the specific divergence named in advance.
Can AI feedback harm performance in any way?
Two known routes: rubric-optimisation, where machine-legible behaviours crowd out patient-responsive ones, and confidence miscalibration from consistently generous scoring; both are caught by the human calibration points doing their job.
Should feedback style differ between the two sources?
Exploit the asymmetry: ask the machine for behaviour-level specificity, counts, timings, missed items, and ask humans for judgement-level comment, safety, rapport, overall impression, so each source reports at the resolution it is actually good at.