Two observers agreeing about a NAC practice encounter does not prove that their judgement is correct. Useful calibration starts earlier: both observers should identify the same task, record what happened and explain how their comments follow from it. Agreement reached by copying the louder observer is not calibration.
The Medical Council of Canada publishes free NAC preparation resources, identified in its public material checked on 13 September 2026. Use the current official examples and candidate guidance to establish the assessment context. The exercise below is a separate, original method for making peer feedback more specific; it is not an official NAC scoring system.
Separate what happened from what you thought of it
"The candidate did not explore the patient's concern" is a claim about an omission. "The candidate sounded abrupt" is an interpretation of the interaction. "I would have asked that question earlier" may be a preference about sequence. All three can be discussed, but they need different evidence.
An observer should be able to locate the behaviour behind the comment. When no supporting moment can be identified, the feedback may still express a useful impression, but it should be labelled as such rather than presented as a missed mark.
The task also matters. An observer cannot fairly criticise a candidate for not completing an entire assessment when the instruction asked for a focused explanation. Calibration begins with the instruction sheet, not an all-purpose checklist applied to every encounter.
Use one controlled fictional encounter
In this original practice scenario, a patient is discussing return to usual activities after an injury. The candidate explores the symptom and daily limitations, then explains a provisional plan. The patient's main concern is an activity required for work, but the candidate never asks what their work involves.
Observer A writes, "Poor empathy and incomplete assessment." Observer B writes, "Good rapport, but the practical effect on work was not explored." The disagreement should not be resolved by averaging those statements into "reasonable communication".
Instead, ask each observer to identify evidence. Did the patient mention work? How did the candidate respond? What relevant question was missing? Was there another point in the conversation where the concern was addressed? This may reveal that both observers noticed the same omission but described its significance differently.
The exercise is about reviewing the encounter, not deciding the fictional patient's treatment. Clinical conclusions would require the appropriate information and current references.
Calibrate before discussing the candidate's overall performance
Have observers review the same fictional role sheet and instruction, then watch the encounter independently. They should write comments before speaking to one another. This prevents the first confident interpretation from becoming the group's shared memory.
During discussion, sort disagreements into four categories: different recollection of what happened, different interpretation of the task, different clinical reasoning, or different communication preference. Rewatching or consulting the transcript can address the first. The instruction and official guidance help with the second. Appropriate clinical sources or a supervisor are needed for the third. The fourth requires discussion with the role-player, not automatic conversion into a clinical error.
It is acceptable to retain a disagreement. Record the reason and what would resolve it rather than inventing consensus.
An original disagreement-resolution sheet
| Field | What to record |
|---|---|
| Task | The exact educational task the candidate was attempting |
| Observed moment | A short quotation or description, with a time point if available |
| Observer A's interpretation | The concern and its stated basis |
| Observer B's interpretation | The alternative view and its stated basis |
| Type of disagreement | Observation, task, clinical reasoning or communication preference |
| Resolution | Agreed correction, remaining uncertainty or supervisor question |
| Next practice target | One behaviour that can be observed in a new encounter |
For the fictional case, a defensible target might be to establish the patient's relevant work demands before agreeing an activity-related plan. "Be more empathetic" is less precise unless the feedback explains what the patient needed and how the response fell short.
Do not turn the worksheet into a homemade numerical pass scale. Its purpose is to improve the quality of the conversation about performance, not replace an examiner's training or judgement.
Test whether the feedback changes the next encounter
Give the learner a new scenario with a different practical concern, such as caring responsibilities rather than employment. Ask the observers to watch for the agreed target without revealing the exact concern in advance.
The question is whether the learner identifies what matters to this patient. Repeating the work-history question in every case would be a superficial response to feedback. The correction should generalise to the task, not become another compulsory line in a script.
Record whether the target was demonstrated and any trade-off introduced. For example, the candidate may explore context thoroughly but lose focus on the instruction. Good feedback can acknowledge improvement while identifying the next bounded issue.
Official material, human observers and iatroX
This article is published by iatroX and includes its own simulation tools in the comparison. Official MCC material establishes the intended assessment context. A peer can observe and discuss a controlled encounter. An experienced supervisor can help resolve clinical or performance judgements beyond the group's competence.
The September 2026 iatroX Simulations release includes a NAC track with transcript-linked domain feedback and Tutor-led remediation. This can provide another account to examine alongside the encounter, but it should not become the tie-breaker simply because it is generated by software. Check what passage supports each comment and whether it matches the task.
Do not assume that a human observer and an AI agreeing on a weakness establishes diagnostic accuracy, clinical validation or a likely examination result. They may share the same incomplete information or interpretation.
Choose support according to the disagreement
When observers disagree about what was said, improve the recording and evidence trail. When they disagree about clinical correctness, use current sources and appropriate supervision. When they disagree about style, ask whether the difference affected understanding or the task rather than enforcing personal preference.
Per iatroX's September 2026 learning information, the UK subscription costs £99 paid upfront annually, equivalent to £8.25 a month billed annually, or £29 monthly. It includes paid banks, Tutor, planning, simulations and CPD tools, with one complete simulation free. These are UK prices; the value for a NAC candidate is access to relevant practice and review, not a substitute for informed human feedback.
Frequently asked questions
Should peer observers give a mark before discussing the encounter?
For this exercise, record specific observations before an overall judgement. That makes it easier to identify whether a disagreement concerns evidence, clinical reasoning or preference.
What happens when the observers still disagree?
Record the unresolved issue and seek the source or supervisor input needed to address it. Forced agreement is less useful than a clearly defined uncertainty.
Can agreement between peer and AI feedback predict a NAC result?
No. Agreement on a practice encounter is not evidence that the feedback is accurate or that it predicts examination performance.