skip to main content
study systems

Critical Appraisal QuickCheck: The 12 Questions That Matter

A non-negotiable appraisal checklist for clinicians: identify design, bias risk, effect size, and whether you should trust the conclusion — fast.

Review date
2026-01-28
Reading time
11 min read
Author
Dr. Kola Tytler MBBS CertHE MBA MSt MRCGP
On this page

Key points

  • Appraisal is not academic theatre: it is risk management for decision-making.
  • Start by identifying the study type correctly (RCT, cohort, case-control, diagnostic accuracy, systematic review).
  • Extract effect size + uncertainty; treat conclusions as marketing until proven otherwise.
  • Use evidence hierarchy to decide how hard you need to work.

Step 0: Identify the study type (correctly)

Most appraisal mistakes come from misclassifying the paper. A “review” is not necessarily a systematic review. An observational association is not a causal claim. Get the design right first, then apply the right checklist.

The QuickCheck (12 questions)

1

1) What is the exact question?

If you can’t state the question in one line, you will misread the paper.
2

2) What is the study design, and is it fit for purpose?

Match design to question: therapy, prognosis, diagnosis, harm, prevalence.
3

3) Who was included (and who was excluded)?

Selection drives generalisability. Look for convenience samples and hidden exclusions.
4

4) What was the comparator?

No meaningful comparator = weak inference, even if the stats look strong.
5

5) Were outcomes pre-specified and clinically meaningful?

Beware outcome switching and proxy outcomes that don’t map to what you care about.
6

6) How big is the effect (absolute, not just relative)?

Absolute risk and absolute benefit often change the interpretation dramatically.
7

7) How uncertain is the estimate?

Confidence intervals matter. A narrow CI supports precision; a wide CI is “maybe.”
8

8) What are the likely biases?

Confounding, measurement bias, missing data, performance bias, reporting bias.
9

9) Are harms and trade-offs reported transparently?

If harms are minimised or absent, you need a reason to trust the paper.
10

10) Is the conclusion proportional to the data?

Strong claims with weak designs are the most common pattern in low-quality literature.
11

11) Funding and conflicts: what incentives exist?

COI doesn’t automatically invalidate results — but it changes your prior.
12

12) Would this change practice (or just add trivia)?

If it doesn’t change a decision, it is not worth deep time right now.
SourceOxford CEBM: Levels of Evidence (to calibrate effort)
Open Link
SourceBMJ: How to read a paper (critical appraisal foundations)
Open Link
Your learning path

Save your place and choose what comes next

sign in to save your learning
Work through this guide (12 steps)

Use the actions from this article to track what you have worked through.

Sign in using the reading controls above to save your steps.

  • If you can’t state the question in one line, you will misread the paper.

    Return to this section

  • Match design to question: therapy, prognosis, diagnosis, harm, prevalence.

    Return to this section

  • Selection drives generalisability. Look for convenience samples and hidden exclusions.

    Return to this section

  • No meaningful comparator = weak inference, even if the stats look strong.

    Return to this section

  • Beware outcome switching and proxy outcomes that don’t map to what you care about.

    Return to this section

  • Absolute risk and absolute benefit often change the interpretation dramatically.

    Return to this section

  • Confidence intervals matter. A narrow CI supports precision; a wide CI is “maybe.”

    Return to this section

  • Confounding, measurement bias, missing data, performance bias, reporting bias.

    Return to this section

  • If harms are minimised or absent, you need a reason to trust the paper.

    Return to this section

  • Strong claims with weak designs are the most common pattern in low-quality literature.

    Return to this section

  • COI doesn’t automatically invalidate results — but it changes your prior.

    Return to this section

  • If it doesn’t change a decision, it is not worth deep time right now.

    Return to this section

Study methods · Article 11 of 48