Table of Contents
You’re three blocks into exam day. Block 2 felt manageable. Block 3 just leveled you. A question about some enzyme pathway you’ve never seen. A genetic syndrome with a name you couldn’t pronounce. A management protocol First Aid never mentioned. You mark them, keep moving, and spend the rest of the block running a silent tally of everything you’ve already gotten wrong.
What nobody told you clearly enough before you walked in: some of those questions — possibly several in that single block — will count for exactly zero toward your final score. Not because the exam is broken. Because that’s precisely how the USMLE is designed to work.
The question is: how many? Where do they come from? And what does the NBME actually know about a question before they put it in front of you? The real answers are more specific than “don’t panic” — and more useful.
NBME Calls Them “Pretest Items” — And the Difference Matters
Most articles, forums, and prep resources call them “experimental questions.” The NBME’s official term is unscored pretest items. That word choice is deliberate, and it tells you exactly what these questions are doing.
The NBME cannot add a new question to its scored pool without real performance data from thousands of actual test-takers. No simulation, no committee review, no expert panel predicting difficulty is an adequate substitute for watching how real students — under real exam conditions — respond to a question. A pretest item is essentially a question going through its job interview. It sits in your exam, gets answered by thousands of students across many test forms, and NBME tracks the statistical results carefully before deciding whether it graduates to a scored item on a future exam.
What most students miss: “unscored for the examinee” doesn’t mean “unanalyzed.” Every pretest item is fully scored internally — NBME tracks exactly what percentage of test-takers got it right, which wrong answers attracted which ability levels, and how performance on that question correlated with performance on the rest of the exam. You’re not answering an unimportant question — every response you give is generating a dataset the NBME will use to decide whether that question deserves to exist on a future exam.
What the USMLE does not publish: how many pretest items are in any given exam. The figure you’ve probably seen — “80 experimental questions” — is widely circulated but is not an official disclosure. It’s an estimate, and as the section below shows, the actual number is likely lower for at least one of the two major exams.
How Many Experimental Questions Are There, Really?
This is where the honest answer diverges from nearly every other prep resource.
What Everyone Claims
The conventional wisdom is approximately 80 experimental questions per exam — roughly 25–28% of both Step 1 (max 280 total questions) and Step 2 CK (max 318 total questions).
What NBME Actually Revealed — Accidentally
In June 2020, as Prometric centers were shuttered during the pandemic, the USMLE announced a plan to offer shortened exams at event-based testing sites at medical schools. The shorter versions would remove unscored pretest items entirely. The NBME’s own announcement specified the result: Step 1 would have 200 items and Step 2 CK would have 240 items under that proposal.
Removing unscored pretest items would yield 200-question Step 1 exams and 240-question Step 2 CK exams. Step 1 math: 280 − 200 = ~80 experimental items. Step 2 CK math: 318 − 240 = ~60–78 experimental items. This is the only time the NBME has ever let the real count slip publicly — and it happened by accident.
The Sheriff of Sodium — whose psychometrics analysis of the USMLE is among the most rigorously sourced in the medical education space — read the same announcement and concluded that roughly ~60 items, or 20% of the standard Step 2 CK form, were experimental. The “up to 318” ceiling for Step 2 CK isn’t always reached, which means the “78” figure assumes a maximum-length form. In practice, closer to 60 is the more defensible estimate for Step 2 CK.
For Step 1, the math lands closer to the conventional 80 — but again, this assumes the maximum-length 280-question form.
The bottom line: USMLE has never officially published the pretest item count for any exam. The 2020 COVID accommodation proposal is the closest thing to an accidental disclosure we’ve ever gotten. For Step 2 CK, approximately 60–78 experimental items (closer to 60, per the best available analysis). For Step 1, approximately 80 experimental items. Every number in every prep resource — including this one — should be treated as an estimate.
One final note: the 2020 shortened forms were ultimately cancelled after significant student backlash. Test-takers argued that one group taking a meaningfully shorter exam than another created an unfair comparison. The NBME reversed course. But the numbers they published before reversing remain the most transparent window into actual pretest item counts we’ve ever seen.
Track your readiness with data that actually reflects scored performance: see our NBME Step 2 CK Score Conversion Calculator.
Why NBME Needs Real Test-Takers — The Psychometric Reason
Item Response Theory (IRT)
The NBME evaluates every question using a statistical framework called Item Response Theory. Before a question can be trusted enough to count toward a score, NBME needs to know three things about it:
b-parameter
What proportion of test-takers answer correctly? A question 95% of students get right has almost no discriminatory value at the pass/fail threshold NBME cares about.
a-parameter
Does performing well on this question predict overall exam performance? A high-discrimination question sharply separates students who know their material from those who don’t.
c-parameter
How often do lower-ability test-takers accidentally get it right? On a five-choice question, random guessing alone gives a 20% correct rate. NBME needs to know if answers are inflated by luck.
Why No Amount of Expert Review Can Replace Live Data
You can assemble a panel of twelve clinician-educators, have them each estimate the difficulty of a question, and average their predictions. NBME does exactly this. But psychometric research shows consistently that expert estimates of question difficulty are often wrong — sometimes dramatically — when compared against what real test-takers actually do under time pressure, in the context of a full exam. Until thousands of examinees answer a question live, predictions are calibrated guesses at best.
This is why pretesting isn’t optional. It’s the mechanism that ensures the questions counting toward your score have been validated against real-world performance data, not just committee opinion.
The Angoff Connection
USMLE passing scores are set via a modified Angoff procedure: expert panels predict what percentage of “minimally competent” physicians would answer each question correctly. Those predictions require calibration against actual data. Pretest items generate exactly that calibration data before a question ever becomes scored.
Understanding this also reveals what gets a question rejected after pretesting:
Answered correctly by >90%
Near-universal correct answers provide minimal discriminatory value. If everyone gets it right, the question can’t separate strong from weak candidates.
Answered correctly by <30%
Likely testing esoteric knowledge rather than clinical competence, or the question itself is flawed. Poor discrimination at the pass/fail threshold.
Zero discrimination
Performance on it doesn’t correlate with overall exam performance. If strong and weak students answer it at the same rate, it’s not measuring what it’s supposed to measure.
Wrong answers attract high performers
When high-ability test-takers cluster on a wrong answer, it’s a sign of ambiguous or misleading question writing. The question gets sent back or retired.
For Step 1 readiness, the NBME Step 1 Score Converter gives you a cleaner performance signal than exam-day question counting ever will.
“Experimental Questions Are Harder” — Why This Belief Costs You Points
The widespread assumption: if a question is impossibly hard or covers something you’ve never encountered, it’s probably experimental — so you can rush through it or dismiss it mentally.
The actual data argues the opposite direction, and acting on this assumption has real consequences.
NBME Favors Questions Most Students Get Right
The USMLE identifies minimally competent physicians, not genius ones. Questions only 15–20% of students answer correctly typically fail pretesting. Questions where 70–85% answer correctly are the most psychometrically valuable — and the most likely to be scored.
“I’ve Never Seen This” Is Not Diagnostic
An IMG with different training might find that “impossible” question completely standard. A subspecialty elective might make an obscure protocol obvious to another student. Your unfamiliarity isn’t evidence of a pretest item.
Real Costs, Zero Benefit
If you rush a question assuming it’s experimental and it’s scored, you’ve surrendered points. If you’re right, you gain nothing — you can’t skip experimental questions or bank the saved time. The assumption is always a losing bet.
Poorly Written Questions Exist Everywhere
Some ambiguous, frustrating questions are pretest items that will get retired. But some are poorly written scored questions that made it through committee review. You cannot tell which category a question belongs to from the question itself.
Block Distribution: What the Randomization Actually Means
The USMLE does not disclose how pretest items are distributed across exam blocks, and the distribution is intentionally randomized. A single 40-question block might contain 5 pretest items or 11. They are designed to be indistinguishable from scored items in format, length, difficulty range, and topic coverage — which is why even experienced physicians reviewing exam questions cannot reliably identify them. The distribution is random, and your only job in every block is to answer each question as well as you can.
Running the Numbers: What Unscored Questions Mean for Your Passing Odds
For students who find the arithmetic helpful:
~137 questions to pass
Total: up to 318 questions. Estimated scored: ~240. Passing threshold: ~57% correct. In practice, roughly 137 correct answers out of ~240 scored questions to pass.
~120 questions to pass
Total: up to 280 questions. Estimated scored: ~200. Passing threshold: ~60% correct. In practice, roughly 120 correct answers out of ~200 scored questions to pass.
The psychological implication: you can miss every single pretest item and it doesn’t change your score at all. If 60–80 questions genuinely vanish from your calculation, the margin for wrong answers is wider than most students calculate when they’re mentally tallying misses after a hard block.
That internal running tally you keep during and after the exam includes an unknown number of questions that don’t count. Stop letting it shape how you approach the blocks that follow.
And if your NBME scores are diverging from your UWorld percentages, here’s why UWorld % doesn’t predict your score the way most students assume.
What This Should Change About How You Study
Understanding the pretest system has two practical implications for prep, not just test day.
Ded.
Don’t chase obscure content. Questions testing truly rare diseases and unusual genetic pathways — the kind where fewer than 20–30% of test-takers answer correctly — tend to fail their pretest evaluation and get retired before becoming scored. The USMLE’s own quality filter systematically eliminates ultra-niche content. Your energy belongs on patterns that show up repeatedly across NBME forms.
Ded.
Start treating “I’ve never seen this” differently. When you hit an unfamiliar question in a timed UWorld block, practice the correct internal response: reason from first principles, commit to your best answer, move on. Don’t spiral. This mental habit is what separates students who stay composed on exam day from those who let one strange question derail an entire block.
Ded.
Resist the temptation to panic-study rare content in your final two weeks. The pretest system argument is clear: questions that only a small fraction of students answer correctly are statistically useless to NBME’s scoring model. Take NBME practice forms seriously — they reflect the scored content that has already passed psychometric review.
Wk
Stop new UWorld blocks. Build the exam-day mental model instead: every question gets the same careful treatment, regardless of how familiar or alien it feels. Use your study plan to structure review, not new exposure.
FAQ: USMLE Experimental Questions
Que. Do experimental questions count against me if I get them wrong?
Ans. No. Pretest items carry zero weight in your final score. You gain nothing for answering them correctly and lose nothing for getting them wrong. Their entire purpose is to generate the statistical data the NBME uses to evaluate questions for future exam forms.
Que. Can I identify which questions are experimental during the exam?
Ans. No — and this is by design. The NBME constructs pretest items to be indistinguishable from scored items in format, length, and difficulty range. Bryan Carmody (the Sheriff of Sodium) makes the point explicitly: just because a question tests a concept not in First Aid doesn’t mean it’s experimental. Even experienced physicians reviewing exam questions cannot reliably identify pretest items. Any attempt to identify them during your exam is cognitive bandwidth spent for zero return.
Que. Are experimental questions harder than regular questions?
Ans. Not by design, and the IRT data suggests the opposite pattern. NBME’s psychometric selection process favors questions that a meaningful majority of test-takers answer correctly, because those questions best discriminate at the pass/fail threshold. Questions almost nobody gets right typically fail their pretest evaluation and get retired — they’re too hard to be useful to NBME’s scoring model.
Que. How many experimental questions will I have on Step 1 vs. Step 2 CK?
Ans. The USMLE doesn’t say — officially, the count has never been published. The best data comes from a 2020 COVID accommodation announcement in which NBME accidentally revealed the numbers: removing pretest items would yield 200-question Step 1 exams and 240-question Step 2 CK exams. That puts the estimate at approximately 60–78 pretest items per exam — closer to 60 for Step 2 CK, closer to 80 for Step 1 — and every figure, including these, should be treated as an estimate.
Que. Should I change my test-taking strategy because of experimental questions?
Ans. No — but you can change how you interpret your exam-day experience. Treat every question as scored. The only adjustment worth making is psychological: when you hit a question that feels genuinely impossible, you can note “this might be a pretest item” as a way to release some anxiety — then still answer it as carefully as you can. Skipping or rushing is never the right response.
Que. I had a terrible block. Does that mean it was full of experimental questions?
Ans. Maybe — but you can’t know, and it doesn’t change anything. A hard block could have an above-average concentration of pretest items, or it could simply be a hard block of scored questions. The distribution is randomized, and one rough block rarely determines outcomes on an exam this long. Flag it mentally, reset, and don’t let it contaminate the blocks that follow.
There’s one thing the “don’t panic” framing always misses about experimental questions. The pretest system isn’t just a buffer protecting your score from impossible questions — it’s a pipeline that every USMLE test-taker contributes to. The questions you answer today, whether you know the answer or not, generate the data NBME uses to decide which questions deserve to appear on future exams. Every examinee who sat before you helped validate the questions that counted toward your score. Every answer you give today — on a question that might feel pointless — is doing the same for the students who come after you.
You’re not just a test-taker running out the clock on unscored questions. You’re part of the calibration. Use your NBME score trend as your readiness signal, walk into the Prometric center knowing the math is better than your in-exam tallying suggests, and answer every question like it counts — because for someone, somewhere down the line, it does.
