Reinforcement Learning for LLM Post-Training · Module 20

Evaluating What RL Did: pass@1 versus pass@k, Spurious Rewards, Contamination, and the Sharpen-or-Expand Debate

A reported RL gain is a pass@1 number measured on one base model, one prompt template and one benchmark, and each of those three can supply the gain without any new capability. This module derives the estimators, puts the pass@k, random-reward, template and contamination evidence side by side, and ends with a measurement that separates a sharpened sampling distribution from a larger set of solvable problems.

Course 50 min read 18 sources As of 2026-10-27 Prerequisites: Module 10, Module 13, Module 19
Measurement flow for a reported RL gain: five gates in order (same protocol, beyond seed noise, beats a random-reward control, clean benchmark, pass@k at large k); failing the first four leads to protocol artifact, noise, prior amplification or contamination; passing all five splits into expansion if pass@k at large k is higher and sharpening if it is flat or lower Reported gain: pass@1 goes up 1. Same protocol? prompt, T, top-p, k, version no protocol artifact yes 2. Beyond seed noise? paired test, 30+ seeds no noise yes 3. Beats random reward? format and random controls no prior amplification yes 4. Clean benchmark? post-cutoff or decontaminated no contamination yes 5. pass@k at large k? per-problem solved sets higher flat or lower expansion new problems solved sharpening same problems, higher odds
Figure 1. The measurement flow this module builds toward (a schematic, not a result). Gate 5 is the pass@k test of Yue et al.1; gate 3 is the reward-control ladder of Shao et al.2; gate 4 combines the partial-prompt contamination audit of Wu et al.3 with the random-reward negative control that Olmo 3 ran on a decontaminated base4; gates 1 and 2 come from the variance findings of Hochlehnert et al.5. Failing a gate does not make a training run useless; it means the measured number cannot be attributed to the RL signal.

1. What each metric measures

An RL report quotes one or more of four numbers for the same benchmark, and they answer different questions. All of them are statistics of one object: the probability \(\rho_x\) that a single sample from the model, at the evaluation decoding settings, is verified correct on problem \(x\). Temperature, top-p and maximum length are part of the definition of \(\rho_x\); change them and you are measuring a different distribution (Section 8.1).

1.1 pass@1 and its aliases

pass@1 is \(\mathbb{E}_x[\rho_x]\), the mean accuracy over problems. It is estimated by sampling k responses per problem and averaging their correctness. DeepSeek-R1 defines \(\text{pass@1}=\frac1k\sum_{i=1}^{k}p_i\) with k = 64 for AIME and GPQA, 16 for MATH and Codeforces, and 8 for LiveCodeBench, at temperature 0.6 and top-p 0.95. It avoids greedy decoding because greedy decoding of long-output reasoning models "results in higher repetition rates and significant variability across different checkpoints"6 (Supplementary D.1). DAPO reports the same estimator as avg@32: the AIME set is repeated 32 times at temperature 1.0 and top-p 0.77 (§4.1). "pass@1", "avg@k" and "mean@k" are one statistic under three names; what differs between reports is k, temperature and top-p, and those change the value.

1.2 pass@k and the unbiased estimator

pass@k is the fraction of problems for which at least one of k samples is correct: per problem \(1-(1-\rho_x)^k\), averaged over problems. The estimator in use comes from the Codex paper8. Draw \(n\ge k\) samples per problem, count the \(c\) that pass, and compute $$\widehat{\text{pass@}k}=\mathbb{E}_x\!\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right],$$ the complement of the probability that k samples drawn without replacement from the n are all wrong (Eq. 1). The tempting plug-in \(1-(1-c/n)^k\) is biased low, and the gap does not close even for \(n>5k\) (Appendix A, Figure 13). The numerically stable form is the running product \(1-\prod_{i=n-c+1}^{n}(1-k/i)\) (their Figure 3), which the calculator evaluates. Yue et al. set n to the largest k on each curve: 128, 256 or 1,024 depending on the benchmark and model1 (Appendix A.2).

pass@k for one problem: unbiased estimator versus plug-in

Unbiased pass@k
Plug-in 1-(1-c/n)^k
Plug-in error

Formulas from Chen et al. (Eq. 1 and Figure 3)8; c and k are clamped to n. At the defaults, n = 64, c = 4, k = 32, the unbiased value is 94.3% and the plug-in 87.3%.

The error is largest for problems solved a few times in n samples, which decide the large-k end of a curve: n = 256, c = 2, k = 128 gives 75.1% unbiased and 63.4% plug-in (derived). At \(k=n\) the estimator collapses to the indicator "at least one of the n samples passed", so the rightmost point of a curve built with n equal to its largest k is a single Bernoulli draw per problem.

1.3 Maj@k and cons@k

Majority voting asks a different question: is the most frequent final answer among k samples the correct one? It needs an answer extractor and a canonicalization rule, and it measures how strongly the answer distribution concentrates on the right answer rather than whether the right answer is reachable. DeepSeekMath plots Maj@K and Pass@K side by side for K up to 649 (Figure 7); R1 calls it cons@646. For any model Maj@k cannot exceed pass@k, because a correct majority contains a correct sample, and pass@k never decreases with k. R1's own numbers on AIME 2024 sit in that order.

79.8%R1 pass@1, AIME 20246
86.7%R1 majority vote over 64 samples6
90.0%R1 pass@64, AIME 20246

Supplementary E.4 of the v2 report. AIME 2024 has 30 problems5, so 86.7% and 90.0% are 26 and 27 problems (derived); 79.8% is a mean over samples and need not be a multiple of 3.33 points.

Each metric matches a deployment: pass@1 is a single-shot product, Maj@k is sampling without a verifier, pass@k is sampling with a checker such as a test suite. Brown et al. show coverage (their name for pass@k) growing log-linearly over four orders of magnitude of samples, and on SWE-bench Lite DeepSeek-Coder-V2-Instruct goes from 15.9% with one sample to 56% with 25010 (abstract). If your stack can afford k samples and a checker, pass@k at that k predicts product quality, so a run that raises pass@1 while lowering pass@256 moves the best operating point.

1.4 Worked derivation: pass@k as a functional of the pass-rate distribution

Because \(\text{pass@}k=\mathbb{E}_x[1-(1-\rho_x)^k]\), the whole curve is fixed by the distribution of \(\rho_x\) across problems, and two models with the same pass@1 can have very different curves. Small k weights the mean. Large k weights the lower tail: \(1-(1-\rho)^k\) saturates at 1 once \(\rho k\gg 1\) and stays near \(\rho k\) while \(\rho k\ll 1\). One identity makes the effect of concentration explicit. Since \(\mathbb{E}[(1-\rho)^2]=(1-\mu)^2+\sigma^2\) with \(\mu=\mathbb{E}\rho_x\) and \(\sigma^2=\operatorname{Var}(\rho_x)\), $$\text{pass@}2 = 1-\big[(1-\mu)^2+\sigma^2\big],$$ and for \(k>2\) Jensen's inequality makes the same expression an upper bound, \(\text{pass@}k\le 1-[(1-\mu)^2+\sigma^2]^{k/2}\). Dang et al. derived this bound and ProRL quotes it11 (§4.4). At fixed mean accuracy, more variance across problems, meaning more problems pinned at 0 and at 1, gives lower pass@k.

Table 1 builds a constructed population to show the effect. It is arithmetic, not data. The "after RL" pass rates imitate the pattern Yue et al. report: RL raises the share of problems near accuracy 1.0 and also raises the share stuck at 01 (Figure 5, §4.1).

Table 1. Constructed population of 100 problems (illustrative arithmetic, not measured). Per-class pass rate \(\rho\) before and after RL, and pass@256 per class from \(1-(1-\rho)^{256}\).
ClassProblems\(\rho\) base\(\rho\) after RLpass@256 base (%)pass@256 after RL (%)
Easy400.600.97100.0100.0
Medium300.150.35100.0100.0
Hard200.010.00192.422.6
Near zero100.000102.50.0
Whole population100pass@1 28.7%pass@1 49.3%88.774.5

The model that is 20.6 points better at k = 1 is 14.2 points worse at k = 256, and the curves cross at k = 16 (70.8% against 70.3%; at k = 15 they are tied, 70.19% against 70.25%). The variances are 0.068 before and 0.170 after, so pass@2 is 42.3% and 57.3%. Had the RL model reached its mean of 0.493 with the base model's variance, pass@2 would be 67.5%; polarization costs it 10.1 points at k = 2, and at k = 256 the loss comes from the hard and near-zero classes.

Constructed pass@k curves: the RL-sharpened model is higher at k from 1 to 8, the base model is higher from k of 16, reaching 88.7 versus 74.5 percent at k of 256 0 25 50 75 100 1 2 4 8 16 32 64 128 256 512 1024 samples per problem k (log scale) pass@k, % of problems 28.7 49.3 88.7 74.5 91.0 82.8 crossover k = 8 to 16 base model after RL (sharpened)
Figure 2. Schematic pass@k curves for the constructed population of Table 1 (computed from \(1-(1-\rho)^k\); not measured). The RL-sharpened model leads for k up to 15, the base model leads from k = 16, and at k = 256 the order is 88.7% against 74.5%. The shape, not the numbers, is what Yue et al. measure on real models1 (Figure 2).

2. The sharpening evidence

2.1 DeepSeekMath saw it first

The sharpen-or-expand question was visible in the paper that introduced GRPO. DeepSeekMath compared the instruction-tuned 7B model with its GRPO-trained successor on GSM8K and MATH at temperature 0.7, for K up to 64, and found that "RL enhances Maj@K's performance but not Pass@K". The authors read this as RL "rendering the output distribution more robust": the improvement "is attributed to boosting the correct response from TopK rather than the enhancement of fundamental capabilities"9 (§5.2.2, Figure 7). They also name a possible cause that is a property of their pipeline rather than of RL: the RL data were questions from the instruction-tuning stage, sampled with plain nucleus sampling, which they suspect is "a potential reason that our RL pipeline only improves the Maj@K performance"9 (§5.2.3). That is one model and two benchmarks1 (§6). Module 9 covers the GRPO objective itself.

2.2 Yue et al.: pass@k at large k

Yue et al. test the claim at scale1. For math they take released zero-RL checkpoints (GRPO from a base model with a correctness reward only, SimpleRLZoo on GSM8K and MATH) for Qwen2.5-7B, 14B and 32B and LLaMA-3.1-8B, plus Oat-Zero-7B and DAPO-32B, and they add released code and vision models (§3). Base and RL models share the same zero-shot prompt, with no few-shot examples for the base, and sampling uses temperature 0.6, top-p 0.95 and up to 16,384 generated tokens (§3). The pattern is the same in every family: the RL model wins at small k, and "as k increases, with steeper curves, base models consistently catch up to and eventually surpass RL-trained models". On Minerva with a 32B model the base is about 9 points ahead at k = 128; Oat-Zero and DAPO start nearly 30 points ahead of their base and are eventually passed by it (§3.1). Code and visual benchmarks behave the same way (§3.2, §3.3).

Two details matter. Base models "often produce unformatted or non-sensical responses" without few-shot prompts, yet with enough samples they produce correctly formatted, correct answers (§3). Part of an early pass@1 gain is therefore the model learning the expected answer form, while pass@k at large k looks through format failures to what is reachable. And because math checking is answer-only, Yue et al. read chains of thought by hand: of 25 GSM8K problems the base solved at accuracy between 0 and 5%, 24 had at least one correct chain (23 of 25 for the RL model), and on low-accuracy AIME 2024 problems the base had a correct chain in 5 of the 6 cases they could judge (§3.1).

Qwen2.5-7B under GRPO: pass@1 on Omni-MATH train rises from 9.9 to 42.5 over 450 steps while pass@256 falls from 67.2 to 64.3; the test split goes from 10.2 to 28.3 and from 69.1 to 63.9 0 10 20 30 40 50 base 150 300 450 GRPO training step pass@1 rises 9.9 26.1 33.6 42.5 10.2 25.1 27.1 28.3 60 62 64 66 68 70 72 base 150 300 450 GRPO training step pass@256 falls 67.2 64.3 69.1 63.9 right panel: axis starts at 60, not 0 Omni-MATH train Omni-MATH test
Figure 3. The controlled run in Yue et al.: Qwen2.5-7B trained with GRPO on Omni-MATH-Rule (2,000 training problems, 8 samples per prompt, 8,192-token rollouts, temperature 1.0, no KL term), evaluated at pass@1 and pass@2561 (§4.3-§4.4, Table 4). Over 450 steps train pass@1 rises 4.3-fold (9.9 to 42.5) while train pass@256 falls 2.9 points (67.2 to 64.3); the in-domain test split moves 10.2 to 28.3 and 69.1 to 63.9.

The controlled experiment in Figure 3 isolates training duration. The same trend shows up per problem. Yue et al. partition problems by whether the base model, the RL model, both or neither solved them at least once, with k = 1,024 for AIME 2024 and k = 128 for MATH-500 (Table 2 of the paper, Figure 4 here). The RL model solved almost nothing the base could not: the RL-only slices are 0.0% of AIME 2024 and 1.0% of MATH-500 (about 5 problems), and the base solves all of those with 1,024 samples (Appendix C.7). Perplexity gives a second view. They score RL-model responses under the base model (median over 32 samples, averaged over 10 problems); the perplexity falls from 1.244 at the base through 1.219 and 1.176 to 1.159 at the end of training (Figure 15). The RL model increasingly writes what the base already considers likely, which is what "sharpening" means operationally.

Share of problems solved at least once by the base model, the RL model, both or neither: on AIME24 at k of 1,024, 63.3 percent both, 13.3 percent only the base, 0.0 percent only RL, 23.3 percent neither; on MATH-500 at k of 128, 92.4, 3.6, 1.0 and 3.0 AIME24, k = 1,024 30 problems 63.3 13.3 23.3 MATH-500, k = 128 500 problems 92.4 3.6 3 100% 0% both solve only base solves only RL solves neither RL-only slice: 0.0 percent on AIME24, 1.0 percent on MATH-500
Figure 4. Share of problems solved at least once by the base model, the RL model, both or neither, from Table 2 of Yue et al.1. AIME 2024 has 30 problems, so the slices are 19, 4, 0 and 7 problems (derived: 63.3, 13.3, 0.0 and 23.3 percent of 30). Base solves 23 problems at k = 1,024, the RL model 19.

2.3 The algorithm does not change the picture

Yue et al. re-implemented six RLVR algorithms in one framework (PPO, GRPO, Reinforce++, RLOO, ReMax and DAPO; KL removed, constant learning rate \(10^{-6}\), 8 responses per prompt) and compared pass@1 and pass@256 on the same Omni-MATH split1 (§4.3, Table 3). Their summary is that all six "perform similarly and remain far from optimal". Table 2 reproduces the numbers and adds one derived column: the sampling-efficiency gap \(\Delta_{SE}\) as the paper defines it, the base model's pass@256 minus the RL model's pass@1.

Table 2. Qwen2.5-7B and six RLVR algorithms on Omni-MATH (percent). Source: Yue et al. Table 31. The last column is derived here as 69.1 minus the algorithm's test pass@1.
ModelTrain pass@1Train pass@256Test pass@1Test pass@256\(\Delta_{SE}\) on test
Qwen2.5-7B (base)9.967.210.269.1n/a
GRPO26.166.325.168.344.0
PPO27.265.826.869.242.3
ReMax24.465.523.867.545.3
RLOO28.666.428.169.241.0
Reinforce++28.267.728.069.741.1
DAPO31.466.126.567.042.6

The spread between algorithms in pass@1 is 7 points on the training split (24.4 to 31.4) and the spread in pass@256 is 2.2 points (65.5 to 67.7), against a base model at 67.2. The paper's text quotes \(\Delta_{SE}\) of 43.9 for GRPO and 42.6 for RLOO on the test split; recomputing from its Table 3 gives 44.0 for GRPO (rounding) and 41.0 for RLOO, while 42.6 is DAPO's, so use the table. Modules 11 and 15 argue that loss and normalization choices move efficiency more than the ceiling; this is the same observation from the pass@k side, on one base model and one benchmark. Other choices did not rescue the boundary: a KL term at 0.001 gave similar pass@1 and a much lower pass@128, 32 rollouts per prompt instead of 8 helped a little, and raising the RL model's temperature to match the base's entropy did not close the gap (§4.4-§4.5).

2.4 Independent checks

Hochlehnert et al. re-evaluated open RL checkpoints under one standardized stack and report "a minor diversity collapse": DeepScaleR-1.5B and FastCuRL-1.5B trained from DeepSeek-R1-Distill-1.5B were below their starting point in pass@k for k up to 128 on AIME24, AMC23 and MATH500, with every \(\Delta\)pass@k negative5 (§5.2, Figure 12). The same study found significant pass@1 gains for DeepScaleR and FastCuRL and robust gains for Nemotron-RR, which by its name is ProRL's released model5 (§4.3). Pass@1 gains and a shrinking boundary are compatible, and several of those recipes show both.

The Invisible Leash study (first author Fang Wu; v5 of 14 October 2026) tracks outcomes question by question instead of averaging12. It defines the empirical support of a model as the set of questions with at least one verified success in n samples, and records for each question whether RL preserved, expanded or shrank access to a verified solution. In the benchmark aggregates it evaluated, "more questions lose a verified success than gain one at the stated sampling budgets", alongside gains in pass@1; on AIME25 and Minerva coverage rose at intermediate checkpoints and ended below the initial level, so expansion can be temporary; and some RLVR models raised average token-level entropy while lowering final-answer entropy12 (abstract, §1). Qwen3's on-policy distillation experiment, which found pass@64 flat under RL, is in Section 713.

A training-free result points the same way. Karan and Du sample from a sharpened version of the base model's own likelihood with an MCMC-inspired procedure, with no training, data or verifier, and report boosts that "nearly match and even outperform" RL on single-shot MATH500, HumanEval and GPQA without RL's diversity collapse14 (abstract). If reweighting base likelihoods reproduces most of the single-shot gain, that gain is a property of the base distribution; it does not show RL does nothing else.

3. The expansion claim: ProRL

3.1 The claim and the recipe

ProRL's authors argue that conclusions like Yue et al.'s "may stem from methodological constraints rather than fundamental limitations of RL": earlier studies leaned on math, where models are heavily trained in both pre-training and post-training, and stopped RL after "no more than hundreds of steps"11 (§1). Their recipe is GRPO with DAPO's decoupled clipping (\(\epsilon_{low}=0.2\), \(\epsilon_{high}=0.4\)) and dynamic sampling that drops prompts with accuracy 0 or 1, plus a KL penalty to a reference policy that is hard-reset, together with the optimizer state, to a recent snapshot of the policy when validation stagnates (§2.3, §3.2-§3.3). The run starts from DeepSeek-R1-Distill-Qwen-1.5B and trains on 136K problems from five domains (math, code, STEM, logic puzzles, instruction following) with 16 rollouts per prompt at temperature 1.2, a 256-prompt batch, and about 16,000 H100 GPU-hours, for more than 2,000 steps (§3.1-§3.3).

3.2 What it measured

Single-sample gains are large and not in dispute (Figure 5): the average over math benchmarks goes from 44.45 to 60.14, code from 23.08 to 37.49, GPQA Diamond from 15.86 to 41.78, IFEval from 44.05 to 66.02, and the Reasoning Gym logic-puzzle reward from 4.24 to 59.06 (Tables 1-3; pass@1 from 16 samples for math, code and GPQA, mean verifier reward for IFEval and the puzzles; temperature 0.6, top-p 0.95, 32K tokens). One reporting inconsistency: the introduction gives average gains of 14.7 (math), 13.9 (code), 25.1 (STEM) and 18.1 (instruction following); the body gives 15.7, 14.4, 25.9 and 22.0, and the tables agree with the body (60.14 minus 44.45 is 15.69; 37.49 minus 23.08 is 14.41; 41.78 minus 15.86 is 25.92; 66.02 minus 44.05 is 21.97, derived). All are absolute points. Module 13 quotes the introduction's figures.

ProRL pass@1 or average reward before and after prolonged RL on a 1.5B model: math 44.45 to 60.14, code 23.08 to 37.49, GPQA 15.86 to 41.78, IFEval 44.05 to 66.02, logic puzzles 4.24 to 59.06, and held-out tasks acre 5.99 to 58.57, boxnet 0.00 to 7.91, game 3.49 to 52.29 0 10 20 30 40 50 60 70 44.5 60.1 Math avg 23.1 37.5 Code avg 15.9 41.8 GPQA 44.0 66.0 IFEval 4.2 59.1 Logic puzzles 6.0 58.6 acre 0.0 7.9 boxnet 3.5 52.3 game trained domains held-out Reasoning Gym tasks DeepSeek-R1-Distill-Qwen-1.5B (base) after ProRL (final)
Figure 5. ProRL pass@1 (math, code, GPQA) or mean rule-based reward (IFEval, logic puzzles and the three held-out tasks) before and after prolonged RL on a 1.5B model11 (Tables 1-3). The base is DeepSeek-R1-Distill-Qwen-1.5B. The held-out tasks (acre, boxnet, game of life halting) are Reasoning Gym tasks not seen in training.

The boundary evidence comes from pass@k. ProRL re-evaluates with 256 samples on 18 of the 96 Reasoning Gym tasks plus the other benchmarks and finds a strong negative relationship between the base model's pass@128 on a task and the gain in pass@128 after RL: tasks the base already solves show "minimal or even negative gains in reasoning breadth", and tasks where it struggles gain most (§4.1, Figure 3). Per task the curves fall into three regimes (§4.2, Figure 4): diminish (math, LiveCodeBench, letter counting, where pass@128 often declines while pass@1 rises), plateau (GPQA Diamond, maze, palindrome, where the final checkpoint adds nothing), and sustained gains (Codecontests, TACO, dice). On the held-out boxnet task the base model "exhibits no capability of solving the task" and the final model does (§4.3). ProRL's own math results therefore agree with Yue et al.; a claim of expansion rests on code and logic tasks where the base starts weak, and on the Creativity Index, which measures how much a response overlaps a pretraining corpus (Dolma in the per-task analysis) and rises with training (§1, §4.1, Figure 1).

3.3 Why expansion needs a signal to start from

RL can only reinforce what it samples. Dr. GRPO: if a base policy "cannot even sample a single trajectory that leads to the correct final answer, it is impossible for RL to improve the policy because there is no reward signal"15 (§2.1). The ProRL budget lets you put numbers on it (derived here). A batch of 256 prompts for about 2,000 steps is 512,000 prompt slots against 136,000 problems, about 3.8 visits per problem on average before dynamic sampling removes prompts. With 16 rollouts per visit, a problem whose pass rate under the training policy is \(\rho\) yields at least one correct rollout with probability \(1-(1-\rho)^{16}\):

Table 3. Chance that one visit with 16 rollouts contains a correct rollout, and expected visits before the first one (derived from \(1-(1-\rho)^{16}\)).
Pass rate \(\rho\)P(at least one correct in 16)Expected visits to first correct
0.0011.6%63.0
0.0114.9%6.7
0.0556.0%1.8
0.297.2%1.0

A problem at \(\rho=0.001\) needs about 63 visits to supply its first reward, roughly 17 times the average number of visits it gets. Direct reinforcement therefore reaches problems the starting policy already solves about 1% of the time or more; a problem near zero can only improve through transfer from other problems. That fits ProRL's finding that gains track base-model weakness without being zero, and Yue et al.'s tiny RL-only slice. It does not decide whether transfer can lift near-zero problems.

3.4 Not a matched comparison

Both studies report real measurements; they differ in at least six ways, so neither isolates a cause.

Table 4. Setup differences between Yue et al.1 and ProRL11, from their stated settings.
AttributeYue et al.ProRL
Starting checkpointBase Qwen2.5 (7B, 14B, 32B), LLaMA-3.1-8B for math; instruct or distilled models for code and visionDeepSeek-R1-Distill-Qwen-1.5B, already fine-tuned on R1 reasoning traces
Training lengthReleased checkpoints; controlled run to 450 stepsMore than 2,000 steps, about 16,000 GPU-hours
Training tasksOne domain per run (GSM8K and MATH, Omni-MATH, code, geometry)One run on 136K problems in five domains
KL termRemoved; a 0.001 KL ablation lowered pass@128Kept, with periodic reference and optimizer resets
Rollouts and temperature8 per prompt, temperature 1.0, 8,192 tokens (controlled run)16 per prompt, temperature 1.2, 8k tokens (16k at the end)
EvaluationTemperature 0.6, top-p 0.95, 16,384 tokens; k to 256 or 1,024Temperature 0.6, top-p 0.95, 32K tokens; pass@k with 256 samples on a subset of tasks

One difference is conceptual. ProRL's baseline is a distilled model that has already absorbed teacher traces, which Yue et al. find can expand the boundary1 (§4.2). Expansion relative to a distilled start does not contradict a bound set by the pre-distillation base, so name the base in every claim. The camps also overlap: Ximing Lu, Mingjie Liu, Yi Dong and Yejin Choi are authors of both ProRL and the Invisible Leash study1112, whose v5 finds shrinkage outweighing expansion in aggregate.

3.5 The evidence on one page

Table 5. What each study measured about the boundary, in the authors' own scope. "Large k" is the largest k the study reports.
StudyStart and recipeMeasureResult at large k
DeepSeekMath9DeepSeekMath-Instruct 7B, GRPO on instruction-tuning questionsMaj@K, Pass@K, K up to 64Maj@K up, Pass@K not
Yue et al.1Base models of several families, six algorithmspass@k to 256 or 1,024Base ahead; RL-only slice 1.0% on MATH-500
Hochlehnert et al.5R1-Distill-1.5B to DeepScaleR, FastCuRLpass@k to 128Every \(\Delta\)pass@k negative
Invisible Leash12Several models, benchmarks and checkpointsPer-question supportMore questions lost than gained
Qwen313Qwen3-8B after off-policy distillation, then RLpass@64, AIME 2024 and 2025Unchanged: 90.0 and 83.3
ProRL11R1-Distill-1.5B, more than 2,000 steps, KL resetspass@k to 128 or 256, per taskDiminish on math, sustained on code and logic
Olmo 3 RL-Zero4Fully open 7B base, decontaminated datapass@32, AIME 2024 and 2025"Decent improvement" (figure only; no numbers in the text)

4. The two-phase reconciliation

Yao et al. (Renmin University of China, Ant Group and Xiamen University; arXiv v1, 5 October 2025) argue that both camps are partly right because policy-gradient updates move probability mass through two stages16. It is a theory with a toy demonstration and a short empirical test by the authors; treat it as an interpretation, not a finding.

4.1 The update rule and what it implies

For a softmax policy over logits \(z\), if action \(v\) is sampled with advantage \(\hat A(v)\), the logits move by \(\Delta z_v=\eta\hat A(v)(1-\pi(v))\) for the sampled action and \(\Delta z_u=-\eta\hat A(v)\pi(u)\) for every other action (Lemma 1). Averaging over a group of samples, Theorem 1 gives the expected update $$\mathbb{E}[\Delta z_v]=\eta\,\pi(v)\Big[(1-\pi(v))\hat A(v)-\sum_{u\ne v}\pi(u)\hat A(u)\Big].$$ If the advantage is centered on the exact mean reward, then \(\sum_u\pi(u)\hat A(u)=0\) and the bracket collapses to \(\hat A(v)\), so \(\mathbb{E}[\Delta z_v]=\eta\,\pi(v)\hat A(v)\) (derived here from the theorem; it is the standard softmax policy gradient in logit space). Two consequences carry the argument. A logit moves in proportion to its own action's probability, so a rare action is slow to change whatever its advantage. And while the common action has positive advantage, its growth pushes the rare action down, because mass is conserved.

The authors read this as two stages (§3.1, Remark 3). In the exploitation stage the policy samples the explored high-reward action and the low-reward one, mass flows to the former, and the rare but better action's probability "remains largely unchanged" or even falls, which they connect to boundary shrinkage. In the exploration stage the common action saturates (\(1-\pi\to0\)), the rare action, when sampled, has positive advantage while the formerly good action becomes the relative negative, and the rare action's probability rises, which they connect to expansion. In pass@k terms they say raising the probability of at least one correct action above \(1/k\) expands the boundary.

4.2 The toy, computed

The paper's toy has three actions: \(a_1\) explored with reward 0.85, \(a_2\) rare with reward 1.0, and \(a_3\) with reward 0 (§3.2). Its own runs sample groups of two with \(\eta=0.1\) over thousands of steps. Figure 6 is the noiseless version computed here from the expected update above, with initial probabilities \(\pi=(0.55,0.05,0.40)\) that I chose.

Three-action toy policy under the expected policy-gradient update: the explored action a1 takes over within the first thousand steps while the rare better action a2 falls to a minimum of about 0.017, then a2 recovers and exceeds 0.5 near step 4,800 and is above 0.99 by step 10,000 0.00 0.25 0.50 0.75 1.00 1 10 100 1,000 10,000 optimization step (log scale, η = 0.1) policy probability phase 1: exploit phase 2: explore min 0.0166 at step 1,195 0.5 at step 4,799 a1: reward 0.85, explored a2: reward 1.0, rare a3: reward 0
Figure 6. Expected-update dynamics of the three-action toy of Yao et al.16, computed here (rewards 0.85, 1.0, 0; \(\eta=0.1\); initial probabilities 0.55, 0.05, 0.40 chosen by me; no sampling noise). The explored action \(a_1\) reaches 0.976 by step 1,000 while the rare better action \(a_2\) falls from 0.05 to a minimum of 0.0166 at step 1,195; then \(a_2\) recovers, passes 0.5 at step 4,799 and reaches 0.994 by step 10,000. Shading marks the authors' two stages, split at the minimum.

The toy exposes a scaling the paper does not state. When \(a_1\) has saturated, \(\bar r\to r_1\), so \(a_2\)'s logit grows at about \(\eta\,\pi_2\,\Delta r\) with \(\Delta r=r_2-r_1=0.15\). The escape time is then of order \(1/(\eta\,\Delta r\,\pi_{2,\min})\), which is \(1/(0.1\times0.15\times0.0166)\approx4{,}000\) steps against 4,799 measured to reach 0.5 (derived; an order-of-magnitude fit, not an exact law). The exploration stage lasts as long as the inverse of the probability that the exploitation stage left behind: more over-exploitation means a longer wait. In the same toy with \(\pi_0=(0.6,0.02,0.38)\) the minimum is 0.0023 and the recovery takes about 33,500 steps to cross 0.5.

A caveat, also derived: the exploration stage exists only because \(r_2>r_1\). With two equally rewarded routes, \(r_1=r_2=1\), the same update gives \(\mathrm{d}(z_2-z_1)/\mathrm{d}t=\eta(\pi_2-\pi_1)\pi_3\), which is negative while \(\pi_1>\pi_2\); in a run from \(\pi_0=(0.6,0.01,0.39)\) the rare route falls from 0.01 to 0.0002 by step 50,000 and does not return. The paper says its binary setting gets differences among positive rewards from the importance ratio (footnote 3). For a binary verifier the stage structure therefore says little about problems where both routes are correct; it speaks to cases where the signal ranks correct routes, such as graded rewards.

4.3 What the paper tested

The empirical part trains Qwen2.5-Math-7B and Llama-3.2-3B-Instruct on the 7,500 MATH training problems with GRPO, GSPO and variants that keep only the relative negative gradients (GRPO-N, GSPO-N), with a uniform chat prompt, KL 0.001, an entropy bonus of \(10^{-4}\), and 256 samples per problem at temperature 0.6 for evaluation (§4, Appendix C.2)16. The training curves in its Figure 2 run to step 42. Plain GRPO lets test entropy fall, while GRPO-N, GSPO and GSPO-N dip and then rise. On Qwen2.5-Math-7B (Table 1), every RL variant lifts AIME 2024 pass@256 from the base 66.7 to 73.3, which is 2 of 30 problems, and GRPO-N lifts AIME 2025 pass@256 from 46.7 to 66.7, which is 6 of 30. That is a short run on the model family where prompts and priors matter most (Section 5), against a base under a chat template, and Dr. GRPO's Table 1 shows chat-style templates are not the best prompt for this model: its Qwen-Math template gives an average of 26.5 against 38.2 with no template15. Whether the AIME gains are expansion or recovery from a poorly matched prompt cannot be separated from the text, and the prolonged-training half of the theory is untested at ProRL scale.

5. Spurious rewards

Module 19 (§2) gives the taxonomy of documented reward exploits. Spurious rewards are a different failure, and a measurement control, so they are covered in full here. The question is how much of an RL gain survives when the reward carries little or no information about correctness.

5.1 The reward ladder

Shao et al. (University of Washington, Allen Institute for AI and Berkeley; arXiv v2, 25 February 2026) run GRPO on Qwen2.5-Math models with the DeepScaleR training data and replace the ground-truth reward with weaker ones, holding everything else fixed2 (§2.1-§2.2). The rewards are binary (Module 10). The ladder, from most to least informative: ground truth; majority vote, where the label is the majority answer of 64 samples from the pre-RL model; format, which rewards any response containing a non-empty \boxed{} whatever it contains; incorrect label, which rewards matching a majority-voted label that is known to be wrong; and random, which gives reward 1 with probability \(\gamma=0.5\) independent of the response (\(\gamma=0\) yields zero gradient, as expected). They report MATH-500 pass@1 and AMC average@8 after 300 steps, and say the weak and spurious rewards are "proposed purely for analytical purposes".

5.2 What it did

On Qwen2.5-Math-7B, random rewards improved MATH-500 by 21.4 points against 29.1 for ground truth, and incorrect labels by 24.12 (Figure 1). On AIME 2024 the same model gains 15.3 with ground truth and 10.2 with both incorrect and random rewards; on AIME 2025, whose questions postdate every model's knowledge cutoff, ground truth has "a clear advantage" and the other rewards give between -0.4 and +4.5 (§2.2, Appendix D). The effect is model-dependent (§3): all non-random rewards, even incorrect ones, help the Qwen2.5 models, OLMo2-7B gains only with ground truth, and each weak or spurious reward fails on at least one non-Qwen model. Smaller models benefit less, and models already RL post-trained, such as Qwen2.5-Math-7B-Instruct, gain little under any reward including ground truth (Appendix J).

MATH-500 gain after 300 GRPO steps under ground-truth, incorrect-label and random rewards: Qwen2.5-Math-7B plus 29.1, 24.1 and 21.4; Qwen2.5-7B plus 34.4, 27.8 and 16.4; Llama3.1-8B-Instruct plus 7.4, minus 6.3 and minus 11.5; OLMo2-7B plus 15.5, minus 6.4 and minus 8.3 -10 0 +10 +20 +30 +40 +29.1 +24.1 +21.4 Qwen2.5-Math-7B base 49.4 +34.4 +27.8 +16.4 Qwen2.5-7B base 41.6 +7.4 -6.3 -11.5 Llama3.1-8B-Instruct base 36.8 +15.5 -6.4 -8.3 OLMo2-7B base 9.0 MATH-500 gain after 300 steps, points ground-truth reward incorrect-label reward random reward
Figure 7. MATH-500 gain in points after 300 GRPO steps under three of the five rewards, for four models2 (Figure 1; base accuracy under each name). The Qwen models gain under every reward; Llama and OLMo lose under incorrect-label and random rewards.
Table 6. MATH-500 accuracy before RL (percent) and gain in points after 300 steps for each reward, from Shao et al. Figure 12. The last column, ground-truth gain minus random-reward gain, is derived here.
ModelBefore RLGround truthMajority voteFormatIncorrect labelRandomGT minus random
Qwen2.5-Math-7B49.4+29.1+27.1+13.8+24.1+21.47.7
Qwen2.5-7B41.6+34.4+31.9+16.7+27.8+16.418.0
Llama3.1-8B-Instruct36.8+7.4+7.2-1.3-6.3-11.518.9
OLMo2-7B9.0+15.5-6.4+0.4-6.4-8.323.8

The last column estimates the gain that an uninformative reward does not reproduce: 7.7 points for the model where the prior does most of the work, 18 to 24 for the others. It bounds what the verifier's information explains only if gains add, which the study does not test.

5.3 Mechanism one: clipping bias

Random rewards have zero mean advantage, so the expected policy gradient should vanish. Shao et al. show it does not, because of the clip2 (§4, Appendix B). They ablate three ways of removing the clipping effect (disabling it, making the minibatch the full rollout, or using one gradient update per rollout so the ratio stays 1) and find that random rewards then give no consistent gain, while with clipping on they give stable gains (Figure 4). Their Eq. 2 says that, for random advantages, the expected gradient is \(+\nabla\log\pi\) when the ratio \(R<1-\epsilon\), zero inside the band, and \(-\nabla\log\pi\) when \(R>1+\epsilon\). Read directly: above the band only negative-advantage samples keep their gradient, so the net push is down; below it only positive-advantage samples do, so the net push is up. The clip is a restoring force toward \(R=1\).

The force is asymmetric because probabilities cannot exceed 1. The upper clip triggers when \(\pi_\theta>(1+\epsilon)\pi_{old}\), which is unreachable once \((1+\epsilon)\pi_{old}\ge1\), that is for \(\pi_{old}\ge1/(1+\epsilon)=0.833\) at \(\epsilon=0.2\) (derived). A token with \(\pi_{old}=0.85\) has its upper threshold at 1.02 and is never pushed back down, while a token with \(\pi_{old}=0.02\) is clipped once \(\pi_\theta\) passes 0.024 and is pushed back (the paper's own example). Under noise, probability mass drifts toward tokens that were already likely: "clipping bias asymmetrically suppresses low-probability tokens and reinforces high-probability ones". DAPO's \(\epsilon_{high}=0.28\) moves the threshold to 0.781 and delays the push-back on rare tokens (Module 13).

The probability a token must reach under the new policy before the upper PPO clip triggers, as a function of its old probability: it is the old probability times one plus epsilon, so for tokens above 0.833 at epsilon 0.2 it exceeds 1 and is never reached 0.00 0.25 0.50 0.75 1.00 0.0 0.5 1.0 probability of the token under the old policy, π_old π_θ where the upper clip starts unreachable: probabilities cannot exceed 1 0.02 to 0.024 0.85 to 1.02 1/1.2 = 0.833 1/1.28 = 0.781 clip ε = 0.2 clip ε_high = 0.28
Figure 8. The new-policy probability at which the upper clip starts, \((1+\epsilon)\pi_{old}\), against the old probability (derived from the clip definition and the worked example of Shao et al.2, §4). Above \(1/(1+\epsilon)\) the threshold exceeds 1 and is never reached, so likely tokens get no restoring push. DAPO's \(\epsilon_{high}=0.28\) is shown for comparison7.

5.4 Mechanism two: an existing behavior gets amplified

Clipping bias says why random rewards sharpen; the next question is what they sharpen toward. For Qwen2.5-Math-7B the answer is writing Python without running it. The model does this in 65.0% of MATH-500 solutions, with no interpreter attached, and answers that do are far more accurate: 60.9% against 35.0% for the rest (Table 1 of the paper; its introduction says 28.0% for the rest, and I use the table)2. Under weak or spurious rewards code frequency rises to about 90% within 15 steps, and to 95.6% under random rewards, tracking the accuracy gain; under ground-truth rewards it rises and later declines as natural-language reasoning improves (§5.2, Figure 6). The behavior is a lever, not just a correlate: forcing the first sentence "Let's solve this using Python." lifts MATH-500 by 15.0 points on Qwen2.5-Math-7B and by 24.2 on the 1.5B model, and rewarding the string "python" alone pushes code frequency above 99% within 20 steps (§5.3, Figure 7). The same intervention hurts the models that lack effective code reasoning.

Table 7. MATH-500 accuracy (percent) with and without forcing the first sentence "Let's solve this using Python.", from Shao et al. Table 22. Differences are absolute points.
ModelOriginalCode-forcedDifference
Qwen2.5-Math-1.5B36.260.4+24.2
Qwen2.5-Math-7B49.464.4+15.0
Qwen2.5-1.5B3.013.0+10.0
Qwen2.5-7B41.622.2-19.4
Llama3.2-3B-Instruct36.88.2-28.6
Llama3.1-8B-Instruct36.815.2-21.6
OLMo2-7B9.07.8-1.2
OLMo2-7B-SFT21.418.6-2.8

5.5 The baseline moves as much as the RL gain

The "before RL" number is not fixed either. Shao et al. evaluate Qwen2.5-Math-7B under five prompts: the Qwen default gives 49.4% on MATH-500 with 78.9% of answers parsable, a "MathProblem:" prompt 55.8%, SimpleRL-Zoo's prompt 63.2%, the Sober Reasoning prompt 61.6% (93.1% parsable), and a block of LaTeX lorem-ipsum text with no task content 68.8% (84.1% parsable)2 (Table 5, Appendix I). The best accuracy does not come from the best parse rate or from task-relevant words. Hochlehnert et al. could not fully reproduce the magnitude but found the same effect on Qwen2.5 models5 (§5.3). A 19.4-point swing (49.4 to 68.8, derived) from a prompt with no content is the size of the 21.4-point gain from random-reward RL. Models trained under three of the prompts converged to similar performance after RL, which the authors take to suggest that what a prompt elicits is a subset of what RL elicits (Appendix I).

5.6 What this does and does not establish

It establishes that for Qwen2.5-Math, a large part of the MATH-500, AMC and AIME 2024 gain needs no information about correctness: any signal that sharpens the policy toward an existing behavior produces it, and "reasoning patterns instilled during pretraining heavily impact the behavior of downstream RLVR training"2. It does not establish that ground-truth RL gains are spurious: ground truth beats random by 7.7 points on the most favorable model, by 18 to 24 on the others, and clearly on post-cutoff AIME 2025. The authors' instruction is to test reward signals on more than one model family and to include format and random rewards as dummy baselines2 (§1, §2.2). Section 6 covers the competing explanation.

6. Contamination and the baseline

Wu et al. (Fudan University and collaborators; arXiv v3, 17 December 2025) explain the Qwen results of Section 5 differently: the benchmark problems leaked into pretraining, and a spurious reward lets GRPO retrieve memorized answers3. Their audit gives the model the first 40, 60 or 80% of a problem and scores the continuation (greedy decoding, no chat template). Given the first 60% of each MATH-500 problem, Qwen2.5-Math-7B regenerates the remaining 40% with 54.60% exact match and answers 53.6% of the truncated problems correctly; Llama3.1-8B manages 3.8% and 2.4%. On LiveMathBench (version 202505), which postdates Qwen2.5, Qwen's completion rate falls to 0.0% and its truncated-problem accuracy to 2.0% (Llama: 0.0% and 1.0%) (Table 2, §1).

Exact-match rate when each model is given the first 60 percent of a benchmark problem and asked to continue: Qwen2.5-Math-7B reproduces 54.6 percent of MATH-500, 42.2 percent of AMC and 20.0 percent of AIME 2024 but 0.0 percent of AIME 2025 and LiveMathBench; Llama3.1-8B reproduces 3.8 percent of MATH-500 and 0.0 percent of the rest 0 10 20 30 40 50 60 54.6 3.8 MATH-500 42.2 0.0 AMC 20.0 0.0 AIME 2024 0.0 0.0 AIME 2025 0.0 0.0 LiveMathBench exact match, % of problems Qwen2.5-Math-7B Llama3.1-8B
Figure 9. Exact-match rate when a model is given the first 60% of a benchmark problem and asked to continue, Qwen2.5-Math-7B against Llama3.1-8B3 (Table 2, greedy decoding, no template). Qwen reproduces more than half of MATH-500, 42.2% of AMC and 20.0% of AIME 2024, and nothing of AIME 2025 or LiveMathBench; Llama is near zero throughout.

They then build a clean benchmark, RandomCalculation: automatically generated arithmetic expressions of 1 to 20 steps, 1,000 problems per sub-dataset, created after Qwen's release, with a continuous reward because exact answers are long decimals. On the 5-step and 10-step sets, correct rewards improve Qwen2.5-Math-7B steadily and beyond the initial model's Max@16, random and incorrect rewards give unstable or no gains, and inverted rewards collapse performance (§4.3). Output similarity before and after RL points the same way: ROUGE-L between pre- and post-RL responses under random reward is 0.601 on MATH-500 but 0.247 and 0.251 on the 5- and 10-step RandomCalculation sets (Table 3), which they read as retrieval of stored answers.

The two accounts agree that no new capability is created and differ on what is amplified, a behavior (Shao et al.: code reasoning) or a memorized item set (Wu et al.). They are not exclusive, and neither paper contains the other's decisive test: Shao et al. do not report their ladder on a decontaminated Qwen-scale set, and Wu et al. do not rule out a behavioral prior. The fully open base of Olmo 3 gives a partial test. Its RL-Zero setup decontaminates the RL data against pretraining and midtraining data and then runs a negative control, random binary rewards on the same data, which "does not improve performance on any of our benchmark evaluations"; performance stays flat or degrades4 (§6.2, Figure 27). Olmo 3 reads this as evidence that its decontamination removed overlap between the base and the evaluation data. It shows this base gives random rewards nothing to amplify; it does not choose between the two accounts of the Qwen result (my reading).

The prompt is the other half of the baseline. Module 10 §6.2 tabulates Dr. GRPO's template results: the R1 template takes Qwen2.5-Math-7B to 0.0 average, no template gives 38.215. Olmo 3 reports the same pattern from the other side: simple templates beat post-training templates on a base mid-trained without special formatting4 (§6.1). The evaluation rule that follows is to score the base under its best prompt as well as under the RL prompt, and to report both.

7. Distillation as a second boundary test

Yue et al. find that distilling R1 into Qwen2.5-Math-7B puts the pass@k curve consistently above the base model's at every k, unlike RL, which they say is bounded by the base1 (§4.2, Figure 7). Qwen3 supplies a controlled version, and Module 21 covers its cost and mechanism. Both arms start from the same off-policy-distilled Qwen3-8B checkpoint and use math and code queries; one continues with RL, the other with on-policy distillation from a stronger teacher13 (§4.7, Table 21).

Qwen3-8B on AIME 2024 and 2025: RL raises pass@1 from 55.0 to 67.6 and from 42.8 to 55.5 but leaves pass@64 at 90.0 and 83.3; on-policy distillation reaches pass@1 74.4 and 65.5 and pass@64 93.3 and 86.7 0 20 40 60 80 100 55.0 67.6 74.4 AIME24 pass@1 90.0 90.0 93.3 AIME24 pass@64 27 / 27 / 28 of 30 42.8 55.5 65.5 AIME25 pass@1 83.3 83.3 86.7 AIME25 pass@64 25 / 25 / 26 of 30 accuracy, % off-policy distilled start + RL (17,920 GPU-h) + on-policy distillation (1,800 GPU-h)
Figure 10. Qwen3-8B on AIME 2024 and 2025 from one off-policy-distilled start, then RL or on-policy distillation13 (Table 21). RL raises pass@1 by 12.6 and 12.7 points and leaves pass@64 at 90.0 and 83.3; on-policy distillation raises pass@1 by 19.4 and 22.7 and pass@64 to 93.3 and 86.7, at 1,800 GPU-hours against 17,920. Problem counts are derived from 30 problems per benchmark.

The paper's reading is that distillation "enables the student model to expand its exploration space" while "reinforcement learning does not lead to any improvement in pass@64 scores". The arithmetic puts a scale on that. AIME has 30 problems5, so 90.0% and 93.3% are 27 and 28 problems, and 83.3% and 86.7% are 25 and 26 (derived). The "expansion" from distillation is exactly one extra problem on each benchmark, and the "no improvement" under RL is the same count before and after, which need not be the same problems. The passage reports no seeds, no intervals and no statement of how pass@64 was estimated. Read it as a direction that agrees with Yue et al., not as a magnitude; the 12 to 23 point pass@1 movements are the robust part. R1's final model itself shows the headroom this metric exposes, 79.8% pass@1 against 90.0% pass@64 on AIME 2024 (Section 1.3). For measurement, the useful question about any teacher-driven gain is whether the newly solved problems lie in the teacher's solved set, which a per-problem partition answers (Section 8.5).

8. A protocol that separates sharpening from expansion

Each gate of Figure 1 matches a documented failure; this section turns the gates into measurements.

8.1 Match the sampling protocol

Reports do not share one. R1, DAPO, Yue et al., ProRL and Olmo 3 differ in temperature, top-p, token cap and sample count (Table 8), so cross-report comparisons such as DAPO's 50 AIME points against 47 for R1-Zero-Qwen-32B7 compare different protocols as well as different recipes. Hochlehnert et al. measured the cost: across nine models and three benchmarks, temperature alone moves accuracy by up to 15%, top-p by up to 8%, and hardware and framework differences leave 2 to 5% even under stringent controls5 (Figures 4 and 5, §3.4).

Table 8. Evaluation protocols as stated in five reports and one standardization study. "n/s" means not stated in the passage read.
ReportMetricTemperaturetop-pMax tokensSamples
R1 v26pass@1 as mean of k; cons@640.60.9532,768k = 64 AIME and GPQA, 16 MATH and Codeforces, 8 LiveCodeBench
DAPO7avg@32 on AIME1.00.7n/sAIME set repeated 32 times
Yue et al.1pass@k0.60.9516,384n up to 1,024
ProRL11pass@1; pass@k0.60.9532K16; 256 on a task subset
Olmo 3 RL-Zero4pass@1; pass@321.0n/s32Kn/s
Hochlehnert et al.5pass@1, sensitivity runs0.80.932,76810 seeds (AIME, AMC), 3 (MATH-500)

8.2 Know the resolution of the benchmark

AIME 2024 has 30 problems and AMC 2023 has 40, so one problem moves pass@1 by 3.33 and 2.5 points; Hochlehnert et al. report seed-to-seed standard deviations of 5 to 15 percentage points, worst on AIME 2024 and AMC 2023, and recommend at least 30 seeds, paired significance tests and per-model tuned decoding5 (§3.2, §4.1). Miller's statistics explain why more samples per problem do not fix this17. The variance of a benchmark mean is \(\operatorname{Var}(\hat\mu)=(\operatorname{Var}(x)+\mathbb{E}[\sigma_i^2])/n\): the variance of true per-problem pass rates, which resampling cannot shrink, plus the mean within-problem variance, which divides by the number of samples K (§3.1). His uniform-difficulty example has \(\operatorname{Var}(x)=1/12\) and \(\mathbb{E}[\sigma_i^2]=1/6\); with \(n=30\), going from one sample to 32 cuts the standard error from 9.13 to 5.43 points, and no number of samples gets below 5.27 (derived). Under that difficulty distribution, an avg@32 comparison of two unpaired models on AIME 2024 cannot resolve a 5-point gap. Scoring both models on the same problems and using question-level differences removes shared difficulty and is Miller's recommended analysis (§4.2).

Table 9. Resolution of common benchmarks at 50% accuracy, one sample per problem (derived from \(\sqrt{p(1-p)/n}\)17, Eq. 2 and Eq. 3). Sizes: AIME 2024 and AMC 20235; LiveMathBench and MATH-5003.
BenchmarkProblemsOne problem (points)Standard error (points)95% CI half-width (points)
AIME 2024303.339.1317.89
AMC 2023402.507.9115.50
LiveMathBench1001.005.009.80
MATH-5005000.202.244.38

8.3 Large-k pass@k counts lucky guesses

With answer-only checking, pass@k at large k also counts guesses. Under uniform guessing over \(N\) possible answers the chance of a hit within k samples is \(1-(1-1/N)^k\) (derived). Real guesses are not uniform, so Table 10 gives a scale, not a bound. Take AIME answers to be integers from 0 to 999, so N = 1,000: a guesser alone would reach 64% at k = 1,024, which is why Yue et al. checked chains of thought by hand1 (§2.2, §3.1). Binary grading making a best guess optimal over abstaining is the training-side version of the same incentive18 (Module 24).

Table 10. Probability that uniform random guessing over \(N\) equally likely answers hits within k samples (derived from \(1-(1-1/N)^k\)).
Answer space Nk = 8k = 64k = 256k = 1,024
490.0%100.0%100.0%100.0%
1007.7%47.4%92.4%100.0%
1,0000.8%6.2%22.6%64.1%

8.4 What large k costs

Large k is a serving workload. Yue et al. could not run pass@k on R1-Zero: their self-hosted copy delivered about 50 tokens per second at a 32k maximum length1 (§4.6). AIME 2024 at n = 1,024 is 30,720 samples, at most 1.01 billion generated tokens at the 32,768-token cap, which is about 233 days at 50 tokens per second (derived upper bound; real lengths are shorter and a batched engine is far faster). Cost scales as problems times n times mean length, so a 1,024-sample partition of MATH-500 is 512,000 samples. Rollout-engine throughput (System Design Module 37) sets how far out the boundary can be measured.

8.5 The checklist, in the order of Figure 1

  1. Same protocol. Score the base and the RL model with the same prompt, temperature, top-p, token cap and k, and also score the base under its own best prompt (Sections 5.5 and 6). Pin report versions: R1-Zero's AIME 2024 pass@1 is 71.0% in v1 and 77.9% in v26 (Module 10).
  2. Estimator and n. Use the unbiased pass@k estimator with \(n\ge k\), report n, and remember that at \(k=n\) the point is a single indicator per problem (Section 1.2).
  3. Noise. Report seeds, standard errors with the benchmark's size in view, and paired differences against the base (Section 8.2).
  4. Control rewards. Run a random-reward and a format-reward run through the same pipeline; their gain is the part of the claim that needs no verifier (Section 5; Olmo 3's negative control).
  5. Clean benchmark. Prefer post-cutoff or decontaminated sets, and run the partial-prompt completion probe on the base model (Section 6).
  6. Boundary. Compute pass@k out to \(k\) in the hundreds or more, partition problems into both, base-only, RL-only and neither, and spot-check chains of thought on the problems that decide the result (Sections 2.2 and 8.3). Track the partition across checkpoints, since expansion can be temporary12.

Where experts disagree

The live question is whether RLVR creates solvable problems or raises the odds on problems the base could already solve. The sources split into three answers and a fourth position that questions what the benchmark gains measure.

RLVR sharpens; the base model sets the boundary

Yue et al.1; DeepSeekMath9; Qwen3's Table 2113; the Invisible Leash study12; Hochlehnert et al.5; Karan and Du14

pass@k at large k, per-problem solved sets and base-model perplexity of RL outputs all show the RL model solving a subset of what the base solves.

  • Replicated with different pipelines and metrics, including per-question tracking and a training-free sampler.
  • Operational: the claim is a measurement, not a story.
  • Mostly short runs, strong math bases and answer-only checking.
  • Distilled starts and weak-base tasks show different behavior.

RLVR can expand with long, diverse, KL-controlled training

ProRL11; Olmo 3's RL-Zero pass@324

Over 2,000 steps on 136K problems from five domains, gains were largest where the base started weak, and a held-out task the base could not solve at all was solved.

  • Sustained pass@k gains on code and logic tasks; a clean 7B base also improved at pass@32.
  • Names a concrete reason earlier studies may have stopped too early.
  • Starts from a distilled 1.5B model, and its own math results diminish.
  • Not a matched comparison with Yue et al. (Table 4); Olmo 3's pass@32 is shown in a figure only.

Both, in two phases

Yao et al.16

Probability mass first concentrates on explored high-reward routes (shrinkage), then, if training continues, flows to rare better ones (expansion).

  • Explains both camps and predicts when each appears.
  • The toy gives a duration scaling, about \(1/(\eta\,\Delta r\,\pi_{\min})\).
  • In my derivation (Section 4.2) its exploration stage needs a reward gap between correct routes, which a binary verifier does not give.
  • Tested for about 42 steps; not tested at ProRL scale.

Most measured gains are prior amplification or contamination

Shao et al.2; Wu et al.3; Dr. GRPO15; Olmo 34

Random and incorrect rewards reproduce most of a Qwen2.5-Math gain; a partial-prompt probe shows MATH-500 is memorized; the template alone moves the baseline by tens of points.

  • Cheap controls any team can run, and they explain why Qwen results do not transfer.
  • Consistent with the post-cutoff AIME 2025 result.
  • Ground truth still beats random by 7.7 to 23.8 points.
  • The two mechanisms (code behavior versus memory) are disputed, and this position says little about the boundary.
Assessment

For strong math bases, single-domain RLVR and runs of hundreds of steps, the sharpening evidence is the best supported, and the Qwen2.5-Math numbers should be treated as unconfirmed until a random-reward control and a second model family agree. The pass@1 gains are real; what they do not show is new problems solved. For weak starts, diverse tasks and long runs the expansion case is plausible but rests on one 1.5B distilled model, and the two-phase theory is an interpretation of it, not evidence. If your deployment is single-shot, sharpening is the goal; if it samples k times against a checker, evaluate pass@k at your k. What would settle it is an experiment I found in none of these sources: an open decontaminated base such as Olmo 3, one task mix and one evaluation protocol, RL from step 0 past 2,000 steps, per-problem partitions at 1,024 samples on a clean held-out set at every checkpoint, chain-of-thought validity checks and a random-reward control, repeated on a second family.

What is changing

This page is as of 2026-10-27, and several sources it depends on are revised preprints: the Invisible Leash study reached v5 on 14 October 2026 with its per-question support framework12, Spurious Rewards is at v2 of February 20262, and Yue et al. is at v51. Check versions before quoting a number. The released open base, data and logs of Olmo 3 make the matched ladder of Figure 1 runnable outside a frontier lab4.

References

  1. paperYue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., Huang, G. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837v5, 24 Nov 2025 (Tsinghua LeapLab, SJTU). https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Used for: pass@k protocol (§2.2, §3), Figures 2, 5, 7, 15, Tables 2-4, controlled run (§4.3-§4.5), CoT validity checks (§3.1), estimator settings (App. A.2), solvable-set partition (App. C.7).
  2. paperShao, R., Li, S. S., Xin, R., Geng, S., et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947v2, 25 Feb 2026 (University of Washington, Allen Institute for AI, UC Berkeley). https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: reward ladder (§2), Figure 1 and Table 6 data, clipping bias (§4, Eq. 2, App. B), code reasoning (§5, Tables 1-2), prompt sensitivity (Table 5, App. I), post-RL models (App. J), AIME results (App. D).
  3. paperWu, M., Zhang, Z., Dong, Q., Xi, Z., et al. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. arXiv:2507.10532v3, 17 Dec 2025 (Fudan University et al.). https://arxiv.org/abs/2507.10532. Accessed 2026-10-27. Used for: partial-prompt completion audit (Table 2), RandomCalculation experiments (§4.3), response similarity (Table 3), benchmark sizes.
  4. paperTeam Olmo. Olmo 3. arXiv:2512.13961v2, 14 Apr 2026 (Allen Institute for AI). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: RL-Zero motivation (§2.2), template finding and evaluation settings (§6.1), pass@32 result and random-reward negative control (§6.2, Figure 27).
  5. paperHochlehnert, A., Bhatnagar, H., Udandarao, V., Albanie, S., Prabhu, A., Bethge, M. A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility. COLM 2025; arXiv:2504.07086v2, 6 Oct 2025. https://arxiv.org/abs/2504.07086. Accessed 2026-10-27. Used for: seed, temperature, top-p and hardware variance (§3, Figures 2, 4, 5, Takeaway 6), best practices (§4.1), RL replication results (§4.3), diversity collapse (§5.2), spurious prompt (§5.3), benchmark sizes.
  6. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (4 Jan 2026; v1 22 Jan 2025 for the 71.0% figure). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: evaluation setup and pass@1 definition (Supplementary D.1), pass@1, cons@64 and pass@64 on AIME 2024 (Supplementary E.4), R1-Zero version numbers.
  7. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 20 May 2025 (ByteDance Seed, Tsinghua AIR). https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: avg@32, temperature 1.0 and top-p 0.7 (§4.1), 50 versus 47 AIME points, Clip-Higher epsilon values.
  8. paperChen, M., Tworek, J., Jun, H., Yuan, Q., et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021 (OpenAI). https://arxiv.org/abs/2107.03374. Accessed 2026-10-27. Used for: the unbiased pass@k estimator (Eq. 1), the numerically stable implementation (Figure 3), and the bias of the plug-in estimator (Appendix A, Figure 13).
  9. paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: Maj@K versus Pass@K (§5.2.2, Figure 7) and the data-source caveat (§5.2.3).
  10. paperBrown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., Mirhoseini, A. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787v3, 30 Dec 2024. https://arxiv.org/abs/2407.21787. Accessed 2026-10-27. Used for: coverage scaling with samples and the SWE-bench Lite figures (abstract).
  11. paperLiu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., Dong, Y. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1, 30 May 2025 (NVIDIA). https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: claim and recipe (§1-§3), Tables 1-3, pass@k analysis (§4, Figures 3-5), Jensen bound on pass@k (§4.4), training and evaluation settings.
  12. paperWu, F., Xuan, W., Xing, D., Lu, X., Liu, M., Dong, Y., Harchaoui, Z., Choi, Y. The Invisible Leash? Why RLVR May or May Not Escape Its Origin. arXiv:2507.14843v5, 14 Oct 2026. https://arxiv.org/abs/2507.14843. Accessed 2026-10-27. Used for: empirical support dynamics, shrinkage versus expansion findings (abstract, §1), temporary expansion at intermediate checkpoints, author list.
  13. paperYang, A., Li, A., Yang, B., Zhang, B., et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: Table 21 and the RL versus on-policy distillation discussion (§4.7).
  14. paperKaran, A., Du, Y. Reasoning with Sampling: Your Base Model is Smarter Than You Think. arXiv:2510.14901v1, 16 Oct 2025 (Harvard). https://arxiv.org/abs/2510.14901. Accessed 2026-10-27. Used for: training-free sharpened sampling result (abstract).
  15. paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: Table 1 (prompt templates, Qwen2.5-Math), the no-reward-signal remark (§2.1), template destroying capability before RL (§3.3).
  16. paperYao, X., Yu, L., Hu, X., Teng, F., Cui, Q., Zhou, J., Liu, Y. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028v1, 5 Oct 2025 (Renmin University of China, Ant Group, Xiamen University). https://arxiv.org/abs/2510.04028. Accessed 2026-10-27. Used for: Lemma 1, Theorem 1, Remark 3 (§3), toy example (§3.2), experiments (§4, Table 1, Figure 2), hyperparameters (App. C.2), footnote 3.
  17. paperMiller, E. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640v1, 1 Nov 2024 (Anthropic). https://arxiv.org/abs/2411.00640. Accessed 2026-10-27. Used for: standard error and confidence interval (Eq. 1-3), variance decomposition and resampling (§3.1), paired analysis (§4.2).
  18. paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. Why Language Models Hallucinate. arXiv:2509.04664v1, 4 Sep 2025 (OpenAI, Georgia Tech). https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-26. Used for: the incentive under binary grading that makes a best guess optimal over abstaining (§4).