Reinforcement Learning for LLM Post-Training · Module 20
Evaluating What RL Did: pass@1 versus pass@k, Spurious Rewards, Contamination, and the Sharpen-or-Expand Debate
A reported RL gain is a pass@1 number measured on one base model, one prompt template and one benchmark, and each of those three can supply the gain without any new capability. This module derives the estimators, puts the pass@k, random-reward, template and contamination evidence side by side, and ends with a measurement that separates a sharpened sampling distribution from a larger set of solvable problems.
1. What each metric measures
An RL report quotes one or more of four numbers for the same benchmark, and they answer different questions. All of them are statistics of one object: the probability \(\rho_x\) that a single sample from the model, at the evaluation decoding settings, is verified correct on problem \(x\). Temperature, top-p and maximum length are part of the definition of \(\rho_x\); change them and you are measuring a different distribution (Section 8.1).
1.1 pass@1 and its aliases
pass@1 is \(\mathbb{E}_x[\rho_x]\), the mean accuracy over problems. It is estimated by sampling k responses per problem and averaging their correctness. DeepSeek-R1 defines \(\text{pass@1}=\frac1k\sum_{i=1}^{k}p_i\) with k = 64 for AIME and GPQA, 16 for MATH and Codeforces, and 8 for LiveCodeBench, at temperature 0.6 and top-p 0.95. It avoids greedy decoding because greedy decoding of long-output reasoning models "results in higher repetition rates and significant variability across different checkpoints"6 (Supplementary D.1). DAPO reports the same estimator as avg@32: the AIME set is repeated 32 times at temperature 1.0 and top-p 0.77 (§4.1). "pass@1", "avg@k" and "mean@k" are one statistic under three names; what differs between reports is k, temperature and top-p, and those change the value.
1.2 pass@k and the unbiased estimator
pass@k is the fraction of problems for which at least one of k samples is correct: per problem \(1-(1-\rho_x)^k\), averaged over problems. The estimator in use comes from the Codex paper8. Draw \(n\ge k\) samples per problem, count the \(c\) that pass, and compute $$\widehat{\text{pass@}k}=\mathbb{E}_x\!\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right],$$ the complement of the probability that k samples drawn without replacement from the n are all wrong (Eq. 1). The tempting plug-in \(1-(1-c/n)^k\) is biased low, and the gap does not close even for \(n>5k\) (Appendix A, Figure 13). The numerically stable form is the running product \(1-\prod_{i=n-c+1}^{n}(1-k/i)\) (their Figure 3), which the calculator evaluates. Yue et al. set n to the largest k on each curve: 128, 256 or 1,024 depending on the benchmark and model1 (Appendix A.2).
pass@k for one problem: unbiased estimator versus plug-in
Formulas from Chen et al. (Eq. 1 and Figure 3)8; c and k are clamped to n. At the defaults, n = 64, c = 4, k = 32, the unbiased value is 94.3% and the plug-in 87.3%.
The error is largest for problems solved a few times in n samples, which decide the large-k end of a curve: n = 256, c = 2, k = 128 gives 75.1% unbiased and 63.4% plug-in (derived). At \(k=n\) the estimator collapses to the indicator "at least one of the n samples passed", so the rightmost point of a curve built with n equal to its largest k is a single Bernoulli draw per problem.
1.3 Maj@k and cons@k
Majority voting asks a different question: is the most frequent final answer among k samples the correct one? It needs an answer extractor and a canonicalization rule, and it measures how strongly the answer distribution concentrates on the right answer rather than whether the right answer is reachable. DeepSeekMath plots Maj@K and Pass@K side by side for K up to 649 (Figure 7); R1 calls it cons@646. For any model Maj@k cannot exceed pass@k, because a correct majority contains a correct sample, and pass@k never decreases with k. R1's own numbers on AIME 2024 sit in that order.
Supplementary E.4 of the v2 report. AIME 2024 has 30 problems5, so 86.7% and 90.0% are 26 and 27 problems (derived); 79.8% is a mean over samples and need not be a multiple of 3.33 points.
Each metric matches a deployment: pass@1 is a single-shot product, Maj@k is sampling without a verifier, pass@k is sampling with a checker such as a test suite. Brown et al. show coverage (their name for pass@k) growing log-linearly over four orders of magnitude of samples, and on SWE-bench Lite DeepSeek-Coder-V2-Instruct goes from 15.9% with one sample to 56% with 25010 (abstract). If your stack can afford k samples and a checker, pass@k at that k predicts product quality, so a run that raises pass@1 while lowering pass@256 moves the best operating point.
1.4 Worked derivation: pass@k as a functional of the pass-rate distribution
Because \(\text{pass@}k=\mathbb{E}_x[1-(1-\rho_x)^k]\), the whole curve is fixed by the distribution of \(\rho_x\) across problems, and two models with the same pass@1 can have very different curves. Small k weights the mean. Large k weights the lower tail: \(1-(1-\rho)^k\) saturates at 1 once \(\rho k\gg 1\) and stays near \(\rho k\) while \(\rho k\ll 1\). One identity makes the effect of concentration explicit. Since \(\mathbb{E}[(1-\rho)^2]=(1-\mu)^2+\sigma^2\) with \(\mu=\mathbb{E}\rho_x\) and \(\sigma^2=\operatorname{Var}(\rho_x)\), $$\text{pass@}2 = 1-\big[(1-\mu)^2+\sigma^2\big],$$ and for \(k>2\) Jensen's inequality makes the same expression an upper bound, \(\text{pass@}k\le 1-[(1-\mu)^2+\sigma^2]^{k/2}\). Dang et al. derived this bound and ProRL quotes it11 (§4.4). At fixed mean accuracy, more variance across problems, meaning more problems pinned at 0 and at 1, gives lower pass@k.
Table 1 builds a constructed population to show the effect. It is arithmetic, not data. The "after RL" pass rates imitate the pattern Yue et al. report: RL raises the share of problems near accuracy 1.0 and also raises the share stuck at 01 (Figure 5, §4.1).
| Class | Problems | \(\rho\) base | \(\rho\) after RL | pass@256 base (%) | pass@256 after RL (%) |
|---|---|---|---|---|---|
| Easy | 40 | 0.60 | 0.97 | 100.0 | 100.0 |
| Medium | 30 | 0.15 | 0.35 | 100.0 | 100.0 |
| Hard | 20 | 0.01 | 0.001 | 92.4 | 22.6 |
| Near zero | 10 | 0.0001 | 0 | 2.5 | 0.0 |
| Whole population | 100 | pass@1 28.7% | pass@1 49.3% | 88.7 | 74.5 |
The model that is 20.6 points better at k = 1 is 14.2 points worse at k = 256, and the curves cross at k = 16 (70.8% against 70.3%; at k = 15 they are tied, 70.19% against 70.25%). The variances are 0.068 before and 0.170 after, so pass@2 is 42.3% and 57.3%. Had the RL model reached its mean of 0.493 with the base model's variance, pass@2 would be 67.5%; polarization costs it 10.1 points at k = 2, and at k = 256 the loss comes from the hard and near-zero classes.
2. The sharpening evidence
2.1 DeepSeekMath saw it first
The sharpen-or-expand question was visible in the paper that introduced GRPO. DeepSeekMath compared the instruction-tuned 7B model with its GRPO-trained successor on GSM8K and MATH at temperature 0.7, for K up to 64, and found that "RL enhances Maj@K's performance but not Pass@K". The authors read this as RL "rendering the output distribution more robust": the improvement "is attributed to boosting the correct response from TopK rather than the enhancement of fundamental capabilities"9 (§5.2.2, Figure 7). They also name a possible cause that is a property of their pipeline rather than of RL: the RL data were questions from the instruction-tuning stage, sampled with plain nucleus sampling, which they suspect is "a potential reason that our RL pipeline only improves the Maj@K performance"9 (§5.2.3). That is one model and two benchmarks1 (§6). Module 9 covers the GRPO objective itself.
2.2 Yue et al.: pass@k at large k
Yue et al. test the claim at scale1. For math they take released zero-RL checkpoints (GRPO from a base model with a correctness reward only, SimpleRLZoo on GSM8K and MATH) for Qwen2.5-7B, 14B and 32B and LLaMA-3.1-8B, plus Oat-Zero-7B and DAPO-32B, and they add released code and vision models (§3). Base and RL models share the same zero-shot prompt, with no few-shot examples for the base, and sampling uses temperature 0.6, top-p 0.95 and up to 16,384 generated tokens (§3). The pattern is the same in every family: the RL model wins at small k, and "as k increases, with steeper curves, base models consistently catch up to and eventually surpass RL-trained models". On Minerva with a 32B model the base is about 9 points ahead at k = 128; Oat-Zero and DAPO start nearly 30 points ahead of their base and are eventually passed by it (§3.1). Code and visual benchmarks behave the same way (§3.2, §3.3).
Two details matter. Base models "often produce unformatted or non-sensical responses" without few-shot prompts, yet with enough samples they produce correctly formatted, correct answers (§3). Part of an early pass@1 gain is therefore the model learning the expected answer form, while pass@k at large k looks through format failures to what is reachable. And because math checking is answer-only, Yue et al. read chains of thought by hand: of 25 GSM8K problems the base solved at accuracy between 0 and 5%, 24 had at least one correct chain (23 of 25 for the RL model), and on low-accuracy AIME 2024 problems the base had a correct chain in 5 of the 6 cases they could judge (§3.1).
The controlled experiment in Figure 3 isolates training duration. The same trend shows up per problem. Yue et al. partition problems by whether the base model, the RL model, both or neither solved them at least once, with k = 1,024 for AIME 2024 and k = 128 for MATH-500 (Table 2 of the paper, Figure 4 here). The RL model solved almost nothing the base could not: the RL-only slices are 0.0% of AIME 2024 and 1.0% of MATH-500 (about 5 problems), and the base solves all of those with 1,024 samples (Appendix C.7). Perplexity gives a second view. They score RL-model responses under the base model (median over 32 samples, averaged over 10 problems); the perplexity falls from 1.244 at the base through 1.219 and 1.176 to 1.159 at the end of training (Figure 15). The RL model increasingly writes what the base already considers likely, which is what "sharpening" means operationally.
2.3 The algorithm does not change the picture
Yue et al. re-implemented six RLVR algorithms in one framework (PPO, GRPO, Reinforce++, RLOO, ReMax and DAPO; KL removed, constant learning rate \(10^{-6}\), 8 responses per prompt) and compared pass@1 and pass@256 on the same Omni-MATH split1 (§4.3, Table 3). Their summary is that all six "perform similarly and remain far from optimal". Table 2 reproduces the numbers and adds one derived column: the sampling-efficiency gap \(\Delta_{SE}\) as the paper defines it, the base model's pass@256 minus the RL model's pass@1.
| Model | Train pass@1 | Train pass@256 | Test pass@1 | Test pass@256 | \(\Delta_{SE}\) on test |
|---|---|---|---|---|---|
| Qwen2.5-7B (base) | 9.9 | 67.2 | 10.2 | 69.1 | n/a |
| GRPO | 26.1 | 66.3 | 25.1 | 68.3 | 44.0 |
| PPO | 27.2 | 65.8 | 26.8 | 69.2 | 42.3 |
| ReMax | 24.4 | 65.5 | 23.8 | 67.5 | 45.3 |
| RLOO | 28.6 | 66.4 | 28.1 | 69.2 | 41.0 |
| Reinforce++ | 28.2 | 67.7 | 28.0 | 69.7 | 41.1 |
| DAPO | 31.4 | 66.1 | 26.5 | 67.0 | 42.6 |
The spread between algorithms in pass@1 is 7 points on the training split (24.4 to 31.4) and the spread in pass@256 is 2.2 points (65.5 to 67.7), against a base model at 67.2. The paper's text quotes \(\Delta_{SE}\) of 43.9 for GRPO and 42.6 for RLOO on the test split; recomputing from its Table 3 gives 44.0 for GRPO (rounding) and 41.0 for RLOO, while 42.6 is DAPO's, so use the table. Modules 11 and 15 argue that loss and normalization choices move efficiency more than the ceiling; this is the same observation from the pass@k side, on one base model and one benchmark. Other choices did not rescue the boundary: a KL term at 0.001 gave similar pass@1 and a much lower pass@128, 32 rollouts per prompt instead of 8 helped a little, and raising the RL model's temperature to match the base's entropy did not close the gap (§4.4-§4.5).
2.4 Independent checks
Hochlehnert et al. re-evaluated open RL checkpoints under one standardized stack and report "a minor diversity collapse": DeepScaleR-1.5B and FastCuRL-1.5B trained from DeepSeek-R1-Distill-1.5B were below their starting point in pass@k for k up to 128 on AIME24, AMC23 and MATH500, with every \(\Delta\)pass@k negative5 (§5.2, Figure 12). The same study found significant pass@1 gains for DeepScaleR and FastCuRL and robust gains for Nemotron-RR, which by its name is ProRL's released model5 (§4.3). Pass@1 gains and a shrinking boundary are compatible, and several of those recipes show both.
The Invisible Leash study (first author Fang Wu; v5 of 14 October 2026) tracks outcomes question by question instead of averaging12. It defines the empirical support of a model as the set of questions with at least one verified success in n samples, and records for each question whether RL preserved, expanded or shrank access to a verified solution. In the benchmark aggregates it evaluated, "more questions lose a verified success than gain one at the stated sampling budgets", alongside gains in pass@1; on AIME25 and Minerva coverage rose at intermediate checkpoints and ended below the initial level, so expansion can be temporary; and some RLVR models raised average token-level entropy while lowering final-answer entropy12 (abstract, §1). Qwen3's on-policy distillation experiment, which found pass@64 flat under RL, is in Section 713.
A training-free result points the same way. Karan and Du sample from a sharpened version of the base model's own likelihood with an MCMC-inspired procedure, with no training, data or verifier, and report boosts that "nearly match and even outperform" RL on single-shot MATH500, HumanEval and GPQA without RL's diversity collapse14 (abstract). If reweighting base likelihoods reproduces most of the single-shot gain, that gain is a property of the base distribution; it does not show RL does nothing else.
3. The expansion claim: ProRL
3.1 The claim and the recipe
ProRL's authors argue that conclusions like Yue et al.'s "may stem from methodological constraints rather than fundamental limitations of RL": earlier studies leaned on math, where models are heavily trained in both pre-training and post-training, and stopped RL after "no more than hundreds of steps"11 (§1). Their recipe is GRPO with DAPO's decoupled clipping (\(\epsilon_{low}=0.2\), \(\epsilon_{high}=0.4\)) and dynamic sampling that drops prompts with accuracy 0 or 1, plus a KL penalty to a reference policy that is hard-reset, together with the optimizer state, to a recent snapshot of the policy when validation stagnates (§2.3, §3.2-§3.3). The run starts from DeepSeek-R1-Distill-Qwen-1.5B and trains on 136K problems from five domains (math, code, STEM, logic puzzles, instruction following) with 16 rollouts per prompt at temperature 1.2, a 256-prompt batch, and about 16,000 H100 GPU-hours, for more than 2,000 steps (§3.1-§3.3).
3.2 What it measured
Single-sample gains are large and not in dispute (Figure 5): the average over math benchmarks goes from 44.45 to 60.14, code from 23.08 to 37.49, GPQA Diamond from 15.86 to 41.78, IFEval from 44.05 to 66.02, and the Reasoning Gym logic-puzzle reward from 4.24 to 59.06 (Tables 1-3; pass@1 from 16 samples for math, code and GPQA, mean verifier reward for IFEval and the puzzles; temperature 0.6, top-p 0.95, 32K tokens). One reporting inconsistency: the introduction gives average gains of 14.7 (math), 13.9 (code), 25.1 (STEM) and 18.1 (instruction following); the body gives 15.7, 14.4, 25.9 and 22.0, and the tables agree with the body (60.14 minus 44.45 is 15.69; 37.49 minus 23.08 is 14.41; 41.78 minus 15.86 is 25.92; 66.02 minus 44.05 is 21.97, derived). All are absolute points. Module 13 quotes the introduction's figures.
The boundary evidence comes from pass@k. ProRL re-evaluates with 256 samples on 18 of the 96 Reasoning Gym tasks plus the other benchmarks and finds a strong negative relationship between the base model's pass@128 on a task and the gain in pass@128 after RL: tasks the base already solves show "minimal or even negative gains in reasoning breadth", and tasks where it struggles gain most (§4.1, Figure 3). Per task the curves fall into three regimes (§4.2, Figure 4): diminish (math, LiveCodeBench, letter counting, where pass@128 often declines while pass@1 rises), plateau (GPQA Diamond, maze, palindrome, where the final checkpoint adds nothing), and sustained gains (Codecontests, TACO, dice). On the held-out boxnet task the base model "exhibits no capability of solving the task" and the final model does (§4.3). ProRL's own math results therefore agree with Yue et al.; a claim of expansion rests on code and logic tasks where the base starts weak, and on the Creativity Index, which measures how much a response overlaps a pretraining corpus (Dolma in the per-task analysis) and rises with training (§1, §4.1, Figure 1).
3.3 Why expansion needs a signal to start from
RL can only reinforce what it samples. Dr. GRPO: if a base policy "cannot even sample a single trajectory that leads to the correct final answer, it is impossible for RL to improve the policy because there is no reward signal"15 (§2.1). The ProRL budget lets you put numbers on it (derived here). A batch of 256 prompts for about 2,000 steps is 512,000 prompt slots against 136,000 problems, about 3.8 visits per problem on average before dynamic sampling removes prompts. With 16 rollouts per visit, a problem whose pass rate under the training policy is \(\rho\) yields at least one correct rollout with probability \(1-(1-\rho)^{16}\):
| Pass rate \(\rho\) | P(at least one correct in 16) | Expected visits to first correct |
|---|---|---|
| 0.001 | 1.6% | 63.0 |
| 0.01 | 14.9% | 6.7 |
| 0.05 | 56.0% | 1.8 |
| 0.2 | 97.2% | 1.0 |
A problem at \(\rho=0.001\) needs about 63 visits to supply its first reward, roughly 17 times the average number of visits it gets. Direct reinforcement therefore reaches problems the starting policy already solves about 1% of the time or more; a problem near zero can only improve through transfer from other problems. That fits ProRL's finding that gains track base-model weakness without being zero, and Yue et al.'s tiny RL-only slice. It does not decide whether transfer can lift near-zero problems.
3.4 Not a matched comparison
Both studies report real measurements; they differ in at least six ways, so neither isolates a cause.
| Attribute | Yue et al. | ProRL |
|---|---|---|
| Starting checkpoint | Base Qwen2.5 (7B, 14B, 32B), LLaMA-3.1-8B for math; instruct or distilled models for code and vision | DeepSeek-R1-Distill-Qwen-1.5B, already fine-tuned on R1 reasoning traces |
| Training length | Released checkpoints; controlled run to 450 steps | More than 2,000 steps, about 16,000 GPU-hours |
| Training tasks | One domain per run (GSM8K and MATH, Omni-MATH, code, geometry) | One run on 136K problems in five domains |
| KL term | Removed; a 0.001 KL ablation lowered pass@128 | Kept, with periodic reference and optimizer resets |
| Rollouts and temperature | 8 per prompt, temperature 1.0, 8,192 tokens (controlled run) | 16 per prompt, temperature 1.2, 8k tokens (16k at the end) |
| Evaluation | Temperature 0.6, top-p 0.95, 16,384 tokens; k to 256 or 1,024 | Temperature 0.6, top-p 0.95, 32K tokens; pass@k with 256 samples on a subset of tasks |
One difference is conceptual. ProRL's baseline is a distilled model that has already absorbed teacher traces, which Yue et al. find can expand the boundary1 (§4.2). Expansion relative to a distilled start does not contradict a bound set by the pre-distillation base, so name the base in every claim. The camps also overlap: Ximing Lu, Mingjie Liu, Yi Dong and Yejin Choi are authors of both ProRL and the Invisible Leash study1112, whose v5 finds shrinkage outweighing expansion in aggregate.
3.5 The evidence on one page
| Study | Start and recipe | Measure | Result at large k |
|---|---|---|---|
| DeepSeekMath9 | DeepSeekMath-Instruct 7B, GRPO on instruction-tuning questions | Maj@K, Pass@K, K up to 64 | Maj@K up, Pass@K not |
| Yue et al.1 | Base models of several families, six algorithms | pass@k to 256 or 1,024 | Base ahead; RL-only slice 1.0% on MATH-500 |
| Hochlehnert et al.5 | R1-Distill-1.5B to DeepScaleR, FastCuRL | pass@k to 128 | Every \(\Delta\)pass@k negative |
| Invisible Leash12 | Several models, benchmarks and checkpoints | Per-question support | More questions lost than gained |
| Qwen313 | Qwen3-8B after off-policy distillation, then RL | pass@64, AIME 2024 and 2025 | Unchanged: 90.0 and 83.3 |
| ProRL11 | R1-Distill-1.5B, more than 2,000 steps, KL resets | pass@k to 128 or 256, per task | Diminish on math, sustained on code and logic |
| Olmo 3 RL-Zero4 | Fully open 7B base, decontaminated data | pass@32, AIME 2024 and 2025 | "Decent improvement" (figure only; no numbers in the text) |
4. The two-phase reconciliation
Yao et al. (Renmin University of China, Ant Group and Xiamen University; arXiv v1, 5 October 2025) argue that both camps are partly right because policy-gradient updates move probability mass through two stages16. It is a theory with a toy demonstration and a short empirical test by the authors; treat it as an interpretation, not a finding.
4.1 The update rule and what it implies
For a softmax policy over logits \(z\), if action \(v\) is sampled with advantage \(\hat A(v)\), the logits move by \(\Delta z_v=\eta\hat A(v)(1-\pi(v))\) for the sampled action and \(\Delta z_u=-\eta\hat A(v)\pi(u)\) for every other action (Lemma 1). Averaging over a group of samples, Theorem 1 gives the expected update $$\mathbb{E}[\Delta z_v]=\eta\,\pi(v)\Big[(1-\pi(v))\hat A(v)-\sum_{u\ne v}\pi(u)\hat A(u)\Big].$$ If the advantage is centered on the exact mean reward, then \(\sum_u\pi(u)\hat A(u)=0\) and the bracket collapses to \(\hat A(v)\), so \(\mathbb{E}[\Delta z_v]=\eta\,\pi(v)\hat A(v)\) (derived here from the theorem; it is the standard softmax policy gradient in logit space). Two consequences carry the argument. A logit moves in proportion to its own action's probability, so a rare action is slow to change whatever its advantage. And while the common action has positive advantage, its growth pushes the rare action down, because mass is conserved.
The authors read this as two stages (§3.1, Remark 3). In the exploitation stage the policy samples the explored high-reward action and the low-reward one, mass flows to the former, and the rare but better action's probability "remains largely unchanged" or even falls, which they connect to boundary shrinkage. In the exploration stage the common action saturates (\(1-\pi\to0\)), the rare action, when sampled, has positive advantage while the formerly good action becomes the relative negative, and the rare action's probability rises, which they connect to expansion. In pass@k terms they say raising the probability of at least one correct action above \(1/k\) expands the boundary.
4.2 The toy, computed
The paper's toy has three actions: \(a_1\) explored with reward 0.85, \(a_2\) rare with reward 1.0, and \(a_3\) with reward 0 (§3.2). Its own runs sample groups of two with \(\eta=0.1\) over thousands of steps. Figure 6 is the noiseless version computed here from the expected update above, with initial probabilities \(\pi=(0.55,0.05,0.40)\) that I chose.
The toy exposes a scaling the paper does not state. When \(a_1\) has saturated, \(\bar r\to r_1\), so \(a_2\)'s logit grows at about \(\eta\,\pi_2\,\Delta r\) with \(\Delta r=r_2-r_1=0.15\). The escape time is then of order \(1/(\eta\,\Delta r\,\pi_{2,\min})\), which is \(1/(0.1\times0.15\times0.0166)\approx4{,}000\) steps against 4,799 measured to reach 0.5 (derived; an order-of-magnitude fit, not an exact law). The exploration stage lasts as long as the inverse of the probability that the exploitation stage left behind: more over-exploitation means a longer wait. In the same toy with \(\pi_0=(0.6,0.02,0.38)\) the minimum is 0.0023 and the recovery takes about 33,500 steps to cross 0.5.
A caveat, also derived: the exploration stage exists only because \(r_2>r_1\). With two equally rewarded routes, \(r_1=r_2=1\), the same update gives \(\mathrm{d}(z_2-z_1)/\mathrm{d}t=\eta(\pi_2-\pi_1)\pi_3\), which is negative while \(\pi_1>\pi_2\); in a run from \(\pi_0=(0.6,0.01,0.39)\) the rare route falls from 0.01 to 0.0002 by step 50,000 and does not return. The paper says its binary setting gets differences among positive rewards from the importance ratio (footnote 3). For a binary verifier the stage structure therefore says little about problems where both routes are correct; it speaks to cases where the signal ranks correct routes, such as graded rewards.
4.3 What the paper tested
The empirical part trains Qwen2.5-Math-7B and Llama-3.2-3B-Instruct on the 7,500 MATH training problems with GRPO, GSPO and variants that keep only the relative negative gradients (GRPO-N, GSPO-N), with a uniform chat prompt, KL 0.001, an entropy bonus of \(10^{-4}\), and 256 samples per problem at temperature 0.6 for evaluation (§4, Appendix C.2)16. The training curves in its Figure 2 run to step 42. Plain GRPO lets test entropy fall, while GRPO-N, GSPO and GSPO-N dip and then rise. On Qwen2.5-Math-7B (Table 1), every RL variant lifts AIME 2024 pass@256 from the base 66.7 to 73.3, which is 2 of 30 problems, and GRPO-N lifts AIME 2025 pass@256 from 46.7 to 66.7, which is 6 of 30. That is a short run on the model family where prompts and priors matter most (Section 5), against a base under a chat template, and Dr. GRPO's Table 1 shows chat-style templates are not the best prompt for this model: its Qwen-Math template gives an average of 26.5 against 38.2 with no template15. Whether the AIME gains are expansion or recovery from a poorly matched prompt cannot be separated from the text, and the prolonged-training half of the theory is untested at ProRL scale.
5. Spurious rewards
Module 19 (§2) gives the taxonomy of documented reward exploits. Spurious rewards are a different failure, and a measurement control, so they are covered in full here. The question is how much of an RL gain survives when the reward carries little or no information about correctness.
5.1 The reward ladder
Shao et al. (University of Washington, Allen Institute for AI and Berkeley; arXiv v2, 25 February 2026) run GRPO on Qwen2.5-Math models with the DeepScaleR training data and replace the ground-truth reward with weaker ones, holding everything else fixed2 (§2.1-§2.2). The rewards are binary (Module 10). The ladder, from most to least informative: ground truth; majority vote, where the label is the majority answer of 64 samples from the pre-RL model; format, which rewards any response containing a non-empty \boxed{} whatever it contains; incorrect label, which rewards matching a majority-voted label that is known to be wrong; and random, which gives reward 1 with probability \(\gamma=0.5\) independent of the response (\(\gamma=0\) yields zero gradient, as expected). They report MATH-500 pass@1 and AMC average@8 after 300 steps, and say the weak and spurious rewards are "proposed purely for analytical purposes".
5.2 What it did
On Qwen2.5-Math-7B, random rewards improved MATH-500 by 21.4 points against 29.1 for ground truth, and incorrect labels by 24.12 (Figure 1). On AIME 2024 the same model gains 15.3 with ground truth and 10.2 with both incorrect and random rewards; on AIME 2025, whose questions postdate every model's knowledge cutoff, ground truth has "a clear advantage" and the other rewards give between -0.4 and +4.5 (§2.2, Appendix D). The effect is model-dependent (§3): all non-random rewards, even incorrect ones, help the Qwen2.5 models, OLMo2-7B gains only with ground truth, and each weak or spurious reward fails on at least one non-Qwen model. Smaller models benefit less, and models already RL post-trained, such as Qwen2.5-Math-7B-Instruct, gain little under any reward including ground truth (Appendix J).
| Model | Before RL | Ground truth | Majority vote | Format | Incorrect label | Random | GT minus random |
|---|---|---|---|---|---|---|---|
| Qwen2.5-Math-7B | 49.4 | +29.1 | +27.1 | +13.8 | +24.1 | +21.4 | 7.7 |
| Qwen2.5-7B | 41.6 | +34.4 | +31.9 | +16.7 | +27.8 | +16.4 | 18.0 |
| Llama3.1-8B-Instruct | 36.8 | +7.4 | +7.2 | -1.3 | -6.3 | -11.5 | 18.9 |
| OLMo2-7B | 9.0 | +15.5 | -6.4 | +0.4 | -6.4 | -8.3 | 23.8 |
The last column estimates the gain that an uninformative reward does not reproduce: 7.7 points for the model where the prior does most of the work, 18 to 24 for the others. It bounds what the verifier's information explains only if gains add, which the study does not test.
5.3 Mechanism one: clipping bias
Random rewards have zero mean advantage, so the expected policy gradient should vanish. Shao et al. show it does not, because of the clip2 (§4, Appendix B). They ablate three ways of removing the clipping effect (disabling it, making the minibatch the full rollout, or using one gradient update per rollout so the ratio stays 1) and find that random rewards then give no consistent gain, while with clipping on they give stable gains (Figure 4). Their Eq. 2 says that, for random advantages, the expected gradient is \(+\nabla\log\pi\) when the ratio \(R<1-\epsilon\), zero inside the band, and \(-\nabla\log\pi\) when \(R>1+\epsilon\). Read directly: above the band only negative-advantage samples keep their gradient, so the net push is down; below it only positive-advantage samples do, so the net push is up. The clip is a restoring force toward \(R=1\).
The force is asymmetric because probabilities cannot exceed 1. The upper clip triggers when \(\pi_\theta>(1+\epsilon)\pi_{old}\), which is unreachable once \((1+\epsilon)\pi_{old}\ge1\), that is for \(\pi_{old}\ge1/(1+\epsilon)=0.833\) at \(\epsilon=0.2\) (derived). A token with \(\pi_{old}=0.85\) has its upper threshold at 1.02 and is never pushed back down, while a token with \(\pi_{old}=0.02\) is clipped once \(\pi_\theta\) passes 0.024 and is pushed back (the paper's own example). Under noise, probability mass drifts toward tokens that were already likely: "clipping bias asymmetrically suppresses low-probability tokens and reinforces high-probability ones". DAPO's \(\epsilon_{high}=0.28\) moves the threshold to 0.781 and delays the push-back on rare tokens (Module 13).
5.4 Mechanism two: an existing behavior gets amplified
Clipping bias says why random rewards sharpen; the next question is what they sharpen toward. For Qwen2.5-Math-7B the answer is writing Python without running it. The model does this in 65.0% of MATH-500 solutions, with no interpreter attached, and answers that do are far more accurate: 60.9% against 35.0% for the rest (Table 1 of the paper; its introduction says 28.0% for the rest, and I use the table)2. Under weak or spurious rewards code frequency rises to about 90% within 15 steps, and to 95.6% under random rewards, tracking the accuracy gain; under ground-truth rewards it rises and later declines as natural-language reasoning improves (§5.2, Figure 6). The behavior is a lever, not just a correlate: forcing the first sentence "Let's solve this using Python." lifts MATH-500 by 15.0 points on Qwen2.5-Math-7B and by 24.2 on the 1.5B model, and rewarding the string "python" alone pushes code frequency above 99% within 20 steps (§5.3, Figure 7). The same intervention hurts the models that lack effective code reasoning.
| Model | Original | Code-forced | Difference |
|---|---|---|---|
| Qwen2.5-Math-1.5B | 36.2 | 60.4 | +24.2 |
| Qwen2.5-Math-7B | 49.4 | 64.4 | +15.0 |
| Qwen2.5-1.5B | 3.0 | 13.0 | +10.0 |
| Qwen2.5-7B | 41.6 | 22.2 | -19.4 |
| Llama3.2-3B-Instruct | 36.8 | 8.2 | -28.6 |
| Llama3.1-8B-Instruct | 36.8 | 15.2 | -21.6 |
| OLMo2-7B | 9.0 | 7.8 | -1.2 |
| OLMo2-7B-SFT | 21.4 | 18.6 | -2.8 |
5.5 The baseline moves as much as the RL gain
The "before RL" number is not fixed either. Shao et al. evaluate Qwen2.5-Math-7B under five prompts: the Qwen default gives 49.4% on MATH-500 with 78.9% of answers parsable, a "MathProblem:" prompt 55.8%, SimpleRL-Zoo's prompt 63.2%, the Sober Reasoning prompt 61.6% (93.1% parsable), and a block of LaTeX lorem-ipsum text with no task content 68.8% (84.1% parsable)2 (Table 5, Appendix I). The best accuracy does not come from the best parse rate or from task-relevant words. Hochlehnert et al. could not fully reproduce the magnitude but found the same effect on Qwen2.5 models5 (§5.3). A 19.4-point swing (49.4 to 68.8, derived) from a prompt with no content is the size of the 21.4-point gain from random-reward RL. Models trained under three of the prompts converged to similar performance after RL, which the authors take to suggest that what a prompt elicits is a subset of what RL elicits (Appendix I).
5.6 What this does and does not establish
It establishes that for Qwen2.5-Math, a large part of the MATH-500, AMC and AIME 2024 gain needs no information about correctness: any signal that sharpens the policy toward an existing behavior produces it, and "reasoning patterns instilled during pretraining heavily impact the behavior of downstream RLVR training"2. It does not establish that ground-truth RL gains are spurious: ground truth beats random by 7.7 points on the most favorable model, by 18 to 24 on the others, and clearly on post-cutoff AIME 2025. The authors' instruction is to test reward signals on more than one model family and to include format and random rewards as dummy baselines2 (§1, §2.2). Section 6 covers the competing explanation.
6. Contamination and the baseline
Wu et al. (Fudan University and collaborators; arXiv v3, 17 December 2025) explain the Qwen results of Section 5 differently: the benchmark problems leaked into pretraining, and a spurious reward lets GRPO retrieve memorized answers3. Their audit gives the model the first 40, 60 or 80% of a problem and scores the continuation (greedy decoding, no chat template). Given the first 60% of each MATH-500 problem, Qwen2.5-Math-7B regenerates the remaining 40% with 54.60% exact match and answers 53.6% of the truncated problems correctly; Llama3.1-8B manages 3.8% and 2.4%. On LiveMathBench (version 202505), which postdates Qwen2.5, Qwen's completion rate falls to 0.0% and its truncated-problem accuracy to 2.0% (Llama: 0.0% and 1.0%) (Table 2, §1).
They then build a clean benchmark, RandomCalculation: automatically generated arithmetic expressions of 1 to 20 steps, 1,000 problems per sub-dataset, created after Qwen's release, with a continuous reward because exact answers are long decimals. On the 5-step and 10-step sets, correct rewards improve Qwen2.5-Math-7B steadily and beyond the initial model's Max@16, random and incorrect rewards give unstable or no gains, and inverted rewards collapse performance (§4.3). Output similarity before and after RL points the same way: ROUGE-L between pre- and post-RL responses under random reward is 0.601 on MATH-500 but 0.247 and 0.251 on the 5- and 10-step RandomCalculation sets (Table 3), which they read as retrieval of stored answers.
The two accounts agree that no new capability is created and differ on what is amplified, a behavior (Shao et al.: code reasoning) or a memorized item set (Wu et al.). They are not exclusive, and neither paper contains the other's decisive test: Shao et al. do not report their ladder on a decontaminated Qwen-scale set, and Wu et al. do not rule out a behavioral prior. The fully open base of Olmo 3 gives a partial test. Its RL-Zero setup decontaminates the RL data against pretraining and midtraining data and then runs a negative control, random binary rewards on the same data, which "does not improve performance on any of our benchmark evaluations"; performance stays flat or degrades4 (§6.2, Figure 27). Olmo 3 reads this as evidence that its decontamination removed overlap between the base and the evaluation data. It shows this base gives random rewards nothing to amplify; it does not choose between the two accounts of the Qwen result (my reading).
The prompt is the other half of the baseline. Module 10 §6.2 tabulates Dr. GRPO's template results: the R1 template takes Qwen2.5-Math-7B to 0.0 average, no template gives 38.215. Olmo 3 reports the same pattern from the other side: simple templates beat post-training templates on a base mid-trained without special formatting4 (§6.1). The evaluation rule that follows is to score the base under its best prompt as well as under the RL prompt, and to report both.
7. Distillation as a second boundary test
Yue et al. find that distilling R1 into Qwen2.5-Math-7B puts the pass@k curve consistently above the base model's at every k, unlike RL, which they say is bounded by the base1 (§4.2, Figure 7). Qwen3 supplies a controlled version, and Module 21 covers its cost and mechanism. Both arms start from the same off-policy-distilled Qwen3-8B checkpoint and use math and code queries; one continues with RL, the other with on-policy distillation from a stronger teacher13 (§4.7, Table 21).
The paper's reading is that distillation "enables the student model to expand its exploration space" while "reinforcement learning does not lead to any improvement in pass@64 scores". The arithmetic puts a scale on that. AIME has 30 problems5, so 90.0% and 93.3% are 27 and 28 problems, and 83.3% and 86.7% are 25 and 26 (derived). The "expansion" from distillation is exactly one extra problem on each benchmark, and the "no improvement" under RL is the same count before and after, which need not be the same problems. The passage reports no seeds, no intervals and no statement of how pass@64 was estimated. Read it as a direction that agrees with Yue et al., not as a magnitude; the 12 to 23 point pass@1 movements are the robust part. R1's final model itself shows the headroom this metric exposes, 79.8% pass@1 against 90.0% pass@64 on AIME 2024 (Section 1.3). For measurement, the useful question about any teacher-driven gain is whether the newly solved problems lie in the teacher's solved set, which a per-problem partition answers (Section 8.5).
8. A protocol that separates sharpening from expansion
Each gate of Figure 1 matches a documented failure; this section turns the gates into measurements.
8.1 Match the sampling protocol
Reports do not share one. R1, DAPO, Yue et al., ProRL and Olmo 3 differ in temperature, top-p, token cap and sample count (Table 8), so cross-report comparisons such as DAPO's 50 AIME points against 47 for R1-Zero-Qwen-32B7 compare different protocols as well as different recipes. Hochlehnert et al. measured the cost: across nine models and three benchmarks, temperature alone moves accuracy by up to 15%, top-p by up to 8%, and hardware and framework differences leave 2 to 5% even under stringent controls5 (Figures 4 and 5, §3.4).
| Report | Metric | Temperature | top-p | Max tokens | Samples |
|---|---|---|---|---|---|
| R1 v26 | pass@1 as mean of k; cons@64 | 0.6 | 0.95 | 32,768 | k = 64 AIME and GPQA, 16 MATH and Codeforces, 8 LiveCodeBench |
| DAPO7 | avg@32 on AIME | 1.0 | 0.7 | n/s | AIME set repeated 32 times |
| Yue et al.1 | pass@k | 0.6 | 0.95 | 16,384 | n up to 1,024 |
| ProRL11 | pass@1; pass@k | 0.6 | 0.95 | 32K | 16; 256 on a task subset |
| Olmo 3 RL-Zero4 | pass@1; pass@32 | 1.0 | n/s | 32K | n/s |
| Hochlehnert et al.5 | pass@1, sensitivity runs | 0.8 | 0.9 | 32,768 | 10 seeds (AIME, AMC), 3 (MATH-500) |
8.2 Know the resolution of the benchmark
AIME 2024 has 30 problems and AMC 2023 has 40, so one problem moves pass@1 by 3.33 and 2.5 points; Hochlehnert et al. report seed-to-seed standard deviations of 5 to 15 percentage points, worst on AIME 2024 and AMC 2023, and recommend at least 30 seeds, paired significance tests and per-model tuned decoding5 (§3.2, §4.1). Miller's statistics explain why more samples per problem do not fix this17. The variance of a benchmark mean is \(\operatorname{Var}(\hat\mu)=(\operatorname{Var}(x)+\mathbb{E}[\sigma_i^2])/n\): the variance of true per-problem pass rates, which resampling cannot shrink, plus the mean within-problem variance, which divides by the number of samples K (§3.1). His uniform-difficulty example has \(\operatorname{Var}(x)=1/12\) and \(\mathbb{E}[\sigma_i^2]=1/6\); with \(n=30\), going from one sample to 32 cuts the standard error from 9.13 to 5.43 points, and no number of samples gets below 5.27 (derived). Under that difficulty distribution, an avg@32 comparison of two unpaired models on AIME 2024 cannot resolve a 5-point gap. Scoring both models on the same problems and using question-level differences removes shared difficulty and is Miller's recommended analysis (§4.2).
| Benchmark | Problems | One problem (points) | Standard error (points) | 95% CI half-width (points) |
|---|---|---|---|---|
| AIME 2024 | 30 | 3.33 | 9.13 | 17.89 |
| AMC 2023 | 40 | 2.50 | 7.91 | 15.50 |
| LiveMathBench | 100 | 1.00 | 5.00 | 9.80 |
| MATH-500 | 500 | 0.20 | 2.24 | 4.38 |
8.3 Large-k pass@k counts lucky guesses
With answer-only checking, pass@k at large k also counts guesses. Under uniform guessing over \(N\) possible answers the chance of a hit within k samples is \(1-(1-1/N)^k\) (derived). Real guesses are not uniform, so Table 10 gives a scale, not a bound. Take AIME answers to be integers from 0 to 999, so N = 1,000: a guesser alone would reach 64% at k = 1,024, which is why Yue et al. checked chains of thought by hand1 (§2.2, §3.1). Binary grading making a best guess optimal over abstaining is the training-side version of the same incentive18 (Module 24).
| Answer space N | k = 8 | k = 64 | k = 256 | k = 1,024 |
|---|---|---|---|---|
| 4 | 90.0% | 100.0% | 100.0% | 100.0% |
| 100 | 7.7% | 47.4% | 92.4% | 100.0% |
| 1,000 | 0.8% | 6.2% | 22.6% | 64.1% |
8.4 What large k costs
Large k is a serving workload. Yue et al. could not run pass@k on R1-Zero: their self-hosted copy delivered about 50 tokens per second at a 32k maximum length1 (§4.6). AIME 2024 at n = 1,024 is 30,720 samples, at most 1.01 billion generated tokens at the 32,768-token cap, which is about 233 days at 50 tokens per second (derived upper bound; real lengths are shorter and a batched engine is far faster). Cost scales as problems times n times mean length, so a 1,024-sample partition of MATH-500 is 512,000 samples. Rollout-engine throughput (System Design Module 37) sets how far out the boundary can be measured.
8.5 The checklist, in the order of Figure 1
- Same protocol. Score the base and the RL model with the same prompt, temperature, top-p, token cap and k, and also score the base under its own best prompt (Sections 5.5 and 6). Pin report versions: R1-Zero's AIME 2024 pass@1 is 71.0% in v1 and 77.9% in v26 (Module 10).
- Estimator and n. Use the unbiased pass@k estimator with \(n\ge k\), report n, and remember that at \(k=n\) the point is a single indicator per problem (Section 1.2).
- Noise. Report seeds, standard errors with the benchmark's size in view, and paired differences against the base (Section 8.2).
- Control rewards. Run a random-reward and a format-reward run through the same pipeline; their gain is the part of the claim that needs no verifier (Section 5; Olmo 3's negative control).
- Clean benchmark. Prefer post-cutoff or decontaminated sets, and run the partial-prompt completion probe on the base model (Section 6).
- Boundary. Compute pass@k out to \(k\) in the hundreds or more, partition problems into both, base-only, RL-only and neither, and spot-check chains of thought on the problems that decide the result (Sections 2.2 and 8.3). Track the partition across checkpoints, since expansion can be temporary12.
Where experts disagree
The live question is whether RLVR creates solvable problems or raises the odds on problems the base could already solve. The sources split into three answers and a fourth position that questions what the benchmark gains measure.
RLVR sharpens; the base model sets the boundary
Yue et al.1; DeepSeekMath9; Qwen3's Table 2113; the Invisible Leash study12; Hochlehnert et al.5; Karan and Du14
pass@k at large k, per-problem solved sets and base-model perplexity of RL outputs all show the RL model solving a subset of what the base solves.
- Replicated with different pipelines and metrics, including per-question tracking and a training-free sampler.
- Operational: the claim is a measurement, not a story.
- Mostly short runs, strong math bases and answer-only checking.
- Distilled starts and weak-base tasks show different behavior.
RLVR can expand with long, diverse, KL-controlled training
ProRL11; Olmo 3's RL-Zero pass@324
Over 2,000 steps on 136K problems from five domains, gains were largest where the base started weak, and a held-out task the base could not solve at all was solved.
- Sustained pass@k gains on code and logic tasks; a clean 7B base also improved at pass@32.
- Names a concrete reason earlier studies may have stopped too early.
- Starts from a distilled 1.5B model, and its own math results diminish.
- Not a matched comparison with Yue et al. (Table 4); Olmo 3's pass@32 is shown in a figure only.
Both, in two phases
Yao et al.16
Probability mass first concentrates on explored high-reward routes (shrinkage), then, if training continues, flows to rare better ones (expansion).
- Explains both camps and predicts when each appears.
- The toy gives a duration scaling, about \(1/(\eta\,\Delta r\,\pi_{\min})\).
- In my derivation (Section 4.2) its exploration stage needs a reward gap between correct routes, which a binary verifier does not give.
- Tested for about 42 steps; not tested at ProRL scale.
Most measured gains are prior amplification or contamination
Shao et al.2; Wu et al.3; Dr. GRPO15; Olmo 34
Random and incorrect rewards reproduce most of a Qwen2.5-Math gain; a partial-prompt probe shows MATH-500 is memorized; the template alone moves the baseline by tens of points.
- Cheap controls any team can run, and they explain why Qwen results do not transfer.
- Consistent with the post-cutoff AIME 2025 result.
- Ground truth still beats random by 7.7 to 23.8 points.
- The two mechanisms (code behavior versus memory) are disputed, and this position says little about the boundary.
For strong math bases, single-domain RLVR and runs of hundreds of steps, the sharpening evidence is the best supported, and the Qwen2.5-Math numbers should be treated as unconfirmed until a random-reward control and a second model family agree. The pass@1 gains are real; what they do not show is new problems solved. For weak starts, diverse tasks and long runs the expansion case is plausible but rests on one 1.5B distilled model, and the two-phase theory is an interpretation of it, not evidence. If your deployment is single-shot, sharpening is the goal; if it samples k times against a checker, evaluate pass@k at your k. What would settle it is an experiment I found in none of these sources: an open decontaminated base such as Olmo 3, one task mix and one evaluation protocol, RL from step 0 past 2,000 steps, per-problem partitions at 1,024 samples on a clean held-out set at every checkpoint, chain-of-thought validity checks and a random-reward control, repeated on a second family.
What is changing
This page is as of 2026-10-27, and several sources it depends on are revised preprints: the Invisible Leash study reached v5 on 14 October 2026 with its per-question support framework12, Spurious Rewards is at v2 of February 20262, and Yue et al. is at v51. Check versions before quoting a number. The released open base, data and logs of Olmo 3 make the matched ladder of Figure 1 runnable outside a frontier lab4.
References
- paperYue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., Huang, G. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837v5, 24 Nov 2025 (Tsinghua LeapLab, SJTU). https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Used for: pass@k protocol (§2.2, §3), Figures 2, 5, 7, 15, Tables 2-4, controlled run (§4.3-§4.5), CoT validity checks (§3.1), estimator settings (App. A.2), solvable-set partition (App. C.7).
- paperShao, R., Li, S. S., Xin, R., Geng, S., et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947v2, 25 Feb 2026 (University of Washington, Allen Institute for AI, UC Berkeley). https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: reward ladder (§2), Figure 1 and Table 6 data, clipping bias (§4, Eq. 2, App. B), code reasoning (§5, Tables 1-2), prompt sensitivity (Table 5, App. I), post-RL models (App. J), AIME results (App. D).
- paperWu, M., Zhang, Z., Dong, Q., Xi, Z., et al. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. arXiv:2507.10532v3, 17 Dec 2025 (Fudan University et al.). https://arxiv.org/abs/2507.10532. Accessed 2026-10-27. Used for: partial-prompt completion audit (Table 2), RandomCalculation experiments (§4.3), response similarity (Table 3), benchmark sizes.
- paperTeam Olmo. Olmo 3. arXiv:2512.13961v2, 14 Apr 2026 (Allen Institute for AI). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: RL-Zero motivation (§2.2), template finding and evaluation settings (§6.1), pass@32 result and random-reward negative control (§6.2, Figure 27).
- paperHochlehnert, A., Bhatnagar, H., Udandarao, V., Albanie, S., Prabhu, A., Bethge, M. A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility. COLM 2025; arXiv:2504.07086v2, 6 Oct 2025. https://arxiv.org/abs/2504.07086. Accessed 2026-10-27. Used for: seed, temperature, top-p and hardware variance (§3, Figures 2, 4, 5, Takeaway 6), best practices (§4.1), RL replication results (§4.3), diversity collapse (§5.2), spurious prompt (§5.3), benchmark sizes.
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (4 Jan 2026; v1 22 Jan 2025 for the 71.0% figure). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: evaluation setup and pass@1 definition (Supplementary D.1), pass@1, cons@64 and pass@64 on AIME 2024 (Supplementary E.4), R1-Zero version numbers.
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 20 May 2025 (ByteDance Seed, Tsinghua AIR). https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: avg@32, temperature 1.0 and top-p 0.7 (§4.1), 50 versus 47 AIME points, Clip-Higher epsilon values.
- paperChen, M., Tworek, J., Jun, H., Yuan, Q., et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021 (OpenAI). https://arxiv.org/abs/2107.03374. Accessed 2026-10-27. Used for: the unbiased pass@k estimator (Eq. 1), the numerically stable implementation (Figure 3), and the bias of the plug-in estimator (Appendix A, Figure 13).
- paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: Maj@K versus Pass@K (§5.2.2, Figure 7) and the data-source caveat (§5.2.3).
- paperBrown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., Mirhoseini, A. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787v3, 30 Dec 2024. https://arxiv.org/abs/2407.21787. Accessed 2026-10-27. Used for: coverage scaling with samples and the SWE-bench Lite figures (abstract).
- paperLiu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., Dong, Y. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1, 30 May 2025 (NVIDIA). https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: claim and recipe (§1-§3), Tables 1-3, pass@k analysis (§4, Figures 3-5), Jensen bound on pass@k (§4.4), training and evaluation settings.
- paperWu, F., Xuan, W., Xing, D., Lu, X., Liu, M., Dong, Y., Harchaoui, Z., Choi, Y. The Invisible Leash? Why RLVR May or May Not Escape Its Origin. arXiv:2507.14843v5, 14 Oct 2026. https://arxiv.org/abs/2507.14843. Accessed 2026-10-27. Used for: empirical support dynamics, shrinkage versus expansion findings (abstract, §1), temporary expansion at intermediate checkpoints, author list.
- paperYang, A., Li, A., Yang, B., Zhang, B., et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: Table 21 and the RL versus on-policy distillation discussion (§4.7).
- paperKaran, A., Du, Y. Reasoning with Sampling: Your Base Model is Smarter Than You Think. arXiv:2510.14901v1, 16 Oct 2025 (Harvard). https://arxiv.org/abs/2510.14901. Accessed 2026-10-27. Used for: training-free sharpened sampling result (abstract).
- paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: Table 1 (prompt templates, Qwen2.5-Math), the no-reward-signal remark (§2.1), template destroying capability before RL (§3.3).
- paperYao, X., Yu, L., Hu, X., Teng, F., Cui, Q., Zhou, J., Liu, Y. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028v1, 5 Oct 2025 (Renmin University of China, Ant Group, Xiamen University). https://arxiv.org/abs/2510.04028. Accessed 2026-10-27. Used for: Lemma 1, Theorem 1, Remark 3 (§3), toy example (§3.2), experiments (§4, Table 1, Figure 2), hyperparameters (App. C.2), footnote 3.
- paperMiller, E. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640v1, 1 Nov 2024 (Anthropic). https://arxiv.org/abs/2411.00640. Accessed 2026-10-27. Used for: standard error and confidence interval (Eq. 1-3), variance decomposition and resampling (§3.1), paired analysis (§4.2).
- paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. Why Language Models Hallucinate. arXiv:2509.04664v1, 4 Sep 2025 (OpenAI, Georgia Tech). https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-26. Used for: the incentive under binary grading that makes a best guess optimal over abstaining (§4).