Reinforcement Learning for LLM Post-Training · Module 27
Case Study: DeepSeek-R1: R1-Zero, the Four-Stage Pipeline, Its Compute, and What the Nature Version Changed
DeepSeek-R1 is a GRPO case, not a TRPO one: GRPO entered through DeepSeekMath in February 2024, and TRPO is the 2015 ancestor of PPO. This module reads the report stage by stage (R1-Zero's rule-based RL, the four-stage R1 pipeline, its GPU-hours, what failed), flags every place where two versions or two sections of the report disagree, and sets later recipes against it. Afterwards you can reproduce R1's schedule arithmetic and say which of its choices later reports kept, changed or dropped.
1. Lineage: what R1 inherited and what it originated
The request behind this course says R1 introduced TRPO. It did not. R1's Section 2.1 names its algorithm and its source: GRPO "was originally proposed to simplify the training process and reduce the resource consumption of Proximal Policy Optimization (PPO)", credited to Shao et al., 20241. That paper is DeepSeekMath, which describes GRPO as a PPO variant that "foregoes the critic model, instead estimating the baseline from group scores"4. TRPO is Schulman et al.'s 2015 trust-region method2, the thing PPO simplified3. Module 3 derives both, Module 9 derives GRPO, and Module 1 counts R1's GPU-hours; this module reads what R1 did.
| Item | Introduced by | Date | R1's relation to it |
|---|---|---|---|
| KL-constrained trust-region update | Schulman et al., TRPO2 | ICML 2015 | Not used; the idea survives as PPO's clip |
| Clipped surrogate, several minibatch passes | Schulman et al., PPO3 | 2017 | R1's Eq. 1 keeps the min and the clip |
| Group-mean baseline, KL term in the loss (GRPO) | Shao et al., DeepSeekMath4 | Feb 2024 | Adopted and credited1 |
| Reasoning model trained with RL | OpenAI o1 | Sep 2024, second-hand110 | Closed; R1 publishes an open recipe |
| The name RLVR | Lambert et al., Tulu 38 | 22 Nov 2024 | R1-Zero is the large-scale case; Lambert calls R1 "famous for popularizing RLVR"10 |
| Pure RL on a base model for long CoT, open weights | DeepSeek-R1, its own claim9 | 22 Jan 2025 | R1-Zero |
The o1 row needs a caveat. OpenAI's announcement page returned HTTP 403 when this course's source map was built, and I did not work around it. The month rests on two second-hand statements: R1's Appendix F.1 says its Qwen2-Math-7B experiment ran before "the first reasoning model, OpenAI-o1 (September 2024)"1, and Lambert's bibliography lists "Introducing OpenAI o1-preview", September 202410.
R1 and Kimi k1.5 appeared on the same day: Lambert's Table 4 dates both 2025-01-2210, and the first version of k1.5 is also 22 January 202511. k1.5 claimed o1-matching numbers with a different objective and no GRPO11 (Module 28), so January 2025 was not a one-recipe result.
What R1 did contribute is narrower and easier to defend. It showed, on a 671B-parameter mixture-of-experts base with 37B parameters active per token and 14.8T pretraining tokens1 (App. A.1), that rule-based rewards and GRPO elicit long reasoning with no SFT stage first. It then published a multi-stage recipe that turns that behavior into a usable assistant, released weights, and, in v2, supplied enough hyperparameters to do the arithmetic. Its conclusion states the claim as a recipe: "the key to unlocking this potential lies not in large-scale human annotation but in the provision of hard reasoning questions, a reliable verifier, and sufficient computational resources for reinforcement learning"1 (§6). Smaller verifiable-domain RL existed earlier. Lambert's survey lists STaR (2022-2023) and VinePPO, and notes that Tulu 3 ran PPO against correctness rewards without losing general performance; what those methods lacked was R1's scale, or they "made sacrifices in overall performance"10 (§7.3.1).
2. R1-Zero: RL on a base model
2.1 What was trained, and where the detail lives
R1-Zero starts from DeepSeek-V3-Base and applies GRPO with no SFT stage1. The reward is a rule-based accuracy term plus a format term (Eq. 4), "combined with the same weight", and the report says it abstains "from applying neural reward models—whether outcome-based or process-based—to reasoning tasks" because they are "susceptible to reward hacking during large-scale reinforcement learning"1 (§2.2). Module 10 reads that reward, the template and the prompt data line by line, and Module 4 §4.5 covers why process reward models did not survive RL at scale; Section 6 below gives R1's own reasons. The optimizer settings are the ones Module 9's Table 4 sets beside DeepSeekMath and later recipes: learning rate 3e-6, KL coefficient 0.001, rollout temperature 1, 16 samples per question, and a maximum length of 32,768 tokens until step 8.2k and 65,536 after it1 (§2.1).
One comparison in the report is about the algorithm itself. Supplementary A.3 trains DeepSeek-Coder-V2-Lite (16B mixture-of-experts, 2.4B active) on MATH with PPO at GAE \(\lambda = 0.95\), PPO at \(\lambda = 1.0\), and GRPO. At the open-source default of 0.95, PPO "performs considerably worse than GRPO"; at 1.0 it is "nearing that of GRPO"1 (A.3, Figure 4). At the last plotted step the three curves sit at about 50.1, 53.5 and 55.4 percent MATH accuracy (read off the plot, accurate to about ±0.3; the same reading as Module 9's Figure 6). The report's argument is about tuning cost and memory, not that a tuned critic loses: "PPO can achieve comparable performance when appropriately tuned", but it demands extra hyperparameter search and a second policy-sized model1. Module 3 sets this against the RLOO and VAPO evidence.
One more thing to carry from the equations. R1 prints its objective (Eq. 1, repeated as Eq. 11) with the ratio \(\pi_\theta(o_i\mid q)/\pi_{\theta_{old}}(o_i\mid q)\) at sequence level and no \(1/|o_i|\) factor, whereas DeepSeekMath's Eq. 3 is per-token with \(1/|o_i|\)4. The report does not say which form ran. Treat it as a source-level conflict (Section 7.3); Module 12 §2.4 works through both readings.
2.2 The schedule, as arithmetic
Read the run-size sentences of §2.1 as a ledger. A training step is "a single policy update operation" (caption of the report's Figure 1)1. Each step uses 32 unique questions with 16 samples each, a batch of 512 outputs. Each rollout generates 8,192 outputs, "randomly split into 16 mini-batches and trained for only a single inner epoch". So 8,192 is 512 questions times 16 samples, one rollout feeds 16 steps, and 10,400 steps are 650 rollouts and 5,324,800 sampled outputs (derived). The reference model is replaced by the latest policy "every 400 steps": 26 multiples of 400 up to 10,400, or every 25 rollouts. The run draws 332,800 questions (10,400 times 32); at the stated 1.6 epochs that implies about 208,000 distinct prompts (derived). Module 10 §2.5 checks that against the 80K reasoning prompts in R1's Table 4, which describe R1; the report does not give R1-Zero's prompt count.
R1-Zero rollout schedule
Defaults are R1-Zero's1. With \(Q\) questions, \(G\) samples, \(K\) minibatches and \(T\) steps: outputs per rollout \(QGK\), rollouts \(T/K\), distinct prompts \(TQ/E\), mean staleness \((K-1)/2\).
Two consequences follow. The schedule is off-policy by construction: minibatch \(j\) is trained after \(j-1\) updates on outputs sampled before the first of them, so staleness runs from 0 to 15 and one minibatch in sixteen is on-policy (derived). That is where the clip begins to act; DeepSeekMath took a single update per rollout, where the clip is inert (Module 12 §1.1). Second, the report does not say whether the random split keeps a question's 16 outputs together. The advantage comes from a question's 16 rewards and is fixed when the group is scored, so the split changes which outputs share an optimizer step, not their advantages (my reading of the equations).
2.3 The rollout side
Supplementary B.1 describes four modules: rollout, inference, rule-based reward and training. Rollout runs vLLM workers holding the actor, with expert parallelism across nodes and "redundant copies of hotspot experts" to balance expert load. The multi-token-prediction head is used "for self-speculative decoding, significantly accelerating the decoding speed and effectively minimizing the completion time for the longest samples"1. The rule-based reward module (code executors, answer matchers, format checkers) needs no GPU memory but "tends to be time-consuming", so it runs asynchronously and overlaps rollout and inference. Training packs sequences into fixed-length chunks with a best-fit strategy, reuses DualPipe, and each module's models are offloaded from VRAM to host memory or disk when its phase ends.
For an inference engineer the omissions matter more than the list. The report gives no tokens per second, no split of GPUs between rollout and training, and no per-phase seconds, so the 68.5 s average per optimizer step on 512 GPUs that Module 1 derives from the 198 hours (derived) is the only timing the text supports. System Design Modules 36 (post-training pipelines), 37 (RL training infrastructure) and 38 (the open-source RL frameworks) cover that side.
2.4 The result, in two versions
Version 2 reports AIME 2024 pass@1 rising "from an initial 15.6% to 77.9%", and 86.7% with self-consistency decoding1 (§2.3). Version 1 reports 15.6% to 71.0%, and 86.7% with majority voting9 (§2.2.4). It gives no step count, only "thousands of RL steps"; its Figure 2, read by eye, ends near step 8,400, while version 2's curve runs to 10,400.
| Metric | v1 (22 Jan 2025) | v2 (4 Jan 2026) | Difference |
|---|---|---|---|
| AIME 2024 pass@1 (%) | 71.0 | 77.9 | +6.9 |
| AIME 2024 consensus (%) | 86.7 (cons@64 in Table 2, cons@16 in Fig. 2) | 86.7 (cons@64 in D.1, cons@16 in Fig. 1) | 0.0 |
| MATH-500 pass@1 (%) | 95.9 | 95.9 | 0.0 |
| GPQA Diamond pass@1 (%) | 73.3 | 75.8 | +2.5 |
| LiveCodeBench pass@1 (%) | 50.0 | 50.0 | 0.0 |
| Codeforces rating | 1444 | 1444 | 0 |
Four of six rows are identical, and the two that moved are AIME and GPQA Diamond. Neither text says why. Both give the same protocol (temperature 0.6, top-p 0.95, "typically between 4 and 64" samples per question); v1's Figure 2 caption says 16 responses per question for the plotted curve, and v2's Supplementary D.1 says k = 64 for AIME19. I cannot tell from the papers whether that, a longer run, or a re-evaluation produced the gap, so cite "77.9% (v2)" or "71.0% (v1)", never bare. Module 10 §6.1 lists what each version does and does not state.
A consensus score is a count. AIME 2024 has 30 problems12 (Fig. 13 caption), so 86.7 is 26 of 30, and v2 attaches the same 86.7 to R1-Zero (§2.3) and to R1 (Supplementary E.4, where R1's pass@64 is 90.0 and its majority vote 86.7)1. Two models can tie on a 30-item test, so the repetition is possible; it also means one problem moves the score by 3.3 points, and the cons@16 versus cons@64 label decides what the number measures. The "aha moment" (§2.3, Table 2) and the length curve are treated in Module 10 §6.3 and Module 13 §5.1.
3. R1: four stages around the same RL core
3.1 Why R1-Zero was not the product
The report says R1-Zero "struggles with challenges like poor readability, and language mixing", which it ties to a base trained mostly on Chinese and English, and that its RL stage "is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering"1 (§1, §3). The cold start was also a product decision. The report says its motivation "is primarily product-driven", because users find first-person reasoning more intuitive, and cautions that such patterns "may elicit unwarranted trust" and reflect "DeepSeek-engineered heuristics" rather than human-like intelligence1 (B.3.2).
3.2 Stage by stage
- Cold-start SFT. Thousands of long chains of thought: R1-Zero outputs sampled at temperature 1.0, kept if the answer is correct and the format readable, rewritten by DeepSeek-V3 and checked by humans. Output: Dev1.
- Reasoning RL. The R1-Zero recipe started from Dev1, with a language-consistency reward and a printed clip of 10 (the report's "first stage of RL"). The report states no step count. Output: Dev2.
- Rejection-sampling SFT. About 600K reasoning and 200K other samples; this SFT restarts from V3-Base, so Dev2 supplies data rather than weights. Output: Dev3.
- RL for all scenarios. The report's "second stage of RL": 1,700 steps at temperature 0.7, with reward models active only in the last 400. Output: R1.
- Distillation. About 800K samples fine-tune Qwen2.5 and Llama students with SFT only; no RL on the students.
| Step | Starts from | Data | Signal | Reported settings | Output |
|---|---|---|---|---|---|
| R1-Zero (side branch) | V3-Base | Not itemized | Accuracy + format | lr 3e-6, KL 0.001, T 1, G 16, 10,400 steps | R1-Zero |
| 1 Cold-start SFT | V3-Base | Thousands of long-CoT samples | None (SFT) | 2-3 epochs, lr 5e-5 to 5e-6 cosine, 32,768 context, batch 128 | Dev1 |
| 2 Reasoning RL | Dev1 | 26K math, 17K code (+8K bug fixing in text), 22K STEM, 15K logic prompts | Rules + language consistency | lr 3e-6, KL 0.001, ε = 10, T 1, G 16, 32,768 tokens, reference reset every 400; steps not stated | Dev2 |
| 3 Rejection-sampling SFT | V3-Base | About 600K reasoning (correct only) + 200K other; 804,745 in Table 5 | Correctness filter, V3 judge for some prompts | As step 1 | Dev3 |
| 4 RL, all scenarios | Dev3 | Reasoning sets + 66K general prompts (the text adds 12,000 harmlessness questions) | Rules; helpful and safety RMs; language | 1,700 steps, T 0.7, RMs in last 400 | R1 |
| 5 Distillation | Qwen2.5, Llama bases | About 800K | None (SFT) | 2-3 epochs, 32,768 context, batch 64, lr 1e-4 (1.5B) to 2e-5 (70B) | Six students |
Step 1, the cold start. Both versions say "thousands" of cold-start samples19. Lambert's textbook summary of the recipe says "100K+"10 (§3.2.3); the report does not support that figure and I follow the report.
Step 2. The first RL stage, started from Dev1, reuses R1-Zero's sampler (16 outputs per question, 32,768 tokens) with two changes. One is the language-consistency reward, "the proportion of target language words in the CoT" (Eq. 7). The other is the printed clip, "the GRPO clip ratio ε to 10"1 (§3.2.1). The report says "a lower value can lead to the truncation of gradients for a significant number of tokens ... while a higher value may cause instability", and gives neither an ablation nor a step count; v1 says only "until it achieves convergence"9. The nearest sentence is a credit line in Section 7: "Zhibin Gou proposed a large PPO clipping strategy to enhance GRPO performance, demonstrating its significance alongside Zhihong Shao and Junxiao Song"1. That is a lead, not an explanation: it does not say the strategy is ε = 10. Module 12 §2.4 works out what 10 means under token-level and sequence-level readings and marks its own reading as unverified. The language reward has a measured cost: in the ablation on a distilled 7B model, math stays comparable and coding shows "a slight degradation"1 (B.6).
Step 3. The second SFT goes back to DeepSeek-V3-Base: B.4.2 says both SFT stages fine-tune V3-Base for 2 to 3 epochs1, and v1 says the data are used to "retrain the DeepSeek-V3-Base model"9. Reasoning samples come from rejection sampling on the step-2 checkpoint, correct answers only; some prompts added at this stage are judged by DeepSeek-V3 against the ground truth, and chains with mixed languages, long paragraphs or code blocks are filtered out1 (B.3.3). The non-reasoning samples reuse DeepSeek-V3's SFT data, with generated chain of thought for some and none for queries such as "hello". Table 5 of the report splits the 804,745 samples into 395,285 math, 211,129 code, 10,124 STEM, 10,395 logic and 177,812 general, averaging 5,355.3 tokens: about 4.31 billion tokens per epoch, or 8.6 to 12.9 billion over two or three epochs (derived).
Step 4. The second RL stage "retains most of the parameters from the first stage", with the temperature lowered to 0.7 because "higher temperatures in this stage lead to incoherent generation". Its reward is a sum (Eqs. 8 to 10): the rule reward for reasoning, a reward-model score plus a format reward for general data, and the language term. Of the 1,700 steps, general data and preference rewards enter "exclusively in the final 400", steps 1,301 to 1,700, because "more training steps with the model based preference reward signal may lead to reward hacking"1 (§3.2.2). Module 16 §1.2 reads the mixture and Module 19 §3 the hacking curve in Supplementary Figure 6, whose pass@1 values (about 0.346 at the peak, about 0.289 at the end) were read by eye from the PDF there. The helpful reward model trains on 66,000 pairs judged by DeepSeek-V3 and the safety model on 106,000 point-wise labels (Module 4).
3.3 What each stage bought
| Benchmark | R1-Zero | Dev1 | Dev2 | Dev3 | R1 | R1 − Zero |
|---|---|---|---|---|---|---|
| AIME 2024 | 77.9 | 59.0 | 74.0 | 78.1 | 79.8 | +1.9 |
| MATH-500 | 95.9 | 94.2 | 95.9 | 95.4 | 97.3 | +1.4 |
| CNMO 2024 | 88.1 | 58.0 | 73.9 | 77.3 | 78.8 | -9.3 |
| GPQA Diamond | 75.8 | 66.1 | 70.7 | 71.2 | 71.5 | -4.3 |
| LiveCodeBench | 50.0 | 57.5 | 63.5 | 64.6 | 65.9 | +15.9 |
| Codeforces rating | 1444 | 1534 | 1687 | 1746 | 2029 | +585 |
| Aider-Polyglot | 12.2 | 6.7 | 25.6 | 44.8 | 53.3 | +41.1 |
| IF-Eval (strict) | 46.6 | 71.7 | 72.0 | 78.1 | 83.3 | +36.7 |
| AlpacaEval 2.0 (LC) | 24.7 | 50.1 | 55.8 | 62.1 | 87.6 | +62.9 |
| ArenaHard | 53.6 | 77.0 | 73.2 | 75.6 | 92.3 | +38.7 |
| MMLU-Pro | 68.9 | 74.1 | 83.8 | 83.1 | 84.0 | +15.1 |
| SimpleQA | 30.3 | 17.8 | 28.2 | 24.9 | 30.1 | -0.2 |
The cold start cost reasoning before RL repaired it. At Dev1, AIME fell from 77.9 to 59.0 and CNMO from 88.1 to 58.0, and the report attributes the Dev1 degradation to "the limited size of the cold-start dataset"; step 2 then recovered 15.0 AIME points, to 74.01 (§4). On LiveCodeBench the gains come from medium and hard problems: the hard-problem pass rate runs 17.09, 23.21, 30.36, 33.16, 34.44 across the five checkpoints, while easy problems sit at 98 to 1001 (Table 14). The last stage mostly moves chat metrics: from Dev3 to R1, AlpacaEval 2.0 rises 25.5 points and ArenaHard 16.7. The report writes "25%" and "17%", which are points on the win-rate scale (derived).
For serving cost, Supplementary E.3 and E.4 report that on 366 problems from 93 competitions held in 2024, R1 averages 8,793 thinking tokens per problem at 61.8% pass@1, fewer than 7,000 on simple problems and more than 18,000 on the hardest, against 711 output tokens at 24.7% for GPT-4o-05131.
4. Compute and cost
Supplementary B.4.4 gives hardware and durations, and Table 7 prices them at "$2 per GPU hour": R1-Zero on 64×8 H800 GPUs for "approximately 198 hours"; R1 on "the same 64*8 H800 GPUs, completing the process in about 4 days, or roughly 80 hours"; and 5K GPU-hours to create the SFT datasets1. Module 1 §4.1 derives the same ledger. Table 5 repeats it so the arithmetic sits beside the case.
| Line | Reported GPU-hours / USD | Check: 512 GPUs × hours | Verdict |
|---|---|---|---|
| R1-Zero, 198 h | 101K / $202K | 101,376 / $202,752 | Agrees |
| SFT-data creation | 5K / $10K | No hours or GPU count given | Not checkable |
| R1, "roughly 80 hours" | 41K / $82K | 40,960 / $81,920 | Agrees |
| R1, "about 4 days" (96 h) | not tabulated | 49,152 / $98,304 | Disagrees by 8,192 GPU-hours |
| Total | 147K / $294K | 147,336 / $294,672 (155,528 / $311,056 with 96 h) | Agrees to rounding at 80 h |
Table 7 and "roughly 80 hours" agree with each other, and "4 days" is the outlier; treat 80 hours as the figure of record, as Module 1 does. The ledger leaves out what the report mentions separately: A100 pilots on a 30B model that gave the confidence to "scale up to 660B R1-Zero and R1"1 (B.4.4; A.1 and the report's Table 8 say 671B), any hyperparameter search, and failed runs. It does not say which of the four training steps the 80 hours covers, and step 2 has no stated step count, so the 80 hours cannot be split by stage.
Lambert puts R1's share at "only about 5%" of V3's compute: 147K H800 GPU-hours against 2.8M for pretraining the base10 (p. 14), which is 147/2,800 = 5.25% (derived). Module 1 §4.4 uses V3's own pretraining figure instead and gets 3.75% for R1-Zero alone. The denominators differ, so do not mix numbers across the two tables. One all-in rate follows: R1-Zero's 5,324,800 sampled outputs over 101,376 GPU-hours is about 52.5 outputs per GPU-hour, and at Module 1's assumed mean of about 6,000 tokens per completion about 88 sampled tokens per GPU-second, rollout and training combined (derived).
5. Distillation versus RL on a small model
Supplementary F trains one base two ways. Qwen2.5-32B-Base gets "large-scale RL training" on math, code and STEM "for over 10K steps", giving Qwen2.5-32B-Zero (named DeepSeek-R1-Zero-Qwen-32B in v1)19. Its settings (B.4.1) are R1-Zero's with a learning rate of 2e-6: KL 0.001, temperature 1, 16 samples of up to 32,768 tokens, 32 questions and 512 outputs per step, reference replaced every 400 steps. The other arm is SFT of the same base on the 800K R1-generated samples, DeepSeek-R1-Distill-Qwen-32B.
| Model | AIME 2024 pass@1 | AIME cons@64 | MATH-500 | GPQA Diamond | LiveCodeBench |
|---|---|---|---|---|---|
| QwQ-32B-Preview | 50.0 | 60.0 | 90.6 | 54.5 | 41.9 |
| Qwen2.5-32B-Zero (RL, over 10K steps) | 47.0 | 60.0 | 91.6 | 55.0 | 40.2 |
| R1-Distill-Qwen-32B (SFT on R1 samples) | 72.6 | 83.3 | 94.3 | 62.1 | 57.2 |
| Distilled minus RL (points) | +25.6 | +23.3 | +2.7 | +7.1 | +17.0 |
The report draws two conclusions: distilling stronger models "yields excellent results, whereas smaller models relying on the large-scale RL ... require enormous computational power and may not even achieve the performance of distillation", and, second, "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning"1 (F.1). The ladder of students shows how far the first goes: AIME 2024 pass@1 of 28.9 for the 1.5B Qwen student (GPT-4o-0513 scores 9.3, Claude-3.5-Sonnet-1022 16.0), 55.5 at 7B, 69.7 at 14B, 72.6 at 32B, and 50.4 and 70.0 for the Llama 8B and 70B students1 (Table 15).
The comparison controls less than its table suggests. The RL arm samples at least \(10{,}000 \times 512 = 5.12\) million completions of up to 32,768 tokens (derived), and the SFT arm reads 804,745 sequences per epoch for two or three epochs, but the distilled student inherits its teacher: 147K GPU-hours of R1 and the generation of the samples. "Economical" is the student's marginal cost. The arms also see different data, since the RL prompts are math, code and STEM while the SFT set includes 177,812 general samples (derived from Table 5). Qwen3 later ran the comparison with GPU-hours at one starting checkpoint: on Qwen3-8B, RL reached 67.6 on AIME'24 with 17,920 GPU-hours and left pass@64 unchanged, while on-policy distillation reached 74.4 with 1,800 GPU-hours13 (§4.7, Table 21). Module 21 sets those rows beside R1's.
A second small-model result sits in Table 17: Qwen2-Math-7B-Zero, trained with "approximately 10,000 policy gradient update steps" on a base released in August 2024, "prior to the launch of the first reasoning model, OpenAI-o1 (September 2024), to ensure the base model was not exposed to any reasoning trajectory data", reaches 22.3% on AIME 2024 and 18.1% on AIME 2025, against 7.9% and 4.6% for Qwen2-Math-7B-Instruct1.
6. What failed, as the report states it
| Attempt | What was tried | Reasons given | Conclusion given |
|---|---|---|---|
| Process reward model | Model-based reward on reasoning steps | A fine-grained step is hard to define; step correctness is hard to judge (model annotation unsatisfactory, manual annotation does not scale); a model-based PRM "inevitably leads to reward hacking" and retraining it adds cost and complexity | Good at reranking top-N or guiding search; advantage "limited compared to the additional computational overhead" in large-scale RL |
| Monte Carlo tree search | AlphaGo-style search with a pre-trained value model; answers split into tagged steps; actor and value trained on search outputs | The token-level search space is "exponentially larger" than chess, and a node-extension limit traps the model in local optima; the value model steers generation and a fine-grained one is hard to train | Helps at inference with a pre-trained value model; iterative improvement by self-search "remains a significant challenge" |
| Small bases | RL on a 7B dense and a 16B mixture-of-experts base | No meaningful AIME gains; repetition as responses lengthened | Moved to 32B dense, 230B and 671B MoE; effectiveness "highly dependent on the underlying model capacity" |
Later evidence points the same way without settling it. Kimi k1.5 set out to avoid "Monte Carlo tree search, value functions, and process reward models"11. Llama 3 still uses outcome and stepwise reward models, but to filter SFT reasoning traces14 (§4.3.3), which is the reranking use R1 concedes. Lightman et al. found that process supervision "significantly outperforms outcome supervision" for training reward models on MATH15. The three are compatible: process labels rank and filter well, and R1's finding concerns using them as the RL reward. Module 4 §4.5 covers the mechanics.
G.1 widens the verifier in one direction: besides rules, an LLM that checks an answer against a ground truth "demonstrates particular effectiveness for tasks with well-defined, concise answers" but "exhibits limited generalizability to more complex tasks, including open-ended generation and long-form writing"1. That is the same boundary R1 draws in its limitations (§6): for tasks "that cannot be effectively evaluated by a reliable reward model, scaling up pure RL methods remains an open challenge", so it used human-annotated SFT data there and "only conduct[ed] RL for hundreds of steps"1.
7. The report across versions, and where it disagrees with itself
7.1 From v1 to v2
Version 1 is 22 pages and version 2 is 8691. The additions, by location: hyperparameters and run length (§2.1, §3.2); the RL data recipe and SFT statistics (Tables 4 and 5, B.3); the PPO-versus-GRPO study (A.3) and the RL framework (B.1); the cost table (Table 7); the reward-hacking and language-consistency ablations (B.5, B.6); reasoning-behavior counts over training (C.1, C.2); Qwen2-Math-7B-Zero (Table 17); a safety report (D.3); and results on problems released after training, 75% pass@1 on AIME 2025 against o1's 80% (E.2). Two headline numbers changed (Table 2), and the run behind the 32B comparison was renamed from DeepSeek-R1-Zero-Qwen-32B to Qwen2.5-32B-Zero. One sentence is gone: v1's contribution bullet calling R1-Zero "the first open research to validate that reasoning capabilities of LLMs can be incentivized purely through RL" does not appear in v2 (I searched its text).
7.2 The Nature copy
Nature published the work in 2025 as volume 645, issue 8081, pages 633 to 638; the reference lists of DeepSeek's own V3.2 and V4 reports cite it that way56. I did not fetch nature.com, which redirected to a cookie page when this course's source map was built. I read the NIH PMC copy of the article through a summarizing fetch tool16, so what follows is that tool's extraction and not my own reading of the page. The peer-review file linked from the PMC page returned only a download placeholder; I did not read it and did not try another route, so I cannot say what the review changed.
What the extraction shows, as differences from arXiv v2. The dates are received 14 February 2025, accepted 17 July 2025, cited as 17 September 2025. The main text is shorter and sends the reader to the Supplementary Information by section number (cost to 2.4.4, pass@1 and cons@16 to 4.1, GRPO versus PPO to 1.3, data statistics to 2.3), where arXiv v2 uses B.4.4, D.1, A.3 and B.3, so v2 is not a verbatim copy of the Nature layout. The figures I could compare match v2: 15.6% to 77.9%, 86.7%, a first-stage clip of 10, and a Figure 1 legend that also says cons@16. On release, the Data availability section points to the DeepSeek-R1 repository for the rejection-sampling and RL-prompt samples, with Zenodo DOI 10.5281/zenodo.15753193, and says weights are "available under an MIT license", while arXiv v2's Appendix I reads "We also release SFT and RL data to the public at xxx"1, a placeholder. The extraction's author-contributions paragraph is generic and does not contain the sentence crediting "a large PPO clipping strategy", which arXiv v2 §7 does. Treat a claim that rests on one document as belonging to that document, and name the version.
7.3 A ledger of conflicts
| Item | One place says | Another says | Handling |
|---|---|---|---|
| R1 training time | "about 4 days, or roughly 80 hours" (B.4.4) | 512 GPUs × 80 h = 40,960, Table 7's "41K"; 4 days is 96 h | Use 80 h; Module 1 §4.1 |
| GRPO ratio | Eq. 1 and Eq. 11: sequence-level ratio, no \(1/|o_i|\) | DeepSeekMath Eq. 3: per-token ratio with \(1/|o_i|\) | Report does not say which ran; Module 12 §2.4 |
| Clip value | "ε to 10" in step 2 (§3.2.1); R1-Zero's ε not stated (§2.1) | PPO's suggested 0.23 | Printed once, unexplained; do not generalize |
| R1-Zero AIME pass@1 | 71.0 (v1) | 77.9 (v2) | Name the version; Module 10 §6.1 |
| The 86.7 label | cons@64 (v1 Table 2 and text; v2 D.1) | cons@16 (v1 Fig. 2 and v2 Fig. 1 legends) | State both; 86.7 is 26 of 30 |
| Code RL prompts | Table 4: 17K | B.3.1: 17K competition problems plus 8K bug fixes | Say 17K or 25K, not one silently |
| SFT split | About 600K reasoning, 200K other (B.3.3) | Table 5: 626,933 in four domain rows, 177,812 general | Domain rows are not the reasoning split |
| The 800K | B.3.3: rejection sampling from the first-RL-stage checkpoint | Appendix F: "samples generated with DeepSeek-R1" | Unclear whether distillation data were regenerated |
| Model size | "660B R1-Zero and R1" (B.4.4) | 671B total (A.1; the report's Table 8) | Use 671B |
| Cold-start size | "thousands" (v1 §2.3.1; v2 §3) | "100K+" (Lambert §3.2.3) | Follow the report |
| Small bases | 7B dense and 16B MoE "consistently failed" on AIME (G.1) | Qwen2-Math-7B-Zero reaches 22.3% (Table 17) | Different bases; unreconciled |
None of these rows changes the story of Sections 2 to 5. Three change what you may safely quote: the clip, the sequence-versus-token ratio, and the unlabeled 86.7.
8. After R1: critiques, and what later recipes kept, changed or dropped
8.1 Four checks on R1's claims
The papers below differ from R1 in base model, data and scale, so each tests a claim or fills a gap and none is a replication. Dr. GRPO hosted DeepSeek-V3-Base and found that it "already exhibit[s] 'Aha moment'", that Qwen2.5 base models "demonstrate strong reasoning capabilities even without prompt templates", and that GRPO's two normalizers bias the update: dividing by response length favors short correct and long wrong responses, and dividing by the group's reward standard deviation up-weights questions that are too easy or too hard17 (abstract, §3.1). Its minimalist recipe reached 43.3% on AIME 2024 from Qwen2.5-Math-7B17. R1's own appendix is closer to that reading than the "aha" framing is: pretraining exposure to reasoning traces "equips the model with the capability to generate plausible solution candidates, from which reinforcement learning can effectively identify and optimize high-quality outputs"1 (A.1; Module 10 §6.3).
Spurious Rewards found that on Qwen2.5-Math-7B, randomly assigned rewards raised MATH-500 by 21.4 points against 29.1 for ground-truth rewards, traced it to a clipping bias that amplifies behaviors with high prior probability, and reported that such rewards "often fail to produce gains for other model families, such as Llama3 or OLMo2"12. It did not test DeepSeek-V3-Base, so it says nothing direct about R1-Zero. It does make an R1-Zero-style gain on a Qwen math base weak evidence about reward quality, which bears on replications built that way, including the 43.3% above. R1 itself flags contamination: n-gram decontamination "cannot prevent the paraphrase of testset", so benchmarks "released before 2024 may suffer from contamination issues"1 (D.1); its answer is the AIME 2025 result of Section 7.1. Olmo 3 argues that open-weight bases with undisclosed mid-training data can contaminate RLVR benchmarks, and releases an RL-Zero run from a fully open base18 (Module 31).
Yue et al. find RLVR-trained models ahead of their base models at small k and behind at large k, with the RLVR reasoning paths already inside the base model's sampling distribution, whereas distillation "can introduce new reasoning patterns from the teacher"19. DeepSeekMath saw the same split between Maj@K and Pass@K in 20244. R1's E.4 gives pass@64 of 90.0 against pass@1 of 79.8 for R1 itself, which is not a base-model comparison1. Module 20 weighs this evidence.
DAPO ran GRPO on Qwen2.5-32B and reached "only 30 points on AIME — a performance significantly below DeepSeek's RL (47 points)", adding that "critical training details may have been omitted in the R1 paper"20. Version 2's added hyperparameters answer part of that. The step count of the first RL stage and the reason for ε = 10 remain absent.
8.2 Kept, changed, dropped
| R1 choice1 | Status | Later reports |
|---|---|---|
| Group-relative advantage, no critic | Kept | V3.2 and V4 run GRPO56; Qwen3's reasoning RL uses it13; Olmo 3's OlmoRL is GRPO plus changes18 |
| Advantage \((r-\mathrm{mean})/\mathrm{std}\) | Changed | V3.2 uses \(A = R - \mathrm{mean}(R)\)5; Olmo 3 drops std18; Dr. GRPO drops std and \(1/|o_i|\)17 |
| KL 0.001 in the loss; reference reset every 400 steps | Changed | V3.2: weak or no KL for math, estimator corrected with the importance ratio5; DAPO and Olmo 3 remove it2018; ProRL keeps it, with hard resets, from a distilled start21 |
| Clip ε = 10, printed | Changed | DAPO: 0.2 below, 0.28 above20; Olmo 3: 0.2 and 0.27218 |
| Sequence-level objective, as printed | Changed | DAPO uses a token-level loss20; Olmo 3 normalizes over all tokens in the batch18 |
| Rule-based reasoning reward, language reward, no neural RM | Kept | Qwen3 keeps rules for their "high precision, preventing issues like reward hacking"13; V3.2 uses rule-based outcome reward, a length penalty and a language-consistency reward5 |
| Scalar helpful and safety RMs for general data | Changed | V3.2: a generative reward model with per-prompt rubrics5; V4: no scalar reward models, the actor doubles as the judge6. A change of reward-model type for general tasks, not a reversal of R1's rule for reasoning |
| SFT, RL, SFT, mixed RL | Changed | V3.2 merges reasoning, agent and alignment RL into one stage5; V4 replaces mixed RL with on-policy distillation6; Qwen3 keeps four stages with thinking-mode fusion13 |
| Distilling R1's 800K into small models | Kept, extended | Qwen3's strong-to-weak distillation, off-policy then on-policy, at about 1/10 the GPU-hours of its four-stage pipeline13 |
| No PRM, no MCTS in RL | Kept | Kimi k1.511; Llama 3 uses stepwise RMs only to filter SFT traces14 |
| No treatment of sampler-trainer mismatch | Added | V3.2: Keep Routing, Keep Sampling Mask, off-policy sequence masking5 (Module 14) |
The last row rests on absence: R1 v2 contains no discussion of routing, mismatch or numerical discrepancy (I searched its text). The pattern in the rest is that the stage structure and the objective changed most, while the verifier-first reward for reasoning and the rejection of step-level RL rewards survived intact.
Where experts disagree
R1 was a method claim (verifiers, GRPO, pure RL from a base) and a set of artifacts (a pipeline, a cost, a failure list). Three disputes it started or sharpened are still open, and each can be read against a table in the report.
What did RL do in R1-Zero?
RL elicited new reasoning behavior
R1's own framing: "emergent development of advanced reasoning patterns", the "aha moment"1
R1-Zero starts from a base with no SFT, and the supplement counts behaviors over training: reflective words rise 5- to 7-fold, and "wait" is nearly absent early, occasional between steps 4,000 and 7,000, and spikes after step 8,000 (C.2).
- A measured trajectory: AIME 15.6 to 77.9 (v2) with response length growing alongside.
- No human reasoning traces in the loop.
- No base-model control for the counted behaviors.
- The same report says pretraining supplies "plausible solution candidates" and that checkpoints "inherently possess substantial potential"1.
RL selects and sharpens what the base can already produce
Yue et al.19; DeepSeekMath's Maj@K versus Pass@K4; Qwen3's pass@6413; Dr. GRPO on V3-Base17
RLVR models win at small k and lose at large k, their reasoning paths lie inside the base distribution, and Dr. GRPO finds the "aha" behavior in V3-Base itself.
- Independent groups, different bases and methods, one direction.
- Fits R1's own appendix on candidates.
- ProRL argues such studies used math-heavy settings and stopped RL after "no more than hundreds of steps"21.
- Large-k comparisons on a 30-problem benchmark are coarse.
On some bases the gain is prior amplification
Spurious Rewards12
Random rewards give +21.4 MATH-500 points on Qwen2.5-Math-7B against +29.1 for ground truth; GRPO's clip amplifies high-prior behaviors; other families mostly do not move.
- A direct control: the reward carries no task signal.
- Explains why a Qwen math base can mislead.
- Does not test V3-Base, so it bears on R1-Zero only by analogy.
Prolonged RL expands the boundary
ProRL21; a two-stage reconciliation22
With KL control, reference resets and diverse tasks, over 2,000 steps on a 1.5B model found strategies "inaccessible to base models, even under extensive sampling". The reconciliation proposes sharpening first, exploration later.
- Addresses the "too short" objection head-on.
- Starts from a distilled checkpoint and changes several things at once, so the cause is not isolated.
- The reconciliation is the authors' own assessment.
The first two are less opposed than they look: R1's appendix already describes selection among candidates, so the live question is whether more compute adds anything beyond selection. Quote R1's gains as pass@1 gains and do not read "emergent" as "absent in the base". What would settle it: pass@k at large k on one contamination-free base, matched data, runs as long as R1-Zero's 10,400 steps, and a base-model control for the counted behaviors. Module 20 holds the full ledger.
Process or outcome supervision for the RL reward?
Outcome and rules only
R1 Supplementary G.21; Kimi k1.511
Step boundaries, step labels and hackability make a model-based PRM a poor RL reward; a verifier is cheaper to trust.
- Simplest pipeline; scaled to 10,400 steps without a learned scorer.
- Two independent reports on the same day reached the same design.
- G.2 gives no numbers or ablation, and frames the attempts as early failures.
Step-level labels help where they exist
Lightman et al.15; Llama 3's stepwise reward models14
Process supervision "significantly outperforms outcome supervision" for MATH reward models; Llama 3 filters SFT traces with stepwise models.
- A measured ranking advantage.
- Used to rank or filter, not as the reward of a long-chain RL run.
- Needs step labels, which cost human effort.
Both hold. R1 concedes that PRMs rerank well, and the two positions describe different uses. None of the reports read for this module trains a long-chain RL run on a learned PRM reward, so the question is untested rather than answered; a matched-compute run with a PRM reward against an outcome reward would settle it.
RL or distillation for a strong small reasoning model?
Distill from a stronger teacher
R1 Supplementary F.11; Qwen313
On one 32B base, SFT on R1 samples beats 10K+ RL steps by 25.6 AIME points; Qwen3-8B reaches 74.4 against 67.6 at 1/10 of the GPU-hours.
- Two reports, same direction.
- Cheap for the student.
- Capped by the teacher, whose cost is not counted.
- R1's arms differ in data as well as method.
RL remains the way past the teacher
R1's second conclusion1; ProRL21
"Advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning"; ProRL reports further gains over a distilled 1.5B start.
- Only route to a model stronger than every available teacher.
- No matched-compute comparison exists in the sources read.
When a stronger teacher exists and the student is small, distillation wins on both quality and cost in the two measured cases, and R1 itself says so. RL is the only route past the teacher, but I found no source that compares it with distillation at matched compute from the same start with the teacher's cost counted; that comparison would settle it.
What is changing
R1's own list of open problems has moved. For tasks without a reliable verifier, where the report says scaling pure RL "remains an open challenge"1, 2026 recipes train rubric-based generative judges inside the RL loop (in V4 the actor itself is the judge), and V4 replaces mixed RL with on-policy distillation of specialists6, a step Lambert says labs are "far from unanimous" about10 (§3.2.4). Three things in R1's own record stay open: the step count and the reason for ε = 10 in the first RL stage, which training steps the 80 hours cover, and what the Nature review changed, which I could not read. I cite V4 without a date because the source map records an unexplained mismatch between its arXiv identifier and the date the arXiv API reports. V3.2 and V4 are the reports in this module most likely to be superseded.
References
- paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v2, 4 Jan 2026; the source map lists v2 as the expanded version of the Nature article (see Section 7.2). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: §1, §2.1 (GRPO credit, Eqs. 1 to 3, run size), Fig. 1, §2.2 (Eq. 4), §2.3 (15.6% to 77.9%, 86.7%), §3 and §3.2 (pipeline, Eqs. 7 to 10, clip 10), §6, §7 (contributions); Table 3; A.1, A.3 (Figs. 3 and 4), B.1, B.3 and Tables 4 and 5, B.4 and Tables 6 and 7, B.5, B.6, C.2, D.1, E.2 to E.5 and Table 14, F.1 and Tables 15 to 17, G.1, G.2, Appendix I.
- paperSchulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv:1502.05477 v5, 20 Apr 2017. https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Used for: the 2015 date and the KL-constrained trust-region update.
- paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv:1707.06347 v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the clipped surrogate and the suggested ε = 0.2 (§3, Eq. 7).
- paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the introduction of GRPO (abstract, §1, §4.1.1), the per-token objective with \(1/|o_i|\) (Eq. 3), Maj@K versus Pass@K (§5.2.2).
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: §3 (mixed RL, reward types, rubric-based generative reward model), §3.1 (advantage without std, corrected KL estimator, Keep Routing, Keep Sampling Mask, sequence masking), and the reference-list entry for R1 (Nature, 645(8081):633-638, 2025).
- paperDeepSeek-AI (Xu, A., Lin, B., Xue, B., et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: §5.1 (pipeline mirrors V3.2 with mixed RL replaced by on-policy distillation; GRPO with hyperparameters aligned to prior work), §5.1.1 (generative reward model, no scalar reward models), and the reference-list entry for R1 (Nat., 645(8081):633-638, 2025).
- paperOuyang, L., Wu, J., Jiang, X., Almeida, D., et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv:2203.02155 v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: the learned-reward-model plus PPO pipeline in Figure 1 (§3.5).
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124 v5, 14 Apr 2025; v1 22 Nov 2024. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the definition and date of RLVR (§1, §6).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (version 1). arXiv:2501.12948 v1, 22 Jan 2025. Same arXiv identifier as reference 1, read here as a separate version. https://arxiv.org/pdf/2501.12948v1. Accessed 2026-10-26. Used for: §1.1 ("first open research" sentence), §2.2.4, Table 2 and Figure 2 (71.0%, 86.7%, cons@64 and cons@16 labels), §2.3 (four-stage pipeline, "convergence", retraining from V3-Base), §3 evaluation setup, §4.1 and Table 6, §4.2.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501 v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: p. 14 (RLVR coined in Tulu 3, R1 "famous for popularizing RLVR", 147K versus 2.8M GPU-hours), §3.2.3 (R1 recipe summary), §3.2.4 (MOPD, "far from unanimous"), §7.3.1 (earlier verifiable-domain work), §7.3.2 Table 4 (release dates), bibliography (o1-preview, Sept. 2024).
- paperKimi Team (Du, A., Gao, B., Xing, B., et al.; Moonshot AI). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599 v4, 3 Jun 2025; v1 22 Jan 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: the 22 January 2025 first-version date; abstract (no MCTS, value functions or process reward models; o1-matching results).
- paperShao, R., Li, S. S., Xin, R., Geng, S., et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947 v2, 25 Feb 2026. https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: abstract and §1 (21.4 versus 29.1 points; model dependence), Figure 13 caption (AIME has 30 questions).
- paperYang, A., Li, A., Yang, B., Zhang, B., et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388 v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: §4 (four-stage pipeline, strong-to-weak distillation at about 1/10 of the GPU-hours), §4.2 (GRPO reasoning RL), §4.4 (rule-based rewards), §4.7 and Table 21 (RL versus on-policy distillation on Qwen3-8B).
- paperLlama Team, AI @ Meta (Grattafiori, A., Dubey, A., Jauhri, A., et al.). The Llama 3 Herd of Models. arXiv:2407.21783 v3, 23 Nov 2024. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: outcome and stepwise reward models used to filter SFT reasoning traces (§4.3.3).
- paperLightman, H., Kosaraju, V., Burda, Y., Edwards, H., et al. Let's Verify Step by Step. ICLR 2024; arXiv:2305.20050 v1, 31 May 2023. https://arxiv.org/abs/2305.20050. Accessed 2026-10-26. Used for: abstract (process supervision outperforms outcome supervision on MATH).
- paperDeepSeek-AI (Guo, D., et al.). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645(8081):633-638, 2025; NIH PubMed Central copy PMC12443585, CC BY 4.0. https://pmc.ncbi.nlm.nih.gov/articles/PMC12443585/. Accessed 2026-10-27. Used for: received and accepted dates, Supplementary section pointers, Figure 1 legend, data and code availability, author-contributions paragraph. Read through a summarizing fetch tool (its extraction, not the page text); the Supplementary Information and peer-review PDFs were not read. Not in the researcher's source map.
- paperLiu, Z., Chen, C., Li, W., Qi, P., et al. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783 v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: abstract (aha moment in V3-Base, templates), §3.1 (length and difficulty biases), §3.2 and Table 4 (minimalist recipe, 43.3% from Qwen2.5-Math-7B).
- paperTeam Olmo (Ettinger, A., Bertsch, A., et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961 v2, 14 Apr 2026; v1 15 Dec 2025. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL (§4.4.1: GRPO plus token-level loss, no KL, clip-higher, no std normalization; hyperparameters 0.2 and 0.272), the RL-Zero contamination argument.
- paperYue, Y., Chen, Z., Lu, R., Zhao, A., et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837 v5, 24 Nov 2025. https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Used for: abstract (pass@k crossover, paths inside the base distribution, distillation introduces new patterns).
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476 v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: §1 (30 versus 47 AIME points; "details may have been omitted in the R1 paper"; 50 points in 50% of the steps), §2.3 (KL removed), §3.1 (clip-higher 0.28), §3.3 (token-level loss).
- paperLiu, M., Diao, S., Lu, X., Hu, J., et al. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864 v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: §1 (methodological-constraints argument, "hundreds of steps", "inaccessible to base models"), §2 (KL control with reference resets, over 2,000 steps).
- paperYao, X., Yu, L., Hu, X., Teng, F., et al. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028 v1, 5 Oct 2025. https://arxiv.org/abs/2510.04028. Accessed 2026-10-26. Used for: abstract (exploitation stage then exploration stage; the authors' own reconciliation).