Reinforcement Learning for LLM Post-Training · Module 27

Case Study: DeepSeek-R1: R1-Zero, the Four-Stage Pipeline, Its Compute, and What the Nature Version Changed

DeepSeek-R1 is a GRPO case, not a TRPO one: GRPO entered through DeepSeekMath in February 2024, and TRPO is the 2015 ancestor of PPO. This module reads the report stage by stage (R1-Zero's rule-based RL, the four-stage R1 pipeline, its GPU-hours, what failed), flags every place where two versions or two sections of the report disagree, and sets later recipes against it. Afterwards you can reproduce R1's schedule arithmetic and say which of its choices later reports kept, changed or dropped.

Course 45 min read 22 sources As of 2026-10-27 Prerequisites: Modules 9, 10, 13, 20
Two lineages meeting in DeepSeek-R1: the optimizer lane runs TRPO, PPO, GRPO in DeepSeekMath, DeepSeek-R1, then DeepSeek-V3.2 and V4; the training-signal lane runs a learned reward model with PPO in InstructGPT, the name RLVR in Tulu 3, rule-based rewards in R1-Zero and R1, and rubric-based generative reward models in V3.2 and V4 Optimizer TRPO 2015 KL trust region PPO 2017 clipped ratio GRPO Feb 2024 DeepSeekMath DeepSeek-R1 Jan 2025 adopts GRPO V3.2 → V4 Dec 2025, 2026 modify, keep GRPO clip, not constraint group, not critic adopted, scaled modified Training signal Learned RM + PPO Mar 2022 InstructGPT RLVR named Nov 2024 Tulu 3 R1-Zero, then R1 Jan 2025 rules first, RMs last Rubric GRMs 2025, 2026 V3.2, V4 verifier replaces RM no SFT first judge model, rubric same report OpenAI o1, Sep 2024: closed; date attested second-hand
Figure 1. Two lineages that meet in DeepSeek-R1 (v1 posted 22 January 2025)1. Top lane, the optimizer: TRPO (ICML 2015)2 and PPO (2017)3, then GRPO, introduced in DeepSeekMath (February 2024)4 and adopted by R1; V3.2 (2 December 2025)5 modifies it and V4 keeps GRPO for its domain specialists6. Bottom lane, the reward: a learned reward model with PPO in InstructGPT7, the name RLVR in Tulu 3 (22 November 2024)8, rule-based rewards first and reward models only late in R1, rubric-based generative reward models in V3.2 and V456. The o1 date rests on R1's own sentence and a bibliography entry, not on OpenAI's page (Section 1).

1. Lineage: what R1 inherited and what it originated

The request behind this course says R1 introduced TRPO. It did not. R1's Section 2.1 names its algorithm and its source: GRPO "was originally proposed to simplify the training process and reduce the resource consumption of Proximal Policy Optimization (PPO)", credited to Shao et al., 20241. That paper is DeepSeekMath, which describes GRPO as a PPO variant that "foregoes the critic model, instead estimating the baseline from group scores"4. TRPO is Schulman et al.'s 2015 trust-region method2, the thing PPO simplified3. Module 3 derives both, Module 9 derives GRPO, and Module 1 counts R1's GPU-hours; this module reads what R1 did.

Table 1. Who introduced what, the date each source gives, and what R1 does with it. "Second-hand" means the date comes from a text that cites the original.
ItemIntroduced byDateR1's relation to it
KL-constrained trust-region updateSchulman et al., TRPO2ICML 2015Not used; the idea survives as PPO's clip
Clipped surrogate, several minibatch passesSchulman et al., PPO32017R1's Eq. 1 keeps the min and the clip
Group-mean baseline, KL term in the loss (GRPO)Shao et al., DeepSeekMath4Feb 2024Adopted and credited1
Reasoning model trained with RLOpenAI o1Sep 2024, second-hand110Closed; R1 publishes an open recipe
The name RLVRLambert et al., Tulu 3822 Nov 2024R1-Zero is the large-scale case; Lambert calls R1 "famous for popularizing RLVR"10
Pure RL on a base model for long CoT, open weightsDeepSeek-R1, its own claim922 Jan 2025R1-Zero

The o1 row needs a caveat. OpenAI's announcement page returned HTTP 403 when this course's source map was built, and I did not work around it. The month rests on two second-hand statements: R1's Appendix F.1 says its Qwen2-Math-7B experiment ran before "the first reasoning model, OpenAI-o1 (September 2024)"1, and Lambert's bibliography lists "Introducing OpenAI o1-preview", September 202410.

R1 and Kimi k1.5 appeared on the same day: Lambert's Table 4 dates both 2025-01-2210, and the first version of k1.5 is also 22 January 202511. k1.5 claimed o1-matching numbers with a different objective and no GRPO11 (Module 28), so January 2025 was not a one-recipe result.

What R1 did contribute is narrower and easier to defend. It showed, on a 671B-parameter mixture-of-experts base with 37B parameters active per token and 14.8T pretraining tokens1 (App. A.1), that rule-based rewards and GRPO elicit long reasoning with no SFT stage first. It then published a multi-stage recipe that turns that behavior into a usable assistant, released weights, and, in v2, supplied enough hyperparameters to do the arithmetic. Its conclusion states the claim as a recipe: "the key to unlocking this potential lies not in large-scale human annotation but in the provision of hard reasoning questions, a reliable verifier, and sufficient computational resources for reinforcement learning"1 (§6). Smaller verifiable-domain RL existed earlier. Lambert's survey lists STaR (2022-2023) and VinePPO, and notes that Tulu 3 ran PPO against correctness rewards without losing general performance; what those methods lacked was R1's scale, or they "made sacrifices in overall performance"10 (§7.3.1).

2. R1-Zero: RL on a base model

2.1 What was trained, and where the detail lives

R1-Zero starts from DeepSeek-V3-Base and applies GRPO with no SFT stage1. The reward is a rule-based accuracy term plus a format term (Eq. 4), "combined with the same weight", and the report says it abstains "from applying neural reward models—whether outcome-based or process-based—to reasoning tasks" because they are "susceptible to reward hacking during large-scale reinforcement learning"1 (§2.2). Module 10 reads that reward, the template and the prompt data line by line, and Module 4 §4.5 covers why process reward models did not survive RL at scale; Section 6 below gives R1's own reasons. The optimizer settings are the ones Module 9's Table 4 sets beside DeepSeekMath and later recipes: learning rate 3e-6, KL coefficient 0.001, rollout temperature 1, 16 samples per question, and a maximum length of 32,768 tokens until step 8.2k and 65,536 after it1 (§2.1).

One comparison in the report is about the algorithm itself. Supplementary A.3 trains DeepSeek-Coder-V2-Lite (16B mixture-of-experts, 2.4B active) on MATH with PPO at GAE \(\lambda = 0.95\), PPO at \(\lambda = 1.0\), and GRPO. At the open-source default of 0.95, PPO "performs considerably worse than GRPO"; at 1.0 it is "nearing that of GRPO"1 (A.3, Figure 4). At the last plotted step the three curves sit at about 50.1, 53.5 and 55.4 percent MATH accuracy (read off the plot, accurate to about ±0.3; the same reading as Module 9's Figure 6). The report's argument is about tuning cost and memory, not that a tuned critic loses: "PPO can achieve comparable performance when appropriately tuned", but it demands extra hyperparameter search and a second policy-sized model1. Module 3 sets this against the RLOO and VAPO evidence.

One more thing to carry from the equations. R1 prints its objective (Eq. 1, repeated as Eq. 11) with the ratio \(\pi_\theta(o_i\mid q)/\pi_{\theta_{old}}(o_i\mid q)\) at sequence level and no \(1/|o_i|\) factor, whereas DeepSeekMath's Eq. 3 is per-token with \(1/|o_i|\)4. The report does not say which form ran. Treat it as a source-level conflict (Section 7.3); Module 12 §2.4 works through both readings.

2.2 The schedule, as arithmetic

Read the run-size sentences of §2.1 as a ledger. A training step is "a single policy update operation" (caption of the report's Figure 1)1. Each step uses 32 unique questions with 16 samples each, a batch of 512 outputs. Each rollout generates 8,192 outputs, "randomly split into 16 mini-batches and trained for only a single inner epoch". So 8,192 is 512 questions times 16 samples, one rollout feeds 16 steps, and 10,400 steps are 650 rollouts and 5,324,800 sampled outputs (derived). The reference model is replaced by the latest policy "every 400 steps": 26 multiples of 400 up to 10,400, or every 25 rollouts. The run draws 332,800 questions (10,400 times 32); at the stated 1.6 epochs that implies about 208,000 distinct prompts (derived). Module 10 §2.5 checks that against the 80K reasoning prompts in R1's Table 4, which describe R1; the report does not give R1-Zero's prompt count.

R1-Zero schedule: one rollout of 8,192 outputs (512 questions by 16 samples) is split into 16 minibatches with staleness 0 to 15 optimizer steps; the 10,400-step run has a tick every 400 steps where the reference model is replaced, and a marker at step 8,200 where the maximum length rises from 32,768 to 65,536 tokens One rollout 512 questions × 16 = 8,192 outputs split 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 minibatch 1 to 16, 512 outputs each (32 questions × 16) number in each box = optimizer steps since the outputs were sampled (mean 7.5) one inner epoch · lr 3e-6 · T = 1 · KL 0.001 · only minibatch 0 is on-policy The run 0 2,000 4,000 6,000 8,000 10,400 step 8.2k: max length 32,768 → 65,536 orange ticks: reference model ← policy every 400 steps (26 marks) 650 rollouts × 16 steps RL steps (one step = one policy update) 1.6 epochs
Figure 2. R1-Zero's schedule from the sentences of §2.11. Top: one rollout of 8,192 outputs becomes 16 minibatches; the number in each box is how many optimizer steps passed between sampling and training on it (derived: 0 to 15, mean 7.5). Bottom: the 10,400-step run, one orange tick per multiple of 400 (26 in all), the step-8.2k length increase, and the 1.6-epoch end.

R1-Zero rollout schedule

Outputs per step
Outputs per rollout
Rollouts in the run
Outputs sampled in total
Question draws
Implied distinct prompts
Reference replacements
Mean staleness (optimizer steps)

Defaults are R1-Zero's1. With \(Q\) questions, \(G\) samples, \(K\) minibatches and \(T\) steps: outputs per rollout \(QGK\), rollouts \(T/K\), distinct prompts \(TQ/E\), mean staleness \((K-1)/2\).

Two consequences follow. The schedule is off-policy by construction: minibatch \(j\) is trained after \(j-1\) updates on outputs sampled before the first of them, so staleness runs from 0 to 15 and one minibatch in sixteen is on-policy (derived). That is where the clip begins to act; DeepSeekMath took a single update per rollout, where the clip is inert (Module 12 §1.1). Second, the report does not say whether the random split keeps a question's 16 outputs together. The advantage comes from a question's 16 rewards and is fixed when the group is scored, so the split changes which outputs share an optimizer step, not their advantages (my reading of the equations).

2.3 The rollout side

Supplementary B.1 describes four modules: rollout, inference, rule-based reward and training. Rollout runs vLLM workers holding the actor, with expert parallelism across nodes and "redundant copies of hotspot experts" to balance expert load. The multi-token-prediction head is used "for self-speculative decoding, significantly accelerating the decoding speed and effectively minimizing the completion time for the longest samples"1. The rule-based reward module (code executors, answer matchers, format checkers) needs no GPU memory but "tends to be time-consuming", so it runs asynchronously and overlaps rollout and inference. Training packs sequences into fixed-length chunks with a best-fit strategy, reuses DualPipe, and each module's models are offloaded from VRAM to host memory or disk when its phase ends.

For an inference engineer the omissions matter more than the list. The report gives no tokens per second, no split of GPUs between rollout and training, and no per-phase seconds, so the 68.5 s average per optimizer step on 512 GPUs that Module 1 derives from the 198 hours (derived) is the only timing the text supports. System Design Modules 36 (post-training pipelines), 37 (RL training infrastructure) and 38 (the open-source RL frameworks) cover that side.

2.4 The result, in two versions

Version 2 reports AIME 2024 pass@1 rising "from an initial 15.6% to 77.9%", and 86.7% with self-consistency decoding1 (§2.3). Version 1 reports 15.6% to 71.0%, and 86.7% with majority voting9 (§2.2.4). It gives no step count, only "thousands of RL steps"; its Figure 2, read by eye, ends near step 8,400, while version 2's curve runs to 10,400.

R1-Zero AIME 2024 accuracy, reported endpoints only: start 15.6 percent; end 77.9 percent in version 2 at 10,400 steps and 71.0 percent in version 1 at a step the text does not give; consensus 86.7 percent; human average 37.8 percent 0 20 40 60 80 100 0 2,000 4,000 6,000 8,000 10,400 RL steps AIME 2024 accuracy (%) 8.2k: max length doubled 37.8 human average (v2 Fig. 10) 86.7 consensus (26 of 30 problems) 15.6 start, both versions v2: 77.9 at step 10,400 v1: 71.0 step ≈ 8,400, by eye R1 final: 79.8 v2 chord, start to end: not the measured curve v1 chord, start to end: not the curve
Figure 3. R1-Zero on AIME 2024: the endpoints each version reports, and nothing in between. Start 15.6 is common to both; the end is 77.9 at step 10,400 in v21 and 71.0 in v19, whose text gives no step (about 8,400, read by eye from its Figure 2). The dashed chords join start to end and are not the measured curves, which are v2 Figure 1(a) and v1 Figure 2. The human average, 37.8, is from v2's Figure 10; R1's final 79.8 is from its Table 31.
Table 2. R1-Zero's headline numbers in the two arXiv versions: v1 Table 29 and v2 Table 31. Differences derived.
Metricv1 (22 Jan 2025)v2 (4 Jan 2026)Difference
AIME 2024 pass@1 (%)71.077.9+6.9
AIME 2024 consensus (%)86.7 (cons@64 in Table 2, cons@16 in Fig. 2)86.7 (cons@64 in D.1, cons@16 in Fig. 1)0.0
MATH-500 pass@1 (%)95.995.90.0
GPQA Diamond pass@1 (%)73.375.8+2.5
LiveCodeBench pass@1 (%)50.050.00.0
Codeforces rating144414440

Four of six rows are identical, and the two that moved are AIME and GPQA Diamond. Neither text says why. Both give the same protocol (temperature 0.6, top-p 0.95, "typically between 4 and 64" samples per question); v1's Figure 2 caption says 16 responses per question for the plotted curve, and v2's Supplementary D.1 says k = 64 for AIME19. I cannot tell from the papers whether that, a longer run, or a re-evaluation produced the gap, so cite "77.9% (v2)" or "71.0% (v1)", never bare. Module 10 §6.1 lists what each version does and does not state.

A consensus score is a count. AIME 2024 has 30 problems12 (Fig. 13 caption), so 86.7 is 26 of 30, and v2 attaches the same 86.7 to R1-Zero (§2.3) and to R1 (Supplementary E.4, where R1's pass@64 is 90.0 and its majority vote 86.7)1. Two models can tie on a 30-item test, so the repetition is possible; it also means one problem moves the score by 3.3 points, and the cons@16 versus cons@64 label decides what the number measures. The "aha moment" (§2.3, Table 2) and the length curve are treated in Module 10 §6.3 and Module 13 §5.1.

3. R1: four stages around the same RL core

3.1 Why R1-Zero was not the product

The report says R1-Zero "struggles with challenges like poor readability, and language mixing", which it ties to a base trained mostly on Chinese and English, and that its RL stage "is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering"1 (§1, §3). The cold start was also a product decision. The report says its motivation "is primarily product-driven", because users find first-person reasoning more intuitive, and cautions that such patterns "may elicit unwarranted trust" and reflect "DeepSeek-engineered heuristics" rather than human-like intelligence1 (B.3.2).

3.2 Stage by stage

V3-Base R1-Zero side branch: 101K GPU-h R1-Zero samples seed the cold-start data Compute reported: R1 about 80 h on 512 H800 = 41K GPU-h (stages not split); SFT-data creation 5K 1 · Cold-startSFTthousands oflong-CoT sampleshuman-checkedno reward→ Dev1 2 · ReasoningRLmath, code, STEM,logic promptsrules + languageε = 10→ Dev2 3 · Rejectionsampling + SFT≈600K reasoning≈200K othersampled from Dev2restart: V3-Base→ Dev3 4 · RL for allscenarios1,700 stepsT = 0.7rules + RMsRM: last 400 steps→ R1 5 · Distillation(small models)≈800K samplesSFT onlyQwen2.5, Llama1.5B to 70Bno RL stage
Figure 4. R1's pipeline in five steps, redrawn from Figure 2 of v21. Dev1 to Dev3 are the report's names for intermediate checkpoints. The compute strip is Table 7 of the report: R1-Zero 101K GPU-hours, SFT-data creation 5K, R1 41K with its stages not split.
  1. Cold-start SFT. Thousands of long chains of thought: R1-Zero outputs sampled at temperature 1.0, kept if the answer is correct and the format readable, rewritten by DeepSeek-V3 and checked by humans. Output: Dev1.
  2. Reasoning RL. The R1-Zero recipe started from Dev1, with a language-consistency reward and a printed clip of 10 (the report's "first stage of RL"). The report states no step count. Output: Dev2.
  3. Rejection-sampling SFT. About 600K reasoning and 200K other samples; this SFT restarts from V3-Base, so Dev2 supplies data rather than weights. Output: Dev3.
  4. RL for all scenarios. The report's "second stage of RL": 1,700 steps at temperature 0.7, with reward models active only in the last 400. Output: R1.
  5. Distillation. About 800K samples fine-tune Qwen2.5 and Llama students with SFT only; no RL on the students.
Table 3. The pipeline with the settings the report states1 (§3.2, B.3, B.4.1 to B.4.3, Tables 4 to 6). I number the five steps; the report's "first stage of RL" is step 2 and its "second stage of RL" is step 4. Table 4 of the report does not say which RL stage each prompt set serves; I place the four reasoning sets at step 2 and the general set at step 4 following §3.2. "Not stated" means the report gives no figure.
StepStarts fromDataSignalReported settingsOutput
R1-Zero (side branch)V3-BaseNot itemizedAccuracy + formatlr 3e-6, KL 0.001, T 1, G 16, 10,400 stepsR1-Zero
1 Cold-start SFTV3-BaseThousands of long-CoT samplesNone (SFT)2-3 epochs, lr 5e-5 to 5e-6 cosine, 32,768 context, batch 128Dev1
2 Reasoning RLDev126K math, 17K code (+8K bug fixing in text), 22K STEM, 15K logic promptsRules + language consistencylr 3e-6, KL 0.001, ε = 10, T 1, G 16, 32,768 tokens, reference reset every 400; steps not statedDev2
3 Rejection-sampling SFTV3-BaseAbout 600K reasoning (correct only) + 200K other; 804,745 in Table 5Correctness filter, V3 judge for some promptsAs step 1Dev3
4 RL, all scenariosDev3Reasoning sets + 66K general prompts (the text adds 12,000 harmlessness questions)Rules; helpful and safety RMs; language1,700 steps, T 0.7, RMs in last 400R1
5 DistillationQwen2.5, Llama basesAbout 800KNone (SFT)2-3 epochs, 32,768 context, batch 64, lr 1e-4 (1.5B) to 2e-5 (70B)Six students

Step 1, the cold start. Both versions say "thousands" of cold-start samples19. Lambert's textbook summary of the recipe says "100K+"10 (§3.2.3); the report does not support that figure and I follow the report.

Step 2. The first RL stage, started from Dev1, reuses R1-Zero's sampler (16 outputs per question, 32,768 tokens) with two changes. One is the language-consistency reward, "the proportion of target language words in the CoT" (Eq. 7). The other is the printed clip, "the GRPO clip ratio ε to 10"1 (§3.2.1). The report says "a lower value can lead to the truncation of gradients for a significant number of tokens ... while a higher value may cause instability", and gives neither an ablation nor a step count; v1 says only "until it achieves convergence"9. The nearest sentence is a credit line in Section 7: "Zhibin Gou proposed a large PPO clipping strategy to enhance GRPO performance, demonstrating its significance alongside Zhihong Shao and Junxiao Song"1. That is a lead, not an explanation: it does not say the strategy is ε = 10. Module 12 §2.4 works out what 10 means under token-level and sequence-level readings and marks its own reading as unverified. The language reward has a measured cost: in the ablation on a distilled 7B model, math stays comparable and coding shows "a slight degradation"1 (B.6).

Step 3. The second SFT goes back to DeepSeek-V3-Base: B.4.2 says both SFT stages fine-tune V3-Base for 2 to 3 epochs1, and v1 says the data are used to "retrain the DeepSeek-V3-Base model"9. Reasoning samples come from rejection sampling on the step-2 checkpoint, correct answers only; some prompts added at this stage are judged by DeepSeek-V3 against the ground truth, and chains with mixed languages, long paragraphs or code blocks are filtered out1 (B.3.3). The non-reasoning samples reuse DeepSeek-V3's SFT data, with generated chain of thought for some and none for queries such as "hello". Table 5 of the report splits the 804,745 samples into 395,285 math, 211,129 code, 10,124 STEM, 10,395 logic and 177,812 general, averaging 5,355.3 tokens: about 4.31 billion tokens per epoch, or 8.6 to 12.9 billion over two or three epochs (derived).

Step 4. The second RL stage "retains most of the parameters from the first stage", with the temperature lowered to 0.7 because "higher temperatures in this stage lead to incoherent generation". Its reward is a sum (Eqs. 8 to 10): the rule reward for reasoning, a reward-model score plus a format reward for general data, and the language term. Of the 1,700 steps, general data and preference rewards enter "exclusively in the final 400", steps 1,301 to 1,700, because "more training steps with the model based preference reward signal may lead to reward hacking"1 (§3.2.2). Module 16 §1.2 reads the mixture and Module 19 §3 the hacking curve in Supplementary Figure 6, whose pass@1 values (about 0.346 at the peak, about 0.289 at the end) were read by eye from the PDF there. The helpful reward model trains on 66,000 pairs judged by DeepSeek-V3 and the safety model on 106,000 point-wise labels (Module 4).

3.3 What each stage bought

Table 4. The report's per-checkpoint results1 (Table 3 of v2). Pass@1 or the metric named; the last column is R1 minus R1-Zero (derived). The best of the five checkpoints in each row is shaded. Bold in the report marks statistical significance (t-test, p < 0.01), not the best value.
BenchmarkR1-ZeroDev1Dev2Dev3R1R1 − Zero
AIME 202477.959.074.078.179.8+1.9
MATH-50095.994.295.995.497.3+1.4
CNMO 202488.158.073.977.378.8-9.3
GPQA Diamond75.866.170.771.271.5-4.3
LiveCodeBench50.057.563.564.665.9+15.9
Codeforces rating14441534168717462029+585
Aider-Polyglot12.26.725.644.853.3+41.1
IF-Eval (strict)46.671.772.078.183.3+36.7
AlpacaEval 2.0 (LC)24.750.155.862.187.6+62.9
ArenaHard53.677.073.275.692.3+38.7
MMLU-Pro68.974.183.883.184.0+15.1
SimpleQA30.317.828.224.930.1-0.2
Six benchmarks across the five checkpoints R1-Zero, Dev1, Dev2, Dev3 and R1: AIME and GPQA are flat or lower at the end while code, instruction following and chat win-rate rise AIME 2024 pass@1 77.9 Zero 59 D1 74 D2 78.1 D3 79.8 R1 GPQA Diamond pass@1 75.8 Zero 66.1 D1 70.7 D2 71.2 D3 71.5 R1 LiveCodeBench pass@1 50 Zero 57.5 D1 63.5 D2 64.6 D3 65.9 R1 Aider-Polyglot 12.2 Zero 6.7 D1 25.6 D2 44.8 D3 53.3 R1 IF-Eval prompt strict 46.6 Zero 71.7 D1 72 D2 78.1 D3 83.3 R1 AlpacaEval 2.0 LC win 24.7 Zero 50.1 D1 55.8 D2 62.1 D3 87.6 R1
Figure 5. Six rows of Table 4 as paths through the five checkpoints (Zero = R1-Zero, D1 to D3 = Dev1 to Dev3); each panel has its own y axis1. Reasoning scores (AIME, GPQA) dip at Dev1 and end near where R1-Zero started; LiveCodeBench, IF-Eval and AlpacaEval rise at every step, and Aider dips at Dev1 before rising.

The cold start cost reasoning before RL repaired it. At Dev1, AIME fell from 77.9 to 59.0 and CNMO from 88.1 to 58.0, and the report attributes the Dev1 degradation to "the limited size of the cold-start dataset"; step 2 then recovered 15.0 AIME points, to 74.01 (§4). On LiveCodeBench the gains come from medium and hard problems: the hard-problem pass rate runs 17.09, 23.21, 30.36, 33.16, 34.44 across the five checkpoints, while easy problems sit at 98 to 1001 (Table 14). The last stage mostly moves chat metrics: from Dev3 to R1, AlpacaEval 2.0 rises 25.5 points and ArenaHard 16.7. The report writes "25%" and "17%", which are points on the win-rate scale (derived).

For serving cost, Supplementary E.3 and E.4 report that on 366 problems from 93 competitions held in 2024, R1 averages 8,793 thinking tokens per problem at 61.8% pass@1, fewer than 7,000 on simple problems and more than 18,000 on the hardest, against 711 output tokens at 24.7% for GPT-4o-05131.

4. Compute and cost

Supplementary B.4.4 gives hardware and durations, and Table 7 prices them at "$2 per GPU hour": R1-Zero on 64×8 H800 GPUs for "approximately 198 hours"; R1 on "the same 64*8 H800 GPUs, completing the process in about 4 days, or roughly 80 hours"; and 5K GPU-hours to create the SFT datasets1. Module 1 §4.1 derives the same ledger. Table 5 repeats it so the arithmetic sits beside the case.

Table 5. R1's cost ledger: Table 7 of the report1 against the arithmetic check (derived; 64 × 8 = 512 GPUs, $2 per H800 GPU-hour as the report assumes).
LineReported GPU-hours / USDCheck: 512 GPUs × hoursVerdict
R1-Zero, 198 h101K / $202K101,376 / $202,752Agrees
SFT-data creation5K / $10KNo hours or GPU count givenNot checkable
R1, "roughly 80 hours"41K / $82K40,960 / $81,920Agrees
R1, "about 4 days" (96 h)not tabulated49,152 / $98,304Disagrees by 8,192 GPU-hours
Total147K / $294K147,336 / $294,672 (155,528 / $311,056 with 96 h)Agrees to rounding at 80 h

Table 7 and "roughly 80 hours" agree with each other, and "4 days" is the outlier; treat 80 hours as the figure of record, as Module 1 does. The ledger leaves out what the report mentions separately: A100 pilots on a 30B model that gave the confidence to "scale up to 660B R1-Zero and R1"1 (B.4.4; A.1 and the report's Table 8 say 671B), any hyperparameter search, and failed runs. It does not say which of the four training steps the 80 hours covers, and step 2 has no stated step count, so the 80 hours cannot be split by stage.

Lambert puts R1's share at "only about 5%" of V3's compute: 147K H800 GPU-hours against 2.8M for pretraining the base10 (p. 14), which is 147/2,800 = 5.25% (derived). Module 1 §4.4 uses V3's own pretraining figure instead and gets 3.75% for R1-Zero alone. The denominators differ, so do not mix numbers across the two tables. One all-in rate follows: R1-Zero's 5,324,800 sampled outputs over 101,376 GPU-hours is about 52.5 outputs per GPU-hour, and at Module 1's assumed mean of about 6,000 tokens per completion about 88 sampled tokens per GPU-second, rollout and training combined (derived).

5. Distillation versus RL on a small model

Supplementary F trains one base two ways. Qwen2.5-32B-Base gets "large-scale RL training" on math, code and STEM "for over 10K steps", giving Qwen2.5-32B-Zero (named DeepSeek-R1-Zero-Qwen-32B in v1)19. Its settings (B.4.1) are R1-Zero's with a learning rate of 2e-6: KL 0.001, temperature 1, 16 samples of up to 32,768 tokens, 32 questions and 512 outputs per step, reference replaced every 400 steps. The other arm is SFT of the same base on the 800K R1-generated samples, DeepSeek-R1-Distill-Qwen-32B.

Table 6. RL against distillation on one 32B base, Table 16 of the report1; the numbers are identical in v1 (Table 6)9. Last row derived. Neither arm has a stated GPU-hour cost.
ModelAIME 2024 pass@1AIME cons@64MATH-500GPQA DiamondLiveCodeBench
QwQ-32B-Preview50.060.090.654.541.9
Qwen2.5-32B-Zero (RL, over 10K steps)47.060.091.655.040.2
R1-Distill-Qwen-32B (SFT on R1 samples)72.683.394.362.157.2
Distilled minus RL (points)+25.6+23.3+2.7+7.1+17.0
Qwen2.5-32B after more than 10,000 RL steps matches QwQ-32B-Preview, while DeepSeek-R1-Distill-Qwen-32B is higher on all five benchmarks, by 25.6 points on AIME pass@1 0 20 40 60 80 100 score (%), model-dependent metric 50 47 72.6 AIME pass@1 60 60 83.3 AIME cons@64 90.6 91.6 94.3 MATH-500 54.5 55 62.1 GPQA Diamond 41.9 40.2 57.2 LiveCodeBench QwQ-32B-Preview Qwen2.5-32B-Zero (RL, > 10K steps) R1-Distill-Qwen-32B (SFT on R1 samples)
Figure 6. Table 6 as bars1. RL on the 32B base lands where QwQ-32B-Preview does, as the report says; the distilled student is higher on every benchmark, most on AIME pass@1 and LiveCodeBench.

The report draws two conclusions: distilling stronger models "yields excellent results, whereas smaller models relying on the large-scale RL ... require enormous computational power and may not even achieve the performance of distillation", and, second, "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning"1 (F.1). The ladder of students shows how far the first goes: AIME 2024 pass@1 of 28.9 for the 1.5B Qwen student (GPT-4o-0513 scores 9.3, Claude-3.5-Sonnet-1022 16.0), 55.5 at 7B, 69.7 at 14B, 72.6 at 32B, and 50.4 and 70.0 for the Llama 8B and 70B students1 (Table 15).

The comparison controls less than its table suggests. The RL arm samples at least \(10{,}000 \times 512 = 5.12\) million completions of up to 32,768 tokens (derived), and the SFT arm reads 804,745 sequences per epoch for two or three epochs, but the distilled student inherits its teacher: 147K GPU-hours of R1 and the generation of the samples. "Economical" is the student's marginal cost. The arms also see different data, since the RL prompts are math, code and STEM while the SFT set includes 177,812 general samples (derived from Table 5). Qwen3 later ran the comparison with GPU-hours at one starting checkpoint: on Qwen3-8B, RL reached 67.6 on AIME'24 with 17,920 GPU-hours and left pass@64 unchanged, while on-policy distillation reached 74.4 with 1,800 GPU-hours13 (§4.7, Table 21). Module 21 sets those rows beside R1's.

A second small-model result sits in Table 17: Qwen2-Math-7B-Zero, trained with "approximately 10,000 policy gradient update steps" on a base released in August 2024, "prior to the launch of the first reasoning model, OpenAI-o1 (September 2024), to ensure the base model was not exposed to any reasoning trajectory data", reaches 22.3% on AIME 2024 and 18.1% on AIME 2025, against 7.9% and 4.6% for Qwen2-Math-7B-Instruct1.

6. What failed, as the report states it

Table 7. The unsuccessful attempts of Supplementary G.1 and G.21; the same text appears in v1 §4.29. The report frames them as failures "in the early stages", not as proofs that the methods cannot work.
AttemptWhat was triedReasons givenConclusion given
Process reward modelModel-based reward on reasoning stepsA fine-grained step is hard to define; step correctness is hard to judge (model annotation unsatisfactory, manual annotation does not scale); a model-based PRM "inevitably leads to reward hacking" and retraining it adds cost and complexityGood at reranking top-N or guiding search; advantage "limited compared to the additional computational overhead" in large-scale RL
Monte Carlo tree searchAlphaGo-style search with a pre-trained value model; answers split into tagged steps; actor and value trained on search outputsThe token-level search space is "exponentially larger" than chess, and a node-extension limit traps the model in local optima; the value model steers generation and a fine-grained one is hard to trainHelps at inference with a pre-trained value model; iterative improvement by self-search "remains a significant challenge"
Small basesRL on a 7B dense and a 16B mixture-of-experts baseNo meaningful AIME gains; repetition as responses lengthenedMoved to 32B dense, 230B and 671B MoE; effectiveness "highly dependent on the underlying model capacity"

Later evidence points the same way without settling it. Kimi k1.5 set out to avoid "Monte Carlo tree search, value functions, and process reward models"11. Llama 3 still uses outcome and stepwise reward models, but to filter SFT reasoning traces14 (§4.3.3), which is the reranking use R1 concedes. Lightman et al. found that process supervision "significantly outperforms outcome supervision" for training reward models on MATH15. The three are compatible: process labels rank and filter well, and R1's finding concerns using them as the RL reward. Module 4 §4.5 covers the mechanics.

G.1 widens the verifier in one direction: besides rules, an LLM that checks an answer against a ground truth "demonstrates particular effectiveness for tasks with well-defined, concise answers" but "exhibits limited generalizability to more complex tasks, including open-ended generation and long-form writing"1. That is the same boundary R1 draws in its limitations (§6): for tasks "that cannot be effectively evaluated by a reliable reward model, scaling up pure RL methods remains an open challenge", so it used human-annotated SFT data there and "only conduct[ed] RL for hundreds of steps"1.

7. The report across versions, and where it disagrees with itself

7.1 From v1 to v2

Version 1 is 22 pages and version 2 is 8691. The additions, by location: hyperparameters and run length (§2.1, §3.2); the RL data recipe and SFT statistics (Tables 4 and 5, B.3); the PPO-versus-GRPO study (A.3) and the RL framework (B.1); the cost table (Table 7); the reward-hacking and language-consistency ablations (B.5, B.6); reasoning-behavior counts over training (C.1, C.2); Qwen2-Math-7B-Zero (Table 17); a safety report (D.3); and results on problems released after training, 75% pass@1 on AIME 2025 against o1's 80% (E.2). Two headline numbers changed (Table 2), and the run behind the 32B comparison was renamed from DeepSeek-R1-Zero-Qwen-32B to Qwen2.5-32B-Zero. One sentence is gone: v1's contribution bullet calling R1-Zero "the first open research to validate that reasoning capabilities of LLMs can be incentivized purely through RL" does not appear in v2 (I searched its text).

7.2 The Nature copy

Nature published the work in 2025 as volume 645, issue 8081, pages 633 to 638; the reference lists of DeepSeek's own V3.2 and V4 reports cite it that way56. I did not fetch nature.com, which redirected to a cookie page when this course's source map was built. I read the NIH PMC copy of the article through a summarizing fetch tool16, so what follows is that tool's extraction and not my own reading of the page. The peer-review file linked from the PMC page returned only a download placeholder; I did not read it and did not try another route, so I cannot say what the review changed.

What the extraction shows, as differences from arXiv v2. The dates are received 14 February 2025, accepted 17 July 2025, cited as 17 September 2025. The main text is shorter and sends the reader to the Supplementary Information by section number (cost to 2.4.4, pass@1 and cons@16 to 4.1, GRPO versus PPO to 1.3, data statistics to 2.3), where arXiv v2 uses B.4.4, D.1, A.3 and B.3, so v2 is not a verbatim copy of the Nature layout. The figures I could compare match v2: 15.6% to 77.9%, 86.7%, a first-stage clip of 10, and a Figure 1 legend that also says cons@16. On release, the Data availability section points to the DeepSeek-R1 repository for the rejection-sampling and RL-prompt samples, with Zenodo DOI 10.5281/zenodo.15753193, and says weights are "available under an MIT license", while arXiv v2's Appendix I reads "We also release SFT and RL data to the public at xxx"1, a placeholder. The extraction's author-contributions paragraph is generic and does not contain the sentence crediting "a large PPO clipping strategy", which arXiv v2 §7 does. Treat a claim that rests on one document as belonging to that document, and name the version.

7.3 A ledger of conflicts

Table 8. Places where the report, or the report and a summary of it, give two answers. Sources: v21, v19, DeepSeekMath4, Lambert10; arithmetic derived.
ItemOne place saysAnother saysHandling
R1 training time"about 4 days, or roughly 80 hours" (B.4.4)512 GPUs × 80 h = 40,960, Table 7's "41K"; 4 days is 96 hUse 80 h; Module 1 §4.1
GRPO ratioEq. 1 and Eq. 11: sequence-level ratio, no \(1/|o_i|\)DeepSeekMath Eq. 3: per-token ratio with \(1/|o_i|\)Report does not say which ran; Module 12 §2.4
Clip value"ε to 10" in step 2 (§3.2.1); R1-Zero's ε not stated (§2.1)PPO's suggested 0.23Printed once, unexplained; do not generalize
R1-Zero AIME pass@171.0 (v1)77.9 (v2)Name the version; Module 10 §6.1
The 86.7 labelcons@64 (v1 Table 2 and text; v2 D.1)cons@16 (v1 Fig. 2 and v2 Fig. 1 legends)State both; 86.7 is 26 of 30
Code RL promptsTable 4: 17KB.3.1: 17K competition problems plus 8K bug fixesSay 17K or 25K, not one silently
SFT splitAbout 600K reasoning, 200K other (B.3.3)Table 5: 626,933 in four domain rows, 177,812 generalDomain rows are not the reasoning split
The 800KB.3.3: rejection sampling from the first-RL-stage checkpointAppendix F: "samples generated with DeepSeek-R1"Unclear whether distillation data were regenerated
Model size"660B R1-Zero and R1" (B.4.4)671B total (A.1; the report's Table 8)Use 671B
Cold-start size"thousands" (v1 §2.3.1; v2 §3)"100K+" (Lambert §3.2.3)Follow the report
Small bases7B dense and 16B MoE "consistently failed" on AIME (G.1)Qwen2-Math-7B-Zero reaches 22.3% (Table 17)Different bases; unreconciled

None of these rows changes the story of Sections 2 to 5. Three change what you may safely quote: the clip, the sequence-versus-token ratio, and the unlabeled 86.7.

8. After R1: critiques, and what later recipes kept, changed or dropped

8.1 Four checks on R1's claims

The papers below differ from R1 in base model, data and scale, so each tests a claim or fills a gap and none is a replication. Dr. GRPO hosted DeepSeek-V3-Base and found that it "already exhibit[s] 'Aha moment'", that Qwen2.5 base models "demonstrate strong reasoning capabilities even without prompt templates", and that GRPO's two normalizers bias the update: dividing by response length favors short correct and long wrong responses, and dividing by the group's reward standard deviation up-weights questions that are too easy or too hard17 (abstract, §3.1). Its minimalist recipe reached 43.3% on AIME 2024 from Qwen2.5-Math-7B17. R1's own appendix is closer to that reading than the "aha" framing is: pretraining exposure to reasoning traces "equips the model with the capability to generate plausible solution candidates, from which reinforcement learning can effectively identify and optimize high-quality outputs"1 (A.1; Module 10 §6.3).

Spurious Rewards found that on Qwen2.5-Math-7B, randomly assigned rewards raised MATH-500 by 21.4 points against 29.1 for ground-truth rewards, traced it to a clipping bias that amplifies behaviors with high prior probability, and reported that such rewards "often fail to produce gains for other model families, such as Llama3 or OLMo2"12. It did not test DeepSeek-V3-Base, so it says nothing direct about R1-Zero. It does make an R1-Zero-style gain on a Qwen math base weak evidence about reward quality, which bears on replications built that way, including the 43.3% above. R1 itself flags contamination: n-gram decontamination "cannot prevent the paraphrase of testset", so benchmarks "released before 2024 may suffer from contamination issues"1 (D.1); its answer is the AIME 2025 result of Section 7.1. Olmo 3 argues that open-weight bases with undisclosed mid-training data can contaminate RLVR benchmarks, and releases an RL-Zero run from a fully open base18 (Module 31).

Yue et al. find RLVR-trained models ahead of their base models at small k and behind at large k, with the RLVR reasoning paths already inside the base model's sampling distribution, whereas distillation "can introduce new reasoning patterns from the teacher"19. DeepSeekMath saw the same split between Maj@K and Pass@K in 20244. R1's E.4 gives pass@64 of 90.0 against pass@1 of 79.8 for R1 itself, which is not a base-model comparison1. Module 20 weighs this evidence.

DAPO ran GRPO on Qwen2.5-32B and reached "only 30 points on AIME — a performance significantly below DeepSeek's RL (47 points)", adding that "critical training details may have been omitted in the R1 paper"20. Version 2's added hyperparameters answer part of that. The step count of the first RL stage and the reason for ε = 10 remain absent.

8.2 Kept, changed, dropped

Table 9. R1's main choices against what the later reports read for this course state. "Kept" means a later report states the same choice. Bases, data and scale differ, so a row records a choice, not a measured improvement. Module 30 reads V3.2 and V4 in full; Modules 29 and 31 read Qwen3 and Olmo 3.
R1 choice1StatusLater reports
Group-relative advantage, no criticKeptV3.2 and V4 run GRPO56; Qwen3's reasoning RL uses it13; Olmo 3's OlmoRL is GRPO plus changes18
Advantage \((r-\mathrm{mean})/\mathrm{std}\)ChangedV3.2 uses \(A = R - \mathrm{mean}(R)\)5; Olmo 3 drops std18; Dr. GRPO drops std and \(1/|o_i|\)17
KL 0.001 in the loss; reference reset every 400 stepsChangedV3.2: weak or no KL for math, estimator corrected with the importance ratio5; DAPO and Olmo 3 remove it2018; ProRL keeps it, with hard resets, from a distilled start21
Clip ε = 10, printedChangedDAPO: 0.2 below, 0.28 above20; Olmo 3: 0.2 and 0.27218
Sequence-level objective, as printedChangedDAPO uses a token-level loss20; Olmo 3 normalizes over all tokens in the batch18
Rule-based reasoning reward, language reward, no neural RMKeptQwen3 keeps rules for their "high precision, preventing issues like reward hacking"13; V3.2 uses rule-based outcome reward, a length penalty and a language-consistency reward5
Scalar helpful and safety RMs for general dataChangedV3.2: a generative reward model with per-prompt rubrics5; V4: no scalar reward models, the actor doubles as the judge6. A change of reward-model type for general tasks, not a reversal of R1's rule for reasoning
SFT, RL, SFT, mixed RLChangedV3.2 merges reasoning, agent and alignment RL into one stage5; V4 replaces mixed RL with on-policy distillation6; Qwen3 keeps four stages with thinking-mode fusion13
Distilling R1's 800K into small modelsKept, extendedQwen3's strong-to-weak distillation, off-policy then on-policy, at about 1/10 the GPU-hours of its four-stage pipeline13
No PRM, no MCTS in RLKeptKimi k1.511; Llama 3 uses stepwise RMs only to filter SFT traces14
No treatment of sampler-trainer mismatchAddedV3.2: Keep Routing, Keep Sampling Mask, off-policy sequence masking5 (Module 14)

The last row rests on absence: R1 v2 contains no discussion of routing, mismatch or numerical discrepancy (I searched its text). The pattern in the rest is that the stage structure and the objective changed most, while the verifier-first reward for reasoning and the rejection of step-level RL rewards survived intact.

Where experts disagree

R1 was a method claim (verifiers, GRPO, pure RL from a base) and a set of artifacts (a pipeline, a cost, a failure list). Three disputes it started or sharpened are still open, and each can be read against a table in the report.

What did RL do in R1-Zero?

RL elicited new reasoning behavior

R1's own framing: "emergent development of advanced reasoning patterns", the "aha moment"1

R1-Zero starts from a base with no SFT, and the supplement counts behaviors over training: reflective words rise 5- to 7-fold, and "wait" is nearly absent early, occasional between steps 4,000 and 7,000, and spikes after step 8,000 (C.2).

  • A measured trajectory: AIME 15.6 to 77.9 (v2) with response length growing alongside.
  • No human reasoning traces in the loop.
  • No base-model control for the counted behaviors.
  • The same report says pretraining supplies "plausible solution candidates" and that checkpoints "inherently possess substantial potential"1.

RL selects and sharpens what the base can already produce

Yue et al.19; DeepSeekMath's Maj@K versus Pass@K4; Qwen3's pass@6413; Dr. GRPO on V3-Base17

RLVR models win at small k and lose at large k, their reasoning paths lie inside the base distribution, and Dr. GRPO finds the "aha" behavior in V3-Base itself.

  • Independent groups, different bases and methods, one direction.
  • Fits R1's own appendix on candidates.
  • ProRL argues such studies used math-heavy settings and stopped RL after "no more than hundreds of steps"21.
  • Large-k comparisons on a 30-problem benchmark are coarse.

On some bases the gain is prior amplification

Spurious Rewards12

Random rewards give +21.4 MATH-500 points on Qwen2.5-Math-7B against +29.1 for ground truth; GRPO's clip amplifies high-prior behaviors; other families mostly do not move.

  • A direct control: the reward carries no task signal.
  • Explains why a Qwen math base can mislead.
  • Does not test V3-Base, so it bears on R1-Zero only by analogy.

Prolonged RL expands the boundary

ProRL21; a two-stage reconciliation22

With KL control, reference resets and diverse tasks, over 2,000 steps on a 1.5B model found strategies "inaccessible to base models, even under extensive sampling". The reconciliation proposes sharpening first, exploration later.

  • Addresses the "too short" objection head-on.
  • Starts from a distilled checkpoint and changes several things at once, so the cause is not isolated.
  • The reconciliation is the authors' own assessment.
Assessment

The first two are less opposed than they look: R1's appendix already describes selection among candidates, so the live question is whether more compute adds anything beyond selection. Quote R1's gains as pass@1 gains and do not read "emergent" as "absent in the base". What would settle it: pass@k at large k on one contamination-free base, matched data, runs as long as R1-Zero's 10,400 steps, and a base-model control for the counted behaviors. Module 20 holds the full ledger.

Process or outcome supervision for the RL reward?

Outcome and rules only

R1 Supplementary G.21; Kimi k1.511

Step boundaries, step labels and hackability make a model-based PRM a poor RL reward; a verifier is cheaper to trust.

  • Simplest pipeline; scaled to 10,400 steps without a learned scorer.
  • Two independent reports on the same day reached the same design.
  • G.2 gives no numbers or ablation, and frames the attempts as early failures.

Step-level labels help where they exist

Lightman et al.15; Llama 3's stepwise reward models14

Process supervision "significantly outperforms outcome supervision" for MATH reward models; Llama 3 filters SFT traces with stepwise models.

  • A measured ranking advantage.
  • Used to rank or filter, not as the reward of a long-chain RL run.
  • Needs step labels, which cost human effort.
Assessment

Both hold. R1 concedes that PRMs rerank well, and the two positions describe different uses. None of the reports read for this module trains a long-chain RL run on a learned PRM reward, so the question is untested rather than answered; a matched-compute run with a PRM reward against an outcome reward would settle it.

RL or distillation for a strong small reasoning model?

Distill from a stronger teacher

R1 Supplementary F.11; Qwen313

On one 32B base, SFT on R1 samples beats 10K+ RL steps by 25.6 AIME points; Qwen3-8B reaches 74.4 against 67.6 at 1/10 of the GPU-hours.

  • Two reports, same direction.
  • Cheap for the student.
  • Capped by the teacher, whose cost is not counted.
  • R1's arms differ in data as well as method.

RL remains the way past the teacher

R1's second conclusion1; ProRL21

"Advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning"; ProRL reports further gains over a distilled 1.5B start.

  • Only route to a model stronger than every available teacher.
  • No matched-compute comparison exists in the sources read.
Assessment

When a stronger teacher exists and the student is small, distillation wins on both quality and cost in the two measured cases, and R1 itself says so. RL is the only route past the teacher, but I found no source that compares it with distillation at matched compute from the same start with the teacher's cost counted; that comparison would settle it.

What is changing

R1's own list of open problems has moved. For tasks without a reliable verifier, where the report says scaling pure RL "remains an open challenge"1, 2026 recipes train rubric-based generative judges inside the RL loop (in V4 the actor itself is the judge), and V4 replaces mixed RL with on-policy distillation of specialists6, a step Lambert says labs are "far from unanimous" about10 (§3.2.4). Three things in R1's own record stay open: the step count and the reason for ε = 10 in the first RL stage, which training steps the 80 hours cover, and what the Nature review changed, which I could not read. I cite V4 without a date because the source map records an unexplained mismatch between its arXiv identifier and the date the arXiv API reports. V3.2 and V4 are the reports in this module most likely to be superseded.

References

  1. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v2, 4 Jan 2026; the source map lists v2 as the expanded version of the Nature article (see Section 7.2). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: §1, §2.1 (GRPO credit, Eqs. 1 to 3, run size), Fig. 1, §2.2 (Eq. 4), §2.3 (15.6% to 77.9%, 86.7%), §3 and §3.2 (pipeline, Eqs. 7 to 10, clip 10), §6, §7 (contributions); Table 3; A.1, A.3 (Figs. 3 and 4), B.1, B.3 and Tables 4 and 5, B.4 and Tables 6 and 7, B.5, B.6, C.2, D.1, E.2 to E.5 and Table 14, F.1 and Tables 15 to 17, G.1, G.2, Appendix I.
  2. paperSchulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv:1502.05477 v5, 20 Apr 2017. https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Used for: the 2015 date and the KL-constrained trust-region update.
  3. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv:1707.06347 v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the clipped surrogate and the suggested ε = 0.2 (§3, Eq. 7).
  4. paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the introduction of GRPO (abstract, §1, §4.1.1), the per-token objective with \(1/|o_i|\) (Eq. 3), Maj@K versus Pass@K (§5.2.2).
  5. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: §3 (mixed RL, reward types, rubric-based generative reward model), §3.1 (advantage without std, corrected KL estimator, Keep Routing, Keep Sampling Mask, sequence masking), and the reference-list entry for R1 (Nature, 645(8081):633-638, 2025).
  6. paperDeepSeek-AI (Xu, A., Lin, B., Xue, B., et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: §5.1 (pipeline mirrors V3.2 with mixed RL replaced by on-policy distillation; GRPO with hyperparameters aligned to prior work), §5.1.1 (generative reward model, no scalar reward models), and the reference-list entry for R1 (Nat., 645(8081):633-638, 2025).
  7. paperOuyang, L., Wu, J., Jiang, X., Almeida, D., et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv:2203.02155 v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: the learned-reward-model plus PPO pipeline in Figure 1 (§3.5).
  8. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124 v5, 14 Apr 2025; v1 22 Nov 2024. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the definition and date of RLVR (§1, §6).
  9. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (version 1). arXiv:2501.12948 v1, 22 Jan 2025. Same arXiv identifier as reference 1, read here as a separate version. https://arxiv.org/pdf/2501.12948v1. Accessed 2026-10-26. Used for: §1.1 ("first open research" sentence), §2.2.4, Table 2 and Figure 2 (71.0%, 86.7%, cons@64 and cons@16 labels), §2.3 (four-stage pipeline, "convergence", retraining from V3-Base), §3 evaluation setup, §4.1 and Table 6, §4.2.
  10. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501 v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: p. 14 (RLVR coined in Tulu 3, R1 "famous for popularizing RLVR", 147K versus 2.8M GPU-hours), §3.2.3 (R1 recipe summary), §3.2.4 (MOPD, "far from unanimous"), §7.3.1 (earlier verifiable-domain work), §7.3.2 Table 4 (release dates), bibliography (o1-preview, Sept. 2024).
  11. paperKimi Team (Du, A., Gao, B., Xing, B., et al.; Moonshot AI). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599 v4, 3 Jun 2025; v1 22 Jan 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: the 22 January 2025 first-version date; abstract (no MCTS, value functions or process reward models; o1-matching results).
  12. paperShao, R., Li, S. S., Xin, R., Geng, S., et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947 v2, 25 Feb 2026. https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: abstract and §1 (21.4 versus 29.1 points; model dependence), Figure 13 caption (AIME has 30 questions).
  13. paperYang, A., Li, A., Yang, B., Zhang, B., et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388 v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: §4 (four-stage pipeline, strong-to-weak distillation at about 1/10 of the GPU-hours), §4.2 (GRPO reasoning RL), §4.4 (rule-based rewards), §4.7 and Table 21 (RL versus on-policy distillation on Qwen3-8B).
  14. paperLlama Team, AI @ Meta (Grattafiori, A., Dubey, A., Jauhri, A., et al.). The Llama 3 Herd of Models. arXiv:2407.21783 v3, 23 Nov 2024. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: outcome and stepwise reward models used to filter SFT reasoning traces (§4.3.3).
  15. paperLightman, H., Kosaraju, V., Burda, Y., Edwards, H., et al. Let's Verify Step by Step. ICLR 2024; arXiv:2305.20050 v1, 31 May 2023. https://arxiv.org/abs/2305.20050. Accessed 2026-10-26. Used for: abstract (process supervision outperforms outcome supervision on MATH).
  16. paperDeepSeek-AI (Guo, D., et al.). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645(8081):633-638, 2025; NIH PubMed Central copy PMC12443585, CC BY 4.0. https://pmc.ncbi.nlm.nih.gov/articles/PMC12443585/. Accessed 2026-10-27. Used for: received and accepted dates, Supplementary section pointers, Figure 1 legend, data and code availability, author-contributions paragraph. Read through a summarizing fetch tool (its extraction, not the page text); the Supplementary Information and peer-review PDFs were not read. Not in the researcher's source map.
  17. paperLiu, Z., Chen, C., Li, W., Qi, P., et al. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783 v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: abstract (aha moment in V3-Base, templates), §3.1 (length and difficulty biases), §3.2 and Table 4 (minimalist recipe, 43.3% from Qwen2.5-Math-7B).
  18. paperTeam Olmo (Ettinger, A., Bertsch, A., et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961 v2, 14 Apr 2026; v1 15 Dec 2025. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL (§4.4.1: GRPO plus token-level loss, no KL, clip-higher, no std normalization; hyperparameters 0.2 and 0.272), the RL-Zero contamination argument.
  19. paperYue, Y., Chen, Z., Lu, R., Zhao, A., et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837 v5, 24 Nov 2025. https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Used for: abstract (pass@k crossover, paths inside the base distribution, distillation introduces new patterns).
  20. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476 v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: §1 (30 versus 47 AIME points; "details may have been omitted in the R1 paper"; 50 points in 50% of the steps), §2.3 (KL removed), §3.1 (clip-higher 0.28), §3.3 (token-level loss).
  21. paperLiu, M., Diao, S., Lu, X., Hu, J., et al. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864 v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: §1 (methodological-constraints argument, "hundreds of steps", "inaccessible to base models"), §2 (KL control with reference resets, over 2,000 steps).
  22. paperYao, X., Yu, L., Hu, X., Teng, F., et al. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028 v1, 5 Oct 2025. https://arxiv.org/abs/2510.04028. Accessed 2026-10-26. Used for: abstract (exploitation stage then exploration stage; the authors' own reconciliation).