Reinforcement Learning for LLM Post-Training · Module 26
Case Study: Llama 3 and Tulu 3 (2024): DPO-First Post-Training, the PPO Comparison, and the Birth of RLVR
Two 2024 reports built on the same Llama 3.1 base models both chose DPO over PPO for preference tuning, and differed on almost everything else. Meta ran six rounds of human-annotated data, rejection sampling and DPO; Ai2 published its data and added a last PPO stage against programmatic checkers, which it named RLVR. This module reads each recipe stage by stage, with data counts, hyperparameters, GPU-hours and measured stage effects. Afterwards you can say what each recipe optimized, what its numbers do and do not show, and which of its choices the 2025 GRPO-style recipes kept or replaced.
1. Two reports, one model family, two objectives
1.1 Same base models, different harnesses
Llama 3 is Meta's report on its 8B, 70B and 405B dense models (arXiv v1 31 Jul 2024, v3 23 Nov 2024), and says that all of its results are for the Llama 3.1 models1 (§1). Tulu 3 is Ai2's report (v1 22 Nov 2024, v5 14 Apr 2025) on post-training those same three Llama 3.1 base models, with Llama 3.1 Instruct among its main baselines2 (§1, §2.4). That shared base is what makes the pair useful as a case study: two recipes, one starting point.
It does not make their numbers comparable. Table 1 lists benchmarks on which both reports score Llama 3.1 405B Instruct, the same released model, in their own harnesses. The gap is 1.4 points on GSM8K, where the nominal setting matches, and 7.2 and 6.9 points on MATH and HumanEval, where the settings differ and the direction flips (derived). Everything below therefore compares the recipes by stage structure, cost and within-report deltas, and uses a cross-report number only where it says so.
| Benchmark | Meta's setting | Meta (%) | Ai2's setting | Ai2 (%) |
|---|---|---|---|---|
| GSM8K | 8-shot, CoT | 96.8 | 8-shot, CoT | 95.4 |
| MATH | 0-shot, CoT | 73.8 | 4-shot, CoT, flex match | 66.6 |
| IFEval | mean of prompt and instruction level, strict and loose | 88.6 | prompt level, loose | 88.4 |
| HumanEval | 0-shot | 89.0 | pass@10 | 95.9 |
| MMLU | 5-shot | 87.3 | 5-shot, CoT | 88.0 |
1.2 What each recipe was steering by
Under the lever it calls "managing complexity", Llama 3 chose a "relatively simple post-training procedure based on supervised finetuning (SFT), rejection sampling (RS), and direct preference optimization (DPO)" over "more complex reinforcement learning algorithms" that "tend to be less stable and harder to scale"1 (§1). Human comparison is the steering signal: annotators' pairs train the reward model, the reward model selects the rejection-sampled SFT data, and the newest pairs are the DPO data (Section 2).
Tulu 3 steers by a measurement instrument. It lists the core skills it wants, builds a development suite of 12 evaluations and a held-out suite of 7 (counted from its Tables 3, 6 and 33), and states that it "did not examine scores on our unseen set when developing our models", so that the gap between the two shows how far its choices overfit2 (§2.2). Data mixes, algorithms and checkpoints are all selected on the development average. Neither recipe optimizes one reward; both optimize a data mix and a stage order against a selection signal. Table 2 sets them side by side; mechanism details are in Modules 4, 5, 7 and 10.
| Item | Llama 3 (Meta) | Tulu 3 (Ai2) |
|---|---|---|
| Stages | reward model, rejection sampling, SFT, DPO; six rounds | SFT, DPO, RLVR; one pass |
| Preference signal | human annotators: two responses from two models, four strength levels, optional edit | GPT-4o-2024-08-06 rates four responses 1 to 5 on four aspects; best versus a random lower-mean response |
| SFT settings | lr 1e-5, 8.5K to 9K steps | lr 5e-6 (8B) and 2e-6 (70B), batch 128, 2 epochs, sum loss |
| DPO settings | β 0.1, lr 1e-5, NLL term 0.2, formatting tokens masked; no length normalization stated | length-normalized, β 5, lr 5e-7 (8B) and 2e-7 (70B), batch 128, 1 epoch; Table 20 lists no NLL term |
| RL stage | none in the recipe; PPO explored, no numbers published | PPO against a verifier; α = 10; β 0.05 (8B), 0.07 (70B) |
| Reward model's job | picks the best of K samples; quality-filters SFT data | initializes the RLVR value model; used in the PPO-versus-DPO test |
2. Llama 3: a six-round data engine with DPO at the end
2.1 One round, five operations
The report's backbone is "a reward model and a language model"1 (§4.1). Each round runs SFT and then DPO on data "collected either via human annotations or generated synthetically", and the whole procedure is applied "in six rounds", collecting "new preference annotations and SFT data, sampling synthetic data from the latest models" in each (§4, §4.1.6). Figure 2 steps through one round. The reward model never appears in the DPO loss; its jobs are selecting the best of K samples for SFT data and scoring data quality (Section 2.3).
- Annotate. Annotators compare two responses sampled from two different models, which "can be trained with different data mixes and alignment recipes", so capability strengths differ by pair. They mark one of four preference strengths and may edit the chosen response, which gives edited > chosen > rejected for some prompts. Prompt complexity is raised each round "to target areas where the model lags"1 (§4.2.1). Module 4 covers the strength labels.
- Fit the reward model. The model starts from the pre-trained checkpoint and trains on "all of our preference data" after similar pairs are filtered out. The loss is Llama 2's without the margin term, "as we observe diminishing improvements after data scaling", and two or three ranked responses share one row, which improved training efficiency "without a loss in accuracy" in the authors' ablations (§4.1.2).
- Rejection-sample. For each annotation prompt the team samples K outputs, "typically between 10 and 30", from the latest chat policy (usually the best checkpoint of the previous iteration, or the best for one capability) and keeps the reward model's top pick. Later rounds add system prompts that steer tone and format (§4.2.2).
- SFT. Cross-entropy on the target tokens with prompt tokens masked, over rejection-sampled responses, synthetic capability data and small human-curated sets. The largest models use a learning rate of 10-5 for 8.5K to 9K steps, a setting the authors found to "work well across different rounds and data mixes" (§4.1.3).
- DPO and averaging. DPO uses "the most recent batches of preference data collected using the best performing models from the previous alignment rounds", with only pairs labeled significantly better or better; checkpoints from different data versions or hyperparameters are averaged at each of the RM, SFT and DPO stages (§4.1.4, §4.1.5, §4.2.1). Module 7 (Sections 3 and 6.2) covers the DPO settings and the on-policy argument.
Figure 2. One Llama 3 round, stepped. Grey rows are the preference data and the reward model, cyan is sampling once from an earlier checkpoint, blue is an offline update of the policy. Source: Llama 3 §4.1-§4.2 and Figure 71.
2.2 What the loop is fed
Table 3 puts the report's two data tables side by side. The preference set is overwhelmingly general English (81.99% of comparisons), with responses of 284.0 tokens on average. The SFT set is a different mix, 52.66% general English, 21.19% reasoning and tools and 14.89% code, and the report notes that "SFT and preference data contain overlapping domains" but "are curated differently". The long-context rows are a case where example counts mislead: they are 0.11% of SFT examples but average 38,135.6 tokens each, about 5% of all SFT tokens (derived from Table 7). The report found that mixing 0.1% of synthetic long-context data into the short-context SFT data "optimizes the performance across both short-context and long-context benchmarks", and that DPO on short-context data alone did not hurt long-context ability, which the authors suspect is because the DPO recipe "has fewer optimizer steps than SFT" (§4.3.4).
| Category | Pref. share (%) | Pref. response (tokens) | SFT share (%) | SFT turns | SFT context (tokens) | SFT final response (tokens) |
|---|---|---|---|---|---|---|
| General English | 81.99 | 271.2 | 52.66 | 6.3 | 656.7 | 317.1 |
| Code | 6.93 | 462.9 | 14.89 | 2.7 | 378.8 | 374.5 |
| Multilingual | 5.19 | 420.9 | 3.01 | 2.7 | 230.8 | 289.7 |
| Reasoning and tools | 5.89 | 129.9 | 21.19 | 3.1 | 359.8 | 301.9 |
| Exam-like | - | - | 8.14 | 2.3 | 124.4 | 173.4 |
| Long context | - | - | 0.11 | 6.7 | 37,395.2 | 740.5 |
| Total | 100 | 284.0 | 100 | 4.7 | 535.7 | 310.4 |
Because most of the data is model-generated, the report spends a subsection on quality control (§4.2.3). Rule-based cleaning removes patterns such as excess emoji and exclamation marks and rebalances overused apology phrases. Model-based pruning then applies four tools: a Llama 3 8B topic classifier; a quality score that keeps examples in the top quartile of reward-model scores or at the maximum of a Llama-based rating, taking the union because the two signals "have high disagreement rates"; a difficulty score from Instag intention counts and a Llama rating; and semantic deduplication that clusters dialogs with RoBERTa embeddings and greedily keeps examples in order of quality times difficulty under a similarity threshold. The last two steps show the intent: the loop does not only drop bad data, it prioritizes the complex examples that remain.
2.3 Rejection sampling is the policy-improvement step
Algorithmically, rejection sampling followed by SFT is a one-step best-of-K operator: sample K outputs, keep the highest-scoring, train on it with cross-entropy. In DeepSeekMath's unified view it is the member of the PPO and GRPO family whose gradient coefficient is a binary keep-or-drop on sampled outputs, where PPO uses a graded advantage3 (§5.2.1; Module 7, Section 7). The reward model's scores never enter a gradient, so the stage runs on an ordinary inference stack; what it costs is generation, K samples per prompt from a 405B model, repeated each round.
Meta's inference-side answer is PagedAttention. To remove swap-out when the cache fills, it defines a maximum output length and admits a request only if memory can hold an output of that length; it also shares a prompt's key-value pages across all of that prompt's outputs. "Together, this leads to a throughput improvement of over 2× during rejection sampling"1 (§4.2.2). The report does not split the 2× among dynamic allocation, admission control and prompt sharing; the sharing part can be bounded from its own tables.
Derivation: how much key-value memory prompt sharing saves per prompt
Llama 3 405B has 126 layers, 8 key-value heads and a head dimension of 128 (model dimension 16,384 over 128 heads)1 (Table 3). The cache dtype is not stated; assume BF16, 2 bytes. Per token:
$$ 2 \times 126 \times 8 \times 128 \times 2\ \text{B} = 516{,}096\ \text{B} \approx 0.52\ \text{MB}. $$Take Table 7's SFT averages as stand-ins for a rejection-sampling prompt (context \(c = 535.7\) tokens, response \(r = 310.4\) tokens; the report gives no lengths for those prompts). With \(K\) samples per prompt:
$$ \text{unshared: } K(c+r) \qquad \text{shared: } c + K r \qquad \text{ratio} = \frac{K(c+r)}{c+Kr} \xrightarrow{K\to\infty} \frac{c+r}{r} = \frac{846.1}{310.4} = 2.73. $$At \(K=30\), the top of the report's range, unshared is 25,383 tokens (13.10 GB) and shared 9,847.7 tokens (5.08 GB), a ratio of 2.58; at \(K=10\), 4.37 GB against 1.88 GB, 2.32 (derived). The ratio cannot pass \((c+r)/r\), because the responses themselves are never shared. It is a footprint ratio, not a throughput, of the same order as the report's measured "over 2×" and no more than that.
Rejection-sampling KV footprint per prompt, Llama 3 405B
Formula: KV bytes per token = \( 2 L H_{kv} d\, b \) with \( L = 126 \), \( H_{kv} = 8 \), \( d = 128 \) (Llama 3 Table 3); shared footprint \( (c + K r)\) tokens, unshared \( K (c + r) \). Defaults are Table 7's average context and response, rounded to integers; the cache dtype is an assumption, not a report figure.
2.4 Where a checkable signal enters: filters on SFT data
The capability sections (§4.3) are where Llama 3 comes closest to verifiable rewards. The motivating observation is blunt: "training Llama 3 405B on its own generated data is not helpful (and can even degrade performance)", so the team "introduced execution feedback as a source of truth"1 (§4.3.1). Parsers, linters, unit tests run in containers, final-answer checks and learned outcome and stepwise reward models decide which generated dialogs survive into SFT (Table 4). About 20% of the roughly one million execution-checked coding dialogs "were initially incorrect but self-corrected".
| Capability | Signal that decides what survives | What it produces |
|---|---|---|
| Code, execution feedback | parser and linter; model-written unit tests in containers; failures revised from stdout, stderr and return code | about 1M coding dialogs |
| Code, rejection-sampled | an earlier Llama 3 as judge, 0 or 1 for correctness and style; keep 2 of 2; hardest examples revised until they pass | SFT data; the strict filter first cost score by removing hard prompts |
| Code, translation and back-translation | parse, compile, execute; or the model scores round-trip faithfulness | about 1.2M back-translated dialogs; over 2.7M synthetic code examples in all |
| Math and reasoning | final-answer match, self-verification, outcome and stepwise RMs, MCTS with stepwise RMs on hard prompts, code execution | filtered and generated reasoning traces |
| Factuality | judge scores answers against the source snippet; refusals written where answers are consistently informative and wrong | SFT data that refuses what the model does not know |
| Tool use | message-level human preference or edit; no rejection sampling, since "we did not observe gains in our tool benchmarks" | preference data and SFT |
| Multilingual | language-match checks before RM selection; temperature fixed at 0.6 in the final round | rejection-sampled data, 18.8% of the multilingual mix |
The pattern is the point: each row is a checker used as an offline filter. Tulu 3's contribution, taken up in Section 3.4, is to put a checker of the same kind in the reward slot of an online loop. Two details bear on later modules. For reasoning data Llama 3 trains "outcome and stepwise reward models" and runs Monte Carlo tree search with step-level reward models on hard prompts, and DeepSeek-R1 would later list process reward models and tree search among its failed attempts for RL4 (Supplementary G.2; Module 4, Section 4.5, and Module 27); the contexts differ, since Llama 3 uses them to curate data and R1 tried them as a training signal. And the factuality recipe, which turns consistently wrong answers into refusals, is an SFT-time attempt at the abstain-when-unsure behavior that binary RL rewards do not teach (Module 24).
2.5 What the report does not publish
The post-training section gives recipes, not ablations. In §4 and §5.2 it publishes no PPO numbers, no benchmark score per round, no separation of rejection-sampling SFT from DPO, no reward-model size, and no example count for either Table 6 or Table 7, only percentages and averages1 (§4.1-§4.2). Its end result is Table 2 of the report: for the 405B model, IFEval 88.6, GSM8K 96.8, MATH 73.8, HumanEval 89.0 and GPQA 51.1, in Meta's harness (see Table 1 above for what that harness does to comparisons). What the DPO stage contributed on top of rejection-sampling SFT is not recoverable from the report; Tulu 3, which publishes per-stage scores, is the better evidence for that question (Section 3.5).
3. Tulu 3: evaluation first, three stages, one verifier stage
3.1 The measuring instrument and the prompt pool
Ai2's first deliverable is the evaluation. The development suite covers knowledge (MMLU, PopQA, TruthfulQA), reasoning (BBH, DROP), math (GSM8K, MATH), code (HumanEval, HumanEval+), instruction following (IFEval, AlpacaEval 2) and a safety average. The held-out suite covers the same skills, except safety, with MMLU-Pro, GPQA, AGIEval, DeepMind Mathematics, BigCodeBench, HREF and IFEval-OOD, whose constraint set is disjoint from IFEval's2 (Tables 3 and 24, §7.3). The prompt pool has 23,327,961 prompts; 939,344 are used in SFT and 425,145 appear in the DPO column (Table 7).
Contamination is handled before training. Ai2 chose 8-gram matching over full-string and embedding matching, because embeddings made it "difficult to distinguish mere distributional similarity from actual paraphrasing". A test instance overlaps a training instance if more than 50% of its tokens sit in shared 8-grams, and a training set is contaminated if its instances overlap more than 2% of any evaluation's instances; datasets contaminated against the held-out suite were dropped, and for the development suite either the whole dataset or the matched instances (§3.2). Table 8 shows the removals: 11.3% of NuminaMath-TIR, 5.4% of the WildChat GPT-4 subset, 3.5% of Evol CodeAlpaca, 1.1% of WildGuardMix and 0.7% of WildJailbreak. The RLVR prompts are the GSM8K and MATH training splits plus constraint prompts built from the Tulu 2 SFT mix (§6.1); the decontamination section does not mention them separately, and Module 20 (Section 6) covers what contamination does to RL measurements.
3.2 SFT: a mix built from per-skill upper bounds
The mix was built backwards from ceilings. For each lagging skill the team trained a specialist on skill-specific data, which approximates the best that evaluation can reach, then mixed datasets until one model came close to every ceiling, adding, removing, decontaminating and downsampling along the way (§4.1.2). Synthetic diversity came from persona prompting: about 250K personas steer GPT-4o to write prompts for instruction following, math and code. The instruction-following set starts from 33 seeds covering 25 IFEval constraint types and yields 29,980 verifiable instruction-response pairs; the math and code sets total about 220K and 35K instances (§3.1.2).
Table 10's ablations on the 8B SFT model show what each slice bought (differences are removal minus the full mix). Dropping math data cuts GSM8K from 76.2 to 64.1 and MATH from 31.5 to 23.5; dropping persona data cuts IFEval from 72.8 to 53.6; dropping WildChat cuts AlpacaEval 2 from 12.4 to 7.5; dropping safety data cuts the safety average from 93.1 to 74.7 and leaves other skills within 2 points (derived), under the report's heading "Safety is Orthogonal"2 (§4.2). Training is two epochs at batch size 128 and 4,096 tokens, learning rate 5e-6 at 8B and 2e-6 at 70B, on 32 GPUs for 6 hours and 64 GPUs for 50 hours (Table 11, §4.3). The checkpoints average 60.1 (8B) and 72.6 (70B) against 48.3 and 63.6 for the Tulu 2 mix (Table 9). One caution: the same 8B SFT checkpoint is scored 62.1 on MMLU in Table 9 and 65.9 in Table 6, with no explanation, so take deltas within one table.
Two training details transfer to RL. First, Ai2's SFT gap to TPU-trained baselines traced to loss aggregation: averaging over tokens within a batch weights tokens equally, while per-sample averaging under gradient accumulation weights examples equally, and the two differ whenever lengths differ. Ai2 switched to a sum loss, which weights tokens equally (§4.3.2); Module 11 shows the same choice reappearing in GRPO. Second, seed noise is large next to the effects compared: five 8B seeds span 59.8 to 60.1 and three 70B seeds span 70.0 to 72.6 (Table 14), and model soups did not reliably beat the best single run (60.2 and 72.5 against 60.1 and 72.6), so Ai2 shipped single runs. Llama 3 averages models at every stage (Section 2.1) and reports no comparison with a single run.
3.3 Preference data: a pipeline, and what its ablations found
The preference pipeline adapts UltraFeedback. For each prompt, four responses are drawn from a pool of 22 models, with one taken from the Tulu 3 SFT model on a subset to make the pair on-policy; GPT-4o-2024-08-06 rates each from 1 to 5 on helpfulness, instruction following, honesty and truthfulness; the best mean rating is chosen and a random lower-mean response is rejected2 (§5.2.1). The SFT prompts alone give 96,911 off-policy and 19,444 on-policy instances; the final mixes add unused prompts, WildChat, UltraFeedback and instruction-following sets for 271,409 pairs at 8B and 334,302 at 70B (Table 15). Module 8 (Section 7) compares this labeler with the other recipes and Module 7 (Section 3.4) covers pair construction.
The report's ablations (§5.3) are unusually specific about what moved the DPO result.
- Unique prompts, not duplicated ones. Average score rose as unique prompts grew from 5% to 100% of the pool, but expanding UltraFeedback's 64k prompts into 180k and 383k pairs by recombining responses gave no better average than 64k.
- New prompts and on-policy responses. Prompts unused by SFT beat reused SFT prompts slightly, the best mix used both, and adding on-policy SFT responses improved the aggregate over a fully off-policy set.
- The judge mattered little. DPO models trained on annotations from five judges averaged 57.3 (GPT-4o), 57.2 (Llama 3.1 405B), 57.0 (GPT-4 Turbo), 56.9 (GPT-4o mini) and 56.6 (Llama 3.1 70B), a 0.7-point spread (derived); Ai2 chose GPT-4o for cost, ease of use and the Batch API (Table 17).
- Verifiers as data filters. Persona-IF pairs rewrite a prompt to relax one constraint and use the response to the relaxed prompt as the rejected answer to the original; of three persona sets only IF helped its target. A second IF set ran chosen completions through constraint verifiers, cutting more than 66k instances to about 26k (39%), and improved IFEval by only 1 point at a small cost to the average, so the unverified set stayed (Figures 13-14).
The algorithm choice rests on Table 18, an early SFT checkpoint with UltraFeedback: of DPO, SimPO and length-normalized DPO, only the last beat the SFT base (57.3 against 55.7 at β = 5, one epoch), and it moved by 10.5 points across settings. Module 7 (Section 4.6 and Figure 6) plots every row. Final settings are in Table 2 above; the 8B run took 10 hours on 8 H100 GPUs and the 70B run 19 hours on 64, using cached reference log-probabilities and separate chosen and rejected forward passes to cut peak GPU memory (§5.4). The verifier-filtered IF pairs are the nearest thing to RLVR inside the DPO stage, and the next stage reuses the same constraint verifiers as a reward.
3.4 RLVR: a small last stage with a checker in the reward slot
Stage three swaps the reward model for a program. Tulu 3 defines RLVR as the RLHF objective "with the reward model replaced by a verification function", with reward α = 10 if the completion verifies and 0 otherwise2 (§6, Eq. 7-8). Module 10 (Section 1) covers the definition and the name; Module 5 (Section 8) covers the PPO configuration, the value-model and KL ablations, and the infrastructure (asynchronous PPO, dedicated vLLM GPUs placed by Ray). This section keeps only what bears on the recipe's order and size, and Figure 3 puts the whole pipeline in one place.
RLVR starts from the best DPO checkpoint. At 8B the team tried the best overall DPO model and the best one with Persona-IF data; at 70B, the best overall one (§6.4). The prompt set is 29,946 prompts: 7,473 GSM8K and 7,500 MATH training questions, with exact match against an extracted answer, and 14,973 instruction-following prompts with prompt-specific verifiers (Table 22). Ablation runs used 100,000 episodes, about 13 passes over GSM8K's 7,473 prompts, and the final 70B run 400,000 episodes at batch 640 and learning rate 1e-7 (Table 21, §6.2, §6.4). One discrepancy matters when you reproduce it: the §6.4 text gives the final 70B run β = 0.7 and a warm-up ratio of 0.1, while the Table 21 caption gives β = 0.07 and ω = 0.07. This module uses the caption's 0.07, as Modules 5 and 6 do; the sweeps in the report run from 0.001 to 0.15, which makes 0.7 the odd value.
The result is a targeted stage. At 8B, RLVR moves GSM8K from 84.3 to 87.6, MATH from 42.0 to 43.7 and IFEval from 81.1 to 82.4; at 70B, GSM8K stays at 93.5, which the report calls close to saturation, MATH moves 62.3 to 63.0 and IFEval 82.6 to 83.2 (Table 23). Final checkpoints were picked on the development suite for the best overall MATH and IFEval; some 8B runs reached 89.4% GSM8K and 84.8% IFEval but lowered the other metrics and the average (§6.4). Lambert reads this as RL "done surgically", with a narrow prompt distribution that does not squash general capability5 (§3.1.6). Section 3.5 asks what the held-out suite says about that. A caution on secondary summaries: Lambert's book describes the recipe as about 1M SFT examples, about 1M preference pairs and about 10K RLVR prompts5 (§3.2.2), while the report's own tables give 939,344, 271,409 (8B) or 334,302 (70B), and 29,946; use the report's counts.
3.5 What each stage added, and what the held-out suite says
The report's summary tables (4, 5 and 6) give each stage's average, which Figure 4 plots. DPO is where the aggregate moves: +4.1 points over SFT at 8B, +3.6 at 70B and +2.7 at 405B (derived; the 405B average excludes safety). RLVR adds +0.4, +0.0 and +1.0. The same-base Instruct model sits between the SFT and DPO checkpoints at all three sizes, and the final models lead it on the development average by 2.2, 2.1 and 1.9 points (derived). For scale, three seeds of the 70B SFT run differed by up to 2.6 points on the report's average (Table 14).
| Skill (development → held-out) | 8B dev | 8B held-out | 70B dev | 70B held-out |
|---|---|---|---|---|
| Knowledge (MMLU → GPQA) | -0.5 | +4.5 | -0.2 | 0.0 |
| Reasoning (BBH → AGIEval) | +0.2 | -2.5 | +0.2 | 0.0 |
| Math (MATH → DeepMind Math) | +1.7 | +2.4 | +0.7 | +0.4 |
| Code (HumanEval → BigCodeBench) | 0.0 | -2.1 | 0.0 | -1.4 |
| Instruction following (IFEval → IFEval-OOD) | +1.3 | +0.4 | +0.6 | +1.4 |
| Average | +0.5 | +0.5 | +0.2 | 0.0 |
Table 5 supports a narrow reading. RLVR trained on GSM8K, MATH and IFEval-style constraints, and those two skills rise on both the development and the held-out benchmark at both sizes, by small amounts: held-out math +2.4 and +0.4, held-out instruction following +0.4 and +1.4. The constraint types in training are the 25 shared with IFEval, so the IFEval gain is in distribution, and at 8B it shrinks from +1.3 to +0.4 on IFEval-OOD. Held-out code falls at both sizes. The 8B GPQA jump of 4.5 points against zero at 70B is single-run movement that the report does not explain.
The held-out suite also qualifies the headline comparison. Table 6 sets Tulu 3 against Llama 3.1 Instruct on benchmarks Tulu 3 did not tune against. Llama 3.1 Instruct leads the held-out average at both sizes, by 2.2 points at 8B and 4.1 at 70B (derived), the reverse of the development lead in Figure 4; Tulu 3 wins only on GPQA. The report calls its results "generally comparable" to the other models, "often falling between" them, and cannot verify that the other models did not train on these benchmarks2 (§7.4.2). Its reading of IFEval is that "models generally overfit to IFEval": every model scores far lower on IFEval-OOD, whose constraint set is disjoint. A recipe selected on a development average will look best on that average; the held-out suite is the fairer test.
| Held-out benchmark | Llama 3.1 8B Inst. | Tulu 3 8B | Llama 3.1 70B Inst. | Tulu 3 70B |
|---|---|---|---|---|
| Average (7 benchmarks) | 36.4 | 34.2 | 51.3 | 47.2 |
| GPQA, 0-shot CoT | 28.8 | 35.7 | 43.8 | 48.0 |
| MMLU-Pro, 0-shot CoT | 49.1 | 44.3 | 68.3 | 65.8 |
| AGIEval English, 0-shot CoT | 64.2 | 59.3 | 77.8 | 75.0 |
| DeepMind Mathematics, 0-shot CoT | 39.3 | 35.4 | 62.4 | 49.8 |
| BigCodeBench-Hard, pass@10 | 15.5 | 7.4 | 26.4 | 21.6 |
| IFEval-OOD, prompt loose | 26.1 | 24.3 | 34.5 | 27.8 |
| HREF, win rate | 38.5 | 32.7 | 45.6 | 42.3 |
3.6 What the stages cost
The report gives wall-clock and GPU counts for most stages, enough to derive GPU-hours (Figure 5). At 8B: SFT 32 GPUs for 6 hours, DPO 8 for 10, RLVR 8 for 65, plus a 9-hour, 8-GPU reward model. At 70B: SFT 64 for 50, DPO 64 for 19, RLVR 48 for 60. The three stages sum to 792 GPU-hours at 8B and 7,296 at 70B, and RLVR is 65.7% of the 8B total but 39.5% of the 70B total, where SFT is the largest line at 43.9% (all derived from §4.3, §5.4.1 and §6.3)2. The 405B RLVR run is 256 GPUs for 46 hours, 11,776 GPU-hours, 4.1 times the 70B run for MATH-only prompts; the report gives no GPU-hours for 405B SFT or DPO.
Two caveats limit these numbers. The report says that for all of the RL runs "we took an earlier than final checkpoint from the run", so the delivered models used fewer GPU-hours than listed (§6.3). And these are final runs only: Lambert's book says that developing a recipe "can easily be 10X to 100X the compute of the final few training runs" and that the Tulu 3 team "ran on the order of thousands of experiments/evaluations at the 7B scale before having the final model"5 (Appendix C.1). For scale against Section 5, R1-Zero's run is 101,376 GPU-hours on H800 GPUs4 (Supplementary B.4.4, Table 7), 8.6 times the 405B RLVR run and 35 times the 70B one (derived; different hardware and response lengths). Module 1 (Section 5.2) analyzes the 405B phase times.
3.7 Scaling to 405B, and what the MATH-only run did
Scaling to 405B needed 32 nodes. The report lists "NCCL timeout and synchronization issues that required meticulous monitoring and intervention (especially with RL training)", restarts after hardware failures, and "limited" hyperparameter tuning2 (§8.1). The RLVR data changed: GSM8K was removed because SFT and DPO had saturated it, the constraint prompts "did not help much in initial RLVR runs", and the final run used the MATH training set only, with an 8B value model. "Even with as few as 25 RLVR steps, MATH performance improved by over 5 points, and continued to increase with more training"; the run stopped at 75 steps because of the "challenges of scaling the asynchronous RL infrastructure".
The 405B hyperparameter table, labelled Table 35 in the PDF text and jumbled there, gives learning rate 1e-7, batch size 1,856, response length 1,024, β = 0.05 and 300,000 total episodes; its caption describes the general-reward-model run while its column header reads 405B RLVR, so the caption looks misplaced. Seventy-five steps at batch 1,856 is 139,200 episodes (derived), 46% of the printed 300,000 (Figure 6). The report's Figure 25 marks the final checkpoint near that point on its episode axis, with the curves continuing somewhat past it (read from the plot); the report does not explain the gap.
| Benchmark | SFT | DPO | RLVR | Δ (RLVR − DPO) |
|---|---|---|---|---|
| MATH, 4-shot flex (in the reward) | 63.4 | 59.9 | 67.3 | +7.4 |
| GSM8K, 8-shot CoT | 93.6 | 94.2 | 95.5 | +1.3 |
| IFEval, prompt loose | 82.4 | 85.0 | 86.0 | +1.0 |
| AlpacaEval 2, LC win rate | 30.4 | 49.8 | 51.4 | +1.6 |
| MMLU, 5-shot CoT | 84.4 | 86.6 | 87.0 | +0.4 |
| BigBenchHard, 0-shot CoT | 88.0 | 88.8 | 88.6 | -0.2 |
| HumanEval, pass@10 | 95.7 | 97.2 | 95.9 | -1.3 |
| HumanEval+, pass@10 | 93.3 | 93.9 | 92.9 | -1.0 |
| Safety, 6-task average | 87.7 | 85.5 | 86.7 | +1.2 |
| Average without safety | 76.3 | 79.0 | 80.0 | +1.0 |
Table 7 shows two things the smaller runs could not. At 405B the DPO stage lowered MATH by 3.5 points against SFT (63.4 to 59.9), the opposite of 8B (31.5 to 42.0) and 70B (53.7 to 62.3), and the report does not say why; MATH-only RLVR recovered it and went 3.9 points past SFT (derived). And a reward on MATH alone moved GSM8K and IFEval up and HumanEval pass@10 down, which is consistent with RL on a narrow prompt set transferring to neighbouring skills, but is one run with no interval and an untested mechanism. Against the same-harness baselines, Tulu 3 405B's MATH of 67.3 is above Llama 3.1 405B Instruct (66.6) and below GPT-4o (68.8) and DeepSeek V3 (72.5).
3.8 What Tulu 3 tried and dropped
Section 8.2 of the report lists two methods that did not make the recipe, and both are central to someone else's.
Rejection sampling. Ai2 used a model trained on its initial SFT and preference mixes to generate n responses per SFT prompt, ranked them with the original response using a reward model or an LLM judge, kept the best as SFT data and made chosen-rejected pairs from the rest. It "found that for our setup the performance gains were minimal for the amount of compute required". Two findings qualify that: "strong judges are vital, and publicly available models can struggle to choose the best response", and including the original response among the candidates "performed much better"2 (§8.2). Llama 3 used Meta's own reward model, trained on its human preference data (Section 2.3), which is consistent with the judge-quality point; neither report tests it.
Online DPO. The recipe samples two responses from the current policy, scores them with a reward model to make a pair, and applies the DPO loss. Ai2 trained an RM on 82K Skywork preference pairs, continued it on synthetic on-policy math preferences, and ran 200K episodes on math prompts from a Tulu 3 DPO checkpoint. It found "no or little improvement on GSM8K and degradation on MATH performance" across temperatures and KL coefficients and dropped it (§8.2). In Figure 7 (Section 6) it sits in the online column with a learned scorer, beside the PPO runs of Section 4.
4. The PPO decision, as each report records it
Both reports tried PPO for preference tuning, found it more expensive, and kept DPO. The evidence is thin by design: one report publishes no numbers, and the other says its PPO was untuned. Table 8 separates what was run from what was concluded; Module 5 (Section 9) and Module 7 (Section 8) hold the ledgers that set these two against studies that found PPO ahead.
| Report | What was run | What is published | Reason given | Outcome |
|---|---|---|---|---|
| Llama 3 | PPO "explored"; model size and setup not stated | no scores, no costs | DPO "required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval" | no RL algorithm in the recipe |
| Tulu 3, early exploration (Table 18) | PPO with an UltraFeedback reward model, lr 1e-6, batch 64, β 0.0325 and 0.05, early SFT checkpoint | averages 54.5 and 55.5; SFT base 55.7, DPO 55.2, best length-normalized DPO 57.3 | "more costly investigations into RL-based methods, such as PPO" | length-normalized DPO throughout development |
| Tulu 3, controlled comparison (§5.4.1, Figure 16) | reward model trained once and untuned on the anchored DPO mix; PPO sweep over β 0.01 to 0.05 and warm-up 0 or 0.1 | plotted only; "comparable ... (albeit slightly lower)"; 28 hours on two nodes against 4 on one | "limited resources and the subtlety in RM evaluation" | DPO for preference tuning; PPO "primarily for RLVR" |
Two experiments are easy to conflate. Per-run PPO scores are printed only in Table 18, the early exploration, where both PPO runs (54.5 and 55.5) finish below the SFT base (55.7) and below the best length-normalized DPO setting (57.3). The later controlled comparison, the source of the 28-hour and 4-hour figures, is a plot of average score against steps (Figure 16) with no numbers in the text; its SFT and DPO baselines (about 57 and 61, read from the plot) are not Table 18's, so it is a different experiment. In it the better PPO runs end between the two baselines, which is the report's "slightly lower". At 8 GPUs per node the cost is 448 against 32 GPU-hours, 14 times, and 7 times in wall-clock (derived; Figure 5, right).
Ai2's reason has two parts: cost, and "the subtlety in RM evaluation". A reward model that scores well on a reward-model benchmark "does not necessarily translate to better downstream performance for PPO", so Ai2 trained its RM once and did not tune it2 (§5.4.1). A verifier changes that, because its reward is exact for the property it checks (it fails in other ways; Modules 10 and 19), and the report adopts PPO for RLVR in the next sentence. In the writer's reading, PPO's cost did not fall between the two decisions; the reward stopped being the thing nobody could evaluate. Tulu 3's own 405B run shows where the cost lands (Section 3.7), and in the writer's reading Meta's "less stable and harder to scale" is a claim about 2024 infrastructure as much as about the algorithm: the 2025 reports that run RL at larger scale spend many pages on stability and rollout-trainer mismatch (Modules 13 and 14).
Neither report runs the comparison that would settle the preference-tuning question: tuned PPO against DPO on matched data at the same scale. Xu et al. report PPO ahead on every testbed, including code competitions6; Ivison et al., from the lab and several of the authors behind Tulu 3, find it ahead by up to 2.5% in math and 1.2% in general domains, while ranking data quality first, then the algorithm, then the reward model7. Tulu 3 cites that study, and the reason it states for choosing DPO is cost. Since data quality ranks above the algorithm, the sharper test holds the data fixed, which Tulu 3's controlled comparison does, with an untuned PPO and an untuned reward model.
5. From a DPO-first recipe to GRPO-style RLVR
5.1 What Tulu 3 left open
The report names its own limits. The critic is an 8B model standing in for a 405B policy, and "future works can benefit from exploring larger value models or alternate value model-free RL algorithms such as GRPO"2 (§8.1). GRPO was already public: DeepSeekMath introduced it in February 2024, nine months earlier, as a PPO variant that "foregoes the critic model, instead estimating the baseline from group scores"3 (abstract). The verifiers are the simplest kind, exact match on an extracted answer and per-constraint checks, with "more complex verifiers" left to future work (§6.1); responses are capped at 1,024 or 2,048 tokens (Table 21); checkpoints are selected on the development suite (§6.4). It is the list of dials the next recipes turned.
5.2 What the 2025 reasoning recipes changed
Table 9 sets Tulu 3's RLVR stage beside R1-Zero and OlmoRL, the RL stage of Ai2's later Olmo 3. No source in the course says one recipe was derived from another, so the table compares design points, not influence. The dials that turned are the critic, the KL term and the response length. The one that did not is the reward: a program that checks the answer.
| Choice | Tulu 3 RLVR (Nov 2024) | R1-Zero (Jan 2025) | OlmoRL (Olmo 3, Dec 2025) |
|---|---|---|---|
| Algorithm | PPO with a value model initialized from a reward model | GRPO, no critic | GRPO with zero-gradient filtering, active sampling, token-level loss, clip-higher, truncated importance sampling, no std normalization |
| Starting point | best DPO checkpoint | DeepSeek-V3-Base, no SFT first | after SFT and DPO: the Think models train "via SFT, DPO, and RLVR" |
| KL to the reference | β 0.05 (8B), 0.07 (70B) | coefficient 0.001 | no KL loss |
| Longest response | 1,024 to 2,048 tokens | 32,768, then 65,536 from step 8.2k (16× to 32× Tulu 3's cap, derived) | rollouts up to 32K tokens |
| Reward | α = 10 if verified; adding RM scores added noise | accuracy plus format, equal weight, no neural reward model | programmatic verifiers (Module 10, Section 1.2) |
| Final-run size | 520 / 2,880 / 11,776 GPU-hours (8B / 70B / 405B, derived) | 101,376 GPU-hours on H800 (Module 1) | see Module 31 |
The critic left first, for reasons already on the table. DeepSeekMath's case against PPO's value model is that it is "another model of comparable size" to the policy, a "substantial memory and computational burden", and hard to train when the reward arrives only on the last token3 (§4.1.1); Tulu 3's 8B stand-in at 405B is the same pressure seen from the other side (Module 9 covers when a critic still wins). The KL term left next: Tulu 3's sweeps showed that more KL did not reliably buy verifiable reward and eventually cost the average score, one reading of why later reports with exact rewards dropped it (Module 13). Length grew by more than an order of magnitude with long chain-of-thought tasks, which shifts the cost toward generation (Olmo 3's average step was 1,000 seconds, of which 125 were training8; Module 1) and brings the rollout-trainer mismatch of Module 14 with it.
5.3 What survived
Three things carried through. The reward stayed programmatic, which is the RLVR idea itself: R1-Zero's accuracy and format rewards are the same construction at a larger scale, and Lambert credits R1 with popularizing RLVR, not inventing it5 (p. 14; Module 10, Section 1). RL stayed a late stage for the skills that have answers: R1's pipeline adds stages around it, and Olmo 3 keeps Tulu 3's order of SFT, then DPO, then RL8 (§1). And for chat behavior, where there is no checker, the reports in this course still use preference signals: DPO in Olmo 3, a preference reward model in R1's last 400 RL steps4 (§3.2.2). What did not survive is the premise that RL algorithms are too unstable to use: for verifiable domains the 2025 reports run online RL as the main stage (Module 10, Section 6). No Meta report after Llama 3 is in the course's source map, so whether Meta's own stance changed is not documented here.
6. The two recipes on two axes
Two properties set a stage's cost and its failure modes: where its training samples come from (a fixed source, an earlier checkpoint sampled once, or the policy being trained) and what scores them (nothing, preference pairs, a learned reward model, a program). Figure 7 places every stage of both recipes on those axes, together with the stages Tulu 3 tried and left out.
Three readings follow. First, cost follows the columns: the left and middle columns need no generation engine inside the training loop (Llama 3 pays for generation in rejection sampling, but as a batch job ahead of training; Section 2.3), and only the right column carries the memory bill of Module 1 (Section 7). Second, both recipes ship almost everything in the left and middle columns, and Tulu 3's one right-column stage has a program as its scorer; the right-column cells with a learned scorer are where Tulu 3 tried PPO and online DPO and shipped neither, and where Llama 3 explored PPO without publishing results. Third, the verifier row is filled at two prices. Llama 3 runs its checkers once, as filters on samples from an earlier checkpoint, and trains on what passed. Tulu 3 runs them as the reward for the policy being trained, which costs a generation engine and a value model and puts a gradient on the checked property, including a push away from failures: the keep-or-drop versus graded-coefficient distinction of Section 2.3.
Nothing in the figure says one cell is better than another; it says which cells each report paid for. The right-column cell with a learned reward model is where classical PPO-RLHF lives (Module 5), and both reports declined to ship it. The disagreement below is partly whether that cell is worth its price. How the stages are chained, gated and registered is System Design Module 36 (Post-Training Pipelines); rollout infrastructure is Modules 37 and 38.
Where experts disagree
Four questions run through these reports. Two are the course's standing disputes, applied here: whether preference tuning needs online RL, and whether PPO's critic earns its cost. Two are internal to the pair: the reports disagree about rejection sampling, and they differ on the value model. Each card states one side; the cons carry the other.
Offline DPO in rounds is enough for preference tuning
Llama 3 team1 (§1, §4.1.4); Tulu 3 authors2 (§5.4.1)
Llama 3 chose SFT, rejection sampling and DPO over RL algorithms that "tend to be less stable and harder to scale", and tried PPO. Tulu 3 measured PPO at about 28 hours on two nodes against 4 on one for a comparable average, and reserved PPO "primarily for RLVR".
- The cost gap is measured: 14 times the GPU-hours and 7 times the wall-clock (derived, Figure 5).
- No reward-model loop, generation engine or value model in the optimizer; used at 405B across six rounds.
- Llama 3 publishes no PPO numbers; Tulu 3's PPO had an untuned reward model, and "it is entirely possible that we can push up the PPO's performance even higher".
- Data quality ranks above the algorithm7, so a DPO-versus-PPO gap on one data set may be a data effect.
Tuned online PPO is stronger, and RL is worth its infrastructure
Xu et al.6 (abstract); Ivison et al.7 (abstract); Lambert5 (§3.1.6)
Xu et al. report PPO ahead of the other methods in all their testbeds, including code competitions. Ivison et al. find it ahead by up to 2.5% in math and 1.2% in general domains. Lambert argues that "despite the simplicity of alternatives, RL-based methods continue to win out", at a far larger infrastructure cost.
- Controlled studies, one from Tsinghua and OpenPsi and one from Ai2 and UW, the Tulu 3 lab (full-text details in Module 5, Section 9).
- Tulu 3's RLVR lifted its targeted benchmarks on a narrow prompt set, which Lambert reads as RL done "surgically" (Section 3.4).
- Margins are small next to the data-quality effect in the same Ivison et al. abstract.
- Tulu 3's 405B PPO run needed a 16-GPU inference group, a 240-GPU learner and an 8B critic, and stopped at 75 steps (Section 3.7).
Rejection sampling with a strong reward model is the core of the data engine
Llama 3 team1 (§4.2.2); opposed by Tulu 32 (§8.2)
Llama 3 samples 10 to 30 outputs per prompt from the latest policy and trains on the reward model's pick, as the loop of its Figure 7 shows, and engineered generation for over 2× throughput. Tulu 3 tried the idea and "found that for our setup the performance gains were minimal for the amount of compute required".
- Cheap next to online RL: a batch job with no gradient through the reward model.
- Composes with capability experts, system prompts and execution or answer checks (Table 4).
- Llama 3's §4 gives no ablation of rejection sampling against none.
- Tulu 3 found public judges "can struggle to choose the best response", so the result depends on reward-model quality; it did not test a Meta-grade one.
A learned value model is worth keeping for RLVR
Tulu 32 (§6.2.1, §8.1); against it, DeepSeekMath3 (§4.1.1) and Ahmadian et al.9 (§1)
Tulu 3 found that initializing the value function from a general reward model gave the best GSM8K score and a higher average, and concluded that "the value function plays an important role". DeepSeekMath and Ahmadian et al. argue the critic is a policy-sized cost that last-token rewards do not justify.
- A measured ablation on the recipe's own prompts, with a clear ordering of the initializations tried.
- Module 9 (Section 8) records controlled comparisons in which a tuned critic-based PPO beats critic-free methods.
- The ablation compares value initializations, not a critic against no critic; the report has no GRPO run.
- At 405B the critic was 8B, and the 2025 reasoning recipes dropped it (Table 9).
On preference tuning the cost claim is settled and the quality claim is not. DPO in rounds is a defensible default when the signal is a reward model you cannot evaluate, when you can run only one inference stack, and when the budget goes to data iteration, which both reports treat as the main lever. A tuned PPO plausibly buys a point or two, inside the range that data choices move, on evidence from studies below 405B scale. A matched-data, matched-compute study at 70B or above, with tuned PPO, held-out evaluations and seed intervals, would settle it; the course's source map has none. On rejection sampling the reports test different judges, not different ideas, and an ablation that fixes a strong reward model and removes the sampling stage is the missing evidence. On the critic the answer is conditional (Module 9): Tulu 3 shows initialization matters if you keep one, the 2025 recipes show that dropping it works at long lengths where it is most expensive, and no report in the source map runs PPO against GRPO at Tulu 3's settings with both tuned. For verifiable skills the field has moved to online RL regardless; what remains open is how much of preference tuning will follow (Section 5).
What is changing
As of 2026-10-27 the course's source map holds no Meta post-training report after Llama 3, so whether the DPO-first loop persists inside Meta is not documented here. Ai2's line is visible: Olmo 3's RL stage, OlmoRL, is a GRPO variant without a KL loss or a critic where Tulu 3 used PPO8 (§4.4.1), and Olmo 3 keeps DPO before RL. Lambert's book (arXiv v12, 11 Sep 2026) argues that RL keeps winning and, in its 2026 recipe survey, notes that multi-teacher on-policy distillation has become a popular final stage in several large reports while "labs are far from unanimous" on it5 (§3.2.4; Modules 21 and 22). Three things to watch: a matched PPO-versus-GRPO ablation at Tulu 3's prompt mix; whether chat-quality tuning moves from DPO to on-policy distillation; and any report that publishes a Llama 3-style multi-round loop with the ablations Meta's did not include.
References
- paperGrattafiori, A., Dubey, A., Jauhri, A., et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv 2407.21783 v3, 23 Nov 2024 (v1 31 Jul 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: design stance and the claim that RL algorithms are less stable and harder to scale (§1); post-training loop and Figure 7 (§4); RM and SFT settings (§4.1.2-§4.1.3); DPO settings and the PPO comparison (§4.1.4); six rounds (§4.1.6); preference and SFT data statistics (Tables 6-7, §4.2.1); rejection sampling and PagedAttention (§4.2.2); data quality control (§4.2.3); capability pipelines (§4.3); benchmark results (Table 2); architecture (Table 3).
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., et al. (Allen Institute for AI and University of Washington). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv 2411.15124 v5, 14 Apr 2025 (v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: recipe overview and result tables (§2, Tables 2-6); prompt pool and decontamination (§3, Tables 7-8); SFT (§4, Tables 9-14); preference data and DPO (§5, Tables 15-20, Figures 7-17); RLVR (§6, Tables 21-23); held-out evaluation (§7.4, Tables 31-33); 405B scaling and negative results (§8, Tables 34-35, Figures 16 and 25 read from the page images); related work (§9.2).
- paperShao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv 2402.03300 v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: GRPO as a critic-free PPO variant (abstract, §4.1.1) and the unified view in which rejection-sampling fine-tuning is a special case (§5.2.1).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948 v2 (expanded Nature version, 4 Jan 2026); v1 22 Jan 2025. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: R1-Zero settings (§2.1: GRPO, KL coefficient 0.001, 32,768 then 65,536 maximum length, 10,400 steps); the 101,376 GPU-hour run (Supplementary B.4.4, Table 7); process reward models and tree search as failed attempts (Supplementary G.2); the stage structure and the last-400-step preference reward (§3.2).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv 2504.12501 v12, 11 Sep 2026; web edition last built 14 Oct 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: RL done surgically and the argument that RL keeps winning (§3.1.6); R1 popularizing RLVR (p. 14); multi-teacher on-policy distillation and the lack of unanimity among labs (§3.2.4); the cost of developing a recipe (Appendix C.1); the rounded Tulu 3 counts (§3.2.2).
- paperXu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., Wu, Y. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv 2404.10719 v3. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: the abstract-level claim that PPO surpasses other alignment methods in all of its testbeds, including code competitions (full-text numbers are in Module 5 §9).
- paperIvison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., Hajishirzi, H. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv 2406.09279 v2. https://arxiv.org/abs/2406.09279. Accessed 2026-10-26. Used for: the ranking of factors (data quality, then algorithm, then reward model, then extra prompts) and the margins by which PPO beat DPO (abstract).
- paperOlmo Team. Olmo 3. arXiv 2512.13961 v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: Olmo 3 Think trained via SFT, DPO and RLVR (§1); OlmoRL as GRPO with the listed changes (§4.4.1); the 1,000-second average step of which 125 seconds was training (§4.4.3, footnote 41).
- paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv 2402.14740 v2. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: the argument that PPO is over-engineered for pre-trained LLMs and the RLOO baseline (§1, §2.3).