Reinforcement Learning for LLM Post-Training · Module 33
Choosing a Method: PPO, DPO, GRPO-Family, On-Policy Distillation, and Hybrids Under Reward, Data, Compute, and Stability Constraints
Each method in this course was built for a constraint: pairs without a reward model, a verifier without a critic budget, a teacher without RL compute. This module maps the constraints you have onto a starting method, states the evidence behind each choice with the stack, scale and strength it was measured at, says what would change it, and lists the choices that no source supports. Afterwards you can pick a starting method and the first three things to monitor for a given reward type, data, compute budget and target.
1. How to read a recommendation
Every recommendation in Sections 3 to 10 is a callout with the same parts: what to start with, the evidence and its tag, the stack and scale it was measured at, what would change it, and the first three signals to watch. The stack and scale come first because the reports differ by orders of magnitude and by engine. Tulu 3's PPO-versus-DPO test ran on two nodes1; Llama 3's DPO ran on models up to 405B2; VinePPO caps responses at 1,024 tokens8; ScaleRL spent more than 400,000 GPU-hours on dense and mixture-of-experts models of 8B to 17B-by-169. A margin measured in one of those settings does not transfer to another without a reason. Infrastructure and framework choice (Ray, Megatron, vLLM, the open-source RL libraries) is System Design Modules 36 to 38 and is not repeated here; this module treats the stack as the place where a result was measured, not as a decision.
1.1 Evidence tags
The tags are this module's rubric, not a convention in the literature. They count independent groups, not papers: Tulu 3 and Olmo 3 come from one lab, and Thinking Machines re-uses the Qwen3 table10, so neither pair counts twice. A tag attaches to a claim, so one method can carry several: DPO's cost advantage is concordant while its quality parity with PPO is conflicting.
- concordant Two or more independent groups measured the same direction, on different stacks.
- single One group measured it; nobody repeated it.
- conflicting Measurements by different groups point in different directions.
- adopted Several reports use the choice and none ablates it. Adoption is not measurement.
- no source Nothing in the course's source map measures or states it. Any view I give there is labelled as mine.
1.2 The eight axes
| Axis | Question | A measured anchor | Section |
|---|---|---|---|
| Reward type | Pairs, a learned score, a verifier, or a judge? | RLVR is the RLHF objective with the reward model replaced by a verification function1 | 2, 8 |
| Data | Do you hold pairs or prompts? | Tulu 3's pipeline yields 354,192 preference instances1; Qwen3's reasoning RL used 3,995 query-verifier pairs6 | 2 |
| Generation compute, critic memory | What does a step cost and what stays resident? | PPO about 28 h on two nodes against DPO about 4 h on one1; value model and its state 46% of GPU memory at 7B11 | 3, 4 |
| Stability, tuning burden | What breaks, and how many knobs? | PPO needed GAE \(\lambda=1.0\) to approach GRPO4; plain PPO scored 5 on AIME 2024 until seven fixes12 | 4, 5 |
| Target | Chat, reasoning, or agent? | Llama 3 kept DPO for chat2; Olmo 3 ends reasoning in RLVR5; GLM-5 adds an agentic RL stage13 | 3, 5, 6 |
| Model size | Distill or run RL? | Qwen3-8B: on-policy distillation 1,800 GPU-hours, RL 17,9206 | 7 |
| Multi-skill integration | One mixed run, stages, or specialists? | Normalized score 0.937 for multi-teacher distillation against 0.882 for mixed RL, three domains7 | 9 |
| Calibration need | Will a consumer read the stated confidence? | ECE 0.37 to 0.03 on HotpotQA with a Brier term14 | 10 |
2. Reward type and data: the first split
The reward source picks the family before anything else, because each family consumes a different object. DPO consumes (prompt, chosen, rejected) triples and does not generate during the update1. PPO, RLOO and GRPO consume prompts, generate on-policy samples and score them with a reward source; that generation is where the time goes, and in Olmo 3 the learner waits for rollouts 75% of the time5 (the report's footnote 41 gives 125 s of training in a 1,000 s step, which implies 87.5%; Module 1 flags the conflict, and both figures say generation dominates). On-policy distillation consumes prompts and a teacher's per-token log-probabilities and needs no reward function at all10.
| What you hold | Start with | Tag | Measured on | What would change it |
|---|---|---|---|---|
| Preference pairs, no reward model | Length-normalized DPO in rounds; add an NLL term if the chosen log-probability falls | conflicting | Llama 3 to 405B, six rounds2; Tulu 3's controlled test1 | The target is a hard verifiable outcome: DPO scored 0.0% on CodeContests where PPO scored 22.4%3 |
| A reward model you trust, prompts, a generation budget | RLOO or REINFORCE-style first; tuned PPO if you can pay for it | conflicting | Pythia-6.9B and Llama-7B, 512-token contexts15; DeepSeekMath-7B8 | A tuned critic is ahead by 3.6 to 6.4 points on short-response math8 |
| Prompts and a programmatic verifier | Group baseline, token-level loss, no KL, mismatch correction | adopted | Qwen2.5-32B16; Olmo 3 at 7B and 32B5; GLM-5 at 744B13 | A long run from a distilled checkpoint: KL with periodic resets17 |
| Prompts and a judge or rubric only | Rubric judge with a verifier-anchored slice | single | RaR with a 7B policy18; Kimi K2 to K319 | The judge shares weights with the policy: add an external accuracy monitor (Module 16) |
| A stronger teacher with your tokenizer, and a small student | SFT on the teacher's output, then on-policy distillation | concordant | Qwen3-8B6; Qwen2.5-32B in R14 | No stronger teacher exists, or the goal is to exceed the teacher |
The data axis is a different measurement for each family. Olmo 3 Think uses 2,268,468 prompts for SFT, 200,000 for DPO and 104,869 for RL at 7B5; Qwen3's reasoning stage used 3,995 query-verifier pairs and GRPO, chosen so that each query was learnable, as hard as possible, not used in the cold start, and broad in coverage6. RL asks for far fewer prompts and far more generation per prompt, and the prompts must carry a pass rate between 0 and 1 for the group to give a gradient (Module 9). For DPO the scarce resource is pair quality: Ivison et al. rank preference-data quality above the algorithm, reward model and extra prompts, with gains up to 8% in instruction following and truthfulness20. Olmo 3 shows how much the construction matters: SFT on the Qwen3-32B responses lowered its development average, while DPO that paired them with weaker rejected responses raised it5 (Table 21).
3. Chat alignment: DPO, PPO, or neither
This is the oldest question in the course and the one with the most conflicting evidence. Table 3 lists the measurements with the stack each was made on, because the answer depends on which row you stand on.
| Study | Stack and scale | Compared | Result | Direction |
|---|---|---|---|---|
| Llama 3 (§1, §4.1.4)2 | Models to 405B; six DPO rounds on batches from the latest models | PPO tried, DPO kept | DPO "required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval"; no PPO numbers published | DPO |
| Tulu 3 (§5.4.1)1 | Controlled, "Non-Tuned Setup"; model size not stated in the passage | PPO, DPO | PPO comparable "albeit slightly lower"; about 28 h on two nodes against about 4 h on one | DPO on cost, parity on score |
| Xu et al. (Tables 7, 8)3 | CodeLlama 7B to 34B on APPS and CodeContests; Llama2-7B on HH | PPO, DPO, DPO-Iter | CodeContests test 10@1k at 34B: SFT 15.2, DPO 0.0, DPO-Iter 3.2, PPO 22.4 (test-case reward) | PPO |
| Ivison et al. (abstract)20 | Stack not in the abstract | DPO, PPO, data, reward model, prompts | PPO ahead by up to 2.5% (math) and 1.2% (general); data quality up to 8% | PPO, small margin |
| Tang et al. (abstract)21 | Stack not in the abstract | Online against offline | Online ahead; the gap persists as the policy network is scaled up | Online |
| Ahmadian et al. (Table 1)15 | Pythia-6.9B on TL;DR and HH; Llama-7B on HH; 512-token contexts | RLOO, PPO, DPO | RLOO 77.9, 43.7, 64.1 against DPO 66.6, 39.0, 61.9: +11.3, +4.7, +2.2 | RLOO |
3.1 The cost side is settled
Llama 3 and Tulu 3, two independent groups, report that PPO costs more for no gain in their setups21. Only Tulu 3 gives a number, and the arithmetic is short. PPO ran 28 h on two nodes, which is 56 node-hours; DPO ran 4 h on one node, which is 4 node-hours. The ratio is \(56/4 = 14\) in node-hours and \(28/4 = 7\) in wall-clock. At 8 GPUs per node, the node size the report states for its SFT runs, that is 448 against 32 GPU-hours (Module 26, derived). Figure 2 plots these next to the Qwen3 pair that Section 7 uses.
3.2 The quality side is not
The PPO-favoring studies and the DPO-favoring ones differ in what they tuned. Tulu 3's authors say their PPO had an untuned reward model and that "it is entirely possible that we can push up the PPO's performance even higher"1. Xu et al. tuned PPO heavily and compared it with DPO trained on the dataset's static correct and incorrect programs; their DPO produced "many meaningless code snippets" after one epoch3. Neither DPO there carries Llama 3's patches: the NLL term of coefficient 0.2, the masking of formatting tokens that stopped tail repetition, and fresh preference batches each round2 (Module 7 collects these). On a general chat target the PPO margin is up to 1.2%, against up to 8% for data quality20; on a code target with a test-suite reward the margin is 19.2 points at 34B (22.4 minus 3.2 for the better DPO variant, derived)3. Both statements are true. What moves the answer is whether the reward can be checked, not the algorithm's name.
Lambert's synthesis is the third view: "despite the simplicity of alternatives, RL-based methods continue to win out", at a far larger infrastructure cost, and RL can be surgical, since Tulu 3's math-only RL did not squash general capability22 (§3.1.6). That is a statement about verifiable skills, and Section 5 picks it up.
4. Critic memory and tuning burden
4.1 What a critic costs
Mixed-precision Adam holds \(2\Psi + 2\Psi + K\Psi\) bytes per trained model with \(K=12\), which is \(16\Psi\) bytes for \(\Psi\) parameters23; a frozen model in half precision holds \(2\Psi\). PPO-RLHF trains a policy and a value model and keeps a frozen reference and reward model: \(16+16+2+2=36\Psi\), or 2,520 GB at 70B. PPO-RLVR as Tulu 3 ran it has no reward model: \(34\Psi\), 2,380 GB. GRPO with a verifier and \(\beta=0\) holds only the policy, \(16\Psi\), 1,120 GB, because TRL's default does not load the reference when the KL coefficient is 0.024 (weights and optimizer state only, derived; Module 1, Table 6 has the full list). The value model's share of PPO-RLHF is \(16/36=44\%\). ReMax measured 46% of GPU memory for the value model and its training state at 7B, and 4% for the reward model11; the derived reward-model share is \(2/36=5.6\%\). The arithmetic and the measurement agree to within two points, and the measurement includes activations.
4.2 What it buys, and why the sign flips
Three things separate the rows. The first is horizon. VinePPO's critic wins are on responses capped at 1,024 tokens8, and the only long chain-of-thought critic win needs seven modifications, among them value pretraining, decoupled GAE and a length-adaptive \(\lambda\); without value pretraining VAPO scores 11, and plain PPO scores 512. The second is tuning. R1's appendix reports that PPO at the common default \(\lambda=0.95\) "performs considerably worse than GRPO", and that only \(\lambda=1.0\) brings it "nearing that of GRPO"4 (Supp. A.3). A fixed \(\lambda=0.95\) multiplies the terminal reward by \(0.95^{100}\approx0.006\) for a token 100 steps from the end (derived from the argument in VAPO §3.112), which is why long-horizon critics learn almost nothing from the reward. The third is the baseline PPO that loses: in Ahmadian et al.'s run PPO's mean output was 16.5 tokens against 60.6 for RLOO on HH-Llama15, which looks like a collapsed run (my reading), so that margin shows PPO's fragility in a short, cheap setup and says little about a tuned critic.
Tulu 3 is outside the figure because it compared critics with each other. Its value model, initialized from a general reward model, gave the best GSM8K and average scores in its ablation of initializations, and the report lists GRPO as untested future work1 (§6.2.1, Limitations). VinePPO adds a middle path, Monte Carlo value estimates from the same states with no value network: 46.0 against PPO's 42.8 on 7B MATH, at up to 2 times the per-iteration cost at 7B8.
5. Reasoning with verifiers: what the recipes converge on
With a verifier and a base or SFT model, the 2025 to 2026 recipes look alike in some columns and differ in others. Table 4 puts the shipped defaults side by side; the agreement set is the nearest thing this course has to a default, and the disagreeing columns are where your first ablation budget should go.
| Recipe | Loss aggregation | Advantage | KL | Clip | Mismatch, staleness | Group filtering | Scale |
|---|---|---|---|---|---|---|---|
| TRL24 | token, over active tokens in the global batch | ÷ group std | \(\beta=0\), reference not loaded | \(\varepsilon=0.2\); 0.28 suggested | vLLM correction on: sequence-level masked IS | none; zero-std share logged | \(G=8\) default |
| DAPO16 | token | ÷ group std | none | 0.2 / 0.28 | not addressed | dynamic sampling; overlong filtering and soft penalty | Qwen2.5-32B; AIME24 30 to 50 |
| Olmo 3 Think5 | token, over the batch | mean-centered, no std | none | 0.2 / 0.272 | truncated IS, cap 2.0 on 32B | zero-gradient filtering; active sampling | 7B, 32B dense |
| ScaleRL9 | prompt | batch-level std | none | CISPO weight cap | FP32 logits; async, max lag \(k=8\) | zero-variance filter; drop prompts with pass rate \(\ge0.9\) | 8B dense, 17B×16 MoE |
| DeepSeek-V3.226 | per-sequence \(1/|o|\) as printed | \(R-\text{mean}\), no std | weak or none for math; unbiased K3 estimate | PPO clip, \(\varepsilon\) not stated in the sections read | off-policy sequence mask; Keep Routing; Keep Sampling Mask | not stated | RL above 10% of pre-training cost |
| GLM-5 (§3.2)13 | per-sequence \(1/|y|\) | ÷ group std | none | 0.2 / 0.28 | IcePop, \(\beta=2\); deterministic top-k | difficulty filter on prompts | 744B; \(G=32\), on-policy |
The agreement set reads off the columns. Five of six drop the KL term or set it to zero, and the sixth, V3.2, keeps a corrected term with domain-dependent strength26. Four of six widen the upper clip or replace it with CISPO. Five of six correct or remove the mismatch between the rollout engine and the trainer; DAPO predates that literature. Three filter zero-variance groups. The columns that disagree are aggregation (token, prompt or sequence) and advantage scale (÷ group std in TRL, DAPO and GLM-5; none in Olmo 3 and V3.2; batch-level in ScaleRL). Module 11 weighs those two; the controlled evidence is small and says second-order. DAPO's cumulative ladder moves AIME24 30, 36, 38, 41, 42, 50 across overlong filtering, Clip-Higher, soft overlong punishment, token-level loss and dynamic sampling, and the token-level step is +116.
5.1 What each shared choice rests on
- No KL. adopted Olmo 3 states that removal "does not lead to over-optimization or destabilized training" and shows no with-and-without run5. ProRL argues the opposite for a distilled 1.5B start, keeping KL with periodic resets17. Module 13's ledger finds no ablation behind any removal.
- Clip-Higher, CISPO. single Each has a controlled comparison in its own paper (DAPO's ladder; ScaleRL's forward tests), with no cross-comparison on one codebase (Module 12).
- Mismatch correction. concordant that it is needed: runs collapse without it in more than one group, and a zero-mismatch control isolates the cause2713. conflicting on how: loss-side weights, FP16 or an FP32 head, or deterministic kernels (Module 14).
- Zero-variance filtering. concordant across DAPO, Olmo 3 and ScaleRL, which all drop dead groups1659.
Magistral and MiMo-V2-Flash are pointers only: neither report was read for this course, so no cell above rests on them (Modules 13 and 22 mention each).
5.2 Efficiency or ceiling
ScaleRL, the one study that fits compute curves, concludes that loss aggregation, normalization, curriculum and the off-policy algorithm "primarily modulate compute efficiency without materially shifting the asymptote", and its leave-one-out reversions all land within \(\pm0.02\) of the fitted ceiling \(A\) (Appendix A.8)9. The same paper is not consistent about the off-policy scheme: Appendix A.11 says the choice "affects the asymptotic performance" and not only the efficiency exponent, and Figure 6 reports that zero-variance filtering reaches a higher asymptote. Treat "details only move speed" as supported for loss aggregation and normalization on one 8B dense model, and not for filtering or the off-policy scheme (Module 15). Olmo 3 ablates only active sampling among its seven changes to GRPO (Module 31).
6. Agentic coding and tool use
Agentic RL is verifier RL plus three costs: the rollout is a multi-turn episode in a sandbox, its length varies by an order of magnitude inside one batch, and the sampler and trainer must agree on the exact tokens. The generation bill is the first-order cost. Olmo 3's 32B run used 8 H100 nodes for training and 20 for inference; its text says inference takes about 5 times the GPU use of training, but 20 nodes against a learner busy 25% of the time gives 10, while the 7B figure of about 14 does follow (Module 1 flags both)5. Table 5 lists the design choices the agentic reports made and what backs each.
| Choice | Options in the reports | Evidence | Tag |
|---|---|---|---|
| Credit unit | Trajectory-level outcome reward (GLM-5, Kimi K3, V3.2, Nemotron's SWE stage); step-level groups from repeated states (GiGPO) | GiGPO: "> 12%" on ALFWorld and "> 9%" on WebShop over GRPO at 1.5B to 7B29; no frontier report ablates trajectory-level against step-level | adopted / single |
| Staleness | Tolerate: K3's per-token regularization19. Cap: GLM-5 discards a sample whose oldest rollout version is too stale13. One-step lag: Nemotron | No lag sweep or staleness histogram in the passages read; GLM-5's threshold is not stated | adopted |
| Rollout unit | End-to-end episodes; or single-turn rollouts "similar to PivotRL" for most agentic tasks in distillation | Nemotron saw "substantial training inefficiency" from end-to-end rollouts in MOPD; whether they gain more "remains an open area"25 | single |
| Token fidelity | TITO gateway: pass token ids, never re-tokenized text | GLM-5 (§4.1.2) introduces it against "re-tokenization mismatches"13 | single |
| Determinism | Deterministic torch.topk in the sparse-attention indexer; indexer frozen | Non-deterministic kernels: drastic degradation after a few steps with an entropy drop13 | single |
| Harness | Several harnesses per task | K3: a single harness "can cause a model to overfit to a particular tool schema" (§4.2.1)19; Nemotron trains each task distribution under at least two of six harnesses (Appendix A.2)25 | adopted |
| Environment mix | Synthesized agent tasks alongside code and search | V3.2's only environment ablation: RL on synthesized general-agent tasks improved τ²-Bench, MCP-Mark and MCP-Universe, while RL on code and search alone did not26 | single |
Scale of environments is the other reported lever: GLM-5 trains on more than 10K real-world software-engineering environments plus terminal and multi-hop search tasks13. Nemotron 3 Ultra's report says it is open-sourcing "the training recipes, data, and RL environments"; the Nemotron repository README lists the program as Pretrain, SFT, RLVR, MOPD and MTP boosting but says "this recipe covers Pretrain → SFT", that it trains "exclusively on the open-sourced subset of training data", and that results "will differ from the tech report benchmarks, which used additional proprietary data"30. So the openness of the RL environments is disputed between the report and the repository, and I do not count Nemotron as an open recipe for the RL stages (Module 32).
7. Small models: distill first
For a student of 32B or below with a stronger teacher available, two independent groups report that a teacher's signal beats RL on the student, and a third reproduces the direction with its own accounting. DeepSeek trained Qwen2.5-32B with RL for more than 10K steps and reached AIME 2024 pass@1 of 47.0, on par with QwQ-32B-Preview (50.0), while distilling R1 into the same base scored 72.64 (Supp. F.1, Table 16). The distillation there is off-policy SFT on 800K samples for 2 to 3 epochs4. Qwen3 ran the on-policy comparison; Table 6 sets the routes side by side.
| Route | AIME'24 | AIME'25 | GPU-hours | Source and limit |
|---|---|---|---|---|
| RL only, Qwen2.5-32B | 47.0 | n/a | not stated | R1 Table 16: over 10K steps4 |
| SFT on R1 samples, 32B | 72.6 | n/a | not stated | R1 Table 16: 800K samples4 |
| + RL, Qwen3-8B | 67.6 (90.0) | 55.5 (83.3) | 17,920 | Table 21; hardware and recipe not stated6 |
| + on-policy distillation, Qwen3-8B | 74.4 (93.3) | 65.5 (86.7) | 1,800 | Table 21; teacher cost not stated6 |
| RL after distillation, 1.5B | n/a | n/a | about 16,000 | ProRL: over 2,000 steps, 4 nodes of 8 H10017; no matched OPD arm |
Thinking Machines reproduces the direction with its own accounting: from a 400K-example SFT checkpoint a Qwen3-8B-Base student reaches 70% on AIME'24 in about 150 on-policy steps, where SFT is extrapolated to need about 2M prompts, and it reports cost reductions of 9 times with the SFT data given, about 18 times in GPU-hours, and about 30 times with teacher sampling counted, all relative to the extrapolated SFT baseline; its table lists RL at about 1 times that baseline. The page used Qwen3-8B as the teacher but counted the FLOPs of the 32B model10.
7.1 A worked cost per point, and what the teacher would have to cost
Qwen3's ratio is 17,920/1,800 = 9.96. A fairer unit is GPU-hours per point of gain, since distillation also gained more. From the 55.0 starting score, RL gained 12.6 points (67.6 minus 55.0) and distillation 19.4 (74.4 minus 55.0). Per point, RL cost \(17{,}920/12.6=1{,}422\) GPU-hours and distillation \(1{,}800/19.4=92.8\), a ratio of 15.3 (derived). Now charge the teacher. Distillation matches RL's cost per point when \((1{,}800+T)/19.4 = 1{,}422\), so \(T\approx25{,}800\) GPU-hours, 14.3 times the distillation run itself. The report states neither the hardware nor whether the teacher is inside the 1,800, and the teacher is the product of four RL-shaped stages that Qwen3 never prices. It does serve six lightweight students (five dense models and one mixture-of-experts)6, so the calculator lets you spread a guessed teacher cost over \(n\) of them. The guess is an assumption, not a source.
GPU-hours per AIME'24 point, Qwen3-8B: RL against on-policy distillation with the teacher charged
Constants from Qwen3 Table 216: 17,920 and 1,800 GPU-hours; gains of 12.6 and 19.4 AIME'24 points over the 55.0 start. \(T\) is not in the report. Single run, 30 questions per AIME set; a one-question change in pass@64 is 3.3 points.
8. Tasks without a verifier
When no program decides correctness, the choice is which model sits in the reward path and how its exploit is guarded. Table 7 routes by task property. The reports agree on the routing and on the guards' existence; none of the model reports gives a hack rate with and without its guard (Module 19), so every guard below is a design the lab states, not a measured effect.
| Task property | Source | What the report states | Guard |
|---|---|---|---|
| A rule or test decides | Verifier | Qwen3: rule-based rewards "assess the correctness of model outputs with high precision, preventing issues like reward hacking"6 | Module 19, Section 9 |
| An answer exists, forms vary | Reference-conditioned judge | Qwen3: a 72B judge sees the reference; a preference model only for queries with none6 | Keep rule checks where possible |
| No reference, many criteria | Rubric judge | RaR: up to 31% (HealthBench) and 7% (GPQA-Diamond) over Likert judges, the authors' numbers18; V3.2 uses per-prompt rubrics26 | Several or rotating rubrics; veto items |
| Open-ended preference | Pairwise or binary-comparison judge | K2: the critic ranks outputs by pairwise comparison and is refined on verifiable rollouts31. K3: "the tournament-style group reward with binary comparisons as in Kimi K2.5"19 | K3: an output longer than \(\sigma\ell_0\) loses the comparison automatically; K2: a hack-check layer |
| Last-mile polish | Scalar preference model | R1 applies it only to the last 400 of 1,700 stage-2 steps4 | Short exposure; length-matched pairs |
| Judge must grow with the policy | Actor as judge | V4: RL applied to the generative reward model itself, with "minimal" human annotation (§5.1.1)32 | None stated; vendor claim |
Two cautions on the evidence. K3 states its aggregation only as the quoted sentence and adds a protocol of reading the output, writing a rubric, scoring each candidate and recording the scores in a scorepad19; the K2.5 text I read does not use the word "tournament", and K2's equivalent is the pairwise ranking above, so I claim nothing beyond the K3 sentence. And RaR's gains come from a 7B policy, point estimates, and a training judge that also grades the benchmark (Module 16). V4's actor-as-judge and K2's closed-loop critic have no published comparison with a frozen judge, so both are vendor claims. R1 observed the failure that justifies a cap: with a preference reward, reward rose while Codeforces performance fell (Supplementary B.5; Module 19).
9. Several skills in one model
Reasoning, agent, chat and safety skills interfere when trained together, and the reports answer in three ways (Figure 4). V3.2 merges reasoning, agent and alignment into one mixed RL stage to avoid "the catastrophic forgetting issues commonly associated with multi-stage training paradigms"26. GLM-5 runs three RL stages in sequence and ends with on-policy cross-stage distillation from its own earlier checkpoints, because "sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities"13 (§3.5). Kimi K3, DeepSeek-V4 and Nemotron 3 Ultra train specialists and distill them on the student's own rollouts: nine experts for K3 (three domains at three effort levels), more than ten for V4 and for Nemotron193225. Lambert's survey records the shift, notes that MiMo-V2-Flash documented it first, and writes that "labs are far from unanimous in its usage"22 (§3.2.4).
| Regime | Approach | Evidence | Tag |
|---|---|---|---|
| Few similar domains, one team | One mixed RL stage | Mix-RL 0.882, the flattest baseline (range 0.064) and 5.5 points behind MOPD; V3.2 and Qwen3's last stage run mixed RL at frontier scale6 | single |
| Serial stages unavoidable | Stages, then distillation from your own earlier checkpoints | GLM-5 §3.5; the controlled cascade alone scored 0.775, with Math closing only 57% of its headroom | single |
| Many domains, agentic ones, parallel teams | Same-origin specialists, then sampled-token MOPD | 0.937, domains within 0.91 to 0.95; adopted by K3, V4, Nemotron; Nemotron's stated reason is per-domain dilution in a mixed batch (§3.3)25 | single + adopted |
| A skill the student cannot sample | MOPD does not import it | Nemotron HLE: recovery 16.9% of the teacher gap, the teacher's gains coming from separate SFT and RL data the student never saw (Table 5)25 | single |
Only one controlled comparison exists, and its numbers are the authors' own. Cascade RL's Math score falls during the later software-engineering stage; off-policy finetuning exceeds the teacher on instruction following (1.01) but closes only 65% of the software-engineering gap7. No frontier report adds a matched-compute ablation against the method it replaced: V4 swaps V3.2's mixed RL for distillation, but the models also differ in architecture and data (Module 30). Teacher choice has a measured failure: replacing the Math RL teacher with the far larger Qwen3-235B-A22B raised the initial per-token KL from about 0.04 to about 0.19 and degraded Math under both loss variants, the top-k variant diverging near step 187 (§4.4.2). A date note on V4: its arXiv identifier 2606.19348 has a 2606 prefix, while the PDF stamp and the API date say 26 April 2026; I could not resolve this, and it does not touch the figures used here.
10. Calibration and abstention
A binary correctness reward pays for guessing: it does not penalize low-confidence wrong answers, so RLVR can degrade calibration14, and under 0-1 grading an abstention is strictly suboptimal33. The public repairs act at different layers (Table 9). The strongest training result is a bounded proper scoring rule added to the verifier reward: RLCR's reward is \(R=\mathbb 1[y=y^*]-(q-\mathbb 1[y=y^*])^2\), where \(q\) is the model's stated confidence, and any bounded proper rule gives both a calibration incentive and a correctness incentive, while the log score is unbounded and can reward wrong answers14 (§3). The theorem's appendix proof carries a sign slip that I do not rely on; Module 24 checks the claim separately.
| Lever | Measured result | Cost or limit | Tag |
|---|---|---|---|
| Brier term on stated confidence (RLCR) | ECE 0.37 to 0.03 (HotpotQA), 0.26 to 0.10 (math); accuracy 63.0% against 62.1%, interval half-width ±3.05 | OOD ECE still 0.21 to 0.25; 249 tokens against 92, +157 per answer; no abstain action | single |
| A second model reads confidence | BCE classifier ECE 0.07 in-domain, 0.24 OOD, on the RLVR model's outputs | A second 7B model; the policy never abstains | single |
| Threshold-conditioned reward (Behaviorally Calibrated RL) | Abstract: a 4B model's log-scale Accuracy-to-Hallucination Ratio gain 0.806 against GPT-5's 0.207 on BeyondAIME, the authors' evaluation34 | Abstract only in the source map; Module 24 reports the policy did not become sensitive to the threshold at 4B | single |
| Change the grading (Kalai et al.) | Of ten audited evaluations only one, WildBench, gives minimal credit for uncertainty; proposed penalty \(t/(1-t)\) for a wrong answer at confidence target \(t\)33 | No training experiment; depends on leaderboard owners; the paper's \(t=0.75\) example misprints the penalty, so use the formula | single |
| TypeSafe "RLCD" | Company statements only35 | No paper, repository, reward function, algorithm or dataset | no source |
TypeSafe AI's "Reinforcement Learning for Calibrated Decisions" is a company claim. Its 15 September 2026 launch post names the method in one sentence and says it optimizes "calibrated decisions: answers with epistemically honest probabilities"35; on the Latent Space podcast the founder, asked whether it was published, answers "No, not yet"36. I give no recommendation for or against it and describe no mechanism, since no source discloses one (Module 25). It is not Yang et al.'s Reinforcement Learning from Contrastive Distillation, which shares the acronym and builds preference pairs from contrasting prompts37.
11. The constraint-by-method matrix, and the cells with no source
| Method | Needs | Resident state, 70B (GB) | Generates? | Stability and tuning | Best-measured target | Tag |
|---|---|---|---|---|---|---|
| DPO, cached reference | Pairs | 1,120 | No | Patches needed: NLL 0.2, token masking, length normalization21 | Chat to 405B2 | conflicting |
| PPO with critic | Reward model or verifier | 2,380 to 2,520 | Yes | \(\lambda=1.0\) needed4; seven fixes for long chains12 | Code with tests3; short-response math8 | conflicting |
| RLOO / REINFORCE | Reward model; \(k\ge2\) samples | 1,400 | Yes | No clip or ratio in the estimator15 | 7B-class chat, 512 tokens15 | single |
| GRPO family, verifier, \(\beta=0\) | Verifier | 1,120 | Yes, 8 to 32 per prompt | Zero-variance groups; mismatch2427 | Math, code, agents513 | adopted |
| On-policy distillation | Teacher, prompts | 1,120 plus teacher | Yes, plus teacher scoring | Same-origin teachers stable; external teacher diverged7 | Small students6 | concordant |
| Multi-teacher distillation | One RL teacher per domain | 1,120 plus teachers | Yes, plus scoring | Cannot import skills the student never samples25 | Many-domain integration7 | single |
| Verifier plus Brier term | Verifier | 1,120 | Yes, +157 tokens | Bounded rule required14 | Short answers, 7B14 | single |
| Decision | Missing evidence | Module |
|---|---|---|
| Tuned PPO against patched DPO at 70B or more | Matched data and compute, held-out human preference, seed intervals | 7, 26 |
| Critic against group baseline | Equal GPU-hours including the critic, two response lengths | 9 |
| Normalization grid | Aggregation by advantage scale, three seeds, short and very long caps | 11 |
| Mismatch fix families | One mixture-of-experts model, zero-mismatch reference, serving-engine evaluation | 14 |
| Frozen against jointly trained judge | Equal compute, scored by humans or a verifier slice | 16 |
| Staleness tolerate against cap; step-level against trajectory credit | Lag sweep; GiGPO-style keys on SWE at 30B or more | 18 |
| KL on rubric-judged domains | Same run with and without the term | 30 |
| RL against distillation, teacher counted | One start, matched total GPU-hours, pass@k over more than 30 problems | 21 |
| Mixed RL against MOPD at ten or more domains | Equal GPU-hours from one SFT checkpoint | 22 |
| Calibration at frontier scale | Agentic or noisy-verifier tasks; independent replication | 24 |
| TypeSafe's RLCD | Everything: no paper, reward, algorithm or dataset | 25 |
12. What the evidence supports, and what it does not
Supported by at least two independent groups. Offline DPO costs far less than PPO for chat tuning, 14 times the node-hours in the one measured pair, and Llama 3 reports the same direction12. Data quality moves preference tuning more than the algorithm205. A teacher's signal beats RL for a small student on cost and pass@1 in each direct comparison46. A mismatch between rollout and trainer engines can collapse a run and must be handled2713.
Supported by one controlled study, or by adoption alone. Specialists plus MOPD beating mixed RL (one study, three domains). The group-baseline default for long chains of thought (adopted; the one critic win there needs seven fixes). Dropping KL (adopted; no with-and-without run). A Brier term for calibration (one paper, 7B). Rubric judging (design statements, no hack-rate ablation).
Not supported by anything read. That any algorithm in this module is best at frontier scale: the frontier reports ablate recipes against their own earlier versions, not against rivals. That RL "expands" capability, or that distillation does so more cheaply, beyond the one-question margins in Table 6 (Module 20). That any teacher cost, hardware, or seed variance is known for the Qwen3 pair. Anything about how TypeSafe's RLCD trains. The numbers are also dated: TRL's defaults, report versions and a V4 identifier that conflicts with its own date will move, and Module 34 lists what to re-check first.
Where experts disagree
The five disputes below are the ones that decide a starting method. Each is treated at length in the module named; here each is reduced to the condition under which one side is the better start.
A. Is DPO enough, or does online RL win for preference tuning? (Modules 7, 26)
DPO in rounds is enough
Llama 3 team; Tulu 3 authors21
Comparable score at 14 times fewer node-hours; spend the saving on data.
- Measured cost; used to 405B.
- Untuned PPO; no PPO numbers published by Meta.
Tuned online RL wins
Xu et al.; Ivison et al.; Tang et al.32021
PPO is ahead everywhere tested, most of all with a checkable reward.
- Three groups agree on direction.
- Margins up to 1.2% on chat against up to 8% for data.
Both hold in their regimes. Without a checkable reward, patched DPO is the efficient start; with one, DPO's 0.0% on CodeContests settles it for RL. A matched 70B study would settle the chat case.
B. Is a value model worth it? (Module 9)
No critic
Ahmadian et al.; DeepSeek; most open reasoning recipes154
Last-token rewards give a critic little to learn; the group is a free baseline.
- Saves about 44% of state in PPO-RLHF (derived).
- Behind a tuned critic by 3.6 to 6.4 points at 7B.
A built critic has the higher ceiling
Per-step credit and lower variance repay the cost once the critic is fixed.
- 60.4 against DAPO's 50 on AIME24.
- Seven fixes; plain PPO scored 5; unreplicated.
Group baseline for long chains with a limited engineering budget; a critic only with VAPO-grade fixes or when credit across steps is the bottleneck. R1's \(\lambda\) result shows the gap is partly tuning.
C. How should the GRPO loss and advantage be normalized? (Module 11)
Remove the biases: no std, constant or token-level loss
Dr. GRPO; Olmo 3; V3.2 on the advantage38526
Per-response length and std division reweight the objective.
- Unbiased; fewer knobs.
- Small-scale evidence; reward scale sets step size.
Normalization changes speed, not the ceiling
ScaleRL9
Choose prompt-level averaging and batch-level scale; spend effort on loss type and numerics.
- Reversions within \(\pm0.02\) of the ceiling.
- One 8B model; its own A.11 and filtering results move the ceiling.
Fix the length term first, since it costs nothing; do not spend an ablation budget on the std question before loss type and mismatch. The grid that would settle it does not exist.
D. Mixed RL, stages, or specialists plus distillation? (Module 22)
One mixed stage
DeepSeek-V3.2; Qwen3's last stage266
One trainer, one recipe, no forgetting by construction.
- Flattest non-distillation baseline (0.882).
- Per-domain signal dilutes as environments multiply.
Specialists, then MOPD
MOPD paper; Kimi K3; DeepSeek-V4; Nemotron7193225
Train each skill to its peak, then merge on the student's rollouts.
- 0.937 and the flattest profile.
- Needs same-origin teachers; HLE recovery 16.9%; one controlled study.
The count of domains and the availability of parallel teams decide it. "Specialists win at frontier scale" is an inference from adoption, as Lambert's "far from unanimous" implies22.
E. RL or distillation for a strong small reasoning model? (Module 21)
Distill first
DeepSeek-R1 authors; Qwen3 team46
A dense teacher signal gives higher pass@1 at about a tenth of the GPU-hours.
- Direction agrees across off- and on-policy variants.
- Teacher cost uncounted; single runs.
RL remains the path past the teacher
R1 authors, same appendix; ProRL417
Distillation cannot exceed its teacher; prolonged RL from a distilled start reports gains.
- The only route beyond the frontier.
- About 16,000 GPU-hours at 1.5B; no matched OPD arm.
Distillation copies a capability; RL is the option when no stronger teacher exists. Cost alone does not separate them once the teacher is charged (Section 7.1).
What is changing
Four inputs to this module are likely to change first: TRL's defaults (its loss menu already lists eight loss types)24; the MOPD recipe family, which Lambert's survey reports with different expert counts per lab22; the open status of Nemotron's RL stages, which depends on a repository that says its recipe covers Pretrain and SFT30; and the Qwen3 teacher accounting. Module 34 carries the dated map of what is settled, contested and open.
References
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025 (v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the PPO-versus-DPO controlled comparison, about 28 h on two nodes against about 4 h on one, "comparable (albeit slightly lower)", and the untuned-setup caveat (§5.4.1); length-normalized DPO result (Table 18); RLVR definition, value-model initialization and 354,192 preference instances (§1, §6.2.1, Limitations).
- paperGrattafiori, A., Dubey, A., Jauhri, A., et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024 (v1 31 Jul 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: the stated stance on RL stability (§1), the PPO comparison and DPO settings (learning rate 1e-5, beta 0.1, NLL coefficient 0.2, token masking) in §4.1.4, six rounds (§4.1.6).
- paperXu, S., Fu, W., Gao, J., Ye, W., et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-27. Used for: CodeContests 10@1k results at CodeLlama-34B (Table 8: SFT 15.2, DPO 0.0, DPO-Iter 3.2, PPO 22.4) and the APPS results (Table 7); "many meaningless code snippets" for DPO (§6). Added beyond the source map (full text; the map had the abstract).
- paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026 (expanded Nature version); v1 22 Jan 2025. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the PPO-versus-GRPO comparison at GAE lambda 0.95 and 1.0 (Supp. A.3, Figure 4), distillation against RL on Qwen2.5-32B (Supp. F.1, Table 16: 47.0 and 72.6 on AIME 2024; 800K samples), the preference-reward window of the last 400 of 1,700 steps (§3.2.2) and reward hacking (Supp. B.5).
- paperOlmo Team. Olmo 3. arXiv v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL changes and "no KL" statement (§4.4.1), the 75% learner wait and footnote 41's 125 s of a 1,000 s step, node counts (§4.4.3), prompt counts (Tables 17, 19, 20), the delta-learning result (Table 21).
- paperQwen Team (Yang, A., et al.). Qwen3 Technical Report. arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: 3,995 query-verifier pairs and the four selection criteria (§4.2), the rule-based reward statement and judge types (§4.4), strong-to-weak distillation for five dense models and one mixture-of-experts model, and Table 21 (§4.7): AIME'24/'25, pass@64 and GPU hours 17,920 and 1,800.
- paperMa, W., et al. (Xiaomi LLM Core). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-26. Used for: the controlled comparison on Qwen3-30B-A3B (normalized scores 0.937, 0.882, 0.824, 0.775; Math 57% in the cascade; per-domain ranges), same-origin teachers and the external-teacher failure (§4.4.2), the framing of Mix-RL, Cascade RL, Off-Policy Finetune and Param-Merge (§1).
- paperKazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., Le Roux, N. VinePPO: Refining Credit Assignment in RL Training of LLMs. ICML 2025; arXiv v2, 3 Jun 2025. https://arxiv.org/abs/2410.01679. Accessed 2026-10-27. Used for: Figure 4 (DeepSeekMath-7B MATH pass@1: PPO 42.8, VinePPO 46.0; GSM8K and MATH for RLOO and GRPO), the 1,024-token response cap (appendix hyperparameters), per-iteration slowdown up to 2x at 7B and 5x at 1.1B. Added beyond the source map.
- paperKhatri, D., et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: over 400,000 GPU-hours, the ScaleRL recipe (§1), efficiency-versus-asymptote conclusion (abstract), leave-one-out ±0.02 (App. A.8), zero-variance filtering and the pass-rate curriculum (Figure 6), PipelineRL k = 8 and the asymptote statement (§3.1, App. A.11).
- blogLu, K., et al. (Thinking Machines). On-Policy Distillation. Blog post, 27 Oct 2025; later update notes models retired. https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-26. Used for: about 150 steps to 70% on AIME'24, SFT extrapolated to about 2M prompts, and the 9x, 18x and 30x cost reductions in FLOPs and GPU-hours; reuse of the Qwen3 table. Quotes from the saved page text.
- paperLi, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., Luo, Z.-Q. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML 2024; arXiv v4, 16 May 2024. https://arxiv.org/abs/2310.10505. Accessed 2026-10-27. Used for: the 46% GPU-memory share of the value model and its state and 4% for the reward model at 7B (footnote 1). Added beyond the source map.
- paperByteDance Seed. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: Table 1 (vanilla PPO 5, DAPO 50, VAPO without value pretraining 11, VAPO 60.4 on AIME24 with Qwen 32B) and the seven modifications (§4, §5). Added beyond the source map.
- paperGLM-5 Team (Zeng, A., Lv, X., et al.). GLM-5: from Vibe Coding to Agentic Engineering. arXiv v2, 24 Feb 2026 (v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the reasoning-RL loss with IcePop, beta = 2, epsilon 0.2/0.28, group 32, KL removed, and the indexer top-k finding (§3.2); agentic RL, TITO gateway, Direct Double-sided Importance Sampling and staleness discard (§3.3 pointer, §4.1, §4.1.2); on-policy cross-stage distillation (§3.5).
- paperDamani, M., Puri, I., Slocum, S., et al. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). ICLR 2026; arXiv v2, 15 May 2026 (v1 22 Jul 2025). https://arxiv.org/pdf/2507.16806v2. Accessed 2026-10-26. Used for: the reward with the Brier term (Eq. 1), the bounded-proper-score argument (§3), Table 1 and Table 2 numbers (ECE, accuracy, tokens, classifier baseline), and the log-score unboundedness caveat. The proof of Theorem 1 carries a sign slip in the appendix (Module 24).
- paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: Table 1 win-rates (RLOO k=4 77.9, 43.7, 64.1; PPO 67.6, 29.2, 32.0; DPO 66.6, 39.0, 61.9), Table 2 output lengths (PPO 16.5 tokens on HH-Llama), the 512-token context (§4). Margins derived; the intro's "3.2% to 20.3%" is not reproduced from the table (Module 9).
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. (ByteDance Seed). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: the cumulative ladder of Table 1 (AIME24 30, 36, 38, 41, 42, 50) and the techniques behind it; no-KL decision (§2.3).
- paperLiu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., Dong, Y. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: KL control with reference resets from a distilled 1.5B start (§2.2), over 2,000 steps, 4 nodes of 8 H100 and about 16k GPU-hours (§3.2).
- paperGunjal, A., Wang, A., Lau, E., Nath, V., et al. (Scale AI). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv v2, 3 Oct 2025. https://arxiv.org/abs/2507.17746. Accessed 2026-10-26. Used for: gains of up to 31% (HealthBench) and 7% (GPQA-Diamond) over Likert judges, the authors' numbers (abstract); the 7B-policy and shared-judge caveats (Module 16).
- paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.). Kimi K3: Open Frontier Intelligence. arXiv v2, 7 Aug 2026 (v1 27 Jul 2026). https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: nine experts, three domains at three effort levels (§4.1.1); the agentic generative reward model and its verbosity control (§4.1.2); multi-teacher on-policy distillation (§4.1.3); partial rollouts and per-token regularization; the single-harness remark (§4.2.1).
- paperIvison, H., Wang, Y., Liu, J., Wu, Z., et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv v2, 7 Oct 2024. https://arxiv.org/abs/2406.09279. Accessed 2026-10-27. Used for: the ranking of factors (data, algorithm, reward model, prompts), PPO ahead by up to 2.5% in math and 1.2% in general domains, data gains up to 8% (abstract; the stack is not in the abstract).
- paperTang, Y., Guo, D. Z., Zheng, Z., Calandriello, D., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv v1, 14 May 2024. https://arxiv.org/abs/2405.08448. Accessed 2026-10-27. Used for: the online-versus-offline gap that scaling the policy does not close (abstract). Added beyond the source map.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026; web edition built 14 Oct 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: "RL-based methods continue to win out" and the surgical-RL remark (§3.1.6); the 2026 MOPD adoption, expert counts and "labs are far from unanimous" (§3.2.4).
- paperRajbhandari, S., Rasley, J., Ruwase, O., He, Y. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020; arXiv v3, 13 May 2020. https://arxiv.org/abs/1910.02054. Accessed 2026-10-27. Used for: mixed-precision Adam memory, 2 + 2 + K bytes per parameter with K = 12, 16 bytes in total (§3.1). Added beyond the source map.
- officialHugging Face TRL maintainers. GRPO Trainer (documentation, main branch; no version stated). Live page, fetched twice. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: GRPOConfig defaults and options: loss_type (dapo default; dr_grpo, bnpo, cispo, sapo, luspo, vespo), beta 0.0 with the reference not loaded, epsilon 0.2 and epsilon_high, scale_rewards, num_generations 8, vLLM importance-sampling correction defaults. Snapshot of a moving page.
- paperNVIDIA (Blakeman, A., Thomas, A., et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: unified RLVR with global batch 8192 and 16 rollouts per sample (unit unstated), dilution argument and more than ten teachers (§3.2, §3.3), the single-turn rollout limitation (§3.3), Table 5 HLE recovery 16.9%, harness list (Appendix A.2).
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the advantage R minus mean (§3.1), the unbiased K3 estimate and domain-dependent KL strength, off-policy sequence masking, Keep Routing and Keep Sampling Mask, the single mixed stage (§3), RL above 10% of pre-training cost (§1), the synthesized-task ablation (§4.3).
- paperQi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., Lin, M. Defeating the Training-Inference Mismatch via FP16. arXiv v1, 30 Oct 2025. https://arxiv.org/pdf/2510.26788v1. Accessed 2026-10-26. Used for: the mismatch collapse under BF16 with vLLM and FSDP and the FP16 remedy (§1; Module 14 holds the dispute).
- paperMiniMax (Chen, A., Li, A., et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: repetition-based early truncation and the combination of sample-level loss with token-level normalization in the 80K-token run (§3.2, §4.2).
- paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training (GiGPO). NeurIPS 2025; arXiv v3, 28 Oct 2025. https://arxiv.org/abs/2505.10978. Accessed 2026-10-26. Used for: anchor-state step-level groups and gains of more than 12% on ALFWorld and 9% on WebShop over GRPO (abstract).
- repoNVIDIA-NeMo. Nemotron repository README (training recipes). Live page. https://github.com/NVIDIA-NeMo/Nemotron. Accessed 2026-10-27. Used for: the Ultra entry (550B total, 55B active; program Pretrain, SFT, RLVR, MOPD, MTP boosting; "this recipe covers Pretrain → SFT") and the open-data-only notice. Added beyond the source map; raw file https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/README.md.
- paperKimi Team (Bai, Y., Bao, Y., Charles, Y., et al.). Kimi K2: Open Agentic Intelligence. arXiv v2, 3 Feb 2026 (v1 28 Jul 2025). https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: the self-critic reward by pairwise comparison, closed-loop critic refinement and the hack-check layer (§3.2.1).
- paperDeepSeek-AI (Xu, A., Lin, B., Xue, B., et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv id 2606.19348 v1; the PDF stamp and API date read 26 Apr 2026 (unresolved against the 2606 prefix). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: specialist training and the actor-as-generative-reward-model paragraph (§5.1.1), more than ten teachers and full-vocabulary distillation (§5.1.2).
- paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. Why Language Models Hallucinate. OpenAI and Georgia Tech, arXiv v1, 4 Sep 2025. https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-26. Used for: the 0-1 grading argument that abstaining is suboptimal, the proposed penalty t/(1-t) in the prompt (§4), and the audit of ten evaluations (Table 2, page 14). The printed t = 0.75 example is not reused.
- paperWu, J., Liu, J., Zeng, Z., Zhan, T., et al. Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. arXiv v3, 28 Jan 2026. https://arxiv.org/abs/2512.19920. Accessed 2026-10-26. Used for: the abstract's claim of a 4B model's log-scale Accuracy-to-Hallucination Ratio gain 0.806 against GPT-5's 0.207 on BeyondAIME (the authors' evaluation). Abstract only in the source map.
- blogTypeSafe AI (Almeida, D.). Introducing System One Models and Jev. Company blog, 15 Sep 2026 (page metadata shows an export date of 19 Oct 2026). https://typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed 2026-10-26. Used for: the one sentence naming RLCD and the comparison-table row "calibrated decisions: answers with epistemically honest probabilities on System One tasks". Company claims only.
- talkswyx (host) with Almeida, D. Jev: System One models for Prod, not God. Latent Space podcast, 2026; publication date not captured. https://latent.space/p/jev/. Accessed 2026-10-26. Used for: the founder's "No, not yet" on whether RLCD is published (transcript around 00:25:05, read from extraction output, not machine-verified).
- paperYang, K., Klein, D., Celikyilmaz, A., Peng, N., Tian, Y. RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. ICLR 2024; arXiv v3, 16 Mar 2024. https://arxiv.org/pdf/2307.12950v3. Accessed 2026-10-26. Used for: the 2023 method that shares the acronym; preference pairs from a positive and a negative prompt. Not TypeSafe's method.
- paperLiu, Z., Chen, C., Li, W., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the length and standard-deviation biases of GRPO and the unbiased variant (§3).