Reinforcement Learning for LLM Post-Training · Module 33

Choosing a Method: PPO, DPO, GRPO-Family, On-Policy Distillation, and Hybrids Under Reward, Data, Compute, and Stability Constraints

Each method in this course was built for a constraint: pairs without a reward model, a verifier without a critic budget, a teacher without RL compute. This module maps the constraints you have onto a starting method, states the evidence behind each choice with the stack, scale and strength it was measured at, says what would change it, and lists the choices that no source supports. Afterwards you can pick a starting method and the first three things to monitor for a given reward type, data, compute budget and target.

Course 45 min read 38 sources As of 2026-10-27 Prerequisites: Modules 7, 9, 11, 15, 21, 22, 24
Decision tree: the reward source selects the family of method to start with, and four modifiers move you off the default; each leaf carries the module that teaches it and an evidence tag Question What you hold Start with, teaching module, evidence tag What produces the reward? Pairs only chosen and rejected Reward model plus prompts Verifier programmatic, plus prompts Judge or rubric no reference answer Patched DPO, in rounds Modules 7, 26 · quality contested, cost settled conflicting RLOO first; tuned PPO if budget Modules 5, 9 · critic evidence points both ways conflicting Group baseline, token loss, no KL Modules 9 to 14 · widely used, rarely ablated adopted Rubric judge + anchored slice Modules 16, 19 · no hack-rate ablation reported single Modifiers that move you off the default A teacher exists on-policy distillation · M21 concordant Many domains specialists + MOPD · M22 single Agent rollouts async, staleness cap · M18 adopted Confidence needed bounded Brier term · M24 single Tags (Section 1.1): concordant, single, conflicting, adopted, no source
Figure 1. First-pass decision tree. The reward source selects the family (left to right); the bottom row lists the four modifiers that move you off that default. Each tag rates the evidence for the choice of starting point, not the method's quality in general. The branches are argued with their sources in Sections 2 to 10: DPO and PPO for preference tuning123, group baselines for verifier rewards45, distillation for small students6, and specialists with multi-teacher distillation7.

1. How to read a recommendation

Every recommendation in Sections 3 to 10 is a callout with the same parts: what to start with, the evidence and its tag, the stack and scale it was measured at, what would change it, and the first three signals to watch. The stack and scale come first because the reports differ by orders of magnitude and by engine. Tulu 3's PPO-versus-DPO test ran on two nodes1; Llama 3's DPO ran on models up to 405B2; VinePPO caps responses at 1,024 tokens8; ScaleRL spent more than 400,000 GPU-hours on dense and mixture-of-experts models of 8B to 17B-by-169. A margin measured in one of those settings does not transfer to another without a reason. Infrastructure and framework choice (Ray, Megatron, vLLM, the open-source RL libraries) is System Design Modules 36 to 38 and is not repeated here; this module treats the stack as the place where a result was measured, not as a decision.

1.1 Evidence tags

The tags are this module's rubric, not a convention in the literature. They count independent groups, not papers: Tulu 3 and Olmo 3 come from one lab, and Thinking Machines re-uses the Qwen3 table10, so neither pair counts twice. A tag attaches to a claim, so one method can carry several: DPO's cost advantage is concordant while its quality parity with PPO is conflicting.

  • concordant Two or more independent groups measured the same direction, on different stacks.
  • single One group measured it; nobody repeated it.
  • conflicting Measurements by different groups point in different directions.
  • adopted Several reports use the choice and none ablates it. Adoption is not measurement.
  • no source Nothing in the course's source map measures or states it. Any view I give there is labelled as mine.

1.2 The eight axes

Table 1. The constraint axes, the question each answers, and one measured quantity from the sources for each.
AxisQuestionA measured anchorSection
Reward typePairs, a learned score, a verifier, or a judge?RLVR is the RLHF objective with the reward model replaced by a verification function12, 8
DataDo you hold pairs or prompts?Tulu 3's pipeline yields 354,192 preference instances1; Qwen3's reasoning RL used 3,995 query-verifier pairs62
Generation compute, critic memoryWhat does a step cost and what stays resident?PPO about 28 h on two nodes against DPO about 4 h on one1; value model and its state 46% of GPU memory at 7B113, 4
Stability, tuning burdenWhat breaks, and how many knobs?PPO needed GAE \(\lambda=1.0\) to approach GRPO4; plain PPO scored 5 on AIME 2024 until seven fixes124, 5
TargetChat, reasoning, or agent?Llama 3 kept DPO for chat2; Olmo 3 ends reasoning in RLVR5; GLM-5 adds an agentic RL stage133, 5, 6
Model sizeDistill or run RL?Qwen3-8B: on-policy distillation 1,800 GPU-hours, RL 17,92067
Multi-skill integrationOne mixed run, stages, or specialists?Normalized score 0.937 for multi-teacher distillation against 0.882 for mixed RL, three domains79
Calibration needWill a consumer read the stated confidence?ECE 0.37 to 0.03 on HotpotQA with a Brier term1410

2. Reward type and data: the first split

The reward source picks the family before anything else, because each family consumes a different object. DPO consumes (prompt, chosen, rejected) triples and does not generate during the update1. PPO, RLOO and GRPO consume prompts, generate on-policy samples and score them with a reward source; that generation is where the time goes, and in Olmo 3 the learner waits for rollouts 75% of the time5 (the report's footnote 41 gives 125 s of training in a 1,000 s step, which implies 87.5%; Module 1 flags the conflict, and both figures say generation dominates). On-policy distillation consumes prompts and a teacher's per-token log-probabilities and needs no reward function at all10.

Table 2. The first split. "Tag" is the evidence tag for the starting choice. "Measured on" gives the stack and scale of the evidence. Details and the full ledgers are in the sections named in Figure 1.
What you holdStart withTagMeasured onWhat would change it
Preference pairs, no reward modelLength-normalized DPO in rounds; add an NLL term if the chosen log-probability fallsconflictingLlama 3 to 405B, six rounds2; Tulu 3's controlled test1The target is a hard verifiable outcome: DPO scored 0.0% on CodeContests where PPO scored 22.4%3
A reward model you trust, prompts, a generation budgetRLOO or REINFORCE-style first; tuned PPO if you can pay for itconflictingPythia-6.9B and Llama-7B, 512-token contexts15; DeepSeekMath-7B8A tuned critic is ahead by 3.6 to 6.4 points on short-response math8
Prompts and a programmatic verifierGroup baseline, token-level loss, no KL, mismatch correctionadoptedQwen2.5-32B16; Olmo 3 at 7B and 32B5; GLM-5 at 744B13A long run from a distilled checkpoint: KL with periodic resets17
Prompts and a judge or rubric onlyRubric judge with a verifier-anchored slicesingleRaR with a 7B policy18; Kimi K2 to K319The judge shares weights with the policy: add an external accuracy monitor (Module 16)
A stronger teacher with your tokenizer, and a small studentSFT on the teacher's output, then on-policy distillationconcordantQwen3-8B6; Qwen2.5-32B in R14No stronger teacher exists, or the goal is to exceed the teacher

The data axis is a different measurement for each family. Olmo 3 Think uses 2,268,468 prompts for SFT, 200,000 for DPO and 104,869 for RL at 7B5; Qwen3's reasoning stage used 3,995 query-verifier pairs and GRPO, chosen so that each query was learnable, as hard as possible, not used in the cold start, and broad in coverage6. RL asks for far fewer prompts and far more generation per prompt, and the prompts must carry a pass rate between 0 and 1 for the group to give a gradient (Module 9). For DPO the scarce resource is pair quality: Ivison et al. rank preference-data quality above the algorithm, reward model and extra prompts, with gains up to 8% in instruction following and truthfulness20. Olmo 3 shows how much the construction matters: SFT on the Qwen3-32B responses lowered its development average, while DPO that paired them with weaker rejected responses raised it5 (Table 21).

3. Chat alignment: DPO, PPO, or neither

This is the oldest question in the course and the one with the most conflicting evidence. Table 3 lists the measurements with the stack each was made on, because the answer depends on which row you stand on.

Table 3. Preference-tuning evidence. "Direction" is which method the study favors. Win-rates are percentages against reference completions judged by GPT-4; margins for Ahmadian et al. are derived (RLOO \(k=4\) minus DPO).
StudyStack and scaleComparedResultDirection
Llama 3 (§1, §4.1.4)2Models to 405B; six DPO rounds on batches from the latest modelsPPO tried, DPO keptDPO "required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval"; no PPO numbers publishedDPO
Tulu 3 (§5.4.1)1Controlled, "Non-Tuned Setup"; model size not stated in the passagePPO, DPOPPO comparable "albeit slightly lower"; about 28 h on two nodes against about 4 h on oneDPO on cost, parity on score
Xu et al. (Tables 7, 8)3CodeLlama 7B to 34B on APPS and CodeContests; Llama2-7B on HHPPO, DPO, DPO-IterCodeContests test 10@1k at 34B: SFT 15.2, DPO 0.0, DPO-Iter 3.2, PPO 22.4 (test-case reward)PPO
Ivison et al. (abstract)20Stack not in the abstractDPO, PPO, data, reward model, promptsPPO ahead by up to 2.5% (math) and 1.2% (general); data quality up to 8%PPO, small margin
Tang et al. (abstract)21Stack not in the abstractOnline against offlineOnline ahead; the gap persists as the policy network is scaled upOnline
Ahmadian et al. (Table 1)15Pythia-6.9B on TL;DR and HH; Llama-7B on HH; 512-token contextsRLOO, PPO, DPORLOO 77.9, 43.7, 64.1 against DPO 66.6, 39.0, 61.9: +11.3, +4.7, +2.2RLOO

3.1 The cost side is settled

Llama 3 and Tulu 3, two independent groups, report that PPO costs more for no gain in their setups21. Only Tulu 3 gives a number, and the arithmetic is short. PPO ran 28 h on two nodes, which is 56 node-hours; DPO ran 4 h on one node, which is 4 node-hours. The ratio is \(56/4 = 14\) in node-hours and \(28/4 = 7\) in wall-clock. At 8 GPUs per node, the node size the report states for its SFT runs, that is 448 against 32 GPU-hours (Module 26, derived). Figure 2 plots these next to the Qwen3 pair that Section 7 uses.

Reported GPU-hours on a logarithmic axis for two pairs of methods: Tulu 3 PPO against DPO, and Qwen3-8B reinforcement learning against on-policy distillation 10 100 1,000 10,000 100,000 GPU-hours (log scale) Tulu 3 PPO 28 h, two nodes (derived) 448 Tulu 3 DPO 4 h, one node (derived) 32 14 times node-hours, derived Qwen3-8B RL hardware not stated 17,920 Qwen3-8B on-policy distill. 1,800 9.96 times
Figure 2. Reported GPU-hours for two matched pairs, on a log axis. Tulu 3: PPO about 28 h on two nodes, DPO about 4 h on one, converted at an assumed 8 GPUs per node (derived)1. Qwen3-8B: reinforcement learning 17,920 GPU-hours against on-policy distillation 1,800, both from the same off-policy distilled checkpoint, hardware and teacher cost not stated6. The two pairs come from different tasks and stacks and say nothing about each other; the shared axis shows only that the measured gap is one order of magnitude in both.

3.2 The quality side is not

The PPO-favoring studies and the DPO-favoring ones differ in what they tuned. Tulu 3's authors say their PPO had an untuned reward model and that "it is entirely possible that we can push up the PPO's performance even higher"1. Xu et al. tuned PPO heavily and compared it with DPO trained on the dataset's static correct and incorrect programs; their DPO produced "many meaningless code snippets" after one epoch3. Neither DPO there carries Llama 3's patches: the NLL term of coefficient 0.2, the masking of formatting tokens that stopped tail repetition, and fresh preference batches each round2 (Module 7 collects these). On a general chat target the PPO margin is up to 1.2%, against up to 8% for data quality20; on a code target with a test-suite reward the margin is 19.2 points at 34B (22.4 minus 3.2 for the better DPO variant, derived)3. Both statements are true. What moves the answer is whether the reward can be checked, not the algorithm's name.

Lambert's synthesis is the third view: "despite the simplicity of alternatives, RL-based methods continue to win out", at a far larger infrastructure cost, and RL can be surgical, since Tulu 3's math-only RL did not squash general capability22 (§3.1.6). That is a statement about verifiable skills, and Section 5 picks it up.

4. Critic memory and tuning burden

4.1 What a critic costs

Mixed-precision Adam holds \(2\Psi + 2\Psi + K\Psi\) bytes per trained model with \(K=12\), which is \(16\Psi\) bytes for \(\Psi\) parameters23; a frozen model in half precision holds \(2\Psi\). PPO-RLHF trains a policy and a value model and keeps a frozen reference and reward model: \(16+16+2+2=36\Psi\), or 2,520 GB at 70B. PPO-RLVR as Tulu 3 ran it has no reward model: \(34\Psi\), 2,380 GB. GRPO with a verifier and \(\beta=0\) holds only the policy, \(16\Psi\), 1,120 GB, because TRL's default does not load the reference when the KL coefficient is 0.024 (weights and optimizer state only, derived; Module 1, Table 6 has the full list). The value model's share of PPO-RLHF is \(16/36=44\%\). ReMax measured 46% of GPU memory for the value model and its training state at 7B, and 4% for the reward model11; the derived reward-model share is \(2/36=5.6\%\). The arithmetic and the measurement agree to within two points, and the measurement includes activations.

4.2 What it buys, and why the sign flips

Margin of the critic-free method over the critic-based method in five comparisons from four reports; the sign changes with response length, tuning and the number of fixes applied to the critic -10 0 10 20 30 40 critic ahead critic-free ahead points, in each report's own metric RLOO vs PPO win-rate, 7B-class, 512 tokens +10.3 to +32.1 GRPO vs tuned PPO MATH, 16B MoE (plot reading) about +2 RLOO, GRPO vs PPO pass@1, 7B, 1,024 tokens -6.4 to -3.6 DAPO vs plain PPO AIME24, Qwen 32B +45, runs off the scale DAPO vs VAPO AIME24, Qwen 32B, 7 fixes -10.4
Figure 3. Margin of the critic-free method over the critic-based one, one row per comparison, each in its own metric, so compare the sign and not the size. Blue right of zero: critic-free ahead. Orange left: the critic is ahead. Rows: RLOO (\(k=4\)) minus PPO on three win-rate columns, derived from Ahmadian et al. Table 115; GRPO against PPO at \(\lambda=1.0\), a value read from R1's Supplementary Figure 4 by Module 9 and therefore approximate4; RLOO and GRPO against PPO on DeepSeekMath-7B GSM8K and MATH pass@1, VinePPO Figure 48; DAPO 50 against plain PPO 5, and against VAPO 60.4, VAPO Table 112.

Three things separate the rows. The first is horizon. VinePPO's critic wins are on responses capped at 1,024 tokens8, and the only long chain-of-thought critic win needs seven modifications, among them value pretraining, decoupled GAE and a length-adaptive \(\lambda\); without value pretraining VAPO scores 11, and plain PPO scores 512. The second is tuning. R1's appendix reports that PPO at the common default \(\lambda=0.95\) "performs considerably worse than GRPO", and that only \(\lambda=1.0\) brings it "nearing that of GRPO"4 (Supp. A.3). A fixed \(\lambda=0.95\) multiplies the terminal reward by \(0.95^{100}\approx0.006\) for a token 100 steps from the end (derived from the argument in VAPO §3.112), which is why long-horizon critics learn almost nothing from the reward. The third is the baseline PPO that loses: in Ahmadian et al.'s run PPO's mean output was 16.5 tokens against 60.6 for RLOO on HH-Llama15, which looks like a collapsed run (my reading), so that margin shows PPO's fragility in a short, cheap setup and says little about a tuned critic.

Tulu 3 is outside the figure because it compared critics with each other. Its value model, initialized from a general reward model, gave the best GSM8K and average scores in its ablation of initializations, and the report lists GRPO as untested future work1 (§6.2.1, Limitations). VinePPO adds a middle path, Monte Carlo value estimates from the same states with no value network: 46.0 against PPO's 42.8 on 7B MATH, at up to 2 times the per-iteration cost at 7B8.

5. Reasoning with verifiers: what the recipes converge on

With a verifier and a base or SFT model, the 2025 to 2026 recipes look alike in some columns and differ in others. Table 4 puts the shipped defaults side by side; the agreement set is the nearest thing this course has to a default, and the disagreeing columns are where your first ablation budget should go.

Table 4. Default recipes for verifier-reward RL. "Not stated" means the sections read do not give it; "none" means removed or zero. TRL values are main-branch defaults on 2026-10-27 and move24. Nemotron 3 Ultra is omitted: its report gives a global batch of 8,192 "with each sample generating 16 rollouts" but not the unit, so it cannot be put in these columns25.
RecipeLoss aggregationAdvantageKLClipMismatch, stalenessGroup filteringScale
TRL24token, over active tokens in the global batch÷ group std\(\beta=0\), reference not loaded\(\varepsilon=0.2\); 0.28 suggestedvLLM correction on: sequence-level masked ISnone; zero-std share logged\(G=8\) default
DAPO16token÷ group stdnone0.2 / 0.28not addresseddynamic sampling; overlong filtering and soft penaltyQwen2.5-32B; AIME24 30 to 50
Olmo 3 Think5token, over the batchmean-centered, no stdnone0.2 / 0.272truncated IS, cap 2.0 on 32Bzero-gradient filtering; active sampling7B, 32B dense
ScaleRL9promptbatch-level stdnoneCISPO weight capFP32 logits; async, max lag \(k=8\)zero-variance filter; drop prompts with pass rate \(\ge0.9\)8B dense, 17B×16 MoE
DeepSeek-V3.226per-sequence \(1/|o|\) as printed\(R-\text{mean}\), no stdweak or none for math; unbiased K3 estimatePPO clip, \(\varepsilon\) not stated in the sections readoff-policy sequence mask; Keep Routing; Keep Sampling Masknot statedRL above 10% of pre-training cost
GLM-5 (§3.2)13per-sequence \(1/|y|\)÷ group stdnone0.2 / 0.28IcePop, \(\beta=2\); deterministic top-kdifficulty filter on prompts744B; \(G=32\), on-policy

The agreement set reads off the columns. Five of six drop the KL term or set it to zero, and the sixth, V3.2, keeps a corrected term with domain-dependent strength26. Four of six widen the upper clip or replace it with CISPO. Five of six correct or remove the mismatch between the rollout engine and the trainer; DAPO predates that literature. Three filter zero-variance groups. The columns that disagree are aggregation (token, prompt or sequence) and advantage scale (÷ group std in TRL, DAPO and GLM-5; none in Olmo 3 and V3.2; batch-level in ScaleRL). Module 11 weighs those two; the controlled evidence is small and says second-order. DAPO's cumulative ladder moves AIME24 30, 36, 38, 41, 42, 50 across overlong filtering, Clip-Higher, soft overlong punishment, token-level loss and dynamic sampling, and the token-level step is +116.

5.1 What each shared choice rests on

  • No KL. adopted Olmo 3 states that removal "does not lead to over-optimization or destabilized training" and shows no with-and-without run5. ProRL argues the opposite for a distilled 1.5B start, keeping KL with periodic resets17. Module 13's ledger finds no ablation behind any removal.
  • Clip-Higher, CISPO. single Each has a controlled comparison in its own paper (DAPO's ladder; ScaleRL's forward tests), with no cross-comparison on one codebase (Module 12).
  • Mismatch correction. concordant that it is needed: runs collapse without it in more than one group, and a zero-mismatch control isolates the cause2713. conflicting on how: loss-side weights, FP16 or an FP32 head, or deterministic kernels (Module 14).
  • Zero-variance filtering. concordant across DAPO, Olmo 3 and ScaleRL, which all drop dead groups1659.

Magistral and MiMo-V2-Flash are pointers only: neither report was read for this course, so no cell above rests on them (Modules 13 and 22 mention each).

5.2 Efficiency or ceiling

ScaleRL, the one study that fits compute curves, concludes that loss aggregation, normalization, curriculum and the off-policy algorithm "primarily modulate compute efficiency without materially shifting the asymptote", and its leave-one-out reversions all land within \(\pm0.02\) of the fitted ceiling \(A\) (Appendix A.8)9. The same paper is not consistent about the off-policy scheme: Appendix A.11 says the choice "affects the asymptotic performance" and not only the efficiency exponent, and Figure 6 reports that zero-variance filtering reaches a higher asymptote. Treat "details only move speed" as supported for loss aggregation and normalization on one 8B dense model, and not for filtering or the off-policy scheme (Module 15). Olmo 3 ablates only active sampling among its seven changes to GRPO (Module 31).

6. Agentic coding and tool use

Agentic RL is verifier RL plus three costs: the rollout is a multi-turn episode in a sandbox, its length varies by an order of magnitude inside one batch, and the sampler and trainer must agree on the exact tokens. The generation bill is the first-order cost. Olmo 3's 32B run used 8 H100 nodes for training and 20 for inference; its text says inference takes about 5 times the GPU use of training, but 20 nodes against a learner busy 25% of the time gives 10, while the 7B figure of about 14 does follow (Module 1 flags both)5. Table 5 lists the design choices the agentic reports made and what backs each.

Table 5. Agentic RL design choices. "Evidence" is what the report shows for the choice, not for the system as a whole. GLM-5 section numbers: §3.3 is a pointer, §4.1 agentic RL, §4.1.2 the TITO gateway, §4.2 environments.
ChoiceOptions in the reportsEvidenceTag
Credit unitTrajectory-level outcome reward (GLM-5, Kimi K3, V3.2, Nemotron's SWE stage); step-level groups from repeated states (GiGPO)GiGPO: "> 12%" on ALFWorld and "> 9%" on WebShop over GRPO at 1.5B to 7B29; no frontier report ablates trajectory-level against step-leveladopted / single
StalenessTolerate: K3's per-token regularization19. Cap: GLM-5 discards a sample whose oldest rollout version is too stale13. One-step lag: NemotronNo lag sweep or staleness histogram in the passages read; GLM-5's threshold is not statedadopted
Rollout unitEnd-to-end episodes; or single-turn rollouts "similar to PivotRL" for most agentic tasks in distillationNemotron saw "substantial training inefficiency" from end-to-end rollouts in MOPD; whether they gain more "remains an open area"25single
Token fidelityTITO gateway: pass token ids, never re-tokenized textGLM-5 (§4.1.2) introduces it against "re-tokenization mismatches"13single
DeterminismDeterministic torch.topk in the sparse-attention indexer; indexer frozenNon-deterministic kernels: drastic degradation after a few steps with an entropy drop13single
HarnessSeveral harnesses per taskK3: a single harness "can cause a model to overfit to a particular tool schema" (§4.2.1)19; Nemotron trains each task distribution under at least two of six harnesses (Appendix A.2)25adopted
Environment mixSynthesized agent tasks alongside code and searchV3.2's only environment ablation: RL on synthesized general-agent tasks improved τ²-Bench, MCP-Mark and MCP-Universe, while RL on code and search alone did not26single

Scale of environments is the other reported lever: GLM-5 trains on more than 10K real-world software-engineering environments plus terminal and multi-hop search tasks13. Nemotron 3 Ultra's report says it is open-sourcing "the training recipes, data, and RL environments"; the Nemotron repository README lists the program as Pretrain, SFT, RLVR, MOPD and MTP boosting but says "this recipe covers Pretrain → SFT", that it trains "exclusively on the open-sourced subset of training data", and that results "will differ from the tech report benchmarks, which used additional proprietary data"30. So the openness of the RL environments is disputed between the report and the repository, and I do not count Nemotron as an open recipe for the RL stages (Module 32).

7. Small models: distill first

For a student of 32B or below with a stronger teacher available, two independent groups report that a teacher's signal beats RL on the student, and a third reproduces the direction with its own accounting. DeepSeek trained Qwen2.5-32B with RL for more than 10K steps and reached AIME 2024 pass@1 of 47.0, on par with QwQ-32B-Preview (50.0), while distilling R1 into the same base scored 72.64 (Supp. F.1, Table 16). The distillation there is off-policy SFT on 800K samples for 2 to 3 epochs4. Qwen3 ran the on-policy comparison; Table 6 sets the routes side by side.

Table 6. Routes to a strong small reasoning model, with what was measured. Qwen3-8B rows start from the same off-policy distilled checkpoint (AIME'24 55.0, AIME'25 42.8); pass@64 in parentheses.
RouteAIME'24AIME'25GPU-hoursSource and limit
RL only, Qwen2.5-32B47.0n/anot statedR1 Table 16: over 10K steps4
SFT on R1 samples, 32B72.6n/anot statedR1 Table 16: 800K samples4
+ RL, Qwen3-8B67.6 (90.0)55.5 (83.3)17,920Table 21; hardware and recipe not stated6
+ on-policy distillation, Qwen3-8B74.4 (93.3)65.5 (86.7)1,800Table 21; teacher cost not stated6
RL after distillation, 1.5Bn/an/aabout 16,000ProRL: over 2,000 steps, 4 nodes of 8 H10017; no matched OPD arm

Thinking Machines reproduces the direction with its own accounting: from a 400K-example SFT checkpoint a Qwen3-8B-Base student reaches 70% on AIME'24 in about 150 on-policy steps, where SFT is extrapolated to need about 2M prompts, and it reports cost reductions of 9 times with the SFT data given, about 18 times in GPU-hours, and about 30 times with teacher sampling counted, all relative to the extrapolated SFT baseline; its table lists RL at about 1 times that baseline. The page used Qwen3-8B as the teacher but counted the FLOPs of the 32B model10.

7.1 A worked cost per point, and what the teacher would have to cost

Qwen3's ratio is 17,920/1,800 = 9.96. A fairer unit is GPU-hours per point of gain, since distillation also gained more. From the 55.0 starting score, RL gained 12.6 points (67.6 minus 55.0) and distillation 19.4 (74.4 minus 55.0). Per point, RL cost \(17{,}920/12.6=1{,}422\) GPU-hours and distillation \(1{,}800/19.4=92.8\), a ratio of 15.3 (derived). Now charge the teacher. Distillation matches RL's cost per point when \((1{,}800+T)/19.4 = 1{,}422\), so \(T\approx25{,}800\) GPU-hours, 14.3 times the distillation run itself. The report states neither the hardware nor whether the teacher is inside the 1,800, and the teacher is the product of four RL-shaped stages that Qwen3 never prices. It does serve six lightweight students (five dense models and one mixture-of-experts)6, so the calculator lets you spread a guessed teacher cost over \(n\) of them. The guess is an assumption, not a source.

GPU-hours per AIME'24 point, Qwen3-8B: RL against on-policy distillation with the teacher charged

RL, per point
Distillation, per point
RL cost over distillation cost

Constants from Qwen3 Table 216: 17,920 and 1,800 GPU-hours; gains of 12.6 and 19.4 AIME'24 points over the 55.0 start. \(T\) is not in the report. Single run, 30 questions per AIME set; a one-question change in pass@64 is 3.3 points.

8. Tasks without a verifier

When no program decides correctness, the choice is which model sits in the reward path and how its exploit is guarded. Table 7 routes by task property. The reports agree on the routing and on the guards' existence; none of the model reports gives a hack rate with and without its guard (Module 19), so every guard below is a design the lab states, not a measured effect.

Table 7. Task property, reward source, and the guard each report states. Routing follows Module 16, Table 7; every row is a single lab's design unless noted.
Task propertySourceWhat the report statesGuard
A rule or test decidesVerifierQwen3: rule-based rewards "assess the correctness of model outputs with high precision, preventing issues like reward hacking"6Module 19, Section 9
An answer exists, forms varyReference-conditioned judgeQwen3: a 72B judge sees the reference; a preference model only for queries with none6Keep rule checks where possible
No reference, many criteriaRubric judgeRaR: up to 31% (HealthBench) and 7% (GPQA-Diamond) over Likert judges, the authors' numbers18; V3.2 uses per-prompt rubrics26Several or rotating rubrics; veto items
Open-ended preferencePairwise or binary-comparison judgeK2: the critic ranks outputs by pairwise comparison and is refined on verifiable rollouts31. K3: "the tournament-style group reward with binary comparisons as in Kimi K2.5"19K3: an output longer than \(\sigma\ell_0\) loses the comparison automatically; K2: a hack-check layer
Last-mile polishScalar preference modelR1 applies it only to the last 400 of 1,700 stage-2 steps4Short exposure; length-matched pairs
Judge must grow with the policyActor as judgeV4: RL applied to the generative reward model itself, with "minimal" human annotation (§5.1.1)32None stated; vendor claim

Two cautions on the evidence. K3 states its aggregation only as the quoted sentence and adds a protocol of reading the output, writing a rubric, scoring each candidate and recording the scores in a scorepad19; the K2.5 text I read does not use the word "tournament", and K2's equivalent is the pairwise ranking above, so I claim nothing beyond the K3 sentence. And RaR's gains come from a 7B policy, point estimates, and a training judge that also grades the benchmark (Module 16). V4's actor-as-judge and K2's closed-loop critic have no published comparison with a frozen judge, so both are vendor claims. R1 observed the failure that justifies a cap: with a preference reward, reward rose while Codeforces performance fell (Supplementary B.5; Module 19).

9. Several skills in one model

Reasoning, agent, chat and safety skills interfere when trained together, and the reports answer in three ways (Figure 4). V3.2 merges reasoning, agent and alignment into one mixed RL stage to avoid "the catastrophic forgetting issues commonly associated with multi-stage training paradigms"26. GLM-5 runs three RL stages in sequence and ends with on-policy cross-stage distillation from its own earlier checkpoints, because "sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities"13 (§3.5). Kimi K3, DeepSeek-V4 and Nemotron 3 Ultra train specialists and distill them on the student's own rollouts: nine experts for K3 (three domains at three effort levels), more than ten for V4 and for Nemotron193225. Lambert's survey records the shift, notes that MiMo-V2-Flash documented it first, and writes that "labs are far from unanimous in its usage"22 (§3.2.4).

Stage sequences of eleven recipes, left to right, with offline supervised stages in blue, reinforcement learning in orange and distillation in cyan SFT, DPO, rejection sampling RL distillation (OPD, MOPD) Llama 3 SFT Rejection sampling DPO ×6 rounds Tulu 3 SFT DPO RLVR (PPO) Olmo 3 Think SFT DPO RLVR (OlmoRL) DeepSeek-R1 Cold-start SFT Reasoning RL 800K SFT All-scenario RL Qwen3 flagship Cold-start SFT Reasoning RL Thinking fusion General RL Qwen3 small Off-policy distill On-policy distill DeepSeek-V3.2 Specialists (RL) Distilled data One mixed RL DeepSeek-V4 SFT >10 specialists Multi-teacher OPD GLM-5 SFT Reasoning RL Agentic RL General RL Cross-stage OPD Nemotron 3 Ultra SFT Unified RLVR MOPD (>10) Kimi K3 SFT 9 RL experts MOPD
Figure 4. Where eleven reports put RL and distillation, stage order left to right. Mixed RL (V3.2, the last Qwen3 stage), staged RL repaired by distillation (GLM-5), and specialists merged by multi-teacher distillation (V4, Nemotron 3 Ultra, Kimi K3) are the three skill-integration patterns of Section 9. Sources: Llama 3 §42; Tulu 31; Olmo 3 §45; R14; Qwen3 §46; V3.2 §326; V4 §532; GLM-5 §313; Nemotron 3 Ultra §325; Kimi K3 §4.119. Box width is not duration.
Table 8. Skill-integration choice by regime. Scores are the MOPD paper's normalized scores on Qwen3-30B-A3B-Base, three domains (math, instruction following, software engineering)7.
RegimeApproachEvidenceTag
Few similar domains, one teamOne mixed RL stageMix-RL 0.882, the flattest baseline (range 0.064) and 5.5 points behind MOPD; V3.2 and Qwen3's last stage run mixed RL at frontier scale6single
Serial stages unavoidableStages, then distillation from your own earlier checkpointsGLM-5 §3.5; the controlled cascade alone scored 0.775, with Math closing only 57% of its headroomsingle
Many domains, agentic ones, parallel teamsSame-origin specialists, then sampled-token MOPD0.937, domains within 0.91 to 0.95; adopted by K3, V4, Nemotron; Nemotron's stated reason is per-domain dilution in a mixed batch (§3.3)25single + adopted
A skill the student cannot sampleMOPD does not import itNemotron HLE: recovery 16.9% of the teacher gap, the teacher's gains coming from separate SFT and RL data the student never saw (Table 5)25single

Only one controlled comparison exists, and its numbers are the authors' own. Cascade RL's Math score falls during the later software-engineering stage; off-policy finetuning exceeds the teacher on instruction following (1.01) but closes only 65% of the software-engineering gap7. No frontier report adds a matched-compute ablation against the method it replaced: V4 swaps V3.2's mixed RL for distillation, but the models also differ in architecture and data (Module 30). Teacher choice has a measured failure: replacing the Math RL teacher with the far larger Qwen3-235B-A22B raised the initial per-token KL from about 0.04 to about 0.19 and degraded Math under both loss variants, the top-k variant diverging near step 187 (§4.4.2). A date note on V4: its arXiv identifier 2606.19348 has a 2606 prefix, while the PDF stamp and the API date say 26 April 2026; I could not resolve this, and it does not touch the figures used here.

10. Calibration and abstention

A binary correctness reward pays for guessing: it does not penalize low-confidence wrong answers, so RLVR can degrade calibration14, and under 0-1 grading an abstention is strictly suboptimal33. The public repairs act at different layers (Table 9). The strongest training result is a bounded proper scoring rule added to the verifier reward: RLCR's reward is \(R=\mathbb 1[y=y^*]-(q-\mathbb 1[y=y^*])^2\), where \(q\) is the model's stated confidence, and any bounded proper rule gives both a calibration incentive and a correctness incentive, while the log score is unbounded and can reward wrong answers14 (§3). The theorem's appendix proof carries a sign slip that I do not rely on; Module 24 checks the claim separately.

Table 9. Calibration levers with their measured effect and cost. RLCR numbers are Qwen2.5-7B (Table 1 and Table 2 of the paper)14; "OOD" averages out-of-domain datasets.
LeverMeasured resultCost or limitTag
Brier term on stated confidence (RLCR)ECE 0.37 to 0.03 (HotpotQA), 0.26 to 0.10 (math); accuracy 63.0% against 62.1%, interval half-width ±3.05OOD ECE still 0.21 to 0.25; 249 tokens against 92, +157 per answer; no abstain actionsingle
A second model reads confidenceBCE classifier ECE 0.07 in-domain, 0.24 OOD, on the RLVR model's outputsA second 7B model; the policy never abstainssingle
Threshold-conditioned reward (Behaviorally Calibrated RL)Abstract: a 4B model's log-scale Accuracy-to-Hallucination Ratio gain 0.806 against GPT-5's 0.207 on BeyondAIME, the authors' evaluation34Abstract only in the source map; Module 24 reports the policy did not become sensitive to the threshold at 4Bsingle
Change the grading (Kalai et al.)Of ten audited evaluations only one, WildBench, gives minimal credit for uncertainty; proposed penalty \(t/(1-t)\) for a wrong answer at confidence target \(t\)33No training experiment; depends on leaderboard owners; the paper's \(t=0.75\) example misprints the penalty, so use the formulasingle
TypeSafe "RLCD"Company statements only35No paper, repository, reward function, algorithm or datasetno source

TypeSafe AI's "Reinforcement Learning for Calibrated Decisions" is a company claim. Its 15 September 2026 launch post names the method in one sentence and says it optimizes "calibrated decisions: answers with epistemically honest probabilities"35; on the Latent Space podcast the founder, asked whether it was published, answers "No, not yet"36. I give no recommendation for or against it and describe no mechanism, since no source discloses one (Module 25). It is not Yang et al.'s Reinforcement Learning from Contrastive Distillation, which shares the acronym and builds preference pairs from contrasting prompts37.

11. The constraint-by-method matrix, and the cells with no source

Table 10. Method by constraint. "Resident state" is weights and optimizer state at 70B, derived from 16 bytes per trained parameter and 2 per frozen one23 (Module 1, Table 6); teacher serving is not counted. Each non-derived cell carries its source.
MethodNeedsResident state, 70B (GB)Generates?Stability and tuningBest-measured targetTag
DPO, cached referencePairs1,120NoPatches needed: NLL 0.2, token masking, length normalization21Chat to 405B2conflicting
PPO with criticReward model or verifier2,380 to 2,520Yes\(\lambda=1.0\) needed4; seven fixes for long chains12Code with tests3; short-response math8conflicting
RLOO / REINFORCEReward model; \(k\ge2\) samples1,400YesNo clip or ratio in the estimator157B-class chat, 512 tokens15single
GRPO family, verifier, \(\beta=0\)Verifier1,120Yes, 8 to 32 per promptZero-variance groups; mismatch2427Math, code, agents513adopted
On-policy distillationTeacher, prompts1,120 plus teacherYes, plus teacher scoringSame-origin teachers stable; external teacher diverged7Small students6concordant
Multi-teacher distillationOne RL teacher per domain1,120 plus teachersYes, plus scoringCannot import skills the student never samples25Many-domain integration7single
Verifier plus Brier termVerifier1,120Yes, +157 tokensBounded rule required14Short answers, 7B14single
Table 11. Decisions on which this module has no source, with the experiment each would need. Each absence is as recorded in the module named; none is a claim that the experiment has not been run anywhere.
DecisionMissing evidenceModule
Tuned PPO against patched DPO at 70B or moreMatched data and compute, held-out human preference, seed intervals7, 26
Critic against group baselineEqual GPU-hours including the critic, two response lengths9
Normalization gridAggregation by advantage scale, three seeds, short and very long caps11
Mismatch fix familiesOne mixture-of-experts model, zero-mismatch reference, serving-engine evaluation14
Frozen against jointly trained judgeEqual compute, scored by humans or a verifier slice16
Staleness tolerate against cap; step-level against trajectory creditLag sweep; GiGPO-style keys on SWE at 30B or more18
KL on rubric-judged domainsSame run with and without the term30
RL against distillation, teacher countedOne start, matched total GPU-hours, pass@k over more than 30 problems21
Mixed RL against MOPD at ten or more domainsEqual GPU-hours from one SFT checkpoint22
Calibration at frontier scaleAgentic or noisy-verifier tasks; independent replication24
TypeSafe's RLCDEverything: no paper, reward, algorithm or dataset25

12. What the evidence supports, and what it does not

Supported by at least two independent groups. Offline DPO costs far less than PPO for chat tuning, 14 times the node-hours in the one measured pair, and Llama 3 reports the same direction12. Data quality moves preference tuning more than the algorithm205. A teacher's signal beats RL for a small student on cost and pass@1 in each direct comparison46. A mismatch between rollout and trainer engines can collapse a run and must be handled2713.

Supported by one controlled study, or by adoption alone. Specialists plus MOPD beating mixed RL (one study, three domains). The group-baseline default for long chains of thought (adopted; the one critic win there needs seven fixes). Dropping KL (adopted; no with-and-without run). A Brier term for calibration (one paper, 7B). Rubric judging (design statements, no hack-rate ablation).

Not supported by anything read. That any algorithm in this module is best at frontier scale: the frontier reports ablate recipes against their own earlier versions, not against rivals. That RL "expands" capability, or that distillation does so more cheaply, beyond the one-question margins in Table 6 (Module 20). That any teacher cost, hardware, or seed variance is known for the Qwen3 pair. Anything about how TypeSafe's RLCD trains. The numbers are also dated: TRL's defaults, report versions and a V4 identifier that conflicts with its own date will move, and Module 34 lists what to re-check first.

Where experts disagree

The five disputes below are the ones that decide a starting method. Each is treated at length in the module named; here each is reduced to the condition under which one side is the better start.

A. Is DPO enough, or does online RL win for preference tuning? (Modules 7, 26)

DPO in rounds is enough

Llama 3 team; Tulu 3 authors21

Comparable score at 14 times fewer node-hours; spend the saving on data.

  • Measured cost; used to 405B.
  • Untuned PPO; no PPO numbers published by Meta.

Tuned online RL wins

Xu et al.; Ivison et al.; Tang et al.32021

PPO is ahead everywhere tested, most of all with a checkable reward.

  • Three groups agree on direction.
  • Margins up to 1.2% on chat against up to 8% for data.
Assessment

Both hold in their regimes. Without a checkable reward, patched DPO is the efficient start; with one, DPO's 0.0% on CodeContests settles it for RL. A matched 70B study would settle the chat case.

B. Is a value model worth it? (Module 9)

No critic

Ahmadian et al.; DeepSeek; most open reasoning recipes154

Last-token rewards give a critic little to learn; the group is a free baseline.

  • Saves about 44% of state in PPO-RLHF (derived).
  • Behind a tuned critic by 3.6 to 6.4 points at 7B.

A built critic has the higher ceiling

VinePPO; VAPO; Tulu 38121

Per-step credit and lower variance repay the cost once the critic is fixed.

  • 60.4 against DAPO's 50 on AIME24.
  • Seven fixes; plain PPO scored 5; unreplicated.
Assessment

Group baseline for long chains with a limited engineering budget; a critic only with VAPO-grade fixes or when credit across steps is the bottleneck. R1's \(\lambda\) result shows the gap is partly tuning.

C. How should the GRPO loss and advantage be normalized? (Module 11)

Remove the biases: no std, constant or token-level loss

Dr. GRPO; Olmo 3; V3.2 on the advantage38526

Per-response length and std division reweight the objective.

  • Unbiased; fewer knobs.
  • Small-scale evidence; reward scale sets step size.

Normalization changes speed, not the ceiling

ScaleRL9

Choose prompt-level averaging and batch-level scale; spend effort on loss type and numerics.

  • Reversions within \(\pm0.02\) of the ceiling.
  • One 8B model; its own A.11 and filtering results move the ceiling.
Assessment

Fix the length term first, since it costs nothing; do not spend an ablation budget on the std question before loss type and mismatch. The grid that would settle it does not exist.

D. Mixed RL, stages, or specialists plus distillation? (Module 22)

One mixed stage

DeepSeek-V3.2; Qwen3's last stage266

One trainer, one recipe, no forgetting by construction.

  • Flattest non-distillation baseline (0.882).
  • Per-domain signal dilutes as environments multiply.

Specialists, then MOPD

MOPD paper; Kimi K3; DeepSeek-V4; Nemotron7193225

Train each skill to its peak, then merge on the student's rollouts.

  • 0.937 and the flattest profile.
  • Needs same-origin teachers; HLE recovery 16.9%; one controlled study.
Assessment

The count of domains and the availability of parallel teams decide it. "Specialists win at frontier scale" is an inference from adoption, as Lambert's "far from unanimous" implies22.

E. RL or distillation for a strong small reasoning model? (Module 21)

Distill first

DeepSeek-R1 authors; Qwen3 team46

A dense teacher signal gives higher pass@1 at about a tenth of the GPU-hours.

  • Direction agrees across off- and on-policy variants.
  • Teacher cost uncounted; single runs.

RL remains the path past the teacher

R1 authors, same appendix; ProRL417

Distillation cannot exceed its teacher; prolonged RL from a distilled start reports gains.

  • The only route beyond the frontier.
  • About 16,000 GPU-hours at 1.5B; no matched OPD arm.
Assessment

Distillation copies a capability; RL is the option when no stronger teacher exists. Cost alone does not separate them once the teacher is charged (Section 7.1).

What is changing

Four inputs to this module are likely to change first: TRL's defaults (its loss menu already lists eight loss types)24; the MOPD recipe family, which Lambert's survey reports with different expert counts per lab22; the open status of Nemotron's RL stages, which depends on a repository that says its recipe covers Pretrain and SFT30; and the Qwen3 teacher accounting. Module 34 carries the dated map of what is settled, contested and open.

References

  1. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025 (v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the PPO-versus-DPO controlled comparison, about 28 h on two nodes against about 4 h on one, "comparable (albeit slightly lower)", and the untuned-setup caveat (§5.4.1); length-normalized DPO result (Table 18); RLVR definition, value-model initialization and 354,192 preference instances (§1, §6.2.1, Limitations).
  2. paperGrattafiori, A., Dubey, A., Jauhri, A., et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024 (v1 31 Jul 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: the stated stance on RL stability (§1), the PPO comparison and DPO settings (learning rate 1e-5, beta 0.1, NLL coefficient 0.2, token masking) in §4.1.4, six rounds (§4.1.6).
  3. paperXu, S., Fu, W., Gao, J., Ye, W., et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-27. Used for: CodeContests 10@1k results at CodeLlama-34B (Table 8: SFT 15.2, DPO 0.0, DPO-Iter 3.2, PPO 22.4) and the APPS results (Table 7); "many meaningless code snippets" for DPO (§6). Added beyond the source map (full text; the map had the abstract).
  4. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026 (expanded Nature version); v1 22 Jan 2025. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the PPO-versus-GRPO comparison at GAE lambda 0.95 and 1.0 (Supp. A.3, Figure 4), distillation against RL on Qwen2.5-32B (Supp. F.1, Table 16: 47.0 and 72.6 on AIME 2024; 800K samples), the preference-reward window of the last 400 of 1,700 steps (§3.2.2) and reward hacking (Supp. B.5).
  5. paperOlmo Team. Olmo 3. arXiv v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL changes and "no KL" statement (§4.4.1), the 75% learner wait and footnote 41's 125 s of a 1,000 s step, node counts (§4.4.3), prompt counts (Tables 17, 19, 20), the delta-learning result (Table 21).
  6. paperQwen Team (Yang, A., et al.). Qwen3 Technical Report. arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: 3,995 query-verifier pairs and the four selection criteria (§4.2), the rule-based reward statement and judge types (§4.4), strong-to-weak distillation for five dense models and one mixture-of-experts model, and Table 21 (§4.7): AIME'24/'25, pass@64 and GPU hours 17,920 and 1,800.
  7. paperMa, W., et al. (Xiaomi LLM Core). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-26. Used for: the controlled comparison on Qwen3-30B-A3B (normalized scores 0.937, 0.882, 0.824, 0.775; Math 57% in the cascade; per-domain ranges), same-origin teachers and the external-teacher failure (§4.4.2), the framing of Mix-RL, Cascade RL, Off-Policy Finetune and Param-Merge (§1).
  8. paperKazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., Le Roux, N. VinePPO: Refining Credit Assignment in RL Training of LLMs. ICML 2025; arXiv v2, 3 Jun 2025. https://arxiv.org/abs/2410.01679. Accessed 2026-10-27. Used for: Figure 4 (DeepSeekMath-7B MATH pass@1: PPO 42.8, VinePPO 46.0; GSM8K and MATH for RLOO and GRPO), the 1,024-token response cap (appendix hyperparameters), per-iteration slowdown up to 2x at 7B and 5x at 1.1B. Added beyond the source map.
  9. paperKhatri, D., et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: over 400,000 GPU-hours, the ScaleRL recipe (§1), efficiency-versus-asymptote conclusion (abstract), leave-one-out ±0.02 (App. A.8), zero-variance filtering and the pass-rate curriculum (Figure 6), PipelineRL k = 8 and the asymptote statement (§3.1, App. A.11).
  10. blogLu, K., et al. (Thinking Machines). On-Policy Distillation. Blog post, 27 Oct 2025; later update notes models retired. https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-26. Used for: about 150 steps to 70% on AIME'24, SFT extrapolated to about 2M prompts, and the 9x, 18x and 30x cost reductions in FLOPs and GPU-hours; reuse of the Qwen3 table. Quotes from the saved page text.
  11. paperLi, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., Luo, Z.-Q. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML 2024; arXiv v4, 16 May 2024. https://arxiv.org/abs/2310.10505. Accessed 2026-10-27. Used for: the 46% GPU-memory share of the value model and its state and 4% for the reward model at 7B (footnote 1). Added beyond the source map.
  12. paperByteDance Seed. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: Table 1 (vanilla PPO 5, DAPO 50, VAPO without value pretraining 11, VAPO 60.4 on AIME24 with Qwen 32B) and the seven modifications (§4, §5). Added beyond the source map.
  13. paperGLM-5 Team (Zeng, A., Lv, X., et al.). GLM-5: from Vibe Coding to Agentic Engineering. arXiv v2, 24 Feb 2026 (v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the reasoning-RL loss with IcePop, beta = 2, epsilon 0.2/0.28, group 32, KL removed, and the indexer top-k finding (§3.2); agentic RL, TITO gateway, Direct Double-sided Importance Sampling and staleness discard (§3.3 pointer, §4.1, §4.1.2); on-policy cross-stage distillation (§3.5).
  14. paperDamani, M., Puri, I., Slocum, S., et al. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). ICLR 2026; arXiv v2, 15 May 2026 (v1 22 Jul 2025). https://arxiv.org/pdf/2507.16806v2. Accessed 2026-10-26. Used for: the reward with the Brier term (Eq. 1), the bounded-proper-score argument (§3), Table 1 and Table 2 numbers (ECE, accuracy, tokens, classifier baseline), and the log-score unboundedness caveat. The proof of Theorem 1 carries a sign slip in the appendix (Module 24).
  15. paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: Table 1 win-rates (RLOO k=4 77.9, 43.7, 64.1; PPO 67.6, 29.2, 32.0; DPO 66.6, 39.0, 61.9), Table 2 output lengths (PPO 16.5 tokens on HH-Llama), the 512-token context (§4). Margins derived; the intro's "3.2% to 20.3%" is not reproduced from the table (Module 9).
  16. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. (ByteDance Seed). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: the cumulative ladder of Table 1 (AIME24 30, 36, 38, 41, 42, 50) and the techniques behind it; no-KL decision (§2.3).
  17. paperLiu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., Dong, Y. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: KL control with reference resets from a distilled 1.5B start (§2.2), over 2,000 steps, 4 nodes of 8 H100 and about 16k GPU-hours (§3.2).
  18. paperGunjal, A., Wang, A., Lau, E., Nath, V., et al. (Scale AI). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv v2, 3 Oct 2025. https://arxiv.org/abs/2507.17746. Accessed 2026-10-26. Used for: gains of up to 31% (HealthBench) and 7% (GPQA-Diamond) over Likert judges, the authors' numbers (abstract); the 7B-policy and shared-judge caveats (Module 16).
  19. paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.). Kimi K3: Open Frontier Intelligence. arXiv v2, 7 Aug 2026 (v1 27 Jul 2026). https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: nine experts, three domains at three effort levels (§4.1.1); the agentic generative reward model and its verbosity control (§4.1.2); multi-teacher on-policy distillation (§4.1.3); partial rollouts and per-token regularization; the single-harness remark (§4.2.1).
  20. paperIvison, H., Wang, Y., Liu, J., Wu, Z., et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv v2, 7 Oct 2024. https://arxiv.org/abs/2406.09279. Accessed 2026-10-27. Used for: the ranking of factors (data, algorithm, reward model, prompts), PPO ahead by up to 2.5% in math and 1.2% in general domains, data gains up to 8% (abstract; the stack is not in the abstract).
  21. paperTang, Y., Guo, D. Z., Zheng, Z., Calandriello, D., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv v1, 14 May 2024. https://arxiv.org/abs/2405.08448. Accessed 2026-10-27. Used for: the online-versus-offline gap that scaling the policy does not close (abstract). Added beyond the source map.
  22. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026; web edition built 14 Oct 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: "RL-based methods continue to win out" and the surgical-RL remark (§3.1.6); the 2026 MOPD adoption, expert counts and "labs are far from unanimous" (§3.2.4).
  23. paperRajbhandari, S., Rasley, J., Ruwase, O., He, Y. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020; arXiv v3, 13 May 2020. https://arxiv.org/abs/1910.02054. Accessed 2026-10-27. Used for: mixed-precision Adam memory, 2 + 2 + K bytes per parameter with K = 12, 16 bytes in total (§3.1). Added beyond the source map.
  24. officialHugging Face TRL maintainers. GRPO Trainer (documentation, main branch; no version stated). Live page, fetched twice. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: GRPOConfig defaults and options: loss_type (dapo default; dr_grpo, bnpo, cispo, sapo, luspo, vespo), beta 0.0 with the reference not loaded, epsilon 0.2 and epsilon_high, scale_rewards, num_generations 8, vLLM importance-sampling correction defaults. Snapshot of a moving page.
  25. paperNVIDIA (Blakeman, A., Thomas, A., et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: unified RLVR with global batch 8192 and 16 rollouts per sample (unit unstated), dilution argument and more than ten teachers (§3.2, §3.3), the single-turn rollout limitation (§3.3), Table 5 HLE recovery 16.9%, harness list (Appendix A.2).
  26. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the advantage R minus mean (§3.1), the unbiased K3 estimate and domain-dependent KL strength, off-policy sequence masking, Keep Routing and Keep Sampling Mask, the single mixed stage (§3), RL above 10% of pre-training cost (§1), the synthesized-task ablation (§4.3).
  27. paperQi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., Lin, M. Defeating the Training-Inference Mismatch via FP16. arXiv v1, 30 Oct 2025. https://arxiv.org/pdf/2510.26788v1. Accessed 2026-10-26. Used for: the mismatch collapse under BF16 with vLLM and FSDP and the FP16 remedy (§1; Module 14 holds the dispute).
  28. paperMiniMax (Chen, A., Li, A., et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: repetition-based early truncation and the combination of sample-level loss with token-level normalization in the 80K-token run (§3.2, §4.2).
  29. paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training (GiGPO). NeurIPS 2025; arXiv v3, 28 Oct 2025. https://arxiv.org/abs/2505.10978. Accessed 2026-10-26. Used for: anchor-state step-level groups and gains of more than 12% on ALFWorld and 9% on WebShop over GRPO (abstract).
  30. repoNVIDIA-NeMo. Nemotron repository README (training recipes). Live page. https://github.com/NVIDIA-NeMo/Nemotron. Accessed 2026-10-27. Used for: the Ultra entry (550B total, 55B active; program Pretrain, SFT, RLVR, MOPD, MTP boosting; "this recipe covers Pretrain → SFT") and the open-data-only notice. Added beyond the source map; raw file https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/README.md.
  31. paperKimi Team (Bai, Y., Bao, Y., Charles, Y., et al.). Kimi K2: Open Agentic Intelligence. arXiv v2, 3 Feb 2026 (v1 28 Jul 2025). https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: the self-critic reward by pairwise comparison, closed-loop critic refinement and the hack-check layer (§3.2.1).
  32. paperDeepSeek-AI (Xu, A., Lin, B., Xue, B., et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv id 2606.19348 v1; the PDF stamp and API date read 26 Apr 2026 (unresolved against the 2606 prefix). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: specialist training and the actor-as-generative-reward-model paragraph (§5.1.1), more than ten teachers and full-vocabulary distillation (§5.1.2).
  33. paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. Why Language Models Hallucinate. OpenAI and Georgia Tech, arXiv v1, 4 Sep 2025. https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-26. Used for: the 0-1 grading argument that abstaining is suboptimal, the proposed penalty t/(1-t) in the prompt (§4), and the audit of ten evaluations (Table 2, page 14). The printed t = 0.75 example is not reused.
  34. paperWu, J., Liu, J., Zeng, Z., Zhan, T., et al. Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. arXiv v3, 28 Jan 2026. https://arxiv.org/abs/2512.19920. Accessed 2026-10-26. Used for: the abstract's claim of a 4B model's log-scale Accuracy-to-Hallucination Ratio gain 0.806 against GPT-5's 0.207 on BeyondAIME (the authors' evaluation). Abstract only in the source map.
  35. blogTypeSafe AI (Almeida, D.). Introducing System One Models and Jev. Company blog, 15 Sep 2026 (page metadata shows an export date of 19 Oct 2026). https://typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed 2026-10-26. Used for: the one sentence naming RLCD and the comparison-table row "calibrated decisions: answers with epistemically honest probabilities on System One tasks". Company claims only.
  36. talkswyx (host) with Almeida, D. Jev: System One models for Prod, not God. Latent Space podcast, 2026; publication date not captured. https://latent.space/p/jev/. Accessed 2026-10-26. Used for: the founder's "No, not yet" on whether RLCD is published (transcript around 00:25:05, read from extraction output, not machine-verified).
  37. paperYang, K., Klein, D., Celikyilmaz, A., Peng, N., Tian, Y. RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. ICLR 2024; arXiv v3, 16 Mar 2024. https://arxiv.org/pdf/2307.12950v3. Accessed 2026-10-26. Used for: the 2023 method that shares the acronym; preference pairs from a positive and a negative prompt. Not TypeSafe's method.
  38. paperLiu, Z., Chen, C., Li, W., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the length and standard-deviation biases of GRPO and the unbiased variant (§3).