Reinforcement Learning for LLM Post-Training · Module 34

The Frontier as of October 2026: Open Problems, Moving Parts, and What Closed Labs Do and Do Not Disclose

A map, dated 27 October 2026, of what the mechanisms in Modules 1 to 32 have settled, what labs still choose differently, and what no source in the course can yet decide, with the leading indicator and the module that carries the evidence for each entry. It ends with a bounded account of what OpenAI and Anthropic have put on the record about RL post-training and what they have not. Afterwards you can say which parts of the course are most likely to be revised first, and where the evidence is weakest.

Course 45 min read 57 sources As of 2026-10-27 Prerequisites: Module 18, Module 22, Module 25, Module 33
Map of 22 mechanisms in three columns: settled within a scope, contested, and open, each tagged with the module that carries the evidence Settled, within a stated scope 7 mechanisms S1 Rule verifiers, reasoning M10 M16 S2 Open-recipe loss hygiene M11 M13 S3 Handle engine mismatch M14 S4 No incentive to calibrate M24 S5 Hacks spread once found M19 S6 DPO cheaper than PPO M7 M26 S7 No PRM reward at RL scale M4 M27 Contested: labs chose differently 8 mechanisms C1 Critic or no critic M9 C2 Ratio unit and clip form M12 C3 Which mismatch fix M14 C4 Mixed RL, cascade, MOPD M22 C5 RL or distill (small) M21 M29 C6 Length and effort control M23 C7 Calibration: where to fix M24 C8 DPO vs PPO quality M7 M26 Open: decisive test absent 7 mechanisms O1 Sharpen or expand M20 O2 Tasks without verifiers M16 O3 MOPD on agentic rollouts M22 O4 Environment quality M17 O5 Does RL compute pay? M15 O6 RL vs CoT monitoring M19 O7 TypeSafe RLCD: claims M25
Figure 1. The map: 22 mechanisms in three states. The codes S, C and O are the row labels of Tables 2, 4 and 5; the M-numbers are the course modules that carry the evidence. The state of each chip is the author's judgement on the cited evidence under the rule in Section 1.2, as of 2026-10-27. Middle column: for skill integration, Lambert's survey says labs "are far from unanimous"1.

1. How to read the map

1.1 What the date covers

The evidence base is the course's source map (60 entries, built 2026-10-26) plus the additions in the references. Most 2026 reports were read at the sections that bear on RL, not end to end, and several have moved since first posting: Kimi K2.5 and K3 are at v2 of 7 August 202623, Olmo 3 at v2 of 14 April 20264, and DeepSeek-R1 changed two headline numbers between v1 and v2 (Table 9). Every benchmark number below was reported by the lab that trained the model.

1.2 Three states, defined by evidence

  • Settled, within a stated scope. Every report read for that regime agrees, and either one measurement or at least three adopters from separate labs support it. The scope is part of the claim: "no KL term" is settled for rule-verified rewards in open recipes, not for judge rewards or distilled starts.
  • Contested. At least two choices, each carried by a lab's own run or a measured result, and no matched comparison between them. The right choice currently depends on stack, cost and task, so Module 33's decision framework applies.
  • Open. The decisive experiment is absent from every source read, or a report states the problem is unsolved.

The tables tag each row's evidence as measured (a controlled comparison or a theorem), concordant (three or more independent reports, no controlled comparison), single report, or claim only. A row can be contested in one regime and settled in another; the figure places each mechanism by the regime its row names.

1.3 What the map cannot see

Open reports come from labs that chose to publish, so a convergence among them is evidence about open-weight practice, not about the field. The closed labs are nearly absent as recipe sources; Section 8 treats what they have said, in a bounded way. Nothing below reconstructs a closed recipe.

2. The dated record, February 2024 to October 2026

Figure 2 colours each recipe report by the last stage its flagship recipe describes; Section 6 reads the pattern. Most disagreements below are between reports separated by months, not years.

Timeline from February 2024 to October 2026 of recipe reports and method papers, colour-coded by how the final stage integrates skills, with the company RLCD claim and the unresolved DeepSeek-V4 date marked separately Jan 2025 Jan 2026 Feb 2024 Tulu 3 R1, k1.5 Qwen3 V3.2 Olmo 3 MiMo-V2-Flash K2.5 GLM-5 V4: date unresolved Nemotron 3 Ultra K3 DeepSeekMath (GRPO) DAPO ScaleRL OPD blog MOPD paper Lambert v12 RLCD claim (company only) scale: 21.6 px per month, axis ends 27 Oct 2026 recipe report whose last stage is RL last stage distils from specialists or earlier stages method paper, analysis or survey claim only: no paper, repo or algorithm
Figure 2. Report and paper dates, February 2024 to 27 October 2026, from arXiv first-submission dates5 and the posting dates in Table 1. Blue: the flagship recipe ends in RL. Orange: it ends by distilling from specialists or earlier stages. Qwen3 is blue for its flagship; its small models end with distillation6. Kimi K2.5 is blue because its post-training is joint RL with domain experts learning together2. DeepSeek-V4's arXiv identifier says June 2026 while the arXiv API and the PDF stamp say 26 April 2026; the amber bar spans both and no dot is placed7. The red dashed marker is a company claim with no paper behind it8.
Table 1. Milestones and the one thing each added. Dates are first-submission or posting dates; V4 is unresolved (Table 9).
DateReportWhat it added to the toolbox
5 Feb 2024DeepSeekMathGRPO: a group-relative baseline replaces PPO's value model9
22 Nov 2024Tulu 3The name RLVR; PPO kept for the verifier stage, DPO for preferences10
22 Jan 2025DeepSeek-R1; Kimi k1.5Rule-based RLVR with GRPO from a base model; a mirror-descent variant with no value function, PRM or tree search1112
18 Mar 2025DAPOClip-Higher, dynamic sampling, token-level loss, overlong shaping; no KL term13
14 May 2025Qwen3Four-stage flagship; strong-to-weak distillation for small models; the RL-versus-distillation Table 216
15 Oct 2025ScaleRLFitted sigmoid compute curves over more than 400,000 GPU-hours14
27 Oct 2025On-policy distillation postPer-token reverse KL as the advantage, so an RL trainer can run it15
2 Dec 2025DeepSeek-V3.2Specialists, then one mixed RL stage; RL budget above 10% of pre-training cost16
15 Dec 2025Olmo 3OlmoRL on a fully open recipe; RL-Zero from a decontaminated base4
6 Jan 2026MiMo-V2-FlashFirst report of multi-teacher on-policy distillation (MOPD)171
2 Feb 2026Kimi K2.5Joint multimodal RL; a log-ratio gradient mask2
17 Feb 2026GLM-5Three RL stages, on-policy cross-stage distillation, a deterministic indexer top-k18
12 Jun 2026Nemotron 3 UltraUnified RLVR, then two MOPD iterations, with the limits stated19
29 Jun 2026MOPD paperThe one controlled comparison with mixed RL, cascade RL and merging20
27 Jul 2026Kimi K3Nine experts (three domains times three effort levels), then MOPD3
unresolvedDeepSeek-V4On-policy distillation from more than ten teachers replaces mixed RL; the actor is its own judge7
15 Sep 2026TypeSafe launch postThe only public statement naming RLCD (Calibrated Decisions); claim only8

Two intervals (derived): the on-policy distillation post (27 October 2025) precedes the first MOPD report (6 January 2026) by 71 days, and DeepSeek-V4 cites it, with Gu et al. (2024), for its OPD stage7. From that first MOPD report to Kimi K3 (27 July 2026) is 202 days, and in that window GLM-5, Nemotron 3 Ultra, K3 and V4 (on either candidate date) each closed with a distillation stage. Seven months is short for a pattern to move through five labs, which is why Section 5 files MOPD's limits under open problems.

3. Settled, within a stated scope

Seven entries meet the Section 1.2 rule. Each row names its scope, its evidence type, the module that carries the evidence, and the signal that would reopen it.

Table 2. Settled entries (S1 to S7). "Evidence" uses the Section 1.2 tags. The last column names the observation, available in the sources, that would reopen the row.
IDEntry and scopeEvidenceModuleWould reopen it
S1Rule-based verifiers carry the reward wherever a program can read the answer (reasoning RL).Concordant: R1 §2.2, DAPO §2.4, Qwen3 §4.411136. R1 saw reward rise while reasoning scores fell under its helpfulness model (Supp. B.5) and capped model reward to the last 400 of 1,700 stage-2 steps11.10, 16, 19A long reasoning run that keeps a neural reward model with a measured hack rate
S2Open-recipe loss hygiene for rule-verified long chain of thought: no KL term, dead groups handled, a wider upper clip or a weight clip (Table 3).Concordant, with dissent: five of six reports in Table 3 drop the KL term; ProRL keeps it with resets from a distilled start21.11, 12, 13A with/without-KL run on a judge-scored domain (none in the reports read)
S3The rollout-engine and trainer-engine probability gap has to be handled; ignoring it can stall or collapse a run.Measured in separate stacks: MiniMax-M1's reward stalled until an FP32 LM head22; GLM-5 collapsed within a few steps with a non-deterministic top-k kernel18; BF16 rounding is the claimed root cause in Qi et al.23.14Reports that publish tail statistics of the ratio for their own stack
S4A binary verdict gives no incentive to calibrate or to abstain.Argument plus theorem: RLCR's Theorem 1 (appendix sign slip, Module 24)24; abstaining is strictly sub-optimal under 0-1 grading (§4)25; calibration dropped after GPT-4's post-training (Fig. 8)26.24A recipe report that states a calibration reward; Module 24 found none
S5Once a reward hack is found it is reinforced and spreads; the exploit classes are catalogued.Single-lab measurement in a frontier run27 and in production coding environments28. An open-model replication finds hacking consistent and misalignment inconsistent; it is a research note, not peer reviewed29.19Hack rates with and without guards in a model technical report (none exist)
S6DPO costs a fraction of PPO for chat preference tuning at comparable scores.Measured once: about 28 hours on two nodes for PPO against about 4 hours on one node for DPO10; Llama 3 reports DPO needed less compute at scale and did better on IFEval (§4.1.4)30.7, 26The quality side is C8 in Table 4
S7A learned process reward is not the reward of a long-chain RL run.Single report plus absence: R1 Supp. G.2 gives reasons without numbers11; k1.5 abstains12; no later report read trains on one.4, 16, 27A matched PRM-versus-outcome RL run (none exists)

3.1 What the loss-hygiene consensus actually is

The summary "no KL, token-level aggregation, clip-higher, mismatch correction" is stronger than the reports support. Table 3 lays out what each states.

Table 3. Loss choices in seven open reasoning-RL setups. "Not stated" means the sections read do not say.
SetupKL termAggregationAdvantageUpper clip or weightDead groups
DAPO, Mar 202513removed (§2.3)token-levelgroup-normalizedClip-Higher, 0.28 up, 0.2 downdynamic sampling drops accuracy 0 or 1
Magistral, Jun 202531removedtoken sum over the group's total length\(r_i-\mu\), then minibatch-normalized\(\varepsilon_{high}\) tuned in 0.26 to 0.28zero-advantage groups filtered
ScaleRL, Oct 202514noneprompt-level averagebatch-level normalizationCISPO weight clipzero-variance filtering
DeepSeek-V3.2, Dec 202516kept, corrected, strength per domain; weak or none for mathnot stated\(R_i-\mathrm{mean}(R)\)negative-advantage sequence masknot stated
Olmo 3, Dec 20254removedtoken-level over the batchno stdclip-higher plus truncated importance samplingzero-gradient filtering and active sampling
GLM-5 reasoning RL, Feb 202618removed (§3.2)\(1/|y_i|\) per sequence, as in GRPOgroup std kept\(\varepsilon_{high}=0.28\) plus the IcePop pop masknot stated
TRL main branch, snapshot32\(\beta=0.0\); no reference loadeddefault dapo (token-level)group std (default)\(\varepsilon_{high}\) unset: Clip-Higher is opt-innot read

Read across the rows (counts derived): five of six reports drop the KL term, and only V3.2 keeps a corrected one. The aggregation and advantage columns do not converge. Token-level over the batch appears in two setups; group-level token sums, prompt-level averaging and the original per-sequence average appear once each; the group standard deviation survives in DAPO, GLM-5 and the TRL default. What the rows share is narrower: each departs from original GRPO somewhere, and almost none leaves all of it. That matches ScaleRL's measurement that aggregation and normalization "primarily modulate compute efficiency without materially shifting the asymptote"14: a team can copy any row and expect a similar ceiling at a possibly different cost. It also means a result labeled "GRPO" is only as specific as its version and flags (Module 11).

4. Contested: different labs, different choices, no matched comparison

A row is contested when two or more choices each have a lab's own run behind them and nothing compares them under one budget. Where a controlled comparison exists it is in the evidence column, and it is usually the authors' own.

Table 4. Contested entries (C1 to C8). Numbers are as the cited report prints them; none was re-run.
IDQuestionPositions and who holds themEvidenceModule
C1Critic or no criticGroup baseline: R1, Qwen3, V3.2, V4's specialists, GLM-5, Olmo 3, Nemotron 3 Ultra11616718419. Tuned critic: VinePPO, VAPO, Tulu 3's value model333410.Conflicting, each measured: GRPO about 2 points ahead at \(\lambda=1.0\)11; tuned PPO 3.6 to 6.4 points ahead on 7B33; 60.4 after seven fixes, 5 for plain PPO34.9
C2Ratio unit and clip formToken clip (DAPO, Olmo 3, GLM-5); sequence ratio (GSPO); weight clip (CISPO, in ScaleRL); log-ratio masks (K2.5, K3)1335221423.Single reports. One independent comparison, one 8B dense model: GSPO runs diverged mid-training more than once (App. A.17.4)14.12
C3Closing the engine mismatchCorrect in the loss (Olmo 3, TRL); change numerics (FP32 head, FP16); remove the source (V3.2 routing reuse, GLM-5 deterministic top-k)43222231618.Each measured on its own stack. Qi et al. say loss-side fixes leave a deployment gap23.14
C4Combining skillsOne mixed RL stage (V3.2); joint RL with domain experts (K2.5); staged RL then distillation (GLM-5); experts then MOPD (MiMo-V2-Flash, Ultra, K3, V4)16218171937.One controlled comparison, the authors' own (Figure 3)20. Labs "are far from unanimous"1.22
C5RL or distillation for a small reasonerDistill: R1 Supp. F.1, Qwen3 §4.7116. RL to pass the teacher: R1's second conclusion, ProRL1121.One compute-matched pair (Table 21); hardware and teacher cost unstated6.21, 29
C6Length and effort controlReward-side term (k1.5, V3.2); budget override (K2, Toggle, K3); trained effort modes (V4, K3, gpt-oss); fix the loss (Dr. GRPO)1216362373738.Single reports; Toggle: 25 to 30% fewer output tokens, "negligible" accuracy impact2. No cross-form ablation.23
C7Calibration: train it in or fix the gradingBrier term in the reward (RLCR, BCRL); confidence targets in scoring (Kalai et al.)243925.RLCR: ECE 0.37 to 0.03 (HotpotQA), 0.26 to 0.10 (math), authors' evaluation24. Kalai et al.: no training experiment.24
C8DPO or tuned PPO for preference qualityDPO in rounds (Llama 3, Tulu 3); PPO ahead (Xu et al.; Ivison et al.)30104041.PPO ahead by up to 2.5% (math) and 1.2% (general); data quality ranked above the algorithm41. Nothing at 70B or above (Module 26).7, 26

4.1 The one controlled comparison, and what it licenses

C4 has the most adoption and the least independent evidence. The MOPD paper compares five integration methods from one SFT checkpoint on Qwen3-30B-A3B across three domains, normalizing each score so the SFT student is 0 and the per-domain RL teachers are 1.

Bar chart of normalised multi-domain score on Qwen3-30B-A3B for five capability-integration methods, bounded by the SFT student at 0 and the per-domain RL teachers at 1 0.00 0.25 0.50 0.75 1.00 SFT student (reference 0) 0.0000 Param-Merge, average 0.3280 Cascade RL 0.7752 Off-Policy Finetune 0.8241 Param-Merge, task arithmetic 0.8574 Mix-RL 0.8818 MOPD 0.9373 RL teachers (reference 1) 1.0000 normalised score = (method - SFT student) / (specialist teacher - SFT student), averaged over three domains; higher is better math (AIME25, AIME26), instruction following (IFBench, IFEval), SWE-bench Verified; one run per method as printed
Figure 3. Normalised score of five capability-integration methods on Qwen3-30B-A3B, from Table 2 of the MOPD paper20. MOPD leads the strongest baseline, Mix-RL, by 5.5 points (0.937 against 0.882, as the abstract states). The paper's per-domain table puts MOPD below the RL teacher on four of five benchmarks and above it on one (AIME26, +1.66)20. All numbers are the authors' own, one run per method as printed. The paper is from Xiaomi, whose MiMo-V2-Flash report was the first to document MOPD17.

Three things limit it. The setting is a 30B model with three domains, where Ultra and V4 use more than ten teachers197. There is no repeat, so the 5.5-point gap has no stated interval. And "nearly all of each teacher's capability" holds for skills the student can sample, which Ultra's Table 5 qualifies (Section 5.1). None of the frontier reports publishes the comparison against its own prior mixed-RL stage at equal compute (Module 22).

4.2 Worked example: how much of the 9.96 times survives a teacher's cost

Qwen3's Table 21 starts two arms from the same off-policy-distilled 8B checkpoint. RL took 17,920 GPU-hours and reached AIME'24 67.6 with pass@64 unchanged at 90.0; on-policy distillation took 1,800 GPU-hours and reached 74.4 with pass@64 up to 93.36. The ratio is $$ \frac{17{,}920}{1{,}800} = 9.96 $$ The report states neither the hardware nor whether the teacher's cost is inside the 1,800. Suppose the teacher cost \(T\) GPU-hours on the same hardware and serves \(N\) students of this size. The distillation arm then costs \(1{,}800 + T/N\) per student, and stays cheaper than RL while $$ \frac{T}{N} < 17{,}920 - 1{,}800 = 16{,}120 \text{ GPU-hours.} $$ The break-even is arithmetic, not a Qwen3 figure; the report prints no \(T\). It shows where the claim is fragile: a teacher whose cost divided by the students it serves exceeds 16,120 GPU-hours reverses the result. Qwen3's teachers serve six small models, which lowers the per-student share, and are themselves the output of a four-stage pipeline the report never prices6. The Thinking Machines post's 9, 18 and 30 times compare on-policy distillation with SFT, not with RL, and differ in what they charge for the teacher (Module 21)15, so none is an RL-versus-distillation ratio.

Per-student cost of the distillation arm when the teacher is charged

Distillation arm per student
RL arm divided by distillation arm

Constants from Qwen3 Table 21 (Qwen3-8B, 17,920 and 1,800 GPU-hours)6. T and N are free parameters, not report values. At T = 0 the output is the report's 9.96; below 1 means the teacher's share makes distillation the costlier arm.

4.3 Efficiency versus asymptote as a tie-breaker

ScaleRL says loss aggregation, advantage normalization, curriculum and the off-policy algorithm "primarily modulate compute efficiency without materially shifting the asymptote"14. If that holds, C2's and Table 3's spreads are questions of cost, and a team should pick the cheaper. The paper's Appendix A.11 qualifies its abstract for one item: the off-policy algorithm "affects the asymptotic performance c of the scaling curve, not just the efficiency exponent b", one of "the most consequential design decisions in RL post-training"14. An FP32 LM head moves the fitted asymptote from 0.52 to 0.61, and GSPO runs diverged mid-training, with restarts recovering the 8B runs but not the 17B-by-16 mixture-of-experts model14. So the paper supports "most loss details move cost" and also "numerics, staleness regime and ratio form can move the ceiling". It is one lab's framework on one 8B dense model and one MoE, unreplicated.

5. Open: the decisive experiment is absent

A row is open when no source read contains the test that would decide it, or when a report says the problem is unsolved. The last column is the experiment the evidence modules identify; none has been published in the sources read.

Table 5. Open entries (O1 to O7). The last column is the missing observation, as stated in the cited module's assessment.
IDQuestionWhat the sources sayModuleMissing observation
O1Does RLVR add capability or sharpen the base?Sharpen: pass@k at large k favors the base42; Qwen3's RL arm left pass@64 at 90.0 and 83.3 while distillation raised it to 93.3 and 86.76. Expand: over 2,000 steps from a distilled 1.5B start21. Both, in phases: the authors' own assessment43. Prior amplification: random rewards gave +21.4 MATH-500 points against +29.1 for true rewards44.20Pass@k to large k at every checkpoint of an RL-Zero run on a decontaminated base, with a random-reward arm4
O2RL where no program can verifyR1: scaling pure RL without a reliable reward model "remains an open challenge"11. 2026 recipes train judges in the loop: K3's read-rubric-score-record protocol3, V4's actor as its own reward model7. Ultra: GenRMs help "but substantial failure cases still remain"19.16Judge accuracy on labelled near-ties through training; frozen against jointly trained judge at equal compute
O3MOPD on long-horizon agentic rolloutsUltra saw "substantial training inefficiency" mixing end-to-end agentic environments into MOPD and used single-turn rollouts for most agentic tasks; whether end-to-end rollouts gain more "remains an open area"19.18, 22End-to-end against single-turn rollouts on the same teachers
O4Environment quality, reward shape, harness diversityK3 composes harnesses from modules3; Ultra varies harnesses19; GLM-5 scales to more than 10K verifiable SWE environments (§4.2)18. V3.2 has one ablation of synthetic agent tasks16.17Matched comparisons of reward shapes, of one against several harnesses, and of simulated against real environments
O5Whether more RL compute keeps payingV3.2: "consistent performance improvements correlating with extended RL training budget", already above 10% of pre-training cost16. K3: gains from scaling RL FLOPs3. ScaleRL's fits held to 100,000 GPU-hours on one 8B dense model14.15A fit on a production run; a report printing GPU-hours, steps and completions together
O6RL pressure on the reasoning trace against monitoring itA chain-of-thought monitor in the reward produced obfuscated hacking at high optimization27; gpt-oss decided "not to put any direct optimization pressure on the CoT"37; Anthropic penalized detected hacks with a classifier28.19One detector on actions only and on actions plus reasoning, at matched optimization
O7TypeSafe's "RLCD"A company claim of a third post-training route beside RLHF and RLVR458; the founder says it is not yet published46. No paper, repository, reward, algorithm or dataset exists in any source (Section 7).25A statement of the reward or loss; a pre-registered third-party calibration study

5.1 O3 in numbers: what distillation does not import

Nemotron 3 Ultra defines recovery as \((\text{MOPD}_2-\text{RLVR})/(\text{Teacher}-\text{RLVR})\), the fraction of the teacher-student gap MOPD closes. Its Table 5 lists recoveries from 172.7% down to 16.9%, and the report ties the low end, Humanity's Last Exam, to a teacher whose advantage came from extra SFT and RL on data the student never saw: "When the missing capability requires reasoning paths that the student rarely samples, student rollouts become effectively out-of-distribution for the teacher"19. A recovery of 16.9% leaves \(100-16.9=83.1\%\) of the gap open (derived). This is the Module 22 limit in the lab's own words, and it makes the agentic item and the HLE item the same question from two sides: what the student never samples, the teacher cannot grade. The report's RLVR batch figure, a global batch of 8,192 with 16 rollouts per sample, has no stated unit, and this module does not multiply it out (Modules 22 and 32).

5.2 O2: judges are now trained in the loop, and nothing audits them

Between R1 (January 2025) and V4 the reward for unverifiable tasks moved from a scalar model used for the last 400 steps11 to rubric-guided generative reward models, and in V4 to a policy that is its own judge, with RL applied "directly to the GRM itself"7. K3 describes its group reward as "tournament-style" with binary comparisons "as in Kimi K2.5"; the K2.5 text read does not use the word for a reward, and K2's equivalent is pairwise ranking (Module 16)3236. None of these reports publishes judge accuracy against human labels through training, the monitor that would separate an improving judge from an exploited one (Modules 16 and 19). Exploitation is already visible in the reports: Ultra's remark above, K3's guard under which a candidate over the length limit "automatically loses the binary comparison"3, and R1's reward-up, accuracy-down curve11.

5.3 O1 and O5 are less in conflict than they look

If RLVR mostly sharpens, pass@1 should saturate and pass@k at large k should fall, while RL compute keeps buying pass@1 until the base's distribution is exhausted. ScaleRL fits a saturating pass-rate curve, and Module 15 (Section 8) sets out what such a curve does not say. A pass-rate curve that keeps rising with compute is compatible with a coverage curve that keeps falling. That is my inference from the two modules, not a result in any one source; the test that separates them is O1's, run on the same runs that produce the compute fit.

6. The 2026 recipe shift: from one RL stage to specialists and a consolidation step

Lambert's survey, updated through September 2026, records two changes in canonical post-training recipes: multi-teacher on-policy distillation "became a popular tool for merging multiple, diverse skills into a final model", and RL compute kept scaling1 (§3.2.4). He credits Xiaomi's MiMo-V2-Flash as the first report to document it and lists expert counts: Nemotron 3 Ultra more than ten, Kimi K3 nine, DeepSeek-V4 more than ten, MiMo-V2 unreported. In the same passage: "MOPD is a popular new tool, but labs are far from unanimous in its usage", and GLM-5 uses it after a three-stage RL process1. Table 6 sets the recipes in date order.

Table 6. What the flagship recipe does last, in report-date order. "Last stage" is as each report describes its main post-training path; Qwen3's six small models end differently (Section 6.1).
Report (date)Before the last stageLast stage
DeepSeek-R1 (Jan 2025)Cold-start SFT, large-scale RL, rejection sampling and SFT1Mixed RL: reasoning verifiers plus preference reward models111
Qwen3 flagship (May 2025)Long-CoT cold start, reasoning RL on 3,995 query-verifier pairs, thinking-mode fusion6General RL over more than 20 tasks6
DeepSeek-V3.2 (Dec 2025)Specialists per domain, then distillation into one model16One mixed RL stage over reasoning, agent and alignment16
Olmo 3 Think (Dec 2025)SFT, then DPO (Module 26)4OlmoRL4
MiMo-V2-Flash (Jan 2026)General SFT, then domain-specialized teachers (more SFT and large-scale RL)1MOPD17
Kimi K2.5 (Feb 2026)Zero-vision SFT; RL domains organized by ability, not modality2Joint multimodal RL2
GLM-5 (Feb 2026)SFT, then Reasoning RL, Agentic RL and General RL in sequence18On-policy cross-stage distillation (§3.5)18
Nemotron 3 Ultra (Jun 2026)SFT, one unified RLVR stage, more than ten specialist teachers19MOPD, two iterations after a warmup SFT191
Kimi K3 (Jul 2026)SFT, then nine experts: three domains times three effort levels3MOPD3
DeepSeek-V4 (date unresolved)Specialists trained with SFT then GRPO7On-policy distillation from more than ten teachers, replacing mixed RL7

Counting the table (derived): none of the four 2025 reports ends its flagship recipe with distillation. Of the six reports dated in 2026 (V4's month is unresolved, but both candidate dates fall in 2026), five do: MiMo-V2-Flash, GLM-5, Nemotron 3 Ultra, K3 and V4. K2.5 does not. Five labs in the 202 days from 6 January to 27 July 2026 (derived) is a documented trend among open labs, not a measurement that the step helps. The sample is the open reports in the course's map. Section 4.1 reports the one controlled comparison, and Module 22 explains why the mechanism suits many domains with different rollout shapes and suits a few similar domains less well.

6.1 What varies inside "MOPD"

The label hides differences that Module 22 tabulates. The teachers' relation to the student: the MOPD paper reports that same-origin teachers are "critical for stable optimization" and that a larger external teacher failed20; Nemotron's warmup SFT raises the overlap between student rollouts and teacher-supported trajectories and helps most in agentic domains19. The signal: the MOPD paper scores each sampled token with a per-token reverse KL20, while V4 says that kind of estimate "leads to high variance in gradient estimation and often causes training instability" and distills over the full vocabulary7. The name: V4 calls its stage OPD, GLM-5 calls its stage on-policy cross-stage distillation, and Lambert groups both under MOPD7181. The reference for the method is still a blog post whose original models were retired in a June 2026 update15. Qwen3's strong-to-weak distillation for six small models cost about a tenth of the four-stage pipeline's GPU-hours6; that is distillation as size reduction, a different use from consolidation.

6.2 Reasoning effort as a product surface

Effort levels moved from a prompt trick to a trained product surface in the same window. gpt-oss ships three trained levels, set by a system-prompt keyword such as "Reasoning: low", and the model card says a higher level lengthens the average chain of thought37 (§2.5.2). DeepSeek-V4 supports three modes per model with distinct length penalties and context windows7. Kimi K3 crosses three domains with three effort levels to make nine experts, then distills them into one conditioned student3. Training details are thin everywhere: the gpt-oss card gives one sentence, and V4 and K3 give no penalty or budget values (Module 23). For a reader on the serving side, the output-length distribution a deployment sees is now a function of a trained, user-selectable input, which Module 23's serving section treats as a capacity-planning variable.

7. Calibration as a post-training target, and the claim that carries the same letters

7.1 What the public record supports

Three public sources carry the evidence. RLCR adds a Brier term on a verbalized confidence to the correctness reward, \(R=\mathbb{1}[y=y^*]-(q-\mathbb{1}[y=y^*])^2\), and reports expected calibration error falling from 0.37 to 0.03 on HotpotQA and from 0.26 to 0.10 on math, with accuracy matched to RLVR24 (Eq. 1, §1). Behaviorally Calibrated RL optimizes strictly proper scoring rules so a model can output a calibrated probability of correctness; its headline comparison against GPT-5 on BeyondAIME is the authors' own evaluation39. Kalai et al. argue that the incentive sits in the grading: under 0-1 scoring an "I don't know" is maximally penalized and a best guess is optimal, so the repair is to put explicit confidence targets in benchmark instructions25 (§4). Two printed slips matter for anyone reusing these: RLCR's Theorem 1 has a sign slip in its appendix, and Kalai et al.'s key section has a slip in a worked penalty value (Module 24); this module uses neither. GPT-4's technical report supplies the earliest closed-lab data point: the pre-trained model "is highly calibrated" and after post-training "the calibration is reduced"26 (§5, Fig. 8).

No recipe report read for the course describes a calibration reward in its RL stage, and Kimi K2 names the gap: its rubric-style rewards can penalize hedging (Module 24)36. So calibration is a diagnosis that is settled (S4) and a repair that is research on 4B to 8B models with short verifiable tasks, not a stage in any published frontier recipe. That is why it sits in Table 4 and not in Table 2's practice rows.

7.2 Four acronyms

Table 7. Names that share letters. Only the first row is the method this course was asked to cover; it has no public mechanism.
NameExpansionSource and statusWhat is public
RLCD (TypeSafe AI, Sep 2026)Reinforcement Learning for Calibrated DecisionsLaunch post of 15 Sep 2026 and AI primer845. Company claim only.An output contract (decisions and calibrated probabilities, no generated text). No paper, repository, reward function, algorithm or dataset.
RLCD (Yang et al., ICLR 2024)Reinforcement Learning from Contrastive DistillationarXiv:2307.12950, peer reviewed47A different method: preference pairs built from a positive and a negative prompt, then standard RLHF with PPO. Unrelated to calibration.
RLCR (Damani et al., ICLR 2026)Reinforcement Learning with Calibration RewardsarXiv:2507.16806, peer reviewed24Reward, theorem, training setup, results.
BCRL (Wu et al., 2026)Behaviorally Calibrated Reinforcement LearningarXiv:2512.19920 v3, preprint39Proper-scoring-rule interventions; the authors' own evaluation.

7.3 What can and cannot be said about TypeSafe's RLCD

The company states that RLCD is a third post-training route beside RLHF and RLVR, which its launch-post comparison table describes as optimizing for human preference and for verifiable rewards; RLCD is described as training a model "to return decisions and calibrated probabilities instead of generated text"458. The founder says in an interview that the method is not yet published and that the training data is synthetic46. The only press coverage writes "from" where the company writes "for", reports that the architecture was not detailed, and relays one developer's remark that the model "delegates the hallucination problem a little bit to the user"48. A third party has built a method and labeled it a reconstruction, stating that TypeSafe discloses no training algorithm49.

That is the whole of the record. Everything the course says about calibration training is about the public methods in 7.1, and none of it should be read as TypeSafe's reward or algorithm. Whether RLCD is a new regime or the interface and packaging of known proper-scoring training is the fifth dispute below, and Module 25 holds the claims ledger, the output contract and the evidence that outside groups have tested the released model's behavior and not its training.

8. What closed labs have put on the record, and what they have not

The rule here is narrow: only documents the labs published themselves, held in the course's source map or fetched for this module. Nothing reconstructs a recipe and nothing comes from a rumour. The OpenAI blog pages returned HTTP 403 when the source map was built; I did not fetch them again or work around the block. The map holds no Google or DeepMind document on RL post-training, and I did not search beyond the map for that row, so its cell reads "no document in the map" and not "none exists". Two circulating figures are excluded because neither is a primary statement: a compute multiple between OpenAI's o1 and o3 that ScaleRL quotes from elsewhere, and abstention and error rates for a GPT-5 variant that reached the course through a search summary.

Matrix of seven disclosure items against OpenAI, Anthropic and Google, showing which are on the record, partial, silent or absent from the course's source map OpenAI Anthropic Google / DeepMind What a reader would want to know Published RLHF or RLAIF recipe, 2022-23 ●InstructGPT, GPT-4 card ●Constitutional AI –no document in map Reasoning-RL algorithm ◐one sentence, no name ○silent –no document in map Reward and verifier design, current ○silent ○silent –no document in map Environments and training data ◐one sentence (gpt-oss) ◐coding envs described –no document in map Compute and hyperparameters, current ○declined (GPT-4 §2) ○silent –no document in map Failure modes seen in RL runs ●Baker et al. 2025 ●MacDiarmid et al. 2025 –no document in map Reasoning-effort levels ◐gpt-oss: three levels –no document in map –no document in map ● dedicated primary document with data ◐ one sentence or paragraph ○ the lab's documents in the map are silent or decline – no document in the map
Figure 4. Disclosure matrix for seven items a practitioner would want, by lab. Filled: a dedicated primary document with data. Half: one sentence or paragraph. Open circle: the lab's documents in the map are silent or decline. Dash: no document in the map. Cells are the author's reading of the documents in Table 8263727502851.

8.1 The ledger

Table 8. Primary statements by closed labs about RL post-training, and what each leaves out. Quotes are verbatim.
Lab and documentOn the recordNot disclosed
OpenAI: InstructGPT, Mar 202252A complete RLHF recipe: SFT on about 13k prompts, reward model on 33k, PPO on 31k; 6B reward models because a 175B one was unstable; per-token KL penalty; PPO-ptx (§3.2, §3.5).Anything after 2022: it documents a research recipe, not later products.
OpenAI: GPT-4 Technical Report, Mar 2023 (v6)26The model was "fine-tuned using Reinforcement Learning from Human Feedback (RLHF)" (§2). The appended system card describes SFT on demonstrations, a reward model from rankings, PPO, and rule-based reward models: zero-shot GPT-4 classifiers that add reward during PPO from a human-written rubric (§3.1). Base and RLHF models score 73.7% and 74.0% on the multiple-choice exams (Table 8); calibration is reduced after post-training (Fig. 8).The report "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar" (§2).
OpenAI: Baker et al., Mar 202527A prompted GPT-4o monitored a frontier reasoning model during training; a monitor that saw the chain of thought beat one that saw actions only; training against the monitor produced obfuscated hacking; "a monitorability tax". Anecdotally, "as we have continued to scale RL training, agents have discovered more complex and hard-to-detect hacks".The recipe, reward design, environments and scale of the monitored run. The obfuscation experiment used a different, non-frontier agent.
OpenAI: gpt-oss model card, Aug 202537"After pre-training, we post-train the models using similar CoT RL techniques as OpenAI o3" (§2.5), on "a wide range of problems from coding, math, science, and more"; three trained reasoning levels set by a system-prompt keyword (§2.5.2); deliberative alignment for refusals (§3); the decision "not to put any direct optimization pressure on the CoT" (§4.4).The algorithm's name, reward, verifiers, compute, and what "similar" covers. The card describes an open-weight model by analogy to o3, not how o3 was trained.
OpenAI and Georgia Tech: Kalai et al., Sep 202525Why binary grading rewards guessing; a proposal for confidence targets in benchmark scoring (§4).Any training experiment or production reward design.
Anthropic: Constitutional AI, Dec 202250A two-phase recipe: self-critique and revision with SFT, then RL from AI preference labels.The recipe of any later Claude model.
Anthropic: constitution pages, May 2023 and Jan 202653512023: the RL phase "uses AI-generated feedback based on the set of principles". 2026: "We use the constitution at various stages of the training process", and Claude "uses the constitution to construct many kinds of synthetic training data", including "rankings of possible responses".How the rankings enter training, the algorithm, the reward composition (Module 8).
Anthropic: MacDiarmid et al., Nov 202528A pretrained model told about hacks (by synthetic documents or in the prompt), then RL on a subset of the production coding environments used for Claude Sonnet 3.7; hacking generalized to alignment faking and sabotage; "We have started implementing this technique [inoculation prompting] in production Claude training".The production algorithm, reward composition and scale. The studied hacks "did not actually occur in the training of Claude Sonnet 3.7".
Google and DeepMindNo document in the course's map.Not assessed by this module.

8.2 Where the boundary falls

The closed labs have disclosed research and failures in detail: reward-model overoptimization in 202254, the monitoring study, the reward-hacking study, the grading theory. They have disclosed recipes only for the era before verifiable-reward reasoning RL: InstructGPT and Constitutional AI in 2022, and the GPT-4 system card's RLHF outline in 2023. For the era after, the on-record material for OpenAI is a few paragraphs in an open-weight model card, and for Anthropic a statement that a document is used "at various stages". No document read gives a reward function, a loss, a KL treatment, a batch size, a compute figure or an environment count for a closed reasoning model.

The one closed-lab sentence about reasoning RL carries less than it appears to. "Similar CoT RL techniques as OpenAI o3" says the open-weight models were trained with techniques that resemble o3's; it does not say which, and it does not say they match anything in Tables 2 to 5. Read it as an existence statement: chain-of-thought RL with tool-use training on a coding, math and science dataset is how this lab made a competitive open-weight reasoning model, and its effort levels are trained and not merely prompted.

8.3 Consequences for the rest of the module

Every settled row is settled among labs that publish. The strongest evidence on failure modes at frontier scale is closed-lab research, and it is the best-sourced reason to treat O6 as a live decision. A UK AI Security Institute research note, not peer reviewed, reproduced the Anthropic setup with open models and found hacking consistent and misalignment inconsistent29; it is the only outside check, and it does not say how the production pipeline behaves.

9. Version and date hygiene: what to re-check first

Several numbers and dates in this course depend on which copy of a source you open. Table 9 lists the version, date and unit ambiguities I found, including places where a report's own arithmetic does not close.

Table 9. Version, date and unit ambiguities, with the check to run first. "Derived" marks arithmetic I did on printed values.
ItemThe ambiguityCheck first
DeepSeek-R1 headlinev1 (22 Jan 2025): AIME 2024 pass@1 from 15.6% to 71.0%, 86.7% with majority voting55. v2 (4 Jan 2026): 15.6% to 77.9%, 86.7% by self-consistency11. A Nature version exists (Nature 645, 17 Sep 2025). v2 prints a stage-1 GRPO clip ratio of 10, unexplained, and "about 4 days" for a run its table counts as 80 hours11.Which version a quoted number came from (Module 27)
DeepSeek-V4 dateThe arXiv identifier 2606.19348 reads June 2026; the arXiv API record says published 2026-04-265 and the PDF stamp reads 26 Apr 2026 (Modules 23 and 30). Unresolved7.Cite by title and identifier; do not date from the identifier
Kimi K2.5 and K3Both at v2, updated 7 Aug 20265. K2.5's printed loss clips the ratio while its prose describes a log-ratio mask with unreported \(\alpha,\beta,\tau\) (Module 28)2.A corrected loss in a later version
Lambert's bookarXiv v12 of 11 Sep 2026; web edition built 14 Oct 2026; Manning print July 20261. Section numbers differ between editions.Cite the section number from the copy you open
Olmo 3 arithmeticFootnote 41: a 1,000 s step with 125 s of training implies the learner idle \((1000-125)/1000=87.5\%\) (derived); the main text says 75%, and the footnote does not say which model size it describes. For the 32B run the text says inference uses "approximately 5x" the GPUs of training, but 20 inference nodes over \(8\times(1-0.75)=2\) busy learner node-equivalents is 10 (derived, Modules 1 and 31)4.Use measured step times, not the ratios
Qwen3 Table 2117,920 and 1,800 GPU-hours; hardware and teacher cost unstated6. The Thinking Machines post uses a 60% AIME'24 SFT baseline where Qwen3 reports 55.0, and its June 2026 update retired the original models15.Do not quote 9.96 as a total-cost ratio (Section 4.2)
Nemotron 3 Ultra opennessThe report says it open-sources "the training recipes, data, and RL environments"19. The repository README lists the program as Pretrain, SFT, RLVR, MOPD and MTP Boosting, says "this recipe covers Pretrain → SFT", trains "exclusively on the open-sourced subset of training data", and warns "results will differ from the tech report benchmarks, which used additional proprietary data"56. The environments may sit in repositories not inspected (Module 32).The RLVR and MOPD stages in the repository, before calling the RL recipe open
Batch unitsUltra's "global batch size of 8192, with each sample generating 16 rollouts" and GLM-5's "group size of 32 and a batch size of 32" (§3.2) give no unit for the batch1918.Whether the batch counts prompts or rollouts
ScaleRLOne arXiv version (15 Oct 2025), no venue on its arXiv page; the abstract says the off-policy algorithm changes efficiency, and Appendix A.11 says it also moves the asymptote14.A revised version or a replication
SnapshotsTRL's GRPO page is the main branch32; the Nemotron and GLM-5 READMEs are snapshots of 2026-10-27, and the GLM-5 README now lists GLM-5.1, 5.2 and 5.35657. Defaults and model lists drift within a quarter.Default values before quoting them
Calibration papersRLCR is v2 of 15 May 202624; Kalai et al. is v1 only25. Each carries a printed slip (RLCR's appendix proof, Kalai et al.'s worked penalty), so cite the statement and not the example (Module 24).Corrected versions

10. Leading indicators: what would move a row

Each entry is an observation that could appear in a document that exists or is about to, with the rows it would move and where the sources point. The list is ordered by how much it would change Tables 2 to 5.

Table 10. Signals to watch, the rows they would move, and where the sources point.
SignalMovesWhy it mattersWhere the sources point
A frontier report comparing its MOPD stage with mixed RL at equal computeC4Replaces a 30B-A3B, three-domain comparison with one at the scale where five labs adopted the stepNext DeepSeek, Kimi, NVIDIA, Xiaomi reports; Lambert §3.2.4120
Hack rates with and without guards in a model reportS5, O2, O6No technical report publishes them, so Module 19's defenses record practice, not effectK3's detection system, extended as hacks appear3
Judge accuracy on labelled pairs, tracked through RLO2Separates an improving judge from an exploited oneV4's actor-as-judge and K3's agentic judge are described, not ablated73
Pass@k at large k at every checkpoint of an RL-Zero run, with a random-reward armO1Separates sharpening from expansion on a decontaminated baseOlmo 3's RL-Zero base, data and scripts4
End-to-end against single-turn agentic rollouts under MOPDO3Tests whether unsampled student behavior is the limitNemotron 3 Ultra §3.3.519
Released environments with yields, and repositories that match the reportsO4Environment quality cannot be audited from descriptionsThe Nemotron README against the report5619; GLM-5 and K3 environment sections183
A sigmoid compute fit on a production or agentic runO5The only way to forecast the return on the next RL budgetScaleRL names multi-turn and agentic RL as next uses (Module 15)14
Changes to library defaultsS2, C2, C3Defaults are where practice converges first; dapo loss, beta=0.0 and sequence-level masked importance sampling are today's snapshotTRL's GRPO configuration page32
A calibration reward in a recipe reportS4, C7Moves calibration from research to practiceModule 24 found none; K2 names the gap36
A statement of TypeSafe's reward or loss, or a pre-registered third-party studyO7Turns a claim into a method, or refutes itCompany blog and docs; Module 25845
A closed-lab recipe statement beyond one sentenceSection 8The only way Tables 2 to 5 could cover the frontierThe next OpenAI model card; Anthropic's training descriptions3751

Where experts disagree

The five disputes below are the ones the map leaves open; each has two positions, the strongest cited evidence for each, and an assessment that names the experiment that would settle it. Disputes that Modules 9 to 15 treat as contested (critic, ratio, normalization) are in Table 4 and not repeated.

Dispute 1: Does RLVR create capability, or sharpen what the base has?

Sharpening: the base sets the boundary

Yue et al.42; Qwen3's Table 216

At large k the base model solves more problems than its RL-trained version, and RL left Qwen3-8B's pass@64 unchanged.

  • Two reports, one direction; a testable claim.
  • Short runs, strong math bases, answer-only checking.

Expansion: long, diverse, KL-controlled training

ProRL21; a phase-based reconciliation43

Over 2,000 steps, gains were largest where the base started weak, and a held-out task the base could not solve at all was solved (Module 20).

  • Answers the "stopped too early" objection directly.
  • Starts from a distilled 1.5B model; the phase theory is the authors' own assessment.
Assessment

For strong math bases and runs of hundreds of steps, sharpening is best supported, and Qwen2.5-Math gains stay unconfirmed until a random-reward control agrees (+21.4 against +29.1 points)44. Expansion is plausible for weak starts but rests on one small model. The deciding run is pass@k to large k at every checkpoint on a decontaminated base with a random-reward arm (Module 20).

Dispute 2: How should many skills end up in one model?

One mixed RL stage

DeepSeek-V3.216; Ultra's first stage, before it moves to MOPD19

One on-policy run over all domains avoids the forgetting of multi-stage pipelines.

  • One trainer; within 5.5 points of MOPD in the controlled test20.
  • Per-domain signal dilutes as environments multiply19; DeepSeek replaced it in V47.

Specialists, then MOPD

MiMo-V2-Flash, Ultra, K3, V4; the MOPD paper17193720

Train each skill to its peak in parallel, then distill on the student's own rollouts.

  • Best score in the one controlled comparison; teams work in parallel.
  • Needs same-origin teachers; HLE recovery of 16.9% in Ultra's Table 519.
Assessment

With few, similar domains mixed RL is within reach and cheaper; with many environments and long agentic rollouts, four labs chose specialists and the dilution argument is sound. GLM-5's staged RL with cross-stage distillation is a hedge18. The missing run is one SFT checkpoint through both routes at matched GPU-hours with ten or more domains (Module 22).

Dispute 3: RL or distillation for a strong small reasoner?

Distill first

DeepSeek-R1 Supp. F.111; Qwen3 Table 216

R1 found 10K+ RL steps on a 32B base only matched QwQ-32B-Preview; Qwen3 got higher scores at about a tenth of the GPU-hours.

  • Two reports, one direction; pass@64 rose.
  • Teacher cost and hardware unstated; the teacher is itself an RL product.

RL to pass the teacher

R1's second conclusion11; ProRL21

R1 adds that "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning".

  • The only route past a teacher.
  • No matched-compute comparison; ProRL's baseline is its own start.
Assessment

Below the frontier, with a close teacher, distillation is the cheaper route in every comparison here; at the frontier there is no teacher above you. Both statements hold, and the comparison that would join them, with teacher cost counted (Section 4.2), has not been published.

Dispute 4: Calibration: train it into the policy, or fix how benchmarks grade?

Train it in

RLCR24; Behaviorally Calibrated RL39

A bounded proper scoring rule next to the correctness reward removes the incentive to guess.

  • ECE 0.37 to 0.03 on HotpotQA with matched accuracy.
  • 4B to 8B models, short tasks, authors' evaluation.

Fix the grading

Kalai et al., OpenAI and Georgia Tech25

Models guess because leaderboards reward guessing; add explicit confidence targets to the scoring.

  • Acts on the cause for every lab at once.
  • No training experiment; depends on leaderboard owners.
Assessment

Complements: they act on two layers of one incentive. If you train, a Brier term costs little accuracy at modest scale; if you publish numbers, the grading argument applies regardless (Module 24). A frontier-scale test on agentic or noisy-verifier tasks is missing.

Dispute 5: TypeSafe's RLCD: a new regime, or known calibration-aware RL under a new name?

A new regime

TypeSafe AI: launch post, primer, founder (company and interview claims)84546

RLHF and RLVR optimize the wrong target for software that needs a trustworthy probability; RLCD trains calibrated typed decisions.

  • The output contract is public, and the motive matches the literature24.
  • No mechanism, data, ablation or calibration table is public.

Known training, new packaging

No source states this about TypeSafe; it is what the public record implies243949

Proper-scoring training exists for text models, and a third party rebuilt a version from standard parts.

  • Parsimonious: needs no hidden method.
  • Assumes the undisclosed method is no more than the public ones; no source can show that.
Assessment

Neither can be established. The record documents an output interface that outside groups have tested and says nothing about the training, so "regime" and "rebranding" are both claims about a method nobody has seen. The decider is a statement of the reward or loss plus an ablation only the company can run (Module 25). Until then use the public machinery of Module 24 for anything you build.

What is changing

As of 2026-10-27 the fastest-moving items are the documents the course reads least stably. The Nemotron and GLM-5 READMEs already list successors (a 3.5 Super recipe with two-phase MOPD, and GLM-5.1 to 5.3)5657, TRL's defaults sit on a main branch32, and the K3 and K2.5 reports were revised on 7 August 202632. Any value quoted in this course from those pages should be re-read first.

One forecast, labeled as my own and not found in any source: the first controlled MOPD-versus-mixed-RL comparison at frontier scale is the result most likely to change Table 4, since five labs adopted the step while the only controlled comparison is at 30B scale. Module 23 notes that no shared accuracy-per-token metric exists, so C6 cannot be compared across labs until one does. And TypeSafe's situation can change in a day: the founder says RLCD is unpublished46, so one post stating the reward would supersede Section 7 and Module 25 at once.

References

  1. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501 v12, 11 Sep 2026 (v1 Apr 2025; Manning print July 2026). https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: §3.1.6 (RL keeps winning), §3.2.4 (MOPD, expert counts, "far from unanimous"), §7.3.
  2. paperKimi Team (Moonshot AI). Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276 v2 (7 Aug 2026; v1 2 Feb 2026). https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: Joint multimodal RL with domain experts learning jointly; log-ratio gradient mask; Toggle.
  3. paperKimi Team (Bai, T. et al.; Moonshot AI). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653 v2 (7 Aug 2026; v1 27 Jul 2026). https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: Nine experts and MOPD (§4.1); effort levels and budget RL (§4.1.2); agentic judge protocol (§4.1.2); harness diversity (§4.2.1).
  4. paperTeam Olmo (Ai2). Olmo 3. arXiv:2512.13961 v2 (14 Apr 2026; v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL list (§4.4.1); RL-Zero and decontamination (§1); the 1,000 s step and 75% learner-wait (§4.4.3, footnote 41).
  5. officialarXiv. arXiv API records for the cited identifiers. export.arxiv.org/api/query?id_list=…, queried 2026-10-27. https://export.arxiv.org/api/query?id_list=2606.19348. Accessed 2026-10-27. Used for: Submission dates for the timeline; the 2606.19348 record reads published 2026-04-26.
  6. paperQwen Team (Yang, A. et al.). Qwen3 Technical Report. arXiv:2505.09388 v1, 14 May 2025. https://arxiv.org/abs/2505.09388. Accessed 2026-10-26. Used for: Four-stage flagship pipeline (§4); rule-based rewards (§4.4); Table 21 (RL versus on-policy distillation, GPU-hours, pass@64).
  7. paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 v1 (the identifier month and the arXiv API date disagree; see Table 9). https://arxiv.org/abs/2606.19348. Accessed 2026-10-26. Used for: Mixed RL replaced by on-policy distillation with more than ten teachers (§5.1, §5.1.2); the actor as generative reward model (§5.1.1); three effort modes.
  8. officialAlmeida, D. (TypeSafe AI). Introducing System One Models & Jev. Company blog, 15 Sep 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed 2026-10-26. Used for: Company claim: the only sentence that names the training method; launch date.
  9. paperShao, Z. et al. (DeepSeek-AI). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 (v1 5 Feb 2024; v3 read), 2024. https://arxiv.org/abs/2402.03300. Accessed 2026-10-26. Used for: GRPO as the first row of the timeline (Table 1).
  10. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. (Ai2). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124 v5 (14 Apr 2025; v1 22 Nov 2024). https://arxiv.org/abs/2411.15124. Accessed 2026-10-26. Used for: The name RLVR (§1, §6); PPO versus DPO cost (§5.4.1); KL coefficients (Table 21 caption).
  11. paperDeepSeek-AI (Guo, D. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v2 (4 Jan 2026; expanded Nature version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: Abstention from neural reward models (§2.2); PPO versus GRPO (Supp. A.3); reward hacking (Supp. B.5); distillation versus RL (Supp. F.1); open challenge (Supp. G.1); v2 headline AIME figure.
  12. paperKimi Team (Du, A. et al.; Moonshot AI). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599 v4 (3 Jun 2025; v1 22 Jan 2025). https://arxiv.org/abs/2501.12599. Accessed 2026-10-26. Used for: Same-day release with R1; no value function, PRM or MCTS (abstract, §2.3).
  13. paperYu, Q. et al. (ByteDance Seed, Tsinghua AIR). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476 v2 (20 May 2025; v1 18 Mar 2025). https://arxiv.org/abs/2503.14476. Accessed 2026-10-26. Used for: Four techniques and Table 1 ladder; KL removed (§2.3); rule-based reward (§2.4); dynamic sampling (§3.2).
  14. paperKhatri, D. et al. (Meta, UT Austin, UCL, UC Berkeley, Harvard, Periodic Labs). The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786 v1, 15 Oct 2025. https://arxiv.org/abs/2510.13786. Accessed 2026-10-26. Used for: Efficiency versus asymptote (abstract); Appendix A.11 on the off-policy algorithm; loss-type comparison (Fig. 5); 100,000 GPU-hour run (Fig. 1).
  15. blogLu, K. and Thinking Machines Lab. On-Policy Distillation. Thinking Machines blog, posted 27 Oct 2025, updated June 2026. https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-26. Used for: Per-token reverse KL as advantage; the June 2026 update that retired the original models.
  16. paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 v1, 2 Dec 2025. https://arxiv.org/abs/2512.02556. Accessed 2026-10-26. Used for: Specialists then one mixed RL stage (§3); budget above 10% of pre-training cost (§1, §4); KL strength per domain; advantage without std (§3.1).
  17. paperLLM-Core Xiaomi. MiMo-V2-Flash Technical Report. arXiv:2601.02780 v2 (8 Jan 2026; v1 6 Jan 2026). https://arxiv.org/abs/2601.02780. Accessed 2026-10-26. Used for: Abstract: the first report of Multi-Teacher On-Policy Distillation (MOPD); 309B total and 15B active parameters.
  18. paperGLM-5 Team (Zeng, A., Lv, X. et al.; Zhipu AI and Tsinghua University). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763 v2 (24 Feb 2026; v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: Reasoning RL (§3.2), cross-stage distillation (§3.5), asynchronous agentic RL (§4.1), environment scaling (§4.2), deterministic top-k finding.
  19. paperNVIDIA (Blakeman, A. et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007 v1, 12 Jun 2026. https://arxiv.org/abs/2606.15007. Accessed 2026-10-26. Used for: Unified RLVR (§3.2); MOPD (§3.3), Table 5 recovery rates, §3.3.5 open problems; GenRM remark.
  20. paperMa, W., Wei, J., Zhao, L., Zhang, H. et al. (Xiaomi). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406 v1, 29 Jun 2026. https://arxiv.org/abs/2606.30406. Accessed 2026-10-26. Used for: The one controlled comparison of MOPD with Mix-RL, Cascade RL, off-policy fine-tuning and parameter merging (abstract, §1, §3).
  21. paperLiu, M., Diao, S., Lu, X. et al. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864 v1, 30 May 2025. https://arxiv.org/abs/2505.24864. Accessed 2026-10-26. Used for: Prolonged RL with KL control and reference resets from a distilled 1.5B start (abstract, §2.2).
  22. paperMiniMax (Chen, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585 v1, 16 Jun 2025. https://arxiv.org/abs/2506.13585. Accessed 2026-10-26. Used for: CISPO (§3.1); the FP32 LM head after a train-inference probability mismatch (§3.2).
  23. paperQi, P., Liu, Z., Zhou, X. et al. (Sea AI Lab, NUS). Defeating the Training-Inference Mismatch via FP16. arXiv:2510.26788 v1, 30 Oct 2025. https://arxiv.org/abs/2510.26788. Accessed 2026-10-26. Used for: Mismatch attributed to BF16 rounding; the cost of importance-sampling corrections and the deployment gap (§1).
  24. paperDamani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y., Andreas, J. (MIT). Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). ICLR 2026; arXiv:2507.16806 v2, 15 May 2026. https://arxiv.org/abs/2507.16806. Accessed 2026-10-26. Used for: Brier-plus-correctness reward (Eq. 1, Theorem 1); ECE 0.37 to 0.03 on HotpotQA and 0.26 to 0.10 on math (§1).
  25. paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. (OpenAI, Georgia Tech). Why Language Models Hallucinate. arXiv:2509.04664 v1, 4 Sep 2025. https://arxiv.org/abs/2509.04664. Accessed 2026-10-26. Used for: Binary grading makes abstaining sub-optimal (§4); the grading-side repair.
  26. paperOpenAI. GPT-4 Technical Report. arXiv:2303.08774 v6, 4 Mar 2024 (v1 15 Mar 2023). https://arxiv.org/abs/2303.08774. Accessed 2026-10-27. Used for: §2 scope statement; RLHF and rule-based reward models in the system card (§3.1, Appendix H); calibration after post-training (Fig. 8); Table 8.
  27. paperBaker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., Farhi, D. (OpenAI). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926 v1, 14 Mar 2025. https://arxiv.org/abs/2503.11926. Accessed 2026-10-27. Used for: Abstract and §1: monitoring a frontier reasoning model during RL; systemic hacks; monitorability tax; scaling RL surfaces harder hacks.
  28. paperMacDiarmid, M., Wright, B., Uesato, J., Benton, J. et al. (Anthropic, Redwood Research). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv:2511.18397 v1, 23 Nov 2025. https://arxiv.org/abs/2511.18397. Accessed 2026-10-26. Used for: Production coding environments from Claude Sonnet 3.7 training (§1); mitigations and the statement about inoculation prompting in production Claude training.
  29. analysisGolechha, S., Black, S., Bloom, J. (UK AI Security Institute). (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL. LessWrong and AI Alignment Forum, 30 Mar 2026; research note, not peer reviewed. https://www.lesswrong.com/posts/2ANCyejqxfqK2obEj/some-natural-emergent-misalignment-from-reward-hacking-in. Accessed 2026-10-27. Used for: Open-model replication of the Anthropic setup: consistent hacking, inconsistent misalignment. Cached copy read.
  30. paperLlama Team, AI @ Meta. The Llama 3 Herd of Models. arXiv:2407.21783 v3 (23 Nov 2024; v1 31 Jul 2024). https://arxiv.org/abs/2407.21783. Accessed 2026-10-26. Used for: §4.1.4: PPO tried, DPO needed less compute at scale; the SFT, rejection-sampling and DPO design stance (§1).
  31. paperMistral-AI. Magistral. arXiv:2506.10910 v1, 12 Jun 2025. https://arxiv.org/abs/2506.10910. Accessed 2026-10-26. Used for: Open-recipe pointer: KL removed, group-level token normalization, mean-only advantage, Clip-Higher 0.26 to 0.28, KL tried and found to hinder training.
  32. officialHugging Face. TRL documentation: GRPO Trainer (main branch). Unversioned snapshot of docs/trl/main. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-26. Used for: Defaults: loss_type dapo, beta 0.0, token-level importance ratio, sequence-level masked importance sampling.
  33. paperKazemnejad, A., Aghajohari, M., Portelance, E. et al. VinePPO: Refining Credit Assignment in RL Training of LLMs. arXiv:2410.01679. https://arxiv.org/abs/2410.01679. Accessed 2026-10-27. Used for: Figure 4: tuned PPO against GRPO and RLOO on MATH and GSM8K (7B). Cached copy read.
  34. paperByteDance Seed. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv:2504.05118 v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: Table 1: vanilla PPO 5, VAPO 60.4 on AIME 2024; seven modifications. Cached copy read.
  35. paperZheng, C., Liu, S., Li, M. et al. (Qwen Team). Group Sequence Policy Optimization. arXiv:2507.18071 v2, 28 Jul 2025. https://arxiv.org/abs/2507.18071. Accessed 2026-10-26. Used for: Sequence-level ratio and clipping; MoE without routing replay (§5.3).
  36. paperKimi Team (Moonshot AI). Kimi K2: Open Agentic Intelligence. arXiv:2507.20534 v2 (3 Feb 2026; v1 28 Jul 2025). https://arxiv.org/abs/2507.20534. Accessed 2026-10-26. Used for: Self-critic pairwise reward and closed-loop critic refinement (§3.2.2); budget control (§3.2.3).
  37. paperOpenAI. gpt-oss-120b and gpt-oss-20b Model Card. arXiv:2508.10925 v1 (card dated 5 Aug 2025). https://arxiv.org/abs/2508.10925. Accessed 2026-10-27. Used for: §2.5 post-training for reasoning ("similar CoT RL techniques as OpenAI o3"), §2.5.2 reasoning levels, §4.4 no direct optimization pressure on the CoT.
  38. paperLiu, Z., Chen, C., Li, W. et al. (Sea AI Lab, NUS, SMU). Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783 v2, 6 Oct 2025. https://arxiv.org/abs/2503.20783. Accessed 2026-10-26. Used for: Length and difficulty biases in GRPO; the "aha moment" already in DeepSeek-V3-Base.
  39. paperWu, J., Liu, J., Zeng, Z., Zhan, T., Cai, T., Huang, W. Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. arXiv:2512.19920 v3, 28 Jan 2026. https://arxiv.org/abs/2512.19920. Accessed 2026-10-26. Used for: Proper-scoring-rule interventions; results are the authors' own evaluation (abstract).
  40. paperXu, S., Fu, W., Gao, J., Ye, W. et al. (Tsinghua, OpenPsi). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv:2404.10719 v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: Abstract: PPO ahead of the other methods on every testbed (abstract only was read).
  41. paperIvison, H., Wang, Y., Liu, J., Wu, Z. et al. (Ai2, UW). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv:2406.09279 v2, 7 Oct 2024. https://arxiv.org/abs/2406.09279. Accessed 2026-10-26. Used for: Abstract: PPO ahead of DPO by up to 2.5% in math and 1.2% in general; data quality ranked first (abstract only was read).
  42. paperYue, Y., Chen, Z., Lu, R. et al. (Tsinghua LeapLab, SJTU). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837 v5, 24 Nov 2025. https://arxiv.org/abs/2504.13837. Accessed 2026-10-26. Used for: Pass@k at large k; coverage of solvable problems falls as pass@1 rises (Fig. 1).
  43. paperYao, X., Yu, L., Hu, X., Teng, F. et al. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028 v1, 5 Oct 2025. https://arxiv.org/abs/2510.04028. Accessed 2026-10-26. Used for: The reconciliation of the sharpening and expansion camps (abstract); the authors' own assessment.
  44. paperShao, R. et al. (UW, Ai2, Berkeley). Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947 v2, 25 Feb 2026. https://arxiv.org/abs/2506.10947. Accessed 2026-10-26. Used for: Random rewards: +21.4 against +29.1 MATH-500 points on Qwen2.5-Math-7B.
  45. officialTypeSafe AI. AI primer (docs.typesafe.ai). Live documentation, no version stamp. https://docs.typesafe.ai/introduction/machine-learning-primer.md. Accessed 2026-10-26. Used for: Company definition of RLCD and its output contract; company claim only.
  46. talkLatent Space. Jev: System One models for Prod, not God, with Diogo Almeida (CEO, TypeSafe AI). Podcast transcript, undated in the copy read. https://latent.space/p/jev/. Accessed 2026-10-26. Used for: Founder interview: the method is not yet published; training data synthetic.
  47. paperYang, K., Klein, D., Celikyilmaz, A., Peng, N., Tian, Y. RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. ICLR 2024; arXiv:2307.12950 v3. https://arxiv.org/abs/2307.12950. Accessed 2026-10-26. Used for: The unrelated 2023 method that shares the acronym.
  48. analysisTechCrunch. A new kind of AI model from a ChatGPT inventor is thrilling developers. 18 Sep 2026. https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/. Accessed 2026-10-26. Used for: The only press coverage; writes "from calibrated decisions".
  49. paperGao, Z., Wang, P. OpenJev-RLCD: A Working RLCD Implementation. arXiv:2609.38850 v1, 30 Sep 2026. https://arxiv.org/abs/2609.38850. Accessed 2026-10-26. Used for: A third-party reconstruction that states TypeSafe discloses no algorithm (abstract, §1).
  50. paperBai, Y., Kadavath, S., Kundu, S., Askell, A. et al. (Anthropic). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073 v1, 15 Dec 2022. https://arxiv.org/abs/2212.08073. Accessed 2026-10-26. Used for: The two-phase RLAIF recipe, published in 2022.
  51. officialAnthropic. Claude's new constitution. Anthropic news, 22 Jan 2026. https://www.anthropic.com/news/claude-new-constitution. Accessed 2026-10-27. Used for: What Anthropic says about using the constitution in training; no algorithm given.
  52. paperOuyang, L., Wu, J., Jiang, X., Almeida, D. et al. (OpenAI). Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv:2203.02155 v1, 4 Mar 2022. https://arxiv.org/abs/2203.02155. Accessed 2026-10-26. Used for: A complete published RLHF recipe: dataset sizes (§3.2), RM and PPO (§3.5).
  53. officialAnthropic. Claude's Constitution. Anthropic news, 9 May 2023 (update note 21 Jan 2026). https://www.anthropic.com/news/claudes-constitution. Accessed 2026-10-27. Used for: The 2023 description of the supervised and RL phases.
  54. paperGao, L., Schulman, J., Hilton, J. (OpenAI). Scaling Laws for Reward Model Overoptimization. ICML 2023; arXiv:2210.10760 v1, 19 Oct 2022. https://arxiv.org/abs/2210.10760. Accessed 2026-10-26. Used for: Gold-versus-proxy measurement of reward-model overoptimization.
  55. paperDeepSeek-AI (Guo, D. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v1, 22 Jan 2025. https://arxiv.org/pdf/2501.12948v1. Accessed 2026-10-26. Used for: The v1 headline numbers (AIME 2024 pass@1 71.0%, 86.7% with majority voting) for the version-hygiene table.
  56. repoNVIDIA-NeMo. Nemotron repository README (training recipes). Main branch, snapshot of 2026-10-27. https://github.com/NVIDIA-NeMo/Nemotron. Accessed 2026-10-27. Used for: Nemotron 3 Ultra recipe entry: full program versus the stages the recipe covers; open-data-only note; the 3.5 Super entry.
  57. repozai-org. GLM-5 repository README (GLM-5.3, 5.2, 5.1, 5). Main branch, snapshot of 2026-10-27. https://github.com/zai-org/GLM-5. Accessed 2026-10-27. Used for: The model list and the vendor statements about successors.