Reinforcement Learning for LLM Post-Training · Module 30

Case Study: DeepSeek-V3.2 to V4: Scaling GRPO Past 10% of Pre-Training Compute, Then Replacing Mixed RL with On-Policy Distillation

DeepSeek's three reasoning-era reports share one optimizer and differ in nearly everything around it. R1 stacked SFT and RL stages; V3.2 merged reasoning, agent and alignment RL into one GRPO stage and spent more than 10% of pre-training cost on it; V4 dropped that stage and distilled more than ten specialists into one student. This module reads the three recipes against each other, including what each report states, leaves out and never tests. Afterwards you can say which R1 choices DeepSeek revised and why, what V4 gave up by dropping mixed RL, and which V3.2 stability fixes an RL loop of your own would still need.

Course 40 min read 20 sources As of 2026-10-27 Prerequisites: Module 9, Module 11, Module 22
Three DeepSeek post-training recipes as lanes of stages; orange marks the step where separately acquired skills are merged DeepSeek-R1: four stages in sequence Cold-start SFT thousands of examples Reasoning RL GRPO, rule rewards Rejection-sampling SFT ~600K reasoning + ~200K other samples Second RL stage 1,700 steps DeepSeek-V3.2: specialists, then one mixed RL stage V3.1-Terminus + DSA continued pre-training Specialists 6 domains + writing + QA large-scale RL for each Train on distilled data specialists' outputs One mixed GRPO stage reason + agent + align thousands of steps DeepSeek-V4: specialists, then on-policy distillation Pre-training 32T (Flash), 33T (Pro) Specialists SFT, then GRPO per domain three effort settings Multi-teacher OPD >10 teachers, reverse KL full-vocabulary logits V4-Pro / V4-Flash 1.6T/49B and 284B/13B Non-think, High, Max hard-to-verify: actor is the GRM SFT or pre-training RL (GRPO) where skills are combined released model
Figure 1. The three recipes as lanes. Orange marks the step where separately acquired skills are merged: an SFT mixture in R1 (about 600K reasoning and 200K other samples1), one mixed GRPO stage in V3.22, multi-teacher on-policy distillation in V43. Stage contents from R1 §3 and Supp. B.3.3, V3.2 §3, and V4 abstract, §1 and §5.1; V4 sizes are total/activated parameters.

1. What R1 settled, and what it left open

R1-Zero ran GRPO4 directly on DeepSeek-V3-Base with a reward that checks only the final answer, and AIME 2024 pass@1 rose from 15.6% to 77.9% (arXiv v2)1. Module 27 reads that recipe stage by stage; this module needs five of its facts, because each is something a later report changes.

  • Optimizer settings. Advantage \(A_i=(r_i-\mathrm{mean})/\mathrm{std}\) over a group of 16, KL coefficient 0.001, 32 questions per step, the reference model replaced by the latest policy every 400 steps, and each 8,192-output rollout split into 16 minibatches with one inner epoch1 (§2.1, Eq. 3).
  • Reasoning rewards. Rule-based accuracy and format only. The report states that it abstains "from applying neural reward models—whether outcome-based or process-based—to reasoning tasks" because they are "susceptible to reward hacking during large-scale reinforcement learning"1 (§2.2).
  • Stages. Cold-start SFT, reasoning RL with a language-consistency reward, rejection sampling plus SFT, then a second RL stage. Helpfulness and harmlessness reward models enter only in the last 400 of that stage's 1,700 steps, because longer exposure to the preference reward "may lead to reward hacking"1 (§3.2.2, Supp. B.5).
  • Context. Maximum output length 32,768 tokens, raised to 65,536 after step 8.2K1 (§2.1).
  • Cost. 101K H800 GPU-hours for R1-Zero, 5K for SFT data creation and 41K for R1, 147K in total, which the report prices at $294K at $2 per GPU-hour1 (Table 7).

V3.2's introduction names three deficiencies of open models: vanilla attention, "insufficient computational investment during the post-training phase", and weak generalization in agent settings2 (§1). The last two are the RL half of the report, and they turn R1's loose ends into four questions that organize the rest of this module. How do you push RL past 10% of pre-training cost, against 3.8% to 5.5% for R1 (derived in Section 5), without the run diverging (Sections 4 and 5)? How do you get reasoning and tool use into one trajectory (Section 3)? Where does the reward for open-ended prompts come from now that a scalar reward model is the thing R1 saw being hacked (Sections 3 and 6)? And how should skills trained separately end up in one set of weights (Sections 2, 7 and 9)? V3.2 answers all four; V4 keeps most of the answers to the first three and reverses the fourth.

2. DeepSeek-V3.2: specialists first, then one mixed RL stage

2.1 The checkpoint RL starts from, and the attention it runs through

V3.2 starts from DeepSeek-V3.1-Terminus, whose context had been extended to 128K, and adds DeepSeek Sparse Attention (DSA) by continued pre-training in two stages: a dense warm-up that trains only the lightning indexer (1,000 steps of 16 sequences of 128K tokens at learning rate \(10^{-3}\)), then a sparse stage that trains everything with 2,048 key-value entries selected per query (15,000 steps of 480 sequences at \(7.3\times10^{-6}\))2 (§2.1.1). The report's token totals, 2.1B and 943.7B, follow from reading 128K as 131,072 tokens: \(1{,}000\times16\times131{,}072=2.10\mathrm{B}\) and \(15{,}000\times480\times131{,}072=943.7\mathrm{B}\) (derived). Post-training "employs sparse attention in the same way as the sparse continued pre-training stage"2 (§3), so rollouts are sampled through the same top-\(k\) selection (\(k=2{,}048\) in the continued pre-training stage; the post-training value is not restated) and the trainer must reproduce that selection when it recomputes probabilities. The report does not discuss how the two engines keep the indexer's choice consistent. GLM-5, which also trains on DSA, reports that a non-deterministic top-k kernel in the indexer degraded RL drastically within a few steps and that a deterministic torch.topk fixed it5 (§3.2). That is an observation about GLM-5's stack, not a finding about DeepSeek's, but it names the failure that Section 4's fixes do not cover.

2.2 Specialists, distilled data, then a single stage

For each domain the team first builds a specialist, all fine-tuned from the same pre-trained V3.2 base checkpoint. The domains are writing and general question answering plus six named areas: mathematics, programming, general logical reasoning, general agentic tasks, agentic coding and agentic search, each in thinking and non-thinking mode, with different models generating the long chain-of-thought data and the direct-response data. Each specialist is "trained with large-scale Reinforcement Learning (RL) computing", and the specialists then produce the domain data for the final checkpoint2 (§3). The report's evidence for this step is one sentence: models trained on the distilled data reach performance "only marginally below those of domain-specific specialists, with the performance gap being effectively eliminated through subsequent RL training". No table shows the gap.

The final stage is GRPO4, with "reasoning, agent, and human alignment training" merged "into one RL stage", which the report says "effectively balances performance across diverse domains while circumventing the catastrophic forgetting issues commonly associated with multi-stage training paradigms". The merged data then go through "thousands of steps of continued RL training to reach the final checkpoints"2 (§3). The mixed stage is therefore RL on top of a model already trained on distilled specialist data, not RL from a cold start. That matters for Section 9, because V4 removes this RL pass.

2.3 What the recipe does not tell you

The report prints an objective, an advantage and four stability operations, and few of the numbers that would let you rerun them. Table 1 lists what a reproducer needs against what the sections give.

Table 1. DeepSeek-V3.2's RL recipe: stated against not stated. "Not stated" means the report was searched in full (main text and appendices) and the quantity does not appear2.
ItemWhat the report givesWhere
Algorithm and advantageGRPO; \(\hat A_{i,t}=R_i-\mathrm{mean}(\boldsymbol R)\), no division by the standard deviation§3, §3.1
Loss aggregationPrinted per token with \(1/|o_i|\) (Eqs. 5, 6, 8); whether the code matches is not stated (Module 11)§3.1
Group size \(G\), prompts per step, batch sizeNot stated; \(G\) appears only as a symbol§3.1
Learning rate, \(\varepsilon\), \(\beta\)Not stated for RL. The only learning rates printed are the DSA ones: \(10^{-3}\) and \(7.3\times10^{-6}\)§2.1.1, §3.1
Mask threshold \(\delta\)Not stated (Module 12)§3.1, Eq. 9
Run length"Thousands of steps"§3
Length penalty, language consistencyNamed; form and weights not stated§3
Judge for general tasksA generative reward model with per-prompt rubrics; which model it is and who writes the rubrics is not stated§3
RL compute"Exceeding 10% of the pre-training cost"; the denominator is not defined§1, §4.1

2.4 Speciale

V3.2-Speciale is the same RL run with the dials turned toward length: "trained exclusively on reasoning data with a reduced length penalty", plus the DeepSeekMath-V2 proof data and reward2 (§3). It scores 35/42 on IMO 2025, 102/126 on CMO 2025 and 492/600 on IOI 2025, and solved 10 of 12 ICPC World Finals 2025 problems, at a 128K generation cap and without tools (Table 4, Appendix D); the report concedes its token efficiency "remains significantly inferior" to Gemini-3.0-Pro's2 (§4.2). Module 23 reads the length-penalty arithmetic.

3. V3.2's task pool and what scores it

3.1 Four populations, three of them in real environments

The introduction's "over 1,800 distinct environments and 85,000 complex prompts"2 (§1) is two numbers about two different things. Table 1 of the report counts agent tasks in four populations, and the rows sum to 85,267 (derived): 24,667 code-agent tasks, 50,275 search-agent tasks, 4,417 general-agent tasks and 5,908 code-interpreter tasks2 (§3.2.3). The 1,827 environments belong to the general-agent row alone; the other three run in real environments (live search APIs, coding tools, Jupyter notebooks) with prompts that are extracted from the internet or synthesized, "rather than obtained from actual user interactions". Figure 2 draws the pool. Module 17 covers how each population is built and filtered.

Two stacked bars of the 85,267 V3.2 agent tasks, split by environment type and by prompt source Environment: real or synthesized General agent: 4,417 (5.2%) Code interpreter: 5,908 (6.9%) Search 50,275 · 59.0% Code 24,667 · 28.9% Prompt source: extracted or synthesized synthesized extracted 1,827 environments, 4,417 tasks 2.42 tasks per environment (derived) real environment, or prompt extracted from the internet synthesized environment or prompt
Figure 2. The 85,267 agent tasks of V3.2's Table 1 (sum derived), drawn to scale. Top bar: environment type. Bottom bar: prompt source. Only the general-agent segment has a synthesized environment; the largest segment pairs a real search environment with synthesized prompts. Counts and labels from V3.2 Table 12.
Table 2. V3.2's agent task populations: size, environment, prompt source and the scoring the report describes2 (Table 1, §3, §3.2.3).
PopulationTasksShare (%)EnvironmentPromptsScoring the report states
Search agent50,27559.0realsynthesizedVerifiable QA pairs kept only when the ground truth checks out and every candidate answer is verifiably wrong; plus helpfulness prompts scored by a rubric GRM
Code agent24,66728.9realextractedRule-based outcome reward; an environment is admitted only if the gold patch yields at least one F2P test and zero P2F tests
Code interpreter5,9086.9real (Jupyter)extractedRule-based outcome reward on math, logic and data-science problems that need code execution
General agent4,4175.2synthesizedsynthesizedA per-task Python verification function; only tasks with non-zero pass@100 after RL with V3.2 are kept

3.2 A rubric judge where there is no checker

For reasoning and agent tasks the reward is a rule-based outcome reward, a length penalty and a language-consistency reward. "For general tasks, we employ a generative reward model where each prompt has its own rubrics for evaluation"2 (§3). The search-agent pipeline shows the hybrid in use: verifiable question-answer pairs reward factual reliability, and "detailed evaluation rubrics across multiple quality dimensions" scored by a generative reward model reward practical helpfulness (§3.2.3). Section 3.1 of the same report says "a set of reward models" scores each output, without saying which2.

Set that against R1, whose general-purpose reward was a scalar model trained on 66,000 preference pairs that DeepSeek-V3 labeled, and whose authors saw reward rise while Codeforces performance fell1 (§3.1, Supp. B.5). V3.2 never says that hacking was the reason for the switch, and it gives no comparison between a scalar model and the rubric judge. The move is documented; the motive and the benefit are not. Module 16 covers rubric judges and their failure modes.

3.3 The one measured claim about synthetic tasks

Section 4.3 is the only environment-design ablation among the seven reports Module 17 searched. It asks whether the synthesized general-agent tasks are hard and whether RL on them transfers. On 50 sampled tasks, V3.2-Exp reaches 12% at pass@1 and 26% at pass@4, against 34% and 62% for Claude Sonnet 4.5, 51% and 74% for Gemini 3.0 Pro, and 62% and 82% for GPT-5 Thinking2 (Table 5). For transfer, RL on only the synthetic tasks in non-thinking mode, starting from the SFT checkpoint, "yields substantial improvements" on τ²-Bench, MCP-Mark and MCP-Universe, whereas RL restricted to code and search "does not improve performance on these benchmarks" (Figure 5, plotted as curves without tabulated values). It is one run per condition on a 50-task sample, so treat it as evidence that the data are not trivially easy and that they add something code-and-search RL does not; Module 17 lists the limits.

3.4 Keeping the reasoning when a tool answers

V3.2 describes R1's strategy as discarding reasoning content when the second round of messages arrives. In tool use that "forces the model to redundantly re-reason through the entire problem for each subsequent tool call", so V3.2 drops historical reasoning only when a new user message arrives, keeps it while only tool outputs are appended, and always keeps the tool-call history2 (§3.2.1). The cold start is deliberately light: system prompts that ask for reasoning, for tool calls, or for tool calls inside the <think> block, with the admission that "the reasoning in tool-use patterns may lack robustness" and the model only "occasionally" produces the wanted trajectories, which is enough to start RL (§3.2.2, Appendix B). V4 goes further and keeps the whole reasoning history across user turns in tool-calling scenarios3 (§5.1.1). Module 18 covers what the harness does to the sequence the policy conditions on.

4. V3.2's stability toolbox, drawn as the boundary between engine and trainer

After the mean-subtracted advantage, V3.2 lists four "additional strategies that stabilize RL scaling, directly building on the GRPO algorithm"2 (§3.1). Read as systems work, each one is either a piece of metadata that has to cross from the rollout engine to the trainer, or a rule for what the trainer does when that metadata says the two have drifted apart. Figure 3 steps through one iteration to show which is which. The mechanisms themselves are derived elsewhere (the KL fix in Module 6 §3.6, the sequence mask in Module 12 §5.3, routing and sampling masks in Module 14); the point here is the pattern.

Rollout engine Trainer what crosses Sampler: top-p, top-k truncates the vocabulary MoE router: top-k experts picks experts per token π_old per token returned by the engine truncation mask Forward under same mask same action subspace expert indices Forward on same routes same experts optimized Rewards R_i rules, length, GRM A = R − mean(R) no std division log π_old M = 0 if A<0, drift>δ whole sequence dropped π_ref frozen reference KL: K3 × π_θ/π_old β set per domain
  1. Sample. The engine draws a group of outputs per prompt with top-p and top-k truncation and MoE routing, and returns each token's probability directly.
  2. Keep Sampling Mask. The truncation mask crosses to the trainer, which restricts \(\pi_\theta\) to the same kept tokens, so both policies "share identical action subspaces".
  3. Keep Routing. The expert paths used at sampling are forced in the trainer's forward pass, so the same expert parameters are optimized.
  4. Reward, then advantage. Rule-based rewards, length penalty, language consistency and the rubric judge give \(R_i\); the advantage subtracts the group mean and does not divide.
  5. Off-policy sequence mask. A sequence with negative advantage whose mean log-ratio of the engine's \(\pi_{old}\) to the current \(\pi_\theta\) exceeds \(\delta\) is masked out whole. \(\delta\) is not stated.
  6. KL to the reference. The K3 estimate is weighted by \(\pi_\theta/\pi_{old}\); the coefficient depends on the domain and is weak or absent for mathematics.

Figure 3. One V3.2 iteration, stepped. Cyan is the rollout engine, blue the trainer, orange the mask that changes which sequences count. What crosses the boundary is named in the report's own terms: probabilities "directly returned by the inference framework", truncation masks, expert routing paths2 (§3.1). The report does not say how masks and paths are stored or transported.

Table 3. The five V3.2 operations, the failure each is said to address, and the evidence the report gives2 (§3.1).
OperationFailure named in the reportEvidence givenDerived in
Advantage \(R_i-\mathrm{mean}(\boldsymbol R)\)None stated for dropping the divisionNoneModule 11
Importance-corrected K3, per-domain \(\beta\)When \(\pi_\theta\ll\pi_{ref}\) the K3 gradient gives "disproportionately large, unbounded weights", producing "noisy gradient updates"None shown; "weak KL penalty or even omitting it entirely can yield improved performance" for mathematicsModule 6 §3.6
Off-policy sequence maskingMinibatch reuse plus engine-versus-trainer differences make rollouts off-policy; "highly off-policy negative samples can be detrimental"Reported to improve stability "in certain training scenarios"; no numbers, \(\delta\) not givenModule 12 §5.3
Keep RoutingInconsistent expert routing between engines causes "abrupt shifts in the active parameter subspace"Operation "found crucial for RL training stability of MoE models"; used since DeepSeek-V3-0324Module 14
Keep Sampling MaskTruncating the action space at sampling but not in training "violates the principles of importance sampling"Top-p with the mask "effectively preserves language consistency"Modules 13, 14

Three observations that the table cannot hold. First, no row has an ablation: every fix is justified by a mechanism and a sentence about observed stability, which is why Module 12 and Module 14 supply the mathematics and the outside evidence. Second, the sequence mask is asymmetric by design. The report masks only negative-advantage sequences because "the model benefits the most by learning from its own mistakes, whereas highly off-policy negative samples can be detrimental"2 (§3.1). That is an argument from intuition, and because \(\pi_{old}\) is the engine's own probability the mask reacts to minibatch staleness and to engine mismatch without telling them apart. Third, the KL term is kept, repaired and tuned per domain. R1 used one coefficient, 0.0011 (§2.1); DAPO, Olmo 3 and GLM-5 delete the term675; V3.2 chose the option with the most knobs and prints none of the values.

5. How much compute: a ladder across four DeepSeek reports

V3.2 states its RL budget as a share, not as GPU-hours: a "post-training computational budget exceeding 10% of the pre-training cost", and later "consistent performance improvements correlating with extended RL training budget, which already exceeds 10% of the pre-training cost", with the hypothesis that "reasoning capabilities could be further enhanced with additional computational budget allocation"2 (§1, §4.1). The report does not define the denominator. DeepSeek's earlier reports give enough to put four points on one scale, if you accept one assumption: the denominator is V3's pre-training, 2,664K H800 GPU-hours8 (Table 1). That is exact for V3 and R1, which both start from V3-Base, and a stand-in for V3.2, whose base is V3.1-Terminus plus continued pre-training. Module 15 uses the same stand-in.

$$\frac{5}{2{,}664}=0.19\%,\qquad \frac{101}{2{,}664}=3.8\%,\qquad \frac{147}{2{,}664}=5.5\%,\qquad 10\%\times2{,}664\mathrm{K}=266.4\mathrm{K}\ \text{GPU-hours}$$

The numerators are V3's whole post-training (SFT and RL together, 5K8), R1-Zero's RL run (101K), and R1's total including SFT data creation (147K)1 (Table 7). Under the assumption, V3.2's floor of 266.4K is 1.8 times everything R1 spent and 53 times what V3 spent on its entire post-training (derived). Figure 4 draws the ladder.

Post-training compute as a percentage of DeepSeek-V3 pre-training cost for V3, R1-Zero, R1 and V3.2 0 2 4 6 8 10 12 V3 post-training (5K) 0.19% R1-Zero RL (101K) 3.8% R1 total (147K) 5.5% V3.2 RL (more than 10%) ≥ 266.4K GPU-hours if the denominator is V3's share of V3 pre-training cost, 2,664K H800 GPU-hours (%)
Figure 4. Post-training compute as a share of one denominator, V3's 2,664K H800 GPU-hours of pre-training8. Blue bars: V3's own SFT and RL (5K), R1-Zero (101K) and R1 including SFT data creation (147K)1; shares are derived. The V3.2 bar sits at its stated lower bound2 and is dashed because the report does not define what it is a share of. Hardware and hour definitions are not matched across reports.

Three limits on reading the ladder. The denominators differ in kind: V3.2's base is V3.1-Terminus plus 945.8B tokens of continued pre-training (2.1B plus 943.7B, derived from §2.1.1), so 10% of its own pre-training cost is a different absolute budget from 10% of V3's. The numerators mix SFT and RL, and V3.2's does not say what it includes. And V4 reports no post-training compute at all in the sections read3, so the ladder stops at V3.2. What the budget bought is also stated rather than plotted: V3.2 reports similar performance to GPT-5-High on reasoning benchmarks, slightly below Gemini-3.0-Pro, and says its headline numbers are "constrained by a length constraint reward model", relaxed only in Speciale2 (§4.1). No source in this course fits a compute curve to a production run (Module 15).

6. DeepSeek-V4: specialists, an actor that judges, three effort modes

V4 comes in two sizes, V4-Pro with 1.6T parameters (49B activated) and V4-Flash with 284B (13B activated), pre-trained on 33T and 32T tokens, both with a one-million-token context3 (abstract, §1). Its post-training "largely mirrored that of DeepSeek-V3.2" except for one substitution, which Section 7 reads. This section covers what carried over and what the specialists became.

6.1 Specialists: the same GRPO, fewer words about it

Each domain expert is "sequentially optimized through an initial fine-tuning phase and subsequent Reinforcement Learning (RL) guided by domain-specific prompts and reward signals", using GRPO with "hyper-parameters closely aligned with our prior research", cited as "(DeepSeek-AI, 2025; DeepSeek-AI, 2025)"3 (§5.1.1). Its reference list has three DeepSeek-AI entries from 2025: R1, V3.2 and the 3FS file system, so the two citations are presumably R1 and V3.2 (my inference). The domains named in the introduction are mathematics, coding, agent and instruction following (§1). The section prints no group size, learning rate, KL coefficient, advantage form or stability operations, so V3.2's Table 1 gaps carry over and widen. The RL-relevant consistency statements I found sit in the infrastructure sections: bitwise batch-invariant, deterministic kernels so that "bitwise alignment among pre-training, post-training, and inference" holds (§3.3, covered in Module 14), and rollouts that use native FP4 expert weights instead of simulated quantization, "ensur[ing] that model behavior during sampling is fully consistent with online deployment" (§5.2.1). That is a different answer to the mismatch problem from V3.2's: remove the numerical gap instead of masking its consequences.

6.2 The actor is the judge

V4 states the reward design for hard-to-verify tasks in one paragraph. It "dispense[s] with these conventional scalar-based reward models", curates "rubric-guided RL data", and employs a generative reward model (GRM) to evaluate trajectories: "Crucially, we apply RL optimization directly to the GRM itself. In this paradigm, the actor network natively functions as the GRM, enabling the joint optimization of the model's evaluative (judging) proficiency alongside its standard generative capabilities"3 (§5.1.1). The stated benefits are that reasoning is "inherently fused into its evaluative process, resulting in highly robust scoring", and that the approach "achieves superior performance with only a minimal set of diverse human annotations".

All of that is the vendor's description. The V4 text contains no ablation of any kind (a search for "ablat" finds nothing), no judge accuracy and no comparison against a separate or frozen judge, so the design is a claim, not a measured result. Two framing facts help. Kimi K2 earlier used the policy as its own critic, but anchored it by refreshing the critic on verifiable-reward rollouts10 (§3.2.2); V4 describes no such anchor. And neither V3.2 nor V4 cites DeepSeek's earlier generative-reward-model paper, so this module asserts no lineage between them (Module 16 draws the same line). Two risks follow directly from the design and are not discussed in the report: the judge moves with the policy it scores, and a policy that learns to produce text its own judge likes has no external check in the loop. Module 16 §7 and Module 19 cover those failure modes.

6.3 Three effort modes

Both V4 models "support three specific reasoning effort modes. For each mode, we apply distinct length penalties and context windows during RL training"3 (§5.1.1). Table 4 collects what the report prints. The penalty values are not stated, and the report does not say whether each mode has its own teachers or how three modes enter one student; Module 23 reads the token counts and the length arithmetic.

Table 4. DeepSeek-V4's reasoning modes as printed3 (Tables 2, 3, 7; §5.3.1). Evaluation window is the context limit used when scoring, not the RL window, which is not stated. Pro scores: HLE Pass@1 and Terminal Bench 2.0 accuracy (%).
ModeResponse formatSystem-prompt changeEval window (tokens)Pro HLEPro TB 2.0
Non-think</think> then summarynone8K7.759.1
Think High<think> reasoning </think> then summarynone128K34.563.3
Think Maxas Highinjected "Reasoning Effort: Absolute maximum with no shortcuts permitted"384K37.767.9

The report's own reading is that "the Max mode, which employs longer contexts and reduced length penalties in RL, outperforms the High mode on the most challenging tasks"3 (§5.3.2). The table shows the shape: on Pro, High recovers most of the distance from Non-think (HLE 7.7 to 34.5), and Max adds 3.2 points on HLE and 4.6 on Terminal Bench for a window three times as large (derived from Table 7). Max is a prompt plus a separate training configuration, so adding its instruction to a High-trained model is not the same thing; the report does not test that.

7. DeepSeek-V4: on-policy distillation replaces the mixed stage

V4's report is blunt about the change: "the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation (OPD; Gu et al., 2024; Lu and Lab, 2025)"3 (§5.1). The stated flow is two stages. First "the independent cultivation of domain-specific experts", each by SFT on domain data and then GRPO against domain rewards; then "a single unified model is trained through on-policy distillation, wherein the unified model acts as the student learning to optimize the reverse KL loss with teacher models" (§1). Modules 21 and 22 derive the method; this section records what V4 specifies, what it gives as reasons, and what the report leaves unsaid.

7.1 The objective as printed

For \(N\) expert models \(\{\pi_{E_1},\dots,\pi_{E_N}\}\) the objective is Eq. 29:

$$\mathcal L_{\mathrm{OPD}}(\theta)=\sum_{i=1}^{N} w_i\cdot D_{\mathrm{KL}}\!\left(\pi_\theta\,\Vert\,\pi_{E_i}\right)$$

with \(w_i\) "the assigned weight for each expert, typically determined by the relative importance of the expert"3 (§5.1.2). Computing reverse KL "requires sampling training trajectories from the student \(\pi_\theta\) to maintain on-policy learning", and the policy "selectively learns from the specialized expert relevant to the current task context (e.g., aligning with the mathematics expert for math reasoning tasks and the coding expert for programming tasks)". Read literally, the sum is over all experts; read with the sentence after it, a math prompt pulls toward the math expert. The report does not say whether \(w_i\) is fixed per domain, set per prompt, or non-zero for several experts at once, and it gives no schedule, batch size or run length for this stage. The sampled-token form is not what V4 uses, as Section 7.3 explains.

Two points of vocabulary. V4 writes "multi-teacher On-Policy Distillation (OPD)" and cites MiniLLM (Gu et al., 2024) and the Thinking Machines post (Lu and Lab, 2025)11; the string "MOPD" does not occur in V4 or in V3.2, and does not occur in GLM-5, which calls its final stage "on-policy cross-stage distillation"5 (§1). Lambert's survey groups V4, GLM-5, Kimi K3, Nemotron 3 Ultra and MiMo-V2-Flash under MOPD and says labs "are far from unanimous in its usage"12 (§3.2.4), and a separate paper from Xiaomi's LLM Core team is titled MOPD13. The label names a family. V4 is one member, with its own choices, and the controlled comparison in the MOPD paper is of a different instance.

7.2 The reasons V4 gives

The case for replacing mixed RL is one clause: aligning at the logit level consolidates "the knowledge from physically distinct expert weights ... into a unified parameter space ... practically circumventing the performance degradation often encountered in traditional weight-merging or mixed RL techniques"3 (§5.1.2). The report presents no mixed-RL run, no merge run and no measurement of degradation. This is the opposite diagnosis from V3.2's, which chose one mixed stage to avoid "catastrophic forgetting issues commonly associated with multi-stage training paradigms"2 (§3). The two claims are about different failure modes: V3.2 worried that sequential stages forget earlier skills, V4 that jointly optimizing many reward signals in one RL run degrades each. Both are plausible and neither report tests its own claim against the other. An outside controlled comparison exists on a small model: on Qwen3-30B-A3B with three domains, the MOPD paper reports a normalized score of 0.937 against 0.882 for mixed RL, 0.775 for cascade RL and 0.824 for off-policy fine-tuning, with MOPD's three per-domain scores in a range of 0.044 (the authors' numbers, on one instance of the method whose form Module 22 details)13 (§4). That supports the direction of V4's choice at small scale; it does not measure V4's trade at 1.6T parameters with more than ten domains. No matched-compute comparison of MOPD against mixed RL appears in V4 or in the Kimi K3 sections Module 22 read.

7.3 Why full-vocabulary, and what it costs

The cheap form of OPD, which V4 describes as what "prior works usually" do, replaces the full-vocabulary KL with "a token-level KL estimate at each token position" and reuses the RL stack by treating \(\mathrm{sg}[\log\pi_{E_i}(y_t\mid x,y_{\lt t})/\pi_\theta(y_t\mid x,y_{\lt t})]\) as the per-token advantage (sg is stop-gradient). The report says this "is resource-efficient" but "leads to high variance in gradient estimation and often causes training instability", and so V4 "adopt[s] full-vocabulary logit distillation"3 (§5.1.2). The claim that the sampled-token estimator is unstable has no ablation or training curve in the passages read (Module 21). Module 22 weighs the alternative evidence, including reports that found no gain from richer signals.

What V4 does show is the engineering that makes the exact form affordable (§5.2.2, §5.2.3), most of it problems an inference engineer will recognize. Teacher weights sit in central distributed storage, are loaded on demand with ZeRO-like sharding, and the framework "supports full-vocabulary On-Policy Distillation (OPD) with an effectively unbounded number of teachers, each potentially comprising trillions of parameters". Only the last-layer hidden states of each teacher are cached; logits are rebuilt through the prediction head at training time with "negligible recomputation overhead" (stated, not measured in the sections read; Module 22 derives that the head is about 1.9% of a Pro-sized forward pass). Samples are ordered by teacher index so each teacher head is loaded once per minibatch and at most one head is resident. The exact KL runs in a TileLang kernel. A further piece concerns rollouts: a token-granular write-ahead log plus saved KV cache lets generation resume after preemption, because "it is mathematically incorrect to regenerate unfinished requests from scratch, as this introduces length bias" (Module 18 §4.2 explains the selection effect).

7.4 Claim ledger

Table 5. V4's post-training claims sorted by the evidence the report itself provides in the sections read3.
ClaimWhereEvidence in the reportStatus
Mixed RL entirely replaced by OPD§5.1Statement of the pipelineDecision, no control
Logit-level alignment avoids degradation from merging or mixed RL§5.1.2NoneVendor claim
Sampled-token OPD is high-variance and unstable§5.1.2NoneVendor claim
More than ten teachers§5.1.2Count onlyStated
Hidden-state caching costs negligible recomputation§5.2.2None quantifiedStated; derivable (Module 22)
The actor as GRM gives robust scoring with minimal annotation§5.1.1NoneVendor claim
Max mode beats High on the hardest tasks§5.3.2, Table 7Benchmark scores, one run eachMeasured; confounded by window and penalty
Regenerating preempted requests biases length§5.2.3Argument (shorter responses survive interruption more often)Reasoned; correct as a selection effect

8. R1, V3.2 and V4 side by side

Table 6 lines up the choices that changed. Cells say "not stated" where the sections read give no value; for R1 they refer to the passages Module 27 reads.

Table 6. What changed from R1 to V3.2 to V4, by component123. Section references are to each report.
ComponentR1V3.2V4
Starting pointV3-Base (§2.1)V3.1-Terminus plus DSA continued pre-training (§2.1)New pre-training, 33T (Pro) and 32T (Flash) tokens (§1)
OptimizerGRPOGRPO (§3)GRPO "closely aligned" with prior work, per specialist (§5.1.1)
Advantage\((r_i-\mathrm{mean})/\mathrm{std}\) (Eq. 3)\(R_i-\mathrm{mean}(\boldsymbol R)\) (§3.1)Not stated
KL to reference\(\beta=0.001\); reference reset every 400 steps (§2.1)Importance-corrected K3; strength per domain, weak or none for math; values not stated (§3.1)Not stated
Off-policy and mismatch control16 minibatches per rollout; clip \(\varepsilon\) printed as 10 (§2.1, §3.2.1)Negative-sequence mask, Keep Routing, Keep Sampling Mask (§3.1)Batch-invariant deterministic kernels; native FP4 rollouts (§3.3, §5.2.1)
Reasoning rewardRule-based accuracy and format; language consistency (§2.2, §3.2.1)Rule-based outcome, length penalty, language consistency (§3)Domain-specific rewards; length penalty per effort mode (§5.1.1)
Open-ended rewardScalar helpfulness model (66K pairs) and safety model (106K prompts), last 400 of 1,700 steps (§3.1, §3.2.2)Generative reward model with per-prompt rubrics (§3)No scalar models; the actor is the GRM and is itself RL-trained (§5.1.1)
Agent tasksNot described in the sections read85,267 tasks; 1,827 synthesized environments inside the 4,417 general-agent tasks (§3.2.3)DSec sandbox platform, hundreds of thousands of concurrent instances; task counts not stated (§5.2.5)
Reasoning kept across tool callsDiscarded at the next round (as V3.2 describes it)Kept until a new user message (§3.2.1)Kept across all turns when tools are used (§5.1.1)
Skill integrationSFT mixture of about 600K reasoning and 200K other samples, then RLSpecialists, then distilled data, then one mixed RL stage (§3)Specialists, then multi-teacher OPD with more than ten teachers (§5.1)
Effort or lengthMax output 32,768, then 65,536 tokens after step 8.2K (R1-Zero, §2.1)Length penalty; Speciale with a reduced one (§3)Three modes, each with its own penalty and window (§5.1.1)
RL compute101K H800 GPU-hours for R1-Zero; 147K total (Table 7)"Exceeding 10% of the pre-training cost" (§1)Not stated in the sections read

What was kept is as informative as what moved. All three keep the critic-free group baseline and the <think> format ("Consistent with our previous version", says V43, §5.1.1), all three lean on rule-based verifiers where answers can be checked, and all three treat language consistency and length as reward terms. The changes are concentrated in four places: how rollouts and trainer are kept consistent, what judges open-ended output, what the harness does to the reasoning trace, and how skills are combined.

8.1 What the benchmark deltas can and cannot say

Table 7 compares V3.2 Thinking with V4-Pro-Max on benchmark names that appear in both reports. It is a record of what the releases scored, not an attribution to post-training: pre-training, architecture, parameter count, context length, evaluation window and agent harness all changed at once.

Table 7. Shared benchmarks, V3.2 Thinking2 (Table 2) against V4-Pro Max3 (Tables 6, 7). Pass@1 or accuracy (%) unless noted. Gain is V4 minus V3.2 (derived).
BenchmarkV3.2 ThinkingV4-Pro MaxGain (points)Comparability note
MMLU-Pro (EM)85.087.5+2.5Same name
GPQA Diamond82.490.1+7.7Same name
HLE (text only)25.137.7+12.6V3.2 also reports 23.9 with the official template
LiveCodeBench83.393.5+10.2V3.2 window 2024.08-2025.04; V4 uses v6
IMOAnswerBench78.389.8+11.5Same name
SWE Verified73.180.6+7.5Each lab's own harness
SWE Multilingual70.276.2+6.0Each lab's own harness
Terminal Bench 2.046.467.9+21.5V3.2 used the Claude Code framework; V4 an internal one; V4 notes environment issues in the benchmark
BrowseComp (discard-all)67.683.4+15.8Same strategy; V3.2 at a 128K window, V4 up to 512K context
Tool-Decathlon / Toolathlon35.251.8+16.6Same benchmark, spelled differently in the two reports

Codeforces is left out on purpose: V3.2 reports 2386 and V4 3206, but V4 describes an internal protocol over 14 Division 1 contests3 (§5.3.1) and the two ratings are not defined the same way in the sections read. The largest gains are on the agentic rows, and the smallest is on MMLU-Pro, a mostly knowledge-based benchmark; V4 itself ties knowledge to parameter count when it compares Flash with Pro3 (§5.3.2). Beyond that, the table does not license a statement about OPD. A 3,206 or 67.9 would look the same if the gain came from a better base model, a longer window or a different harness.

9. What dropping mixed RL buys and costs

V4 states the substitution and one reason. The trade-offs below combine what the three reports say with what follows from the mechanism; the second kind is marked as my reading, because none of it is ablated in the DeepSeek reports.

9.1 What the change plausibly buys

  • Independence of the specialists. Each expert is a separate SFT-then-GRPO run, so a failed or restarted domain does not disturb the others (V4 §1; the MOPD paper makes the same argument for parallel pipelines13). In V3.2's single stage, a reward-hacking episode in the rubric judge shares a run with the code and search tasks.
  • No cross-domain reward weighting inside one run. A mixed GRPO stage must choose how rule-based rewards, length penalties and rubric scores trade off on one scale. In OPD the target is a teacher's distribution per domain (V4 Eq. 29), which sidesteps that choice and moves it into \(w_i\) (my reading).
  • A dense signal. Every student token is graded against a teacher on the vocabulary, instead of one scalar per trajectory spread over thousands of tokens. Module 21 derives the information argument and its caveats11.
  • Effort modes as separate specialists. V4 trains specialists "under divergent RL configurations" to realize its three modes3 (§5.1.1). Training them separately lets each get its own length penalty and window without the others' rewards in the same run (my reading; how the modes map onto teachers is not stated).

9.2 What it plausibly costs

  • A ceiling at the teachers. The student is trained to match experts, not to maximize the reward directly. V3.2's mixed stage continued optimizing the real reward on the merged model and, in its words, eliminated the gap to the specialists2 (§3). V4's report says the mixed stage was "entirely replaced", so the student's last stage has no outcome reward of its own. That means it can inherit teacher errors, including reward-hacked behavior that a specialist's RL picked up, and the report does not describe a check for that (my reading).
  • Skills the student does not sample. A per-token log-ratio only grades tokens the student can emit. In Nemotron 3 Ultra's MOPD, HLE recovered only 16.9% of the teacher-student gap, which the authors attribute to a general-reasoning teacher that gained from "additional large-scale SFT and RL on a separate reasoning mixture generated by DeepSeek-V4-Pro" that the student never saw14 (§3.3.4, Table 5), as Module 22 records. V4's teachers all begin from one base model per its §1, but the report does not say which checkpoint the student starts from, so whether the same-origin condition that the MOPD paper found critical holds is not stated13.
  • The teacher fleet. More than ten teachers, some "potentially comprising trillions of parameters", loaded on demand3 (§5.2.2). From the inference side, the OPD stage is a teacher forward pass over every student sample: prefill-shaped, compute-bound work whose last-layer states are cached and whose head is applied per minibatch. V4's hidden-state cache avoids storing \(V\)-sized logits but not the forward pass itself (my reading). The report gives no OPD run time or GPU-hours, so the cost of this stage relative to a mixed RL stage is unknown.
  • Agentic coverage. Nemotron 3 Ultra reports that end-to-end agentic rollouts inside MOPD caused large inefficiency and used single-turn rollouts for most agentic tasks14 (§3.3). V4 builds a sandbox platform for agentic work in post-training, "during post-training and evaluation"3 (§5.2.5), but does not say whether its OPD stage rolls out agentic episodes end to end.

Where experts disagree

The two DeepSeek reports read like a natural experiment, one lab and one optimizer with different surrounding choices, but both are descriptions without ablations. Four disagreements run through them. Each block below states the positions as the sources hold them and ends with an assessment that says what would settle it.

A. How should separately useful skills end up in one model?

A1. One mixed RL stage

DeepSeek-V3.22 (§3); Qwen3's last stage covers more than 20 task types15 (§4.4)

Merge reasoning, agent and alignment RL so no stage overwrites another, and V3.2 says this "effectively balances performance across diverse domains".

  • One trainer and one recipe; real reward optimized on the final model.
  • Each domain gets fewer samples per batch as environments multiply, the dilution argument Nemotron 3 Ultra gives14 (§3.3).
  • V3.2 prints no measurement of the balance it claims.

A2. Specialists, then distillation into one student

DeepSeek-V43 (§5.1); Kimi K3's nine experts, three domains at three efforts16 (§4.1.2); the MOPD paper13

Train each skill to its peak independently and merge by matching the teachers' distributions on the student's own samples.

  • Parallel, restartable specialists; dense signal.
  • The one controlled comparison favors it at small scale (0.937 against 0.882)13.
  • V4 gives no ablation for the claim that mixed RL degrades.
  • Skills outside the student's sampling distribution transfer poorly (HLE recovery 16.9% in Nemotron)14.

A3. Both: unified RL first, distillation second

Nemotron 3 Ultra14 (§3.2, §3.3)

Start the student from a unified RLVR run, then use specialist teachers to close the remaining per-domain gaps.

  • The student starts strong, and MOPD adds recovery on agentic and instruction-following tasks.
  • Pays for both stages, and the first still has the dilution problem.
Assessment

The choice depends on the number of domains, how different their rollouts are, and whether the missing skill is inside the student's own samples. With few similar domains, A1 is defensible and cheaper. With many environments including long agentic ones, three of the frontier reports read here (V4, Kimi K3, Nemotron 3 Ultra) use specialists as teachers, and the dilution mechanism is sound. A3 is the conservative hedge. V3.2 and V4 are the same lab's two bets, and neither report contains a matched-compute run of the other's method, so "A2 wins at frontier scale" is an inference from adoption. The experiment that would settle it is one SFT checkpoint trained both ways with equal GPU-hours at ten or more domains (Module 22 gives the same verdict at more length).

B. Should the KL term to the reference model stay?

B1. Keep it, repair it, tune it by domain

DeepSeek-V3.22 (§3.1); ProRL17 (§2.2)

V3.2 corrects the K3 estimator18 with \(\pi_\theta/\pi_{old}\) and lets mathematics use a weak or absent term. ProRL argues a KL term (with reference resets) is a more stable guard against entropy collapse than clipping tricks, and that advice to remove it applies mainly to base models before any SFT.

  • A leash against drift in non-verifiable domains such as writing and alignment.
  • Extra knobs per domain; V3.2 gives no values and no ablation.

B2. Remove it

DAPO6 (§2.3); Olmo 37 (§4.4.1); GLM-55 (§3.2)

In long-CoT reasoning the policy should move far from the reference, so the constraint is unnecessary (DAPO), and removing it speeds improvement (GLM-5).

  • One fewer estimator to get wrong; no reference-model forward pass.
  • No built-in restraint where the reward is soft.
Assessment

The reports split by reward type. Verifiable reasoning RL drops KL, and V3.2's own sentence about mathematics agrees. V3.2 says only that the strength varies by domain; a nonzero term should matter most where a soft judge can be exploited, as with rubric-scored writing and alignment prompts (my reading). V3.2's contribution is the correction, not the verdict: if you keep the K3 term in an off-policy loop, importance-weight it (derived in Module 6 §3.6). What would settle the question is the same run with and without the term on a rubric-judged domain; none of these reports contains it.

C. Should the advantage be divided by the group standard deviation?

C1. Do not divide

DeepSeek-V3.22 (§3.1); Dr. GRPO19 (§3.1); Olmo 37 (§4.4.1)

Dividing weights questions that are too easy or too hard more heavily, a difficulty bias. V3.2 uses \(R_i-\mathrm{mean}(\boldsymbol R)\) and states no reason.

  • Removes the bias; keeps the scale of each reward term meaningful.
  • The demonstrations are at 1.5B to 3B scale (Module 11).

C2. Keep GRPO's normalization

R11 (Eq. 3); DAPO6, which changes the loss aggregation and leaves the group division in place

The original form; every group's advantages have unit scale.

  • Comparable updates across prompts of different reward spread.
  • Can magnify noise in nearly all-correct or all-wrong groups, and can swamp a small length term (Module 23).
Assessment

V3.2's mixed reward adds a length penalty, a language term and rubric scores to an outcome reward. My reading, not stated in the report, is that this is where not dividing helps: a division by the standard deviation can change how large those small terms are relative to correctness. The evidence is mechanism plus small-scale demonstrations; no frontier report ablates it, and V4 does not say what its specialists use.

D. Who scores open-ended output?

D1. Rules for reasoning, a scalar model late and briefly

R11 (§2.2, §3.2.2, Supp. B.5)

No neural reward model on reasoning because of hacking; helpfulness and safety models only in the last 400 steps.

  • Observed hacking: reward rose while Codeforces performance fell.
  • Leaves open-ended quality to a short, late stage.

D2. A rubric judge per prompt

DeepSeek-V3.22 (§3); Rubrics as Rewards20; Kimi K2's rubric critic10

Score against instance-specific criteria. RaR reports up to 31% relative gain on HealthBench over direct Likert judging (the authors' numbers).

  • Structured criteria reduce variance across judges.
  • V3.2 does not say which model judges or who writes the rubrics.

D3. The actor is the judge, trained jointly

DeepSeek-V43 (§5.1.1)

No scalar models; RL applied to the GRM itself; "minimal" human annotation.

  • One model; reasoning inside the judgment; low annotation cost (claimed).
  • No ablation, no judge accuracy, no comparison with a separate or frozen judge; the judge shifts with the policy.
Assessment

Only D2 has outside support, and that support is for rubrics, not for the judge's identity. D3 is a vendor claim until someone publishes judge accuracy against human labels, tracked through the RL run, for a joint judge and a frozen one. No citation links V3.2 or V4 to DeepSeek's earlier GRM paper, so treat the three positions as separate designs, not a lineage.

What is changing

V4 is a v1 "preview"; its identifier and date disagree (Section 6) and its post-training sections may be revised. Three open items as of 2026-10-27. First, whether the OPD choice survives replication: the controlled evidence is on a 30B-A3B model, Nemotron and Kimi report their own versions, and no matched-compute run against mixed RL exists in the reports read (Module 22). Second, whether the actor-as-judge design is ablated by anyone, since it moves the reward model inside the policy. Third, how RL compute is reported: V3.2's "more than 10%" has an undefined denominator and V4 gives none, so the ladder in Figure 4 stops at V3.2. Modules 31 and 32 read two other recipes (Olmo 3; GLM-5 and Nemotron 3) that took parts of the same toolbox.

References

  1. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948 v2 (4 Jan 2026; the expanded version of the Nature 645, 633-638 article). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: GRPO settings and advantage (§2.1, Eq. 3), rule-based rewards and the stance against neural reward models (§2.2), stages and reward models (§3.1, §3.2.1, §3.2.2), reward hacking (Supp. B.5), SFT data size (Supp. B.3.3), training cost (Supp. B.4.4, Table 7).
  2. paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv 2512.02556 v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the whole of §3 (specialist distillation, mixed RL, rewards), §3.1 (GRPO objective Eqs. 5-9, the five stability operations), §3.2 (thinking in tool use, task counts in Table 1, synthesis pipeline), §4.1-§4.3 (results, Table 5), §2.1.1 (DSA training), §1 (budget above 10%).
  3. paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348 v1 (the PDF stamp reads 26 Apr 2026; the identifier prefix is 2606). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: §1 and §5.1 (pipeline, mixed RL replaced by OPD), §5.1.1 (specialists, effort modes, GRM, interleaved thinking), §5.1.2 (Eq. 29, full-vocabulary OPD), §5.2 (FP4 QAT, teacher scheduling, rollout WAL, DSec), §3.3 (deterministic kernels), §5.3 and Tables 6-7 (evaluation).
  4. paperShao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv 2402.03300 v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the origin of GRPO, the optimizer R1, V3.2 and V4 all name (§4.1).
  5. paperGLM-5 Team (Zhipu AI). GLM-5: from Vibe Coding to Agentic Engineering. arXiv 2602.15763 v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the non-deterministic indexer top-k failure and the torch.topk fix on a DSA model (§3.2), KL term removed (§3.2), "on-policy cross-stage distillation" (§1).
  6. paperYu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv 2503.14476 v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: KL term removed because long-CoT policies are expected to diverge (§2.3); token-level loss that leaves the group-level advantage normalization in place (§3.3, Eq. 8; see Module 11).
  7. paperTeam Olmo (Allen Institute for AI). Olmo 3. arXiv 2512.13961 v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL with no KL loss and no standard-deviation normalization (§4.4.1).
  8. paperDeepSeek-AI. DeepSeek-V3 Technical Report. arXiv 2412.19437. https://arxiv.org/abs/2412.19437. Accessed 2026-10-27. Used for: Table 1, pre-training 2,664K, context extension 119K, post-training 5K, total 2,788K H800 GPU-hours (not in the source map; fetched for this module).
  9. officialarXiv. arXiv API query for id 2606.19348 (record for "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence", v1). https://export.arxiv.org/api/query?id_list=2606.19348. Accessed 2026-10-27 (server timestamp 2026-10-26 23:14 UTC). Used for: the published and updated fields, both 2026-04-26, against the 2606 identifier prefix (not in the source map; fetched for this module).
  10. paperMoonshot AI. Kimi K2: Open Agentic Intelligence. arXiv 2507.20534 v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: the self-critic reward refreshed on verifiable-reward rollouts (§3.2.2).
  11. blogLu, K. et al. (Thinking Machines Lab). On-Policy Distillation. Thinking Machines Lab: Connectionism, 27 Oct 2025 (updated June 2026). https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-26. Used for: the reference V4 cites as "Lu and Lab, 2025"; the dense per-token signal argument behind Section 9.1 (derived in Module 21).
  12. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv 2504.12501 v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: grouping V4, GLM-5, Kimi K3, Nemotron 3 Ultra and MiMo-V2-Flash under MOPD, and "far from unanimous" (§3.2.4).
  13. paperMa, W. et al. (Xiaomi LLM Core). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv 2606.30406 v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-26. Used for: the controlled comparison on Qwen3-30B-A3B (0.937 against 0.882, 0.775 and 0.824; per-domain range 0.044), parallel teachers, the same-origin finding (abstract, §1, §4).
  14. paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv 2606.15007 v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: unified RLVR followed by MOPD, the dilution argument (§3.2, §3.3), HLE recovery 16.9% and its explanation (§3.3.4, Table 5), end-to-end agentic rollouts in MOPD (§3.3).
  15. paperYang, A. et al. (Qwen Team). Qwen3 Technical Report. arXiv 2505.09388 v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: general RL over more than 20 tasks as a mixed final stage (§4.4).
  16. paperMoonshot AI. Kimi K3: Open Frontier Intelligence. arXiv 2607.24653 v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: nine experts (three domains at three efforts) consolidated by multi-teacher on-policy distillation (§4.1, §4.1.2).
  17. paperLiu, M. et al. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv 2505.24864 v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: keeping a KL term with reference resets, and the scope of the advice to remove it (§2.2).
  18. blogSchulman, J. Approximating KL Divergence. 2020. http://joschu.net/blog/kl-approx.html. Accessed 2026-10-26. Used for: the K3 estimator that V3.2 corrects with an importance ratio (derived in Module 6).
  19. paperLiu, Z. R. et al. Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO). arXiv 2503.20783 v2, 6 Oct 2025; COLM 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the difficulty bias from dividing by the group standard deviation, and the length bias (§3.1).
  20. paperGunjal, A. et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv 2507.17746 v2, 3 Oct 2025. https://arxiv.org/abs/2507.17746. Accessed 2026-10-26. Used for: instance-specific rubric rewards and the reported relative gain over Likert-style judging on HealthBench (abstract; authors' numbers).