Reinforcement Learning for LLM Post-Training · Module 30
Case Study: DeepSeek-V3.2 to V4: Scaling GRPO Past 10% of Pre-Training Compute, Then Replacing Mixed RL with On-Policy Distillation
DeepSeek's three reasoning-era reports share one optimizer and differ in nearly everything around it. R1 stacked SFT and RL stages; V3.2 merged reasoning, agent and alignment RL into one GRPO stage and spent more than 10% of pre-training cost on it; V4 dropped that stage and distilled more than ten specialists into one student. This module reads the three recipes against each other, including what each report states, leaves out and never tests. Afterwards you can say which R1 choices DeepSeek revised and why, what V4 gave up by dropping mixed RL, and which V3.2 stability fixes an RL loop of your own would still need.
1. What R1 settled, and what it left open
R1-Zero ran GRPO4 directly on DeepSeek-V3-Base with a reward that checks only the final answer, and AIME 2024 pass@1 rose from 15.6% to 77.9% (arXiv v2)1. Module 27 reads that recipe stage by stage; this module needs five of its facts, because each is something a later report changes.
- Optimizer settings. Advantage \(A_i=(r_i-\mathrm{mean})/\mathrm{std}\) over a group of 16, KL coefficient 0.001, 32 questions per step, the reference model replaced by the latest policy every 400 steps, and each 8,192-output rollout split into 16 minibatches with one inner epoch1 (§2.1, Eq. 3).
- Reasoning rewards. Rule-based accuracy and format only. The report states that it abstains "from applying neural reward models—whether outcome-based or process-based—to reasoning tasks" because they are "susceptible to reward hacking during large-scale reinforcement learning"1 (§2.2).
- Stages. Cold-start SFT, reasoning RL with a language-consistency reward, rejection sampling plus SFT, then a second RL stage. Helpfulness and harmlessness reward models enter only in the last 400 of that stage's 1,700 steps, because longer exposure to the preference reward "may lead to reward hacking"1 (§3.2.2, Supp. B.5).
- Context. Maximum output length 32,768 tokens, raised to 65,536 after step 8.2K1 (§2.1).
- Cost. 101K H800 GPU-hours for R1-Zero, 5K for SFT data creation and 41K for R1, 147K in total, which the report prices at $294K at $2 per GPU-hour1 (Table 7).
V3.2's introduction names three deficiencies of open models: vanilla attention, "insufficient computational investment during the post-training phase", and weak generalization in agent settings2 (§1). The last two are the RL half of the report, and they turn R1's loose ends into four questions that organize the rest of this module. How do you push RL past 10% of pre-training cost, against 3.8% to 5.5% for R1 (derived in Section 5), without the run diverging (Sections 4 and 5)? How do you get reasoning and tool use into one trajectory (Section 3)? Where does the reward for open-ended prompts come from now that a scalar reward model is the thing R1 saw being hacked (Sections 3 and 6)? And how should skills trained separately end up in one set of weights (Sections 2, 7 and 9)? V3.2 answers all four; V4 keeps most of the answers to the first three and reverses the fourth.
2. DeepSeek-V3.2: specialists first, then one mixed RL stage
2.1 The checkpoint RL starts from, and the attention it runs through
V3.2 starts from DeepSeek-V3.1-Terminus, whose context had been extended to 128K, and adds DeepSeek Sparse Attention (DSA) by continued pre-training in two stages: a dense warm-up that trains only the lightning indexer (1,000 steps of 16 sequences of 128K tokens at learning rate \(10^{-3}\)), then a sparse stage that trains everything with 2,048 key-value entries selected per query (15,000 steps of 480 sequences at \(7.3\times10^{-6}\))2 (§2.1.1). The report's token totals, 2.1B and 943.7B, follow from reading 128K as 131,072 tokens: \(1{,}000\times16\times131{,}072=2.10\mathrm{B}\) and \(15{,}000\times480\times131{,}072=943.7\mathrm{B}\) (derived). Post-training "employs sparse attention in the same way as the sparse continued pre-training stage"2 (§3), so rollouts are sampled through the same top-\(k\) selection (\(k=2{,}048\) in the continued pre-training stage; the post-training value is not restated) and the trainer must reproduce that selection when it recomputes probabilities. The report does not discuss how the two engines keep the indexer's choice consistent. GLM-5, which also trains on DSA, reports that a non-deterministic top-k kernel in the indexer degraded RL drastically within a few steps and that a deterministic torch.topk fixed it5 (§3.2). That is an observation about GLM-5's stack, not a finding about DeepSeek's, but it names the failure that Section 4's fixes do not cover.
2.2 Specialists, distilled data, then a single stage
For each domain the team first builds a specialist, all fine-tuned from the same pre-trained V3.2 base checkpoint. The domains are writing and general question answering plus six named areas: mathematics, programming, general logical reasoning, general agentic tasks, agentic coding and agentic search, each in thinking and non-thinking mode, with different models generating the long chain-of-thought data and the direct-response data. Each specialist is "trained with large-scale Reinforcement Learning (RL) computing", and the specialists then produce the domain data for the final checkpoint2 (§3). The report's evidence for this step is one sentence: models trained on the distilled data reach performance "only marginally below those of domain-specific specialists, with the performance gap being effectively eliminated through subsequent RL training". No table shows the gap.
The final stage is GRPO4, with "reasoning, agent, and human alignment training" merged "into one RL stage", which the report says "effectively balances performance across diverse domains while circumventing the catastrophic forgetting issues commonly associated with multi-stage training paradigms". The merged data then go through "thousands of steps of continued RL training to reach the final checkpoints"2 (§3). The mixed stage is therefore RL on top of a model already trained on distilled specialist data, not RL from a cold start. That matters for Section 9, because V4 removes this RL pass.
2.3 What the recipe does not tell you
The report prints an objective, an advantage and four stability operations, and few of the numbers that would let you rerun them. Table 1 lists what a reproducer needs against what the sections give.
| Item | What the report gives | Where |
|---|---|---|
| Algorithm and advantage | GRPO; \(\hat A_{i,t}=R_i-\mathrm{mean}(\boldsymbol R)\), no division by the standard deviation | §3, §3.1 |
| Loss aggregation | Printed per token with \(1/|o_i|\) (Eqs. 5, 6, 8); whether the code matches is not stated (Module 11) | §3.1 |
| Group size \(G\), prompts per step, batch size | Not stated; \(G\) appears only as a symbol | §3.1 |
| Learning rate, \(\varepsilon\), \(\beta\) | Not stated for RL. The only learning rates printed are the DSA ones: \(10^{-3}\) and \(7.3\times10^{-6}\) | §2.1.1, §3.1 |
| Mask threshold \(\delta\) | Not stated (Module 12) | §3.1, Eq. 9 |
| Run length | "Thousands of steps" | §3 |
| Length penalty, language consistency | Named; form and weights not stated | §3 |
| Judge for general tasks | A generative reward model with per-prompt rubrics; which model it is and who writes the rubrics is not stated | §3 |
| RL compute | "Exceeding 10% of the pre-training cost"; the denominator is not defined | §1, §4.1 |
2.4 Speciale
V3.2-Speciale is the same RL run with the dials turned toward length: "trained exclusively on reasoning data with a reduced length penalty", plus the DeepSeekMath-V2 proof data and reward2 (§3). It scores 35/42 on IMO 2025, 102/126 on CMO 2025 and 492/600 on IOI 2025, and solved 10 of 12 ICPC World Finals 2025 problems, at a 128K generation cap and without tools (Table 4, Appendix D); the report concedes its token efficiency "remains significantly inferior" to Gemini-3.0-Pro's2 (§4.2). Module 23 reads the length-penalty arithmetic.
3. V3.2's task pool and what scores it
3.1 Four populations, three of them in real environments
The introduction's "over 1,800 distinct environments and 85,000 complex prompts"2 (§1) is two numbers about two different things. Table 1 of the report counts agent tasks in four populations, and the rows sum to 85,267 (derived): 24,667 code-agent tasks, 50,275 search-agent tasks, 4,417 general-agent tasks and 5,908 code-interpreter tasks2 (§3.2.3). The 1,827 environments belong to the general-agent row alone; the other three run in real environments (live search APIs, coding tools, Jupyter notebooks) with prompts that are extracted from the internet or synthesized, "rather than obtained from actual user interactions". Figure 2 draws the pool. Module 17 covers how each population is built and filtered.
| Population | Tasks | Share (%) | Environment | Prompts | Scoring the report states |
|---|---|---|---|---|---|
| Search agent | 50,275 | 59.0 | real | synthesized | Verifiable QA pairs kept only when the ground truth checks out and every candidate answer is verifiably wrong; plus helpfulness prompts scored by a rubric GRM |
| Code agent | 24,667 | 28.9 | real | extracted | Rule-based outcome reward; an environment is admitted only if the gold patch yields at least one F2P test and zero P2F tests |
| Code interpreter | 5,908 | 6.9 | real (Jupyter) | extracted | Rule-based outcome reward on math, logic and data-science problems that need code execution |
| General agent | 4,417 | 5.2 | synthesized | synthesized | A per-task Python verification function; only tasks with non-zero pass@100 after RL with V3.2 are kept |
3.2 A rubric judge where there is no checker
For reasoning and agent tasks the reward is a rule-based outcome reward, a length penalty and a language-consistency reward. "For general tasks, we employ a generative reward model where each prompt has its own rubrics for evaluation"2 (§3). The search-agent pipeline shows the hybrid in use: verifiable question-answer pairs reward factual reliability, and "detailed evaluation rubrics across multiple quality dimensions" scored by a generative reward model reward practical helpfulness (§3.2.3). Section 3.1 of the same report says "a set of reward models" scores each output, without saying which2.
Set that against R1, whose general-purpose reward was a scalar model trained on 66,000 preference pairs that DeepSeek-V3 labeled, and whose authors saw reward rise while Codeforces performance fell1 (§3.1, Supp. B.5). V3.2 never says that hacking was the reason for the switch, and it gives no comparison between a scalar model and the rubric judge. The move is documented; the motive and the benefit are not. Module 16 covers rubric judges and their failure modes.
3.3 The one measured claim about synthetic tasks
Section 4.3 is the only environment-design ablation among the seven reports Module 17 searched. It asks whether the synthesized general-agent tasks are hard and whether RL on them transfers. On 50 sampled tasks, V3.2-Exp reaches 12% at pass@1 and 26% at pass@4, against 34% and 62% for Claude Sonnet 4.5, 51% and 74% for Gemini 3.0 Pro, and 62% and 82% for GPT-5 Thinking2 (Table 5). For transfer, RL on only the synthetic tasks in non-thinking mode, starting from the SFT checkpoint, "yields substantial improvements" on τ²-Bench, MCP-Mark and MCP-Universe, whereas RL restricted to code and search "does not improve performance on these benchmarks" (Figure 5, plotted as curves without tabulated values). It is one run per condition on a 50-task sample, so treat it as evidence that the data are not trivially easy and that they add something code-and-search RL does not; Module 17 lists the limits.
3.4 Keeping the reasoning when a tool answers
V3.2 describes R1's strategy as discarding reasoning content when the second round of messages arrives. In tool use that "forces the model to redundantly re-reason through the entire problem for each subsequent tool call", so V3.2 drops historical reasoning only when a new user message arrives, keeps it while only tool outputs are appended, and always keeps the tool-call history2 (§3.2.1). The cold start is deliberately light: system prompts that ask for reasoning, for tool calls, or for tool calls inside the <think> block, with the admission that "the reasoning in tool-use patterns may lack robustness" and the model only "occasionally" produces the wanted trajectories, which is enough to start RL (§3.2.2, Appendix B). V4 goes further and keeps the whole reasoning history across user turns in tool-calling scenarios3 (§5.1.1). Module 18 covers what the harness does to the sequence the policy conditions on.
4. V3.2's stability toolbox, drawn as the boundary between engine and trainer
After the mean-subtracted advantage, V3.2 lists four "additional strategies that stabilize RL scaling, directly building on the GRPO algorithm"2 (§3.1). Read as systems work, each one is either a piece of metadata that has to cross from the rollout engine to the trainer, or a rule for what the trainer does when that metadata says the two have drifted apart. Figure 3 steps through one iteration to show which is which. The mechanisms themselves are derived elsewhere (the KL fix in Module 6 §3.6, the sequence mask in Module 12 §5.3, routing and sampling masks in Module 14); the point here is the pattern.
- Sample. The engine draws a group of outputs per prompt with top-p and top-k truncation and MoE routing, and returns each token's probability directly.
- Keep Sampling Mask. The truncation mask crosses to the trainer, which restricts \(\pi_\theta\) to the same kept tokens, so both policies "share identical action subspaces".
- Keep Routing. The expert paths used at sampling are forced in the trainer's forward pass, so the same expert parameters are optimized.
- Reward, then advantage. Rule-based rewards, length penalty, language consistency and the rubric judge give \(R_i\); the advantage subtracts the group mean and does not divide.
- Off-policy sequence mask. A sequence with negative advantage whose mean log-ratio of the engine's \(\pi_{old}\) to the current \(\pi_\theta\) exceeds \(\delta\) is masked out whole. \(\delta\) is not stated.
- KL to the reference. The K3 estimate is weighted by \(\pi_\theta/\pi_{old}\); the coefficient depends on the domain and is weak or absent for mathematics.
Figure 3. One V3.2 iteration, stepped. Cyan is the rollout engine, blue the trainer, orange the mask that changes which sequences count. What crosses the boundary is named in the report's own terms: probabilities "directly returned by the inference framework", truncation masks, expert routing paths2 (§3.1). The report does not say how masks and paths are stored or transported.
| Operation | Failure named in the report | Evidence given | Derived in |
|---|---|---|---|
| Advantage \(R_i-\mathrm{mean}(\boldsymbol R)\) | None stated for dropping the division | None | Module 11 |
| Importance-corrected K3, per-domain \(\beta\) | When \(\pi_\theta\ll\pi_{ref}\) the K3 gradient gives "disproportionately large, unbounded weights", producing "noisy gradient updates" | None shown; "weak KL penalty or even omitting it entirely can yield improved performance" for mathematics | Module 6 §3.6 |
| Off-policy sequence masking | Minibatch reuse plus engine-versus-trainer differences make rollouts off-policy; "highly off-policy negative samples can be detrimental" | Reported to improve stability "in certain training scenarios"; no numbers, \(\delta\) not given | Module 12 §5.3 |
| Keep Routing | Inconsistent expert routing between engines causes "abrupt shifts in the active parameter subspace" | Operation "found crucial for RL training stability of MoE models"; used since DeepSeek-V3-0324 | Module 14 |
| Keep Sampling Mask | Truncating the action space at sampling but not in training "violates the principles of importance sampling" | Top-p with the mask "effectively preserves language consistency" | Modules 13, 14 |
Three observations that the table cannot hold. First, no row has an ablation: every fix is justified by a mechanism and a sentence about observed stability, which is why Module 12 and Module 14 supply the mathematics and the outside evidence. Second, the sequence mask is asymmetric by design. The report masks only negative-advantage sequences because "the model benefits the most by learning from its own mistakes, whereas highly off-policy negative samples can be detrimental"2 (§3.1). That is an argument from intuition, and because \(\pi_{old}\) is the engine's own probability the mask reacts to minibatch staleness and to engine mismatch without telling them apart. Third, the KL term is kept, repaired and tuned per domain. R1 used one coefficient, 0.0011 (§2.1); DAPO, Olmo 3 and GLM-5 delete the term675; V3.2 chose the option with the most knobs and prints none of the values.
5. How much compute: a ladder across four DeepSeek reports
V3.2 states its RL budget as a share, not as GPU-hours: a "post-training computational budget exceeding 10% of the pre-training cost", and later "consistent performance improvements correlating with extended RL training budget, which already exceeds 10% of the pre-training cost", with the hypothesis that "reasoning capabilities could be further enhanced with additional computational budget allocation"2 (§1, §4.1). The report does not define the denominator. DeepSeek's earlier reports give enough to put four points on one scale, if you accept one assumption: the denominator is V3's pre-training, 2,664K H800 GPU-hours8 (Table 1). That is exact for V3 and R1, which both start from V3-Base, and a stand-in for V3.2, whose base is V3.1-Terminus plus continued pre-training. Module 15 uses the same stand-in.
$$\frac{5}{2{,}664}=0.19\%,\qquad \frac{101}{2{,}664}=3.8\%,\qquad \frac{147}{2{,}664}=5.5\%,\qquad 10\%\times2{,}664\mathrm{K}=266.4\mathrm{K}\ \text{GPU-hours}$$The numerators are V3's whole post-training (SFT and RL together, 5K8), R1-Zero's RL run (101K), and R1's total including SFT data creation (147K)1 (Table 7). Under the assumption, V3.2's floor of 266.4K is 1.8 times everything R1 spent and 53 times what V3 spent on its entire post-training (derived). Figure 4 draws the ladder.
Three limits on reading the ladder. The denominators differ in kind: V3.2's base is V3.1-Terminus plus 945.8B tokens of continued pre-training (2.1B plus 943.7B, derived from §2.1.1), so 10% of its own pre-training cost is a different absolute budget from 10% of V3's. The numerators mix SFT and RL, and V3.2's does not say what it includes. And V4 reports no post-training compute at all in the sections read3, so the ladder stops at V3.2. What the budget bought is also stated rather than plotted: V3.2 reports similar performance to GPT-5-High on reasoning benchmarks, slightly below Gemini-3.0-Pro, and says its headline numbers are "constrained by a length constraint reward model", relaxed only in Speciale2 (§4.1). No source in this course fits a compute curve to a production run (Module 15).
6. DeepSeek-V4: specialists, an actor that judges, three effort modes
V4 comes in two sizes, V4-Pro with 1.6T parameters (49B activated) and V4-Flash with 284B (13B activated), pre-trained on 33T and 32T tokens, both with a one-million-token context3 (abstract, §1). Its post-training "largely mirrored that of DeepSeek-V3.2" except for one substitution, which Section 7 reads. This section covers what carried over and what the specialists became.
6.1 Specialists: the same GRPO, fewer words about it
Each domain expert is "sequentially optimized through an initial fine-tuning phase and subsequent Reinforcement Learning (RL) guided by domain-specific prompts and reward signals", using GRPO with "hyper-parameters closely aligned with our prior research", cited as "(DeepSeek-AI, 2025; DeepSeek-AI, 2025)"3 (§5.1.1). Its reference list has three DeepSeek-AI entries from 2025: R1, V3.2 and the 3FS file system, so the two citations are presumably R1 and V3.2 (my inference). The domains named in the introduction are mathematics, coding, agent and instruction following (§1). The section prints no group size, learning rate, KL coefficient, advantage form or stability operations, so V3.2's Table 1 gaps carry over and widen. The RL-relevant consistency statements I found sit in the infrastructure sections: bitwise batch-invariant, deterministic kernels so that "bitwise alignment among pre-training, post-training, and inference" holds (§3.3, covered in Module 14), and rollouts that use native FP4 expert weights instead of simulated quantization, "ensur[ing] that model behavior during sampling is fully consistent with online deployment" (§5.2.1). That is a different answer to the mismatch problem from V3.2's: remove the numerical gap instead of masking its consequences.
6.2 The actor is the judge
V4 states the reward design for hard-to-verify tasks in one paragraph. It "dispense[s] with these conventional scalar-based reward models", curates "rubric-guided RL data", and employs a generative reward model (GRM) to evaluate trajectories: "Crucially, we apply RL optimization directly to the GRM itself. In this paradigm, the actor network natively functions as the GRM, enabling the joint optimization of the model's evaluative (judging) proficiency alongside its standard generative capabilities"3 (§5.1.1). The stated benefits are that reasoning is "inherently fused into its evaluative process, resulting in highly robust scoring", and that the approach "achieves superior performance with only a minimal set of diverse human annotations".
All of that is the vendor's description. The V4 text contains no ablation of any kind (a search for "ablat" finds nothing), no judge accuracy and no comparison against a separate or frozen judge, so the design is a claim, not a measured result. Two framing facts help. Kimi K2 earlier used the policy as its own critic, but anchored it by refreshing the critic on verifiable-reward rollouts10 (§3.2.2); V4 describes no such anchor. And neither V3.2 nor V4 cites DeepSeek's earlier generative-reward-model paper, so this module asserts no lineage between them (Module 16 draws the same line). Two risks follow directly from the design and are not discussed in the report: the judge moves with the policy it scores, and a policy that learns to produce text its own judge likes has no external check in the loop. Module 16 §7 and Module 19 cover those failure modes.
6.3 Three effort modes
Both V4 models "support three specific reasoning effort modes. For each mode, we apply distinct length penalties and context windows during RL training"3 (§5.1.1). Table 4 collects what the report prints. The penalty values are not stated, and the report does not say whether each mode has its own teachers or how three modes enter one student; Module 23 reads the token counts and the length arithmetic.
| Mode | Response format | System-prompt change | Eval window (tokens) | Pro HLE | Pro TB 2.0 |
|---|---|---|---|---|---|
| Non-think | </think> then summary | none | 8K | 7.7 | 59.1 |
| Think High | <think> reasoning </think> then summary | none | 128K | 34.5 | 63.3 |
| Think Max | as High | injected "Reasoning Effort: Absolute maximum with no shortcuts permitted" | 384K | 37.7 | 67.9 |
The report's own reading is that "the Max mode, which employs longer contexts and reduced length penalties in RL, outperforms the High mode on the most challenging tasks"3 (§5.3.2). The table shows the shape: on Pro, High recovers most of the distance from Non-think (HLE 7.7 to 34.5), and Max adds 3.2 points on HLE and 4.6 on Terminal Bench for a window three times as large (derived from Table 7). Max is a prompt plus a separate training configuration, so adding its instruction to a High-trained model is not the same thing; the report does not test that.
7. DeepSeek-V4: on-policy distillation replaces the mixed stage
V4's report is blunt about the change: "the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation (OPD; Gu et al., 2024; Lu and Lab, 2025)"3 (§5.1). The stated flow is two stages. First "the independent cultivation of domain-specific experts", each by SFT on domain data and then GRPO against domain rewards; then "a single unified model is trained through on-policy distillation, wherein the unified model acts as the student learning to optimize the reverse KL loss with teacher models" (§1). Modules 21 and 22 derive the method; this section records what V4 specifies, what it gives as reasons, and what the report leaves unsaid.
7.1 The objective as printed
For \(N\) expert models \(\{\pi_{E_1},\dots,\pi_{E_N}\}\) the objective is Eq. 29:
$$\mathcal L_{\mathrm{OPD}}(\theta)=\sum_{i=1}^{N} w_i\cdot D_{\mathrm{KL}}\!\left(\pi_\theta\,\Vert\,\pi_{E_i}\right)$$with \(w_i\) "the assigned weight for each expert, typically determined by the relative importance of the expert"3 (§5.1.2). Computing reverse KL "requires sampling training trajectories from the student \(\pi_\theta\) to maintain on-policy learning", and the policy "selectively learns from the specialized expert relevant to the current task context (e.g., aligning with the mathematics expert for math reasoning tasks and the coding expert for programming tasks)". Read literally, the sum is over all experts; read with the sentence after it, a math prompt pulls toward the math expert. The report does not say whether \(w_i\) is fixed per domain, set per prompt, or non-zero for several experts at once, and it gives no schedule, batch size or run length for this stage. The sampled-token form is not what V4 uses, as Section 7.3 explains.
Two points of vocabulary. V4 writes "multi-teacher On-Policy Distillation (OPD)" and cites MiniLLM (Gu et al., 2024) and the Thinking Machines post (Lu and Lab, 2025)11; the string "MOPD" does not occur in V4 or in V3.2, and does not occur in GLM-5, which calls its final stage "on-policy cross-stage distillation"5 (§1). Lambert's survey groups V4, GLM-5, Kimi K3, Nemotron 3 Ultra and MiMo-V2-Flash under MOPD and says labs "are far from unanimous in its usage"12 (§3.2.4), and a separate paper from Xiaomi's LLM Core team is titled MOPD13. The label names a family. V4 is one member, with its own choices, and the controlled comparison in the MOPD paper is of a different instance.
7.2 The reasons V4 gives
The case for replacing mixed RL is one clause: aligning at the logit level consolidates "the knowledge from physically distinct expert weights ... into a unified parameter space ... practically circumventing the performance degradation often encountered in traditional weight-merging or mixed RL techniques"3 (§5.1.2). The report presents no mixed-RL run, no merge run and no measurement of degradation. This is the opposite diagnosis from V3.2's, which chose one mixed stage to avoid "catastrophic forgetting issues commonly associated with multi-stage training paradigms"2 (§3). The two claims are about different failure modes: V3.2 worried that sequential stages forget earlier skills, V4 that jointly optimizing many reward signals in one RL run degrades each. Both are plausible and neither report tests its own claim against the other. An outside controlled comparison exists on a small model: on Qwen3-30B-A3B with three domains, the MOPD paper reports a normalized score of 0.937 against 0.882 for mixed RL, 0.775 for cascade RL and 0.824 for off-policy fine-tuning, with MOPD's three per-domain scores in a range of 0.044 (the authors' numbers, on one instance of the method whose form Module 22 details)13 (§4). That supports the direction of V4's choice at small scale; it does not measure V4's trade at 1.6T parameters with more than ten domains. No matched-compute comparison of MOPD against mixed RL appears in V4 or in the Kimi K3 sections Module 22 read.
7.3 Why full-vocabulary, and what it costs
The cheap form of OPD, which V4 describes as what "prior works usually" do, replaces the full-vocabulary KL with "a token-level KL estimate at each token position" and reuses the RL stack by treating \(\mathrm{sg}[\log\pi_{E_i}(y_t\mid x,y_{\lt t})/\pi_\theta(y_t\mid x,y_{\lt t})]\) as the per-token advantage (sg is stop-gradient). The report says this "is resource-efficient" but "leads to high variance in gradient estimation and often causes training instability", and so V4 "adopt[s] full-vocabulary logit distillation"3 (§5.1.2). The claim that the sampled-token estimator is unstable has no ablation or training curve in the passages read (Module 21). Module 22 weighs the alternative evidence, including reports that found no gain from richer signals.
What V4 does show is the engineering that makes the exact form affordable (§5.2.2, §5.2.3), most of it problems an inference engineer will recognize. Teacher weights sit in central distributed storage, are loaded on demand with ZeRO-like sharding, and the framework "supports full-vocabulary On-Policy Distillation (OPD) with an effectively unbounded number of teachers, each potentially comprising trillions of parameters". Only the last-layer hidden states of each teacher are cached; logits are rebuilt through the prediction head at training time with "negligible recomputation overhead" (stated, not measured in the sections read; Module 22 derives that the head is about 1.9% of a Pro-sized forward pass). Samples are ordered by teacher index so each teacher head is loaded once per minibatch and at most one head is resident. The exact KL runs in a TileLang kernel. A further piece concerns rollouts: a token-granular write-ahead log plus saved KV cache lets generation resume after preemption, because "it is mathematically incorrect to regenerate unfinished requests from scratch, as this introduces length bias" (Module 18 §4.2 explains the selection effect).
7.4 Claim ledger
| Claim | Where | Evidence in the report | Status |
|---|---|---|---|
| Mixed RL entirely replaced by OPD | §5.1 | Statement of the pipeline | Decision, no control |
| Logit-level alignment avoids degradation from merging or mixed RL | §5.1.2 | None | Vendor claim |
| Sampled-token OPD is high-variance and unstable | §5.1.2 | None | Vendor claim |
| More than ten teachers | §5.1.2 | Count only | Stated |
| Hidden-state caching costs negligible recomputation | §5.2.2 | None quantified | Stated; derivable (Module 22) |
| The actor as GRM gives robust scoring with minimal annotation | §5.1.1 | None | Vendor claim |
| Max mode beats High on the hardest tasks | §5.3.2, Table 7 | Benchmark scores, one run each | Measured; confounded by window and penalty |
| Regenerating preempted requests biases length | §5.2.3 | Argument (shorter responses survive interruption more often) | Reasoned; correct as a selection effect |
8. R1, V3.2 and V4 side by side
Table 6 lines up the choices that changed. Cells say "not stated" where the sections read give no value; for R1 they refer to the passages Module 27 reads.
| Component | R1 | V3.2 | V4 |
|---|---|---|---|
| Starting point | V3-Base (§2.1) | V3.1-Terminus plus DSA continued pre-training (§2.1) | New pre-training, 33T (Pro) and 32T (Flash) tokens (§1) |
| Optimizer | GRPO | GRPO (§3) | GRPO "closely aligned" with prior work, per specialist (§5.1.1) |
| Advantage | \((r_i-\mathrm{mean})/\mathrm{std}\) (Eq. 3) | \(R_i-\mathrm{mean}(\boldsymbol R)\) (§3.1) | Not stated |
| KL to reference | \(\beta=0.001\); reference reset every 400 steps (§2.1) | Importance-corrected K3; strength per domain, weak or none for math; values not stated (§3.1) | Not stated |
| Off-policy and mismatch control | 16 minibatches per rollout; clip \(\varepsilon\) printed as 10 (§2.1, §3.2.1) | Negative-sequence mask, Keep Routing, Keep Sampling Mask (§3.1) | Batch-invariant deterministic kernels; native FP4 rollouts (§3.3, §5.2.1) |
| Reasoning reward | Rule-based accuracy and format; language consistency (§2.2, §3.2.1) | Rule-based outcome, length penalty, language consistency (§3) | Domain-specific rewards; length penalty per effort mode (§5.1.1) |
| Open-ended reward | Scalar helpfulness model (66K pairs) and safety model (106K prompts), last 400 of 1,700 steps (§3.1, §3.2.2) | Generative reward model with per-prompt rubrics (§3) | No scalar models; the actor is the GRM and is itself RL-trained (§5.1.1) |
| Agent tasks | Not described in the sections read | 85,267 tasks; 1,827 synthesized environments inside the 4,417 general-agent tasks (§3.2.3) | DSec sandbox platform, hundreds of thousands of concurrent instances; task counts not stated (§5.2.5) |
| Reasoning kept across tool calls | Discarded at the next round (as V3.2 describes it) | Kept until a new user message (§3.2.1) | Kept across all turns when tools are used (§5.1.1) |
| Skill integration | SFT mixture of about 600K reasoning and 200K other samples, then RL | Specialists, then distilled data, then one mixed RL stage (§3) | Specialists, then multi-teacher OPD with more than ten teachers (§5.1) |
| Effort or length | Max output 32,768, then 65,536 tokens after step 8.2K (R1-Zero, §2.1) | Length penalty; Speciale with a reduced one (§3) | Three modes, each with its own penalty and window (§5.1.1) |
| RL compute | 101K H800 GPU-hours for R1-Zero; 147K total (Table 7) | "Exceeding 10% of the pre-training cost" (§1) | Not stated in the sections read |
What was kept is as informative as what moved. All three keep the critic-free group baseline and the <think> format ("Consistent with our previous version", says V43, §5.1.1), all three lean on rule-based verifiers where answers can be checked, and all three treat language consistency and length as reward terms. The changes are concentrated in four places: how rollouts and trainer are kept consistent, what judges open-ended output, what the harness does to the reasoning trace, and how skills are combined.
8.1 What the benchmark deltas can and cannot say
Table 7 compares V3.2 Thinking with V4-Pro-Max on benchmark names that appear in both reports. It is a record of what the releases scored, not an attribution to post-training: pre-training, architecture, parameter count, context length, evaluation window and agent harness all changed at once.
| Benchmark | V3.2 Thinking | V4-Pro Max | Gain (points) | Comparability note |
|---|---|---|---|---|
| MMLU-Pro (EM) | 85.0 | 87.5 | +2.5 | Same name |
| GPQA Diamond | 82.4 | 90.1 | +7.7 | Same name |
| HLE (text only) | 25.1 | 37.7 | +12.6 | V3.2 also reports 23.9 with the official template |
| LiveCodeBench | 83.3 | 93.5 | +10.2 | V3.2 window 2024.08-2025.04; V4 uses v6 |
| IMOAnswerBench | 78.3 | 89.8 | +11.5 | Same name |
| SWE Verified | 73.1 | 80.6 | +7.5 | Each lab's own harness |
| SWE Multilingual | 70.2 | 76.2 | +6.0 | Each lab's own harness |
| Terminal Bench 2.0 | 46.4 | 67.9 | +21.5 | V3.2 used the Claude Code framework; V4 an internal one; V4 notes environment issues in the benchmark |
| BrowseComp (discard-all) | 67.6 | 83.4 | +15.8 | Same strategy; V3.2 at a 128K window, V4 up to 512K context |
| Tool-Decathlon / Toolathlon | 35.2 | 51.8 | +16.6 | Same benchmark, spelled differently in the two reports |
Codeforces is left out on purpose: V3.2 reports 2386 and V4 3206, but V4 describes an internal protocol over 14 Division 1 contests3 (§5.3.1) and the two ratings are not defined the same way in the sections read. The largest gains are on the agentic rows, and the smallest is on MMLU-Pro, a mostly knowledge-based benchmark; V4 itself ties knowledge to parameter count when it compares Flash with Pro3 (§5.3.2). Beyond that, the table does not license a statement about OPD. A 3,206 or 67.9 would look the same if the gain came from a better base model, a longer window or a different harness.
9. What dropping mixed RL buys and costs
V4 states the substitution and one reason. The trade-offs below combine what the three reports say with what follows from the mechanism; the second kind is marked as my reading, because none of it is ablated in the DeepSeek reports.
9.1 What the change plausibly buys
- Independence of the specialists. Each expert is a separate SFT-then-GRPO run, so a failed or restarted domain does not disturb the others (V4 §1; the MOPD paper makes the same argument for parallel pipelines13). In V3.2's single stage, a reward-hacking episode in the rubric judge shares a run with the code and search tasks.
- No cross-domain reward weighting inside one run. A mixed GRPO stage must choose how rule-based rewards, length penalties and rubric scores trade off on one scale. In OPD the target is a teacher's distribution per domain (V4 Eq. 29), which sidesteps that choice and moves it into \(w_i\) (my reading).
- A dense signal. Every student token is graded against a teacher on the vocabulary, instead of one scalar per trajectory spread over thousands of tokens. Module 21 derives the information argument and its caveats11.
- Effort modes as separate specialists. V4 trains specialists "under divergent RL configurations" to realize its three modes3 (§5.1.1). Training them separately lets each get its own length penalty and window without the others' rewards in the same run (my reading; how the modes map onto teachers is not stated).
9.2 What it plausibly costs
- A ceiling at the teachers. The student is trained to match experts, not to maximize the reward directly. V3.2's mixed stage continued optimizing the real reward on the merged model and, in its words, eliminated the gap to the specialists2 (§3). V4's report says the mixed stage was "entirely replaced", so the student's last stage has no outcome reward of its own. That means it can inherit teacher errors, including reward-hacked behavior that a specialist's RL picked up, and the report does not describe a check for that (my reading).
- Skills the student does not sample. A per-token log-ratio only grades tokens the student can emit. In Nemotron 3 Ultra's MOPD, HLE recovered only 16.9% of the teacher-student gap, which the authors attribute to a general-reasoning teacher that gained from "additional large-scale SFT and RL on a separate reasoning mixture generated by DeepSeek-V4-Pro" that the student never saw14 (§3.3.4, Table 5), as Module 22 records. V4's teachers all begin from one base model per its §1, but the report does not say which checkpoint the student starts from, so whether the same-origin condition that the MOPD paper found critical holds is not stated13.
- The teacher fleet. More than ten teachers, some "potentially comprising trillions of parameters", loaded on demand3 (§5.2.2). From the inference side, the OPD stage is a teacher forward pass over every student sample: prefill-shaped, compute-bound work whose last-layer states are cached and whose head is applied per minibatch. V4's hidden-state cache avoids storing \(V\)-sized logits but not the forward pass itself (my reading). The report gives no OPD run time or GPU-hours, so the cost of this stage relative to a mixed RL stage is unknown.
- Agentic coverage. Nemotron 3 Ultra reports that end-to-end agentic rollouts inside MOPD caused large inefficiency and used single-turn rollouts for most agentic tasks14 (§3.3). V4 builds a sandbox platform for agentic work in post-training, "during post-training and evaluation"3 (§5.2.5), but does not say whether its OPD stage rolls out agentic episodes end to end.
Where experts disagree
The two DeepSeek reports read like a natural experiment, one lab and one optimizer with different surrounding choices, but both are descriptions without ablations. Four disagreements run through them. Each block below states the positions as the sources hold them and ends with an assessment that says what would settle it.
A. How should separately useful skills end up in one model?
A1. One mixed RL stage
DeepSeek-V3.22 (§3); Qwen3's last stage covers more than 20 task types15 (§4.4)
Merge reasoning, agent and alignment RL so no stage overwrites another, and V3.2 says this "effectively balances performance across diverse domains".
- One trainer and one recipe; real reward optimized on the final model.
- Each domain gets fewer samples per batch as environments multiply, the dilution argument Nemotron 3 Ultra gives14 (§3.3).
- V3.2 prints no measurement of the balance it claims.
A2. Specialists, then distillation into one student
DeepSeek-V43 (§5.1); Kimi K3's nine experts, three domains at three efforts16 (§4.1.2); the MOPD paper13
Train each skill to its peak independently and merge by matching the teachers' distributions on the student's own samples.
- Parallel, restartable specialists; dense signal.
- The one controlled comparison favors it at small scale (0.937 against 0.882)13.
- V4 gives no ablation for the claim that mixed RL degrades.
- Skills outside the student's sampling distribution transfer poorly (HLE recovery 16.9% in Nemotron)14.
A3. Both: unified RL first, distillation second
Nemotron 3 Ultra14 (§3.2, §3.3)
Start the student from a unified RLVR run, then use specialist teachers to close the remaining per-domain gaps.
- The student starts strong, and MOPD adds recovery on agentic and instruction-following tasks.
- Pays for both stages, and the first still has the dilution problem.
The choice depends on the number of domains, how different their rollouts are, and whether the missing skill is inside the student's own samples. With few similar domains, A1 is defensible and cheaper. With many environments including long agentic ones, three of the frontier reports read here (V4, Kimi K3, Nemotron 3 Ultra) use specialists as teachers, and the dilution mechanism is sound. A3 is the conservative hedge. V3.2 and V4 are the same lab's two bets, and neither report contains a matched-compute run of the other's method, so "A2 wins at frontier scale" is an inference from adoption. The experiment that would settle it is one SFT checkpoint trained both ways with equal GPU-hours at ten or more domains (Module 22 gives the same verdict at more length).
B. Should the KL term to the reference model stay?
B1. Keep it, repair it, tune it by domain
DeepSeek-V3.22 (§3.1); ProRL17 (§2.2)
V3.2 corrects the K3 estimator18 with \(\pi_\theta/\pi_{old}\) and lets mathematics use a weak or absent term. ProRL argues a KL term (with reference resets) is a more stable guard against entropy collapse than clipping tricks, and that advice to remove it applies mainly to base models before any SFT.
- A leash against drift in non-verifiable domains such as writing and alignment.
- Extra knobs per domain; V3.2 gives no values and no ablation.
B2. Remove it
DAPO6 (§2.3); Olmo 37 (§4.4.1); GLM-55 (§3.2)
In long-CoT reasoning the policy should move far from the reference, so the constraint is unnecessary (DAPO), and removing it speeds improvement (GLM-5).
- One fewer estimator to get wrong; no reference-model forward pass.
- No built-in restraint where the reward is soft.
The reports split by reward type. Verifiable reasoning RL drops KL, and V3.2's own sentence about mathematics agrees. V3.2 says only that the strength varies by domain; a nonzero term should matter most where a soft judge can be exploited, as with rubric-scored writing and alignment prompts (my reading). V3.2's contribution is the correction, not the verdict: if you keep the K3 term in an off-policy loop, importance-weight it (derived in Module 6 §3.6). What would settle the question is the same run with and without the term on a rubric-judged domain; none of these reports contains it.
C. Should the advantage be divided by the group standard deviation?
C1. Do not divide
DeepSeek-V3.22 (§3.1); Dr. GRPO19 (§3.1); Olmo 37 (§4.4.1)
Dividing weights questions that are too easy or too hard more heavily, a difficulty bias. V3.2 uses \(R_i-\mathrm{mean}(\boldsymbol R)\) and states no reason.
- Removes the bias; keeps the scale of each reward term meaningful.
- The demonstrations are at 1.5B to 3B scale (Module 11).
C2. Keep GRPO's normalization
R11 (Eq. 3); DAPO6, which changes the loss aggregation and leaves the group division in place
The original form; every group's advantages have unit scale.
- Comparable updates across prompts of different reward spread.
- Can magnify noise in nearly all-correct or all-wrong groups, and can swamp a small length term (Module 23).
V3.2's mixed reward adds a length penalty, a language term and rubric scores to an outcome reward. My reading, not stated in the report, is that this is where not dividing helps: a division by the standard deviation can change how large those small terms are relative to correctness. The evidence is mechanism plus small-scale demonstrations; no frontier report ablates it, and V4 does not say what its specialists use.
D. Who scores open-ended output?
D1. Rules for reasoning, a scalar model late and briefly
R11 (§2.2, §3.2.2, Supp. B.5)
No neural reward model on reasoning because of hacking; helpfulness and safety models only in the last 400 steps.
- Observed hacking: reward rose while Codeforces performance fell.
- Leaves open-ended quality to a short, late stage.
D2. A rubric judge per prompt
DeepSeek-V3.22 (§3); Rubrics as Rewards20; Kimi K2's rubric critic10
Score against instance-specific criteria. RaR reports up to 31% relative gain on HealthBench over direct Likert judging (the authors' numbers).
- Structured criteria reduce variance across judges.
- V3.2 does not say which model judges or who writes the rubrics.
D3. The actor is the judge, trained jointly
DeepSeek-V43 (§5.1.1)
No scalar models; RL applied to the GRM itself; "minimal" human annotation.
- One model; reasoning inside the judgment; low annotation cost (claimed).
- No ablation, no judge accuracy, no comparison with a separate or frozen judge; the judge shifts with the policy.
Only D2 has outside support, and that support is for rubrics, not for the judge's identity. D3 is a vendor claim until someone publishes judge accuracy against human labels, tracked through the RL run, for a joint judge and a frozen one. No citation links V3.2 or V4 to DeepSeek's earlier GRM paper, so treat the three positions as separate designs, not a lineage.
What is changing
V4 is a v1 "preview"; its identifier and date disagree (Section 6) and its post-training sections may be revised. Three open items as of 2026-10-27. First, whether the OPD choice survives replication: the controlled evidence is on a 30B-A3B model, Nemotron and Kimi report their own versions, and no matched-compute run against mixed RL exists in the reports read (Module 22). Second, whether the actor-as-judge design is ablated by anyone, since it moves the reward model inside the policy. Third, how RL compute is reported: V3.2's "more than 10%" has an undefined denominator and V4 gives none, so the ladder in Figure 4 stops at V3.2. Modules 31 and 32 read two other recipes (Olmo 3; GLM-5 and Nemotron 3) that took parts of the same toolbox.
References
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948 v2 (4 Jan 2026; the expanded version of the Nature 645, 633-638 article). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: GRPO settings and advantage (§2.1, Eq. 3), rule-based rewards and the stance against neural reward models (§2.2), stages and reward models (§3.1, §3.2.1, §3.2.2), reward hacking (Supp. B.5), SFT data size (Supp. B.3.3), training cost (Supp. B.4.4, Table 7).
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv 2512.02556 v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the whole of §3 (specialist distillation, mixed RL, rewards), §3.1 (GRPO objective Eqs. 5-9, the five stability operations), §3.2 (thinking in tool use, task counts in Table 1, synthesis pipeline), §4.1-§4.3 (results, Table 5), §2.1.1 (DSA training), §1 (budget above 10%).
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348 v1 (the PDF stamp reads 26 Apr 2026; the identifier prefix is 2606). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: §1 and §5.1 (pipeline, mixed RL replaced by OPD), §5.1.1 (specialists, effort modes, GRM, interleaved thinking), §5.1.2 (Eq. 29, full-vocabulary OPD), §5.2 (FP4 QAT, teacher scheduling, rollout WAL, DSec), §3.3 (deterministic kernels), §5.3 and Tables 6-7 (evaluation).
- paperShao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv 2402.03300 v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the origin of GRPO, the optimizer R1, V3.2 and V4 all name (§4.1).
- paperGLM-5 Team (Zhipu AI). GLM-5: from Vibe Coding to Agentic Engineering. arXiv 2602.15763 v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the non-deterministic indexer top-k failure and the
torch.topkfix on a DSA model (§3.2), KL term removed (§3.2), "on-policy cross-stage distillation" (§1). - paperYu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv 2503.14476 v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: KL term removed because long-CoT policies are expected to diverge (§2.3); token-level loss that leaves the group-level advantage normalization in place (§3.3, Eq. 8; see Module 11).
- paperTeam Olmo (Allen Institute for AI). Olmo 3. arXiv 2512.13961 v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL with no KL loss and no standard-deviation normalization (§4.4.1).
- paperDeepSeek-AI. DeepSeek-V3 Technical Report. arXiv 2412.19437. https://arxiv.org/abs/2412.19437. Accessed 2026-10-27. Used for: Table 1, pre-training 2,664K, context extension 119K, post-training 5K, total 2,788K H800 GPU-hours (not in the source map; fetched for this module).
- officialarXiv. arXiv API query for id 2606.19348 (record for "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence", v1). https://export.arxiv.org/api/query?id_list=2606.19348. Accessed 2026-10-27 (server timestamp 2026-10-26 23:14 UTC). Used for: the
publishedandupdatedfields, both 2026-04-26, against the 2606 identifier prefix (not in the source map; fetched for this module). - paperMoonshot AI. Kimi K2: Open Agentic Intelligence. arXiv 2507.20534 v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: the self-critic reward refreshed on verifiable-reward rollouts (§3.2.2).
- blogLu, K. et al. (Thinking Machines Lab). On-Policy Distillation. Thinking Machines Lab: Connectionism, 27 Oct 2025 (updated June 2026). https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-26. Used for: the reference V4 cites as "Lu and Lab, 2025"; the dense per-token signal argument behind Section 9.1 (derived in Module 21).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv 2504.12501 v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: grouping V4, GLM-5, Kimi K3, Nemotron 3 Ultra and MiMo-V2-Flash under MOPD, and "far from unanimous" (§3.2.4).
- paperMa, W. et al. (Xiaomi LLM Core). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv 2606.30406 v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-26. Used for: the controlled comparison on Qwen3-30B-A3B (0.937 against 0.882, 0.775 and 0.824; per-domain range 0.044), parallel teachers, the same-origin finding (abstract, §1, §4).
- paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv 2606.15007 v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: unified RLVR followed by MOPD, the dilution argument (§3.2, §3.3), HLE recovery 16.9% and its explanation (§3.3.4, Table 5), end-to-end agentic rollouts in MOPD (§3.3).
- paperYang, A. et al. (Qwen Team). Qwen3 Technical Report. arXiv 2505.09388 v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: general RL over more than 20 tasks as a mixed final stage (§4.4).
- paperMoonshot AI. Kimi K3: Open Frontier Intelligence. arXiv 2607.24653 v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: nine experts (three domains at three efforts) consolidated by multi-teacher on-policy distillation (§4.1, §4.1.2).
- paperLiu, M. et al. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv 2505.24864 v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: keeping a KL term with reference resets, and the scope of the advice to remove it (§2.2).
- blogSchulman, J. Approximating KL Divergence. 2020. http://joschu.net/blog/kl-approx.html. Accessed 2026-10-26. Used for: the K3 estimator that V3.2 corrects with an importance ratio (derived in Module 6).
- paperLiu, Z. R. et al. Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO). arXiv 2503.20783 v2, 6 Oct 2025; COLM 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the difficulty bias from dividing by the group standard deviation, and the length bias (§3.1).
- paperGunjal, A. et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv 2507.17746 v2, 3 Oct 2025. https://arxiv.org/abs/2507.17746. Accessed 2026-10-26. Used for: instance-specific rubric rewards and the reported relative gain over Likert-style judging on HealthBench (abstract; authors' numbers).