Reinforcement Learning for LLM Post-Training · Module 22
Capability Integration: Mixed RL, Cascade RL, Parameter Merging, and Multi-Teacher On-Policy Distillation (MOPD)
Frontier models are now trained as a set of RL specialists (math, instruction following, software engineering, search) that must end up in one set of weights. This module compares the five ways to do that, derives the multi-teacher on-policy distillation objective, reads the one controlled comparison and the frontier reports that use the method, and lays the Kimi K3, DeepSeek-V4, Nemotron 3 Ultra, GLM-5 and DeepSeek-V3.2 recipes side by side. Afterwards you can choose between mixed RL, a cascade and MOPD for a multi-skill target, and say what each costs and where each fails.
1. The integration problem
1.1 Why skills are trained separately
Each target skill has its own RL pipeline. The MOPD paper lists them: verifiable-answer RL for mathematics, agent-style RL in executable sandboxes for software engineering, rubric-based RL for instruction following and writing, and web-environment RL for search agents1 (§1). The pipelines differ in what a rollout is (one response versus fifty tool turns), how long it takes to verify, and which hyperparameters survive. NVIDIA's Cascade RL paper puts the cost bluntly: cross-domain heterogeneity, including large variation in response length and verification latency, "complicates the RL infrastructure, slows training, and makes training curriculum … and hyperparameter selection challenging"2 (abstract). Modules 17 and 18 cover why the rollout shapes differ; this module takes the consequence. If every domain has its own recipe, there are N trained policies and one deliverable, and some operation has to turn N into one.
1.2 Five routes and the failure each one is blamed for
The MOPD paper sorts prior work into four families and adds its own1. Table 1 reproduces its three axes (read from the paper's Table 1; the PDF text extraction scrambles the column order, so the symbols were checked against the rendered page) and adds the failure the authors name and where the route appears in the reports used in this module.
| Route | Dense signal | On-policy | Parallel specialists | Failure named by the MOPD paper | Appears in |
|---|---|---|---|---|---|
| Mix-RL | no | yes | no | cross-domain signals interfere ("see-saw"); joint model stays below per-domain teachers | one RL stage over mixed prompts: DeepSeek-V3.23; Qwen3's last RL stage4 |
| Cascade RL | no | yes | no | earlier skills decay as later stages run; a long run amplifies stability risk | Nemotron-Cascade2; Qwen3's four stages4 |
| Off-policy finetune | yes | no | yes | exposure bias: the student trains on text it would not itself generate | DeepSeek-V3.2 specialist distillation3 |
| Parameter merge | no | n/a | yes | fused model "typically unstable", fails to match every teacher | task arithmetic5; Llama 3 averages runs of one task6 |
| MOPD | yes | yes | yes | same-origin teachers needed (Section 4) | MiMo-V2-Flash, Kimi K3, Nemotron 3 Ultra, DeepSeek-V4 (Section 6) |
Three of the attributions in that last column are the MOPD paper's own: it cites the Qwen3 report for Mix-RL, Nemotron-Cascade for Cascade RL, and DeepSeek-V3.2 for off-policy finetuning1. They are fair labels for the shape of each pipeline but not for everything those reports do. Qwen3's flagship recipe is four sequential stages whose last RL stage covers more than 20 task types4 (§4.4), so it is a cascade ending in a mixed stage. DeepSeek-V3.2 trains specialists for six domains (plus writing and general question answering), distils their outputs into data, then runs one mixed GRPO stage that merges reasoning, agent and human-alignment training3 (§3). Section 6 returns to that report, because DeepSeek-V4 reversed it.
Parameter merging deserves one line of algebra, since it is the route that needs no training. A task vector is the fine-tuned weights minus the pre-trained weights, and task vectors can be combined by addition5 (abstract). For specialists \(\theta_1,\dots,\theta_N\) fine-tuned from a shared checkpoint \(\theta_0\):
$$\theta_{\text{merged}}=\theta_0+\sum_{i=1}^{N}\lambda_i(\theta_i-\theta_0),\qquad \lambda_i=\tfrac1N\;\Rightarrow\;\theta_{\text{merged}}=\tfrac1N\sum_i\theta_i .$$Plain weight averaging is task arithmetic with equal coefficients, so the merge baselines in the MOPD comparison (Section 3) are two settings of one family, and the coefficients are the knob. Merging has a long production record when the inputs are runs of the same task: Llama 3 averages "models obtained from experiments using various versions of data or hyperparameters" at the reward-model, SFT and DPO stages6 (§4.1.5). Merging specialists that moved in different directions under RL is a different regime, and the evidence in this module is that it is the least reliable one.
1.3 The see-saw in a published table
The decay that Table 1 attributes to sequential training is visible in a primary source. Qwen3-32B passes through reasoning RL (stage 2), thinking-mode fusion (stage 3) and general RL (stage 4); the report tabulates the thinking-mode scores after each4 (Table 22). General benchmarks rise and the two hardest reasoning benchmarks fall.
| Benchmark | Stage 2: reasoning RL | Stage 3: fusion | Stage 4: general RL | Net (points) |
|---|---|---|---|---|
| AIME'24 | 83.8 | 81.9 | 81.4 | -2.4 |
| LiveCodeBench v5 | 68.4 | 67.2 | 65.7 | -2.7 |
| LiveBench 2024-11-25 | 68.6 | 70.9 | 74.9 | +6.3 |
| Arena-Hard | 86.8 | 89.4 | 93.8 | +7.0 |
The drops are small, and stage 3 is supervised fusion rather than RL, so this is a weaker test of "cascade RL" than the MOPD paper's own (Section 3). What it shows is that Qwen3's authors accepted a measured regression on AIME'24 to buy breadth, which is the trade the integration literature is trying to remove.
2. MOPD: the mechanism
2.1 Three stages, one initialisation, routing by label
The MOPD paper structures post-training as three stages1 (§3.1). Stage 1 is general SFT on a corpus covering every target capability. Stage 2 trains one expert \(\pi_{\phi_d}\) per domain \(d\), each started from the Stage-1 checkpoint and each using "the RL recipe most natural for its domain"; the runs are independent and fully parallel. Stage 3 starts the student from the same Stage-1 checkpoint, freezes the experts as the teacher group, and trains on a multi-domain prompt set. Routing is a lookup: each trajectory is "dispatched to its corresponding domain teacher according to the task's domain". The paper describes no learned router and no mixture over teachers inside a prompt. Figure 2 steps through one iteration.
- Sample prompts. A batch comes from the multi-domain prompt set at a fixed domain ratio: math 0.35, instruction following 0.35, software engineering 0.30 in the Qwen3-30B-A3B runs1 (Appendix A).
- Student rolls out. The student generates one trajectory per prompt (\(N=1\)) from its own distribution and records its per-token probabilities.
- Route and prefill. The trajectory goes to the frozen teacher for the prompt's domain, which prefills it and returns its log-probability at every position. The teacher never decodes.
- Per-token advantage. The advantage at position \(t\) is the teacher-minus-student log-probability of the sampled token, stop-gradient, clipped to \(\pm A_{\max}\). The bar heights drawn here are illustrative, not measured.
- Update. An ordinary policy-gradient step with those advantages; the trainer is the existing PPO or GRPO stack with the advantage computation replaced1 (§3.2.1).
2.2 The objective, and why the advantage is a log-ratio
Write the prefix as \(s_t=(x,y_{\lt t})\). The per-token reverse KL to the routed teacher, averaged over the student's own trajectory, is
$$\mathcal L_{\text{rev-KL}}=\mathbb E_{x,\;y\sim\pi_\theta}\Big[\tfrac1{|y|}\sum_{t}\sum_{v}\pi_\theta(v\mid s_t)\log\frac{\pi_\theta(v\mid s_t)}{\pi_{\phi_d}(v\mid s_t)}\Big]$$(the paper's Eq. 11). Differentiate one token's term at a fixed prefix. Because \(\sum_v\nabla\pi_\theta(v)=0\), the second term of the product rule vanishes and
$$\nabla_\theta\sum_v\pi_\theta(v)\log\frac{\pi_\theta(v)}{\pi_T(v)}=-\,\mathbb E_{v\sim\pi_\theta}\Big[\log\frac{\pi_T(v)}{\pi_\theta(v)}\;\nabla_\theta\log\pi_\theta(v)\Big].$$That is a policy-gradient step whose per-token advantage is \(\hat A_t=\mathrm{sg}[\log\pi_T(y_t)-\log\pi_\theta(y_t)]\), which is the paper's Eq. 2 and 31. The final loss clips the advantage to \([-A_{\max},+A_{\max}]\) "for training stability" (Eq. 4), with a default \(A_{\max}=5\) in the Qwen3-30B-A3B experiments (Appendix A). A clip at 5 in log space is a probability ratio of \(e^5\approx148\), so it only touches tokens where teacher and student disagree by about two orders of magnitude. Two properties follow from the form. First, the sampled-token advantage is the negative of the \(k_1\) estimator of Module 6, so the cheap estimator is unbiased for the KL value and, as the display shows, for its gradient at a fixed prefix. Second, the estimator keeps only the token-level term and has no return-to-go sum over later tokens, the "discount zero" choice that Module 21 examines. The template (student samples, teacher grades each token) is the one Thinking Machines' on-policy distillation post describes7; what MOPD adds is the routing in Stage 3 and, more importantly, the way the teachers are produced in Stage 2.
2.3 What the reports changed
Every deployment keeps the sampled log-ratio but wraps it differently. Table 3 lists the variants; Module 12 covers the clip and ratio machinery and Module 14 the importance-sampling corrections.
| Report | Signal | Stabilisers | Batch and rollout |
|---|---|---|---|
| MOPD paper, Qwen3-30B-A3B1 | \(\mathrm{sg}[\log\pi_T-\log\pi_\theta]\) (Eq. 3); alternative top-\(k\) form with a bias-correcting term (Eq. 5) | two-sided clip \(\pm A_{\max}\), default 5; top-\(k\) with \(k=64\) | batch 2,048, \(N=1\), no dynamic sampling (App. A) |
| MiMo-V2-Flash8 | same log-ratio plus \(\alpha\hat A_{\text{ORM}}\), a GRPO outcome-reward advantage (Eq. 9) | training-inference importance weight, zeroed outside \([\epsilon_{\text{low}},\epsilon_{\text{high}}]\) (Eq. 7, 8) | not stated in the section read (§4.4) |
| Kimi K39 | \(\mathrm{clip}(\mathrm{sg}[\log\pi^{(d,e)}_{\text{teacher}}-\log\pi_\theta(\cdot\mid e,\dots)],\pm R_{\max})\) (Eq. 15); student sees effort \(e\) | clip \(R_{\max}\) (value not given) | not stated (§4.1.3) |
| Nemotron 3 Ultra10 | \(\mathrm{sg}[\ell^{T}-\ell^{\text{prox}}]\), then PPO clip on \(\pi_\theta/\pi_{\text{prox}}\) (Eq. 2, 3) | behaviour-to-proximal correction \(c_t\), IcePop mask, domain weights \(\lambda_i\) | 1,024 prompts, 1 rollout each; up to 192K tokens (§3.3.1) |
| GLM-511 | \(\mathrm{sg}[\log\pi^{\text{infer}}_{\text{teacher}}-\log\pi^{\text{train}}_\theta]\) replaces the GRPO advantage (Eq. 2) | GRPO loss with IcePop mask carried over from Eq. 1 | group size 1, batch 1,024 (§3.5) |
| DeepSeek-V412 | \(\sum_i w_i D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{E_i})\), full-vocabulary (Eq. 29) | none stated; exact KL kernel | not stated (§5.1.2) |
Two entries change the problem rather than the details. MiMo-V2-Flash adds the outcome-reward advantage to the teacher term by default8 (§4.4), so the student is pulled toward the teacher and toward verified correctness at once; its Figure 6 compares ORM-only RL, MOPD without outcome rewards, and MOPD on AIME 2025 and LiveCodeBench, and the report says MOPD "matches or exceeds the strongest teacher in most domains" without tabulating the three curves. GLM-5 and Nemotron set the group size or the rollouts per prompt to one, because "it is no longer necessary to maintain a large group of samples per prompt to estimate advantages; the advantage is computed directly from the gap with the teacher models instead"11 (§3.5). That is a structural saving against GRPO (Module 9): one rollout per prompt instead of \(G\).
2.4 Where the teacher runs
The only new operation in Stage 3 is the teacher prefill. The MOPD authors note that it "has the same nature as reward computation in RL", so they deploy each domain teacher as a stand-alone prefill service outside the trainer1 (§3.3). As soon as one student sequence finishes, the trainer fires an asynchronous request; the service returns per-token log-probabilities, and because prefill overlaps with other sequences still being sampled, "the teacher cost is essentially hidden behind student sampling" and contributed "nearly no measurable wall-clock overhead" in their deployments. Nemotron 3 Ultra pipelines rollout workers, teacher-scoring workers and learner workers the same way10 (§3.3.1), and GLM-5 fetches the teachers' logits through its inference engine11 (§3.5). The hiding claim depends on the teacher service keeping pace with sampling; Section 5 shows the cost when there are more than ten teachers, each potentially with trillions of parameters. How the trainer, rollout engines and teacher services are wired is System Design Modules 36 to 38.
3. The evidence: Qwen3-30B-A3B and MiMo-V2-Flash
3.1 The controlled comparison and its metric
The MOPD paper compares all five routes from one Stage-1 SFT checkpoint of Qwen3-30B-A3B-Base on three domains: math (AIME25 and AIME26, 32 samples per question), instruction following (IFBench and IFEval, one sample each) and software engineering (SWE-bench Verified, one run per task), at temperature 1.0 with no top-\(p\) or top-\(k\)1 (§4.1, Appendix B). The specialists use GRPO with dynamic sampling at learning rate \(3\times10^{-6}\): batch size 144 with \(N=8\) rollouts per prompt for math and IF (about 175K sequences), batch size 80 with \(N=8\) for SWE (about 150K sequences). Cascade RL runs them in the order IF, math, SWE. Mix-RL uses batch 256, \(N=8\) and a 0.35:0.35:0.3 mix (Appendix A).
Raw accuracy over-weights the domains with more headroom, so the paper normalises: \(\tilde s_d=(s_d-s^{\text{student}}_d)/(s^{\text{teacher}}_d-s^{\text{student}}_d)\), which is 0 at the Stage-1 student and 1 at the per-domain specialist, and reports the uniform mean over domains1 (§4.1).
Reproducing the headline 0.9373 from the paper's Table 2 (derived)
The paper does not say how a domain's accuracy is formed from its two benchmarks. Averaging them reproduces every printed score in that table, so that is what this module uses. Student domain accuracies: math \((45.42+54.48)/2=49.95\), IF \((42.69+84.17)/2=63.43\), SWE \(35.80\). Teachers: \(59.22\), \(86.95\), \(51.20\). MOPD: \((51.46+65.31)/2=58.385\), \((77.89+93.84)/2=85.865\), \(50.40\). Then
$$\tilde s_{\text{math}}=\tfrac{58.385-49.95}{59.22-49.95}=0.910,\quad \tilde s_{\text{IF}}=\tfrac{85.865-63.43}{86.95-63.43}=0.954,\quad \tilde s_{\text{SWE}}=\tfrac{50.40-35.80}{51.20-35.80}=0.948,$$mean \(0.937\), as printed. The same arithmetic gives 0.8818 (Mix-RL), 0.7752 (Cascade), 0.8241 (off-policy finetune), 0.3280 (averaged merge) and 0.8574 (task-arithmetic merge). Note that the SWE term rests on one run of one benchmark: a 0.8-point gap there is \(0.8/15.4\approx5\%\) of the SWE headroom.
3.2 What the comparison shows, and what it does not
The authors read four things off the paper's Table 2 and Figure 21 (§4.2). Mix-RL is the most balanced baseline (per-domain range 0.064) but still trails MOPD by 5.5 points overall, 0.882 against 0.937. Cascade RL in the order IF, math, SWE closes 98% of the IF headroom first, then only 57% of the math headroom, and the training curves show math accuracy falling during the SWE stage; the paper reads this as cross-domain interference and decay. Off-policy finetuning beats the IF teacher (1.01) yet closes only 65% of SWE. Parameter merging is "highly sensitive to the merging recipe": averaging gives 0.328, task arithmetic recovers to 0.857 but closes only 73% of math. MOPD's three per-domain scores sit in 0.91 to 0.95 and the authors report closing "91–95% of the student–teacher headroom on every domain".
Two limits matter before anyone treats this as settled. First, the sample-efficiency claim is about the integration stage alone: MOPD reaches the teacher-level plateau within about 25K IF samples and about 30K SWE samples, where Mix-RL "requires the full 150–180K sample budget in each domain"1 (§4.2). The three specialists were trained first, on roughly 175K, 175K and 150K sequences (Appendix A), about 500K in total (derived). If the paper's "samples" and "sequences" are comparable units, the specialists alone cost about what the whole Mix-RL run did, and MOPD's student stage sits on top. The paper reports no total compute, so the defensible claims are that the integration stage is cheap and that the specialists can be built in parallel, not that MOPD uses fewer rollouts end to end. Second, every number is the authors' own, from single runs with no intervals; the gaps of one or two points on IF and SWE in the paper's Table 2 are inside what a single run can move (Section 3.3 gives a scale).
3.3 MiMo-V2-Flash: the frontier deployment
The same recipe was applied to MiMo-V2-Flash, a 309B-parameter, 15B-active mixture-of-experts model8 (abstract), with teachers for math, code, instruction following, software engineering and tool use1 (§4.3). The MiMo report's own table is more complete than the MOPD paper's seven-column subset, and it tags what kind of model the "best teacher" is.
| Benchmark | Student before (%) | Best teacher | Teacher (%) | Student after (%) | After minus teacher (points) |
|---|---|---|---|---|---|
| AIME 2025 | 89.3 | RL | 93.9 | 94.1 | +0.2 |
| HMMT Feb. 2025 | 76.9 | RL | 82.6 | 84.4 | +1.8 |
| LiveCodeBench | 77.5 | RL | 82.6 | 83.2 | +0.6 |
| MMLU-Pro | 84.7 | Self | 84.7 | 84.9 | +0.2 |
| GPQA-Diamond | 84.9 | Self | 84.9 | 84.3 | -0.6 |
| HLE (no tools) | 21.2 | Self | 21.2 | 22.1 | +0.9 |
| Arena-Hard (hard prompt) | 50.0 | Self | 50.0 | 54.1 | +4.1 |
| Arena-Hard (creative writing) | 90.1 | Self | 90.1 | 86.2 | -3.9 |
| SWE-Bench Verified | 67.8 | RL | 74.2 | 73.4 | -0.8 |
| Tau2-Bench | 75.9 | RL | 79.6 | 80.3 | +0.7 |
| Tau2-Bench (Telecom) | 92.7 | RL | 95.0 | 95.3 | +0.3 |
| BrowseComp | 42.5 | SFT | 51.7 | 45.4 | -6.3 |
Read the table for what it says about the mechanism. On the six benchmarks with RL teachers, the student ends at most 0.8 points below the teacher and above it on five of the six: the integration stage works when the teacher is an RL specialist trained from the student's own checkpoint. On the five rows where the "teacher" is the student's own pre-MOPD checkpoint, the target is the student's old behaviour, yet the scores moved by between -3.9 and +4.1 points. That is a drift scale (derived from the table, not stated by the authors): one-point gaps in this table should not be read as signal, given single runs and the outcome-reward term MiMo adds to the loss (Table 3). The one large miss, BrowseComp, is a case where the best teacher was an SFT model and the student gained 2.9 points over its pre-MOPD self but stayed 6.3 below the teacher. The MOPD paper's own summary of the same run names two regressions, IFBench (-2.2) and SWE-Bench Verified (-0.8), "modest relative to the gains on the remaining benchmarks"1 (§4.3); its table is a seven-column subset that adds an IFBench column (student 55.4, teacher 68.9, MOPD 66.7) absent from the MiMo report's Table 7 and omits the MMLU-Pro, GPQA, HLE, Arena-Hard and BrowseComp rows.
4. Same-origin teachers, and what MOPD cannot import
4.1 A stronger teacher that breaks the run
The obvious variant is to use the best available teacher rather than a specialist derived from the student. The MOPD authors test it directly: in the math domain they replace the Qwen3-30B-A3B math teacher with Qwen3-235B-A22B, "a significantly larger model with stronger math capability", and keep everything else fixed1 (§4.4.2). Math gets worse under both loss forms.
| Math teacher | Loss | AIME25 | AIME26 | IFBench | IFEval | SWE-bench Verified | Norm. score |
|---|---|---|---|---|---|---|---|
| none (Stage-1 student) | n/a | 45.42 | 54.48 | 42.69 | 84.17 | 35.80 | 0.0000 |
| same-origin RL teacher | PG | 51.46 | 65.31 | 77.89 | 93.84 | 50.40 | 0.9373 |
| same-origin RL teacher | top-k | 51.77 | 64.79 | 75.85 | 93.07 | 50.20 | 0.9093 |
| Qwen3-235B-A22B | PG | 45.63 | 51.56 | 79.25 | 93.99 | 50.60 | 0.6003 |
| Qwen3-235B-A22B | top-k | 0.94 | 0.42 | 72.96 | 88.97 | 51.20 | -1.1898 |
The 235B teacher with the policy-gradient loss leaves AIME25 at 45.63, which is 0.21 above the SFT student's 45.42, and drops AIME26 to 51.56, below the student's 54.48. The paper's Figure 3 gives the mechanism1 (§4.4.2). The initial per-token student-teacher KL is about 0.19 with the external teacher against about 0.04 for the same-origin one, roughly five times as large. Under the policy-gradient loss, math accuracy "degrades progressively while entropy contracts from 0.30 to 0.21": the student "receives predominantly punitive gradient signal from the teacher's low-probability region" and narrows toward one mode. Under top-\(k\) the run diverges "around step 18" with wild oscillation in KL and entropy. With the same-origin teacher, KL falls monotonically from its low start and entropy holds near 0.30. The authors' conclusion is that same-origin teachers (each produced by domain RL from the very checkpoint that initialises the student) give "low initial KL divergence and smooth, stable optimization" (§3.1).
Two independent abstracts point the same way. Li et al. state that on-policy distillation works when "the student and teacher should share compatible thinking patterns" and "the teacher must offer genuinely new capabilities beyond what the student has seen during training"13 (abstract; the full paper was not read). Ko et al. say the method "remains prone to instability and negative transfer", frame the teacher-student log-likelihood ratio as a token reward, and stabilise it with mixture-based reward clipping14 (abstract). The advantage clip of Section 2 (\(A_{\max}=5\) in the MOPD paper) is related in spirit: both bound how hard a single token's teacher score can push the student.
4.2 When teachers are not same-origin: Nemotron 3 Ultra's warmup
Nemotron 3 Ultra did not hold the same-origin condition for many of its teachers, and reports what that cost. Its teachers were developed in parallel with the student, many through their own specialised SFT pipelines, and the authors found that "teacher models trained with substantially different training pipelines cannot be effectively combined through a straightforward MOPD merge"10 (§3.3.3). Their hypothesis: different SFT data produce different reasoning behaviours, so student rollouts are out-of-distribution for the teacher and its token-level scores become unreliable. The fix is a brief SFT of the student on data from the teacher's training distribution before MOPD, kept "intentionally limited in scale" so that regressions elsewhere are recovered during MOPD.
| Benchmark | RLVR student | With warmup | Without warmup | Teacher | Gap closed, with (%) | Gap closed, without (%) |
|---|---|---|---|---|---|---|
| GDPVal | 28.9 | 46.7 | 35.3 | 49.5 | 86.4 | 31.1 |
| BrowseComp | 31.0 | 44.4 | 33.0 | 51.0 | 67.0 | 10.0 |
| HLE (no tools) | 25.6 | 26.7 | 26.3 | 32.1 | 16.9 | 10.8 |
Warmup nearly triples the gap closed on GDPVal and multiplies it by 6.7 on BrowseComp, and does almost nothing on HLE. The report's own "Foundations for MOPD" paragraph lists the cleaner alternatives it did not run: a shared SFT stage before specialisation, or generating the student's SFT data from the specialists10 (§3.3.5). Both are ways of restoring same-origin.
4.3 The support argument
Figure 4 shows the recovery column of the report's main MOPD table, which the authors define as \((\text{MOPD2}-\text{RLVR})/(\text{Teacher}-\text{RLVR})\)10 (Table 5). All eleven printed values reproduce from the table's scores.
The report's reading is explicit. MOPD "is particularly effective when the teacher's advantage can be expressed as token-level preferences over trajectories that the student is already able to sample, such as tool-use decisions, environment interactions, abstention behavior, and multi-step execution patterns". It is weakest on HLE (16.9%) because the general-reasoning teacher gained through "additional large-scale SFT and RL on a separate reasoning mixture generated by DeepSeek-V4-Pro" that the student never saw, so the teacher's advantage "also comes from capabilities acquired through additional off-policy data exposure" and the student's rollouts are "effectively out-of-distribution for the teacher"10 (§3.3.4). In other words, a per-token log-ratio only grades tokens the student can emit. A skill that lives in reasoning paths the student almost never samples cannot be pulled in by scoring its own samples. That is the same boundary as the exposure-bias argument, seen from the teacher's side, and it is why the warmup helps agentic domains and not HLE. This module's inference, not the report's claim: it predicts that off-policy SFT on teacher rollouts (Table 1's third route) is the better tool for exactly those skills, with MOPD used afterwards to repair what that SFT disturbed.
4.4 Failure modes beyond teacher quality
A May 2026 follow-up names two more failure modes of vanilla MOPD, in a setting where a general-purpose teacher must restore broad ability while domain behaviour is preserved. The first is "recovery-preservation counteraction from mixing conflicting recovery and preservation gradients"; the second is "weak-signal flattening from uniformly averaging samples with unequal correction demand". Its remedy, CaMOPD, gives recovery dedicated updates, interleaves periodic preservation updates, and prioritises samples with larger average token-level teacher-student log-probability gaps15 (abstract only; role-play and medical QA, not the reasoning-and-agent setting of the frontier reports). The relevance here is the second failure: a batch mixing domains with very different correction demand averages the signal where it matters least, a relative of the dilution Nemotron raises against mixed RL (Section 6).
5. Sampled-token versus distribution-level distillation
5.1 What one sampled token costs in noise
The cheap estimator of Section 2 reads one number per position: the log-ratio at the token the student happened to sample. The alternative reads the teacher's whole next-token distribution and computes the exact per-position KL. The difference is easiest to see on a four-token vocabulary (constructed numbers, not from a report). At one prefix let the student be \(\pi_\theta=(0.70,0.20,0.05,0.05)\) and the teacher \(\pi_T=(0.50,0.40,0.09,0.01)\). The per-position reverse KL is \(0.148\) nats, and the advantage a sampled token would receive is \(\log\pi_T-\log\pi_\theta=(-0.34,+0.69,+0.59,-1.61)\), Figure 5.
Take the gradient of the loss with respect to the four logits. The exact value, \(\pi_\theta\odot(\log\pi_\theta-\log\pi_T-\mathrm{KL})\), is \((0.132,-0.168,-0.037,0.073)\), and averaging the single-sample estimator over the student's own draws reproduces it exactly. A single draw does not: the four possible estimates have norms 0.12, 0.74, 0.70 and 1.93, the total variance is 0.278, and the standard deviation (0.53) is 2.3 times the norm of the true gradient (0.23). A full-vocabulary computation removes the variance from the token draw at this prefix. Real batches average thousands of positions, and if the draws were independent the extra variance would shrink as \(1/\sqrt n\); whether the remainder matters is an empirical question the reports answer differently (Section 5.3).
5.2 DeepSeek-V4: full-vocabulary OPD
DeepSeek-V4 takes the exact route. Its report writes the objective as \(\sum_i w_i D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{E_i})\) over \(N\) experts, with \(w_i\) "typically determined by the relative importance of the expert", and says the unified policy "selectively learns from the specialized expert relevant to the current task context"12 (§5.1.2, Eq. 29). It does not say how \(w_i\) is set beyond that, so whether a prompt carries one nonzero weight or several is not specified. More than ten teachers are used. The stated reason for the full vocabulary: prior work reduces the KL to "a token-level KL estimate" reused as a per-token advantage; that is "resource-efficient" but "leads to high variance in gradient estimation and often causes training instability". The report gives no ablation numbers for this choice in the sections read, so it is a design claim, not a measured result.
The claim comes with an engineering bill. Materialising logits over a vocabulary above 100K for every teacher is "prohibitive, even when spooled to disk", so the system caches only each teacher's last-layer hidden states and rebuilds the logits at training time12 (§5.2.2). Figure 6 shows the path.
The arithmetic behind the cache is a ratio. V4 keeps a 128K vocabulary12 (§4.1) and has hidden size \(d=7{,}168\) in Pro and \(4{,}096\) in Flash (§4.2.1). Per token, a cached hidden state is \(d\) numbers and the logits are \(V\) numbers, so at equal precision caching saves a factor \(V/d=17.9\) for Pro and \(31.3\) for Flash, taking 128K as 128,000. The price is recomputing the head: \(2dV\) FLOPs per token, 1.84 GFLOP for Pro. Against a forward pass of about \(2N_{\text{active}}\) FLOPs per token (a standard approximation that ignores attention; 49B active parameters for Pro and 13B for Flash12, §4.2.1), that is 98 GFLOP, so the head is about 1.9% of a Pro-sized teacher forward and 4.0% of a Flash-sized one (derived). That supports the report's "negligible recomputation overhead". The calculator exposes the knobs; the cache precision is not stated in the report, so it is an input.
Per-sequence bytes for one teacher: full logits versus cached hidden states
Formulas: \(n\,V\,b_\ell\) against \(n\,d\,b_h\). \(V\) and \(d\) are the DeepSeek-V4 values12 (§4.1, §4.2.1); the report does not state the cache dtype or whether logits would be BF16 or FP32, so both are inputs.
At the 32,768-token maximum sequence length of the MOPD paper's math and IF data1 (Appendix A), with BF16 logits, one teacher's logits for one sequence are 8.39 GB against 0.47 GB of Pro-sized hidden states. At a million tokens the same BF16 logits are 268 GB per teacher per sequence (derived), and V4 serves more than ten teachers; that is why sorting samples by teacher index and holding one head at a time matters.
5.3 Who tested the alternatives
The data movement is also the MOPD authors' argument for their top-\(k\) variant: it "keeps the teacher-prefill payload small enough to be transmitted like a lightweight reward signal, in contrast to full-vocabulary distillation which requires shipping the entire vocabulary distribution per token"1 (§3.2.2). Table 7 puts numbers on the three payloads and collects what each report found when it tried more than one.
| Objective | Payload per token | Per 32,768-token sequence | Reported comparison |
|---|---|---|---|
| Sampled token | 4 B | 0.13 MB | MOPD paper default (norm. 0.937)1; used by MiMo, K3, Nemotron and GLM-5 (Table 3) |
| Top-64 | 512 B | 16.8 MB | MOPD paper: 0.909, "comparable on Math and slightly worse on IF and SWE", and no added stability with same-origin teachers1 (§4.4.1). K3: "no clear advantage in either convergence speed or final performance"9 (§4.1.3) |
| Top-k or full vocabulary ("logit matching") | 512 B to 256 kB | 16.8 MB to 8.39 GB | Nemotron 3 Ultra: logit matching "did not improve MOPD performance and consistently underperformed the sampled-token objective on some agentic benchmarks such as Terminal Bench"10 (§3.3.5) |
| Full vocabulary | 256 kB | 8.39 GB | DeepSeek-V4 adopts it for stability12; no ablation numbers in the sections read |
Nemotron offers a mechanism for why more information could hurt: "full-distribution matching may impose an overly strong local constraint on prefixes sampled from the student policy, particularly when those prefixes have limited support under the teacher distribution", where teacher logits "can become poorly calibrated or less informative"10 (§3.3.5). That is the same off-support problem as Section 4. This module's reading: exactness removes sampling noise, not teacher error, so if the teacher's distribution is unreliable off its own support, matching all of it fits that error more faithfully. The report calls its explanation a hypothesis and the question open. The MOPD paper's top-\(k\) numbers support the narrower statement that with same-origin teachers, the extra information bought nothing measurable in their setting. DeepSeek-V4 is the only report on the other side, with more than ten teachers and no published ablation.
6. The recipes side by side
6.1 Seven pipelines on one axis
Figure 7 places each report's integration design in report-date order. Read it for one thing: where the RL happens relative to the distillation, and whether the teachers are produced in parallel (Xiaomi, DeepSeek-V4, Nemotron 3 Ultra, Kimi K3) or in sequence (GLM-5, Nemotron-Cascade 2). Lambert's survey identifies MiMo-V2-Flash as the first technical report to document this process17 (§3.2.4). Per-report recipe detail is Modules 28 (Kimi), 30 (DeepSeek) and 32 (GLM-5, Nemotron).
6.2 DeepSeek reverses itself
The cleanest before-and-after in the literature is inside one lab. DeepSeek-V3.2 distils specialists into data and then merges "reasoning, agent, and human alignment training into one RL stage", a design said to balance performance "while circumventing the catastrophic forgetting issues commonly associated with multi-stage training paradigms"; the same report says models trained on the distilled specialist data landed "only marginally below" the specialists, with "the performance gap … effectively eliminated through subsequent RL training"3 (§3). Its mixed stage rewards reasoning and agent tasks with rule-based outcomes plus a length penalty and a language-consistency reward, and general tasks with a rubric-based generative reward model. DeepSeek-V4 says the pipeline "largely mirrored" V3.2 but that "the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation"12 (§5.1). Each specialist still gets SFT then GRPO, keeping "hyper-parameters closely aligned with our prior research" (§5.1.1). The claimed benefit is that logits-level alignment is "practically circumventing the performance degradation often encountered in traditional weight-merging or mixed RL techniques" (§5.1.2).
Two cautions. V4 reports no V3.2-style mixed-RL control, so the replacement is a decision the report states, not an ablation it shows (in the sections read). And V3.2's own claim, that one mixed stage avoids the forgetting of multi-stage training, is the opposite diagnosis from the one MOPD makes of mixed RL, so the two reports differ on what mixed RL does as well as on what to replace it with. Neither contains a matched-compute comparison between the two.
6.3 Distilling from your own earlier checkpoints
GLM-5 does not train its teachers in parallel. It runs SFT, then Reasoning RL, Agentic RL and General RL in sequence, and finishes with "on-policy cross-stage distillation" in which "the final checkpoints from the preceding training stages serve as teacher models", with prompts "sampled from the corresponding teachers' RL training sets and mixed in appropriate proportions"11 (§3.5). The stated motive is the cascade failure of Table 1: "sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities". The loss replaces the GRPO advantage with the teacher-student log-ratio (Eq. 2), uses a group size of 1 and a batch of 1,024, and fetches teacher logits through the inference engine. Because each teacher is an earlier checkpoint of the same run, the same-origin condition of Section 4 is satisfied by construction, but no parallelism is gained: the stages are still serial. Nemotron-Cascade 2 does something similar, running "multi-domain on-policy distillation from the strongest intermediate teacher models for each domain" throughout its Cascade RL to "recover benchmark regressions"16 (abstract only; schedule and numbers not read).
The evidence on whether cascades decay is split. The MOPD paper's cascade lost math accuracy during its SWE stage (Section 3.2). The original Nemotron-Cascade paper says that "subsequent domain-wise RLVR stages rarely degrade the benchmark performance attained in earlier domains and may even improve it", after an RLHF pre-step2 (abstract). The orders differ (IF, math, SWE against an RLHF-first order), the domains differ, and the abstract does not give numbers, so these are not contradictory measurements. They do show that cascade decay depends on the order and the pair of domains, which is what makes it hard to plan around.
6.4 How experts are defined and routed
| Report | Experts | What defines an expert | Routing and weighting |
|---|---|---|---|
| MOPD paper (Qwen3-30B-A3B)1 | 3 | domain: math, instruction following, SWE | by the task's domain label; batch mix 0.35:0.35:0.3 (§3.1, App. A) |
| MiMo-V2-Flash8 | not stated | agentic (search, coding, tool use) and non-agentic (math, general reasoning, safety); teacher may be RL, SFT or the student itself | domain-specific teachers; outcome-reward advantage added (§4.1, §4.4) |
| GLM-511 | not stated | an earlier stage's final checkpoint | prompts from each teacher's RL set, mixed in proportions (§3.5) |
| DeepSeek-V412 | more than 10 | domain, "such as mathematics, coding, agent, and instruction following" | weights \(w_i\) "by the relative importance of the expert" (Eq. 29) |
| Nemotron 3 Ultra10 | more than 10 | domain (SWE, office work, search, terminal use, conversational tool use, usability, ...); the RLVR student is the self-teacher for uncovered areas | per-domain prompt set \(\mathcal D_i\), weights \(\lambda_i\); second round adds teachers branched from MOPD 1 (§3.3, Fig. 10) |
| Kimi K39 | 9 | domain (general, general agents, coding agents) crossed with effort (low, high, max) | teacher chosen by \((d,e)\); the student is conditioned on \(e\) (Eq. 15) |
Kimi K3's grid makes effort an explicit axis of the expert definition, and its nine experts come from running the budget-controlled RL of Module 23 per domain. DeepSeek-V4 also trains specialists "under divergent RL configurations" for its three reasoning-effort modes12 (§5.1.1) but does not say how those map onto its distillation teachers. The general expert's reward for non-verifiable tasks is the agentic judge that the report describes only as retaining "the tournament-style group reward with binary comparisons as in Kimi K2.5"9 (§4.1.2); how that aggregation works is not given there, and Module 16 records that the K2.5 text does not use the word. The report also says expert trajectories at all effort levels are "jointly collected for supervised fine-tuning and multi-teacher on-policy distillation" without placing the SFT in the order.
Nemotron states the argument for experts at all: with unified RLVR, "as the number of environments continues to grow, each domain contributes only a relatively small number of samples to any given training batch, diluting the per-domain learning signal and making it increasingly difficult to balance training across domains"10 (§3.3). The arithmetic is a share. Its unified run lists fifteen items (terminal use through model usability) and reports a global batch of 8,192 with 16 rollouts per sample, which the report does not specify as prompts or rollouts, so no per-domain count is derived here. An even split would give each listed item 1/15 of whatever the batch counts (an illustration only: the report mixes data with a Gaussian-based curriculum). A specialist run trains one domain on its whole batch. The MOPD student batch is also mixed, but each token carries a teacher signal rather than one outcome per rollout, so a domain's gradient does not wait for a rare success.
6.5 Second rounds, and agentic rollouts
Both Nemotron 3 Ultra (two MOPD iterations, Figure 10) and the MOPD paper run a second round: retrain each teacher from the first-round student, then distill again. In the MOPD paper the second-round teachers score 1.030 on the paper's metric (above the original specialists) against the first-round student's 0.937, and the second-round student reaches 0.986 (+0.049). Only math and IF were retrained; SWE moved from 50.40 to 50.20 with no SWE training1 (§4.4.3, Table 5; both normalised scores reproduce from the table's accuracies). The iteration is what makes the cost of Section 7 recurrent rather than one-off.
For agentic domains, Nemotron states a practical limit: mixing end-to-end agentic environments with reasoning environments in MOPD, "we observed substantial training inefficiency because rollout times can differ dramatically", so "we use single-turn rollouts, similar to PivotRL … for most of the agentic tasks"; whether end-to-end rollouts give more is "an open area for exploration"10 (§3.3.5). The turn-local training that this relies on is Module 18's Section 5; the Kimi K3 report instead keeps partial rollouts with distillation in the same RL framework, noting that the dense reward "naturally enabl[es] infrastructure-level optimizations such as partial rollout training for long-horizon tasks"9 (§4.1.3).
7. What each route costs
None of the reports gives total GPU-hours for an integration route, so cost here is a set of derived counts and structural facts, each labelled. The sequence counts come from the MOPD paper's setup (Appendix A): the three specialists consumed about 175K, 175K and 150K sequences, 500K in all. Figure 2's axes run to roughly 175K math, 175K IF and 150K SWE samples, which at Mix-RL's 0.35:0.35:0.3 mix is the same total (\(175/0.35=150/0.30=500\) thousand; derived from the axes), and the Figure 3 runs last 60 steps at batch 2,048 with one rollout per prompt, about 123K student sequences if a step is one batch (derived). So the student stage adds up to roughly a quarter to the sequence count of a specialist build. It is also an upper bound on student sequences only: sequence length differs by domain (SWE runs to 65,536 tokens over up to 50 turns), and the teacher prefill adds a forward pass the RL baselines do not have. Under the usual \(2N\) (forward) and \(6N\) (forward and backward) FLOPs-per-token approximation of Module 1, a teacher of the student's own size adds one third of the student's update FLOPs per token (derived, attention ignored), which the authors hide behind sampling by running prefill as an asynchronous service1 (§3.3).
| Route | Training runs | Wall-clock shape | Extra infrastructure | A restart costs | Evidence |
|---|---|---|---|---|---|
| Mix-RL | 1 pooled run | serial | one verifier per domain in one trainer | the whole run | controlled, authors' own: 0.882 |
| Cascade RL | N runs in sequence | sum of stages | none | every later stage | controlled, authors' own: 0.775; split elsewhere (Section 6.3) |
| Off-policy finetune | N specialists, then SFT | max of specialists, then SFT | teacher sampling | one specialist | controlled: 0.824; V3.2 pairs it with RL3 |
| Parameter merge | N specialists, no training after | max of specialists | none | one specialist | controlled: 0.328 to 0.857 by recipe |
| MOPD | N specialists, then one student pass | max of specialists, then student pass | teacher prefill service per teacher | one specialist | controlled: 0.937; four frontier reports, no matched ablation |
Three decision inputs follow from the evidence. (1) The number and heterogeneity of domains. At three domains with similar rollout shape, mixed RL was 5.5 points behind on a metric the MOPD authors defined; at the dozen-plus environments of Nemotron 3 Ultra, the report's reason for specialists is per-domain dilution10. The sources do not locate the crossover. (2) Whether the teachers share the student's origin. The paper's diagnostic is the initial per-token reverse KL, about 0.04 against 0.191 (§4.4.2); measuring it on a few hundred student rollouts before committing to a teacher is this module's suggestion, not a procedure any report states. (3) Whether the missing skill is something the student can already sample. If not, the HLE row of Figure 4 says the dense signal has little to grade, and the report's alternatives (a shared SFT stage, or SFT on specialist outputs first) apply.
For an inference engineer the new workload is the teacher service: a prefill-only job that scores completed sequences and returns a log-probability at every position, at up to 192K tokens per sequence in Nemotron's runs10 (§3.3.1). That shape (long prefill, per-position outputs, no decode) differs from both a serving engine and an RL rollout engine. This is an inference from the reports' descriptions, not a statement from them; infrastructure choices are System Design Modules 36 to 38.
Where experts disagree
The 2026 reports do not agree on how to combine skills, and Lambert's survey says so directly: MOPD "is a popular new tool, but labs are far from unanimous in its usage"17 (§3.2.4). The disagreement has four live positions. Only one controlled comparison exists (the MOPD authors', on three domains), and none of the frontier reports published here includes a matched-compute ablation against the alternative it replaced.
A. Keep one mixed RL stage
DeepSeek-V3.23 (§3); Qwen3's final general-RL stage4; Nemotron 3 Ultra's own first stage is a unified RLVR run10
One on-policy run over all domains, which V3.2 says "balances performance across diverse domains while circumventing the catastrophic forgetting issues" of multi-stage pipelines.
- Simplest to operate: one trainer, one recipe, no teacher fleet.
- Most balanced of the non-distillation baselines in the controlled test (range 0.064, score 0.882)1.
B. Sequential stages, repaired by distilling from earlier checkpoints
GLM-511 (§3.5); Nemotron-Cascade and Cascade 2216 (abstracts)
Accept that stages interfere and fix it at the end, with the model's own earlier checkpoints as teachers.
- Same-origin by construction; no parallel teacher fleet to build.
- The original Cascade RL paper reports later stages "rarely degrade" earlier domains2.
C. Parallel specialists, sampled-token MOPD
MOPD paper1; MiMo-V2-Flash8; Kimi K39; Nemotron 3 Ultra10
Train each skill to its peak independently, then distill with one log-ratio per token on the student's own rollouts.
D. Parallel specialists, full-vocabulary distillation
DeepSeek-V412 (§5.1.2, §5.2.2)
The sampled-token estimate is resource-efficient but "leads to high variance in gradient estimation and often causes training instability", so use the exact KL over the whole vocabulary.
- Removes sampling variance (the toy example's 2.3x noise-to-signal at one prefix).
- Engineering shown to work with more than ten teachers.
- No ablation published; Nemotron's logit matching underperformed on Terminal Bench10.
- Needs hidden-state caching, per-teacher head loading and a custom KL kernel.
The choice turns on three inputs: how many domains there are and how different their rollouts are, whether the teachers can be made same-origin, and whether the missing skill lies inside the student's own sampling distribution. With few, similar domains, mixed RL is within reach of MOPD (5.5 points on the authors' metric) at far lower engineering cost, so position A is defensible there. With many environments including long agentic ones, four independent reports (Xiaomi, Moonshot, NVIDIA, DeepSeek) chose specialists, and the dilution argument is mechanistically sound. Position B is a reasonable compromise when parallel teams are not available. Between C and D the evidence favours C as the default and treats D as unproven: three reports tested richer signals and found no gain, and the one report that adopted them published no comparison. What would settle it: a matched-GPU-hour comparison, with several seeds, of continued mixed RL, cascade-plus-repair, sampled-token MOPD and full-vocabulary OPD from one SFT checkpoint at ten or more domains, including at least two agentic ones. That experiment does not exist in the sources read.
What is changing
Four items are open. First, the same-origin requirement is being worked around: Nemotron's warmup SFT and its proposed shared SFT stage, Ko et al.'s relaxed imitation through reward clipping14, and Li et al.'s off-policy cold start with teacher-aligned prompt selection13 (abstracts) all try to make distillation tolerate a teacher that differs from the student. Second, whether end-to-end agentic rollouts beat single-turn ones under distillation is untested (Section 6.5). Third, the naming is unsettled: DeepSeek-V4 calls its stage "multi-teacher On-Policy Distillation (OPD)" and GLM-5 "on-policy cross-stage distillation"1112, while Lambert groups both under MOPD17; the term is a family, not a spec. Fourth, the iterative loop (retrain specialists from the last student, then distill again) is measured for one extra round in the MOPD paper1 and for two MOPD iterations in Nemotron 3 Ultra10 (Table 5), so how many rounds keep paying is unknown. The MOPD paper is a June 2026 preprint, v1; its numbers come from the authors' own runs.
References
- paperMa, W., Wei, J., Zhao, L., Zhang, H., et al. (Xiaomi LLM Core; Peking University). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-26. Used for: Table 1 axes (checked on the rendered page), §3 objective and infrastructure, Tables 2 to 5, Appendix A hyperparameters, §4.4 analyses, §5 workflow argument.
- paperWang, B., Lee, C., Lee, N., Lin, S.-C., Dai, W., Chen, Y., Chen, Y., Yang, Z., Liu, Z., Shoeybi, M., Catanzaro, B., Ping, W. Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models. arXiv:2512.13607, v1 15 Dec 2025, v2 27 Mar 2026. https://arxiv.org/abs/2512.13607. Accessed 2026-10-27. Used for: abstract only (cross-domain heterogeneity, Cascade RL, "rarely degrade"); full text not read.
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: §3 specialist distillation and mixed RL training, reward composition.
- paperYang, A., et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: §4 four-stage pipeline, §4.4 General RL task list, Table 22 stage-by-stage scores.
- paperIlharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., Farhadi, A. Editing Models with Task Arithmetic. ICLR 2023; arXiv:2212.04089. https://arxiv.org/abs/2212.04089. Accessed 2026-10-27. Used for: task-vector definition and addition (abstract only).
- paperLlama Team, AI @ Meta. The Llama 3 Herd of Models. arXiv:2407.21783v3. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: §4.1.5 model averaging across data and hyperparameter variants.
- blogLu, K. and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 27 Oct 2025 (page notes a June 2026 update). https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-26. Used for: the student-samples, teacher-grades-each-token template (single-teacher treatment is Module 21).
- paperXiaomi LLM-Core. MiMo-V2-Flash Technical Report. arXiv:2601.02780v2, 8 Jan 2026. https://arxiv.org/pdf/2601.02780v2. Accessed 2026-10-27. Used for: §4.1 three-stage MOPD, Table 7 (12 benchmarks), §4.4 Eq. 5 to 9, Figure 6.
- paperKimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: §4.1.2 nine experts and agentic GRM sentence, §4.1.3 MOPD reward (Eq. 15) and top-k remark.
- paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: §3.2 to §3.3 (unified RLVR, MOPD, warmup, Tables 4 and 5, limitations), Figure 10.
- paperGLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: §3 pipeline, §3.5 on-policy cross-stage distillation (Eq. 2).
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1 (PDF stamp 26 Apr 2026). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: §5.1 specialists and OPD (Eq. 29), §5.2.2 teacher scheduling, §4.1 and §4.2.1 vocabulary and hidden sizes.
- paperLi, Y., Zuo, Y., He, B., Zhang, J., Xiao, C., Qian, C., Yu, T., Gao, H., Yang, W., Liu, Z., Ding, N. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 14 Apr 2026. https://arxiv.org/abs/2604.13016. Accessed 2026-10-27. Used for: abstract only (conditions for success, cold start).
- paperKo, J., Abdali, S., Kim, Y. J., Chen, T., Cameron, P. Scaling Reasoning Efficiently via Relaxed On-Policy Distillation. arXiv:2603.11137, 11 Mar 2026. https://arxiv.org/abs/2603.11137. Accessed 2026-10-27. Used for: abstract only (instability, log-ratio as token reward, reward clipping).
- paperChen, T., Ou, J., Liu, Z., Tang, R., Liang, J., Li, H. Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation. arXiv:2605.27115, v1 26 May 2026, v2 14 Sep 2026. https://arxiv.org/abs/2605.27115. Accessed 2026-10-27. Used for: abstract only (two failure modes of vanilla MOPD, CaMOPD).
- paperYang, Z., et al. (NVIDIA). Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. arXiv:2603.19220, v1 19 Mar 2026, v2 22 Mar 2026. https://arxiv.org/abs/2603.19220. Accessed 2026-10-27. Used for: abstract only (distillation from the strongest intermediate teachers inside a cascade).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: §3.2.4 MOPD recipe shift, expert counts, "far from unanimous".