Reinforcement Learning for LLM Post-Training · Module 21
On-Policy Distillation as Dense-Reward RL: Per-Token Reverse KL, Cost Arithmetic, and When It Beats RL
On-policy distillation (OPD) keeps the part of RL that matters for a small model, training on its own samples, and replaces the one scalar reward per episode with a teacher's verdict on every token. This module derives the objective and its gradient, reads the two reported cost comparisons against RL (Thinking Machines and Qwen3) with their arithmetic and their gaps, and collects what later papers found about when the recipe fails. Afterwards you can implement OPD on top of an RL trainer and say, for a given teacher and budget, whether distillation or RL is the cheaper route.
1. Two axes: whose trajectories, and how dense the signal
Post-training methods differ along two axes that are easy to conflate. One is where the sequences come from: a fixed dataset or a teacher's samples (off-policy), or the model being trained (on-policy). The other is how much of each sequence receives a training signal: one number for the whole response, or a number per token. Thinking Machines' post, which is the reference DeepSeek-V4 cites for OPD2, uses exactly this grid1. This module is about the cell that SFT and RL each cover half of.
1.1 The taxonomy and what each row optimizes
| Method | Who samples | Signal | Objective on the student | Known weakness |
|---|---|---|---|---|
| SFT / off-policy distillation | dataset or teacher | dense: teacher's token at every position | forward KL \(\mathrm{KL}(\pi_T\|\pi_\theta)\) on teacher-sampled prefixes (cross-entropy differs from it by the constant \(H(\pi_T)\)) | trains on states the teacher visits, not the student1 |
| RL (RLVR, RLHF) | student | sparse: one reward per episode | maximize expected reward, with the credit machinery of Modules 2 and 9 | "a fixed number of bits per training episode regardless of the number of tokens used"1 |
| On-policy distillation | student | dense: teacher's verdict on every sampled token | reverse KL \(\mathrm{KL}(\pi_\theta\|\pi_T)\) along the student's own samples | needs a teacher whose distribution the student can reach (section 5) |
The distinction in the last column of the first row is the reason the field cared. Cross-entropy on teacher samples is the forward KL up to a constant that depends only on the fixed teacher3; the student is graded on the teacher's prefixes. In the OPD row the expectation is taken under the student, which is the one-line change in the objective and, as section 2 shows, the reason it can reuse an RL trainer.
1.2 Exposure bias and the DAgger bound
The off-policy student never sees its own mistakes. The post's description: "if the student makes an early mistake that the teacher never makes, it finds itself diverging ever farther from the states it observed in training"1. Imitation learning has a formal version of this. Ross, Gordon and Bagnell show that a learner whose per-step 0-1 loss is \(\epsilon\) on the expert's state distribution can cost \(J(\pi)\le J(\pi^*)+T^2\epsilon\) over a horizon of \(T\) steps (their Theorem 2.1), and that DAgger, which labels the states the learner itself visits, reaches \(J(\hat\pi)\le J(\pi^*)+uT\epsilon_N+O(1)\) (Theorem 3.2), where \(\epsilon_N\) is a loss measured under the learner's own state distribution4. For equal \(\epsilon\) and \(u=O(1)\) the ratio of the two bounds is \(T\): at 4,000 tokens the quadratic bound is 4,000 times the linear one (derived).
1.3 Lineage
- DAgger
The student acts, the expert labels the states the student reached, and the student retrains on the aggregate dataset4.
- GKD and MiniLLM
Generalized Knowledge Distillation trains the student on its own samples with teacher probabilities, does "not backpropagate through the student's sampling distribution", mixes on-policy and dataset sequences through a student-data fraction \(\lambda\), and allows other divergences such as reverse KL and generalized JSD; it reports relative gains over baseline distillation of 2.1× on summarization, 1.7× on translation and 1.9× on arithmetic reasoning, averaged across T5 student sizes5 (ICLR 2024). MiniLLM minimizes the sequence-level reverse KL with a policy gradient and three stabilizers, covered in section 2.56.
- Qwen3 strong-to-weak
Small Qwen3 models get an off-policy distillation phase and then an on-policy phase that aligns student logits with a teacher7 (§4.5).
- Thinking Machines post
OPD recast as an RL-shaped objective: advantage equals negative per-token reverse KL, so an RL trainer runs it with one changed line1.
- OPD as a final stage
DeepSeek-V4's mixed-RL stage is "entirely replaced" by OPD2 (§5.1) and the MOPD paper integrates domain teachers into one student by the same mechanism8. Module 22 covers the multi-teacher case.
Two facts matter for what follows. GKD uses the teacher's full next-token distribution at every visited prefix and back-propagates it directly5, while the Thinking Machines recipe uses only the sampled token's log-ratio as a reward1; section 5.2 returns to the cost of that choice. And the Qwen3 report describes its on-policy phase as fine-tuning "by aligning its logits with those of a teacher model (Qwen3-32B or Qwen3-235B-A22B) to minimize the KL divergence"7 (§4.5), without stating the KL direction or the estimator, so the two headline results in section 4 come from related but not identical objectives.
2. The objective as a reward in an RL trainer
2.1 Per-token reverse KL along the student's sample
Write \(c_t=(x,y_{<t})\) for the context at position \(t\). The sequence-level reverse KL from student to teacher splits exactly, by the chain rule, into per-token terms evaluated on contexts the student generated3 (§12.3.2, Eq. 136):
$$ \mathrm{KL}\big(\pi_\theta(\cdot|x)\,\|\,\pi_T(\cdot|x)\big)=\mathbb{E}_{y\sim\pi_\theta}\sum_{t=1}^{T}\mathrm{KL}_t,\qquad \mathrm{KL}_t=\sum_{v}\pi_\theta(v|c_t)\log\frac{\pi_\theta(v|c_t)}{\pi_T(v|c_t)}. $$The Thinking Machines recipe estimates each \(\mathrm{KL}_t\) from the one token the student sampled, uses it as a negative reward, and sets the discount factor to zero: "the student only optimizes the immediate next token, with no consideration for future tokens". The post adds that a positive discount is "more mathematically correct" but that "we do not find discount factors > 0 to improve performance in practice"1. The per-token advantage is the negative sampled log-ratio:
$$ A_t=\log\pi_T(y_t|c_t)-\log\pi_\theta(y_t|c_t). $$Nothing else is needed from the environment. No verifier, no reward model, no critic: the teacher's log-probabilities on the student's tokens are the entire reward. The post notes two consequences: a rollout need not finish before it is scored, so partial or shorter rollouts work, and the teacher is queried with a single forward pass while the cheaper student does the generating1.
2.2 The gradient, and what discount zero drops
At a fixed context the policy-gradient identity applies position by position. Because \(\mathbb{E}_{v\sim\pi_\theta}[\nabla\log\pi_\theta(v)]=0\), the constant in \(\nabla(\pi\log\pi)\) vanishes and
$$ \nabla_\theta\mathrm{KL}_t=\mathbb{E}_{y_t\sim\pi_\theta}\Big[\big(\log\pi_\theta(y_t)-\log\pi_T(y_t)\big)\nabla_\theta\log\pi_\theta(y_t)\Big]=-\,\mathbb{E}\big[A_t\,\nabla_\theta\log\pi_\theta(y_t)\big]. $$So an RL trainer that multiplies \(\nabla\log\pi_\theta(y_t)\) by \(A_t\) follows an unbiased estimate of \(-\nabla\mathrm{KL}_t\) at that context (derived; Module 6 section 3.3 shows the same identity for the \(k_2\) estimator). What it does not follow is the gradient of the whole sum, because the token \(y_t\) also changes which contexts \(c_{t+1},\dots\) the student visits next, and so changes their KL terms. The estimator that keeps this dependence uses the return-to-go. Fu et al. define the interpolation9 (Eq. 1 to 3), with \(r_t=\log\pi_\theta(y_t|c_t)-\log\pi_T(y_t|c_t)=-A_t\) and \(s_t=\nabla_\theta\log\pi_\theta(y_t|c_t)\):
$$ \hat g_\gamma=\sum_{t=1}^{T}\Big(\sum_{t'=t}^{T}\gamma^{t'-t}\,r_{t'}\Big)s_t,\qquad \gamma\in[0,1]. $$With \(\gamma=0\) the inner sum collapses to \(r_t\) alone, so no return-to-go over later tokens enters the weight on \(s_t\); this is the token-level estimator, which is what the Thinking Machines recipe computes; with \(\gamma=1\) it is the causal sequence-level policy gradient, which is what MiniLLM derives (its Eq. 2, with reward-to-go \(R_t\))6. The two cost the teacher the same payload, one log-probability per sampled token. They differ in bias and variance. Fu et al. show that under bounded rewards and gradients the worst-case variance bound of the token-level estimator grows as \(O(T^2)\) in the response length and that of the sequence-level estimator as \(O(T^4)\), and their toy experiment shows variance rising with \(\gamma\)9 (§2.1, Appendix D). Tang and Munos find the opposite ordering on a different metric: distilling a Llama 3.1 70B policy into an 8B one on 7,500 MATH prompts with \(n=4\) samples per gradient estimate, a properly implemented sequence-level estimator lowered the sequence-level KL faster than the best token-level one, and leave-one-out and cumulative variance reduction lowered it further10 (§5.2 to §5.3, Figure 8). The first result is about gradient noise and stability, the second is about how fast the KL objective itself falls. Neither measures downstream accuracy, so they do not contradict each other; the recent recipes in Table 2 all keep supervision per token (section 5.2).
| Form | Per-token signal | Teacher payload | Used or analyzed in |
|---|---|---|---|
| Sampled token, \(\gamma=0\) | \(A_t=\log\pi_T(y_t)-\log\pi_\theta(y_t)\) | one log-probability | Thinking Machines1; MOPD's policy-gradient form, with the advantage clipped to \([-A_{\max},A_{\max}]\)8 (Eqs. 3 and 4) |
| Sampled token, \(\gamma=1\) | \(\sum_{t'\ge t}A_{t'}\) (reward-to-go) | one log-probability | MiniLLM, with single-step decomposition, teacher-mixed sampling and length normalization6; Tang and Munos10 |
| Top-\(k\) of the teacher | reverse KL over \(k\) tokens, renormalized or bias-corrected | \(k\) token ids and \(k\) log-probabilities | MOPD, \(k=64\) in its comparison8 (§3.2.2, Eq. 5; §4.4.1); Fu et al.'s local support matching9 |
| Full vocabulary | exact \(\mathrm{KL}_t\) | \(|V|\) values, or last-layer hidden states to rebuild them | GKD5; DeepSeek-V42 (§5.1.2); Qwen3 describes "aligning its logits"7 |
2.3 A worked trajectory
Table 3 is a constructed eight-token response, scored by a hypothetical teacher. It mirrors the illustration in the Thinking Machines post, where the teacher penalizes the tokens that start the wrong approach and does not penalize the final wrong answer, "though wrong, isn't penalized — it is entirely predictable conditional on the whole preceding sequence"1. The probabilities are the probabilities each model assigns to the token the student sampled, in the same context; \(G_t\) is the \(\gamma=1\) return-to-go. Values are computed from the probabilities (derived).
| \(t\) | token | \(\pi_\theta(y_t)\) | \(\pi_T(y_t)\) | \(A_t=\ln(\pi_T/\pi_\theta)\) | \(G_t\) |
|---|---|---|---|---|---|
| 1 | The | 0.40 | 0.38 | −0.051 | −2.470 |
| 2 | ice | 0.55 | 0.60 | +0.087 | −2.419 |
| 3 | cubes | 0.90 | 0.92 | +0.022 | −2.506 |
| 4 | will | 0.30 | 0.45 | +0.405 | −2.528 |
| 5 | just | 0.35 | 0.02 | −2.862 | −2.933 |
| 6 | multiply | 0.85 | 0.80 | −0.061 | −0.071 |
| 7 | to | 0.95 | 0.95 | 0.000 | −0.010 |
| 8 | 20 | 0.97 | 0.96 | −0.010 | −0.010 |
Three readings. First, \(\sum_tA_t=-2.470\): the sum of the advantages is minus the single-sample estimate of the sequence reverse KL, 2.470 nats, and 2.862 of the magnitude sits on token 5. Under \(\gamma=0\) the update lowers the probability of "just" in that context and leaves the final answer token almost untouched (\(-0.010\)), which is the behavior the post highlights. Second, under \(\gamma=1\) tokens 1 to 4 receive a return of \(-2.42\) to \(-2.53\) although their own log-ratios are \(-0.051\), \(+0.087\), \(+0.022\) and \(+0.405\): the fork's penalty is charged back to every earlier token. That is correct credit assignment in the sense of the exact gradient, since those tokens did lead the student to the fork, and it is also the source of the extra variance in \(O(T^4)\) versus \(O(T^2)\). Third, the expectation of \(A_t\) at any context is \(-\mathrm{KL}_t\le0\) (derived), so the average signal is negative by construction. Fu et al. report the sharper empirical fact that "most sampled tokens receive negative reward"9 (§2.2, Figure 2), with Qwen2.5-7B-Instruct as student and OpenThinker3-7B as teacher.
2.4 The loop, as a stepper and as code
- Student decode. The student samples responses exactly as in RL, and the sampler records \(\log\pi_\theta(y_t)\) for each token, "as part of the importance-sampling loss"1. This is the serving workload and the expensive half of the loop (Module 1, section 5).
- Teacher prefill. The teacher scores the finished sequence in one forward pass with no gradient and returns \(\log\pi_T(y_t)\) on the student's tokens. MOPD calls this "teacher prefill" and notes it is the one operation added to an RL loop8.
- Advantage. \(A_t=\log\pi_T(y_t)-\log\pi_\theta(y_t)\), the negative of the sampled reverse-KL term1. MOPD clips it two-sided to \([-A_{\max},A_{\max}]\) "for training stability"8 (Eq. 3).
- Policy-gradient update. The advantages go into the trainer's ordinary loss. The post calls its importance-sampling loss1; MOPD says the form "drops directly into existing PPO/GRPO training frameworks; the advantage computation is the only change"8.
- Weight sync. The updated student replaces the sampler's weights. How that hand-over is organized, and where the teacher is placed, is System Design Modules 36 and 37.
The mechanism is small enough that the code is the specification. This is the pseudocode from the post, which runs on Tinker's API, with comments added:
teacher_client = service_client.create_sampling_client(
base_model=teacher_config.base_model,
model_path=teacher_config.load_checkpoint_path,
)
trajectories = do_group_rollout(student_client, env_group_builder) # student samples, as in RL
sampled_logprobs = trajectories.loss_fn_inputs["logprobs"] # log pi_student(y_t), from the sampler
teacher_logprobs = teacher_client.compute_logprobs(trajectories) # log pi_teacher(y_t): one forward pass
reverse_kl = sampled_logprobs - teacher_logprobs
trajectories["advantages"] = -reverse_kl # A_t = log pi_T - log pi_S
training_client.forward_backward(trajectories, loss_fn="importance_sampling")
The post describes this as "a one-line change on top of RL implementations which use KL regularization: we merely swap out the regularizer model"1. That is the cleanest way to hold the idea: KL-regularized RL has a reward plus a small penalty \(\beta\,\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\) (Module 6); OPD sets the reward to zero, sets \(\beta=1\), and replaces the reference with the teacher. Lambert notes the advantage can also be combined with other advantage terms, such as GRPO's group normalization, for more elaborate reward shaping3 (§12.3.3, p. 153).
2.5 Why the direction is reverse, and what that does not protect
Minimizing the forward KL against a teacher the student cannot fully represent produces a student that covers the teacher's modes and places mass between them; minimizing the reverse KL produces one that locks onto a mode. MiniLLM states the consequence for text generation: the forward KL "causes qθ to place large probability masses on the zero-probability regions of p", "corresponding to the generation of low-quality text in practice", while the reverse KL "focuses on p's major modes"6 (§2.1); and Module 6 section 1.3 draws the geometry on a two-mode example. The Thinking Machines post adds that the target is "unhackable" because "low KL always corresponds to a high probability of desirable behavior from the teacher model's point of view", unlike a learned reward model1.
Two papers qualify "unhackable". MiniLLM observed reward hacking when it trained with the sequence-level policy gradient: the student sometimes produced degenerate sentences, such as repeated phrases, that the teacher scored highly, "especially for small student models". Its fix is teacher-mixed sampling, \(\tilde p=\alpha\,p_T+(1-\alpha)\,q_\theta\) at each step, with an importance weight, plus length normalization because the return-to-go favored short outputs and the student learned to emit empty responses6 (§2.2). Fu et al. show the same family of failure for the sampled-token form: in one run the student fell into a repetition loop while "the teacher remains locally aligned with the student on repeated tokens", so the sampled-token signal did not penalize it9 (Figure 3). My reading: the teacher's probability is a trustworthy grade only on prefixes the teacher could have produced, and a student that drifts off them is where it stops being one. "Unhackable" holds for the objective's optimum, which is the teacher, not for the path to it.
Two links to Module 6. The sampled-token advantage is the negative of the \(k_1\) estimator of the per-token reverse KL (Module 6 section 2.2); used as a detached reward it is gradient-equivalent on-policy to backpropagating \(k_2\) (section 3.3), and Shao et al. confirm the same direction for the OPD loss in the form Kimi K3 uses, which clips the reward for stability11. So the \(k_3\) trap of Module 6, regularizing toward the wrong KL direction, does not apply here. The partial-gradient caveat does, and \(\gamma=0\) adopts it on purpose.
3. What the extra signal is worth: bits per episode
The Thinking Machines post justifies the cost claims with an information argument it takes from an earlier post by Schulman and colleagues, "LoRA Without Regret". The argument for RL: the policy-gradient update has the form (score function) times (advantage), the score is independent of the reward given the history, so the only channel through which reward information reaches the weights is the scalar advantage. By the data-processing inequality the information about the reward function is at most the entropy of the advantage; quantize it into \(B\) bins and that is about \(\log B\) bits, which the post summarizes as "\(O(1)\) bits per episode", independent of model size12. Supervised learning, by contrast, "arguably provides \(O(\text{number of tokens})\) bits per episode"12. The distillation post applies the second count to OPD1.
The arithmetic is worth doing once because it changes how you read the RL cost figures of Module 1. If a whole RL run on MATH absorbs about 320,000 bits, the information content of the training signal is smaller than the parameter count of a rank-1 LoRA adapter for Llama-3.1-8B, which the post puts at roughly 3M parameters12. RL's compute goes into finding which samples to reward, not into delivering bits to the weights. The distillation post draws that conclusion directly: RL should be thought of as "spending most of its compute on search", meaning rolling out a policy and assigning credit rather than making updates, and once a good strategy exists, on-policy distillation need not model the intermediate strategies of the RL curriculum but "only the final strategy learned"1.
Three limits, none hidden by the posts. First, it is an upper bound on what the signal can carry. The LoRA post says its capacity figures concern "the maximum amount of information absorbed in the long-training limit", not compute efficiency or learning speed, and that the one-bit figure "may only apply narrowly to policy gradient algorithms"12. Second, \(O(N)\) counts positions, not realized information. A token on which teacher and student agree carries an advantage near zero (tokens 1 to 3, 7 and 8 in Table 3), and in that example one token of eight holds 82% of the total absolute advantage (derived: \(2.862/3.498\)). The post's illustration agrees: the penalized tokens are "forking tokens" that start a wrong phrase1. Third, the sampled-token advantage is a single-sample estimate with mean \(-\mathrm{KL}_t\), noisy enough that DeepSeek-V4 replaced it (section 5.2). OPD's signal has far more degrees of freedom per episode than a scalar reward; it does not teach a bit per token.
4. The reported numbers, and what each one does and does not count
Two sources report that OPD reaches an RL-level or better result at a fraction of the cost: the Thinking Machines post (October 2025, with a June 2026 update) and the Qwen3 report (May 2025). They are not independent in the way the phrase "replicated" suggests. The post says it uses Tinker to "replicate Qwen3's result of achieving equivalent performance on reasoning benchmarks with on-policy distillation for a fraction the cost of RL", and it takes its RL cost from the Qwen3 report1. Each has its own accounting, and neither is a controlled study with repeated seeds.
4.1 Thinking Machines: Qwen3-8B-Base on math
The setting1: student Qwen3-8B-Base, off-policy initialization by SFT on OpenThoughts-3 (reasoning traces generated by QwQ-32B), benchmark AIME'24. Full fine-tuning on 400K prompts gives 60%. SFT performance then rises log-linearly in the number of prompts after the first 50K to 100K, and extrapolating that line the post estimates about 70% at about 2M prompts, with the caveat that this "requires the scaling law to hold up without stalling" and is "an optimistic estimate" for off-policy distillation. Two reference points bound it: DeepSeek-R1-0528-Qwen3-8B reaches 86% with an unstated number of prompts, and Qwen2.5-7B and 14B reach 55.5% and 69.7% after 800K R1 distillation prompts. For RL, the post takes the Qwen3 report's 67.6% at 17,920 GPU-hours and estimates that "given some reasonable assumptions about the SFT training stack" this is similar to the cost of 2M off-policy prompts, while saying it "is hard to compare this directly to the cost of distillation".
OPD from the 400K checkpoint "achieves an AIME'24 of 70% in about 150 steps", which the post equates to about 77K prompts at 4 samples per prompt. The teacher actually used was Qwen3-8B, "as it performs slightly better"; the FLOPs in the cost table are nonetheless those of the 32B model1. A LoRA run at rank 32 trails full fine-tuning by 13% after SFT but by only 6% after OPD1.
4.2 The FLOPs table and the 9×, 18×, 30×
| Method | AIME'24 (%) | Teacher (FLOPs) | Student (FLOPs) | CE vs SFT-2M (printed) | Cost vs OPD, student+teacher (derived) |
|---|---|---|---|---|---|
| SFT-400K (initialization) | 60 | 8.5×1020 | 3.8×1020 | – | – |
| SFT-2M (extrapolated) | ~70 | 3.4×1021 | 1.5×1021 | 1× | 29.5× with teacher, 9.0× without |
| RL (Qwen3 report) | 68 | – | – | ≈1× | not reported in FLOPs |
| On-policy distillation | 70 | 8.4×1019 | 8.2×1019 | 9 to 30× | 1× |
The three multipliers are three accountings of one table. The post's definitions1:
- 9×: "when the SFT dataset is given, as in our example with OpenThoughts-3, or is amortized across many training runs". The teacher FLOPs of off-policy training are not counted, but the teacher's log-probability pass is counted for OPD, "since we must run the teacher model to compute log-probs". Derived: \(1.5\times10^{21}/(8.4\times10^{19}+8.2\times10^{19})=9.04\).
- 18×: the same comparison in GPU-hours, "since this computation can be cheaply parallelized across GPUs", so the teacher pass is effectively free. Derived: \(1.5\times10^{21}/8.2\times10^{19}=18.3\).
- 30×: the task is new and no distillation dataset exists, so the SFT side also pays for teacher sampling. Derived: \((3.4\times10^{21}+1.5\times10^{21})/(8.4\times10^{19}+8.2\times10^{19})=29.5\).
All three use only the incremental OPD cost, leaving out the shared 400K SFT initialization (the post does not say so; the numbers only work out that way). One more observation from the table, mine: the SFT-2M row is 3.95 times (student) and 4.0 times (teacher) the SFT-400K row, which is what 1.6M additional prompts at the same per-prompt cost would give, not 2M. The row therefore appears to be the increment over the 400K checkpoint, the right quantity for a like-for-like comparison, but the label says 2M.
Can the FLOPs columns be reconstructed? A token-count check (derived)
The post gives no FLOPs-per-token formula. Use the usual dense convention: \(2N\) FLOPs per token for a forward pass, \(6N\) for forward plus backward (Module 1, section 5.3). For OPD the 8B student decodes and trains on each sampled token, \(8N_S=6.4\times10^{10}\) FLOPs; the 32B teacher does one forward pass, \(2N_T=6.4\times10^{10}\) FLOPs. The two printed columns then imply \(8.2\times10^{19}/6.4\times10^{10}=1.28\times10^{9}\) and \(8.4\times10^{19}/6.4\times10^{10}=1.31\times10^{9}\) tokens, within 2.5% of each other, so the columns are consistent with one token count of about \(1.3\times10^{9}\). Over 77K prompts × 4 samples (308,000 samples) that is about 4,200 tokens per sample; over 9,600 × 4 (38,400 samples) it would be about 34,000, a very long average for training rollouts that the post says run at shorter context than evaluation1. The FLOPs favor the 77K figure (my reading).
The SFT rows reconcile less well: at \(6N_S\) per token the student column implies about 19,800 tokens per prompt over 400K prompts, and at \(2N_T\) the teacher column implies about 33,000. The post gives no formula that explains the factor of 1.7.
The FLOP-to-time conversion is where an inference engineer should push. A FLOP ratio is a GPU-hour ratio only if every FLOP runs at the same utilization, and these do not. Take the split above, 2 of the 8 student FLOPs per token in sampling and 6 in training, price training and SFT at 30% MFU, and price decode at Olmo 3's measured 1.01% MFU (43.21% MBU) for its final inference configuration13 (Table 23). At equal utilization the arithmetic reproduces the post's numbers, 18.3× without the teacher pass and 9.0× with it. With decode at 1.01% it falls to 2.2× and 2.0×; at 5% decode MFU it is 8.1× and 5.6×; at 10%, 12.2× and 7.2× (all derived, illustrative, and sensitive to the 30% assumption). The post's 18× therefore rests on both the teacher pass being cheap and student sampling running about as efficiently as training, and OPD's own cost is dominated by sampling, as Module 1 shows for Olmo 3's RL run. The one measured GPU-hour comparison is Qwen3's RL versus OPD, where both arms pay for sampling.
4.3 Qwen3's own comparison
The Qwen3 report is the origin of both the 17,920 and the 1,800. Its small models (0.6B, 1.7B, 4B, 8B, 14B and the 30B-A3B MoE) skip the four-stage pipeline. They get an off-policy phase on teacher outputs generated in both /think and /no think modes, then an on-policy phase in which the student generates in either mode and "is then fine-tuned by aligning its logits with those of a teacher model (Qwen3-32B or Qwen3-235B-A22B) to minimize the KL divergence"7 (§4.5). The report states the saving twice, as "only 1/10 of the GPU hours compared to the four-stage training method" for the whole strong-to-weak route (§4 introduction) and as the Table 21 experiment (§4.7)7.
| Method | AIME'24 | AIME'25 | MATH500 | LiveCodeBench v5 | MMLU-Redux | GPQA-Diamond | GPU-hours | GPU-hours per AIME'24 point |
|---|---|---|---|---|---|---|---|---|
| Off-policy distillation | 55.0 (90.0) | 42.8 (83.3) | 92.4 | 42.0 | 86.4 | 55.6 | – | – |
| + RL | 67.6 (90.0) | 55.5 (83.3) | 94.8 | 52.9 | 86.9 | 61.3 | 17,920 | 1,422 |
| + on-policy distillation | 74.4 (93.3) | 65.5 (86.7) | 97.0 | 60.3 | 88.3 | 63.3 | 1,800 | 93 |
Derived from the table: \(17{,}920/1{,}800=9.96\). Per point gained on AIME'24 over the shared starting checkpoint, RL spent \(17{,}920/12.6=1{,}422\) GPU-hours and OPD \(1{,}800/19.4=93\), a ratio of 15.3; on AIME'25 the figures are 1,411 and 79, a ratio of 17.8. Both are larger than 9.96 because OPD also ended higher. OPD led on all six benchmarks, by 6.8 and 10.0 points on AIME'24 and AIME'25 and by 1.4 to 7.4 points on the other four.
Three things the table does not tell you. First, whether the 1,800 GPU-hours include the teacher. The passage I read does not say, and it gives no hardware, so 9.96× is the ratio of the two numbers as printed, not a total-cost ratio. Treat it like the 18× of the Thinking Machines accounting, where the teacher pass is assumed cheap, and not like the 30×. Second, which teacher: §4.5 names Qwen3-32B or Qwen3-235B-A22B for the pipeline, and the Table 21 text does not say which one served the 8B experiment. Third, what the RL arm was. The report says "direct reinforcement learning" from the same checkpoint on math and code queries and gives no recipe for that arm in this comparison, so you cannot tell whether 17,920 GPU-hours is a tuned run or the flagship's stage-2 recipe applied to the 8B. The Thinking Machines post calls the RL result "similar" in cost to 2M SFT prompts and warns it is "hard to compare"1.
The pass@64 column is the report's capability argument. RL left it unchanged at 90.0 and 83.3; OPD raised it to 93.3 and 86.7. The report reads this as distillation from teacher logits letting the student "expand its exploration space and enhance its reasoning potential", while RL "does not lead to any improvement in pass@64 scores"7 (§4.7). Figure 5 shows how thin the margin is. With 30 questions per benchmark, the OPD gain in pass@64 is exactly one question on each (27 to 28 of 30 on AIME'24, 25 to 26 of 30 on AIME'25), and the report gives no seeds or intervals (derived). The pass@1 gaps of 6.8 and 10.0 points between OPD and RL are much larger than that, but they too come from one run each. Whether RL expands or sharpens, and how to design a measurement that separates the two, is Module 20's question; this module only records that the one place where on-policy distillation and RL were compared at fixed checkpoint and reported pass@k found the two different, on a one-question margin.
4.4 Self-distillation: when the teacher is an earlier copy
The post's two remaining experiments change what the teacher is, and they show the cost advantage in a different light1.
Recovering behavior after mid-training. Qwen3-8B, already post-trained, is fine-tuned on internal company documents. Knowledge goes up, instruction following goes down, and no mix of document and chat data keeps the original IF-eval. OPD with the earlier Qwen3-8B as teacher, run on Tulu 3 prompts unrelated to the documents, restores most of it.
| Model | Internal QA (%) | IF-eval (%) |
|---|---|---|
| Qwen3-8B | 18 | 85 |
| + mid-train, 100% documents | 43 | 45 |
| + mid-train, 70% documents / 30% chat | 36 | 79 |
| + mid-train 70/30, then OPD from earlier Qwen3-8B | 41 | 83 |
Replaying an RL result. The post trains Qwen3-8B-Base with RL on DeepMath at LoRA rank 128, and uses the result as the teacher for OPD back into the base model. It reports that the distilled student reaches the teacher's level "in approximately 7-10x fewer gradient steps", that the reverse KL falls to near zero and the AIME score is recovered "in under 10 gradient steps, while RL took 70 steps". It then adds two savings (RL must train near the evaluation context length, while distillation "learns reasonably at shorter context lengths", and OPD tolerates smaller batches) and states a compute efficiency "on the order of 50-100x"1. The factors are not broken out; only the step ratio (70/10, at least 7) is checkable.
Two smaller results. Training on one math prompt for 20 steps at 256 rollouts each (5,120 graded sequences) "approximately" matches the teacher's AIME'24 score, less efficiently than the multi-prompt default1. And SFT on the model's own temperature-1 samples degrades IF-eval at every practical learning rate, which the post attributes to finite-batch noise turning self-training into off-policy drift, while OPD against a fixed teacher "always stays on-policy"1. These are single runs, partly on an internal setup, and I treat them as hypotheses with a plausible mechanism.
5. What the recipe needs: teacher, estimator, horizon
The cost results hold when three conditions hold. Later work, mostly 2026 preprints, tested each. All of it is recent and little of it is peer reviewed, which I flag where it matters.
5.1 The teacher must be reachable from the student
Reverse KL is mode-seeking within the student's support: the post notes that SFT "adds support for new tokens" and "reverse-KL methods can then perform mode seeking within the initialization's support", which is why its experiments start from a mid-trained or SFT'd student1. The MOPD paper tested what happens when that fails. Its student is Qwen3-30B-A3B, and its math teachers are RL-trained descendants of the same SFT checkpoint ("same-origin"). Swapping in Qwen3-235B-A22B, "a significantly larger model with stronger math capability", degraded the student8 (§4.4.2). Figure 6 shows the numbers: the initial per-token KL was about 5× higher (~0.19 versus ~0.04), entropy under the policy-gradient loss contracted from 0.30 to 0.21 as the student "narrows toward a single mode as it receives predominantly punitive gradient signal", and the top-k variant collapsed near step 18.
The tension with Qwen3's own recipe is real and has a reading. Qwen3 distills 235B-A22B and 32B into its 30B-A3B, but only after an off-policy phase that fine-tunes the student on the teacher's outputs7 (§4.5). That phase moves the student into the teacher's support before the on-policy phase starts. The MOPD swap did not do that. Li et al. test the same idea from the other side and report that an "off-policy cold start" and "teacher-aligned prompt selection" recover failing OPD runs14. I read these as one finding: OPD needs low initial divergence, and an SFT stage on the teacher's samples is how you buy it. That is my synthesis, not a claim any of the three papers makes in those words.
Li et al. add that benchmark rank is not what matters. Same-pipeline teachers helped little and RL-post-trained ones much more, measured as the fraction of the student-teacher gap recovered: 5.3% versus 16.9% in the DeepSeek-distill family (student R1-Distill-1.5B) and 15.6% versus 58.6% in the Qwen3 family (student Qwen3-1.7B non-thinking), even though the RL teachers had slightly lower initial token overlap with the student (71.5% versus 74.7%; 70.3% versus 75.7%)14 (Figure 4). In a reverse experiment, JustRL-1.5B (an RL-improved model) as student was distilled from its own pre-RL checkpoint and "regress[ed] almost exactly to its pre-RL performance"; with R1-Distill-7B as teacher, a larger model that scored slightly higher than the student on benchmarks, it regressed to the same level14 (§3.3). That follows from the objective: the minimizer is the teacher's local distribution, so a teacher that sits in the wrong place pulls the student there whatever its score. In the authors' words OPD "fundamentally learns thinking patterns"; they also report that 97% to 99% of the probability mass sits in a small token set shared by student and teacher in successful runs14.
5.2 The sampled-token estimator is cheap and noisy
The estimator question is how much of the teacher's distribution each position returns. Sampled-token needs one number per token. Exact reverse KL needs \(|V|\) numbers, and Qwen3's vocabulary is 151,6697 (§2). In fp32 that is \(151{,}669\times4=606{,}676\) B, about 592 KiB per token per teacher; a batch of 1M tokens would hold about 607 GB of logits if materialized, against 4 MB for sampled-token log-probabilities (derived; real implementations chunk). Three groups made three choices:
- DeepSeek-V4 says the sampled-token form is "resource-efficient" but "leads to high variance in gradient estimation and often causes training instability", so it uses full-vocabulary logit distillation, made feasible by caching only the teacher's last-layer hidden states, rebuilding logits through the prediction head on the fly, ordering samples by teacher index so at most one teacher head is resident, and computing the exact KL in a TileLang kernel2 (§5.1.2, §5.2.2). The passage I read gives no ablation behind the instability claim.
- The MOPD paper compares its policy-gradient (sampled-token) form against top-\(k\) with \(k=64\) on same-origin teachers and finds them "comparable", normalized scores 0.937 versus 0.909, and says top-\(k\) "does not yield additional stability or variance reduction"8 (§4.4.1).
- Fu et al. replace the sampled token with teacher top-\(K\) local support matching (a truncated reverse KL over the teacher's \(K\) most likely tokens, renormalized on both sides), plus top-\(p\) rollouts and special-token masking, and report "+19.8%". That is the relative gain in average math score, 34.8 to 41.7, in an alternating math and ALFWorld run. In single-task math the average is 36.4 for sampled-token OPD, 40.7 with special-token masking alone and 41.7 with local support matching, so masking accounts for about 81% of the gain (derived: 4.3 of 5.3 points)9 (Tables 1 and 2). The student is Qwen2.5-7B-Instruct and the teacher OpenThinker3-7B, not a same-origin pair; the paper is marked "Under review".
These are compatible if variance matters most when teacher and student are far apart or rollouts are long, which is where Fu et al. and V4 operate, and matters little when the teacher is a close descendant, which is the MOPD setting. That is my reading; no paper runs all three on one setup. Table 7 collects the failure modes the sources report and the fix each proposes.
| Failure | What was measured | Proposed fix | Source |
|---|---|---|---|
| Teacher far from student | initial per-token KL ~0.19 vs ~0.04; entropy 0.30 to 0.21; top-k collapse near step 18 | same-origin teacher; off-policy cold start; teacher-aligned prompts | MOPD8; Li et al.14 |
| Imbalanced sampled-token signal | most sampled tokens receive negative reward (Figure 2 of the source) | teacher top-\(K\) local support matching | Fu et al.9 |
| Repetition not penalized | teacher "locally aligned" with the student's repeated tokens | top-\(p\) rollouts; teacher-mixed sampling (\(\alpha\) mix-in) | Fu et al.9; MiniLLM6 |
| Tokenizer mismatch | student's <think> split into three tokens: teacher log-probability −19.16 against student −0.07; end-of-sequence −58.71 against −0.00 | special-token masking; shared tokenizer (Lambert notes KD "often" requires one) | Fu et al.9; Lambert3 |
| Long responses | best results at 3K to 7K maximum length; 10K and 15K collapse late; teacher's accuracy advantage over the student's continuation falls from +0.37 at a 1K prefix to +0.02 at 16K | cap length; open problem for agentic horizons | Li et al.14 (§6.1) |
| Gradient concentrated on surprising tokens | gradient L1 norm factors into \(2|\Delta_t|\) and \((1-p_S)\) for the \(k_2\) loss; low-student-probability tokens carry a disproportionate share | surprise-aware reweighting (SuRe), up to +6.7 points on AIME'24 pass@8 for a 1.7B student | Shao et al.11 |
5.3 Horizon, and what stays outside this module
Li et al.'s long-response result limits the recipe where RL is most valuable. The teacher's grade depends on a prefix the student wrote, and the further it drifts from what the teacher would produce, the less informative the grade; instability starts at the end of long responses and propagates backward, and the authors conclude that OPD "may not extend cleanly" to extended chain-of-thought or agentic multi-turn interaction14 (§6.1). That is a statement about their setup, a 1.5B student on math. Mismatch between the engine that produces \(\log\pi_\theta\) and the one that trains is Module 14's subject; multiple teachers, prompt routing and V4's teacher scheduling are Module 22.
6. When distillation, not RL, is the cheaper route
6.1 The comparisons that exist
| Source | Setting | Result | Cost as reported |
|---|---|---|---|
| DeepSeek-R1, Supp. F.1, Table 1615 | Qwen2.5-32B; RL (Qwen2.5-32B-Zero, over 10K steps) versus SFT on 800K R1 samples | AIME'24 pass@1: QwQ-32B-Preview 50.0, RL 47.0, distilled 72.6; GPQA-Diamond 54.5, 55.0, 62.1; LiveCodeBench 41.9, 40.2, 57.2 | RL "require[s] enormous computational power"; no GPU-hours given in the table |
| Qwen3, Table 217 | 8B, same checkpoint; RL versus OPD | AIME'24 67.6 versus 74.4; pass@64 90.0 versus 93.3 | 17,920 versus 1,800 GPU-hours (teacher cost not stated) |
| Thinking Machines1 | Qwen3-8B-Base, from SFT-400K (60%) | OPD 70% at about 150 steps; RL 68% (from the Qwen3 report); SFT-2M extrapolated 70% | 9× to 30× in FLOPs; 18× in GPU-hours |
| Yue et al., §4.216 | Qwen2.5-Math-7B; base, RLVR, R1-distilled | distilled model's pass@k curve "consistently and significantly above" the base; RLVR bounded by the base | not compared |
| ProRL17 | 1.5B from R1-Distill-Qwen-1.5B; RL for over 2,000 steps | gains over the start of +15.7% (math), +14.4% (code), +25.9% (GPQA-Diamond), +22.0% (IFEval) and +54.8% (Reasoning Gym reward), from the body and Tables 1-3; the introduction prints 14.7%, 13.9%, 25.1% and 18.1% for the first four domains | about 16K GPU-hours on 4×8 H100 |
Read as a set, the rows agree on one thing and leave one thing open. They agree that for a small student with a stronger teacher available, a teacher-supplied signal beats RL from the same start on both quality and cost. They leave open whether RL, run long enough from a distilled start, passes what distillation reaches: ProRL's 16K GPU-hours is the same order as Qwen3's 17,920 for RL on the 8B, and ProRL claims gains "inaccessible to base models, even under extensive sampling"17, while R1's authors close the comparison by saying "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning"15. Module 20 weighs the capability evidence.
6.2 Break-even: when a teacher pays for itself
The quoted ratios hide that a teacher is a sunk cost somebody paid. If one teacher serves \(n\) students, per-student cost is what you compare; if you must build it, the break-even is \(n^*=C_T/(C_{\mathrm{RL}}-C_{\mathrm{OPD}}(1+o))\), with \(C_T\) the teacher's build cost, \(C_{\mathrm{OPD}}\) the per-student OPD cost, \(C_{\mathrm{RL}}\) the per-student RL cost, and \(o\) the teacher-pass overhead as a fraction of the OPD cost. Qwen3 amortizes teachers over six small models (five dense and one MoE)7; the calculator below defaults to the 8B figures and an unknown, hence zero, teacher cost.
Distillation versus RL, with the teacher counted
Formulas: \(n C_{\mathrm{RL}}\); \(C_T+nC_{\mathrm{OPD}}(1+o)\); \(C_{\mathrm{RL}}/(C_{\mathrm{OPD}}(1+o))\); \(C_T/(C_{\mathrm{RL}}-C_{\mathrm{OPD}}(1+o))\). Defaults are the Qwen3-8B figures of Table 57; the report states neither the teacher cost nor whether the 1,800 includes it. Setting \(C_T\) to the cost of an RL run for the teacher shows why Thinking Machines' 50-100× applies to copies, not to discovery.
6.3 Decision inputs
| Situation | Cheaper or better route | Evidence |
|---|---|---|
| Stronger teacher exists, student already SFT'd on its outputs, same tokenizer | OPD | Qwen3 Table 217; Thinking Machines1 |
| Teacher is a different, larger model and the student is cold | off-policy SFT first, then OPD; OPD alone can degrade | MOPD §4.4.28; Li et al.14 |
| You must restore a behavior lost in fine-tuning, and an earlier checkpoint has it | OPD with that checkpoint as teacher | Table 61 |
| No model stronger than your target exists | RL; OPD cannot exceed its teacher | R1 Supp. F.115 |
| Student and teacher use different tokenizers | avoid sampled-token OPD, or mask special tokens | Fu et al.9 |
| Responses of 10K tokens or more, or multi-turn agent episodes | no OPD evidence at this length in the sources read | Li et al.14 |
Where experts disagree
The question the sources contest is where a fixed budget goes when you want a strong small reasoning model: RL on the model, or a teacher's signal. A second question, which estimator to use, is covered in section 5.2.
Distill first; RL costs more for less
DeepSeek-R1 authors, Supp. F.115; Qwen3 team7; Thinking Machines1
R1 trained Qwen2.5-32B with RL for over 10K steps and matched only QwQ-32B-Preview, while SFT on R1's samples scored far higher. Qwen3 and the post report OPD beating RL at about a tenth of the GPU-hours, with higher pass@64 in Qwen3's case.
- Every direct comparison favors the teacher signal on cost and on pass@1.
- Dense signal works at smaller batches and shorter context than RL needs1.
- The ratios omit or do not state the teacher's cost, and the 30× version is the only one that counts teacher sampling.
- Single runs without seeds; R1's "distillation" is off-policy SFT.
RL is still needed to pass the teacher, and it works from a distilled start
DeepSeek-R1 authors, same passage15; ProRL (NVIDIA)17
R1's own second conclusion is that "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning". ProRL runs over 2,000 RL steps from a distilled 1.5B checkpoint, with KL control, reference resets and diverse tasks, and reports gains across five domains.
- OPD's optimum is the teacher, so OPD alone cannot exceed it.
- ProRL shows large RL gains on top of distillation at about 16K GPU-hours17.
- ProRL's baseline is its own start, not OPD from a stronger teacher.
- The claim that these gains are new capability is contested (Module 20).
OPD is conditional: it moves a student toward a close teacher, nothing more
Li et al.14; the MOPD paper (Xiaomi)8; Fu et al.9
A larger, stronger teacher failed when it was distributionally far from the student; same-pipeline teachers gave small gains; a pre-RL checkpoint erased an RL student's gains; long rollouts destabilized training.
- Explains the failures that the headline ratios hide, and offers fixes (cold start, same-origin teachers).
- Small models, mostly math, 2026 preprints, one marked "under review"; none measures cost against RL.
The positions answer different questions, so they mostly do not collide. Below the frontier, with a teacher that exists, shares the tokenizer, and sits close to the student after an SFT stage, OPD is the cheaper route in every comparison here, by 9.96× on the one that reports GPU-hours, a ratio that may overstate the saving because the report does not say whether the teacher is in the 1,800. At or past the frontier, or with no teacher stronger than the target, RL is the only route, and the third position says the conditions for the first are narrower than the headlines suggest. The evidence that would settle it is a study the sources do not contain: one starting checkpoint, teacher build and serving cost counted, RL and OPD each given matched total GPU-hours, several seeds, and pass@k at large k over more than 30 problems.
What is changing
The reproducibility footing is thin. The post's June 2026 update says Qwen3-32B and Qwen3-8B-Base "have since been retired from the Tinker model lineup" and that the cookbook recipes now use Qwen3.5-9B-Base as the student and Qwen3.5-9B as the teacher; the printed numbers come from the original models1. The analysis literature is moving faster than the recipe: Fu et al. (March 2026, revised April), Li et al. (April 2026) and Shao et al. (August 2026, third version October) all post-date the post and all are preprints91411. In production the recipe has moved from a single teacher to OPD as a final stage with many teachers (DeepSeek-V4's more than ten2), which is Module 22. A separate branch drops the external teacher: Lambert describes on-policy self-distillation, where one model with privileged information in its context, such as a hint or a judge's feedback, acts as its own teacher, and attributes a use of it to Cursor's Composer 2.53 (§12.3.3). I have not checked Cursor's own account.
References
- blogLu, K. and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 27 Oct 2025; June 2026 update note on retired models. https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-27. Used for: taxonomy table, per-token reverse KL objective, discount factor zero, pseudocode, Qwen3-8B-Base experiments, FLOPs table and the 9x/18x/30x accounting, personalization table, 7-10x step and 50-100x compute claims, single-prompt run, June 2026 update.
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-27. Used for: mixed RL replaced by OPD (§5.1), full-vocabulary OPD and the stated reason (§5.1.2, Eq. 29), hidden-state caching and teacher-head scheduling (§5.2.2), more than ten teachers.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12 (11 Sep 2026). https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-27. Used for: forward-KL identity for SFT-style distillation (§12.3.1, p. 150), exposure bias and the DAgger analogy (§12.3.2, pp. 150-152), advantage as negative per-token reverse KL, combination with GRPO advantages and the shared-tokenizer remark (§12.3.3, p. 153), on-policy self-distillation and the Cursor attribution.
- paperRoss, S., Gordon, G. J., Bagnell, J. A. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. arXiv:1011.0686v3, 16 Mar 2011 (AISTATS 2011). https://arxiv.org/abs/1011.0686. Accessed 2026-10-27. Used for: the \(T^2\epsilon\) bound for supervised imitation (Theorem 2.1) and DAgger's \(uT\epsilon_N\) bound (Theorem 3.2).
- paperAgarwal, R., Vieillard, N. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD). ICLR 2024; arXiv:2306.13649v3, 17 Jan 2024. https://arxiv.org/abs/2306.13649. Accessed 2026-10-27. Used for: on-policy KD without backpropagating through sampling (§3.1, Eq. 4), the student-data fraction \(\lambda\), divergence choices, relative gains of 2.1x, 1.7x and 1.9x (§1, Figure 1).
- paperGu, Y., Dong, L., Wei, F., Huang, M. MiniLLM: On-Policy Distillation of Large Language Models. arXiv:2306.08543v6, 31 Jan 2026. https://arxiv.org/abs/2306.08543. Accessed 2026-10-27. Used for: reverse-KL policy gradient with reward-to-go (§2.2, Eq. 2), single-step decomposition, teacher-mixed sampling and length normalization, reward hacking by degenerate text, the mode-seeking argument (§2.1).
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-27. Used for: vocabulary size 151,669 (§2), strong-to-weak distillation, off-policy then on-policy phases, teacher choices, 1/10 GPU-hours (§4 introduction, §4.5), evaluation protocol of 30 AIME questions with 64 samples (§4.6), RL versus on-policy distillation (§4.7, Table 21, p. 21).
- paperMa, W. et al. (Peking University and Xiaomi LLM Core). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-27. Used for: per-token reverse KL loss, policy-gradient and top-k forms with clipped advantage (§3.2, Eqs. 1-5), teacher prefill (§3.3), same-origin teacher experiment (§4.4.2, Table 4, Figure 3), policy-gradient versus top-k comparison (§4.4.1).
- paperFu, Y. et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv:2603.25562v2, 27 Apr 2026 (preprint, "under review"). https://arxiv.org/abs/2603.25562. Accessed 2026-10-27. Used for: the \(\gamma\)-interpolated estimator and the \(O(T^2)\) versus \(O(T^4)\) variance bounds (§2.1, Eqs. 1-3, Appendix D), three failure modes and the tokenizer example (§2.2, Figures 2-5), local support matching (§3), the +19.8% claim (abstract).
- paperTang, Y., Munos, R. On a few pitfalls in KL divergence gradient estimation for RL. arXiv:2506.09477v1, 11 Jun 2025. https://arxiv.org/abs/2506.09477. Accessed 2026-10-27. Used for: sequence-level versus token-level KL gradient in the 70B-to-8B on-policy distillation experiment (§5.2-§5.3, Figure 8, Appendix E).
- paperShao, B. et al. (Fudan University). A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation. arXiv:2608.25643v3, 13 Oct 2026 (preprint). https://arxiv.org/abs/2608.25643. Accessed 2026-10-27. Used for: gradient of the per-token \(K_2\) estimator, its direction matching the detached sampled-token reward of Kimi K3 (which clips it), the \(2|\Delta_t|(1-p_S)\) factorization, SuRe result (abstract, Figure 1).
- blogSchulman, J. and Thinking Machines Lab. LoRA Without Regret. Thinking Machines Lab: Connectionism, 29 Sep 2025. https://thinkingmachines.ai/blog/lora/. Accessed 2026-10-27. Used for: the O(1) bits per episode argument for policy gradient, 320,000 bits for the MATH run, 5.3M episodes for R1-Zero, the stated limits of the argument (§"How much capacity is needed by supervised and reinforcement learning?").
- paperOlmo Team. Olmo 3. arXiv:2512.13961v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-27. Used for: inference-side MFU 1.01% and MBU 43.21% in the RL infrastructure benchmark (Table 23).
- paperLi, Y. et al. (Tsinghua University). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016v2, 15 Apr 2026 (preprint). https://arxiv.org/abs/2604.13016. Accessed 2026-10-27. Used for: thinking-pattern consistency and "new knowledge" conditions (§3), gap-recovery rates and initial overlap (Figure 4), reverse distillation (§3.3, Figure 5), overlap-token mechanism (§4), cold start and teacher-aligned prompts (§5), reward quality versus trajectory depth (§6.1).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (expanded Nature version), 4 Jan 2026. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-27. Used for: distillation versus RL on Qwen2.5-32B, over 10K RL steps, Table 16 (Supp. F.1), the two conclusions drawn there.
- paperYue, Y. et al. (Tsinghua University). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837v5, 24 Nov 2025. https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-27. Used for: distilled R1-Distill-Qwen-7B pass@k above the base model and above RLVR (§4.2, Figure 7).
- paperLiu, M. et al. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-27. Used for: RL from DeepSeek-R1-Distill-Qwen-1.5B, over 2,000 steps, about 16K GPU-hours on 4 nodes of 8 H100 (§3.2), per-domain gains from the body text and Tables 1-3 (§3.3), with the differing figures of the introduction (§1), the claim of strategies inaccessible to base models (abstract).