Reinforcement Learning for LLM Post-Training · Module 21

On-Policy Distillation as Dense-Reward RL: Per-Token Reverse KL, Cost Arithmetic, and When It Beats RL

On-policy distillation (OPD) keeps the part of RL that matters for a small model, training on its own samples, and replaces the one scalar reward per episode with a teacher's verdict on every token. This module derives the objective and its gradient, reads the two reported cost comparisons against RL (Thinking Machines and Qwen3) with their arithmetic and their gaps, and collects what later papers found about when the recipe fails. Afterwards you can implement OPD on top of an RL trainer and say, for a given teacher and budget, whether distillation or RL is the cheaper route.

Course 50 min read 17 sources As of 2026-10-27 Prerequisites: Module 9, Module 10, Module 20
Two-by-two grid: columns are off-policy and on-policy trajectories, rows are dense per-token signal and sparse per-episode signal; SFT sits in off-policy dense, RL in on-policy sparse, on-policy distillation in on-policy dense, and the off-policy sparse cell is empty in the source's table Off-policy teacher's or dataset's trajectories On-policy the student's own trajectories Dense signal on every token Sparse one scalar per episode SFT, off-policy distillation forward KL on teacher traces trains on states the teacher visits exposure bias On-policy distillation reverse KL on each sampled token O(N) bits per episode student's states, teacher's grade empty in the source's table Reinforcement learning reward on the finished response O(1) bits per episode student's states, no per-token grade who generated the trajectory that is being graded
Figure 1. The taxonomy the Thinking Machines post uses: SFT is off-policy and dense, RL is on-policy and sparse, on-policy distillation is on-policy and dense1. The two arrows are the two moves that produce OPD: sample from the student instead of the teacher, or grade every token instead of the whole response. The post's table has no entry for the fourth cell. The per-episode bit counts are the post's information-theoretic argument1, examined in section 3.

1. Two axes: whose trajectories, and how dense the signal

Post-training methods differ along two axes that are easy to conflate. One is where the sequences come from: a fixed dataset or a teacher's samples (off-policy), or the model being trained (on-policy). The other is how much of each sequence receives a training signal: one number for the whole response, or a number per token. Thinking Machines' post, which is the reference DeepSeek-V4 cites for OPD2, uses exactly this grid1. This module is about the cell that SFT and RL each cover half of.

1.1 The taxonomy and what each row optimizes

Table 1. The three rows of the Thinking Machines taxonomy1, with the objective each one minimizes on the student \(\pi_\theta\) against a teacher \(\pi_T\). The forward-KL identity for SFT-style distillation is Lambert's (§12.3.1, p. 150)3.
MethodWho samplesSignalObjective on the studentKnown weakness
SFT / off-policy distillationdataset or teacherdense: teacher's token at every positionforward KL \(\mathrm{KL}(\pi_T\|\pi_\theta)\) on teacher-sampled prefixes (cross-entropy differs from it by the constant \(H(\pi_T)\))trains on states the teacher visits, not the student1
RL (RLVR, RLHF)studentsparse: one reward per episodemaximize expected reward, with the credit machinery of Modules 2 and 9"a fixed number of bits per training episode regardless of the number of tokens used"1
On-policy distillationstudentdense: teacher's verdict on every sampled tokenreverse KL \(\mathrm{KL}(\pi_\theta\|\pi_T)\) along the student's own samplesneeds a teacher whose distribution the student can reach (section 5)

The distinction in the last column of the first row is the reason the field cared. Cross-entropy on teacher samples is the forward KL up to a constant that depends only on the fixed teacher3; the student is graded on the teacher's prefixes. In the OPD row the expectation is taken under the student, which is the one-line change in the objective and, as section 2 shows, the reason it can reuse an RL trainer.

1.2 Exposure bias and the DAgger bound

The off-policy student never sees its own mistakes. The post's description: "if the student makes an early mistake that the teacher never makes, it finds itself diverging ever farther from the states it observed in training"1. Imitation learning has a formal version of this. Ross, Gordon and Bagnell show that a learner whose per-step 0-1 loss is \(\epsilon\) on the expert's state distribution can cost \(J(\pi)\le J(\pi^*)+T^2\epsilon\) over a horizon of \(T\) steps (their Theorem 2.1), and that DAgger, which labels the states the learner itself visits, reaches \(J(\hat\pi)\le J(\pi^*)+uT\epsilon_N+O(1)\) (Theorem 3.2), where \(\epsilon_N\) is a loss measured under the learner's own state distribution4. For equal \(\epsilon\) and \(u=O(1)\) the ratio of the two bounds is \(T\): at 4,000 tokens the quadratic bound is 4,000 times the linear one (derived).

1.3 Lineage

  1. DAgger

    The student acts, the expert labels the states the student reached, and the student retrains on the aggregate dataset4.

  2. GKD and MiniLLM

    Generalized Knowledge Distillation trains the student on its own samples with teacher probabilities, does "not backpropagate through the student's sampling distribution", mixes on-policy and dataset sequences through a student-data fraction \(\lambda\), and allows other divergences such as reverse KL and generalized JSD; it reports relative gains over baseline distillation of 2.1× on summarization, 1.7× on translation and 1.9× on arithmetic reasoning, averaged across T5 student sizes5 (ICLR 2024). MiniLLM minimizes the sequence-level reverse KL with a policy gradient and three stabilizers, covered in section 2.56.

  3. Qwen3 strong-to-weak

    Small Qwen3 models get an off-policy distillation phase and then an on-policy phase that aligns student logits with a teacher7 (§4.5).

  4. Thinking Machines post

    OPD recast as an RL-shaped objective: advantage equals negative per-token reverse KL, so an RL trainer runs it with one changed line1.

  5. OPD as a final stage

    DeepSeek-V4's mixed-RL stage is "entirely replaced" by OPD2 (§5.1) and the MOPD paper integrates domain teachers into one student by the same mechanism8. Module 22 covers the multi-teacher case.

Two facts matter for what follows. GKD uses the teacher's full next-token distribution at every visited prefix and back-propagates it directly5, while the Thinking Machines recipe uses only the sampled token's log-ratio as a reward1; section 5.2 returns to the cost of that choice. And the Qwen3 report describes its on-policy phase as fine-tuning "by aligning its logits with those of a teacher model (Qwen3-32B or Qwen3-235B-A22B) to minimize the KL divergence"7 (§4.5), without stating the KL direction or the estimator, so the two headline results in section 4 come from related but not identical objectives.

2. The objective as a reward in an RL trainer

2.1 Per-token reverse KL along the student's sample

Write \(c_t=(x,y_{<t})\) for the context at position \(t\). The sequence-level reverse KL from student to teacher splits exactly, by the chain rule, into per-token terms evaluated on contexts the student generated3 (§12.3.2, Eq. 136):

$$ \mathrm{KL}\big(\pi_\theta(\cdot|x)\,\|\,\pi_T(\cdot|x)\big)=\mathbb{E}_{y\sim\pi_\theta}\sum_{t=1}^{T}\mathrm{KL}_t,\qquad \mathrm{KL}_t=\sum_{v}\pi_\theta(v|c_t)\log\frac{\pi_\theta(v|c_t)}{\pi_T(v|c_t)}. $$

The Thinking Machines recipe estimates each \(\mathrm{KL}_t\) from the one token the student sampled, uses it as a negative reward, and sets the discount factor to zero: "the student only optimizes the immediate next token, with no consideration for future tokens". The post adds that a positive discount is "more mathematically correct" but that "we do not find discount factors > 0 to improve performance in practice"1. The per-token advantage is the negative sampled log-ratio:

$$ A_t=\log\pi_T(y_t|c_t)-\log\pi_\theta(y_t|c_t). $$

Nothing else is needed from the environment. No verifier, no reward model, no critic: the teacher's log-probabilities on the student's tokens are the entire reward. The post notes two consequences: a rollout need not finish before it is scored, so partial or shorter rollouts work, and the teacher is queried with a single forward pass while the cheaper student does the generating1.

2.2 The gradient, and what discount zero drops

At a fixed context the policy-gradient identity applies position by position. Because \(\mathbb{E}_{v\sim\pi_\theta}[\nabla\log\pi_\theta(v)]=0\), the constant in \(\nabla(\pi\log\pi)\) vanishes and

$$ \nabla_\theta\mathrm{KL}_t=\mathbb{E}_{y_t\sim\pi_\theta}\Big[\big(\log\pi_\theta(y_t)-\log\pi_T(y_t)\big)\nabla_\theta\log\pi_\theta(y_t)\Big]=-\,\mathbb{E}\big[A_t\,\nabla_\theta\log\pi_\theta(y_t)\big]. $$

So an RL trainer that multiplies \(\nabla\log\pi_\theta(y_t)\) by \(A_t\) follows an unbiased estimate of \(-\nabla\mathrm{KL}_t\) at that context (derived; Module 6 section 3.3 shows the same identity for the \(k_2\) estimator). What it does not follow is the gradient of the whole sum, because the token \(y_t\) also changes which contexts \(c_{t+1},\dots\) the student visits next, and so changes their KL terms. The estimator that keeps this dependence uses the return-to-go. Fu et al. define the interpolation9 (Eq. 1 to 3), with \(r_t=\log\pi_\theta(y_t|c_t)-\log\pi_T(y_t|c_t)=-A_t\) and \(s_t=\nabla_\theta\log\pi_\theta(y_t|c_t)\):

$$ \hat g_\gamma=\sum_{t=1}^{T}\Big(\sum_{t'=t}^{T}\gamma^{t'-t}\,r_{t'}\Big)s_t,\qquad \gamma\in[0,1]. $$

With \(\gamma=0\) the inner sum collapses to \(r_t\) alone, so no return-to-go over later tokens enters the weight on \(s_t\); this is the token-level estimator, which is what the Thinking Machines recipe computes; with \(\gamma=1\) it is the causal sequence-level policy gradient, which is what MiniLLM derives (its Eq. 2, with reward-to-go \(R_t\))6. The two cost the teacher the same payload, one log-probability per sampled token. They differ in bias and variance. Fu et al. show that under bounded rewards and gradients the worst-case variance bound of the token-level estimator grows as \(O(T^2)\) in the response length and that of the sequence-level estimator as \(O(T^4)\), and their toy experiment shows variance rising with \(\gamma\)9 (§2.1, Appendix D). Tang and Munos find the opposite ordering on a different metric: distilling a Llama 3.1 70B policy into an 8B one on 7,500 MATH prompts with \(n=4\) samples per gradient estimate, a properly implemented sequence-level estimator lowered the sequence-level KL faster than the best token-level one, and leave-one-out and cumulative variance reduction lowered it further10 (§5.2 to §5.3, Figure 8). The first result is about gradient noise and stability, the second is about how fast the KL objective itself falls. Neither measures downstream accuracy, so they do not contradict each other; the recent recipes in Table 2 all keep supervision per token (section 5.2).

Table 2. Four ways to turn "grade the student's sample with a teacher" into a loss. "Payload" is what the teacher must return per generated token. Sources for each row are in the last column; the tensor sizes are derived in section 5.2.
FormPer-token signalTeacher payloadUsed or analyzed in
Sampled token, \(\gamma=0\)\(A_t=\log\pi_T(y_t)-\log\pi_\theta(y_t)\)one log-probabilityThinking Machines1; MOPD's policy-gradient form, with the advantage clipped to \([-A_{\max},A_{\max}]\)8 (Eqs. 3 and 4)
Sampled token, \(\gamma=1\)\(\sum_{t'\ge t}A_{t'}\) (reward-to-go)one log-probabilityMiniLLM, with single-step decomposition, teacher-mixed sampling and length normalization6; Tang and Munos10
Top-\(k\) of the teacherreverse KL over \(k\) tokens, renormalized or bias-corrected\(k\) token ids and \(k\) log-probabilitiesMOPD, \(k=64\) in its comparison8 (§3.2.2, Eq. 5; §4.4.1); Fu et al.'s local support matching9
Full vocabularyexact \(\mathrm{KL}_t\)\(|V|\) values, or last-layer hidden states to rebuild themGKD5; DeepSeek-V42 (§5.1.2); Qwen3 describes "aligning its logits"7

2.3 A worked trajectory

Table 3 is a constructed eight-token response, scored by a hypothetical teacher. It mirrors the illustration in the Thinking Machines post, where the teacher penalizes the tokens that start the wrong approach and does not penalize the final wrong answer, "though wrong, isn't penalized — it is entirely predictable conditional on the whole preceding sequence"1. The probabilities are the probabilities each model assigns to the token the student sampled, in the same context; \(G_t\) is the \(\gamma=1\) return-to-go. Values are computed from the probabilities (derived).

Table 3. Constructed example: per-token advantage under \(\gamma=0\) (\(A_t\)) and \(\gamma=1\) (\(G_t=\sum_{t'\ge t}A_{t'}\)), in nats. Token 5 is the fork where the student and teacher disagree.
\(t\)token\(\pi_\theta(y_t)\)\(\pi_T(y_t)\)\(A_t=\ln(\pi_T/\pi_\theta)\)\(G_t\)
1The0.400.38−0.051−2.470
2ice0.550.60+0.087−2.419
3cubes0.900.92+0.022−2.506
4will0.300.45+0.405−2.528
5just0.350.02−2.862−2.933
6multiply0.850.80−0.061−0.071
7to0.950.950.000−0.010
8200.970.96−0.010−0.010

Three readings. First, \(\sum_tA_t=-2.470\): the sum of the advantages is minus the single-sample estimate of the sequence reverse KL, 2.470 nats, and 2.862 of the magnitude sits on token 5. Under \(\gamma=0\) the update lowers the probability of "just" in that context and leaves the final answer token almost untouched (\(-0.010\)), which is the behavior the post highlights. Second, under \(\gamma=1\) tokens 1 to 4 receive a return of \(-2.42\) to \(-2.53\) although their own log-ratios are \(-0.051\), \(+0.087\), \(+0.022\) and \(+0.405\): the fork's penalty is charged back to every earlier token. That is correct credit assignment in the sense of the exact gradient, since those tokens did lead the student to the fork, and it is also the source of the extra variance in \(O(T^4)\) versus \(O(T^2)\). Third, the expectation of \(A_t\) at any context is \(-\mathrm{KL}_t\le0\) (derived), so the average signal is negative by construction. Fu et al. report the sharper empirical fact that "most sampled tokens receive negative reward"9 (§2.2, Figure 2), with Qwen2.5-7B-Instruct as student and OpenThinker3-7B as teacher.

2.4 The loop, as a stepper and as code

Student decode y, log π_S(y_t) Teacher prefill log π_T(y_t) Advantage A_t = log π_T− log π_S PG update no critic,no reward model New weights next rollout samples from the updated student
Figure 2. One iteration of sampled-token OPD in an RL trainer. Stage structure from the Thinking Machines pseudocode1 and the MOPD infrastructure description8 (§3.3).
  1. Student decode. The student samples responses exactly as in RL, and the sampler records \(\log\pi_\theta(y_t)\) for each token, "as part of the importance-sampling loss"1. This is the serving workload and the expensive half of the loop (Module 1, section 5).
  2. Teacher prefill. The teacher scores the finished sequence in one forward pass with no gradient and returns \(\log\pi_T(y_t)\) on the student's tokens. MOPD calls this "teacher prefill" and notes it is the one operation added to an RL loop8.
  3. Advantage. \(A_t=\log\pi_T(y_t)-\log\pi_\theta(y_t)\), the negative of the sampled reverse-KL term1. MOPD clips it two-sided to \([-A_{\max},A_{\max}]\) "for training stability"8 (Eq. 3).
  4. Policy-gradient update. The advantages go into the trainer's ordinary loss. The post calls its importance-sampling loss1; MOPD says the form "drops directly into existing PPO/GRPO training frameworks; the advantage computation is the only change"8.
  5. Weight sync. The updated student replaces the sampler's weights. How that hand-over is organized, and where the teacher is placed, is System Design Modules 36 and 37.

The mechanism is small enough that the code is the specification. This is the pseudocode from the post, which runs on Tinker's API, with comments added:

teacher_client = service_client.create_sampling_client(
    base_model=teacher_config.base_model,
    model_path=teacher_config.load_checkpoint_path,
)
trajectories = do_group_rollout(student_client, env_group_builder)   # student samples, as in RL
sampled_logprobs = trajectories.loss_fn_inputs["logprobs"]           # log pi_student(y_t), from the sampler
teacher_logprobs = teacher_client.compute_logprobs(trajectories)     # log pi_teacher(y_t): one forward pass
reverse_kl = sampled_logprobs - teacher_logprobs
trajectories["advantages"] = -reverse_kl                             # A_t = log pi_T - log pi_S
training_client.forward_backward(trajectories, loss_fn="importance_sampling")

The post describes this as "a one-line change on top of RL implementations which use KL regularization: we merely swap out the regularizer model"1. That is the cleanest way to hold the idea: KL-regularized RL has a reward plus a small penalty \(\beta\,\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\) (Module 6); OPD sets the reward to zero, sets \(\beta=1\), and replaces the reference with the teacher. Lambert notes the advantage can also be combined with other advantage terms, such as GRPO's group normalization, for more elaborate reward shaping3 (§12.3.3, p. 153).

2.5 Why the direction is reverse, and what that does not protect

Minimizing the forward KL against a teacher the student cannot fully represent produces a student that covers the teacher's modes and places mass between them; minimizing the reverse KL produces one that locks onto a mode. MiniLLM states the consequence for text generation: the forward KL "causes qθ to place large probability masses on the zero-probability regions of p", "corresponding to the generation of low-quality text in practice", while the reverse KL "focuses on p's major modes"6 (§2.1); and Module 6 section 1.3 draws the geometry on a two-mode example. The Thinking Machines post adds that the target is "unhackable" because "low KL always corresponds to a high probability of desirable behavior from the teacher model's point of view", unlike a learned reward model1.

Two papers qualify "unhackable". MiniLLM observed reward hacking when it trained with the sequence-level policy gradient: the student sometimes produced degenerate sentences, such as repeated phrases, that the teacher scored highly, "especially for small student models". Its fix is teacher-mixed sampling, \(\tilde p=\alpha\,p_T+(1-\alpha)\,q_\theta\) at each step, with an importance weight, plus length normalization because the return-to-go favored short outputs and the student learned to emit empty responses6 (§2.2). Fu et al. show the same family of failure for the sampled-token form: in one run the student fell into a repetition loop while "the teacher remains locally aligned with the student on repeated tokens", so the sampled-token signal did not penalize it9 (Figure 3). My reading: the teacher's probability is a trustworthy grade only on prefixes the teacher could have produced, and a student that drifts off them is where it stops being one. "Unhackable" holds for the objective's optimum, which is the teacher, not for the path to it.

Two links to Module 6. The sampled-token advantage is the negative of the \(k_1\) estimator of the per-token reverse KL (Module 6 section 2.2); used as a detached reward it is gradient-equivalent on-policy to backpropagating \(k_2\) (section 3.3), and Shao et al. confirm the same direction for the OPD loss in the form Kimi K3 uses, which clips the reward for stability11. So the \(k_3\) trap of Module 6, regularizing toward the wrong KL direction, does not apply here. The partial-gradient caveat does, and \(\gamma=0\) adopts it on purpose.

3. What the extra signal is worth: bits per episode

The Thinking Machines post justifies the cost claims with an information argument it takes from an earlier post by Schulman and colleagues, "LoRA Without Regret". The argument for RL: the policy-gradient update has the form (score function) times (advantage), the score is independent of the reward given the history, so the only channel through which reward information reaches the weights is the scalar advantage. By the data-processing inequality the information about the reward function is at most the entropy of the advantage; quantize it into \(B\) bins and that is about \(\log B\) bits, which the post summarizes as "\(O(1)\) bits per episode", independent of model size12. Supervised learning, by contrast, "arguably provides \(O(\text{number of tokens})\) bits per episode"12. The distillation post applies the second count to OPD1.

One 20-token trajectory shown twice: with RL a single scalar reward arrives after the last token; with on-policy distillation every token carries its own advantage, drawn as bars of uneven height RL: one scalar after the last token r O(1) bits OPD: one signal on every sampled token schematic |A_t|, uneven: a few forking tokens carry most of it O(N) bits token position t = 1 ... N (N = 20 here) bar height: magnitude of the per-token advantage (schematic, not measured)
Figure 3. The information-theoretic argument as a picture. In policy-gradient RL the reward information in an episode is a single scalar advantage, "O(1) bits per episode"12; the Thinking Machines post counts "O(N) bits per episode" for distillation, where N is the number of tokens1. Bar heights are illustrative; Table 3 gives a computed example in which one token of eight holds 82% of the total absolute advantage (derived).
320,000 bitswhole MATH RL run: about 10,000 problems × 32 samples × 1 bit12
5.3M bitsR1-Zero: 5.3M episodes, one bit each12
~1,000×less information per token for RL than for supervised learning, at thousands of tokens per episode12

The arithmetic is worth doing once because it changes how you read the RL cost figures of Module 1. If a whole RL run on MATH absorbs about 320,000 bits, the information content of the training signal is smaller than the parameter count of a rank-1 LoRA adapter for Llama-3.1-8B, which the post puts at roughly 3M parameters12. RL's compute goes into finding which samples to reward, not into delivering bits to the weights. The distillation post draws that conclusion directly: RL should be thought of as "spending most of its compute on search", meaning rolling out a policy and assigning credit rather than making updates, and once a good strategy exists, on-policy distillation need not model the intermediate strategies of the RL curriculum but "only the final strategy learned"1.

Three limits, none hidden by the posts. First, it is an upper bound on what the signal can carry. The LoRA post says its capacity figures concern "the maximum amount of information absorbed in the long-training limit", not compute efficiency or learning speed, and that the one-bit figure "may only apply narrowly to policy gradient algorithms"12. Second, \(O(N)\) counts positions, not realized information. A token on which teacher and student agree carries an advantage near zero (tokens 1 to 3, 7 and 8 in Table 3), and in that example one token of eight holds 82% of the total absolute advantage (derived: \(2.862/3.498\)). The post's illustration agrees: the penalized tokens are "forking tokens" that start a wrong phrase1. Third, the sampled-token advantage is a single-sample estimate with mean \(-\mathrm{KL}_t\), noisy enough that DeepSeek-V4 replaced it (section 5.2). OPD's signal has far more degrees of freedom per episode than a scalar reward; it does not teach a bit per token.

4. The reported numbers, and what each one does and does not count

Two sources report that OPD reaches an RL-level or better result at a fraction of the cost: the Thinking Machines post (October 2025, with a June 2026 update) and the Qwen3 report (May 2025). They are not independent in the way the phrase "replicated" suggests. The post says it uses Tinker to "replicate Qwen3's result of achieving equivalent performance on reasoning benchmarks with on-policy distillation for a fraction the cost of RL", and it takes its RL cost from the Qwen3 report1. Each has its own accounting, and neither is a controlled study with repeated seeds.

4.1 Thinking Machines: Qwen3-8B-Base on math

The setting1: student Qwen3-8B-Base, off-policy initialization by SFT on OpenThoughts-3 (reasoning traces generated by QwQ-32B), benchmark AIME'24. Full fine-tuning on 400K prompts gives 60%. SFT performance then rises log-linearly in the number of prompts after the first 50K to 100K, and extrapolating that line the post estimates about 70% at about 2M prompts, with the caveat that this "requires the scaling law to hold up without stalling" and is "an optimistic estimate" for off-policy distillation. Two reference points bound it: DeepSeek-R1-0528-Qwen3-8B reaches 86% with an unstated number of prompts, and Qwen2.5-7B and 14B reach 55.5% and 69.7% after 800K R1 distillation prompts. For RL, the post takes the Qwen3 report's 67.6% at 17,920 GPU-hours and estimates that "given some reasonable assumptions about the SFT training stack" this is similar to the cost of 2M off-policy prompts, while saying it "is hard to compare this directly to the cost of distillation".

OPD from the 400K checkpoint "achieves an AIME'24 of 70% in about 150 steps", which the post equates to about 77K prompts at 4 samples per prompt. The teacher actually used was Qwen3-8B, "as it performs slightly better"; the FLOPs in the cost table are nonetheless those of the 32B model1. A LoRA run at rank 32 trails full fine-tuning by 13% after SFT but by only 6% after OPD1.

4.2 The FLOPs table and the 9×, 18×, 30×

Table 4. The Thinking Machines cost table for lifting Qwen3-8B-Base from 60% to about 70% on AIME'24, as printed1, with the last column derived here. "Teacher FLOPs" is the cost of generating the off-policy data for SFT rows, and of the log-probability pass for OPD.
MethodAIME'24 (%)Teacher (FLOPs)Student (FLOPs)CE vs SFT-2M (printed)Cost vs OPD, student+teacher (derived)
SFT-400K (initialization)608.5×10203.8×1020––
SFT-2M (extrapolated)~703.4×10211.5×10211×29.5× with teacher, 9.0× without
RL (Qwen3 report)68––≈1×not reported in FLOPs
On-policy distillation708.4×10198.2×10199 to 30×1×

The three multipliers are three accountings of one table. The post's definitions1:

  • 9×: "when the SFT dataset is given, as in our example with OpenThoughts-3, or is amortized across many training runs". The teacher FLOPs of off-policy training are not counted, but the teacher's log-probability pass is counted for OPD, "since we must run the teacher model to compute log-probs". Derived: \(1.5\times10^{21}/(8.4\times10^{19}+8.2\times10^{19})=9.04\).
  • 18×: the same comparison in GPU-hours, "since this computation can be cheaply parallelized across GPUs", so the teacher pass is effectively free. Derived: \(1.5\times10^{21}/8.2\times10^{19}=18.3\).
  • 30×: the task is new and no distillation dataset exists, so the SFT side also pays for teacher sampling. Derived: \((3.4\times10^{21}+1.5\times10^{21})/(8.4\times10^{19}+8.2\times10^{19})=29.5\).

All three use only the incremental OPD cost, leaving out the shared 400K SFT initialization (the post does not say so; the numbers only work out that way). One more observation from the table, mine: the SFT-2M row is 3.95 times (student) and 4.0 times (teacher) the SFT-400K row, which is what 1.6M additional prompts at the same per-prompt cost would give, not 2M. The row therefore appears to be the increment over the 400K checkpoint, the right quantity for a like-for-like comparison, but the label says 2M.

Can the FLOPs columns be reconstructed? A token-count check (derived)

The post gives no FLOPs-per-token formula. Use the usual dense convention: \(2N\) FLOPs per token for a forward pass, \(6N\) for forward plus backward (Module 1, section 5.3). For OPD the 8B student decodes and trains on each sampled token, \(8N_S=6.4\times10^{10}\) FLOPs; the 32B teacher does one forward pass, \(2N_T=6.4\times10^{10}\) FLOPs. The two printed columns then imply \(8.2\times10^{19}/6.4\times10^{10}=1.28\times10^{9}\) and \(8.4\times10^{19}/6.4\times10^{10}=1.31\times10^{9}\) tokens, within 2.5% of each other, so the columns are consistent with one token count of about \(1.3\times10^{9}\). Over 77K prompts × 4 samples (308,000 samples) that is about 4,200 tokens per sample; over 9,600 × 4 (38,400 samples) it would be about 34,000, a very long average for training rollouts that the post says run at shorter context than evaluation1. The FLOPs favor the 77K figure (my reading).

The SFT rows reconcile less well: at \(6N_S\) per token the student column implies about 19,800 tokens per prompt over 400K prompts, and at \(2N_T\) the teacher column implies about 33,000. The post gives no formula that explains the factor of 1.7.

The FLOP-to-time conversion is where an inference engineer should push. A FLOP ratio is a GPU-hour ratio only if every FLOP runs at the same utilization, and these do not. Take the split above, 2 of the 8 student FLOPs per token in sampling and 6 in training, price training and SFT at 30% MFU, and price decode at Olmo 3's measured 1.01% MFU (43.21% MBU) for its final inference configuration13 (Table 23). At equal utilization the arithmetic reproduces the post's numbers, 18.3× without the teacher pass and 9.0× with it. With decode at 1.01% it falls to 2.2× and 2.0×; at 5% decode MFU it is 8.1× and 5.6×; at 10%, 12.2× and 7.2× (all derived, illustrative, and sensitive to the 30% assumption). The post's 18× therefore rests on both the teacher pass being cheap and student sampling running about as efficiently as training, and OPD's own cost is dominated by sampling, as Module 1 shows for Olmo 3's RL run. The one measured GPU-hour comparison is Qwen3's RL versus OPD, where both arms pay for sampling.

Left: Qwen3-8B GPU-hours for reinforcement learning (17,920) and on-policy distillation (1,800), ratio 9.96. Right: the three cost-reduction multipliers in the Thinking Machines post, 9, 18 and 30 times, each with a different accounting Qwen3-8B, same starting checkpoint (GPU-hours) 0 5,000 10,000 15,000 RL 17,920 OPD 1,800 17,920 / 1,800 = 9.96× (derived) Thinking Machines cost reduction (CE) FLOPs, SFT data given9× GPU-hours18× incl. teacher sampling30× FLOP-based unless stated Qwen3 report: hardware and teacher cost not stated
Figure 4. Left: the Qwen3-8B comparison from the Qwen3 report, RL 17,920 versus on-policy distillation 1,800 GPU-hours7 (Table 21); the ratio is derived. Right: the three cost-effectiveness multipliers of the Thinking Machines post for reaching about 70% on AIME'24 from a 400K-prompt SFT checkpoint1. The two panels are different experiments with different starting points and different accounting; they are drawn together to show how much the multiplier moves with what is counted.

4.3 Qwen3's own comparison

The Qwen3 report is the origin of both the 17,920 and the 1,800. Its small models (0.6B, 1.7B, 4B, 8B, 14B and the 30B-A3B MoE) skip the four-stage pipeline. They get an off-policy phase on teacher outputs generated in both /think and /no think modes, then an on-policy phase in which the student generates in either mode and "is then fine-tuned by aligning its logits with those of a teacher model (Qwen3-32B or Qwen3-235B-A22B) to minimize the KL divergence"7 (§4.5). The report states the saving twice, as "only 1/10 of the GPU hours compared to the four-stage training method" for the whole strong-to-weak route (§4 introduction) and as the Table 21 experiment (§4.7)7.

Table 5. Qwen3 report Table 21: RL versus on-policy distillation on Qwen3-8B, both starting from the same off-policy-distilled 8B checkpoint, math and code queries only. Parentheses are pass@64. GPU-hours as printed; hardware not stated7 (p. 21). The last column is derived.
MethodAIME'24AIME'25MATH500LiveCodeBench v5MMLU-ReduxGPQA-DiamondGPU-hoursGPU-hours per AIME'24 point
Off-policy distillation55.0 (90.0)42.8 (83.3)92.442.086.455.6––
+ RL67.6 (90.0)55.5 (83.3)94.852.986.961.317,9201,422
+ on-policy distillation74.4 (93.3)65.5 (86.7)97.060.388.363.31,80093

Derived from the table: \(17{,}920/1{,}800=9.96\). Per point gained on AIME'24 over the shared starting checkpoint, RL spent \(17{,}920/12.6=1{,}422\) GPU-hours and OPD \(1{,}800/19.4=93\), a ratio of 15.3; on AIME'25 the figures are 1,411 and 79, a ratio of 17.8. Both are larger than 9.96 because OPD also ended higher. OPD led on all six benchmarks, by 6.8 and 10.0 points on AIME'24 and AIME'25 and by 1.4 to 7.4 points on the other four.

Three things the table does not tell you. First, whether the 1,800 GPU-hours include the teacher. The passage I read does not say, and it gives no hardware, so 9.96× is the ratio of the two numbers as printed, not a total-cost ratio. Treat it like the 18× of the Thinking Machines accounting, where the teacher pass is assumed cheap, and not like the 30×. Second, which teacher: §4.5 names Qwen3-32B or Qwen3-235B-A22B for the pipeline, and the Table 21 text does not say which one served the 8B experiment. Third, what the RL arm was. The report says "direct reinforcement learning" from the same checkpoint on math and code queries and gives no recipe for that arm in this comparison, so you cannot tell whether 17,920 GPU-hours is a tuned run or the flagship's stage-2 recipe applied to the 8B. The Thinking Machines post calls the RL result "similar" in cost to 2M SFT prompts and warns it is "hard to compare"1.

Qwen3-8B Table 21 as grouped bars: pass@1 and pass@64 on AIME 2024 and AIME 2025 for the off-policy distilled checkpoint, after RL, and after on-policy distillation AIME 2024 0 50 100 55.0 90.0 off-policy 67.6 90.0 + RL 17,920 GPU-h 74.4 93.3 + OPD 1,800 GPU-h AIME 2025 0 50 100 42.8 83.3 off-policy 55.5 83.3 + RL 17,920 GPU-h 65.5 86.7 + OPD 1,800 GPU-h pass@1 pass@64 30 questions per year, 64 samples each: pass@64 moves in steps of 3.3 points, one question
Figure 5. Table 21 of the Qwen3 report: starting from the same off-policy-distilled 8B checkpoint, RL raised pass@1 on AIME'24 from 55.0 to 67.6 and on AIME'25 from 42.8 to 55.5 with pass@64 unchanged (90.0 and 83.3); on-policy distillation raised pass@1 to 74.4 and 65.5 and pass@64 to 93.3 and 86.77; GPU-hours as printed, hardware and teacher cost not stated. The step size of pass@64 follows from the evaluation protocol of 30 questions and 64 samples per question7 (§4.6): 90.0, 83.3, 93.3 and 86.7 are 27, 25, 28 and 26 questions of 30 (derived).

The pass@64 column is the report's capability argument. RL left it unchanged at 90.0 and 83.3; OPD raised it to 93.3 and 86.7. The report reads this as distillation from teacher logits letting the student "expand its exploration space and enhance its reasoning potential", while RL "does not lead to any improvement in pass@64 scores"7 (§4.7). Figure 5 shows how thin the margin is. With 30 questions per benchmark, the OPD gain in pass@64 is exactly one question on each (27 to 28 of 30 on AIME'24, 25 to 26 of 30 on AIME'25), and the report gives no seeds or intervals (derived). The pass@1 gaps of 6.8 and 10.0 points between OPD and RL are much larger than that, but they too come from one run each. Whether RL expands or sharpens, and how to design a measurement that separates the two, is Module 20's question; this module only records that the one place where on-policy distillation and RL were compared at fixed checkpoint and reported pass@k found the two different, on a one-question margin.

4.4 Self-distillation: when the teacher is an earlier copy

The post's two remaining experiments change what the teacher is, and they show the cost advantage in a different light1.

Recovering behavior after mid-training. Qwen3-8B, already post-trained, is fine-tuned on internal company documents. Knowledge goes up, instruction following goes down, and no mix of document and chat data keeps the original IF-eval. OPD with the earlier Qwen3-8B as teacher, run on Tulu 3 prompts unrelated to the documents, restores most of it.

Table 6. Personalization experiment: internal-document QA (knowledge) and IF-eval (instruction following), Qwen3-8B1. Internal QA is the authors' own eval and is not reproducible from outside.
ModelInternal QA (%)IF-eval (%)
Qwen3-8B1885
+ mid-train, 100% documents4345
+ mid-train, 70% documents / 30% chat3679
+ mid-train 70/30, then OPD from earlier Qwen3-8B4183

Replaying an RL result. The post trains Qwen3-8B-Base with RL on DeepMath at LoRA rank 128, and uses the result as the teacher for OPD back into the base model. It reports that the distilled student reaches the teacher's level "in approximately 7-10x fewer gradient steps", that the reverse KL falls to near zero and the AIME score is recovered "in under 10 gradient steps, while RL took 70 steps". It then adds two savings (RL must train near the evaluation context length, while distillation "learns reasonably at shorter context lengths", and OPD tolerates smaller batches) and states a compute efficiency "on the order of 50-100x"1. The factors are not broken out; only the step ratio (70/10, at least 7) is checkable.

Two smaller results. Training on one math prompt for 20 steps at 256 rollouts each (5,120 graded sequences) "approximately" matches the teacher's AIME'24 score, less efficiently than the multi-prompt default1. And SFT on the model's own temperature-1 samples degrades IF-eval at every practical learning rate, which the post attributes to finite-batch noise turning self-training into off-policy drift, while OPD against a fixed teacher "always stays on-policy"1. These are single runs, partly on an internal setup, and I treat them as hypotheses with a plausible mechanism.

5. What the recipe needs: teacher, estimator, horizon

The cost results hold when three conditions hold. Later work, mostly 2026 preprints, tested each. All of it is recent and little of it is peer reviewed, which I flag where it matters.

5.1 The teacher must be reachable from the student

Reverse KL is mode-seeking within the student's support: the post notes that SFT "adds support for new tokens" and "reverse-KL methods can then perform mode seeking within the initialization's support", which is why its experiments start from a mid-trained or SFT'd student1. The MOPD paper tested what happens when that fails. Its student is Qwen3-30B-A3B, and its math teachers are RL-trained descendants of the same SFT checkpoint ("same-origin"). Swapping in Qwen3-235B-A22B, "a significantly larger model with stronger math capability", degraded the student8 (§4.4.2). Figure 6 shows the numbers: the initial per-token KL was about 5× higher (~0.19 versus ~0.04), entropy under the policy-gradient loss contracted from 0.30 to 0.21 as the student "narrows toward a single mode as it receives predominantly punitive gradient signal", and the top-k variant collapsed near step 18.

Left: AIME 2025 accuracy of a Qwen3-30B-A3B student after multi-teacher on-policy distillation of its math domain, with the same-origin RL teacher (51.46 and 51.77) versus Qwen3-235B-A22B as the math teacher (45.63 and 0.94). Right: initial per-token student-teacher KL, about 0.04 versus about 0.19 Math accuracy, AIME'25 (%) 0 20 40 60 51.46 same-originpolicy gradient 51.77 same-origintop-k (k=64) 45.63 235B teacherpolicy gradient 0.94 235B teachertop-k (k=64) Initial per-token KL 0.0 0.1 0.2 ~0.04 same-origin ~0.19 235B teacher Same-origin = math teacher obtained by RL from the student's own SFT checkpoint. Top-k diverged near step 18 with the 235B teacher.
Figure 6. The same-origin experiment of the MOPD paper on Qwen3-30B-A3B: replacing the math teacher (an RL-trained copy descended from the student's SFT checkpoint) with the larger, stronger Qwen3-235B-A22B lowers AIME'25 from 51.46 to 45.63 under the policy-gradient loss and to 0.94 under the top-k loss, which collapsed around step 18. The initial per-token student-teacher KL was about 0.19 versus about 0.04, roughly 5×8 (Table 4, Figure 3, §4.4.2). Units of the KL are not stated in the passage I read.

The tension with Qwen3's own recipe is real and has a reading. Qwen3 distills 235B-A22B and 32B into its 30B-A3B, but only after an off-policy phase that fine-tunes the student on the teacher's outputs7 (§4.5). That phase moves the student into the teacher's support before the on-policy phase starts. The MOPD swap did not do that. Li et al. test the same idea from the other side and report that an "off-policy cold start" and "teacher-aligned prompt selection" recover failing OPD runs14. I read these as one finding: OPD needs low initial divergence, and an SFT stage on the teacher's samples is how you buy it. That is my synthesis, not a claim any of the three papers makes in those words.

Li et al. add that benchmark rank is not what matters. Same-pipeline teachers helped little and RL-post-trained ones much more, measured as the fraction of the student-teacher gap recovered: 5.3% versus 16.9% in the DeepSeek-distill family (student R1-Distill-1.5B) and 15.6% versus 58.6% in the Qwen3 family (student Qwen3-1.7B non-thinking), even though the RL teachers had slightly lower initial token overlap with the student (71.5% versus 74.7%; 70.3% versus 75.7%)14 (Figure 4). In a reverse experiment, JustRL-1.5B (an RL-improved model) as student was distilled from its own pre-RL checkpoint and "regress[ed] almost exactly to its pre-RL performance"; with R1-Distill-7B as teacher, a larger model that scored slightly higher than the student on benchmarks, it regressed to the same level14 (§3.3). That follows from the objective: the minimizer is the teacher's local distribution, so a teacher that sits in the wrong place pulls the student there whatever its score. In the authors' words OPD "fundamentally learns thinking patterns"; they also report that 97% to 99% of the probability mass sits in a small token set shared by student and teacher in successful runs14.

5.2 The sampled-token estimator is cheap and noisy

The estimator question is how much of the teacher's distribution each position returns. Sampled-token needs one number per token. Exact reverse KL needs \(|V|\) numbers, and Qwen3's vocabulary is 151,6697 (§2). In fp32 that is \(151{,}669\times4=606{,}676\) B, about 592 KiB per token per teacher; a batch of 1M tokens would hold about 607 GB of logits if materialized, against 4 MB for sampled-token log-probabilities (derived; real implementations chunk). Three groups made three choices:

  • DeepSeek-V4 says the sampled-token form is "resource-efficient" but "leads to high variance in gradient estimation and often causes training instability", so it uses full-vocabulary logit distillation, made feasible by caching only the teacher's last-layer hidden states, rebuilding logits through the prediction head on the fly, ordering samples by teacher index so at most one teacher head is resident, and computing the exact KL in a TileLang kernel2 (§5.1.2, §5.2.2). The passage I read gives no ablation behind the instability claim.
  • The MOPD paper compares its policy-gradient (sampled-token) form against top-\(k\) with \(k=64\) on same-origin teachers and finds them "comparable", normalized scores 0.937 versus 0.909, and says top-\(k\) "does not yield additional stability or variance reduction"8 (§4.4.1).
  • Fu et al. replace the sampled token with teacher top-\(K\) local support matching (a truncated reverse KL over the teacher's \(K\) most likely tokens, renormalized on both sides), plus top-\(p\) rollouts and special-token masking, and report "+19.8%". That is the relative gain in average math score, 34.8 to 41.7, in an alternating math and ALFWorld run. In single-task math the average is 36.4 for sampled-token OPD, 40.7 with special-token masking alone and 41.7 with local support matching, so masking accounts for about 81% of the gain (derived: 4.3 of 5.3 points)9 (Tables 1 and 2). The student is Qwen2.5-7B-Instruct and the teacher OpenThinker3-7B, not a same-origin pair; the paper is marked "Under review".

These are compatible if variance matters most when teacher and student are far apart or rollouts are long, which is where Fu et al. and V4 operate, and matters little when the teacher is a close descendant, which is the MOPD setting. That is my reading; no paper runs all three on one setup. Table 7 collects the failure modes the sources report and the fix each proposes.

Table 7. Reported failure modes of OPD, with the measurement and the proposed fix. All sources except the Thinking Machines post are 2026 preprints; the Li et al. and Shao et al. results are the authors' own.
FailureWhat was measuredProposed fixSource
Teacher far from studentinitial per-token KL ~0.19 vs ~0.04; entropy 0.30 to 0.21; top-k collapse near step 18same-origin teacher; off-policy cold start; teacher-aligned promptsMOPD8; Li et al.14
Imbalanced sampled-token signalmost sampled tokens receive negative reward (Figure 2 of the source)teacher top-\(K\) local support matchingFu et al.9
Repetition not penalizedteacher "locally aligned" with the student's repeated tokenstop-\(p\) rollouts; teacher-mixed sampling (\(\alpha\) mix-in)Fu et al.9; MiniLLM6
Tokenizer mismatchstudent's <think> split into three tokens: teacher log-probability −19.16 against student −0.07; end-of-sequence −58.71 against −0.00special-token masking; shared tokenizer (Lambert notes KD "often" requires one)Fu et al.9; Lambert3
Long responsesbest results at 3K to 7K maximum length; 10K and 15K collapse late; teacher's accuracy advantage over the student's continuation falls from +0.37 at a 1K prefix to +0.02 at 16Kcap length; open problem for agentic horizonsLi et al.14 (§6.1)
Gradient concentrated on surprising tokensgradient L1 norm factors into \(2|\Delta_t|\) and \((1-p_S)\) for the \(k_2\) loss; low-student-probability tokens carry a disproportionate sharesurprise-aware reweighting (SuRe), up to +6.7 points on AIME'24 pass@8 for a 1.7B studentShao et al.11

5.3 Horizon, and what stays outside this module

Li et al.'s long-response result limits the recipe where RL is most valuable. The teacher's grade depends on a prefix the student wrote, and the further it drifts from what the teacher would produce, the less informative the grade; instability starts at the end of long responses and propagates backward, and the authors conclude that OPD "may not extend cleanly" to extended chain-of-thought or agentic multi-turn interaction14 (§6.1). That is a statement about their setup, a 1.5B student on math. Mismatch between the engine that produces \(\log\pi_\theta\) and the one that trains is Module 14's subject; multiple teachers, prompt routing and V4's teacher scheduling are Module 22.

6. When distillation, not RL, is the cheaper route

6.1 The comparisons that exist

Table 8. The head-to-head evidence between RL and distillation. "Distillation" in the R1 and Yue et al. rows means off-policy SFT on teacher samples, not OPD; only the Qwen3 and Thinking Machines rows are OPD. Accounting differs by row.
SourceSettingResultCost as reported
DeepSeek-R1, Supp. F.1, Table 1615Qwen2.5-32B; RL (Qwen2.5-32B-Zero, over 10K steps) versus SFT on 800K R1 samplesAIME'24 pass@1: QwQ-32B-Preview 50.0, RL 47.0, distilled 72.6; GPQA-Diamond 54.5, 55.0, 62.1; LiveCodeBench 41.9, 40.2, 57.2RL "require[s] enormous computational power"; no GPU-hours given in the table
Qwen3, Table 2178B, same checkpoint; RL versus OPDAIME'24 67.6 versus 74.4; pass@64 90.0 versus 93.317,920 versus 1,800 GPU-hours (teacher cost not stated)
Thinking Machines1Qwen3-8B-Base, from SFT-400K (60%)OPD 70% at about 150 steps; RL 68% (from the Qwen3 report); SFT-2M extrapolated 70%9× to 30× in FLOPs; 18× in GPU-hours
Yue et al., §4.216Qwen2.5-Math-7B; base, RLVR, R1-distilleddistilled model's pass@k curve "consistently and significantly above" the base; RLVR bounded by the basenot compared
ProRL171.5B from R1-Distill-Qwen-1.5B; RL for over 2,000 stepsgains over the start of +15.7% (math), +14.4% (code), +25.9% (GPQA-Diamond), +22.0% (IFEval) and +54.8% (Reasoning Gym reward), from the body and Tables 1-3; the introduction prints 14.7%, 13.9%, 25.1% and 18.1% for the first four domainsabout 16K GPU-hours on 4×8 H100

Read as a set, the rows agree on one thing and leave one thing open. They agree that for a small student with a stronger teacher available, a teacher-supplied signal beats RL from the same start on both quality and cost. They leave open whether RL, run long enough from a distilled start, passes what distillation reaches: ProRL's 16K GPU-hours is the same order as Qwen3's 17,920 for RL on the 8B, and ProRL claims gains "inaccessible to base models, even under extensive sampling"17, while R1's authors close the comparison by saying "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning"15. Module 20 weighs the capability evidence.

6.2 Break-even: when a teacher pays for itself

The quoted ratios hide that a teacher is a sunk cost somebody paid. If one teacher serves \(n\) students, per-student cost is what you compare; if you must build it, the break-even is \(n^*=C_T/(C_{\mathrm{RL}}-C_{\mathrm{OPD}}(1+o))\), with \(C_T\) the teacher's build cost, \(C_{\mathrm{OPD}}\) the per-student OPD cost, \(C_{\mathrm{RL}}\) the per-student RL cost, and \(o\) the teacher-pass overhead as a fraction of the OPD cost. Qwen3 amortizes teachers over six small models (five dense and one MoE)7; the calculator below defaults to the 8B figures and an unknown, hence zero, teacher cost.

Distillation versus RL, with the teacher counted

RL, all students (GPU-h)
OPD, all students plus teacher (GPU-h)
Per-student ratio RL / OPD
Break-even number of students

Formulas: \(n C_{\mathrm{RL}}\); \(C_T+nC_{\mathrm{OPD}}(1+o)\); \(C_{\mathrm{RL}}/(C_{\mathrm{OPD}}(1+o))\); \(C_T/(C_{\mathrm{RL}}-C_{\mathrm{OPD}}(1+o))\). Defaults are the Qwen3-8B figures of Table 57; the report states neither the teacher cost nor whether the 1,800 includes it. Setting \(C_T\) to the cost of an RL run for the teacher shows why Thinking Machines' 50-100× applies to copies, not to discovery.

6.3 Decision inputs

Table 9. Which route the evidence favors, by situation. These are my readings of the cited results, not rules any source states; the decision framework for the whole course is Module 33.
SituationCheaper or better routeEvidence
Stronger teacher exists, student already SFT'd on its outputs, same tokenizerOPDQwen3 Table 217; Thinking Machines1
Teacher is a different, larger model and the student is coldoff-policy SFT first, then OPD; OPD alone can degradeMOPD §4.4.28; Li et al.14
You must restore a behavior lost in fine-tuning, and an earlier checkpoint has itOPD with that checkpoint as teacherTable 61
No model stronger than your target existsRL; OPD cannot exceed its teacherR1 Supp. F.115
Student and teacher use different tokenizersavoid sampled-token OPD, or mask special tokensFu et al.9
Responses of 10K tokens or more, or multi-turn agent episodesno OPD evidence at this length in the sources readLi et al.14

Where experts disagree

The question the sources contest is where a fixed budget goes when you want a strong small reasoning model: RL on the model, or a teacher's signal. A second question, which estimator to use, is covered in section 5.2.

Distill first; RL costs more for less

DeepSeek-R1 authors, Supp. F.115; Qwen3 team7; Thinking Machines1

R1 trained Qwen2.5-32B with RL for over 10K steps and matched only QwQ-32B-Preview, while SFT on R1's samples scored far higher. Qwen3 and the post report OPD beating RL at about a tenth of the GPU-hours, with higher pass@64 in Qwen3's case.

  • Every direct comparison favors the teacher signal on cost and on pass@1.
  • Dense signal works at smaller batches and shorter context than RL needs1.
  • The ratios omit or do not state the teacher's cost, and the 30× version is the only one that counts teacher sampling.
  • Single runs without seeds; R1's "distillation" is off-policy SFT.

RL is still needed to pass the teacher, and it works from a distilled start

DeepSeek-R1 authors, same passage15; ProRL (NVIDIA)17

R1's own second conclusion is that "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning". ProRL runs over 2,000 RL steps from a distilled 1.5B checkpoint, with KL control, reference resets and diverse tasks, and reports gains across five domains.

  • OPD's optimum is the teacher, so OPD alone cannot exceed it.
  • ProRL shows large RL gains on top of distillation at about 16K GPU-hours17.
  • ProRL's baseline is its own start, not OPD from a stronger teacher.
  • The claim that these gains are new capability is contested (Module 20).

OPD is conditional: it moves a student toward a close teacher, nothing more

Li et al.14; the MOPD paper (Xiaomi)8; Fu et al.9

A larger, stronger teacher failed when it was distributionally far from the student; same-pipeline teachers gave small gains; a pre-RL checkpoint erased an RL student's gains; long rollouts destabilized training.

  • Explains the failures that the headline ratios hide, and offers fixes (cold start, same-origin teachers).
  • Small models, mostly math, 2026 preprints, one marked "under review"; none measures cost against RL.
Assessment

The positions answer different questions, so they mostly do not collide. Below the frontier, with a teacher that exists, shares the tokenizer, and sits close to the student after an SFT stage, OPD is the cheaper route in every comparison here, by 9.96× on the one that reports GPU-hours, a ratio that may overstate the saving because the report does not say whether the teacher is in the 1,800. At or past the frontier, or with no teacher stronger than the target, RL is the only route, and the third position says the conditions for the first are narrower than the headlines suggest. The evidence that would settle it is a study the sources do not contain: one starting checkpoint, teacher build and serving cost counted, RL and OPD each given matched total GPU-hours, several seeds, and pass@k at large k over more than 30 problems.

What is changing

The reproducibility footing is thin. The post's June 2026 update says Qwen3-32B and Qwen3-8B-Base "have since been retired from the Tinker model lineup" and that the cookbook recipes now use Qwen3.5-9B-Base as the student and Qwen3.5-9B as the teacher; the printed numbers come from the original models1. The analysis literature is moving faster than the recipe: Fu et al. (March 2026, revised April), Li et al. (April 2026) and Shao et al. (August 2026, third version October) all post-date the post and all are preprints91411. In production the recipe has moved from a single teacher to OPD as a final stage with many teachers (DeepSeek-V4's more than ten2), which is Module 22. A separate branch drops the external teacher: Lambert describes on-policy self-distillation, where one model with privileged information in its context, such as a hint or a judge's feedback, acts as its own teacher, and attributes a use of it to Cursor's Composer 2.53 (§12.3.3). I have not checked Cursor's own account.

References

  1. blogLu, K. and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 27 Oct 2025; June 2026 update note on retired models. https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-27. Used for: taxonomy table, per-token reverse KL objective, discount factor zero, pseudocode, Qwen3-8B-Base experiments, FLOPs table and the 9x/18x/30x accounting, personalization table, 7-10x step and 50-100x compute claims, single-prompt run, June 2026 update.
  2. paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-27. Used for: mixed RL replaced by OPD (§5.1), full-vocabulary OPD and the stated reason (§5.1.2, Eq. 29), hidden-state caching and teacher-head scheduling (§5.2.2), more than ten teachers.
  3. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12 (11 Sep 2026). https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-27. Used for: forward-KL identity for SFT-style distillation (§12.3.1, p. 150), exposure bias and the DAgger analogy (§12.3.2, pp. 150-152), advantage as negative per-token reverse KL, combination with GRPO advantages and the shared-tokenizer remark (§12.3.3, p. 153), on-policy self-distillation and the Cursor attribution.
  4. paperRoss, S., Gordon, G. J., Bagnell, J. A. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. arXiv:1011.0686v3, 16 Mar 2011 (AISTATS 2011). https://arxiv.org/abs/1011.0686. Accessed 2026-10-27. Used for: the \(T^2\epsilon\) bound for supervised imitation (Theorem 2.1) and DAgger's \(uT\epsilon_N\) bound (Theorem 3.2).
  5. paperAgarwal, R., Vieillard, N. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD). ICLR 2024; arXiv:2306.13649v3, 17 Jan 2024. https://arxiv.org/abs/2306.13649. Accessed 2026-10-27. Used for: on-policy KD without backpropagating through sampling (§3.1, Eq. 4), the student-data fraction \(\lambda\), divergence choices, relative gains of 2.1x, 1.7x and 1.9x (§1, Figure 1).
  6. paperGu, Y., Dong, L., Wei, F., Huang, M. MiniLLM: On-Policy Distillation of Large Language Models. arXiv:2306.08543v6, 31 Jan 2026. https://arxiv.org/abs/2306.08543. Accessed 2026-10-27. Used for: reverse-KL policy gradient with reward-to-go (§2.2, Eq. 2), single-step decomposition, teacher-mixed sampling and length normalization, reward hacking by degenerate text, the mode-seeking argument (§2.1).
  7. paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-27. Used for: vocabulary size 151,669 (§2), strong-to-weak distillation, off-policy then on-policy phases, teacher choices, 1/10 GPU-hours (§4 introduction, §4.5), evaluation protocol of 30 AIME questions with 64 samples (§4.6), RL versus on-policy distillation (§4.7, Table 21, p. 21).
  8. paperMa, W. et al. (Peking University and Xiaomi LLM Core). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-27. Used for: per-token reverse KL loss, policy-gradient and top-k forms with clipped advantage (§3.2, Eqs. 1-5), teacher prefill (§3.3), same-origin teacher experiment (§4.4.2, Table 4, Figure 3), policy-gradient versus top-k comparison (§4.4.1).
  9. paperFu, Y. et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv:2603.25562v2, 27 Apr 2026 (preprint, "under review"). https://arxiv.org/abs/2603.25562. Accessed 2026-10-27. Used for: the \(\gamma\)-interpolated estimator and the \(O(T^2)\) versus \(O(T^4)\) variance bounds (§2.1, Eqs. 1-3, Appendix D), three failure modes and the tokenizer example (§2.2, Figures 2-5), local support matching (§3), the +19.8% claim (abstract).
  10. paperTang, Y., Munos, R. On a few pitfalls in KL divergence gradient estimation for RL. arXiv:2506.09477v1, 11 Jun 2025. https://arxiv.org/abs/2506.09477. Accessed 2026-10-27. Used for: sequence-level versus token-level KL gradient in the 70B-to-8B on-policy distillation experiment (§5.2-§5.3, Figure 8, Appendix E).
  11. paperShao, B. et al. (Fudan University). A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation. arXiv:2608.25643v3, 13 Oct 2026 (preprint). https://arxiv.org/abs/2608.25643. Accessed 2026-10-27. Used for: gradient of the per-token \(K_2\) estimator, its direction matching the detached sampled-token reward of Kimi K3 (which clips it), the \(2|\Delta_t|(1-p_S)\) factorization, SuRe result (abstract, Figure 1).
  12. blogSchulman, J. and Thinking Machines Lab. LoRA Without Regret. Thinking Machines Lab: Connectionism, 29 Sep 2025. https://thinkingmachines.ai/blog/lora/. Accessed 2026-10-27. Used for: the O(1) bits per episode argument for policy gradient, 320,000 bits for the MATH run, 5.3M episodes for R1-Zero, the stated limits of the argument (§"How much capacity is needed by supervised and reinforcement learning?").
  13. paperOlmo Team. Olmo 3. arXiv:2512.13961v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-27. Used for: inference-side MFU 1.01% and MBU 43.21% in the RL infrastructure benchmark (Table 23).
  14. paperLi, Y. et al. (Tsinghua University). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016v2, 15 Apr 2026 (preprint). https://arxiv.org/abs/2604.13016. Accessed 2026-10-27. Used for: thinking-pattern consistency and "new knowledge" conditions (§3), gap-recovery rates and initial overlap (Figure 4), reverse distillation (§3.3, Figure 5), overlap-token mechanism (§4), cold start and teacher-aligned prompts (§5), reward quality versus trajectory depth (§6.1).
  15. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (expanded Nature version), 4 Jan 2026. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-27. Used for: distillation versus RL on Qwen2.5-32B, over 10K RL steps, Table 16 (Supp. F.1), the two conclusions drawn there.
  16. paperYue, Y. et al. (Tsinghua University). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837v5, 24 Nov 2025. https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-27. Used for: distilled R1-Distill-Qwen-7B pass@k above the base model and above RLVR (§4.2, Figure 7).
  17. paperLiu, M. et al. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-27. Used for: RL from DeepSeek-R1-Distill-Qwen-1.5B, over 2,000 steps, about 16K GPU-hours on 4 nodes of 8 H100 (§3.2), per-domain gains from the body text and Tables 1-3 (§3.3), with the differing figures of the introduction (§1), the claim of strategies inaccessible to base models (abstract).