Reinforcement Learning for LLM Post-Training · Module 13
Training Dynamics: Entropy Collapse, KL Drift, Length Growth, and the Controls That Work
A long RL run on a verifier is not a smooth climb: entropy falls within a few hundred steps, KL to the start drifts, response length grows (sometimes into repetition), and each has a documented mechanism and a documented fix with limited evidence behind it. This module derives the covariance mechanism behind entropy collapse, sets the interventions side by side with the models they were tested on, lays out the keep-or-drop-KL ledger, and ends with a run-health guide. After it you can read a run's entropy, KL, length, clip-fraction and zero-advantage curves and say which intervention a report would have tried, and how far to trust it.
1. Entropy collapse as a measured curve
1.1 What is being measured
Policy entropy in these papers is the entropy of the next-token distribution, averaged over the generated positions of the training prompts: \(H(\pi_\theta,\mathcal D)=-\mathbb E_{x\sim\mathcal D,\,y\sim\pi_\theta}\big[\tfrac1{|y|}\sum_t\log\pi_\theta(y_t\mid y_{<t},x)\big]\)1 (§2.1, Eq. 5). TRL logs the same quantity from the logits as entropy6. Two choices change the number you read. The logits: TRL builds its fused log-probability head with the same temperature it passes to generation7, so with \(T\ne1\) the reported entropy is that of the distribution actually sampled; for logits \((2,1,0,-1)\) it is 0.588, 0.948 and 1.051 nats at \(T=0.6,\,1.0,\,1.2\) (derived), so a temperature change alone moves the logged curve. The prompts: Cui et al. measure on training batches1, ScaleRL on a held-out validation set8 (App. A.12).
import torch
def token_stats(logits, tokens, adv, temperature=1.0):
"""logits [N, V] at generated positions; tokens [N]; adv [N] per-token advantage."""
logp = torch.log_softmax(logits.float() / temperature, dim=-1) # the distribution that was sampled
ent = -(logp.exp() * logp).sum(-1) # H(pi(.|s_t)) in nats, per position
lp_tok = logp.gather(-1, tokens[:, None]).squeeze(-1) # log pi(y_t | s_t)
cov = (lp_tok - lp_tok.mean()) * (adv - adv.mean()) # Cui et al. Eq. 10, per token
return ent, lp_tok, cov
ent is the quantity of §1.2–§1.3 and cov the per-token term of §2 and §3.3. Checked against a pure-Python softmax on random logits.
1.2 The curve, in numbers
Cui et al. ran 11 base models in four families (Qwen2.5 from 0.5B to 32B, three Mistral models, two LLaMA models, DeepSeekMath-7B) on math and code with GRPO, RLOO, REINFORCE++ and PRIME. The runs had no entropy bonus and no reference-KL term (KL coefficient 0, \(\epsilon=0.2\)), 512 prompts with 8 responses each, and all-correct or all-wrong prompts filtered1 (§2.2). Averaged over 2,400-step runs, 73% of the entropy consumed and 76% of the performance gained occurred in the first 200 gradient steps, one twelfth of training; by step 800, one third, the figures are 94% and over 93%, so "over 2/3 of the training steps yielded marginal returns"1 (§2.3, Fig. 2). DAPO's naive PPO and GRPO baselines show the same shape on Qwen2.5-32B: entropy "decreases quickly", and the sampled responses of some prompts become "nearly identical"2 (§3.1, Fig. 2b).
1.3 The empirical law and what it implies
Across those runs, validation accuracy \(R\) and entropy \(H\) fit \(R=-a\,e^{H}+b\) with two coefficients over more than 200 data points1 (§2.4, Eq. 6). Three consequences follow by calculus (derived, not stated in that form by the paper):
- Ceiling. At \(H=0\), \(R=b-a\): the policy has nothing left to explore and the curve ends. The paper states this ceiling as "fully predictable"1 (abstract).
- Diminishing price. \(dR/dH=-a\,e^{H}\), so one nat of entropy spent at \(H=1\) buys \(e\,a\approx2.72a\) of performance at the margin, and at \(H=0.1\) it buys \(1.105a\): a factor of 2.46 less.
- Headroom is proportional to what is left. \(R(0)-R(H)=a\,(e^{H}-1)\approx aH\) for small \(H\). When entropy is down to a tenth of a nat, the fit predicts about \(0.105a\) of performance still to come.
The authors use the form as a forecaster: coefficients fitted on the first 36 steps of the Qwen2.5 runs predicted the next 200 steps with root-mean-square error 0.9% (math) and 1.2% (code)1 (§2.4, Fig. 5). The coefficients did not depend on the algorithm (GRPO, RLOO, PRIME and REINFORCE++ gave one curve on Qwen2.5-7B) and varied log-linearly with parameter count across the Qwen2.5 family1 (§2.5, Figs. 6–7).
2. Why entropy falls: a covariance between probability and advantage
2.1 The derivation
Take one state (a prefix) with a softmax policy over next tokens, \(\pi_a=e^{z_a}/\sum_b e^{z_b}\), and entropy \(H=-\sum_a\pi_a\log\pi_a\). The derivative of entropy with respect to a logit is \(\partial H/\partial z_k=-\pi_k(\log\pi_k+H)\). Because \(\mathbb E_\pi[\log\pi]=-H\), a first-order change of the logits \(\Delta z\) therefore changes entropy by
$$ \Delta H\;\approx\;-\sum_k \pi_k\,(\log\pi_k+H)\,\Delta z_k\;=\;-\,\mathrm{Cov}_{a\sim\pi}\big(\log\pi_a,\;\Delta z_a\big). $$This is Lemma 1 of Cui et al., adapted from Liu (2025)1 (§3.1). It holds for any update and says only that entropy falls when the tokens whose logits rise are already the likely ones. What the update does to the logits depends on the algorithm. For a tabular softmax with centered advantages (\(\mathbb E_\pi[A]=0\)), the gradient of \(J=\mathbb E_\pi[A]\) with respect to \(z_k\) is \(\pi_k A_k\), so one policy-gradient step of size \(\eta\) gives \(\Delta z_k=\eta\,\pi_kA_k\) (their Proposition 1), and a natural-gradient step gives \(\Delta z_k=\eta A_k\) up to a constant that softmax ignores. Substituting:
$$ \Delta H_{\mathrm{PG}}\approx-\eta\,\mathrm{Cov}_{a\sim\pi}\big(\log\pi_a,\;\pi_a A_a\big),\qquad \Delta H_{\mathrm{NPG}}\approx-\eta\,\mathrm{Cov}_{a\sim\pi}\big(\log\pi_a,\;A_a\big). $$These are their Theorems 1 and 21 (§3.2). The sampled-token update that a trainer actually applies, \(\Delta z=\eta A_y(e_y-\pi)\) for a drawn token \(y\), has expectation \(\eta\pi_k(A_k-\mathbb E_\pi A)=\eta\pi_kA_k\), which is Proposition 1 again (derived). The abstract states the reading: "a high-probability action with high advantage would reduce policy entropy, while a rare action with high advantage would increase policy entropy"1. One refinement the algebra makes precise: "high probability" means above the state's own average log-probability, \(\pi_a>e^{-H(s)}\), the inverse perplexity of that state.
The derivation assumes a tabular policy, a small step and the exact expected gradient. A transformer shares parameters across positions, the optimizer rescales steps and PPO clipping removes some tokens, so the formula is a diagnosis, not a simulator; the paper's check is empirical (§2.4 below).
2.2 A three-token state, worked
Let the policy at one position be \(\pi=(0.70,\,0.25,\,0.05)\). Then \(H=0.7460\) nats and the "likely" threshold is \(e^{-H}=0.474\): only the first token is above it. Suppose a verifier rewards exactly one token with \(Q=1\) and the others get 0, so \(A_a=Q_a-\mathbb E_\pi Q\). Take one policy-gradient step with \(\eta=1\), \(\Delta z_a=\pi_aA_a\). Table 1 compares the first-order prediction with the exact entropy after the step (derived; the script checks the first-order formula against exact softmax recomputation).
| Rewarded token | \(A\) (token 1, 2, 3) | \(\Delta z=\pi A\) | New \(\pi\) | First-order \(\Delta H\) | Exact \(\Delta H\) |
|---|---|---|---|---|---|
| 1 (p = 0.70, above threshold) | +0.30, −0.70, −0.70 | +0.210, −0.175, −0.035 | 0.770, 0.187, 0.043 | −0.0892 | −0.0957 |
| 2 (p = 0.25, below threshold) | −0.25, +0.75, −0.25 | −0.175, +0.188, −0.013 | 0.626, 0.321, 0.053 | +0.0763 | +0.0669 |
| 3 (p = 0.05, rare) | −0.05, −0.05, +0.95 | −0.035, −0.013, +0.048 | 0.693, 0.253, 0.054 | +0.0129 | +0.0130 |
Same state, same reward size, three entropy outcomes. Rewarding the dominant token costs 0.096 nats in one step, 12.8% of the state's entropy at \(\eta=1\); rewarding the 0.25 token raises it by 0.067. Rewarding the rare token raises entropy by only 0.013 even though its advantage is 0.95, because the policy-gradient step scales the logit change by \(\pi_a\): the rare token's logit moves by 0.048 while the dominant token's moves by 0.210 on an advantage three times smaller. A natural-gradient step (\(\Delta z=\eta A\)) would move the rare token by 0.95 and raise entropy by 0.135, ten times as much, so the optimizer's geometry matters as well as the sign (derived; first-order +0.1125).
2.3 Four quadrants
Each sampled token adds \((\log\pi-\overline{\log\pi})(A-\bar A)\) to the covariance, and the net sign decides the direction of the entropy change. Figure 3 sorts the four cases. Rewarding a likely token and penalizing a rare one are the ordinary business of reinforcement, and both sharpen the policy; rewarding a rare token and penalizing a likely one spread probability mass out. Early in a run the policy's confidence is informative, so the sharpening quadrants dominate: Cui et al. observe a large positive covariance at the start, read it as confidence being "well-calibrated", and find it declines but stays positive, "continually dragging policy entropy even lower"1 (§1, §3.3).
2.4 What was measured
The paper checks Theorem 1 on Qwen2.5-7B with on-policy GRPO (no PPO surrogate), treating the whole response as the action, normalizing log-probability by length (their Eqs. 8–9), and averaging covariance per prompt group. The empirical \(-\Delta H\) and the covariance "exhibit highly similar dynamics"; covariance stays positive for the whole run, so entropy falls monotonically1 (§3.3, Fig. 8). Split by prompt difficulty, covariance is smaller for hard prompts (accuracy 0.125) and larger for easy ones (0.875): when the model struggles, "high-probability actions are not reliably associated with higher expected returns"1. The writer's inference is that an easy training set should therefore collapse entropy faster; the paper does not run that comparison.
The token-level distribution is heavy-tailed. At step 1 of the same run the mean covariance per token is 5.654 in the top 0.02% of tokens, 3.112 in the top 0.2%, 1.385 in the top 2%, 0.351 in the top 20%, 0.152 in the top 50%, and 0.003 over all tokens1 (Table 1). The top 2% alone contribute \(0.02\times1.385=0.028\) per token of the batch, nine times the all-token mean (derived; the paper rounds that mean to 0.003), so the net covariance is a small residual of a large positive tail minus a negative bulk (the second half is the writer's inference). Controlling that tail is what Clip-Cov and KL-Cov do (§3.3).
3. Controls on the entropy side, and the evidence for each
3.1 The textbook tools: an entropy bonus and a reference KL
The default RL answer to collapsing entropy is a bonus term, \(\mathcal L_{\mathrm{ent}}=\mathcal L-\alpha H(\pi_\theta)\). Cui et al. tried \(\alpha\in\{10^{-4},10^{-3},5\times10^{-3},10^{-2}\}\): the two smallest had "a minor influence on entropy", 0.01 produced "entropy explosion", and 0.005 stabilized entropy but "does not outperform other baselines". A KL penalty to the reference (0.001 to 0.1) held entropy stable but "fails to improve policy and instead leads to a degradation in performance"1 (§4.1, Figs. 9–10). Magistral saw the fragility from another side: the same coefficient made entropy fall on a math-only set and rise "excessively" on a math-and-code set, and a KL term, even against an exponential moving average of the weights, "primarily hinders training"5 (§7.4.2, Fig. 12).
The bonus is not gone. TRL ships a static one (entropy_coef, default 0.0) and an adaptive one "introduced in Skywork-OR1": the coefficient rises by entropy_coef_delta per optimizer step while logged entropy is at or below entropy_target and falls otherwise, so the bonus engages only near collapse6. That turns a sensitive constant into a feedback loop, the move Ziegler et al. made for the KL coefficient (Module 6 §4.3); I found no ablation of it in the sources read. A trap in its docs: the default entropy_target is 0.2, while the docs say typical models have 2–10 nats and use 5.0 in their example6; DAPO's Qwen2.5-32B run plots entropy on a 0.3 to 0.7 axis (Fig. 7c)2. Calibrate the target on your model's early-run entropy (§6.2).
3.2 Clip-Higher and dead-group filters
DAPO's diagnosis is that the upper clip, not the entropy bonus, is what throttles exploration. With \(\hat A>0\) a likely token can climb almost to 1 while a token at 0.01 is stopped at 0.012, and the up-clipped tokens have a mean probability below 0.22 (§3.1, Fig. 3a). The fix raises only the upper bound, \(\epsilon_{\mathrm{high}}=0.28\) with \(\epsilon_{\mathrm{low}}=0.2\), because a higher lower bound would "suppress the probability of these tokens to 0, resulting in the collapse of the sampling space"2. Module 3 (Table 4) has the headroom arithmetic and Module 12 (§2.3) the evidence on how the constant interacts with off-policy reuse; Figure 5 adds the frontier at which the up-clip can bind at all, and the two larger values later reports used.
Four reports converged on this lever with different values. Olmo 3 uses 0.2 and 0.272 in all four RL runs9 (Table 49); GLM-5 uses 0.2 and 0.2810 (§3.2). Magistral Medium adjusted \(\epsilon_{\mathrm{high}}\) "between 0.26 and 0.28 during the training to keep the group entropy stable", and Magistral Small, which starts from a cold-started SFT checkpoint, used 0.3 "as the cold-started model yielded responses with far lower entropy"5 (§2.1, §5.3). ProRL, from a distilled 1.5B model, used 0.43 (§3.2). The two largest values belong to the two reports that start from low-entropy checkpoints, and one says why; two data points are a pattern to test, not a rule. Qwen3 states only the target: its reasoning RL controls entropy "to increase steadily or remain stable"11 (§4.2).
How much does it buy? In DAPO's ladder on Qwen2.5-32B (AIME 2024 avg@32), Clip-Higher moves 36 to 38 points2 (Table 1; Module 11 carries the ladder). In Cui et al.'s runs it beat plain GRPO by 0.2 average points on Qwen2.5-7B and 1.4 on 32B, and "although it can also increase entropy and lead to performance improvement in the early stage of training, it gradually becomes unstable, with performance saturating and declining"1 (§4.3, Table 2). The strongest statement is in Module 12 §2.3: in ScaleRL the upper bound moves the fitted asymptote itself8 (App. A.17.1). Cui et al. explain why it raises entropy in the terms of Figure 3: the upper threshold only touches positive-advantage tokens, so raising it adds "low-covariance (low probability, high advantage, with average covariance of ∼−0.03)" tokens1 (§4.5), the top-right cell.
The second half of DAPO's recipe works on the batch. Once entropy has fallen, the \(G\) samples of a prompt tend to be near-identical, the reward is the same for all of them, and the group advantage is zero; DAPO's dynamic sampling oversamples and drops groups with accuracy 0 or 12 (§3.2), and Olmo 3's active sampling does the same continuously, with "greatly reduced" loss variance in its RL-Zero ablation9 (§6.2, Fig. 26). Module 11 §4 derives the dead-group arithmetic. Here it matters as a monitor: the zero-advantage share is the second-order symptom of an entropy collapse (the causal direction, entropy first and dead groups second, is the writer's reading of DAPO's description).
3.3 Clip-Cov and KL-Cov: select the tokens that drain entropy
If a few tokens carry the covariance (Figure 4), restrict only those. With the token covariance \(\mathrm{Cov}(y_i)=\big(\log\pi_\theta(y_i)-\tfrac1N\sum_j\log\pi_\theta(y_j)\big)\big(A(y_i)-\tfrac1N\sum_jA(y_j)\big)\) over the \(N\) rollout tokens of a batch, Clip-Cov picks \(\lfloor rN\rfloor\) tokens uniformly at random among those with covariance in \([\omega_{\mathrm{low}},\omega_{\mathrm{high}}]\) and detaches them from the policy gradient; KL-Cov ranks tokens by covariance and adds \(-\beta\,D_{\mathrm{KL}}(\pi_{\theta_{\mathrm{old}}}\Vert\pi_\theta)\) to the top fraction \(k\)1 (§4.2, Eqs. 10–14). The KL here is to the rollout policy, a per-iteration trust region, not to the reference model. Settings: \(r=2\times10^{-4}\), \(\omega_{\mathrm{low}}=1\), \(\omega_{\mathrm{high}}=5\) (over 500 times the average covariance); \(k=2\times10^{-3}\) for 7B and \(2\times10^{-4}\) for 32B; \(\beta=1\); DAPO-MATH prompts, 256 prompts with 8 responses at temperature 1, 8 updates per rollout, dead groups filtered, 8,192-token maximum1 (§4.3).
| Model | Method | AIME24 | AIME25 | MATH-500 | Avg | Δ Avg |
|---|---|---|---|---|---|---|
| Qwen2.5-7B | GRPO | 21.2 | 9.6 | 78.8 | 38.6 | – |
| + Clip-Higher | 18.1 | 11.5 | 79.2 | 38.8 | +0.2 | |
| + Clip-Cov | 22.1 | 15.8 | 80.4 | 40.4 | +1.8 | |
| + KL-Cov | 22.6 | 12.9 | 80.8 | 40.6 | +2.0 | |
| Qwen2.5-32B | GRPO | 21.8 | 16.2 | 84.2 | 45.8 | – |
| + Clip-Higher | 35.6 | 22.3 | 77.2 | 47.2 | +1.4 | |
| + Clip-Cov | 32.3 | 22.7 | 87.0 | 50.3 | +4.5 | |
| + KL-Cov | 36.8 | 30.8 | 84.6 | 52.2 | +6.4 |
Both methods raise the average on both models, KL-Cov by 2.0 and 6.4 points; on the 32B model AIME24 rises by 15.0 points and AIME25 by 14.6 (derived from Table 2). The entropy curves are the stronger evidence: when the baseline's entropy has plateaued, KL-Cov "still sustains an entropy level over 10× higher", response length keeps rising, and held-out accuracy does not plateau1 (§4.3, Fig. 11). Entropy becomes a dial: more clipped tokens (Clip-Cov) or a larger \(\beta\) (KL-Cov) gives higher entropy, with KL-Cov the steadier of the two (§4.4, Fig. 12). The method touches only \(10^{-4}\) to \(10^{-3}\) of tokens "yet totally change[s] the entropy curve", which the authors read as a handful of "pivotal" tokens; they also report that they "don't observe a relationship between the intervened entropy and model performance", leaving open whether an optimal entropy exists1 (§4.5).
3.4 Selecting by entropy rather than covariance
A neighboring line of work selects tokens by the entropy of the state's distribution. Wang et al. report that only a minority of tokens have high entropy, that these act as forks steering the reasoning, and that restricting the update to about the top 20% of tokens by entropy matches full-gradient training on Qwen3-8B and beats it on Qwen3-14B and 32B (AIME'24 and AIME'25), while training only on the lowest-entropy 80% hurts performance markedly (abstract level; I did not read the full paper)12. TRL exposes it as top_entropy_quantile (default 1.0; "The paper recommends a value of 0.2")6. The two selections differ in one respect: Clip-Cov and KL-Cov brake the update on confident, rewarded tokens, while the top-entropy rule drops confident tokens' updates altogether; both leave the uncertain forks in the gradient (the unification is the writer's reading).
3.5 Temperature as an entropy lever
Raising the rollout temperature is the cheapest knob and the least trusted. ProRL "did employ high rollout temperature" (1.2) because it "encourages exploration by increasing the initial entropy", but found it "only delays the onset of entropy collapse rather than preventing it altogether, as entropy continues to decline steadily"3 (§2.2, §3.2). Magistral settled on 1.0 as "the best balance between avoiding the lack of diversity seen at lower temperatures and the incoherent outputs generated at higher temperatures"5 (§5.3). R1-Zero, Olmo 3 and DAPO's evaluation use 1.01392. Kimi K2 schedules it: a high temperature early is "crucial" for creative writing and complex reasoning because it facilitates "the discovery of effective strategies and reducing the risk of premature convergence to suboptimal solutions", but keeping it high later "introduces excessive randomness", so "we employ a temperature decay schedule, to shift from exploration to exploitation"4 (§3.2.3). The report gives neither the schedule nor an ablation.
Temperature acts before the gradient: it changes which tokens get sampled, and so which cells of Figure 3 the batch populates, not how an update treats a sampled token (the writer's reading). It has a systems consequence an inference engineer will recognize: the sampler's temperature and the trainer's log-probabilities must agree. TRL passes the same temperature to the vLLM call and to the fused head that computes the trainer's log-probabilities and entropy7; if they differed, the importance ratio and the logged entropy would describe a distribution the sampler never used (Module 14).
| Intervention | Acts on | Tested on | Reported effect | Limit | Source |
|---|---|---|---|---|---|
| Entropy bonus, static | loss | Qwen2.5 (Cui); 3B, math and math+code (Magistral) | 0.005 stable but no better; 0.01 explodes | coefficient- and dataset-sensitive | Cui §4.1; Magistral §7.4.2 |
| Entropy bonus, adaptive | loss | none read | implemented in TRL (Skywork-OR1) | no ablation in sources | TRL docs |
| KL to reference as entropy tool | loss | Qwen2.5, \(\beta\) 0.001 to 0.1 | stable entropy, worse accuracy | one setup | Cui §4.1 |
| Clip-Higher | clip | Qwen2.5-32B (DAPO); 7B and 32B (Cui) | AIME24 36 to 38; avg +0.2 and +1.4 | unstable later (Cui); ScaleRL: moves asymptote | DAPO Tab. 1; Cui Tab. 2 |
| Drop dead groups | batch | Qwen2.5-32B (DAPO); Olmo 3 | 42 to 50 AIME24 in ladder; lower loss variance | needs oversampling | DAPO Tab. 1; Olmo 3 Fig. 26 |
| Clip-Cov, KL-Cov | clip, penalty | Qwen2.5-7B and 32B, 8,192 tokens | avg +1.8/+2.0 (7B), +4.5/+6.4 (32B) | one family, short context, unreplicated | Cui Tab. 2 |
| Top-entropy token mask | loss mask | Qwen3-8B, 14B, 32B | 20% of tokens match or beat full gradient | abstract level only | Wang et al.; TRL |
| Temperature, high or decayed | sampling | 1.5B (ProRL); K2 (decay) | delays collapse; K2 gives no ablation | no schedule or ablation | ProRL §2.2; K2 §3.2.3 |
4. KL drift: three different KLs, what the curve does, and the ledger
Module 6 teaches the KL-regularized objective, the \(k_1/k_2/k_3\) estimators and Table 6 of coefficients and placements. This section takes the question it leaves open: when the policy is meant to travel far, what does the leash do, and what do reports that dropped it still watch?
4.1 "KL" names three different things
- KL to a fixed reference (the SFT or starting checkpoint): the RLHF leash of Module 6. It is the quantity Tulu 3's \(\beta\) sweep and Olmo 3's logged curve measure149.
- KL to the rollout policy \(\pi_{\mathrm{old}}\): a per-iteration trust region. The PPO ratio bounds it implicitly; KL-Cov adds it explicitly on selected tokens1; Kimi K2's squared loss, inherited from K1.5, carries a \(\tau\log(\pi_\theta/\pi_{\mathrm{old}})\) term4 (§3.2.3; Module 9 derives it). It limits step size and says nothing about distance from the start.
- KL to a moving reference: a fixed-anchor KL whose anchor is periodically replaced. R1 replaces the reference with the latest policy every 400 steps13 (§2.1); ProRL hard-resets it, with the optimizer state, when validation metrics degrade or plateau3 (App. E); DeepSeekMath's iterative GRPO resets it each outer iteration15 (§4.1.4); TRL implements the TR-DPO rule \(\pi_{\mathrm{ref}}\leftarrow\alpha\pi_\theta+(1-\alpha)\pi_{\mathrm{ref}}\) with \(\alpha=0.6\) every 512 steps, off by default6.
A KL curve in a log means one of these, and which one decides how to read it (Figure 6).
4.2 What the curve does in runs that do not penalize it
Olmo 3 removes the KL loss and still plots the KL divergence for its 7B Think run: it "grows gradually and reflects stronger deviation from the reference policy", while response length "becomes longer and stabilizes at a higher level"9 (App. A.6.3, Fig. 41). Read from the figure (approximate), KL rises from near 0 to about 0.09 by roughly step 1,000 and ends near 0.08 at about 1,400 steps, while overall reward climbs from about 4.2 to 5.8. A smooth rise then plateau, tracking reward, is the healthy picture without a leash. A catch: TRL logs kl "only if beta is nonzero", and its default \(\beta=0\) does not load the reference model6, so the default removes the penalty and the instrument together. Watching KL without penalizing it still costs the reference forward pass (Module 1).
Tulu 3 shows what the leash protected against. In its PPO-based RLVR from the DPO checkpoint, lowering \(\beta\) from 0.1 toward 0.01 raised KL from the initial model, and "more KL divergence typically results in lower average scores", with one exception: "as the model diverges more from the initial model, overoptimization happens as the average scores drop significantly"14 (§6.2.1, Figs. 21–22). What drops is the average over all evaluations, not only the rewarded task. Olmo 3 sees the same symptom with no KL term and fixes it with data: OlmoRL on IFEval alone traded higher IFEval for lower AlpacaEval, while "training on a mix yields steady improvements across different domains"9 (§4.5, Fig. 20). Same failure (off-target regression as the policy drifts), two cures: a price on drift, or a broader reward.
4.3 The ledger
Table 4 lists the keep-or-drop decisions with what the report says and what it shows. It deliberately omits estimators, placement and coefficient values, which are in Module 6 Table 6; the new columns are the starting checkpoint and the kind of evidence.
| Recipe | Starts from | Decision | Reason given | Evidence |
|---|---|---|---|---|
| DeepSeekMath-RL | instruction-tuned 7B | keep, \(\beta=0.04\); iterative GRPO resets reference | in the loss so the advantage stays simple | practice15 |
| Tulu 3 RLVR | DPO checkpoint, 8B and 70B | keep, 0.05 and 0.07 | over-optimization as KL grows | sweep, \(\beta\) 0.1 to 0.0114 |
| R1-Zero, R1 stage 1 | DeepSeek-V3-Base (Zero) | keep, 0.001; reference replaced every 400 steps | not stated | practice13 |
| DAPO | Qwen2.5-32B base | remove | the model "can diverge significantly from the initial model, thus this restriction is not necessary" | practice; no KL row in the ablation ladder2 |
| ProRL | R1-Distill-Qwen-1.5B | keep, with hard resets | "stronger and more stable" against entropy collapse; "remove KL" applies to base models | practice; confounded with Clip-Higher, T = 1.23 |
| DeepSeek-V3.2 | specialist-distilled data, then mixed RL | keep, domain-dependent; weak or none for math | "different domains benefit from varying strengths" | practice; no numbers16 |
| Olmo 3 | DPO checkpoint (Think); base (RL-Zero) | remove | "allows less-restricted policy updates"; no over-optimization seen | practice; KL still logged9 |
| GLM-5 reasoning RL | first RL stage | remove | "to accelerate RL improvement" | practice10 |
| Magistral | Mistral Medium 3 Instruct; SFT-bootstrapped Small | remove | the policy diverges anyway; reference copy "incurs a compute cost we find unjustified" | tried a KL term and an EMA reference: it "primarily hinders training"5 |
| MiniMax-M1, ScaleRL | M1 base; ScaleRL base algorithm | remove | "similar to other recent works"; "in line with large-scale training reports" | practice178 |
| TRL GRPOTrainer | user's choice | default \(\beta=0\); reference not loaded | memory and speed | documented default6 |
Two things stand out. First, none of the "remove" decisions carries an ablation in the source: DAPO's ladder has no KL row and Olmo 3 states the decision as common practice. Magistral tested a KL term and reports harm, and Cui et al.'s entropy experiments (§3.1) add a second negative result for a KL penalty used to hold entropy. Second, ProRL's rule (keep KL from a distilled checkpoint, drop it for a raw base) does not sort the rows: Tulu 3 and Olmo 3 Think both start from a DPO checkpoint and decide opposite ways. What separates them, if anything, is the task: Tulu 3's RLVR uses 1,024 to 2,048-token response caps on a narrow verifiable set14 (Table 21), Olmo 3 runs 32,768-token responses over a mixed reward (Table 49). That is the writer's reading; nothing in the sources isolates it.
4.4 Resets, and the ProRL case
ProRL's is the only sustained argument, in the sources read, for keeping the term in long-chain-of-thought RL. Its premise: DAPO-style components and a high temperature "help slow entropy collapse", but "explicit regularization via a KL divergence penalty provides a stronger and more stable solution", and it also "prevent[s] the online policy from drifting too far from a stable reference"3 (§2.3.1). The cost it observes is that "the KL term may increasingly dominate the loss, leading to diminishing policy updates"; the cure is a periodic hard reset of the reference to a more recent snapshot of the online policy, with the optimizer state reinitialized3. Its qualification of the "remove KL" advice is that the perspective "often applies to base models prior to any supervised fine-tuning", whereas ProRL starts from DeepSeek-R1-Distill-Qwen-1.5B, a checkpoint "already capable of generating coherent CoT outputs".
ProRL's Appendix E shows the resets inside a sequence of eight runs. Run 1 (four task types, 8k cap) ran until "instability and degradation in validation performance"; Run 2 hard-reset the reference and continued; Run 3 added instruction-following data until "a sudden increase in response length, primarily due to the model repeating answers and failing to terminate with an <eos> token"; Runs 4 and 5 penalized non-terminating responses; Runs 6 and 7 doubled rollouts from 16 to 32 with two hard resets; Run 8 extended the context to 16k3 (App. E). The resets are bundled with data, reward and hyperparameter changes, which the authors present as a feature ("an opportunity to adjust training hyperparameters") and which makes the KL term's own contribution unmeasurable from this record. The entropy trace stays near 1 on a 0–6 axis through more than 2,000 steps (Fig. 2; approximate). The final model's gains, from Tables 1–3, are in Figure 7.
5. Length growth and language drift as run-health signals
Module 11 §2 derives why a per-response length normalizer pushes length one way or the other depending on pass rate, and Module 23 covers controlling length on purpose. This section is about reading the length curve, because it is the one monitor that is also a bill: every extra token is decode work in the rollout engine.
5.1 What the reports show
R1-Zero is the reference curve. Its average response length rises steadily through 10,400 steps, from under 1,000 tokens near step 0 to roughly 14,000 near step 10,000, with a jump at step 8.2k, exactly where the maximum length was raised from 32,768 to 65,536 (approximate reading of Fig. 1b)13 (§2.1). The report reads it as the model learning "to solve reasoning tasks with more thinking time" (Fig. 1 caption) and ties it to growing reflection, including a sudden increase in the word "wait" (§2.3). The rollout bill follows: each rollout generates 8,192 outputs, so 14,000 tokens each is about 115 million generated tokens, against about 4 million at 500 tokens, roughly 28 times the decode work per rollout between start and end (derived from approximate readings; 500 is an assumed start). Table 5 collects what other reports saw.
| Report | Length behavior | Stated interpretation | Control or check |
|---|---|---|---|
| R1-Zero | under 1,000 to about 14,000 tokens, jump at step 8.2k | "more thinking time"; reflection words rise | cap raised 32,768 to 65,53613 |
| DAPO | not monotone; stagnation or decline for "considerable periods" | length and validation accuracy together tell you if a run is deteriorating | overlong filtering; soft punishment over a 4,096-token cache; token-level loss2 |
| Olmo 3 Think 7B | dips, then "slowly increase over time"; about 14,500 trough, about 18,000 at end | SFT and DPO already trained 32K-token traces | none specific (§4.5)9 |
| Olmo 3 RL-Zero, weak mid-training mix | flat | the base lacks backtracking and verification behaviors to amplify | change the mid-training data9 |
| ProRL | correlates with validation score; one run spikes | "does not appear to be decisive"; spike is repetition without termination | 8k cap; penalty for non-terminating answers3 |
| Magistral | grows with reward until it meets the length penalty and cap | "increasing completion length is the main resource that improves the performance" | \(l_{\max}-l_{\mathrm{cache}}\) soft penalty5 |
| R1-Zero as analyzed by Dr. GRPO | correct 621 to 4,965, incorrect 1,039 to 8,206 (string length) | wrong answers come from harder prompts that need longer responses, or from the \(1/|o_i|\) bias | constant normalizer18 |
| Kimi K2, DeepSeek-V3.2 | "substantial increase" in RL generally | cost not justified outside reasoning domains | per-task token budget; length penalty in the reward416 |
5.2 Four readings of a rising length curve
- Capability. Longer reasoning is the thing being learned. R1-Zero's account, and Magistral's: along the weight-space direction RL moves, "mean reward and output length grow up", and raw reward without the length penalty scales logarithmically with output length for checkpoints averaging 1,500 to 8,000 tokens5 (§7.1, Figs. 8–9). That is a cross-section of perturbed checkpoints, so it shows co-movement along one direction, not that length causes reward.
- Loss artifact. The \(1/|o_i|\) normalizer rewards short correct and long wrong answers; Dr. GRPO finds GRPO keeps lengthening "even when the reward improvement slows down" while the unbiased variant does not18 (§3.2). The effect is shown at 1.5B to 3B with 3,000-token caps (Module 11 §2.3).
- Selection. Wrong answers are longer than right ones partly because they come from harder prompts. Dr. GRPO states this hypothesis itself for R1-Zero's Table 518 (Appendix, Table 5), so a gap between incorrect and correct length is not by itself evidence of a bias.
- Degeneration. The model repeats itself and never emits end-of-sequence. This one is a failure, and on a mean-length plot it looks like the capability reading.
5.3 Telling runaway from reasoning
ProRL's Run 3 shows the degenerate case: length rose suddenly "primarily due to the model repeating answers and failing to terminate", and reward shaping that penalizes non-terminating responses reduced it modestly3 (App. E). R1's notes record the small-model version: with a 7B dense or 16B MoE base, "as response lengths increased, these smaller models exhibited a tendency toward repetition and were unable to effectively leverage long chains of thought (CoT)"13 (Supp. G.1). Tulu 3 charged a flat −10 for any response without an end-of-sequence token14 (Table 21). DAPO ties the runaway to entropy. With sample-level loss, long low-quality samples with "gibberish and repetitive words" go unpunished, which "leads to an unhealthy increase in entropy and response length": entropy climbs from about 0.6 to about 3 within 5,000 steps, while the token-level-loss run creeps from about 0.4 to about 1.0 over 9,000 (approximate readings of Fig. 4a). Without overlong filtering, entropy sat near zero until roughly step 3,500 and then jumped to about 4.5 (approximate reading of Fig. 5b)2 (§3.3, §3.4). So entropy can fail in both directions, and an upward excursion alongside a length spike is the signature of runaway, not of healthier exploration.
TRL's logged metrics make the distinction measurable (the use is the writer's suggestion): completions/mean_length averages all completions, completions/mean_terminated_length only those that ended with EOS, and completions/clipped_ratio is the truncated share6. If the mean rises while the terminated mean stays flat and the clipped ratio climbs, the growth is truncated runaways; if the terminated mean rises with accuracy and the clipped ratio is flat, longer reasoning is being learned. DAPO's overlong filtering masks the loss of truncated samples and "significantly stabilizes training"; TRL exposes it as mask_truncated_completions6.
5.4 Language mixing
R1-Zero's report lists "poor readability and language mixing, occasionally combining English and Chinese" and suggests DeepSeek-V3-Base's Chinese and English data "might be the cause"13 (§1, Supp. A). The remedy was a reward equal to the fraction of target-language words in the chain of thought (Eq. 7); without it consistency "gradually deteriorates", with it consistency holds, math is comparable and coding drops slightly13 (§3.2.1, Supp. B.6; Module 10 §2.4). Magistral saw "outputs that mixed English, Chinese, and Russian words" and added 0.1 reward when a fastText classifier labels problem, thoughts and answer as one language, after translating 10% of English problems into six languages5 (§2.2). DeepSeek-V3.2 found a sampling-side contributor: truncating the action space with top-p at rollout but not in training breaks importance sampling, and top-p with the Keep Sampling Mask "effectively preserves language consistency"16 (§3.1; Module 14). A per-step language-ID rate over the chain of thought is a cheap monitor (the writer's suggestion; none of the reports reads one out).
6. Reading a run
6.1 The signals the reports actually use
Every monitor below appears in at least one report or in TRL's logged-metric list: entropy (Cui et al., DAPO, ProRL, Magistral, Qwen3), KL to the reference (Olmo 3, Tulu 3, ProRL), response length and its truncation split (DAPO, R1, Olmo 3, ProRL), the share of prompts with zero reward variance (DAPO Fig. 3b, Olmo 3 Fig. 26), clip fraction (DAPO's mean up-clipped probability, GSPO's clipped-token fraction), the gap between training reward and validation accuracy (DAPO), and the log-probability gap between sampler and trainer (TRL's sampling/sampling_logp_difference/mean)2619. DAPO's rule for entropy: it "needs to be maintained within an appropriate range", since too low loses exploration and too high brings "gibberish and repetitive generation", and "a slow upward trend in entropy is conducive to the improvement of model performance"; it also notes that training reward "often exhibits little correlation with the accuracy on the validation set"2 (§4.3). ProRL monitors a blended validation set (AIME 2024, Codeforces, GPQA-Diamond, IFEval, one Reasoning Gym task) and uses it to trigger reference resets3 (App. E). Figure 8 draws the healthy and failing shapes.
| Symptom | Where it shows | Documented response | Source and its limit |
|---|---|---|---|
| Entropy falls toward zero early; accuracy plateaus | entropy; accuracy curve | Clip-Higher; dynamic or active sampling; Clip-Cov or KL-Cov; KL with resets | DAPO, Olmo 3, Cui, ProRL (Table 3; mostly Qwen2.5, single runs) |
| Entropy rises sharply with length | entropy, completions/mean_length, completions/clipped_ratio | token-level loss; overlong filtering with soft punishment; end-of-sequence penalty | DAPO §3.3–3.4, ProRL Run 4–5, Tulu 3 (Qwen2.5-32B, 1.5B) |
| Sudden entropy drop and degradation within a few steps, no algorithm change | entropy, sampling/sampling_logp_difference/mean | check engine determinism: GLM-5's non-deterministic top-k in a sparse-attention indexer; torch.topk fixed it | GLM-5 §3.210; Module 14 |
| KL keeps rising; off-domain evaluations fall | kl (needs \(\beta\ne0\)); held-out evals | raise \(\beta\) (Tulu 3); mix reward and data across domains (Olmo 3); reset the anchor (ProRL) | one DPO-start 8B setup; Olmo 3 reports no matched ablation |
| KL term dominates the loss; updates shrink | kl, update size | hard reset of reference and optimizer state | ProRL only; 1.5B |
| Training reward up, validation flat | reward against held-out accuracy | watch a validation blend; reset on plateau | DAPO §4.3, ProRL App. E |
| Zero-advantage share climbs | frac_reward_zero_std | drop dead groups and refill; make prompts harder | DAPO, Olmo 3 (Module 11 §4) |
| Reasoning mixes languages | language-ID rate (not logged by default) | language-consistency reward; keep the sampling mask with top-p | R1, Magistral, V3.2 (§5.4) |
6.2 Absolute values do not transfer
The same curve can sit at very different numbers in different setups, so compare a run with its own early trajectory and with a sibling run, not with a published value.
- Entropy. DAPO's final run on Qwen2.5-32B plots entropy between about 0.3 and 0.7 (axis of Fig. 7c)2; ProRL's 1.5B run sits near 1 on a 0–6 axis3; TRL's docs use an
entropy_targetof 5.0 in their example and say typical models have 2–10 nats6. Temperature, the tokens included (TRL excludes truncated completions only when asked) and the prompt set all move it (§1.1). - Clip fraction. It is not comparable across loss types. GSPO clips whole sequences and reports an average clipped fraction of 0.15 against 0.0013 for GRPO, two orders of magnitude apart, and "adjusting the clipping ranges does not alter the disparity in magnitude"19 (§5.2, Fig. 2). A fraction that signals trouble for token-level clipping is routine for the sequence-level kind (Module 12).
Where experts disagree
Two questions are open. First, how to stop entropy collapsing, and whether entropy is what should be protected. Second, whether a long-chain-of-thought run should carry a KL term at all. Most reports decide both by assertion; the tests that exist are short and single-model. Module 6's disagreement section covers the KL estimator and placement; this one covers the long-CoT decision.
A. How to prevent entropy collapse
Reshape the clip and the batch
DAPO, Olmo 3, Magistral, GLM-5; ScaleRL's base loss295108
The upper clip throttles rare tokens, so raise it (0.26 to 0.4 across reports) and drop zero-advantage groups so the batch keeps a gradient.
- No new term or model; four independent recipes use it.
- Magistral gives an operational rule: tune \(\epsilon_{\mathrm{high}}\) to hold group entropy stable.
- Modest measured gain (36 to 38 AIME points in DAPO's ladder; +0.2 and +1.4 average points in Cui et al., where it turns unstable late).
- The constant moves the asymptote (ScaleRL) and depends on off-policy reuse (Module 12).
Select tokens by covariance
Cui et al. (Clip-Cov, KL-Cov)1; related token-selective work by Wang et al.12
A few confident, rewarded tokens carry the covariance that drains entropy; brake those only, and entropy becomes a dial.
- Derived from a mechanism checked against measured covariance.
- Largest reported gains (+6.4 average points at 32B).
- Two Qwen2.5 models, math, 8,192 tokens; unreplicated, and no production recipe read adopts it.
- No relationship found between the controlled entropy and accuracy, so the target is unknown.
Anchor with KL, reset it, and sample hot
ProRL3
From a distilled start an explicit KL term is the stronger fix; reset the reference when it dominates the loss.
- Entropy held near 1 for over 2,000 steps; large per-domain gains at 1.5B.
- Cui et al. and Magistral found KL hurt accuracy; no isolating ablation; temperature alone "only delays" collapse.
Entropy is a symptom, not the target
ScaleRL (App. A.12)8; Agarwal et al.20; Olmo 3's DPO-start observation9
Two ScaleRL runs with the same entropy path differed in performance; Agarwal et al. report that rewarding negative entropy alone matches or beats GRPO and RLOO on Qwen-7B (abstract level); Olmo 3's DPO checkpoint had lower entropy and higher pass@k than its SFT checkpoint.
- Explains why Cui et al.'s interventions showed no entropy-accuracy link.
- ScaleRL's own base loss still carries asymmetric clipping "to avoid entropy collapse".
The positions answer different questions. To stop an early collapse cheaply, Clip-Higher plus dead-group filtering is the only answer with several independent adopters. To steer entropy to a chosen level, covariance-selective control is the only method with a mechanism behind a dial, on two models and a short context. For long runs from a distilled start, resets are the only documented practice and are unablated. The fourth position is the important correction: entropy is a necessary health signal and a poor score, because the same path can carry different accuracy (ScaleRL) and a sharper start can have a better ceiling (Olmo 3). I would log entropy, keep a floor with Clip-Higher, and not optimize toward a number. What would settle it: one recipe at 16k or more context, two base models, {none, Clip-Higher, Clip-Cov or KL-Cov, adaptive bonus} crossed, with held-out entropy and pass@k (Module 20) so sharpening and expansion are separated.
B. Keep or remove the KL-to-reference term in long-CoT RL
Remove it
DAPO, Olmo 3, GLM-5, Magistral, MiniMax-M1, ScaleRL's base, TRL's default29105176
A reasoning policy must diverge from its start, a verifier gives no learned proxy to exploit, and the reference copy costs memory and a forward pass.
- Convergent practice across open recipes; TRL does not even load the reference by default.
- Magistral tested a KL term, including an EMA reference, and reports it hindered training.
- Mostly assertion: DAPO's ladder has no KL row, Olmo 3 cites practice.
- Tulu 3 saw off-domain scores fall as KL grew; Olmo 3 avoids that by mixing data, which a narrow run cannot copy.
Keep it, and reset the anchor
ProRL; R1's 400-step replacement; DeepSeekMath's iterative GRPO31315
A restoring force protects stability over thousands of steps; resets keep it from freezing progress. "Remove KL" applies to raw base models.
- R1 ran 10,400 steps with this scheme; ProRL ran more than 2,000 on a 1.5B model.
- Cheap to add where the reference is already loaded.
- Resets turn the leash into a trust region (Module 6 §4.2), so it no longer bounds distance from the start.
- Tulu 3 and Olmo 3 share a DPO start yet decide opposite ways (Table 4), which weakens the base-versus-distilled rule.
Keep it small and per domain, with a corrected estimator
DeepSeek-V3.2; Tulu 3's sweep1614
Different domains want different strengths, weak or none for math, and the \(k_3\) gradient must be importance-corrected when kept.
- Matches the evidence that drift hurts off-target evaluations but not the rewarded task.
- V3.2 fixes a known estimator bias instead of dropping the term.
- V3.2 gives no coefficients or ablation; per-domain tuning multiplies hyperparameters.
- Tulu 3's evidence is a sweep at 1,024 to 2,048-token responses, not long chains of thought.
The decision turns on what the reward cannot see, not on the checkpoint. With a rule-based verifier, broad mixed data and held-out evaluations in the loop, removal is convergent and cheap, and the KL curve should still be logged as an instrument (Olmo 3 does). With a narrow single-domain reward, with a learned or generative judge (Modules 16 and 19), or when off-domain scores fall as KL rises, keep a small term or reset the anchor, as Tulu 3 and ProRL did. The sources contain no matched on, off and reset comparison at long context; Magistral's test and Cui et al.'s Figure 10 are the nearest, and both are negative results for a KL penalty used to hold entropy, not controlled studies of drift. That experiment would settle it: one recipe, thousands of steps, {no KL, fixed \(\beta\), \(\beta\) with resets}, with entropy, KL, off-domain accuracy and pass@k logged.
What is changing
The default recipe has converged faster than the evidence for it: Clip-Higher with an upper bound of 0.26 to 0.4, no KL loss, token-level aggregation and dead-group filtering appear together in DAPO, Olmo 3, Magistral and GLM-5, between March 2025 and February 2026. TRL's defaults lag part of that: loss_type is dapo and beta is 0.0, but epsilon_high is unset, so Clip-Higher is opt-in6; these pages move, so check your version. Entropy control is moving into libraries (an adaptive bonus and a top-entropy-token mask), on evidence that in the sources read is a docstring and an abstract. Open as of 2026-10-27: whether an optimal entropy exists (Cui et al. find none; ScaleRL finds entropy does not predict performance across recipes), whether the entropy-performance law survives off-policy data and mixed rewards, and how much of a "collapse" is numerical rather than algorithmic, since GLM-5's entropy cliff came from a non-deterministic top-k kernel (Module 14). Module 15 reframes the loss-side tweaks as efficiency changes rather than ceiling changes, with the clip bound as the exception. Cui et al. is a v1 preprint (May 2025) and Magistral a v1 (June 2025).
References
- paperCui, G. et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617v1, 28 May 2025 (preprint, Shanghai AI Laboratory et al.). https://arxiv.org/pdf/2505.22617v1. Accessed 2026-10-27. Used for: entropy definition (Eq. 5), the law R = -a exp(H) + b (Eq. 6, §2.3-2.6), the covariance lemma and theorems (§3), the entropy-bonus and reference-KL experiments (§4.1, Figs. 9-10), Clip-Cov and KL-Cov (§4.2-4.5, Eqs. 10-14, Table 2, Table 1 covariance percentiles).
- paperYu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-27. Used for: entropy collapse and Clip-Higher (§3.1, Figs. 2-3), dynamic sampling (§3.2), token-level loss and its entropy and length effect (§3.3, Fig. 4), overlong filtering (§3.4, Fig. 5), no-KL decision (§2.3), ablation ladder (Table 1), monitoring guidance (§4.3, Fig. 7).
- paperLiu, M. et al. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1, 30 May 2025 (NVIDIA). https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-27. Used for: KL penalty with reference resets and the "remove KL" qualification (§2.3.1), setup including epsilon_high 0.4 and temperature 1.2 (§3.2), training dynamics (§3.3, Fig. 2), results (Tables 1-3, §3.4), the eight-run recipe (App. E).
- paperKimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534v2, 3 Feb 2026 (v1 28 Jul 2025). https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-27. Used for: temperature decay, budget control and the squared-loss objective (§3.2.3).
- paperMistral AI. Magistral. arXiv:2506.10910v1, 12 Jun 2025. https://arxiv.org/pdf/2506.10910v1. Accessed 2026-10-27. Used for: removal of KL, Clip-Higher values and language-consistency reward (§2.1-2.2), epsilon_high 0.3 and temperature 1.0 (§5.3), the entropy-bonus and KL failures (§7.4.2, Fig. 12), length-reward co-movement (§7.1, Figs. 8-9). Not in the source map; read from the PDF.
- officialHugging Face TRL maintainers. TRL documentation: GRPO Trainer. main branch, fetched 2026-10-27 (no version stated). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: logged metrics (entropy, kl, completions/*, frac_reward_zero_std, sampling_logp_difference), entropy_coef and adaptive entropy, top_entropy_quantile, sync_ref_model, beta, epsilon_high and loss_type defaults.
- repoHugging Face. trl/trainer/grpo_trainer.py. main branch, same URL as Module 9, fetched 2026-10-27. https://raw.githubusercontent.com/huggingface/trl/main/trl/trainer/grpo_trainer.py. Accessed 2026-10-27. Used for: the fused log-probability and entropy head built with the generation temperature (lines near add_fused_lm_head); the sampler temperature passed to vLLM. Not in the source map; also used by Module 9.
- paperKhatri, D. et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-27. Used for: base algorithm without KL and with asymmetric clipping (§3), entropy on a held-out set and the batch-size finding (App. A.12), epsilon_max effect on the asymptote (App. A.17.1).
- paperTeam Olmo. Olmo 3. arXiv:2512.13961v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-27. Used for: OlmoRL list including no KL loss and clip-higher (§4.4.1), key findings on length, DPO-start entropy and domain mixing (§4.5), RL-Zero active-sampling ablation (§6.2, Fig. 26), hyperparameters (Table 49) and the 7B Think curves (Fig. 41, App. A.6.3).
- paperGLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-27. Used for: KL removed relative to IcePop, epsilon 0.2/0.28 (§3.2), non-deterministic top-k entropy cliff (§3.2, DSA RL insights).
- paperQwen Team. Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-27. Used for: the stated entropy target in reasoning RL (§4.2).
- paperWang, S. et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. arXiv:2506.01939, 2 Jun 2025; NeurIPS 2025. https://arxiv.org/abs/2506.01939. Accessed 2026-10-27. Used for: restricting updates to the top-entropy 20% of tokens (abstract level only; page fetched through a summarizing tool). Not in the source map.
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2, 4 Jan 2026 (expanded Nature version; v1 22 Jan 2025). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-27. Used for: R1-Zero settings, length curve and 8.2k jump (§2.1, §2.3, Fig. 1), reference replacement every 400 steps (§2.1, §3.2.1), language-mixing and language-consistency reward (§1, §3.2.1, Supp. B.6), small-model repetition (Supp. G.1).
- paperLambert, N. et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124v5, 14 Apr 2025 (v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-27. Used for: beta sweep and over-optimization as KL grows (§6.2.1, Figs. 21-22), response caps and the -10 end-of-sequence penalty (Table 21).
- paperShao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-27. Used for: KL in the loss, 0.04 coefficient and the iterative-GRPO reference reset (§4.1.4, §4.2).
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-27. Used for: domain-dependent KL strength, length penalty and language-consistency reward (§3, §3.1), Keep Sampling Mask and language consistency (§3.1).
- paperMiniMax. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-27. Used for: no KL penalty in CISPO (§3.1).
- paperLiu, Z. et al. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783v2, 6 Oct 2025; COLM 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-27. Used for: response-length growth with and without the length normalizer (§3.1-3.2), R1-Zero correct versus incorrect response length and the harder-prompt hypothesis (Appendix, Table 5).
- paperZheng, C. et al. (Qwen Team). Group Sequence Policy Optimization. arXiv:2507.18071v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-27. Used for: clipped-token fractions of GSPO (0.15) and GRPO (0.0013) (§5.2, Fig. 2).
- paperAgarwal, S., Zhang, Z., Yuan, L., Han, J., Peng, H. The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. arXiv:2505.15134, 21 May 2025. https://arxiv.org/abs/2505.15134. Accessed 2026-10-27. Used for: the claim that rewarding negative entropy alone matches GRPO and RLOO on Qwen-7B (abstract level only; page fetched through a summarizing tool). Not in the source map.