Reinforcement Learning for LLM Post-Training · Module 12
Importance Ratios and Clipping: Token-Level, Sequence-Level (GSPO), IS-Weight Clipping (CISPO), and Log-Ratio Masks
The PPO ratio is a different object in LLM RL than in MuJoCo: it is exactly 1 for one-update recipes, a staleness meter for 16-minibatch recipes, and a mismatch detector once the rollout engine and the trainer disagree on probabilities. This module writes the token-level clip, GSPO's sequence ratio, CISPO's weight clipping, the log-ratio masks of Kimi K2.5 and GLM-5, DeepSeek-V3.2's sequence mask and SAPO's soft gate in one form, so you can say which tokens receive gradient under each, and which design tolerates MoE routing noise and stale rollouts.
1. When the ratio is not 1
Module 3 derived the PPO clip for a ratio \(r_t=\pi_\theta(a_t\mid s_t)/\pi_{old}(a_t\mid s_t)\) that starts at 1 and drifts as the optimizer takes steps on a fixed batch. Whether that drift exists at all is a property of the recipe, not of the algorithm, and the reports differ more on this than on any clipping constant. Everything later in this module is a different answer to the question of what to do with a ratio that has drifted, so the first job is to say where the drift comes from.
1.1 Reuse per rollout is a staleness dial
DeepSeekMath-RL took one update per exploration stage: "the policy model only has a single update following each exploration stage"11 (§4.2), and the appendix drops the \(\min\) and the clip from the objective under exactly that assumption, "thereby ensuring that \(\pi_{\theta_{old}}=\pi_\theta\)"11 (App. A.1, Eq. 16). The clip in GRPO as published is inert. R1's first RL stage did the opposite: each rollout of 8,192 outputs is "randomly split into 16 minibatches and trained for only a single inner epoch"12 (§2.1, §3.2.1). Minibatch \(j\) is trained after \(j-1\) optimizer steps on data that were sampled before the first of them, so the staleness is 0, 1, …, 15 steps (derived). Averaged over the rollout that is \((16-1)/2=7.5\) steps, and only \(1/16=6.25\%\) of the minibatches are on-policy. With four minibatches the figures are 1.5 steps and 25%. Table 1 lists what the sources report.
| Recipe | Updates per rollout and lag | Ratio the loss sees | Source |
|---|---|---|---|
| DeepSeekMath-RL | 1 update per exploration stage | \(\equiv 1\); clip removed in the analysis | §4.2, App. A.111 |
| R1, first RL stage | 16 minibatches of 512 from 8,192 outputs, 1 epoch | drift, 0 to 15 steps | §2.1, §3.2.112 |
| DAPO | 16 gradient updates per rollout step (512 prompts × 16 responses, minibatch 512) | drift, 0 to 15 steps | §4.12 |
| MiniMax-M1 (CISPO study) | "16 rounds of off-policy updates per generation batch" | drift, up to 15 steps | §3.13 |
| GSPO paper | 4 minibatches per rollout | drift, 0 to 3 steps | §5.17 |
| Olmo 3 32B Think | 1 minibatch; max asynchrony 8; truncated-IS cap 2.0 | async lag up to 8 steps, plus engine mismatch | Table 4913 |
| Olmo 3 7B Instruct | 4 minibatches; max asynchrony 8 | drift plus async lag | Table 4913 |
| ScaleRL (PipelineRL-8) | generators take new weights mid-generation; trainers wait if they get \(k=8\) steps ahead | async lag up to 8 steps | §3.114 |
| GLM-5 reasoning RL | described as "entirely on-policy", group 32, batch 32 | mismatch only (Section 1.2) | §3.25 |
| GLM-5 agentic RL | fully asynchronous; a trajectory can span several policy versions; sample dropped if too stale | \(\pi_\theta/\pi_{rollout}\) | §4.1.25 |
| Kimi K3 | partial rollouts: generation pauses once a fraction \(\lambda\) of trajectories finish; the rest resume next iteration | a trajectory spans several iterations | §4.1.215 |
| TRL default | num_iterations = 1 | \(r\equiv 1\) in value (old log-probs = current, detached); with vLLM, a separate mismatch weight \(\rho\) multiplies the loss | docs, source1016 |
A ratio equal to 1 in value is not a zero-information case for the gradient. Because \(\nabla_\theta r_t=r_t\nabla_\theta\log\pi_\theta\), the gradient at \(r_t=1\) is the REINFORCE term \(\hat A\,\nabla\log\pi\) of Module 2, which is what DeepSeekMath's Eq. 16 keeps. TRL implements the same shortcut: when there is one iteration per batch and the generation batch fits in one accumulation window, the old log-probabilities are the current ones, detached, and the clip fraction is identically zero. With vLLM generation TRL always recomputes the old log-probabilities in the trainer, still before the update, so \(r\) is 1 in value and the engine mismatch enters as the separate weight \(\rho\) (Section 1.2)16 (_compute_loss, _generate_and_score_completions). A logged clip fraction of 0 in such a run says nothing about whether \(\epsilon\) was well chosen.
1.2 Two ratios hide inside one
There is a second source of \(r\neq 1\) that has nothing to do with the optimizer. The rollout engine (vLLM or SGLang) and the trainer run different kernels, precisions and parallelism, so even with identical weights their token probabilities differ. Lambert separates the two corrections: the policy-drift ratio \(\pi_\theta/\pi_{\theta_{old}}\) "corrects for policy drift across multiple gradient steps within one RL batch", and the implementation ratio \(\pi^{learner}_\theta/\pi^{sampler}_\theta\) corrects for "implementation-induced" differences; they are "complementary" and enter policy-gradient code "for different reasons"9 (§6.3.4). Writing the old policy as evaluated by the trainer, the sampler-to-current ratio factors exactly:
$$\underbrace{\frac{\pi^{train}_{\theta}(a_t)}{\pi^{infer}_{\theta_{old}}(a_t)}}_{\text{what the data were sampled from}}=\underbrace{\frac{\pi^{train}_{\theta}(a_t)}{\pi^{train}_{\theta_{old}}(a_t)}}_{r_t:\ \text{optimizer drift}}\times\underbrace{\frac{\pi^{train}_{\theta_{old}}(a_t)}{\pi^{infer}_{\theta_{old}}(a_t)}}_{\rho_t:\ \text{engine mismatch}}$$Designs differ in which factor they act on, and Figure 2 places them. Olmo 3 multiplies the loss by a truncated mismatch weight, with a cap of 2.0 for the 32B Think run, and still clips \(r\) with \(\epsilon_{low}=0.2,\ \epsilon_{high}=0.272\)13 (OlmoRL, Table 49). GLM-5's reasoning stage applies IcePop, a mask \(\mathrm{pop}(\rho,1/\beta,\beta)\) with \(\beta=2\) on \(\rho\), then the usual clip on \(r\) with 0.2 and 0.285 (§3.2, Eq. 1). ScaleRL writes its ratio as \(\pi^{train}_\theta/\pi^{gen}_{\theta_{old}}\), the left-hand side above14 (Eq. 2). GLM-5's asynchronous stage discards \(\pi_{old}\) altogether and uses the logged rollout probability as the denominator5 (§4.1.2). DeepSeek-V3.2 builds its sequence mask from the probability "directly returned by the inference framework"8 (§3.1). Module 14 covers where \(\rho\) comes from and how to correct it; here it matters because several of the designs below are, in effect, mismatch filters that also happen to bound drift.
2. Token-level clipping as a weight and a mask
2.1 One form for every variant
Every design in this module changes the same per-token term. Write the gradient of one token as a weight, times a mask, times the usual policy-gradient term:
$$\nabla_\theta J\;=\;\frac1Z\sum_{i,t}\;w_{i,t}\;M_{i,t}\;\hat A_{i,t}\;\nabla_\theta\log\pi_\theta(o_{i,t}\mid q,o_{i,2.2 The gradient multiplier of each variant, drawn
Figure 3 draws \(w(r)M(r)\), the factor that multiplies \(\hat A\,\nabla\log\pi\), for one token under four of the variants. For SAPO the factor is not obvious from its definition. TRL documents the gate as \(f(r)=\sigma(\tau(r-1))\cdot 4/\tau\) and the loss as \(-f(r)\hat A\)10, so by the chain rule the gradient is \(\hat A\,f'(r)\,r\,\nabla\log\pi\) with \(f'(r)=4\sigma(1-\sigma)\), giving \(w(r)=4\sigma(\tau(r-1))\bigl(1-\sigma(\tau(r-1))\bigr)\,r\) (derived). It equals 1 at \(r=1\), so SAPO reproduces the policy gradient on-policy, and it decays like \(r\,e^{-\tau(r-1)}\) for large \(r\).
Three things stand out. First, only CISPO and SAPO give every token a non-zero gradient at every ratio; the PPO-style clip and the band mask both switch tokens off. Second, for \(\hat A<0\) the PPO-style multiplier is unbounded: a bad token whose probability has doubled is pushed down twice as hard, and one that has grown tenfold ten times as hard. That is the case TRL's optional delta handles, a second, upper bound on the ratio for negative advantages that it attributes to the INTELLECT-2 report10; CISPO bounds it by construction. Third, the band mask differs from PPO exactly where PPO is deliberately pessimistic. In Module 3's Table 3 the cells "good token whose probability fell below \(1-\epsilon\)" and "bad token whose probability rose above \(1+\epsilon\)" still carry gradient, because PPO's \(\min\) keeps correcting moves in the wrong direction. A sign-blind band removes those too. Kimi K2.5 says so in as many words: its mechanism "relies strictly on the log-ratio to explicitly bound off-policy drift, regardless of the sign of the advantages"4 (§4.4.2).
2.3 How much the constants matter
Module 3 (Table 4) gave the headroom arithmetic behind Clip-Higher: with \(\epsilon=0.2\) a token at \(p=0.01\) can rise only to 0.012 before its gradient dies, and DAPO measured a mean up-clipped probability below 0.22 (§3.1). DAPO keeps \(\epsilon_{low}=0.2\) because raising the lower bound "will suppress the probability of these tokens to 0, resulting in the collapse of the sampling space"2 (§3.1). Two later reports measured how much the upper bound moves results. ScaleRL swept \(\epsilon_{max}\) in a DAPO-style loss and found that it "directly alters the scaling behavior": as it grows the terminal reward \(A\) rises to an optimum and then falls, so the constant "governs the asymptotic error itself", unlike many hyperparameters, which merely shift the convergence speed14 (App. A.17.1). Its DAPO baseline used 0.26. MiniMax-M1 found that raising the upper bound was "less effective in our setup, which involved 16 rounds of off-policy updates per generation batch"3 (§3.1), and later lowered its own IS-weight bound to stabilize long-context scaling3 (§5). The pattern is that the right width depends on reuse per rollout (Table 1) and on the model, and ScaleRL reports the PPO-style clip as markedly more sensitive to its constant than either CISPO or GSPO is to theirs14 (App. A.17).
2.4 What R1 printed
R1's first RL stage lists "the GRPO clip ratio \(\varepsilon\) to 10" and adds that "the clip ratio plays a crucial role in training. A lower value can lead to the truncation of gradients for a significant number of tokens, thereby degrading the model's performance, while a higher value may cause instability during training"12 (§3.2.1). Under a PPO parameterization the interval is \([1-\varepsilon,1+\varepsilon]=[-9,11]\); a ratio is never negative, so the lower clip cannot fire, and the upper clip fires only if a ratio exceeds 11 (derived). Two readings of the equation make this more puzzling. R1 prints the GRPO objective with a sequence-level ratio \(\pi_\theta(o_i\mid q)/\pi_{\theta_{old}}(o_i\mid q)\), with no token sum and no \(1/|o_i|\), both in the main text and in the appendix12 (§2.1 Eq. 1, Supp. A.3 Eq. 11). DeepSeekMath's Eq. 3 and DeepSeek-V3.2's Eq. 5 are per-token with a \(1/|o_i|\) average118. The report does not say whether the implementation differs from DeepSeekMath's or the equation is shorthand. The two readings give different meanings to 10. Per token, it lets a token's probability grow elevenfold before its gradient stops. Per sequence, the unnormalized product of token ratios would be allowed a total log-ratio of \(\ln 11=2.40\) nats upward, against \(\ln 1.2=0.18\) nats for the conventional 0.2 (derived).
The report's author list adds a lead without resolving it: "Zhibin Gou proposed a large PPO clipping strategy to enhance GRPO performance, demonstrating its significance alongside Zhihong Shao and Junxiao Song"12 (§7, Contributions of the Core Authors). Whether that strategy is the \(\varepsilon=10\) of stage 1, and whether the ratio is per token or per sequence, is not documented; I read a clip that is effectively open, with step control carried by the learning rate of 3e-6 and the 16-minibatch structure, but that reading is the writer's. The practical lesson is narrower and sourced: among the reports read for this module, R1 alone prints a clip value far outside the conventional range, and it says why a small value is a problem.
3. CISPO: clip the weight, not the update
3.1 The objective and what it changes in the gradient
MiniMax-M1 starts from REINFORCE with an importance correction for off-policy minibatches, \(\frac1{|o_i|}\sum_t \mathrm{sg}(r_{i,t})\,\hat A_{i,t}\log\pi_\theta(o_{i,t})\), where \(\mathrm{sg}\) is stop-gradient3 (§3.1, Eq. 3). CISPO ("Clipped IS-weight Policy Optimization") "abandons the trust region constraint and instead clips the importance sampling weights"3 (§1); it keeps that form and clips the weight instead of the objective:
$$J_{CISPO}=\mathbb E\Big[\tfrac{1}{\sum_i|o_i|}\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}\mathrm{sg}\big(\hat r_{i,t}\big)\,\hat A_{i,t}\,\log\pi_\theta(o_{i,t}\mid q,o_{i,loss_type="cispo" is that expression with epsilon_high as the cap16; the docs recommend 5.010. Lambert's table classifies it as an importance-sampling granularity of "Token" with "Weights (stop-grad)" as the clipping style, against "Objective (bilateral)" for PPO and GRPO9 (§6.2.11, Table 3).
The gradient is the PPO-style gradient of Section 2.1 with two changes. The mask is always 1, and the weight is \(\min(r,\epsilon_{max})\) for both signs of the advantage. The bias is easy to state (derived): the exact importance-corrected gradient is \(\mathbb E_{old}[\,r\,\hat A\nabla\log\pi\,]\), the CISPO gradient is \(\mathbb E_{old}[\,\min(r,c)\,\hat A\nabla\log\pi\,]\), and the difference is \(\mathbb E_{old}[(r-c)^+\hat A\nabla\log\pi]\). It is nonzero only for tokens whose probability has grown by more than the cap since sampling, a factor of 5 at TRL's recommended \(\epsilon_{max}\). With no weight above the cap, CISPO is the unbiased corrected gradient; M1 notes the same limit: without weight clipping the objective "reduces to the standard policy gradient objective"3 (§3.1).
3.2 Fork tokens: the case it was built for
M1's diagnosis came from zero-RL runs on its hybrid-attention model. GRPO did not elicit long chain-of-thought behavior, and controlled ablations traced the cause to the clip: "tokens associated with reflective behaviors (e.g., However, Recheck, Wait, Aha), which often serve as 'forks' in reasoning paths, were typically rare and assigned low probabilities by our base model", so after the first update they "exhibit high \(r_{i,t}\) values" and "were clipped out after the first on-policy update, preventing them from contributing to subsequent off-policy gradient updates"3 (§3.1). The mechanism is the probability-headroom arithmetic of Module 3 (Table 4) seen from the weight's side. Figure 4 draws it for a common token and a rare one.
M1 adds that these low-probability tokens "are often crucial for stabilizing entropy" and for "scalable RL", and that increasing the upper bound, DAPO's remedy, was less effective with 16 off-policy rounds. Module 13 takes up the entropy side. The cost of keeping their gradient is, on the writer's reading, variance: a rare token pushed with weight up to 5 contributes five times the usual gradient, which is why the cap exists and why ScaleRL sweeps \(\epsilon_{max}\).
3.3 What the comparisons show
M1's own study trains Qwen2.5-32B base on DAPO's math data and scores AIME 2024: CISPO beats both GRPO and DAPO at equal steps and "achieves comparable performance to DAPO using 50% of the training steps", the "2x speedup" that its introduction and Figure 2 report3 (§1, §3.1). It is one model and one benchmark. ScaleRL provides the independent comparison, on an 8B dense model with the Polaris-53K math set and a 16,384-token sequence budget, fitting a sigmoid to pass rate against GPU-hours14 (§2, §3.2).
| Loss | Asymptote \(A\) | \(B\) | Sensitivity to the clip constant | Stability note |
|---|---|---|---|---|
| CISPO | 0.595 | 2.18 | no significant difference for \(\epsilon_{max}\in\{4,5,8\}\) | "best balance of stability and robustness"; ran to about 7k GPU-hours |
| GSPO | 0.590 | 1.80 | the \(10^{-4}\) scale from its paper "did not work well"; \(10^{-3}\) scale did, upper 5e-3 slightly better than 4e-3 | "diverged mid-training" on multiple occasions (Section 4.5) |
| DAPO loss | 0.520 | 4.55 | "highly sensitive" to \(\epsilon_{max}\); the constant moves the asymptote | \(\epsilon_{max}=0.26\) ran to about 5k GPU-hours |
Two caveats from the same paper keep the table honest. Inside the full ScaleRL recipe, reverting to the DAPO loss "yields similar asymptotic performance to CISPO", and the authors prefer CISPO because it is "markedly more robust" to \(\epsilon_{max}\) and more efficient in the leave-one-out run (\(B=2.01\) against 1.77); they add that "a carefully tuned DAPO variant can perform similar asymptotically"14 (§4). So the evidence supports "CISPO and GSPO beat an untuned token-level clip, and CISPO is the least fussy", not "token-level clipping cannot reach the same asymptote". The second caveat is that the gap in Table 2 is measured with ScaleRL's other choices already fixed (prompt-level averaging, FP32 logits, zero-variance filtering), the ones Module 11 and Module 15 weigh.
4. GSPO: the sequence is the unit
4.1 The argument, and what is and is not proven
The Qwen team's diagnosis is that GRPO's instability "stems from the fundamental misapplication and invalidation of importance sampling weights"7 (§1). Importance sampling estimates \(\mathbb E_{\pi_{tar}}[f]\) by reweighting samples from \(\pi_{beh}\), and the correction "relies on averaging over multiple samples (\(N\gg1\))". GRPO's token weight \(\pi_\theta(y_{i,t}\mid\cdot)/\pi_{\theta_{old}}(y_{i,t}\mid\cdot)\) is computed from one sample at each position, so it "fails to perform the intended distribution-correction role" and instead "introduces high-variance noise into the training gradients, which accumulates over long sequences and is exacerbated by the clipping mechanism"7 (§3). The authors report that the resulting collapse is often irreversible, even "when reverting to a previous checkpoint and meticulously tuning hyperparameters". Their principle is that "the unit of optimization objective should match the unit of reward", and since the reward is granted to the whole sequence the correction belongs at the sequence level7 (§3). Lambert's textbook gives the same two motivations, a practical one (per-token ratios are "numerically unstable") and a conceptual one (granularity of correction should match granularity of reward)9 (§6.2.9).
Two points sit between that argument and the loss. First, the per-token ratio is not a mistake in the TRPO derivation. TRPO's practical estimator replaces "the sum over the actions by an importance sampling estimator" at each state visited by the old policy17 (§5, Eq. 14), one sampled action per state in its single-path variant; what that surrogate approximates is the state distribution, not the action weights (Module 3, Section 2.1). TRPO's vine variant draws several actions from one state, which is the multi-sample situation the GSPO paper says importance sampling needs; GRPO-family recipes sample one next token per prefix, and Module 3 (Section 3.3) covers the vine-style exception for advantage estimation. Second, the quantity GSPO clips is not the sequence importance weight. The exact weight for a whole response is the product \(\prod_t r_t\), whose variance grows with length. GSPO takes its \(1/|y|\) power, \(s_i=\exp(\tfrac1{|y_i|}\sum_t\log r_{i,t})\), the geometric mean of the token ratios, and says why: "we adopt length normalization in \(s_i(\theta)\) to reduce the variance and to control \(s_i(\theta)\) within a unified numerical range"7 (§4.1). Because \(\mathbb E_{old}[s_if(y_i)]\neq\mathbb E_\theta[f]\) for \(|y_i|>1\) (derived), \(s_i\) is a sequence-level trust-region statistic rather than an unbiased correction. That does not make it a worse design. It does mean the theoretical argument for GSPO and the loss GSPO runs are not the same object, and the evidence for it has to be empirical.
4.2 The loss and its gradient
With \(\hat A_i\) the group-normalized advantage, GSPO maximizes \(\frac1G\sum_i\min\big(s_i\hat A_i,\ \mathrm{clip}(s_i,1-\epsilon,1+\epsilon)\hat A_i\big)\) (Eq. 5-7)7. Using \(\nabla s_i=s_i\nabla\log s_i\), the unclipped gradient is \(\frac1G\sum_i s_i\hat A_i\,\frac1{|y_i|}\sum_t\nabla\log\pi_\theta(y_{i,t}\mid\cdot)\) (Eq. 10). Every token of response \(i\) has the same gradient weight \(s_i/|y_i|\), where GRPO weights each token by its own ratio, a value that "can vary among \((0,1+\epsilon]\) (for \(\hat A_i>0\)) or \([1-\epsilon,+\infty)\) (for \(\hat A_i<0\))" (Eq. 12). I checked the weight \(s_i/|y_i|\) against autograd on a 7-token example: the difference is \(3\times10^{-17}\) in float64. Two consequences follow.
- The \(1/|y_i|\) is still there. Tokens in longer responses receive less gradient weight under GSPO exactly as under GRPO's per-sequence average (Module 11), because GSPO keeps the mean over tokens inside the exponent. TRL warns that token-summed loss types combined with
importance_sampling_level="sequence"instead weight each sequence "by its completion length", and says to useloss_type="grpo"to reproduce the paper16. TRL also ships LUSPO, described there as a length-unbiased modification of GSPO; it is a pointer only here10. - Advantages stay at sequence level. For multi-turn work the paper adds GSPO-token, \(s_{i,t}=\mathrm{sg}[s_i]\,\pi_\theta(y_{i,t})/\mathrm{sg}[\pi_\theta(y_{i,t})]\), which is numerically equal to \(s_i\) and gives the same gradient when all token advantages are equal, but allows per-token advantages7 (§4.3, Eq. 13-17). Module 18 returns to it.
4.3 Why the clip range is 3e-4, and what that does to a response
GSPO uses left and right ranges of 3e-4 and 4e-4, against 0.2 and 0.27 for the GRPO baseline it was compared with; the authors note that the ranges "differ in order of magnitude due to the distinct definitions of importance ratios"7 (§4.1, §5.1). The sequence statistic is a mean of \(L\) token log-ratios, so a band of \([-3\times10^{-4},\,+4\times10^{-4}]\) on \(\log s_i\) allows a total log-likelihood change of at most 0.4 nats at \(L=1{,}000\) and 3.2 nats at \(L=8{,}000\), and the corresponding sequence-likelihood ratio can reach \(e^{0.4}=1.49\) or \(e^{3.2}=24.5\) (derived). The band is a budget in total nats that grows with length. Table 3 and Figure 5 run the arithmetic for a 1,000-token response with a positive advantage.
| Case | GSPO \(s\) | GRPO tokens with gradient | GSPO tokens with gradient | CISPO tokens with gradient |
|---|---|---|---|---|
| 1 token at \(r=1.4\) | 1.000336 | 999 | 1,000 | 1,000 |
| 1 token at \(r=1.5\) | 1.000406 | 999 | 0 | 1,000 |
| 5 tokens at \(r=2\) | 1.003472 | 995 | 0 | 1,000 |
| 50 tokens at \(r=1.1\) | 1.004777 | 1,000 | 0 | 1,000 |
| all 1,000 tokens at \(r=1.0005\) | 1.000500 | 1,000 | 0 | 1,000 |
Token ratios against the sequence ratio
\(s=\exp(m\ln r/L)\) with \(A>0\). GSPO's clip fires when \(s>1+h\cdot10^{-4}\); a single token trips it once \(r>e^{hL\cdot10^{-4}}\), so the same token ratio that GRPO clips (about 1.27) leaves a 1,000-token response intact under GSPO until \(r\) reaches 1.49, and trips a 100-token response at 1.04. Defaults are the paper's 4e-4 and the Figure 5 case.
Reading Table 3 as a trade-off: GSPO's budget is generous for one jumpy token (row 1 keeps all 1,000 tokens where GRPO drops one) and punishing for any response whose total drift crosses 0.4 nats, however it is spread (rows 2 to 5). That is the behavior the SAPO abstract objects to: "when a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence"6 (abstract). The GSPO paper measured the consequence and reads it the other way. Averaged over training, GSPO clips a fraction of 0.15 of tokens against 0.0013 for GRPO, "a difference of two orders of magnitude", and still trains more efficiently; the authors conclude that GRPO's token-level gradient estimates "are inherently noisy and inefficient for sample exploitation"7 (§5.2, Figure 2). The two readings are compatible with the same curve: dropping the sequences that drifted most can act as a stricter, better-aimed trust region, and it can also discard useful signal. The paper does not run the ablation that would separate them, such as GSPO with the band widened until the clip fraction matches GRPO's.
4.4 MoE routing: replay it, or average it away
The GSPO paper's strongest practical argument is about mixture-of-experts models. Its measurement: in the 48-layer Qwen3-30B-A3B-Base, "after each RL gradient update and for the same rollout sample, there are roughly 10% of the experts activated under the new policy \(\pi_\theta\) that are different from those under the old policy", a phenomenon the authors say becomes more prominent in deeper MoE models7 (§5.3). A token whose router picks different experts has a different output distribution for reasons unrelated to the gradient step, so its ratio \(w_{i,t}\) "fluctuate[s] drastically", and GRPO's token-level ratios stop meaning "this token's probability moved". The Qwen team's earlier fix was Routing Replay: cache the experts activated under \(\pi_{old}\) and force \(\pi_\theta\) to use them when computing the ratio. Their Figure 3 shows GRPO converging on MoE only with it; they also say it "incurs additional memory and communication overhead and can also limit the actual capacity of the MoE model"7 (§5.3). GSPO drops the cache. The stated reason is that it "focuses only on the sequence likelihood and is not sensitive to the individual token likelihood", and since the MoE "always maintains its language modeling capability, the sequence likelihood will not fluctuate drastically"; Figure 1 plots GSPO against GRPO with Routing Replay, and GSPO's training reward, AIME 2024, LiveCodeBench and CodeForces curves sit above the GRPO curves at equal compute7 (§5.1, §5.3).
The other lab with a published MoE recipe chose the other branch. DeepSeek-V3.2 keeps token-level GRPO and adds Keep Routing: "we preserve the expert routing paths used during sampling in the inference framework and enforce the same routing paths during training", an operation "found crucial for RL training stability of MoE models" and "adopted in our RL training pipeline since DeepSeek-V3-0324"8 (§3.1). Neither report runs the other's method, so the choice is a design bet and not a measured comparison. The mechanisms do differ in what they pay: replay needs the inference engine to export routing decisions and the trainer to consume them, a systems cost (Module 14 and System Design Module 37), while GSPO pays in the band arithmetic of Section 4.3. GSPO also claims a second benefit, that sequence likelihoods tolerate training-versus-inference precision differences well enough to use the inference engine's own probabilities without recomputing them in the trainer7 (§5.4). The paper states this as "intuitively" true and reports no experiment, so it belongs with the numerics discussion of Module 14, not with the evidence here.
4.5 Independent evidence, and where it disagrees
Outside the Qwen team, the main measurement is ScaleRL's (Table 2). Its GSPO asymptote, 0.590, is within 0.005 of CISPO's and 0.070 above the DAPO loss on the same 8B dense model, so the sequence-level ratio does help relative to an untuned token-level clip14 (Fig. 5a). The same appendix reports problems that the GSPO paper does not. The paper's \(10^{-4}\) clip scale "did not work well in our setting", and a \(10^{-3}\) scale was needed. Past that, "GSPO runs diverged mid-training" on multiple occasions; for 8B models restarting from a stable checkpoint recovered, but on the 17B×16 MoE "instability persisted despite repeated resetting to a stable checkpoint", with no implementation bug found14 (App. A.17.3-A.17.4). That is the opposite of GSPO's headline claim of stability, from a group that originated neither method, in one lab's training framework. Two other notes belong here. The GSPO paper says it contributed to "the latest Qwen3 models"7; the Qwen3 technical report on arXiv (v1, May 2025) predates GSPO and does not mention it (text search), so the Qwen3 case study (Module 29) cannot be used as independent confirmation18. And Lambert's textbook notes that CISPO and GSPO come from groups "pushing the limits of applying RL on large-scale MoE models" and are "less studied and beneficial within smaller, academic experiments"9 (§6.2.10).
5. Masks and gates: Kimi, GLM-5, DeepSeek-V3.2 and SAPO
The remaining designs share one idea with Section 2.2's band mask: stop trusting a token or a sequence once it has drifted too far from what generated it. Kimi K2.5 and GLM-5 do so regardless of the sign of the advantage; DeepSeek-V3.2 and SAPO keep a sign dependence. They differ in what "drifted" is measured against, and that is the useful axis (Figure 2).
5.1 Kimi K2.5 and K3: a log-ratio band
K2.5 prints its objective as a squared-error surrogate in the lineage of K1.5's mirror-descent loss (Module 9), with a clip added:
$$L_{RL}(\theta)=\mathbb E\Big[\tfrac1N\sum_{j=1}^{K}\sum_{i=1}^{|y_j|}\Big(\mathrm{Clip}\big(\tfrac{\pi_\theta(y^i_j\mid\cdot)}{\pi_{old}(y^i_j\mid\cdot)},\alpha,\beta\big)\,\big(r(x,y_j)-\bar r(x)\big)-\tau\log\tfrac{\pi_\theta(y^i_j\mid\cdot)}{\pi_{old}(y^i_j\mid\cdot)}\Big)^{2}\Big]$$with \(N\) the number of generated tokens in the batch and \(\alpha,\beta,\tau>0\) hyperparameters4 (§4.4.2, Eq. 1). The prose describes something slightly different from the formula: "a simple gradient masking scheme: policy gradients are computed normally for tokens with log-ratios within the interval \([\alpha,\beta]\), while gradients for tokens falling outside this range are zeroed out", introduced "to mitigate the off-policy divergence amplified by discrepancies between training and inference frameworks"4. The printed clip acts on the ratio and the prose on the log-ratio. I follow the prose as the operational definition. Read literally, though, the printed squared form does not zero every gradient outside the band: there the clipped ratio is a constant, so the gradient through it vanishes, but the \(-\tau\log\)-ratio term inside the square still carries a gradient, and the prose's "zeroed out" is the stronger statement (derived reading, not checked against an implementation; Module 28, Section 4.1 derives it). The values of \(\alpha\), \(\beta\) and \(\tau\) are not given in the text searched, and the report offers no ablation beyond saying the mechanism is "essential for maintaining training stability in complex domains requiring long-horizon, multi-step tool-use reasoning". Kimi K3 states that its policy optimization "follows the algorithm in Kimi K2.5" and adds partial rollouts, which make a trajectory span several policy versions; staleness is handled "through a per-token regularization" that "constrains policy updates within a localized neighborhood"15 (§4.1.2). That is a claim of tolerance, with no staleness measurement or comparison in the text searched.
5.2 GLM-5: two stages, two answers
GLM-5 uses different ratio machinery for its two RL regimes. For reasoning RL it runs GRPO with IcePop and no KL term "to accelerate RL improvement". The loss multiplies each token's usual clipped term by \(\mathrm{pop}(\rho_{i,t},1/\beta,\beta)\), where \(\rho_{i,t}=\pi^{train}_{\theta_{old}}/\pi^{infer}_{\theta_{old}}\) is the mismatch ratio of Section 1.2 and \(\mathrm{pop}(\rho,1/\beta,\beta)\) equals \(\rho\) when \(1/\beta\le\rho\le\beta\) and 0 otherwise; the hyperparameters are \(\beta=2\), \(\epsilon_{low}=0.2\), \(\epsilon_{high}=0.28\)5 (§3.2, Eq. 1). Olmo 3's truncated importance weight with a cap of 2.0 is the one-sided cousin: IcePop zeroes tokens whose mismatch is beyond a factor of 2 in either direction, where truncation only caps the upper side13. The report describes this stage as "entirely on-policy"5 (§3.2); if that means one update per rollout, \(r=1\) in value and the PPO clip is inert, and the mismatch mask is the only active ratio term (the writer's inference from Table 1).
For asynchronous agentic RL the problem is different. A rollout engine "may undergo multiple updates during a single trajectory generation", so exact \(\pi_{old}\) would require "an extensive history of model checkpoints". GLM-5 therefore drops \(\pi_{old}\) and uses the log-probability the engine returned at generation time as the denominator:
$$L(\theta)=\mathbb E_t\big[f(r_t;\epsilon_\ell,\epsilon_h)\,\hat A_t\log\pi_\theta(a_t\mid s_t)\big],\quad r_t=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{rollout}(a_t\mid s_t)},\quad f(x)=\begin{cases}x&1-\epsilon_\ell5.3 DeepSeek-V3.2: mask the negatives that drifted
V3.2 keeps token-level GRPO with its clip and adds a binary mask on whole sequences (Eq. 8-9):
$$M_{i,t}=\begin{cases}0&\hat A_{i,t}<0\ \text{and}\ \dfrac1{|o_i|}\sum_{t'=1}^{|o_i|}\log\dfrac{\pi_{old}(o_{i,t'}\mid q,o_{i,where \(\pi_{old}\) "denotes the sampling probability directly returned by the inference framework, thus the KL divergence between the old and current policy accounts for both sources of off-policyness", drift and mismatch8 (§3.1). The report masks only negative advantages because "the model benefits the most by learning from its own mistakes, whereas highly off-policy negative samples can be detrimental", and says only that the mask "improves stability in certain training scenarios that would otherwise exhibit instability"; the value of \(\delta\) and any ablation are not given. In GSPO's notation the left side of the condition is \(-\log s_i\), so the mask is "\(\hat A_i<0\) and \(s_i
5.4 SAPO, and the rest of TRL's menu
SAPO replaces the hard band with the smooth gate of Section 2.2, \(f(r)=\sigma(\tau(r-1))\cdot4/\tau\) with \(\tau_{pos}=1.0\) and \(\tau_{neg}=1.05\) by default, "penalized more strictly for 'bad' actions to prevent instability, while being more permissive with 'good' actions"10. The abstract positions it against both predecessors: relative to GRPO it swaps hard token clipping for "smooth, temperature-controlled scaling", and relative to GSPO it "selectively down-weights only the offending tokens" instead of suppressing all gradients for a sequence, "improving sample efficiency"; the authors report better stability and Pass@1 on math benchmarks and use it to train the Qwen3-VL series6. I read only the abstract and TRL's documentation and code, so the abstract's claim that SAPO is "sequence-coherent" (TRL's formula is per token) and every number behind the efficiency claim are unverified here. TRL's current loss menu is a good index of how fast this area moves: grpo, dr_grpo, dapo (default), bnpo, cispo, sapo, luspo and vespo, plus the INTELLECT-2 delta bound and the V3.2 mask10.
6. Eight designs side by side
Table 4 puts the designs of Sections 2 to 5 in the same columns; Table 5 lists the constants each report ships.
| Design | Ratio unit | Clipped or masked | Sign-dependent | Gradient outside the band | MoE routing noise | Stale rollouts | Source |
|---|---|---|---|---|---|---|---|
| PPO / GRPO clip (0.2) | token | update, one side | yes | zero | needs Routing Replay (measured by Qwen) | bounds favourable drift only | 17 |
| Clip-Higher (0.2 / 0.28) | token | update, wider upside | yes | zero | not addressed | "less effective" at 16 rounds; \(\epsilon_{max}\) moves the asymptote | 2314 |
| GSPO | sequence, length-normalized | whole response, band 3e-4 / 4e-4 | yes | zero for all tokens of the response | claim: no replay needed; Figures 1 and 3 | claim: precision-tolerant (no experiment); diverged in ScaleRL | 714 |
| CISPO | token | weight capped, stop-gradient | no | kept, weight at cap | not addressed | 16 off-policy rounds; used with PipelineRL-8 | 314 |
| Kimi K2.5 mask | token | log-ratio outside \([\alpha,\beta]\) | no | zero, both sides | not addressed | claim: essential in long tool-use runs; K3 claims extreme staleness is tolerated | 415 |
| GLM-5 double-sided IS | token, vs \(\pi_{rollout}\) | ratio outside \((1-\epsilon_\ell,1+\epsilon_h)\); stale samples dropped | no | zero | not addressed in this stage | asynchronous by design; bias accepted | 5 |
| V3.2 sequence mask | sequence, engine \(\pi_{old}\) | negative-advantage responses with drift above \(\delta\) | yes (negatives only) | zero for that response | paired with Keep Routing | claim: "improves stability in certain training scenarios" | 8 |
| SAPO | token | nothing masked; smooth gate | yes (temperature) | decays smoothly | abstract: variance "exacerbated" in MoE | abstract: "attenuates off-policy updates" | 610 |
6.1 Values that ship
| Recipe | Lower bound | Upper bound or cap | Other ratio-related settings |
|---|---|---|---|
| GSPO paper, Qwen3-30B-A3B cold start7 | 3e-4 (sequence) | 4e-4 (sequence) | GRPO baseline 0.2 / 0.27 with Routing Replay; 4 minibatches |
| DAPO2 | 0.2 | 0.28 | 16 updates per rollout; no KL |
| Olmo 3 Think and Instruct13 | 0.2 | 0.272 | truncated-IS cap 2.0 on 32B Think and RL-Zero; max asynchrony 1 to 8 |
| GLM-5 reasoning RL5 | 0.2 | 0.28 | IcePop \(\beta=2\) on the mismatch ratio; no KL |
| MiniMax-M1 CISPO3 | none (set large) | \(\epsilon^{IS}_{high}\) tuned, value not stated | 16 off-policy rounds; lowered during length scaling |
| ScaleRL14 | CISPO: none | CISPO \(\epsilon_{max}\in\{4,5,8\}\) tried; DAPO baseline 0.26 | GSPO needed a \(10^{-3}\) scale, upper 5e-3; PipelineRL-8 |
| R1, first RL stage12 | \(\varepsilon=10\), printed once; ratio printed at sequence level | 16 minibatches; KL 0.001 | |
| TRL defaults10 | 0.2 | same as lower; DAPO suggests 0.28; CISPO suggests 5.0 | token level; delta off; SAPO \(\tau\) 1.0 / 1.05; vLLM mismatch weight on: sequence-level product, zeroed above 3.0; V3.2 mask off |
6.2 The variants as code
The listing writes the five per-token or per-sequence losses and the V3.2 keep-mask from the equations above and TRL's trainer. I checked the gradient of each loss against \(w(r)\,M\,\hat A\) (and \(s_i/|y_i|\) for GSPO) on a random batch of 4 responses with padding in float64; the maximum difference is \(2\times10^{-16}\).
import torch
def variant_loss(kind, logp, logp_old, adv, mask, eps_lo=0.2, eps_hi=0.28, eps_max=5.0,
band=(0.8, 1.28), tau_pos=1.0, tau_neg=1.05, s_lo=3e-4, s_hi=4e-4):
"""logp, logp_old, mask: (B, T); adv: (B, 1). Per-token loss (B, T), to be summed."""
log_r = logp - logp_old
r = log_r.exp() # token ratio
if kind == "ppo": # clip the update: PPO, GRPO, DAPO
return -torch.minimum(r * adv, r.clamp(1 - eps_lo, 1 + eps_hi) * adv)
if kind == "cispo": # clip the weight, stop-gradient, no floor
return -r.clamp(max=eps_max).detach() * adv * logp
if kind == "band": # sign-blind mask: K2.5, GLM-5 agentic
keep = ((r >= band[0]) & (r <= band[1])).detach()
return -(keep * r * adv)
if kind == "sapo": # soft gate, TRL's formulation
tau = torch.where(adv > 0, torch.full_like(adv, tau_pos), torch.full_like(adv, tau_neg))
return -(torch.sigmoid(tau * (r - 1)) * 4 / tau) * adv
if kind == "gspo": # one length-normalised ratio per response
n = mask.sum(-1, keepdim=True)
s = ((log_r * mask).sum(-1, keepdim=True) / n).exp() # (B, 1)
per_seq = -torch.minimum(s * adv, s.clamp(1 - s_lo, 1 + s_hi) * adv)
return per_seq.expand_as(logp) / n # each token gets weight s / |y|
raise ValueError(kind)
def v32_keep(adv, logp, logp_sampler, mask, delta):
"""DeepSeek-V3.2 off-policy sequence mask: True = keep. logp_sampler: engine's old log-probs."""
drift = ((logp_sampler - logp.detach()) * mask).sum(-1, keepdim=True) / mask.sum(-1, keepdim=True)
return (adv >= 0) | (drift <= delta)
Where experts disagree
Everyone in this module agrees that reuse, staleness and engine mismatch make the ratio do real work. They disagree about what to do with it: keep the token ratio and tune the band, change its unit, stop clipping the update, or add a drift test. The evidence is thin and uneven. Each lab reports its own method, and the one independent comparison covers a single 8B dense model.
A. Keep the token ratio; tune the band and fix the source of drift
DAPO, Olmo 3, GLM-5 reasoning RL and DeepSeek-V3.221358; DeepSeek-V4's specialist RL keeps GRPO with hyperparameters "closely aligned" with its earlier reports19 (§5.1.1)
The per-token ratio is TRPO's own estimator, so treat engine mismatch and MoE routing as separate defects and fix them where they arise: Clip-Higher for exploration, truncated or masked importance weights for \(\rho\), Keep Routing and Keep Sampling Mask for the MoE and top-p gaps.
- Smallest change from the PPO lineage; per-token credit and the familiar clip fraction survive.
- With one update per rollout and matched engines, as in DeepSeekMath and GLM-5's reasoning stage, the clip is inert and nothing here matters.
B. Make the sequence the unit of the ratio
Qwen team (GSPO)7; Lambert's framing that correction granularity should match reward granularity9; SAPO as a softened descendant6
One sample per token position cannot correct a distribution, and routing noise makes token ratios meaningless; a length-normalized sequence ratio is stable and removes the need for Routing Replay.
- Qwen's MoE runs converge without replay, and ScaleRL's asymptote for GSPO (0.590) is far above its DAPO loss (0.520)14.
- One scalar per response is easy to log and to ship from the inference engine.
- The geometric-mean ratio is not an unbiased importance weight, and a single token above 1.49 trips the band at \(L=1{,}000\) (Table 3).
- Independent runs diverged and the clip scale did not transfer from \(10^{-4}\) to \(10^{-3}\)14; the \(1/|y|\) length weight remains.
C. Clip the importance weight and keep every token's gradient
MiniMax (CISPO)3; adopted in ScaleRL14 and recommended with \(\epsilon_{max}=5\) in TRL10
Fork tokens such as "Wait" carry the exploration that reasoning RL needs, and a hard clip removes them after one update; cap the weight instead and accept a small bias.
- Best asymptote in ScaleRL (0.595) and no significant change across \(\epsilon_{max}\) in {4, 5, 8}; 2x over DAPO in M1's own study.
- Nothing is dropped, so the clip fraction cannot hide discarded data.
- Biased gradient by the authors' own account, and rare tokens get up to 5x weight.
- Inside ScaleRL's full recipe a carefully tuned DAPO loss reaches a similar asymptote14.
D. Test drift directly, with a mask or a gate
Kimi K2.5 and K3415; GLM-5 agentic RL5; DeepSeek-V3.2's sequence mask8; SAPO6
In asynchronous, tool-using, mismatch-prone runs the ratio measures how far a token is from the policy that generated it, not how much the policy learned; bound that distance, against the logged rollout probability where possible.
- Handles staleness without checkpoint history (GLM-5) and targets the negative-advantage tail where large weights are most harmful (V3.2).
- Matches the production reality that \(\rho\) and version lag are measured quantities.
- No public ablations or values for \(\alpha,\beta,\delta,\epsilon_\ell,\epsilon_h\) in the text searched; stability claims rest on the authors' word.
- The sign-blind masks (K2.5, GLM-5) discard the corrective gradients that PPO's \(\min\) keeps (Section 2.2).
Where reuse is one update per rollout and the engines agree, the ratio is inert and the choice is a matter of taste. Where there are 4 to 16 minibatches, asynchrony or MoE routing, the evidence supports a ranking with different confidence at each rung. An untuned token-level clip is the weakest-supported option, but ScaleRL's own leave-one-out run shows a tuned DAPO loss closing the asymptote gap, so the sturdier finding is robustness and efficiency, not ceiling. CISPO is the best-supported default: an independent comparison, the least sensitivity to its constant, a recommendation in TRL, and failure modes that are documented and bounded (bias past the cap, rare-token variance). GSPO has the most coherent MoE story but only its authors' runs show the benefit, while the one independent study saw divergence and a clip scale that did not transfer; test it with a per-model band and a length-unbiased variant, not as a drop-in. The masks and gates are responses to measured mismatch and staleness and are best used as safety nets after \(\rho\) has been measured (Module 14). What would settle the question is a same-codebase, same-data comparison on one dense and one MoE model that logs staleness and \(\rho\) and reports asymptote and efficiency, including GSPO with its band widened until its clip fraction matches GRPO's, CISPO with and without routing replay, and a K2.5-style mask against PPO's.
What is changing
As of 2026-10-27, TRL's main branch ships eight loss types and keeps dapo with token-level ratios as the default; the menu now includes cispo, sapo, luspo and vespo alongside the older variants, so any default recorded here is likely to move10. Kimi K3 reuses K2.5's mask instead of adopting GSPO or CISPO15, and DeepSeek's V4 specialists remain GRPO19, so among the reports read for this course only Qwen (GSPO) and MiniMax (CISPO) train with a sequence-level or weight-clipped loss, and ScaleRL adopts CISPO in a research recipe. The SAPO full paper and any multi-lab comparison with logged mismatch are the two things that would most change the ranking above. The systems side of the same problem is in System Design Modules 37 and 38.
References
- paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv:1707.06347v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the clipped objective, Eq. 7, and ε = 0.2 (§3).
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. (ByteDance Seed / Tsinghua AIR). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: Clip-Higher, εlow = 0.2 and εhigh = 0.28, mean up-clipped probability below 0.2 (§3.1, Fig. 3a); 512 prompts × 16 responses, minibatch 512, 16 gradient updates per rollout (§4.1).
- paperMiniMax (Chen, A., Li, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: CISPO objective Eq. 3-7, the fork-token diagnosis, 16 off-policy rounds, comparison with DAPO and GRPO (§3.1, Fig. 2); lowering the IS-weight bound during length scaling (§5).
- paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.; Moonshot AI). Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276v2 (7 Aug 2026; v1 2 Feb 2026). https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: the policy-optimization objective Eq. 1 and the log-ratio gradient mask, sign-blind by design (§4.4.2, p. 8).
- paperGLM-5 Team (Zeng, A., Lv, X. et al.; Zhipu AI / Tsinghua). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2 (24 Feb 2026; v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: reasoning-RL loss with IcePop, β = 2, ε 0.2 / 0.28, on-policy, no KL (§3.2, Eq. 1); direct double-sided importance sampling, Eq. 3-5, and version-lag dropping (§4.1.2).
- paperGao, C., Zheng, C., Chen, X.-H., Dang, K., Liu, S., Yu, B., Yang, A., Bai, S., Zhou, J., Lin, J. (Qwen Team). Soft Adaptive Policy Optimization. arXiv:2511.20347 (v1 25 Nov 2025; v2 1 Dec 2025). https://arxiv.org/abs/2511.20347. Accessed 2026-10-27. Used for: abstract only (read through the arXiv API): the soft gate, the comparison with GSPO and GRPO, the Qwen3-VL use. Source added beyond the researcher's map.
- paperZheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B. et al. (Qwen Team, Alibaba Inc.). Group Sequence Policy Optimization. arXiv:2507.18071v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: Eq. 5-7 and gradients Eq. 10-12 (§4); clip ranges 3e-4 / 4e-4 against 0.2 / 0.27 and four minibatches (§5.1); clip fractions 0.15 and 0.0013 (§5.2, Fig. 2); roughly 10% of experts changing per update, Routing Replay (§5.3, Fig. 3); inference-engine likelihoods (§5.4).
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: per-token GRPO objective Eq. 5-6, Off-Policy Sequence Masking Eq. 8-9, Keep Routing and Keep Sampling Mask (§3.1, pp. 7-8).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12 (11 Sep 2026); Manning print edition 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: §6.2.9 (GSPO), §6.2.10 (CISPO, TOPR), §6.2.11 Table 3 (granularity and clipping style), §6.3.4 (truncated importance sampling, policy-drift and implementation ratios).
- officialHugging Face TRL maintainers. TRL documentation: GRPO Trainer (main branch; v1.16.0 was the latest tagged release listed). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: loss_type options, the SAPO formula and temperatures, epsilon, epsilon_high, delta, the CISPO recommendation of 5.0, off_policy_mask_threshold, vLLM importance-sampling defaults, logged clip metrics.
- paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: per-token GRPO objective Eq. 3 (§4.1.1); single update per exploration stage (§4.2); dropping min and clip when πθold = πθ (App. A.1, Eq. 15-16).
- paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv v2, the expanded Nature version). arXiv:2501.12948v2, 4 Jan 2026. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the sequence-level objective Eq. 1 (§2.1) and Eq. 11 (Supp. A.3); 16 minibatches and ε = 10 with the rationale sentence (§2.1, §3.2.1); the author-contribution sentence on a large PPO clipping strategy (§7).
- paperTeam Olmo (Ettinger, A., Bertsch, A. et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2 (14 Apr 2026; v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: the OlmoRL objective with truncated importance sampling and clip-higher (§4); Table 49: minibatches, max asynchrony, TIS cap, clip-lower 0.2 and clip-higher 0.272.
- paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R. et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: setup and ratio notation, Eq. 2-4 (§2, §3.2); PipelineRL-k (§3.1); loss-type comparison, Figure 5a (A and B values); leave-one-out discussion (§4); sweeps of εmax and GSPO clip scale, divergence of GSPO runs (App. A.6, A.17).
- paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.; Moonshot AI). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2 (7 Aug 2026; v1 27 Jul 2026). https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: partial rollouts, staleness and per-token regularization (§4.1.2).
- repoHugging Face. trl/trainer/grpo_trainer.py, main branch (201,296 bytes when read). https://raw.githubusercontent.com/huggingface/trl/main/trl/trainer/grpo_trainer.py. Accessed 2026-10-27. Used for:
_compute_loss: the on-policy shortcut for old log-probs, sequence-level log-weight, CISPO, SAPO and GRPO branches,get_off_policy_mask(V3.2 mask), the GSPO warning. - paperSchulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv:1502.05477v5 (20 Apr 2017). https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Used for: the importance-sampling estimator that replaces the sum over actions at each sampled state (§5, Eq. 14-16).
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: text search for GSPO or sequence-level policy optimization (none found).
- paperDeepSeek-AI (Xu, A., Lin, B., Xue, B. et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: specialist RL uses GRPO with hyperparameters closely aligned with prior reports (§5.1.1).