Reinforcement Learning for LLM Post-Training · Module 12

Importance Ratios and Clipping: Token-Level, Sequence-Level (GSPO), IS-Weight Clipping (CISPO), and Log-Ratio Masks

The PPO ratio is a different object in LLM RL than in MuJoCo: it is exactly 1 for one-update recipes, a staleness meter for 16-minibatch recipes, and a mismatch detector once the rollout engine and the trainer disagree on probabilities. This module writes the token-level clip, GSPO's sequence ratio, CISPO's weight clipping, the log-ratio masks of Kimi K2.5 and GLM-5, DeepSeek-V3.2's sequence mask and SAPO's soft gate in one form, so you can say which tokens receive gradient under each, and which design tolerates MoE routing noise and stale rollouts.

Course 45 min read 19 sources As of 2026-10-27 Prerequisites: Module 3, Module 9, Module 11
A two-by-four map of ratio designs: the unit that carries the ratio (token or sequence) against what happens outside the trust band (clip the update, clip the weight, mask by band, soft gate) What happens to a token or sequence outside the trust band Clip the updategradient goes to 0 Clip the weightgradient kept, capped Mask by bandzero outside the band Soft gatesmooth decay Token ratior_t = π_θ / π_oldone per token Sequence ratios_i, length-normalizedone per response PPO · GRPODAPO ε 0.2 / 0.28 upsign-dependent:one side only CISPOmin(r, ε_max)stop-gradient,no lower floor Kimi K2.5log r in [α, β] GLM-5 agenticπ_θ/π_rollout in bandsign-blind SAPOσ(τ(r−1)) · 4/ττ by advantage signtoken-adaptive GSPOclip s_i at3e-4 / 4e-4whole response TOPRclips IS weight,by reward signpointer only DeepSeek-V3.2mask response ifA < 0 and drift > δnegatives only VESPOGamma weighton s_i (TRL)pointer only Columns follow Figure 3's colours: blue clip, orange weight clip, cyan band mask, green soft gate.
Figure 1. Every design in this module answers two questions: which unit carries the ratio (row) and what happens outside the trust band (column). Entries: PPO-style clip1 and Clip-Higher2; CISPO3; Kimi K2.54; GLM-55; SAPO6; GSPO7; DeepSeek-V3.28; TOPR (as described by Lambert)9; VESPO as documented in TRL10. Placement is the writer's classification.

1. When the ratio is not 1

Module 3 derived the PPO clip for a ratio \(r_t=\pi_\theta(a_t\mid s_t)/\pi_{old}(a_t\mid s_t)\) that starts at 1 and drifts as the optimizer takes steps on a fixed batch. Whether that drift exists at all is a property of the recipe, not of the algorithm, and the reports differ more on this than on any clipping constant. Everything later in this module is a different answer to the question of what to do with a ratio that has drifted, so the first job is to say where the drift comes from.

1.1 Reuse per rollout is a staleness dial

DeepSeekMath-RL took one update per exploration stage: "the policy model only has a single update following each exploration stage"11 (§4.2), and the appendix drops the \(\min\) and the clip from the objective under exactly that assumption, "thereby ensuring that \(\pi_{\theta_{old}}=\pi_\theta\)"11 (App. A.1, Eq. 16). The clip in GRPO as published is inert. R1's first RL stage did the opposite: each rollout of 8,192 outputs is "randomly split into 16 minibatches and trained for only a single inner epoch"12 (§2.1, §3.2.1). Minibatch \(j\) is trained after \(j-1\) optimizer steps on data that were sampled before the first of them, so the staleness is 0, 1, …, 15 steps (derived). Averaged over the rollout that is \((16-1)/2=7.5\) steps, and only \(1/16=6.25\%\) of the minibatches are on-policy. With four minibatches the figures are 1.5 steps and 25%. Table 1 lists what the sources report.

Table 1. How much reuse and lag each recipe builds in, and which ratio its loss sees. "Updates per rollout" counts the optimizer steps taken on one rollout; lag is stated only where the report states it.
RecipeUpdates per rollout and lagRatio the loss seesSource
DeepSeekMath-RL1 update per exploration stage\(\equiv 1\); clip removed in the analysis§4.2, App. A.111
R1, first RL stage16 minibatches of 512 from 8,192 outputs, 1 epochdrift, 0 to 15 steps§2.1, §3.2.112
DAPO16 gradient updates per rollout step (512 prompts × 16 responses, minibatch 512)drift, 0 to 15 steps§4.12
MiniMax-M1 (CISPO study)"16 rounds of off-policy updates per generation batch"drift, up to 15 steps§3.13
GSPO paper4 minibatches per rolloutdrift, 0 to 3 steps§5.17
Olmo 3 32B Think1 minibatch; max asynchrony 8; truncated-IS cap 2.0async lag up to 8 steps, plus engine mismatchTable 4913
Olmo 3 7B Instruct4 minibatches; max asynchrony 8drift plus async lagTable 4913
ScaleRL (PipelineRL-8)generators take new weights mid-generation; trainers wait if they get \(k=8\) steps aheadasync lag up to 8 steps§3.114
GLM-5 reasoning RLdescribed as "entirely on-policy", group 32, batch 32mismatch only (Section 1.2)§3.25
GLM-5 agentic RLfully asynchronous; a trajectory can span several policy versions; sample dropped if too stale\(\pi_\theta/\pi_{rollout}\)§4.1.25
Kimi K3partial rollouts: generation pauses once a fraction \(\lambda\) of trajectories finish; the rest resume next iterationa trajectory spans several iterations§4.1.215
TRL defaultnum_iterations = 1\(r\equiv 1\) in value (old log-probs = current, detached); with vLLM, a separate mismatch weight \(\rho\) multiplies the lossdocs, source1016

A ratio equal to 1 in value is not a zero-information case for the gradient. Because \(\nabla_\theta r_t=r_t\nabla_\theta\log\pi_\theta\), the gradient at \(r_t=1\) is the REINFORCE term \(\hat A\,\nabla\log\pi\) of Module 2, which is what DeepSeekMath's Eq. 16 keeps. TRL implements the same shortcut: when there is one iteration per batch and the generation batch fits in one accumulation window, the old log-probabilities are the current ones, detached, and the clip fraction is identically zero. With vLLM generation TRL always recomputes the old log-probabilities in the trainer, still before the update, so \(r\) is 1 in value and the engine mismatch enters as the separate weight \(\rho\) (Section 1.2)16 (_compute_loss, _generate_and_score_completions). A logged clip fraction of 0 in such a run says nothing about whether \(\epsilon\) was well chosen.

1.2 Two ratios hide inside one

There is a second source of \(r\neq 1\) that has nothing to do with the optimizer. The rollout engine (vLLM or SGLang) and the trainer run different kernels, precisions and parallelism, so even with identical weights their token probabilities differ. Lambert separates the two corrections: the policy-drift ratio \(\pi_\theta/\pi_{\theta_{old}}\) "corrects for policy drift across multiple gradient steps within one RL batch", and the implementation ratio \(\pi^{learner}_\theta/\pi^{sampler}_\theta\) corrects for "implementation-induced" differences; they are "complementary" and enter policy-gradient code "for different reasons"9 (§6.3.4). Writing the old policy as evaluated by the trainer, the sampler-to-current ratio factors exactly:

$$\underbrace{\frac{\pi^{train}_{\theta}(a_t)}{\pi^{infer}_{\theta_{old}}(a_t)}}_{\text{what the data were sampled from}}=\underbrace{\frac{\pi^{train}_{\theta}(a_t)}{\pi^{train}_{\theta_{old}}(a_t)}}_{r_t:\ \text{optimizer drift}}\times\underbrace{\frac{\pi^{train}_{\theta_{old}}(a_t)}{\pi^{infer}_{\theta_{old}}(a_t)}}_{\rho_t:\ \text{engine mismatch}}$$

Designs differ in which factor they act on, and Figure 2 places them. Olmo 3 multiplies the loss by a truncated mismatch weight, with a cap of 2.0 for the 32B Think run, and still clips \(r\) with \(\epsilon_{low}=0.2,\ \epsilon_{high}=0.272\)13 (OlmoRL, Table 49). GLM-5's reasoning stage applies IcePop, a mask \(\mathrm{pop}(\rho,1/\beta,\beta)\) with \(\beta=2\) on \(\rho\), then the usual clip on \(r\) with 0.2 and 0.285 (§3.2, Eq. 1). ScaleRL writes its ratio as \(\pi^{train}_\theta/\pi^{gen}_{\theta_{old}}\), the left-hand side above14 (Eq. 2). GLM-5's asynchronous stage discards \(\pi_{old}\) altogether and uses the logged rollout probability as the denominator5 (§4.1.2). DeepSeek-V3.2 builds its sequence mask from the probability "directly returned by the inference framework"8 (§3.1). Module 14 covers where \(\rho\) comes from and how to correct it; here it matters because several of the designs below are, in effect, mismatch filters that also happen to bound drift.

Three evaluation points of one policy: the rollout engine, the trainer before the update, and the trainer at minibatch j, joined by the mismatch ratio rho and the drift ratio r, with brackets showing which ratio each design uses Rollout engineπ_infer(θ_old) Trainer, old passπ_train(θ_old) Trainer, minibatch jπ_train(θ_j) ρkernels, precision rj − 1 steps, async lag PPO, GRPO, DAPO, GSPO, CISPO: r GLM-5 reasoning: mask on ρ Olmo 3: min(ρ, cap) ScaleRL: π_train(θ) / π_gen(θ_old), one ratio GLM-5 agentic: π_θ / π_rollout, no π_old V3.2 mask: drift measured from the engine's own π_old Brackets span the two evaluation points whose ratio the design uses.
Figure 2. One policy evaluated at three points. The factorization \(\pi^{train}_\theta/\pi^{infer}_{old}=r\cdot\rho\) is exact; designs differ in which factor they clip, mask or ignore. Placement from Olmo 3's OlmoRL objective13, GLM-5 Eq. 1 and §4.1.25, ScaleRL Eq. 214 and DeepSeek-V3.2 Eq. 98; the two-ratio framing follows Lambert9.

2. Token-level clipping as a weight and a mask

2.1 One form for every variant

Every design in this module changes the same per-token term. Write the gradient of one token as a weight, times a mask, times the usual policy-gradient term:

$$\nabla_\theta J\;=\;\frac1Z\sum_{i,t}\;w_{i,t}\;M_{i,t}\;\hat A_{i,t}\;\nabla_\theta\log\pi_\theta(o_{i,t}\mid q,o_{i,where \(Z\) is the normalizer that Module 11 compares. MiniMax-M1 writes the same decomposition: its "general formulation" multiplies a stop-gradient clipped weight by a token mask \(M_{i,t}\) and shows that PPO's trust region is the mask that is 0 when \(\hat A>0\) and \(r>1+\epsilon_{high}\), or \(\hat A<0\) and \(r<1-\epsilon_{low}\), and 1 otherwise3 (§3.1, Eq. 6-7). In PPO-style clipping the weight is \(w=r\) (the importance correction, since \(\nabla r=r\nabla\log\pi\)) and the mask is the one in Module 3's Table 3. What follows is a list of \((w,M)\) pairs: CISPO sets \(M=1\) and bounds \(w\); Kimi K2.5 and GLM-5 mask on a band whatever the sign of \(\hat A\); GSPO and DeepSeek-V3.2 compute both from a sequence statistic; SAPO replaces the pair with one smooth function.

2.2 The gradient multiplier of each variant, drawn

Figure 3 draws \(w(r)M(r)\), the factor that multiplies \(\hat A\,\nabla\log\pi\), for one token under four of the variants. For SAPO the factor is not obvious from its definition. TRL documents the gate as \(f(r)=\sigma(\tau(r-1))\cdot 4/\tau\) and the loss as \(-f(r)\hat A\)10, so by the chain rule the gradient is \(\hat A\,f'(r)\,r\,\nabla\log\pi\) with \(f'(r)=4\sigma(1-\sigma)\), giving \(w(r)=4\sigma(\tau(r-1))\bigl(1-\sigma(\tau(r-1))\bigr)\,r\) (derived). It equals 1 at \(r=1\), so SAPO reproduces the policy gradient on-policy, and it decays like \(r\,e^{-\tau(r-1)}\) for large \(r\).

Per-token gradient multiplier against the probability ratio for PPO-style clipping, CISPO, SAPO and a sign-blind band, for positive and negative advantage A > 0: a better-than-baseline token A < 0: a worse-than-baseline token 1 2 3 0 0 1 2 3 1 2 3 0 0 1 2 3 gradient multiplier w(r) ratio r = π_θ / π_old ratio r = π_θ / π_old PPO-style clip CISPO (cap ε_max) SAPO soft gate Sign-blind band
Figure 3. Gradient multiplier \(w(r)\) on \(\hat A\,\nabla\log\pi\) for one token, by the sign of the advantage. The PPO-style clip uses \(\epsilon_{low}=0.2,\ \epsilon_{high}=0.28\)12. CISPO is \(\min(r,\epsilon_{max})\) with TRL's recommended \(\epsilon_{max}=5\), which lies off the plot310. SAPO uses TRL's defaults \(\tau_{pos}=1.0,\ \tau_{neg}=1.05\)10. The band mask keeps \(w=r\) inside \([0.8,1.28]\) and zero outside for both signs; its width is illustrative, because Kimi K2.5 and GLM-5 do not state theirs in the text searched45. The four bars under each panel mark where each variant's gradient is non-zero. Derived; each curve was checked against autograd on the loss as TRL's trainer writes it16 (13 values of \(r\), both signs, four variants, no mismatch above \(10^{-9}\)).

Three things stand out. First, only CISPO and SAPO give every token a non-zero gradient at every ratio; the PPO-style clip and the band mask both switch tokens off. Second, for \(\hat A<0\) the PPO-style multiplier is unbounded: a bad token whose probability has doubled is pushed down twice as hard, and one that has grown tenfold ten times as hard. That is the case TRL's optional delta handles, a second, upper bound on the ratio for negative advantages that it attributes to the INTELLECT-2 report10; CISPO bounds it by construction. Third, the band mask differs from PPO exactly where PPO is deliberately pessimistic. In Module 3's Table 3 the cells "good token whose probability fell below \(1-\epsilon\)" and "bad token whose probability rose above \(1+\epsilon\)" still carry gradient, because PPO's \(\min\) keeps correcting moves in the wrong direction. A sign-blind band removes those too. Kimi K2.5 says so in as many words: its mechanism "relies strictly on the log-ratio to explicitly bound off-policy drift, regardless of the sign of the advantages"4 (§4.4.2).

2.3 How much the constants matter

Module 3 (Table 4) gave the headroom arithmetic behind Clip-Higher: with \(\epsilon=0.2\) a token at \(p=0.01\) can rise only to 0.012 before its gradient dies, and DAPO measured a mean up-clipped probability below 0.22 (§3.1). DAPO keeps \(\epsilon_{low}=0.2\) because raising the lower bound "will suppress the probability of these tokens to 0, resulting in the collapse of the sampling space"2 (§3.1). Two later reports measured how much the upper bound moves results. ScaleRL swept \(\epsilon_{max}\) in a DAPO-style loss and found that it "directly alters the scaling behavior": as it grows the terminal reward \(A\) rises to an optimum and then falls, so the constant "governs the asymptotic error itself", unlike many hyperparameters, which merely shift the convergence speed14 (App. A.17.1). Its DAPO baseline used 0.26. MiniMax-M1 found that raising the upper bound was "less effective in our setup, which involved 16 rounds of off-policy updates per generation batch"3 (§3.1), and later lowered its own IS-weight bound to stabilize long-context scaling3 (§5). The pattern is that the right width depends on reuse per rollout (Table 1) and on the model, and ScaleRL reports the PPO-style clip as markedly more sensitive to its constant than either CISPO or GSPO is to theirs14 (App. A.17).

2.4 What R1 printed

R1's first RL stage lists "the GRPO clip ratio \(\varepsilon\) to 10" and adds that "the clip ratio plays a crucial role in training. A lower value can lead to the truncation of gradients for a significant number of tokens, thereby degrading the model's performance, while a higher value may cause instability during training"12 (§3.2.1). Under a PPO parameterization the interval is \([1-\varepsilon,1+\varepsilon]=[-9,11]\); a ratio is never negative, so the lower clip cannot fire, and the upper clip fires only if a ratio exceeds 11 (derived). Two readings of the equation make this more puzzling. R1 prints the GRPO objective with a sequence-level ratio \(\pi_\theta(o_i\mid q)/\pi_{\theta_{old}}(o_i\mid q)\), with no token sum and no \(1/|o_i|\), both in the main text and in the appendix12 (§2.1 Eq. 1, Supp. A.3 Eq. 11). DeepSeekMath's Eq. 3 and DeepSeek-V3.2's Eq. 5 are per-token with a \(1/|o_i|\) average118. The report does not say whether the implementation differs from DeepSeekMath's or the equation is shorthand. The two readings give different meanings to 10. Per token, it lets a token's probability grow elevenfold before its gradient stops. Per sequence, the unnormalized product of token ratios would be allowed a total log-ratio of \(\ln 11=2.40\) nats upward, against \(\ln 1.2=0.18\) nats for the conventional 0.2 (derived).

The report's author list adds a lead without resolving it: "Zhibin Gou proposed a large PPO clipping strategy to enhance GRPO performance, demonstrating its significance alongside Zhihong Shao and Junxiao Song"12 (§7, Contributions of the Core Authors). Whether that strategy is the \(\varepsilon=10\) of stage 1, and whether the ratio is per token or per sequence, is not documented; I read a clip that is effectively open, with step control carried by the learning rate of 3e-6 and the 16-minibatch structure, but that reading is the writer's. The practical lesson is narrower and sourced: among the reports read for this module, R1 alone prints a clip value far outside the conventional range, and it says why a small value is a problem.

3. CISPO: clip the weight, not the update

3.1 The objective and what it changes in the gradient

MiniMax-M1 starts from REINFORCE with an importance correction for off-policy minibatches, \(\frac1{|o_i|}\sum_t \mathrm{sg}(r_{i,t})\,\hat A_{i,t}\log\pi_\theta(o_{i,t})\), where \(\mathrm{sg}\) is stop-gradient3 (§3.1, Eq. 3). CISPO ("Clipped IS-weight Policy Optimization") "abandons the trust region constraint and instead clips the importance sampling weights"3 (§1); it keeps that form and clips the weight instead of the objective:

$$J_{CISPO}=\mathbb E\Big[\tfrac{1}{\sum_i|o_i|}\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}\mathrm{sg}\big(\hat r_{i,t}\big)\,\hat A_{i,t}\,\log\pi_\theta(o_{i,t}\mid q,o_{i,(Eq. 4-5). The advantage is GRPO's group-relative one, the loss is token-level, and the run also uses DAPO's dynamic sampling and length penalty and no KL term3 (§3.1). In the experiments "we did not impose a lower bound on the IS weight by setting \(\epsilon^{IS}_{low}\) to a large value; instead, we only tuned \(\epsilon^{IS}_{high}\)", and the authors concede that "the gradient of Eq. 4 is slightly biased due to weight clipping" while preserving "gradient contributions from all tokens, especially in long responses"3 (§3.1). ScaleRL writes the same loss as truncated importance sampling plus vanilla policy gradient, \(\mathrm{sg}(\min(\rho_{i,t},\epsilon_{max}))\,\hat A_i\log\pi\)14 (Eq. 4), and TRL's loss_type="cispo" is that expression with epsilon_high as the cap16; the docs recommend 5.010. Lambert's table classifies it as an importance-sampling granularity of "Token" with "Weights (stop-grad)" as the clipping style, against "Objective (bilateral)" for PPO and GRPO9 (§6.2.11, Table 3).

The gradient is the PPO-style gradient of Section 2.1 with two changes. The mask is always 1, and the weight is \(\min(r,\epsilon_{max})\) for both signs of the advantage. The bias is easy to state (derived): the exact importance-corrected gradient is \(\mathbb E_{old}[\,r\,\hat A\nabla\log\pi\,]\), the CISPO gradient is \(\mathbb E_{old}[\,\min(r,c)\,\hat A\nabla\log\pi\,]\), and the difference is \(\mathbb E_{old}[(r-c)^+\hat A\nabla\log\pi]\). It is nonzero only for tokens whose probability has grown by more than the cap since sampling, a factor of 5 at TRL's recommended \(\epsilon_{max}\). With no weight above the cap, CISPO is the unbiased corrected gradient; M1 notes the same limit: without weight clipping the objective "reduces to the standard policy gradient objective"3 (§3.1).

3.2 Fork tokens: the case it was built for

M1's diagnosis came from zero-RL runs on its hybrid-attention model. GRPO did not elicit long chain-of-thought behavior, and controlled ablations traced the cause to the clip: "tokens associated with reflective behaviors (e.g., However, Recheck, Wait, Aha), which often serve as 'forks' in reasoning paths, were typically rare and assigned low probabilities by our base model", so after the first update they "exhibit high \(r_{i,t}\) values" and "were clipped out after the first on-policy update, preventing them from contributing to subsequent off-policy gradient updates"3 (§3.1). The mechanism is the probability-headroom arithmetic of Module 3 (Table 4) seen from the weight's side. Figure 4 draws it for a common token and a rare one.

Gradient weight of a positive-advantage token against its current probability, for a common token and a rare fork token, under PPO-style clipping and CISPO Common token: p_old = 0.9 Fork token: p_old = 0.02 1 2 3 4 5 0 0.9 0.95 1 1 2 3 4 5 0 0.02 0.04 0.06 0.08 0.1 0.12 r ≤ 1.11: the 1.28 clip never fires p = 0.0256: PPO-style gradient dies weight capped at 5 (p = 0.10) gradient multiplier w(r), A > 0 current probability π_θ of the token current probability π_θ of the token PPO-style clip, ε_high = 0.28 CISPO, ε_max = 5
Figure 4. Gradient multiplier of a positive-advantage token against its current probability, for a common token (\(p_{old}=0.9\)) and a rare fork token (\(p_{old}=0.02\)); both probabilities are illustrative. A ratio of 1.28 is reached at \(p=0.0256\) for the rare token, after which the PPO-style clip (\(\epsilon_{high}=0.28\)) gives it no gradient; the common token cannot reach 1.28 at all, since \(r\le1/0.9=1.11\) (derived from the clip definition12). CISPO keeps pushing the rare token with a weight that grows to the cap \(\epsilon_{max}=5\) at \(p=0.10\)310. The fork-token diagnosis is MiniMax-M1's3 (§3.1).

M1 adds that these low-probability tokens "are often crucial for stabilizing entropy" and for "scalable RL", and that increasing the upper bound, DAPO's remedy, was less effective with 16 off-policy rounds. Module 13 takes up the entropy side. The cost of keeping their gradient is, on the writer's reading, variance: a rare token pushed with weight up to 5 contributes five times the usual gradient, which is why the cap exists and why ScaleRL sweeps \(\epsilon_{max}\).

3.3 What the comparisons show

M1's own study trains Qwen2.5-32B base on DAPO's math data and scores AIME 2024: CISPO beats both GRPO and DAPO at equal steps and "achieves comparable performance to DAPO using 50% of the training steps", the "2x speedup" that its introduction and Figure 2 report3 (§1, §3.1). It is one model and one benchmark. ScaleRL provides the independent comparison, on an 8B dense model with the Polaris-53K math set and a 16,384-token sequence budget, fitting a sigmoid to pass rate against GPU-hours14 (§2, §3.2).

Table 2. ScaleRL's loss-type comparison: 8B dense model, Polaris-53K, pass rate as mean@16 on 1,000 held-out prompts. \(A\) is the fitted asymptotic pass rate and \(B\) the sigmoid's steepness parameter, as printed in Figure 5a14; robustness notes from its Appendix A.1714. The ranking holds for this one setup.
LossAsymptote \(A\)\(B\)Sensitivity to the clip constantStability note
CISPO0.5952.18no significant difference for \(\epsilon_{max}\in\{4,5,8\}\)"best balance of stability and robustness"; ran to about 7k GPU-hours
GSPO0.5901.80the \(10^{-4}\) scale from its paper "did not work well"; \(10^{-3}\) scale did, upper 5e-3 slightly better than 4e-3"diverged mid-training" on multiple occasions (Section 4.5)
DAPO loss0.5204.55"highly sensitive" to \(\epsilon_{max}\); the constant moves the asymptote\(\epsilon_{max}=0.26\) ran to about 5k GPU-hours

Two caveats from the same paper keep the table honest. Inside the full ScaleRL recipe, reverting to the DAPO loss "yields similar asymptotic performance to CISPO", and the authors prefer CISPO because it is "markedly more robust" to \(\epsilon_{max}\) and more efficient in the leave-one-out run (\(B=2.01\) against 1.77); they add that "a carefully tuned DAPO variant can perform similar asymptotically"14 (§4). So the evidence supports "CISPO and GSPO beat an untuned token-level clip, and CISPO is the least fussy", not "token-level clipping cannot reach the same asymptote". The second caveat is that the gap in Table 2 is measured with ScaleRL's other choices already fixed (prompt-level averaging, FP32 logits, zero-variance filtering), the ones Module 11 and Module 15 weigh.

4. GSPO: the sequence is the unit

4.1 The argument, and what is and is not proven

The Qwen team's diagnosis is that GRPO's instability "stems from the fundamental misapplication and invalidation of importance sampling weights"7 (§1). Importance sampling estimates \(\mathbb E_{\pi_{tar}}[f]\) by reweighting samples from \(\pi_{beh}\), and the correction "relies on averaging over multiple samples (\(N\gg1\))". GRPO's token weight \(\pi_\theta(y_{i,t}\mid\cdot)/\pi_{\theta_{old}}(y_{i,t}\mid\cdot)\) is computed from one sample at each position, so it "fails to perform the intended distribution-correction role" and instead "introduces high-variance noise into the training gradients, which accumulates over long sequences and is exacerbated by the clipping mechanism"7 (§3). The authors report that the resulting collapse is often irreversible, even "when reverting to a previous checkpoint and meticulously tuning hyperparameters". Their principle is that "the unit of optimization objective should match the unit of reward", and since the reward is granted to the whole sequence the correction belongs at the sequence level7 (§3). Lambert's textbook gives the same two motivations, a practical one (per-token ratios are "numerically unstable") and a conceptual one (granularity of correction should match granularity of reward)9 (§6.2.9).

Two points sit between that argument and the loss. First, the per-token ratio is not a mistake in the TRPO derivation. TRPO's practical estimator replaces "the sum over the actions by an importance sampling estimator" at each state visited by the old policy17 (§5, Eq. 14), one sampled action per state in its single-path variant; what that surrogate approximates is the state distribution, not the action weights (Module 3, Section 2.1). TRPO's vine variant draws several actions from one state, which is the multi-sample situation the GSPO paper says importance sampling needs; GRPO-family recipes sample one next token per prefix, and Module 3 (Section 3.3) covers the vine-style exception for advantage estimation. Second, the quantity GSPO clips is not the sequence importance weight. The exact weight for a whole response is the product \(\prod_t r_t\), whose variance grows with length. GSPO takes its \(1/|y|\) power, \(s_i=\exp(\tfrac1{|y_i|}\sum_t\log r_{i,t})\), the geometric mean of the token ratios, and says why: "we adopt length normalization in \(s_i(\theta)\) to reduce the variance and to control \(s_i(\theta)\) within a unified numerical range"7 (§4.1). Because \(\mathbb E_{old}[s_if(y_i)]\neq\mathbb E_\theta[f]\) for \(|y_i|>1\) (derived), \(s_i\) is a sequence-level trust-region statistic rather than an unbiased correction. That does not make it a worse design. It does mean the theoretical argument for GSPO and the loss GSPO runs are not the same object, and the evidence for it has to be empirical.

4.2 The loss and its gradient

With \(\hat A_i\) the group-normalized advantage, GSPO maximizes \(\frac1G\sum_i\min\big(s_i\hat A_i,\ \mathrm{clip}(s_i,1-\epsilon,1+\epsilon)\hat A_i\big)\) (Eq. 5-7)7. Using \(\nabla s_i=s_i\nabla\log s_i\), the unclipped gradient is \(\frac1G\sum_i s_i\hat A_i\,\frac1{|y_i|}\sum_t\nabla\log\pi_\theta(y_{i,t}\mid\cdot)\) (Eq. 10). Every token of response \(i\) has the same gradient weight \(s_i/|y_i|\), where GRPO weights each token by its own ratio, a value that "can vary among \((0,1+\epsilon]\) (for \(\hat A_i>0\)) or \([1-\epsilon,+\infty)\) (for \(\hat A_i<0\))" (Eq. 12). I checked the weight \(s_i/|y_i|\) against autograd on a 7-token example: the difference is \(3\times10^{-17}\) in float64. Two consequences follow.

  • The \(1/|y_i|\) is still there. Tokens in longer responses receive less gradient weight under GSPO exactly as under GRPO's per-sequence average (Module 11), because GSPO keeps the mean over tokens inside the exponent. TRL warns that token-summed loss types combined with importance_sampling_level="sequence" instead weight each sequence "by its completion length", and says to use loss_type="grpo" to reproduce the paper16. TRL also ships LUSPO, described there as a length-unbiased modification of GSPO; it is a pointer only here10.
  • Advantages stay at sequence level. For multi-turn work the paper adds GSPO-token, \(s_{i,t}=\mathrm{sg}[s_i]\,\pi_\theta(y_{i,t})/\mathrm{sg}[\pi_\theta(y_{i,t})]\), which is numerically equal to \(s_i\) and gives the same gradient when all token advantages are equal, but allows per-token advantages7 (§4.3, Eq. 13-17). Module 18 returns to it.

4.3 Why the clip range is 3e-4, and what that does to a response

GSPO uses left and right ranges of 3e-4 and 4e-4, against 0.2 and 0.27 for the GRPO baseline it was compared with; the authors note that the ranges "differ in order of magnitude due to the distinct definitions of importance ratios"7 (§4.1, §5.1). The sequence statistic is a mean of \(L\) token log-ratios, so a band of \([-3\times10^{-4},\,+4\times10^{-4}]\) on \(\log s_i\) allows a total log-likelihood change of at most 0.4 nats at \(L=1{,}000\) and 3.2 nats at \(L=8{,}000\), and the corresponding sequence-likelihood ratio can reach \(e^{0.4}=1.49\) or \(e^{3.2}=24.5\) (derived). The band is a budget in total nats that grows with length. Table 3 and Figure 5 run the arithmetic for a 1,000-token response with a positive advantage.

Table 3. One response of \(L=1{,}000\) tokens with \(\hat A>0\): how many tokens still receive gradient. GRPO uses \(\epsilon_{high}=0.27\) and GSPO the upper range 4e-4, both as in the GSPO paper's comparison7 (§5.1); CISPO uses the cap \(\epsilon_{max}=5\)10. \(s=\exp(m\ln r/L)\) for \(m\) tokens at ratio \(r\) and the rest at 1. Derived.
CaseGSPO \(s\)GRPO tokens with gradientGSPO tokens with gradientCISPO tokens with gradient
1 token at \(r=1.4\)1.0003369991,0001,000
1 token at \(r=1.5\)1.00040699901,000
5 tokens at \(r=2\)1.00347299501,000
50 tokens at \(r=1.1\)1.0047771,00001,000
all 1,000 tokens at \(r=1.0005\)1.0005001,00001,000
One 1,000-token response with five tokens at ratio 2: which tokens receive gradient under GRPO, GSPO and CISPO One response, 1,000 tokens, A > 0: five tokens have r = 2, the rest r = 1 ratio r_t token 11,000 r = 2 at five tokensr = 1 elsewhere GRPOtoken clip, 0.27 995 / 1,000 GSPOs = 1.00347 > 1.0004 whole response clipped 0 / 1,000 CISPOweight min(r, 5) 1,000 / 1,000 five tokens at weight 2 receives gradient clipped token clipped response
Figure 5. Row 3 of Table 3 as a picture. Token-level clipping removes the five tokens that moved; sequence-level clipping removes all 1,000 because the mean log-ratio, 5 ln 2 / 1,000 = 0.00347, exceeds 4e-4; CISPO removes none and gives the five tokens twice the usual weight. Positions of the five tokens are arbitrary. Derived from GSPO Eq. 5-77, the GRPO clip1 and CISPO Eq. 4-53.

Token ratios against the sequence ratio

Sequence ratio s
GSPO: tokens with gradient
GRPO (0.27): tokens with gradient
Single-token ratio that trips GSPO

\(s=\exp(m\ln r/L)\) with \(A>0\). GSPO's clip fires when \(s>1+h\cdot10^{-4}\); a single token trips it once \(r>e^{hL\cdot10^{-4}}\), so the same token ratio that GRPO clips (about 1.27) leaves a 1,000-token response intact under GSPO until \(r\) reaches 1.49, and trips a 100-token response at 1.04. Defaults are the paper's 4e-4 and the Figure 5 case.

Reading Table 3 as a trade-off: GSPO's budget is generous for one jumpy token (row 1 keeps all 1,000 tokens where GRPO drops one) and punishing for any response whose total drift crosses 0.4 nats, however it is spread (rows 2 to 5). That is the behavior the SAPO abstract objects to: "when a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence"6 (abstract). The GSPO paper measured the consequence and reads it the other way. Averaged over training, GSPO clips a fraction of 0.15 of tokens against 0.0013 for GRPO, "a difference of two orders of magnitude", and still trains more efficiently; the authors conclude that GRPO's token-level gradient estimates "are inherently noisy and inefficient for sample exploitation"7 (§5.2, Figure 2). The two readings are compatible with the same curve: dropping the sequences that drifted most can act as a stricter, better-aimed trust region, and it can also discard useful signal. The paper does not run the ablation that would separate them, such as GSPO with the band widened until the clip fraction matches GRPO's.

4.4 MoE routing: replay it, or average it away

The GSPO paper's strongest practical argument is about mixture-of-experts models. Its measurement: in the 48-layer Qwen3-30B-A3B-Base, "after each RL gradient update and for the same rollout sample, there are roughly 10% of the experts activated under the new policy \(\pi_\theta\) that are different from those under the old policy", a phenomenon the authors say becomes more prominent in deeper MoE models7 (§5.3). A token whose router picks different experts has a different output distribution for reasons unrelated to the gradient step, so its ratio \(w_{i,t}\) "fluctuate[s] drastically", and GRPO's token-level ratios stop meaning "this token's probability moved". The Qwen team's earlier fix was Routing Replay: cache the experts activated under \(\pi_{old}\) and force \(\pi_\theta\) to use them when computing the ratio. Their Figure 3 shows GRPO converging on MoE only with it; they also say it "incurs additional memory and communication overhead and can also limit the actual capacity of the MoE model"7 (§5.3). GSPO drops the cache. The stated reason is that it "focuses only on the sequence likelihood and is not sensitive to the individual token likelihood", and since the MoE "always maintains its language modeling capability, the sequence likelihood will not fluctuate drastically"; Figure 1 plots GSPO against GRPO with Routing Replay, and GSPO's training reward, AIME 2024, LiveCodeBench and CodeForces curves sit above the GRPO curves at equal compute7 (§5.1, §5.3).

Schematic of MoE routing in the ratio: GRPO with Routing Replay forces the same experts in the old and new pass, while GSPO lets the router move and averages the token log-ratios over the sequence GRPO with Routing Replay GSPO, router left free t1 t2 t3 E1 E2 E3 E4 routes replayed from the old pass r_tsame experts cost: cache + transfer t1 t2 t3 E1 E2 E3 E4 t2 re-routes: E2 → E3 s_imean of log r_t about 10% of experts move
Figure 6. Schematic of where routing enters the ratio. Left: Routing Replay caches the experts chosen by \(\pi_{old}\) and forces them in the \(\pi_\theta\) pass, so each token ratio compares the same experts. Right: GSPO lets the router move; one token's ratio jumps, and the sequence ratio averages the \(L\) log-ratios. The only measured quantity is the figure of roughly 10% of activated experts differing after one update in Qwen3-30B-A3B-Base7 (§5.3); the token and expert counts drawn are illustrative.

The other lab with a published MoE recipe chose the other branch. DeepSeek-V3.2 keeps token-level GRPO and adds Keep Routing: "we preserve the expert routing paths used during sampling in the inference framework and enforce the same routing paths during training", an operation "found crucial for RL training stability of MoE models" and "adopted in our RL training pipeline since DeepSeek-V3-0324"8 (§3.1). Neither report runs the other's method, so the choice is a design bet and not a measured comparison. The mechanisms do differ in what they pay: replay needs the inference engine to export routing decisions and the trainer to consume them, a systems cost (Module 14 and System Design Module 37), while GSPO pays in the band arithmetic of Section 4.3. GSPO also claims a second benefit, that sequence likelihoods tolerate training-versus-inference precision differences well enough to use the inference engine's own probabilities without recomputing them in the trainer7 (§5.4). The paper states this as "intuitively" true and reports no experiment, so it belongs with the numerics discussion of Module 14, not with the evidence here.

4.5 Independent evidence, and where it disagrees

Outside the Qwen team, the main measurement is ScaleRL's (Table 2). Its GSPO asymptote, 0.590, is within 0.005 of CISPO's and 0.070 above the DAPO loss on the same 8B dense model, so the sequence-level ratio does help relative to an untuned token-level clip14 (Fig. 5a). The same appendix reports problems that the GSPO paper does not. The paper's \(10^{-4}\) clip scale "did not work well in our setting", and a \(10^{-3}\) scale was needed. Past that, "GSPO runs diverged mid-training" on multiple occasions; for 8B models restarting from a stable checkpoint recovered, but on the 17B×16 MoE "instability persisted despite repeated resetting to a stable checkpoint", with no implementation bug found14 (App. A.17.3-A.17.4). That is the opposite of GSPO's headline claim of stability, from a group that originated neither method, in one lab's training framework. Two other notes belong here. The GSPO paper says it contributed to "the latest Qwen3 models"7; the Qwen3 technical report on arXiv (v1, May 2025) predates GSPO and does not mention it (text search), so the Qwen3 case study (Module 29) cannot be used as independent confirmation18. And Lambert's textbook notes that CISPO and GSPO come from groups "pushing the limits of applying RL on large-scale MoE models" and are "less studied and beneficial within smaller, academic experiments"9 (§6.2.10).

5. Masks and gates: Kimi, GLM-5, DeepSeek-V3.2 and SAPO

The remaining designs share one idea with Section 2.2's band mask: stop trusting a token or a sequence once it has drifted too far from what generated it. Kimi K2.5 and GLM-5 do so regardless of the sign of the advantage; DeepSeek-V3.2 and SAPO keep a sign dependence. They differ in what "drifted" is measured against, and that is the useful axis (Figure 2).

5.1 Kimi K2.5 and K3: a log-ratio band

K2.5 prints its objective as a squared-error surrogate in the lineage of K1.5's mirror-descent loss (Module 9), with a clip added:

$$L_{RL}(\theta)=\mathbb E\Big[\tfrac1N\sum_{j=1}^{K}\sum_{i=1}^{|y_j|}\Big(\mathrm{Clip}\big(\tfrac{\pi_\theta(y^i_j\mid\cdot)}{\pi_{old}(y^i_j\mid\cdot)},\alpha,\beta\big)\,\big(r(x,y_j)-\bar r(x)\big)-\tau\log\tfrac{\pi_\theta(y^i_j\mid\cdot)}{\pi_{old}(y^i_j\mid\cdot)}\Big)^{2}\Big]$$

with \(N\) the number of generated tokens in the batch and \(\alpha,\beta,\tau>0\) hyperparameters4 (§4.4.2, Eq. 1). The prose describes something slightly different from the formula: "a simple gradient masking scheme: policy gradients are computed normally for tokens with log-ratios within the interval \([\alpha,\beta]\), while gradients for tokens falling outside this range are zeroed out", introduced "to mitigate the off-policy divergence amplified by discrepancies between training and inference frameworks"4. The printed clip acts on the ratio and the prose on the log-ratio. I follow the prose as the operational definition. Read literally, though, the printed squared form does not zero every gradient outside the band: there the clipped ratio is a constant, so the gradient through it vanishes, but the \(-\tau\log\)-ratio term inside the square still carries a gradient, and the prose's "zeroed out" is the stronger statement (derived reading, not checked against an implementation; Module 28, Section 4.1 derives it). The values of \(\alpha\), \(\beta\) and \(\tau\) are not given in the text searched, and the report offers no ablation beyond saying the mechanism is "essential for maintaining training stability in complex domains requiring long-horizon, multi-step tool-use reasoning". Kimi K3 states that its policy optimization "follows the algorithm in Kimi K2.5" and adds partial rollouts, which make a trajectory span several policy versions; staleness is handled "through a per-token regularization" that "constrains policy updates within a localized neighborhood"15 (§4.1.2). That is a claim of tolerance, with no staleness measurement or comparison in the text searched.

5.2 GLM-5: two stages, two answers

GLM-5 uses different ratio machinery for its two RL regimes. For reasoning RL it runs GRPO with IcePop and no KL term "to accelerate RL improvement". The loss multiplies each token's usual clipped term by \(\mathrm{pop}(\rho_{i,t},1/\beta,\beta)\), where \(\rho_{i,t}=\pi^{train}_{\theta_{old}}/\pi^{infer}_{\theta_{old}}\) is the mismatch ratio of Section 1.2 and \(\mathrm{pop}(\rho,1/\beta,\beta)\) equals \(\rho\) when \(1/\beta\le\rho\le\beta\) and 0 otherwise; the hyperparameters are \(\beta=2\), \(\epsilon_{low}=0.2\), \(\epsilon_{high}=0.28\)5 (§3.2, Eq. 1). Olmo 3's truncated importance weight with a cap of 2.0 is the one-sided cousin: IcePop zeroes tokens whose mismatch is beyond a factor of 2 in either direction, where truncation only caps the upper side13. The report describes this stage as "entirely on-policy"5 (§3.2); if that means one update per rollout, \(r=1\) in value and the PPO clip is inert, and the mismatch mask is the only active ratio term (the writer's inference from Table 1).

For asynchronous agentic RL the problem is different. A rollout engine "may undergo multiple updates during a single trajectory generation", so exact \(\pi_{old}\) would require "an extensive history of model checkpoints". GLM-5 therefore drops \(\pi_{old}\) and uses the log-probability the engine returned at generation time as the denominator:

$$L(\theta)=\mathbb E_t\big[f(r_t;\epsilon_\ell,\epsilon_h)\,\hat A_t\log\pi_\theta(a_t\mid s_t)\big],\quad r_t=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{rollout}(a_t\mid s_t)},\quad f(x)=\begin{cases}x&1-\epsilon_\ell(Eq. 3-5), with tokens outside the band "entirely masked from gradient computation"5 (§4.1.2). The report states the trade directly: reusing rollout log-probabilities "accepts a controlled degree of off-policy bias to circumvent the need for historical policy tracking". It also drops whole samples whose oldest policy version lags the current one by more than a threshold \(\tau\), and the values of \(\epsilon_\ell\), \(\epsilon_h\) and \(\tau\) are not stated in the text searched. Read as a constant weight (the equation does not write a stop-gradient, so this is the writer's reading), the gradient inside the band is \(r\,\hat A\nabla\log\pi\) and zero outside: MiniMax-M1's unified form of Section 3.1 with a two-sided mask where CISPO has a one-sided cap. What it buys over PPO-style clipping is the denominator: \(r\) here measures distance from the policy that actually produced each token, however many versions that crossed.

5.3 DeepSeek-V3.2: mask the negatives that drifted

V3.2 keeps token-level GRPO with its clip and adds a binary mask on whole sequences (Eq. 8-9):

$$M_{i,t}=\begin{cases}0&\hat A_{i,t}<0\ \text{and}\ \dfrac1{|o_i|}\sum_{t'=1}^{|o_i|}\log\dfrac{\pi_{old}(o_{i,t'}\mid q,o_{i,\delta\\[2mm]1&\text{otherwise,}\end{cases}$$

where \(\pi_{old}\) "denotes the sampling probability directly returned by the inference framework, thus the KL divergence between the old and current policy accounts for both sources of off-policyness", drift and mismatch8 (§3.1). The report masks only negative advantages because "the model benefits the most by learning from its own mistakes, whereas highly off-policy negative samples can be detrimental", and says only that the mask "improves stability in certain training scenarios that would otherwise exhibit instability"; the value of \(\delta\) and any ablation are not given. In GSPO's notation the left side of the condition is \(-\log s_i\), so the mask is "\(\hat A_i<0\) and \(s_ioff_policy_mask_threshold, off by default, keeping a sequence if its advantage is non-negative or its mean log-ratio is at most \(\delta\); the docs' example value of 0.5 is TRL's, not DeepSeek's1016.

5.4 SAPO, and the rest of TRL's menu

SAPO replaces the hard band with the smooth gate of Section 2.2, \(f(r)=\sigma(\tau(r-1))\cdot4/\tau\) with \(\tau_{pos}=1.0\) and \(\tau_{neg}=1.05\) by default, "penalized more strictly for 'bad' actions to prevent instability, while being more permissive with 'good' actions"10. The abstract positions it against both predecessors: relative to GRPO it swaps hard token clipping for "smooth, temperature-controlled scaling", and relative to GSPO it "selectively down-weights only the offending tokens" instead of suppressing all gradients for a sequence, "improving sample efficiency"; the authors report better stability and Pass@1 on math benchmarks and use it to train the Qwen3-VL series6. I read only the abstract and TRL's documentation and code, so the abstract's claim that SAPO is "sequence-coherent" (TRL's formula is per token) and every number behind the efficiency claim are unverified here. TRL's current loss menu is a good index of how fast this area moves: grpo, dr_grpo, dapo (default), bnpo, cispo, sapo, luspo and vespo, plus the INTELLECT-2 delta bound and the V3.2 mask10.

6. Eight designs side by side

Table 4 puts the designs of Sections 2 to 5 in the same columns; Table 5 lists the constants each report ships.

Table 4. What each design clips or masks and what its source says about MoE routing and stale rollouts. "Claim" means the authors assert tolerance without a controlled comparison in the text searched; "not addressed" means the report does not discuss it. Sign-dependent means the rule differs for \(\hat A>0\) and \(\hat A<0\).
DesignRatio unitClipped or maskedSign-dependentGradient outside the bandMoE routing noiseStale rolloutsSource
PPO / GRPO clip (0.2)tokenupdate, one sideyeszeroneeds Routing Replay (measured by Qwen)bounds favourable drift only17
Clip-Higher (0.2 / 0.28)tokenupdate, wider upsideyeszeronot addressed"less effective" at 16 rounds; \(\epsilon_{max}\) moves the asymptote2314
GSPOsequence, length-normalizedwhole response, band 3e-4 / 4e-4yeszero for all tokens of the responseclaim: no replay needed; Figures 1 and 3claim: precision-tolerant (no experiment); diverged in ScaleRL714
CISPOtokenweight capped, stop-gradientnokept, weight at capnot addressed16 off-policy rounds; used with PipelineRL-8314
Kimi K2.5 masktokenlog-ratio outside \([\alpha,\beta]\)nozero, both sidesnot addressedclaim: essential in long tool-use runs; K3 claims extreme staleness is tolerated415
GLM-5 double-sided IStoken, vs \(\pi_{rollout}\)ratio outside \((1-\epsilon_\ell,1+\epsilon_h)\); stale samples droppednozeronot addressed in this stageasynchronous by design; bias accepted5
V3.2 sequence masksequence, engine \(\pi_{old}\)negative-advantage responses with drift above \(\delta\)yes (negatives only)zero for that responsepaired with Keep Routingclaim: "improves stability in certain training scenarios"8
SAPOtokennothing masked; smooth gateyes (temperature)decays smoothlyabstract: variance "exacerbated" in MoEabstract: "attenuates off-policy updates"610

6.1 Values that ship

Table 5. Ratio-related settings as reported. Where a report gives no value in the text searched, the cell says so. TRL values are the main branch's GRPOConfig defaults and recommendations, which move.
RecipeLower boundUpper bound or capOther ratio-related settings
GSPO paper, Qwen3-30B-A3B cold start73e-4 (sequence)4e-4 (sequence)GRPO baseline 0.2 / 0.27 with Routing Replay; 4 minibatches
DAPO20.20.2816 updates per rollout; no KL
Olmo 3 Think and Instruct130.20.272truncated-IS cap 2.0 on 32B Think and RL-Zero; max asynchrony 1 to 8
GLM-5 reasoning RL50.20.28IcePop \(\beta=2\) on the mismatch ratio; no KL
MiniMax-M1 CISPO3none (set large)\(\epsilon^{IS}_{high}\) tuned, value not stated16 off-policy rounds; lowered during length scaling
ScaleRL14CISPO: noneCISPO \(\epsilon_{max}\in\{4,5,8\}\) tried; DAPO baseline 0.26GSPO needed a \(10^{-3}\) scale, upper 5e-3; PipelineRL-8
R1, first RL stage12\(\varepsilon=10\), printed once; ratio printed at sequence level16 minibatches; KL 0.001
TRL defaults100.2same as lower; DAPO suggests 0.28; CISPO suggests 5.0token level; delta off; SAPO \(\tau\) 1.0 / 1.05; vLLM mismatch weight on: sequence-level product, zeroed above 3.0; V3.2 mask off

6.2 The variants as code

The listing writes the five per-token or per-sequence losses and the V3.2 keep-mask from the equations above and TRL's trainer. I checked the gradient of each loss against \(w(r)\,M\,\hat A\) (and \(s_i/|y_i|\) for GSPO) on a random batch of 4 responses with padding in float64; the maximum difference is \(2\times10^{-16}\).

import torch

def variant_loss(kind, logp, logp_old, adv, mask, eps_lo=0.2, eps_hi=0.28, eps_max=5.0,
                 band=(0.8, 1.28), tau_pos=1.0, tau_neg=1.05, s_lo=3e-4, s_hi=4e-4):
    """logp, logp_old, mask: (B, T); adv: (B, 1). Per-token loss (B, T), to be summed."""
    log_r = logp - logp_old
    r = log_r.exp()                                    # token ratio
    if kind == "ppo":                                  # clip the update: PPO, GRPO, DAPO
        return -torch.minimum(r * adv, r.clamp(1 - eps_lo, 1 + eps_hi) * adv)
    if kind == "cispo":                                # clip the weight, stop-gradient, no floor
        return -r.clamp(max=eps_max).detach() * adv * logp
    if kind == "band":                                 # sign-blind mask: K2.5, GLM-5 agentic
        keep = ((r >= band[0]) & (r <= band[1])).detach()
        return -(keep * r * adv)
    if kind == "sapo":                                 # soft gate, TRL's formulation
        tau = torch.where(adv > 0, torch.full_like(adv, tau_pos), torch.full_like(adv, tau_neg))
        return -(torch.sigmoid(tau * (r - 1)) * 4 / tau) * adv
    if kind == "gspo":                                 # one length-normalised ratio per response
        n = mask.sum(-1, keepdim=True)
        s = ((log_r * mask).sum(-1, keepdim=True) / n).exp()            # (B, 1)
        per_seq = -torch.minimum(s * adv, s.clamp(1 - s_lo, 1 + s_hi) * adv)
        return per_seq.expand_as(logp) / n             # each token gets weight s / |y|
    raise ValueError(kind)

def v32_keep(adv, logp, logp_sampler, mask, delta):
    """DeepSeek-V3.2 off-policy sequence mask: True = keep. logp_sampler: engine's old log-probs."""
    drift = ((logp_sampler - logp.detach()) * mask).sum(-1, keepdim=True) / mask.sum(-1, keepdim=True)
    return (adv >= 0) | (drift <= delta)

Where experts disagree

Everyone in this module agrees that reuse, staleness and engine mismatch make the ratio do real work. They disagree about what to do with it: keep the token ratio and tune the band, change its unit, stop clipping the update, or add a drift test. The evidence is thin and uneven. Each lab reports its own method, and the one independent comparison covers a single 8B dense model.

A. Keep the token ratio; tune the band and fix the source of drift

DAPO, Olmo 3, GLM-5 reasoning RL and DeepSeek-V3.221358; DeepSeek-V4's specialist RL keeps GRPO with hyperparameters "closely aligned" with its earlier reports19 (§5.1.1)

The per-token ratio is TRPO's own estimator, so treat engine mismatch and MoE routing as separate defects and fix them where they arise: Clip-Higher for exploration, truncated or masked importance weights for \(\rho\), Keep Routing and Keep Sampling Mask for the MoE and top-p gaps.

  • Smallest change from the PPO lineage; per-token credit and the familiar clip fraction survive.
  • With one update per rollout and matched engines, as in DeepSeekMath and GLM-5's reasoning stage, the clip is inert and nothing here matters.
  • ScaleRL finds the upper bound moves the asymptote itself, so the constant must be re-tuned per setup14.
  • MiniMax found Clip-Higher less effective at 16 off-policy rounds3; MoE needs replay, with its systems cost7.

B. Make the sequence the unit of the ratio

Qwen team (GSPO)7; Lambert's framing that correction granularity should match reward granularity9; SAPO as a softened descendant6

One sample per token position cannot correct a distribution, and routing noise makes token ratios meaningless; a length-normalized sequence ratio is stable and removes the need for Routing Replay.

  • Qwen's MoE runs converge without replay, and ScaleRL's asymptote for GSPO (0.590) is far above its DAPO loss (0.520)14.
  • One scalar per response is easy to log and to ship from the inference engine.
  • The geometric-mean ratio is not an unbiased importance weight, and a single token above 1.49 trips the band at \(L=1{,}000\) (Table 3).
  • Independent runs diverged and the clip scale did not transfer from \(10^{-4}\) to \(10^{-3}\)14; the \(1/|y|\) length weight remains.

C. Clip the importance weight and keep every token's gradient

MiniMax (CISPO)3; adopted in ScaleRL14 and recommended with \(\epsilon_{max}=5\) in TRL10

Fork tokens such as "Wait" carry the exploration that reasoning RL needs, and a hard clip removes them after one update; cap the weight instead and accept a small bias.

  • Best asymptote in ScaleRL (0.595) and no significant change across \(\epsilon_{max}\) in {4, 5, 8}; 2x over DAPO in M1's own study.
  • Nothing is dropped, so the clip fraction cannot hide discarded data.
  • Biased gradient by the authors' own account, and rare tokens get up to 5x weight.
  • Inside ScaleRL's full recipe a carefully tuned DAPO loss reaches a similar asymptote14.

D. Test drift directly, with a mask or a gate

Kimi K2.5 and K3415; GLM-5 agentic RL5; DeepSeek-V3.2's sequence mask8; SAPO6

In asynchronous, tool-using, mismatch-prone runs the ratio measures how far a token is from the policy that generated it, not how much the policy learned; bound that distance, against the logged rollout probability where possible.

  • Handles staleness without checkpoint history (GLM-5) and targets the negative-advantage tail where large weights are most harmful (V3.2).
  • Matches the production reality that \(\rho\) and version lag are measured quantities.
  • No public ablations or values for \(\alpha,\beta,\delta,\epsilon_\ell,\epsilon_h\) in the text searched; stability claims rest on the authors' word.
  • The sign-blind masks (K2.5, GLM-5) discard the corrective gradients that PPO's \(\min\) keeps (Section 2.2).
Assessment

Where reuse is one update per rollout and the engines agree, the ratio is inert and the choice is a matter of taste. Where there are 4 to 16 minibatches, asynchrony or MoE routing, the evidence supports a ranking with different confidence at each rung. An untuned token-level clip is the weakest-supported option, but ScaleRL's own leave-one-out run shows a tuned DAPO loss closing the asymptote gap, so the sturdier finding is robustness and efficiency, not ceiling. CISPO is the best-supported default: an independent comparison, the least sensitivity to its constant, a recommendation in TRL, and failure modes that are documented and bounded (bias past the cap, rare-token variance). GSPO has the most coherent MoE story but only its authors' runs show the benefit, while the one independent study saw divergence and a clip scale that did not transfer; test it with a per-model band and a length-unbiased variant, not as a drop-in. The masks and gates are responses to measured mismatch and staleness and are best used as safety nets after \(\rho\) has been measured (Module 14). What would settle the question is a same-codebase, same-data comparison on one dense and one MoE model that logs staleness and \(\rho\) and reports asymptote and efficiency, including GSPO with its band widened until its clip fraction matches GRPO's, CISPO with and without routing replay, and a K2.5-style mask against PPO's.

What is changing

As of 2026-10-27, TRL's main branch ships eight loss types and keeps dapo with token-level ratios as the default; the menu now includes cispo, sapo, luspo and vespo alongside the older variants, so any default recorded here is likely to move10. Kimi K3 reuses K2.5's mask instead of adopting GSPO or CISPO15, and DeepSeek's V4 specialists remain GRPO19, so among the reports read for this course only Qwen (GSPO) and MiniMax (CISPO) train with a sequence-level or weight-clipped loss, and ScaleRL adopts CISPO in a research recipe. The SAPO full paper and any multi-lab comparison with logged mismatch are the two things that would most change the ranking above. The systems side of the same problem is in System Design Modules 37 and 38.

References

  1. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv:1707.06347v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the clipped objective, Eq. 7, and ε = 0.2 (§3).
  2. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. (ByteDance Seed / Tsinghua AIR). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: Clip-Higher, εlow = 0.2 and εhigh = 0.28, mean up-clipped probability below 0.2 (§3.1, Fig. 3a); 512 prompts × 16 responses, minibatch 512, 16 gradient updates per rollout (§4.1).
  3. paperMiniMax (Chen, A., Li, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: CISPO objective Eq. 3-7, the fork-token diagnosis, 16 off-policy rounds, comparison with DAPO and GRPO (§3.1, Fig. 2); lowering the IS-weight bound during length scaling (§5).
  4. paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.; Moonshot AI). Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276v2 (7 Aug 2026; v1 2 Feb 2026). https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: the policy-optimization objective Eq. 1 and the log-ratio gradient mask, sign-blind by design (§4.4.2, p. 8).
  5. paperGLM-5 Team (Zeng, A., Lv, X. et al.; Zhipu AI / Tsinghua). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2 (24 Feb 2026; v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: reasoning-RL loss with IcePop, β = 2, ε 0.2 / 0.28, on-policy, no KL (§3.2, Eq. 1); direct double-sided importance sampling, Eq. 3-5, and version-lag dropping (§4.1.2).
  6. paperGao, C., Zheng, C., Chen, X.-H., Dang, K., Liu, S., Yu, B., Yang, A., Bai, S., Zhou, J., Lin, J. (Qwen Team). Soft Adaptive Policy Optimization. arXiv:2511.20347 (v1 25 Nov 2025; v2 1 Dec 2025). https://arxiv.org/abs/2511.20347. Accessed 2026-10-27. Used for: abstract only (read through the arXiv API): the soft gate, the comparison with GSPO and GRPO, the Qwen3-VL use. Source added beyond the researcher's map.
  7. paperZheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B. et al. (Qwen Team, Alibaba Inc.). Group Sequence Policy Optimization. arXiv:2507.18071v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: Eq. 5-7 and gradients Eq. 10-12 (§4); clip ranges 3e-4 / 4e-4 against 0.2 / 0.27 and four minibatches (§5.1); clip fractions 0.15 and 0.0013 (§5.2, Fig. 2); roughly 10% of experts changing per update, Routing Replay (§5.3, Fig. 3); inference-engine likelihoods (§5.4).
  8. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: per-token GRPO objective Eq. 5-6, Off-Policy Sequence Masking Eq. 8-9, Keep Routing and Keep Sampling Mask (§3.1, pp. 7-8).
  9. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12 (11 Sep 2026); Manning print edition 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: §6.2.9 (GSPO), §6.2.10 (CISPO, TOPR), §6.2.11 Table 3 (granularity and clipping style), §6.3.4 (truncated importance sampling, policy-drift and implementation ratios).
  10. officialHugging Face TRL maintainers. TRL documentation: GRPO Trainer (main branch; v1.16.0 was the latest tagged release listed). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: loss_type options, the SAPO formula and temperatures, epsilon, epsilon_high, delta, the CISPO recommendation of 5.0, off_policy_mask_threshold, vLLM importance-sampling defaults, logged clip metrics.
  11. paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: per-token GRPO objective Eq. 3 (§4.1.1); single update per exploration stage (§4.2); dropping min and clip when πθold = πθ (App. A.1, Eq. 15-16).
  12. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv v2, the expanded Nature version). arXiv:2501.12948v2, 4 Jan 2026. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the sequence-level objective Eq. 1 (§2.1) and Eq. 11 (Supp. A.3); 16 minibatches and ε = 10 with the rationale sentence (§2.1, §3.2.1); the author-contribution sentence on a large PPO clipping strategy (§7).
  13. paperTeam Olmo (Ettinger, A., Bertsch, A. et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2 (14 Apr 2026; v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: the OlmoRL objective with truncated importance sampling and clip-higher (§4); Table 49: minibatches, max asynchrony, TIS cap, clip-lower 0.2 and clip-higher 0.272.
  14. paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R. et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: setup and ratio notation, Eq. 2-4 (§2, §3.2); PipelineRL-k (§3.1); loss-type comparison, Figure 5a (A and B values); leave-one-out discussion (§4); sweeps of εmax and GSPO clip scale, divergence of GSPO runs (App. A.6, A.17).
  15. paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.; Moonshot AI). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2 (7 Aug 2026; v1 27 Jul 2026). https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: partial rollouts, staleness and per-token regularization (§4.1.2).
  16. repoHugging Face. trl/trainer/grpo_trainer.py, main branch (201,296 bytes when read). https://raw.githubusercontent.com/huggingface/trl/main/trl/trainer/grpo_trainer.py. Accessed 2026-10-27. Used for: _compute_loss: the on-policy shortcut for old log-probs, sequence-level log-weight, CISPO, SAPO and GRPO branches, get_off_policy_mask (V3.2 mask), the GSPO warning.
  17. paperSchulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv:1502.05477v5 (20 Apr 2017). https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Used for: the importance-sampling estimator that replaces the sum over actions at each sampled state (§5, Eq. 14-16).
  18. paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: text search for GSPO or sequence-level policy optimization (none found).
  19. paperDeepSeek-AI (Xu, A., Lin, B., Xue, B. et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: specialist RL uses GRPO with hyperparameters closely aligned with prior reports (§5.1.1).