Reinforcement Learning for LLM Post-Training · Module 11

The GRPO Loss and Advantage Under the Microscope: Length Bias, Difficulty Bias, Token-Level Averaging, and Zero-Variance Groups

Two denominators decide how much gradient each token of a GRPO update receives: the response length |oi| in the loss, and the group's reward standard deviation in the advantage. This module derives the weight a token gets under every published aggregation, shows what Dr. GRPO, DAPO, DeepSeek-V3.2, Olmo 3 and ScaleRL each changed and why, and states what their ablations do and do not show. You can compute the per-token weight and the dead-group fraction for your own batch and pick a normalization with the evidence in hand.

Course 45 min read 13 sources As of 2026-10-27 Prerequisite: Module 9
One GRPO gradient: group rewards become advantages through a scale choice s, then each token of a response is weighted by 1/Z, and the two choices differ across recipes G rewardsone per sample Advantage A_i(R_i − mean) / s Token weight1 / Z, per token GradientΣ (A_i/Z) ∇log π Knob s: scale of the advantage group stdGRPO, DAPO, GLM-5, TRL default noneDr. GRPO, DeepSeek-V3.2, Olmo 3 batch stdScaleRL, TRL scale_rewards=batch Changes how much each prompt pulls, not the ranking inside it. Knob Z: denominator of the token average G |o_i|original GRPO, V3.2 as printed Σ |o| in the groupDAPO Eq. 8 Σ |o| in the batchOlmo 3, TRL loss_type=dapo G L_maxDr. GRPO, a constant Changes how much each response pulls, and which one wins. Before the lossdrop groups whose rewards are all equal (A = 0 for every token); mask or soft-penalise truncated samples (DAPO, ScaleRL, Olmo 3 differ).
Figure 1. The two normalization knobs in a GRPO-family update. Original GRPO sets s to the group standard deviation and Z to G|oi|1; Dr. GRPO removes both2; DAPO changes Z but keeps s3. Recipes named in the panels are mapped, with sources, in Table 7.

1. Where the gradient weight of a token comes from

1.1 One formula for every variant

Take one prompt \(q\), a group of \(G\) samples \(o_1,\dots,o_G\) drawn from the rollout policy, lengths \(|o_i|\), and one scalar reward \(R_i\) per sample. DeepSeekMath's GRPO objective averages a clipped per-token surrogate first over the tokens of each response and then over the group1 (§4.1.1, Eq. 3), with the advantage \(\hat A_i = (R_i-\mathrm{mean}(R))/\mathrm{std}(R)\) copied to every token of response \(i\)1 (§4.1.2). Dr. GRPO writes the same objective in its Eq. 32.

When the policy takes one gradient step per rollout the ratio is exactly 1, the clip is inactive and the surrogate's gradient is \(\hat A\,\nabla\log\pi\) (Module 9, §3.3 derives this; DeepSeekMath-RL uses a single update per exploration stage1, §4.2). Drop the KL term and every aggregation covered in this module collapses to one expression:

$$\nabla\mathcal J=\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}\frac{\hat A_i}{Z_i}\,\nabla\log\pi_\theta(o_{i,t}\mid q,o_{i,\lt t}),\qquad \hat A_i=\frac{R_i-\mathrm{mean}(R)}{s}.$$

Two choices remain: the scale \(s\) in the advantage and the denominator \(Z_i\) of the token average. They are the two knobs of Figure 1. (For a batch of \(P\) prompts, sum over all \(B=PG\) responses and read \(Z_i\) as in Table 1; for a single prompt, \(P=1\) and \(B=G\).) Every token of response \(i\) receives the same coefficient \(\hat A_i/Z_i\) in front of its score function, so "how much does this token matter" is the same question as "what is \(Z_i\)". The rest of the module asks what each choice of \(Z\) and \(s\) does to the weights, and what the experiments say.

1.2 Four denominators in use

Table 1 lists the denominators that appear in the reports and in Hugging Face TRL, which exposes most of them as loss_type values45. Lambert's textbook describes the same three families of aggregation6 (§6.3.2).

Table 1. Aggregation schemes for a batch of \(P\) prompts with \(G\) samples each. \(L_i=|o_i|\); \(B=PG\) responses; \(N\) response tokens in the batch; \(L_{\max}\) the generation cap. "Total" is the weight of the whole response, \(L_i\) times the per-token weight. For one prompt, set \(P=1\).
SchemeDenominator \(Z_i\)Per-token weightTotal for response \(i\)Where it appears
Sample-level\(B\,L_i\)\(1/(B L_i)\)\(1/B\), any lengthDeepSeekMath Eq. 31; GLM-5 Eq. 17; V3.2 as printed8; TRL grpo4
Prompt-level token mean\(P\sum_{j\in\text{group}} L_j\)\(1/(P\sum_j L_j)\)\(L_i/(P\sum_j L_j)\)DAPO Eq. 83; "prompt average" in ScaleRL9
Batch-level token mean\(N\) (all tokens in the batch)\(1/N\)\(L_i/N\)Olmo 310; TRL dapo (default)4; "token average" in ScaleRL9
Constant\(B\,L_{\max}\)\(1/(B L_{\max})\)\(L_i/(B L_{\max})\)Dr. GRPO2; TRL dr_grpo5

Three relations follow directly (derived). First, only the sample-level scheme makes the total weight of a response independent of its length; all other schemes make it proportional to \(L_i\). Second, the batch-level token mean and the constant normalizer give every token in a batch the same weight, so they differ by one scalar per batch, \(N/(B L_{\max})\), and by nothing else. Third, DAPO's Eq. 8 normalizes inside each prompt, so tokens of a prompt whose group happened to be long get a smaller weight than tokens of a prompt whose group was short: it removes the length weighting among the samples of one prompt and introduces a prompt-to-prompt weighting by \(1/\sum_j L_j\), the same kind of effect as the standard-deviation division of Section 3.

Dr. GRPO's Listing 1 comments that per-batch normalization, as in TRL and verl at the time, is also biased2; TRL's documentation summarizes the paper as saying the DAPO formulation "reduces this bias" but does "not eliminate it completely"4. By the second relation above, the residual for the batch-level mean is only that \(N\) is itself random and contains the sampled tokens (a ratio estimator), an effect that shrinks with batch size; neither source quantifies it.

Per-token and per-sequence gradient weight versus response length under sample-level, token-level and constant normalization 0.5 1 2 5 10 20 0 5,000 10,000 15,000 20,000 mean length 6,000 Weight per token response length |o| (tokens) G x weight, 1e-4 per token (log) 0 1 2 3 0 5,000 10,000 15,000 20,000 Weight per sequence response length |o| (tokens) G x total weight of one sequence sample-level 1/(G|o_i|) token-mean 1/Σ|o| constant 1/(G L_max)
Figure 2. Left: the weight one token receives, scaled by G, as a function of the length of the response it sits in. Sample-level averaging (blue) falls as 1/|o|; token-level and constant normalization are flat. Right: the total weight a whole response receives. Sample-level gives every response the same total; the other two grow linearly with length. Illustrative group: mean length 6,000 tokens (orange dot), L_max = 20,480 (DAPO’s generation cap3). Derived from the normalizers in Table 1; no measured data.

1.3 One group, worked through

Take \(G=8\) samples of one prompt, three correct (reward 1) and five wrong (reward 0), with wrong answers longer than correct ones, as is typical when wrong attempts ramble or run to the cap. The lengths below are illustrative; the shape matches Dr. GRPO's measurements, where the y-axes of its Figure 5 run from roughly 200 to 400 tokens for correct and 1.0k to 1.8k tokens for incorrect responses2 (read off the axes). The centered advantage is \(+0.625\) for a correct sample and \(-0.375\) for a wrong one (derived: mean \(3/8\)).

Table 2. Weights inside one group (P = 1, G = 8, k = 3 correct, L_max = 8,192; illustrative lengths). "Relative per-token weight" is the weight of one token of this response divided by the weight under token-level averaging, which is \(\bar L/L_i\) with \(\bar L=2{,}822.5\) tokens. Derived from Table 1.
iOutcomeTokens \(L_i\)\(A_i\), centeredRelative per-token weight, sample-levelShare of group weight, sample-levelShare of group weight, token-level or constant
1correct640+0.6254.4112.5%2.8%
2correct910+0.6253.1012.5%4.0%
3correct1,280+0.6252.2112.5%5.7%
4wrong1,150−0.3752.4512.5%5.1%
5wrong2,100−0.3751.3412.5%9.3%
6wrong3,500−0.3750.8112.5%15.5%
7wrong5,200−0.3750.5412.5%23.0%
8wrong7,800−0.3750.3612.5%34.5%
Total push on correct samples2,830​​0.2340.078
Total push on wrong samples19,750​​0.2340.328

Read the table in two directions. Per token, the first correct response (640 tokens) counts 12.2 times as much as a token of the last wrong one (7,800 tokens) under sample-level averaging (\(7{,}800/640\)); under the other schemes the two weigh the same. Per response, sample-level averaging gives all eight the same 12.5% of the group's weight, and the positive and negative sides balance exactly because \(\sum_i A_i=0\). Under token-level averaging the weights follow length, and the negative side carries \(0.3280/0.0783=4.19\) times the positive side. That ratio is not an accident: because the advantages sum to zero, it equals the ratio of mean lengths of wrong to correct responses, \(3{,}950/943=4.19\) (derived). So the choice of \(Z\) decides whether a long wrong answer is pushed down once, as one response, or in proportion to its length. MiniMax-M1 reports this effect from the other side in Section 7.

1.4 The aggregation as code

The four schemes differ in one line each. The function below mirrors Dr. GRPO's Listing 12 and the normalizers in TRL's compute_loss5 (single-device case; TRL's dapo and cispo normalize by the token count of the global accumulated batch). I checked that autograd gives the per-token weights of Table 1 for random lengths with torch 2.10.

import torch

def aggregate(per_token_loss, mask, kind, group_size=None, l_max=None):
    """per_token_loss, mask: (B, T). Rows are consecutive groups of group_size samples."""
    if kind == "sample":      # original GRPO; TRL loss_type="grpo"
        return ((per_token_loss * mask).sum(-1) / mask.sum(-1).clamp(min=1)).mean()
    if kind == "prompt":      # DAPO Eq. 8; ScaleRL "prompt average"
        P = per_token_loss.size(0) // group_size
        num = (per_token_loss * mask).reshape(P, -1).sum(-1)
        den = mask.reshape(P, -1).sum(-1).clamp(min=1)
        return (num / den).mean()
    if kind == "batch":       # Olmo 3; TRL loss_type="dapo" on one device
        return (per_token_loss * mask).sum() / mask.sum().clamp(min=1)
    if kind == "constant":    # Dr. GRPO; TRL loss_type="dr_grpo"
        return (per_token_loss * mask).sum() / (per_token_loss.size(0) * l_max)

2. Response-level length bias

2.1 GRPO ascends a different reward

Dr. GRPO's first diagnosis is that dividing by \(|o_i|\) makes the update favor short responses when the advantage is positive and long responses when it is negative: "for positive advantages [...] this bias results in greater gradient updates for shorter responses", while for negative ones "longer responses are penalized less due to their larger \(|o_i|\)"2 (§3.1). The same mechanism can be stated exactly, which also shows what the bias is a bias of.

Fix a baseline \(b\) (the group mean in the limit of a large group) and a scale \(s\). The sample-level estimator is \(\frac1G\sum_i \frac{R_i-b}{s\,|o_i|}\nabla\log\pi_\theta(o_i)\), where \(\nabla\log\pi_\theta(o_i)=\sum_t\nabla\log\pi_\theta(o_{i,t}\mid\cdot)\). For samples drawn from \(\pi_\theta\) its expectation is the policy gradient of a modified reward, treated as a fixed function of the response (derived; the identity \(\mathbb E[f(o)\nabla\log\pi_\theta(o)]=\nabla\mathbb E[f(o)]\) is Module 2's score-function step):

$$\mathbb E\Big[\tfrac{R-b}{s|o|}\,\nabla\log\pi_\theta(o)\Big]=\nabla_\theta\,\mathbb E_{o\sim\pi_\theta}\big[\tilde r(o)\big],\qquad \tilde r(o)=\frac{R(o)-b}{s\,|o|}.$$

A constant denominator (Dr. GRPO) leaves \(\tilde r=(R-b)/s\), the reward you meant. The sample-level denominator makes the reward depend on length. Compare two responses with the same outcome. If both are correct (\(R-b>0\)), \(\tilde r\) is larger for the shorter one. If both are wrong (\(R-b<0\)), \(\tilde r\) is closer to zero, that is larger, for the longer one. Inside each outcome class the objective has acquired a length preference whose sign depends on the sign of the advantage. That is the Dr. GRPO statement, restated as an objective.

The standard-deviation division is a different kind of change. Dividing by \(s_q\), a constant per prompt, multiplies every \(\tilde r\) of that prompt by the same positive number, so it cannot reorder two samples of the same prompt; it only changes how loudly the prompt speaks relative to others (Section 3). The length term does reorder samples inside a prompt.

The length normalization did not originate in GRPO. Dr. GRPO checked five open-source PPO implementations (TRL, OpenRLHF, verl, SimpleRL-Zero, Open-Reasoner-Zero) and found all of them normalize the loss by response length, "even before the publication of GRPO"; it speculates the habit comes from pretraining, where loss.mean(-1) over a fixed packed context is a harmless stabilizer2 (§3.1, Table 2).

2.2 Which way does length move? A toy with a closed form

The identity says the objective prefers short-among-correct and long-among-wrong. Whether mean length rises or falls depends on the mix, and a small model shows how (derived here; not in the source). One prompt, four response types: correct or wrong, crossed with short (\(S\) tokens) or long (\(L\) tokens). The probability of being correct is \(p\) and does not depend on length, so a correct reward has no reason to change length. The policy is a softmax over the four types, updated by gradient flow on the expected modified reward, with the baseline at its large-group limit \(b=p\). With long and short equally likely, the expected change in mean length is

$$\frac{d\,\mathbb E[\text{length}]}{dt}\;\propto\;(1-2p)\,p(1-p)\,\frac{(L-S)^2}{8\,S\,L},$$

and exactly zero when the denominator is a constant, because then the reward ignores length. The sign is the point: sample-level averaging lengthens responses on prompts the policy mostly fails (\(p<\tfrac12\)) and shortens them on prompts it mostly solves (\(p>\tfrac12\)), with no length signal in the reward at all.

Table 3. The toy of Section 2.2 evaluated for S = 500, L = 2,000 tokens, equal chance of long and short. Entries are the expected drift of mean length in the units of the gradient flow (only signs and ratios carry meaning). Derived; a tabular softmax model, not an LLM.
Pass rate \(p\)0.10.20.30.50.70.80.9
Sample-level \(1/|o_i|\)+0.0203+0.0270+0.02360−0.0236−0.0270−0.0203
Constant or token-level0000000

Real prompts mix both regimes, and real wrong answers are longer than right ones for reasons that are not an artifact (more flailing, more truncation), so the net effect on a training run depends on the pass-rate distribution. What the toy establishes is narrower: the bias has a direction that depends on accuracy, it needs no correlation between length and reward, and the token-level and constant normalizers remove it in expectation. The toy also says why the symptom appears on hard prompts and early in training: that is where \(p<\tfrac12\).

2.3 What Dr. GRPO measured, and how far it reaches

Dr. GRPO's main training comparison uses Qwen2.5-1.5B with the R1 prompt template, MATH training questions, a 0/1 Math-Verify reward, eight samples per question, a 3,000-token generation cap, learning rate \(10^{-6}\), no KL term and clip 0.2, on 8 A100 GPUs2 (§3.2; App. G, Table 6). Four observations matter here.

  • Both algorithms show response length rising with training reward, as in R1-Zero, whose report presents the rise as the model learning "to solve reasoning tasks with more thinking time"11 (Fig. 1b). GRPO keeps lengthening "even when the reward improvement slows down"; Dr. GRPO does not, and the length of incorrect evaluation responses is substantially lower2 (§3.2, Fig. 5). The authors' claim is that the rise "may also be attributed to" the bias and is "confounded" by it, not that it is only the bias.
  • On Llama-3.2-3B continued-pretrained on math, GRPO produces a "double-increase" of accuracy and length that could be misread as emerging long chain-of-thought; Dr. GRPO limits the length growth2 (§3.4, Fig. 7).
  • In the ablation (Qwen2.5-1.5B, 3K mixed questions), removing only the length normalization and full Dr. GRPO both produce shorter responses than the other two variants, and all three modified variants beat vanilla GRPO on training reward and evaluation accuracy2 (App. C, Fig. 8). This matches Section 2.1: the length term moves length; the std term does not.
  • Across three independent runs Dr. GRPO is better in both token efficiency and final accuracy, "statistically significant" by the paper's account2 (App. C, Fig. 9).

DAPO arrives at the same place from the other direction. It argues that with sample-level averaging "tokens within longer responses [...] may have a disproportionately lower contribution to the overall loss", which prevents learning from good long samples and fails to punish gibberish in bad ones, and its Figure 4 shows entropy and length growing less wildly with a token-level loss3 (§3.3). The ablation table credits the change with one point on AIME but says it "enhances training stability and makes the length increase more healthily"3 (§4.2, Table 1; Section 7 below).

3. Division by the group standard deviation

3.1 The algebra for binary rewards

With a 0/1 verifier and \(k\) correct answers out of \(G\), let \(\hat p=k/G\). The group mean is \(\hat p\) and the population standard deviation is \(\sigma=\sqrt{\hat p(1-\hat p)}\). A correct sample then has advantage \((1-\hat p)/\sigma=\sqrt{(1-\hat p)/\hat p}\) and a wrong one \(-\hat p/\sigma=-\sqrt{\hat p/(1-\hat p)}\); without the division they are \(1-\hat p\) and \(-\hat p\) (Module 9, Table 3 tabulates these at \(G=8\)). The mean absolute advantage over the group, which measures how hard the prompt pulls on the update, is

$$\overline{|A|}_{\text{centered}}=2\hat p(1-\hat p),\qquad \overline{|A|}_{\text{divided}}=\frac{2\hat p(1-\hat p)}{\sigma}=2\sqrt{\hat p(1-\hat p)}.$$

The division multiplies a prompt's pull by \(1/\sigma\). Figure 3 plots both. Without the division, a prompt with one success in 64 samples pulls with weight 0.031 against 0.500 for a balanced prompt, a ratio of 16. With it the weights are 0.248 against 1.000, a ratio of 4. In general the weight of a one-success group relative to a balanced one rises from \(1/(2\sigma)=1.51\) at \(G=8\) to 2.07 at \(G=16\) and 4.03 at \(G=64\) (derived; the single-success weight is \(G/\sqrt{G-1}\) against 2).

Mean absolute advantage per sample versus group pass rate, with and without division by the group standard deviation 0 0.25 0.5 0.75 1 0 0.25 0.5 0.75 1 group pass rate k/G mean |advantage| per sample divided by std centered only One success in the group (k = 1) G centered divided by std 8 0.2190.661 16 0.1170.484 64 0.0310.248 at k/G = 1/2 both are 0.5 / 1.0 population std; binary reward
Figure 3. The average size of an advantage on one prompt, as a function of how many of its G samples are correct. Centered rewards (blue, 2p(1−p)) vanish at the extremes; dividing by the group standard deviation (orange, 2√(p(1−p))) keeps them large. Dots mark a single success in G samples. Derived from the definitions of the advantage in DeepSeekMath and Dr. GRPO12 for binary rewards and population standard deviation; Section 3.1 gives the algebra.

3.2 Which standard deviation?

The sources are not uniform, and the choice matters at small \(G\). DeepSeekMath writes \(\mathrm{std}(\mathbf r)\) and R1 writes \(\mathrm{std}(\{r_1,\dots,r_G\})\), with neither stating population or sample convention nor any epsilon111 (§4.1.2; Eq. 3 of the R1 paper). DAPO (Eq. 9) and Dr. GRPO (Eq. 3) likewise32. ScaleRL's Eq. 5 adds an unspecified \(\epsilon\) to the denominator and does not state the convention9. I read the TRL source rather than its documentation: grpo_trainer.py computes the group statistic with nanstd, which applies Bessel's correction, and divides by std + 1e-45 (commit 63d9074, 2026-10-21). Bessel's correction multiplies \(\sigma\) by \(\sqrt{G/(G-1)}\): 1.155 at \(G=4\), 1.069 at \(G=8\), 1.033 at \(G=16\), 1.008 at \(G=64\) (derived). The closed forms in this module use the population convention; the sample convention shrinks every divided advantage by that factor and changes no ranking. The \(10^{-4}\) epsilon matters only for groups whose reward spread is of that order, such as near-ties of a continuous reward.

"Batch-level" normalization also has two meanings. ScaleRL divides by the standard deviation of all advantages in the batch, \(\hat A_i/\hat A_{\mathrm{std}}\), where the advantages are already group-centered9 (§4). TRL's scale_rewards="batch" divides the group-centered advantage by the standard deviation of the raw rewards over the whole batch5, which also contains the spread between prompts of different difficulty. Both are a single scalar per batch, so they rescale the update without changing the relative pull of any two prompts (derived), unlike the per-group division.

3.3 What it changes, what it does not, and whether to call it a bias

Dr. GRPO's complaint is that questions with low reward spread, "those that are too easy or too hard", "are given higher weights during policy updates", whereas advantage normalization is usually computed "across an entire batch"2 (§3.1). Figure 3 sharpens the statement. Relative to the centered advantage, extreme prompts are indeed amplified (by 8.1 at one success in 64); relative to a balanced prompt they still pull less (0.25 against 1.0). The division does not make the extremes dominate; it softens their suppression. It also cannot reorder samples within a prompt, so unlike the length term it leaves each prompt's own optimum unchanged (Section 2.1).

Whether softening is desirable is a judgment about the training distribution. Lambert's textbook argues that removing the division "comes at the cost of downweighting questions that were all incorrect with a few correct answers", the hardest cases, whose rare success is the most informative sample in the batch6 (pp. 73–74). The opposite worry is noise. When the reward is continuous, a group of near-ties has a tiny standard deviation, and dividing by it hands unit-variance advantages to what is essentially measurement noise (derived). Section 6 shows where this bites. And no normalization can validate a reward: after division, \(\sum_i A_i^2\) equals \(G\) (population convention) for every non-degenerate group regardless of what the reward means. Training with randomly assigned rewards still raised Qwen2.5-Math-7B's MATH-500 score by 21.4 points against 29.1 for ground-truth rewards; the authors attribute that to a clipping bias in GRPO rather than to normalization12, but it is a reminder that normalized advantages are only as informative as the correlation between reward and response.

The controlled evidence is thin and says the effect is second-order: in ScaleRL's forward ablation, no normalization, batch-level and prompt-level division reach asymptotic pass rates of 0.525, 0.530 and 0.520, "similar performance"9 (App. A.9, Fig. 14b); Section 7 reads those numbers against their noise.

4. Zero-variance groups

4.1 How many groups are dead

If every sample of a prompt gets the same reward, the centered advantage is zero for all of them, and so is every token's coefficient \(\hat A_i/Z_i\) in the gradient. DAPO states the consequence plainly: "A zero advantage results in zero policy gradients, shrinking the magnitude and increasing the noise sensitivity of the batch gradient"3 (§3.2). For a prompt with pass rate \(p\) and a binary verifier the chance of a dead group is

$$P_0(p,G)=p^{G}+(1-p)^{G}.$$

Module 2 derived this, tabulated its complement and plotted it (Table 6 and Figure 6 there); Figure 4 here shows the dead side for \(G=4,8,16,64\). The two features that matter are that increasing \(G\) empties the middle of the curve but barely touches the ends (a prompt solved 1% of the time is dead 85% of the time at \(G=16\) and 53% at \(G=64\)), and that the curve is symmetric, so prompts that are nearly always solved are as useless as prompts that are never solved.

Probability that all G samples for a prompt get the same binary reward, versus the prompt pass rate, for G = 4, 8, 16 and 64 0 0.25 0.5 0.75 1 0 0.25 0.5 0.75 1 prompt pass rate p P(zero-variance group) = p^G + (1−p)^G Dead-group probability p = 0.05p = 0.01 4 81.5%96.1% 8 66.3%92.3% 16 44.0%85.1% 64 3.8%52.6% symmetric in p and 1−p G = 8: 0.8% at p = 0.5
Figure 4. The chance that a group of G samples contains no learning signal (all correct or all wrong), for G = 4 (blue), 8 (cyan), 16 (orange) and 64 (green). A larger group empties the middle of the curve but the ends stay high: a prompt solved 1% of the time is dead 85% of the time at G = 16. Derived from p^G + (1−p)^G; Module 2 plots the complement (Figure 6 there).

A batch holds prompts of many difficulties, so the batch-level dead fraction is the average of \(P_0\) over the pass-rate distribution. For a Beta\((a,b)\) population it has a closed form, \(\mathbb E[P_0]=\big[B(a+G,b)+B(a,b+G)\big]/B(a,b)\) (derived). Table 4 evaluates it for four assumed populations. They are illustrations of shape, not measurements: a mid-heavy population, a uniform one, and two U-shaped ones in which most prompts are nearly always or nearly never solved.

Table 4. Expected fraction of dead groups, and the oversampling factor \(1/(1-P_0)\) a "drop and refill" scheme needs, for assumed Beta pass-rate populations. Derived from \(P_0\); not measured data.
Pass-rate population\(G=4\)\(G=8\)\(G=16\)\(G=64\)
Beta(2, 2), mid-heavy28.6% (1.40x)10.9% (1.12x)3.5% (1.04x)0.3% (1.00x)
Beta(1, 1), uniform40.0% (1.67x)22.2% (1.29x)11.8% (1.13x)3.1% (1.03x)
Beta(0.5, 0.5), U-shaped54.7% (2.21x)39.3% (1.65x)28.0% (1.39x)14.1% (1.16x)
Beta(0.3, 0.3), strongly U-shaped65.9% (2.93x)53.4% (2.15x)43.4% (1.77x)28.6% (1.40x)

The one measured trace in the source set drifts toward the solved end. In DAPO's run the share of prompts with avg@32 equal to 100% rises from near zero to roughly 0.55 to 0.6 by step 8,000 (read off its Figure 3b; approximate)3. With more than half the prompts dead from that one side alone, refilling needs at least \(1/(1-0.55)\approx 2.2\) to \(2.5\) times as many prompts as the batch holds (derived; a lower bound, since accuracy-0 prompts are dropped too). Olmo 3 describes DAPO's dynamic sampling as generating "three times the number of prompts used in each training batch"10 (§4.4.3), the same order.

A larger \(G\) is not a free fix. Informative prompts per rollout fall as \(G\) grows: for the U-shaped Beta(0.5, 0.5) population, 11.3, 7.6, 4.5 and 1.3 informative prompts per 100 rollouts at \(G=4,8,16,64\) (derived from Table 4). Group size buys a better baseline and rarer dead groups at the price of fewer prompts. ScaleRL's one test at fixed total batch, sweeping 8, 16, 24 and 32 generations per prompt, left the fitted scaling curves "essentially unchanged"9 (§5).

Dead groups and the cost of refilling a batch, for one pass rate

Dead groups
Informative prompts in the batch
Prompts to sample to fill it
Rollouts to sample to fill it

Formula: \(P_0=p^G+(1-p)^G\); prompts to fill \(=B/(1-P_0)\). Assumes every prompt has the same pass rate \(p\); real batches mix rates (Table 4).

4.2 What a dead group costs

Two different bills arrive. The first is generation. Olmo 3's 32B reasoner used 8 H100 nodes for training and 20 for inference; its learner waits for data 75% of the time, about five times as many GPUs serve inference as training, and in one representative run a step averaged 1,000 s of which 125 s was training10 (§4.4.3, footnote 41). A dead group is paid for entirely in that dominant phase and returns nothing. The second is statistical. Under batch-level token averaging the dead group's tokens stay in the denominator \(N\), so the gradient is the informative groups' sum divided by a count that includes the dead ones: the nominal batch of 512 prompts becomes an effective batch of roughly \(512\times(1-\text{dead fraction})\), and gradient noise grows as one over the effective count (derived). Adam, which divides by a running scale of the gradient, largely undoes the smaller magnitude that DAPO points to (derived), so what survives is the noise.

4.3 Four remedies

Table 5. Remedies for zero-variance groups in the reports. "Extra rollouts" means generation beyond the nominal batch. LOO = leave-one-out from the full ScaleRL recipe (8B dense, math); A is the fitted asymptotic pass rate with run-to-run noise of about 0.02.
RemedyMechanismExtra rolloutsReported effectSource
Dynamic samplingOversample, drop groups with accuracy 0 or 1, repeat until the buffer holds N informative groupsYes, about 2 to 3 times the prompts late in training (Table 4)Last step of the ladder, 42 to 50 on AIME24 avg@32; reaches the same score in fewer steps and, the authors say, not much more wall-clock timeDAPO §3.2, Alg. 1, Fig. 63
Zero-gradient filtering with active samplingFilter dead groups; the asynchronous actors keep pulling completions and prompts until the batch is full of non-zero-advantage samplesYes, but continuous batching leaves no idle slotsKeeps the batch size from decaying; "greatly reduced loss variance" in an RL-Zero math runOlmo 3 §4.4.1, §4.4.3, Fig. 2610
Zero-variance filteringDrop dead groups from the loss; the batch simply shrinksNoHigher asymptote in the forward test; LOO without it: A 0.595 against 0.610, B 1.69 against 2.01ScaleRL §3.2, Fig. 6a, Fig. 79
No-Positive-ResamplingRemove a prompt for good once its historical pass rate reaches 0.9Fewer: dead groups are never generatedHigher asymptote in the forward test; LOO without it: A 0.605, B 1.62, the lowest B of the nine variantsScaleRL §3.2, Fig. 6b, Fig. 79

The two filters are different in kind. Dynamic sampling and active sampling keep the number of informative prompts constant and pay in generation. Zero-variance filtering accepts a smaller effective batch. ScaleRL notes the distinction (filtering "merely drop[s] the prompts" while DAPO "resamples more prompts until the batch is full") and its DAPO baseline filters rather than refills, because refilling stalled generators in its pre-assigned PipelineRL layout9 (§3.2, App. A.16). Hugging Face TRL's GRPOTrainer implements none of these; it logs frac_reward_zero_std, the fraction of samples whose group has zero reward spread, and leaves the choice to the user45. A zero-variance filter keyed on the total reward fires only when the whole reward is constant, which Section 6 shows is a weaker condition than "accuracy is constant".

5. Truncated samples: overlong filtering and soft punishment

A generation cap \(L_{\max}\) makes an unfinished chain of thought score as wrong; DAPO's default assigns truncated samples a punitive reward. DAPO's objection is that this is noise: "a sound reasoning process can be penalized solely due to its excessive length"3 (§3.4). The loss aggregation makes it worse (derived). Truncated samples are the longest samples in the batch. Under sample-level averaging each of their tokens gets the smallest weight in the group; under token-level averaging they hold the largest share, as the 7,800-token response holds 34.5% of the weight in Table 2. A scheme that fixes the length bias therefore sends most of the negative push to the samples most likely to be unfinished rather than wrong, which is why DAPO ships the fix together with truncation handling.

DAPO has two remedies. Overlong filtering masks the loss of truncated samples. Soft overlong punishment keeps them but adds a graded length reward to the correctness reward3 (§3.4, Eq. 13):

$$R_{\text{length}}(y)=\begin{cases}0,&|y|\le L_{\max}-L_{\text{cache}}\\[2pt] \dfrac{(L_{\max}-L_{\text{cache}})-|y|}{L_{\text{cache}}},&L_{\max}-L_{\text{cache}}<|y|\le L_{\max}\\[6pt] -1,&L_{\max}<|y|.\end{cases}$$

The paper sets an expected maximum of 16,384 tokens and a punishment cache of 4,096, so the hard cap is \(L_{\max}=16{,}384+4{,}096=20{,}480\)3 (§4.1). A response of 18,432 tokens, halfway through the ramp, receives \((16{,}384-18{,}432)/4{,}096=-0.5\), so a correct answer there scores 0.5 where a correct answer under 16,384 tokens scores 1 and a wrong one scores 0 (derived). TRL ships the same function as get_soft_overlong_punishment(max_completion_len, soft_punish_cache) and a mask_truncated_completions option for the filtering variant, default False54.

Soft overlong punishment as a function of response length with a 16,384-token threshold and 4,096-token ramp 0.0 −0.5 −1.0 12,000 14,000 18,000 22,000 16,384 20,480 18,432 tokens: −0.5 no penalty soft zone, L_cache = 4,096 truncated: −1 response length |y| (tokens) length reward R_length
Figure 5. DAPO’s soft overlong punishment (Eq. 13) with the paper’s settings: an expected maximum of 16,384 tokens, a 4,096-token punishment interval and a hard cap L_max = 20,480. The penalty is added to the correctness reward. Plotted from Eq. 13 and §4.1 of the paper3; the TRL helper implements the same piecewise function5.

The evidence is mixed. In DAPO's ladder, overlong filtering is the first and, after dynamic sampling, the second largest step (30 to 36), and soft punishment adds three more (38 to 41)3 (Table 1). Olmo 3 tried "additional changes (e.g., overlong filtering)" and "did not find these gave consistent improvements"10 (footnote 34). ScaleRL compared two ways to bound length under its recipe, forced interruption (inserting an end-of-thinking phrase at a random point between 10,000 and 12,000 tokens) and a DAPO-style penalty applied to correct traces only, and found that replacing interruption with the penalty "does not improve performance"; the penalty's leave-one-out run has \(B=1.75\) against 2.01 for the full recipe at an unchanged \(A=0.610\)9 (App. A.10, Fig. 7). ScaleRL also treats truncation rate as a health signal: at batch size 768, truncation of 10% to 15% "typically destabilized training", while its own runs stayed below 5% for over 90% of training9 (App. A.15). DeepSeek-V3.2 uses a length penalty inside the reward for reasoning and agent tasks and trains its Speciale variant "with a reduced length penalty"8 (§3), so length control by reward is alive at the frontier next to the loss-side fixes of Section 2.

6. Advantage without the standard deviation

DeepSeek-V3.2 sets the advantage to "subtracting the average reward of the group from the reward of output", \(\hat A_{i,t}=R_i-\mathrm{mean}(\boldsymbol R)\), with no division8 (§3.1). The objective printed beside it still averages with \(1/|o_i|\) and carries the KL term, as a review of GRPO, and the report does not say whether its implementation aggregates at sample or token level or why the standard deviation was dropped8. Olmo 3 states its reason: "no standard deviation normalization [...] removes a difficulty bias", following Dr. GRPO, alongside a token-level loss normalized over the batch's tokens and no KL loss10 (§4.4.1). Two consequences follow, both derived here.

Update size now follows the reward scale. TRL's documentation says so directly: turning off std scaling "removes variance normalization, so update magnitudes depend directly on the raw reward scale and batch composition"4. For a binary reward the centered advantage has standard deviation \(\sqrt{\hat p(1-\hat p)}\le 0.5\), so a balanced prompt's advantages are at most half the size they would be after division (a quarter of the energy \(\sum_iA_i^2\)), and those of extreme prompts are far smaller (Figure 3). A global factor like that is absorbed by Adam; the relative scale between prompts and between reward components is what remains, and with reward terms of different magnitude that is a design decision rather than a side effect.

Composite rewards change what "dead" means. Suppose the reward is accuracy (0 or 1) plus \(0.1\,c\), where \(c\in[0,1]\) is a continuous auxiliary term such as a language-consistency fraction, and take an all-correct group whose \(c\) values have a standard deviation of 0.015. The reward spread is 0.0015. Centered, the advantages are about \(\pm 0.0015\), roughly 300 times smaller than the \(\pm0.5\) of a balanced accuracy group. Divided by the group standard deviation they become unit-variance and the jitter in a 10%-weight bonus drives the gradient of a prompt the verifier calls solved with the full strength of an informative one (illustrative numbers; Module 10, §2.4 works the language-consistency reward in detail). The same composite reward also defeats a filter keyed on total reward: groups are "all identical" only when every component ties, so dead-in-accuracy groups pass through Olmo 3's filter ("rewards are all identical")10 or TRL's frac_reward_zero_std counter5. DAPO's constraint is written on accuracy, \(0<|\{o_i:\text{is\_equivalent}(a,o_i)\}|\lt G\)3 (Eq. 8), which ignores the auxiliary term by construction. Which condition to filter on is a choice that interacts with whether you divide by the standard deviation.

7. What the controlled comparisons show

7.1 DAPO's ladder

DAPO's headline is 50 AIME 2024 points on Qwen2.5-32B against 30 for "naive GRPO" on the same base and 47 for DeepSeek-R1-Zero-Qwen-32B3 (Fig. 1, Table 1). Figure 6 plots the ladder. Of its five rungs, two concern this module (token-level loss, dynamic sampling), two are truncation fixes (Section 5) and one belongs to Module 12 (Clip-Higher).

DAPO ablation ladder on AIME 2024 avg@32 with Qwen2.5-32B base 0 10 20 30 40 50 AIME 2024 avg@32 (%) DeepSeek-R1-Zero-Qwen-32B 47 Naive GRPO 30 + overlong filtering 36 +6 + Clip-Higher 38 +2 + soft overlong punishment 41 +3 + token-level loss 42 +1 + dynamic sampling (DAPO) 50 +8 step gain
Figure 6. DAPO’s cumulative ablation: each row adds one technique to the row above. Qwen2.5-32B base, AIME 2024, avg@32 at temperature 1.0 and top-p 0.7; one number per row, no error bars; the gain column is the difference between adjacent rows, so it depends on the order. Data: DAPO Table 1 and §4.13.

Three limits apply before using the ladder to rank techniques. It is cumulative, so each gain is conditional on the rows above it and would differ in another order; dynamic sampling's +8 comes last, when more than half the prompts are dead (Section 4.1), which is when it has the most to recover (derived reading). It is one base model and one benchmark: AIME 2024 has 30 problems, and avg@32 removes most sampling noise but not problem-set noise. Even so, sampling alone leaves a standard error of up to \(\sqrt{30\times0.25/32}/30=1.6\) points when each problem's accuracy is near one half (derived), so the +1 of the token-level loss and the +2 of Clip-Higher cannot be separated from zero without repeated runs, and the table reports one number per row. The authors themselves describe the token-level loss as bringing "less performance improvement" but better stability3 (§4.2).

7.2 ScaleRL's two tests

ScaleRL (Khatri et al.) is the only study in the source set that varies loss aggregation and advantage normalization in a controlled way and fits a compute curve to each: pass rate \(R\) as a saturating function of GPU-hours with an asymptote \(A\) and an efficiency exponent \(B\)9 (Eq. 1). Its forward ablations start from a PipelineRL baseline on an 8B dense model, with 48 prompts of 16 generations (batch 768) from the Polaris-53K math set. Loss aggregation: prompt average and token average both reach \(A=0.535\) (\(B=2.98\) and \(2.04\)), sample average \(A=0.520\) (\(B=4.55\)). Advantage normalization: none \(A=0.525\) (\(B=3.69\)), batch-level 0.530 (3.15), prompt-level 0.520 (4.55)9 (App. A.9, Fig. 14; values read from the printed legends). The paper adopts prompt average because it "achieves the highest asymptotic performance", and batch-level normalization because it is "theoretically sound and marginally better" while "all three methods are observed to yield similar performance"9 (§3.2).

The spreads are 0.015 and 0.010 in \(A\), smaller than the \(\pm0.02\) seen across three independent runs of the final recipe9 (Fig. 8a). \(B\) cannot be compared across runs with different \(A\), because the two fitted parameters trade off: sample average and prompt-level normalization both fit \(B=4.55\) at the lowest \(A\). The paper therefore pins \(A=0.605\) in its leave-one-out table (Figure 7), where each choice is reverted from the full recipe at 16,000 GPU-hours per run.

ScaleRL leave-one-out: fitted efficiency exponent B at fixed asymptote A = 0.605 for nine reverted design choices 1.5 1.6 1.7 1.8 1.9 2.0 2.1 fitted B at fixed A = 0.605 (axis starts at 1.5) own fit A ScaleRL (reference) 2.01 0.610 no FP32 logits 1.97 0.610 off-policy k = 8 (PPO) 1.89 0.605 sample-average loss 1.84 0.600 DAPO loss (not CISPO) 1.83 0.610 prompt-level adv. norm 1.82 0.590 length penalty 1.75 0.610 no zero-variance filter 1.69 0.595 uniform prompt sampling 1.62 0.605
Figure 7. ScaleRL’s leave-one-out table: starting from the full recipe, revert one choice and refit. Dots are the efficiency exponent B with the asymptote pinned at A = 0.605 (higher is faster); the right column is each run’s own fitted A. Orange dots are the choices this module covers. Run-to-run variation in A across three ScaleRL runs is ±0.02. 8B dense model, math, 16,000 GPU-hours per run. Values transcribed from the printed table in Figure 7 of the paper9.

Every reverted choice here lands within 0.02 of the full recipe's \(A=0.610\); the largest gap, prompt-level standard-deviation division at 0.590, sits on the noise margin. Pinned-\(A\) efficiency falls from 2.01 to 1.84 (sample-average loss), 1.82 (prompt-level division), 1.69 (no zero-variance filtering) and 1.62 (uniform prompt sampling), that is by 8%, 9%, 16% and 19% (derived), though the paper gives no margin for \(B\) and does not translate the differences into GPU-hours. This is the evidence behind the claim that these choices "primarily modulate compute efficiency without materially shifting the asymptote"9 (abstract). Two reservations. The asymptote is a fitted parameter extrapolated from runs of at most 16,000 GPU-hours (fits up to 8,000), and the 100,000 GPU-hour validation run is the full recipe only. And the same study finds that the loss type (CISPO or GSPO against DAPO) and FP32 logits do move the fitted asymptote in the forward tests, from 0.52 to 0.61 for the latter9 (§3.2, Fig. 5), so "mostly efficiency" describes aggregation and normalization specifically (Modules 12 and 14 cover the others). One model family, one data domain, one benchmark protocol.

7.3 The ledger

Table 6. Controlled evidence on the choices of this module, with what limits each entry. Sources are cited in the first column.
EvidenceSettingContrastResultLimit
DAPO Table 13Qwen2.5-32B base, DAPO-Math-17K, AIME24 avg@32six cumulative rows30, 36, 38, 41, 42, 50one run per row, order-dependent, 30 problems; sampling s.e. up to 1.6 (derived)
Dr. GRPO Figs. 8 and 92Qwen2.5-1.5B, 3K questions, 8 samples, 3,000 tokensGRPO, without length norm, without std norm, Dr. GRPOboth single removals beat vanilla; length term drives length; Dr. GRPO better across 3 seedssmall model, short cap, curves rather than a table
Dr. GRPO Table 42Qwen2.5-Math-7B, MATH levels 3 to 5, greedy, 3,000-token budgetOat-Zero-7B against other 7B recipesAIME24 43.3 (13 of 30 problems), five-benchmark average 51.4different data, templates and frameworks across rows: not a loss ablation
ScaleRL Fig. 1498B dense, math, batch 768, PipelineRL baseline3 aggregations, 3 advantage scalingsA spread 0.015 and 0.010inside the 0.02 run-to-run margin; B not comparable across A
ScaleRL Fig. 79same, 16,000 GPU-hours per runrevert one choice from the full recipeA 0.590 to 0.610; pinned-A B 1.62 to 2.01one run per variant; no margin on B; fitted asymptote extrapolated
Olmo 3 Fig. 26, footnote 3410RL-Zero math, 7Bactive sampling on and off; overlong filtering triedactive sampling keeps the batch full and lowers loss variance; overlong filtering no consistent gainfigure and one sentence; no table
MiniMax-M1 §513M1 at 40K to 80K output lengthtoken-level loss with GRPO advantage normalization at long lengthnegative samples lengthen faster, "disproportionately large negative gradients" in sequence tailsnarrative; no ablation numbers for the fix

The last row is the counterweight to Section 2. MiniMax-M1 scaled generation from 40K to 80K tokens and observed pattern collapse late in each length window, with the tails of generations turning into incoherent text and perplexity rising. Its diagnosis: negative samples "increase in length substantially faster than positive samples", so "disproportionately large negative gradients accumulate in the latter segments of generation sequences", an imbalance it traces to "the inherently unequal nature of GRPO's advantage normalization and the token-level loss we adopt"13 (§5). That is the 4.19-fold negative-to-positive imbalance of Table 2, seen in a production run. Its remedies were early stopping on repetitive patterns, "combined sample-level loss and token-level normalization", and lower gradient-clipping and \(\epsilon_{\text{high}}\) thresholds13. The report does not publish an ablation of any of the three.

8. The choices side by side

Table 7. Loss aggregation and advantage normalization in nine documented recipes, as printed in each source. "Not stated" means not stated in the sections read; none of these reports defines the std convention except TRL's code (Section 3.2).
RecipeLoss aggregationAdvantage scaleKLDead groupsGSource
DeepSeekMath-RL (original GRPO)sample-levelgroup stdin the loss, 0.04not discussed64§4.1 to §4.21
DAPOprompt-level token mean (Eq. 8)group std (Eq. 9)nonedynamic sampling on accuracy16§2.3, §3, §4.13
Dr. GRPOconstant (generation budget, 3,000 tokens in its runs)nonenonenot discussed8§3.2, Table 62
DeepSeek-V3.2sample-level as printed; implementation not stated\(R_i-\mathrm{mean}(R)\)unbiased K3 estimate; weak or none for mathnot statednot stated§3.18
OlmoRLbatch-level token meannonenonezero-gradient filtering, active sampling8§4.4.1, Table 4910
ScaleRLprompt-level averaging, CISPO lossbatch std of the advantagesnone in the printed losszero-variance filtering, no-positive-resampling16§3.2, §49
GLM-5 reasoning RLsample-level (Eq. 1)group stdremovednot stated32§3.27
MiniMax-M1token-level (Eq. 4); sample-plus-token hybrid at 80Kgroup std (Eq. 2)nonedynamic sampling from DAPOnot stated§3.1, §513
TRL defaultsdapo: token mean over the global accumulated batchgroup: Bessel std plus 1e-40.0none; logs frac_reward_zero_std8config and trainer source54

No two rows are the same recipe. The three with controlled experiments behind them (DAPO, Olmo 3, ScaleRL) share a direction: aggregate by tokens or by prompt rather than by response, handle dead groups explicitly, and drop the KL term for verifiable rewards. They disagree on the standard deviation: group-level in DAPO, absent in Olmo 3, batch-level in ScaleRL. The library default is a mixture, DAPO's loss with GRPO's group division, a combination whose std half Dr. GRPO argues against and Olmo 3 and V3.2 do not use.

Where experts disagree

The question is how the GRPO loss and advantage should be normalized. Everyone who has measured it agrees that original GRPO's per-response length average is unsatisfactory in long-chain-of-thought runs. They disagree on the replacement, on whether the standard deviation belongs in the advantage, and on whether token-level weighting is safe when responses reach tens of thousands of tokens.

A. Remove both normalizers

Dr. GRPO2; Olmo 3 (token-level loss over the batch, no std)10; DeepSeek-V3.2 on the advantage8; TRL's documentation recommends no scaling on the Dr. GRPO authority4

Dividing by \(|o_i|\) changes the objective inside each prompt and dividing by the standard deviation changes the weight of each prompt. Take both out and the estimator is the plain Monte Carlo policy gradient with a leave-one-out-equivalent baseline, equal to RLOO up to a factor \(G/(G-1)\)2 (App. A).

  • Unbiased by construction; fewer knobs; the length term moves length and the std term does not (Dr. GRPO ablation).
  • Adopted independently by a fully open pipeline (Olmo 3) and a frontier report (V3.2, advantage only).
  • Evidence is small-scale (1.5B to 7B, 3,000-token caps) and the final comparison is not a loss ablation.
  • Down-weights rare-success prompts6; update size now tracks the raw reward scale4.

B. Token-level loss fixes length; keep the group std

DAPO3; TRL's defaults5; MiniMax-M1's CISPO loss13

Sample-level averaging underweights long samples and cannot penalize gibberish in them; a token-level loss fixes that and leaves the advantage as in GRPO, with dynamic sampling removing dead groups.

  • The best-documented full recipe on a hard benchmark: 30 to 50 AIME24 points on one base model.
  • Keeps every non-dead prompt at unit advantage scale.
  • Eq. 8's per-prompt denominator brings back a prompt-to-prompt weighting; the std division is the difficulty bias of Section 3.
  • The token-level gain is +1 on the ladder, inside sampling noise.

C. These are efficiency knobs; look elsewhere for the asymptote

ScaleRL (Khatri et al.)9

Under fitted compute curves, loss aggregation, advantage normalization, curriculum and off-policy algorithm "primarily modulate compute efficiency without materially shifting the asymptote"; use prompt-level averaging and batch-level normalization and spend effort on loss type and numerics.

  • The only study that fits curves, reports a noise margin and reverts one choice at a time at 16,000 GPU-hours.
  • Consistent with the small A spreads in Table 6.
  • One 8B dense model, one math domain, one protocol; the asymptote is extrapolated.
  • An 8% to 19% change in the efficiency exponent is not nothing, and B has no stated margin.

D. Token-level weighting is unsafe at extreme length; keep sample-level or a hybrid

MiniMax-M1 at 80K tokens13; GLM-5 reasoning RL keeps \(1/|y_i|\) and group std7; V3.2's printed objective8

When negative samples are far longer than positive ones, token-level weighting lets long, degenerate tails dominate the negative gradient; sample-level averaging keeps positive and negative mass balanced (1.0 against 4.19 in Table 2).

  • Observed failure mode in a real 80K run, with a stated mechanism.
  • Balanced push per response regardless of length.
  • Reintroduces the length preference of Section 2.1 and its accuracy-dependent drift.
  • No published ablation for M1's hybrid or GLM-5's choice; V3.2's implementation is unstated.
Assessment

On the length term, A and B are right and the toy of Section 2.2 shows why: the 1/|o| factor changes each prompt's optimum with a sign that depends on accuracy, and every controlled comparison that touched it points the same way. D is right about a different problem. The token-level estimator is unbiased, but unbiasedness does not make a very long repetition loop a good thing to weight in proportion to its length. At 80K the remedy M1 reports for that (repetition detection and early stopping, with truncation handling) addresses the cause; reintroducing \(1/|o|\) trades one known bias for another, and no one has measured that trade. On the standard deviation, the evidence is thin and says second-order. Use a batch-level scale or none; if you keep the group division, filter dead groups on accuracy and look at the spread of each reward component before trusting the normalized advantage (Section 6). C is the best-calibrated reading of the data, with the scope warning that it is one model family. The experiment that would settle this is a matched grid of \(\{\)sample, prompt, token, constant\(\}\times\{\)group, batch, none\(\}\) with at least three seeds and error bars at both a short and a very long generation cap, logging the wrong-to-correct length ratio and the dead-group fraction. It does not exist yet.

A second, smaller disagreement is how to control response length. Dr. GRPO argues the unbiased loss itself curbs overthinking2; DAPO, V3.2 and others add a length penalty38; ScaleRL prefers forced interruption9. No source reverts the loss fix and the penalty jointly (derived from the evidence ledger), so the defensible order is to fix the loss first, since it costs nothing, and enforce a length budget with a penalty or interruption separately.

What is changing

Hugging Face TRL's loss_type now lists variants beyond the papers read here, among them luspo, "Length-Unbiased Sequence Policy Optimization", a sequence-level loss that scales each sequence's loss by its length and modifies GSPO, and vespo and sapo (pointer only: described in the TRL configuration and documentation, not read at the paper level)54. That a length-unbiased loss was written for sequence-level ratios shows the length question returns whenever a method averages at the response level (Module 12 covers GSPO and its relatives). TRL also offers multi_objective_aggregation="normalize_then_sum", which normalizes each reward component within the group before summing, aimed at the composite-reward problem of Section 65. Open as of 2026-10-27: whether the aggregation results hold at 32B and above and for mixture-of-experts policies, since the ScaleRL ablations are dense 8B, and which aggregation DeepSeek-V3.2 actually runs.

References

  1. paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the GRPO objective (§4.1.1, Eq. 3) and the outcome-supervision advantage written with std(r) (§4.1.2); DeepSeekMath-RL settings (§4.2: KL 0.04, 64 outputs per question, one policy update per exploration stage).
  2. paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the two biases (§3.1), Dr. GRPO and Listing 1 (§3.2), Table 2 (PPO implementations), Figs. 5 and 7, Table 4 (App. B), Figs. 8 and 9 (App. C), RLOO equivalence (App. A), hyperparameters (App. G, Table 6).
  3. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. (ByteDance Seed, Tsinghua AIR). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: Eqs. 8, 9 and 13; dynamic sampling and Algorithm 1 (§3.2); token-level loss (§3.3); overlong reward shaping (§3.4); training details (§4.1); Table 1 ladder (§4.2); Fig. 3b read off the rendered plot.
  4. officialHugging Face. TRL documentation: GRPO Trainer (main branch; no version stated on the page). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the loss-type formulas and the DAPO and Dr. GRPO remarks ("Loss Types"), the scale_rewards note on raw reward scale, the frac_reward_zero_std metric, defaults.
  5. repoHugging Face. TRL source: trl/trainer/grpo_trainer.py (last changed in commit 63d9074e, 2026-10-21), grpo_config.py, utils.py (nanstd) and rewards/other_rewards.py, main branch. https://github.com/huggingface/trl/tree/main/trl. Accessed 2026-10-27. Used for: per-loss-type normalizers in compute_loss, nanstd with Bessel’s correction and the +1e-4 epsilon in advantage scaling, the scale_rewards="batch" computation, get_soft_overlong_punishment, mask_truncated_completions default, loss_type and multi_objective_aggregation options. Files fetched from raw.githubusercontent.com/huggingface/trl/main/.
  6. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: the trade-off of removing std normalization (pp. 73–74) and the three loss-aggregation strategies (§6.3.2).
  7. paperGLM-5 Team (Zeng, A., Lv, X. et al.; Zhipu AI and Tsinghua University). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the reasoning-RL loss with 1/|y_i|, group-std advantage, KL removed, group size 32 (§3.2, Eq. 1).
  8. paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: Mixed RL rewards and the Speciale variant (§3); the printed GRPO objective and the advantage \(R_i-\mathrm{mean}(R)\) (§3.1, Eqs. 5 and 6).
  9. paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R. et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: loss-aggregation and advantage-normalization ablations (§3.2, Fig. 14), the ScaleRL recipe and leave-one-out table (§4, Figs. 7 and 8), length control (App. A.10), truncation rates (App. A.15), recipe descriptions (App. A.2, A.16). Values in Figs. 7 and 14 read from the rendered PDF.
  10. paperTeam Olmo (Ettinger, A., Bertsch, A. et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL algorithm details and footnote 34 (§4.4.1), active sampling and rollout cost (§4.4.3, footnote 41), Table 49 hyperparameters, Fig. 26 (RL-Zero active sampling).
  11. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (expanded Nature version), 4 Jan 2026. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the group-normalized advantage (Eq. 3) and the thinking-time reading of the response-length curve (Fig. 1b).
  12. paperShao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., Du, S. S., Lambert, N., Min, S., Krishna, R., Tsvetkov, Y., Hajishirzi, H., Koh, P. W., Zettlemoyer, L. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947v2, 25 Feb 2026. https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: the random-reward result (abstract: +21.4 against +29.1 points on MATH-500) and its clipping-bias explanation.
  13. paperMiniMax (Chen, A., Li, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: the CISPO objective with token-level loss and group-std advantage (§3.1, Eqs. 2–4); the 80K length-scaling instability and its remedies (§5).