Reinforcement Learning for LLM Post-Training · Module 11
The GRPO Loss and Advantage Under the Microscope: Length Bias, Difficulty Bias, Token-Level Averaging, and Zero-Variance Groups
Two denominators decide how much gradient each token of a GRPO update receives: the response length |oi| in the loss, and the group's reward standard deviation in the advantage. This module derives the weight a token gets under every published aggregation, shows what Dr. GRPO, DAPO, DeepSeek-V3.2, Olmo 3 and ScaleRL each changed and why, and states what their ablations do and do not show. You can compute the per-token weight and the dead-group fraction for your own batch and pick a normalization with the evidence in hand.
1. Where the gradient weight of a token comes from
1.1 One formula for every variant
Take one prompt \(q\), a group of \(G\) samples \(o_1,\dots,o_G\) drawn from the rollout policy, lengths \(|o_i|\), and one scalar reward \(R_i\) per sample. DeepSeekMath's GRPO objective averages a clipped per-token surrogate first over the tokens of each response and then over the group1 (§4.1.1, Eq. 3), with the advantage \(\hat A_i = (R_i-\mathrm{mean}(R))/\mathrm{std}(R)\) copied to every token of response \(i\)1 (§4.1.2). Dr. GRPO writes the same objective in its Eq. 32.
When the policy takes one gradient step per rollout the ratio is exactly 1, the clip is inactive and the surrogate's gradient is \(\hat A\,\nabla\log\pi\) (Module 9, §3.3 derives this; DeepSeekMath-RL uses a single update per exploration stage1, §4.2). Drop the KL term and every aggregation covered in this module collapses to one expression:
$$\nabla\mathcal J=\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}\frac{\hat A_i}{Z_i}\,\nabla\log\pi_\theta(o_{i,t}\mid q,o_{i,\lt t}),\qquad \hat A_i=\frac{R_i-\mathrm{mean}(R)}{s}.$$Two choices remain: the scale \(s\) in the advantage and the denominator \(Z_i\) of the token average. They are the two knobs of Figure 1. (For a batch of \(P\) prompts, sum over all \(B=PG\) responses and read \(Z_i\) as in Table 1; for a single prompt, \(P=1\) and \(B=G\).) Every token of response \(i\) receives the same coefficient \(\hat A_i/Z_i\) in front of its score function, so "how much does this token matter" is the same question as "what is \(Z_i\)". The rest of the module asks what each choice of \(Z\) and \(s\) does to the weights, and what the experiments say.
1.2 Four denominators in use
Table 1 lists the denominators that appear in the reports and in Hugging Face TRL, which exposes most of them as loss_type values45. Lambert's textbook describes the same three families of aggregation6 (§6.3.2).
| Scheme | Denominator \(Z_i\) | Per-token weight | Total for response \(i\) | Where it appears |
|---|---|---|---|---|
| Sample-level | \(B\,L_i\) | \(1/(B L_i)\) | \(1/B\), any length | DeepSeekMath Eq. 31; GLM-5 Eq. 17; V3.2 as printed8; TRL grpo4 |
| Prompt-level token mean | \(P\sum_{j\in\text{group}} L_j\) | \(1/(P\sum_j L_j)\) | \(L_i/(P\sum_j L_j)\) | DAPO Eq. 83; "prompt average" in ScaleRL9 |
| Batch-level token mean | \(N\) (all tokens in the batch) | \(1/N\) | \(L_i/N\) | Olmo 310; TRL dapo (default)4; "token average" in ScaleRL9 |
| Constant | \(B\,L_{\max}\) | \(1/(B L_{\max})\) | \(L_i/(B L_{\max})\) | Dr. GRPO2; TRL dr_grpo5 |
Three relations follow directly (derived). First, only the sample-level scheme makes the total weight of a response independent of its length; all other schemes make it proportional to \(L_i\). Second, the batch-level token mean and the constant normalizer give every token in a batch the same weight, so they differ by one scalar per batch, \(N/(B L_{\max})\), and by nothing else. Third, DAPO's Eq. 8 normalizes inside each prompt, so tokens of a prompt whose group happened to be long get a smaller weight than tokens of a prompt whose group was short: it removes the length weighting among the samples of one prompt and introduces a prompt-to-prompt weighting by \(1/\sum_j L_j\), the same kind of effect as the standard-deviation division of Section 3.
Dr. GRPO's Listing 1 comments that per-batch normalization, as in TRL and verl at the time, is also biased2; TRL's documentation summarizes the paper as saying the DAPO formulation "reduces this bias" but does "not eliminate it completely"4. By the second relation above, the residual for the batch-level mean is only that \(N\) is itself random and contains the sampled tokens (a ratio estimator), an effect that shrinks with batch size; neither source quantifies it.
1.3 One group, worked through
Take \(G=8\) samples of one prompt, three correct (reward 1) and five wrong (reward 0), with wrong answers longer than correct ones, as is typical when wrong attempts ramble or run to the cap. The lengths below are illustrative; the shape matches Dr. GRPO's measurements, where the y-axes of its Figure 5 run from roughly 200 to 400 tokens for correct and 1.0k to 1.8k tokens for incorrect responses2 (read off the axes). The centered advantage is \(+0.625\) for a correct sample and \(-0.375\) for a wrong one (derived: mean \(3/8\)).
| i | Outcome | Tokens \(L_i\) | \(A_i\), centered | Relative per-token weight, sample-level | Share of group weight, sample-level | Share of group weight, token-level or constant |
|---|---|---|---|---|---|---|
| 1 | correct | 640 | +0.625 | 4.41 | 12.5% | 2.8% |
| 2 | correct | 910 | +0.625 | 3.10 | 12.5% | 4.0% |
| 3 | correct | 1,280 | +0.625 | 2.21 | 12.5% | 5.7% |
| 4 | wrong | 1,150 | −0.375 | 2.45 | 12.5% | 5.1% |
| 5 | wrong | 2,100 | −0.375 | 1.34 | 12.5% | 9.3% |
| 6 | wrong | 3,500 | −0.375 | 0.81 | 12.5% | 15.5% |
| 7 | wrong | 5,200 | −0.375 | 0.54 | 12.5% | 23.0% |
| 8 | wrong | 7,800 | −0.375 | 0.36 | 12.5% | 34.5% |
| Total push on correct samples | 2,830 | | | 0.234 | 0.078 | |
| Total push on wrong samples | 19,750 | | | 0.234 | 0.328 | |
Read the table in two directions. Per token, the first correct response (640 tokens) counts 12.2 times as much as a token of the last wrong one (7,800 tokens) under sample-level averaging (\(7{,}800/640\)); under the other schemes the two weigh the same. Per response, sample-level averaging gives all eight the same 12.5% of the group's weight, and the positive and negative sides balance exactly because \(\sum_i A_i=0\). Under token-level averaging the weights follow length, and the negative side carries \(0.3280/0.0783=4.19\) times the positive side. That ratio is not an accident: because the advantages sum to zero, it equals the ratio of mean lengths of wrong to correct responses, \(3{,}950/943=4.19\) (derived). So the choice of \(Z\) decides whether a long wrong answer is pushed down once, as one response, or in proportion to its length. MiniMax-M1 reports this effect from the other side in Section 7.
1.4 The aggregation as code
The four schemes differ in one line each. The function below mirrors Dr. GRPO's Listing 12 and the normalizers in TRL's compute_loss5 (single-device case; TRL's dapo and cispo normalize by the token count of the global accumulated batch). I checked that autograd gives the per-token weights of Table 1 for random lengths with torch 2.10.
import torch
def aggregate(per_token_loss, mask, kind, group_size=None, l_max=None):
"""per_token_loss, mask: (B, T). Rows are consecutive groups of group_size samples."""
if kind == "sample": # original GRPO; TRL loss_type="grpo"
return ((per_token_loss * mask).sum(-1) / mask.sum(-1).clamp(min=1)).mean()
if kind == "prompt": # DAPO Eq. 8; ScaleRL "prompt average"
P = per_token_loss.size(0) // group_size
num = (per_token_loss * mask).reshape(P, -1).sum(-1)
den = mask.reshape(P, -1).sum(-1).clamp(min=1)
return (num / den).mean()
if kind == "batch": # Olmo 3; TRL loss_type="dapo" on one device
return (per_token_loss * mask).sum() / mask.sum().clamp(min=1)
if kind == "constant": # Dr. GRPO; TRL loss_type="dr_grpo"
return (per_token_loss * mask).sum() / (per_token_loss.size(0) * l_max)
2. Response-level length bias
2.1 GRPO ascends a different reward
Dr. GRPO's first diagnosis is that dividing by \(|o_i|\) makes the update favor short responses when the advantage is positive and long responses when it is negative: "for positive advantages [...] this bias results in greater gradient updates for shorter responses", while for negative ones "longer responses are penalized less due to their larger \(|o_i|\)"2 (§3.1). The same mechanism can be stated exactly, which also shows what the bias is a bias of.
Fix a baseline \(b\) (the group mean in the limit of a large group) and a scale \(s\). The sample-level estimator is \(\frac1G\sum_i \frac{R_i-b}{s\,|o_i|}\nabla\log\pi_\theta(o_i)\), where \(\nabla\log\pi_\theta(o_i)=\sum_t\nabla\log\pi_\theta(o_{i,t}\mid\cdot)\). For samples drawn from \(\pi_\theta\) its expectation is the policy gradient of a modified reward, treated as a fixed function of the response (derived; the identity \(\mathbb E[f(o)\nabla\log\pi_\theta(o)]=\nabla\mathbb E[f(o)]\) is Module 2's score-function step):
$$\mathbb E\Big[\tfrac{R-b}{s|o|}\,\nabla\log\pi_\theta(o)\Big]=\nabla_\theta\,\mathbb E_{o\sim\pi_\theta}\big[\tilde r(o)\big],\qquad \tilde r(o)=\frac{R(o)-b}{s\,|o|}.$$A constant denominator (Dr. GRPO) leaves \(\tilde r=(R-b)/s\), the reward you meant. The sample-level denominator makes the reward depend on length. Compare two responses with the same outcome. If both are correct (\(R-b>0\)), \(\tilde r\) is larger for the shorter one. If both are wrong (\(R-b<0\)), \(\tilde r\) is closer to zero, that is larger, for the longer one. Inside each outcome class the objective has acquired a length preference whose sign depends on the sign of the advantage. That is the Dr. GRPO statement, restated as an objective.
The standard-deviation division is a different kind of change. Dividing by \(s_q\), a constant per prompt, multiplies every \(\tilde r\) of that prompt by the same positive number, so it cannot reorder two samples of the same prompt; it only changes how loudly the prompt speaks relative to others (Section 3). The length term does reorder samples inside a prompt.
The length normalization did not originate in GRPO. Dr. GRPO checked five open-source PPO implementations (TRL, OpenRLHF, verl, SimpleRL-Zero, Open-Reasoner-Zero) and found all of them normalize the loss by response length, "even before the publication of GRPO"; it speculates the habit comes from pretraining, where loss.mean(-1) over a fixed packed context is a harmless stabilizer2 (§3.1, Table 2).
2.2 Which way does length move? A toy with a closed form
The identity says the objective prefers short-among-correct and long-among-wrong. Whether mean length rises or falls depends on the mix, and a small model shows how (derived here; not in the source). One prompt, four response types: correct or wrong, crossed with short (\(S\) tokens) or long (\(L\) tokens). The probability of being correct is \(p\) and does not depend on length, so a correct reward has no reason to change length. The policy is a softmax over the four types, updated by gradient flow on the expected modified reward, with the baseline at its large-group limit \(b=p\). With long and short equally likely, the expected change in mean length is
$$\frac{d\,\mathbb E[\text{length}]}{dt}\;\propto\;(1-2p)\,p(1-p)\,\frac{(L-S)^2}{8\,S\,L},$$and exactly zero when the denominator is a constant, because then the reward ignores length. The sign is the point: sample-level averaging lengthens responses on prompts the policy mostly fails (\(p<\tfrac12\)) and shortens them on prompts it mostly solves (\(p>\tfrac12\)), with no length signal in the reward at all.
| Pass rate \(p\) | 0.1 | 0.2 | 0.3 | 0.5 | 0.7 | 0.8 | 0.9 |
|---|---|---|---|---|---|---|---|
| Sample-level \(1/|o_i|\) | +0.0203 | +0.0270 | +0.0236 | 0 | −0.0236 | −0.0270 | −0.0203 |
| Constant or token-level | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Real prompts mix both regimes, and real wrong answers are longer than right ones for reasons that are not an artifact (more flailing, more truncation), so the net effect on a training run depends on the pass-rate distribution. What the toy establishes is narrower: the bias has a direction that depends on accuracy, it needs no correlation between length and reward, and the token-level and constant normalizers remove it in expectation. The toy also says why the symptom appears on hard prompts and early in training: that is where \(p<\tfrac12\).
2.3 What Dr. GRPO measured, and how far it reaches
Dr. GRPO's main training comparison uses Qwen2.5-1.5B with the R1 prompt template, MATH training questions, a 0/1 Math-Verify reward, eight samples per question, a 3,000-token generation cap, learning rate \(10^{-6}\), no KL term and clip 0.2, on 8 A100 GPUs2 (§3.2; App. G, Table 6). Four observations matter here.
- Both algorithms show response length rising with training reward, as in R1-Zero, whose report presents the rise as the model learning "to solve reasoning tasks with more thinking time"11 (Fig. 1b). GRPO keeps lengthening "even when the reward improvement slows down"; Dr. GRPO does not, and the length of incorrect evaluation responses is substantially lower2 (§3.2, Fig. 5). The authors' claim is that the rise "may also be attributed to" the bias and is "confounded" by it, not that it is only the bias.
- On Llama-3.2-3B continued-pretrained on math, GRPO produces a "double-increase" of accuracy and length that could be misread as emerging long chain-of-thought; Dr. GRPO limits the length growth2 (§3.4, Fig. 7).
- In the ablation (Qwen2.5-1.5B, 3K mixed questions), removing only the length normalization and full Dr. GRPO both produce shorter responses than the other two variants, and all three modified variants beat vanilla GRPO on training reward and evaluation accuracy2 (App. C, Fig. 8). This matches Section 2.1: the length term moves length; the std term does not.
- Across three independent runs Dr. GRPO is better in both token efficiency and final accuracy, "statistically significant" by the paper's account2 (App. C, Fig. 9).
DAPO arrives at the same place from the other direction. It argues that with sample-level averaging "tokens within longer responses [...] may have a disproportionately lower contribution to the overall loss", which prevents learning from good long samples and fails to punish gibberish in bad ones, and its Figure 4 shows entropy and length growing less wildly with a token-level loss3 (§3.3). The ablation table credits the change with one point on AIME but says it "enhances training stability and makes the length increase more healthily"3 (§4.2, Table 1; Section 7 below).
3. Division by the group standard deviation
3.1 The algebra for binary rewards
With a 0/1 verifier and \(k\) correct answers out of \(G\), let \(\hat p=k/G\). The group mean is \(\hat p\) and the population standard deviation is \(\sigma=\sqrt{\hat p(1-\hat p)}\). A correct sample then has advantage \((1-\hat p)/\sigma=\sqrt{(1-\hat p)/\hat p}\) and a wrong one \(-\hat p/\sigma=-\sqrt{\hat p/(1-\hat p)}\); without the division they are \(1-\hat p\) and \(-\hat p\) (Module 9, Table 3 tabulates these at \(G=8\)). The mean absolute advantage over the group, which measures how hard the prompt pulls on the update, is
$$\overline{|A|}_{\text{centered}}=2\hat p(1-\hat p),\qquad \overline{|A|}_{\text{divided}}=\frac{2\hat p(1-\hat p)}{\sigma}=2\sqrt{\hat p(1-\hat p)}.$$The division multiplies a prompt's pull by \(1/\sigma\). Figure 3 plots both. Without the division, a prompt with one success in 64 samples pulls with weight 0.031 against 0.500 for a balanced prompt, a ratio of 16. With it the weights are 0.248 against 1.000, a ratio of 4. In general the weight of a one-success group relative to a balanced one rises from \(1/(2\sigma)=1.51\) at \(G=8\) to 2.07 at \(G=16\) and 4.03 at \(G=64\) (derived; the single-success weight is \(G/\sqrt{G-1}\) against 2).
3.2 Which standard deviation?
The sources are not uniform, and the choice matters at small \(G\). DeepSeekMath writes \(\mathrm{std}(\mathbf r)\) and R1 writes \(\mathrm{std}(\{r_1,\dots,r_G\})\), with neither stating population or sample convention nor any epsilon111 (§4.1.2; Eq. 3 of the R1 paper). DAPO (Eq. 9) and Dr. GRPO (Eq. 3) likewise32. ScaleRL's Eq. 5 adds an unspecified \(\epsilon\) to the denominator and does not state the convention9. I read the TRL source rather than its documentation: grpo_trainer.py computes the group statistic with nanstd, which applies Bessel's correction, and divides by std + 1e-45 (commit 63d9074, 2026-10-21). Bessel's correction multiplies \(\sigma\) by \(\sqrt{G/(G-1)}\): 1.155 at \(G=4\), 1.069 at \(G=8\), 1.033 at \(G=16\), 1.008 at \(G=64\) (derived). The closed forms in this module use the population convention; the sample convention shrinks every divided advantage by that factor and changes no ranking. The \(10^{-4}\) epsilon matters only for groups whose reward spread is of that order, such as near-ties of a continuous reward.
"Batch-level" normalization also has two meanings. ScaleRL divides by the standard deviation of all advantages in the batch, \(\hat A_i/\hat A_{\mathrm{std}}\), where the advantages are already group-centered9 (§4). TRL's scale_rewards="batch" divides the group-centered advantage by the standard deviation of the raw rewards over the whole batch5, which also contains the spread between prompts of different difficulty. Both are a single scalar per batch, so they rescale the update without changing the relative pull of any two prompts (derived), unlike the per-group division.
3.3 What it changes, what it does not, and whether to call it a bias
Dr. GRPO's complaint is that questions with low reward spread, "those that are too easy or too hard", "are given higher weights during policy updates", whereas advantage normalization is usually computed "across an entire batch"2 (§3.1). Figure 3 sharpens the statement. Relative to the centered advantage, extreme prompts are indeed amplified (by 8.1 at one success in 64); relative to a balanced prompt they still pull less (0.25 against 1.0). The division does not make the extremes dominate; it softens their suppression. It also cannot reorder samples within a prompt, so unlike the length term it leaves each prompt's own optimum unchanged (Section 2.1).
Whether softening is desirable is a judgment about the training distribution. Lambert's textbook argues that removing the division "comes at the cost of downweighting questions that were all incorrect with a few correct answers", the hardest cases, whose rare success is the most informative sample in the batch6 (pp. 73–74). The opposite worry is noise. When the reward is continuous, a group of near-ties has a tiny standard deviation, and dividing by it hands unit-variance advantages to what is essentially measurement noise (derived). Section 6 shows where this bites. And no normalization can validate a reward: after division, \(\sum_i A_i^2\) equals \(G\) (population convention) for every non-degenerate group regardless of what the reward means. Training with randomly assigned rewards still raised Qwen2.5-Math-7B's MATH-500 score by 21.4 points against 29.1 for ground-truth rewards; the authors attribute that to a clipping bias in GRPO rather than to normalization12, but it is a reminder that normalized advantages are only as informative as the correlation between reward and response.
The controlled evidence is thin and says the effect is second-order: in ScaleRL's forward ablation, no normalization, batch-level and prompt-level division reach asymptotic pass rates of 0.525, 0.530 and 0.520, "similar performance"9 (App. A.9, Fig. 14b); Section 7 reads those numbers against their noise.
4. Zero-variance groups
4.1 How many groups are dead
If every sample of a prompt gets the same reward, the centered advantage is zero for all of them, and so is every token's coefficient \(\hat A_i/Z_i\) in the gradient. DAPO states the consequence plainly: "A zero advantage results in zero policy gradients, shrinking the magnitude and increasing the noise sensitivity of the batch gradient"3 (§3.2). For a prompt with pass rate \(p\) and a binary verifier the chance of a dead group is
$$P_0(p,G)=p^{G}+(1-p)^{G}.$$Module 2 derived this, tabulated its complement and plotted it (Table 6 and Figure 6 there); Figure 4 here shows the dead side for \(G=4,8,16,64\). The two features that matter are that increasing \(G\) empties the middle of the curve but barely touches the ends (a prompt solved 1% of the time is dead 85% of the time at \(G=16\) and 53% at \(G=64\)), and that the curve is symmetric, so prompts that are nearly always solved are as useless as prompts that are never solved.
A batch holds prompts of many difficulties, so the batch-level dead fraction is the average of \(P_0\) over the pass-rate distribution. For a Beta\((a,b)\) population it has a closed form, \(\mathbb E[P_0]=\big[B(a+G,b)+B(a,b+G)\big]/B(a,b)\) (derived). Table 4 evaluates it for four assumed populations. They are illustrations of shape, not measurements: a mid-heavy population, a uniform one, and two U-shaped ones in which most prompts are nearly always or nearly never solved.
| Pass-rate population | \(G=4\) | \(G=8\) | \(G=16\) | \(G=64\) |
|---|---|---|---|---|
| Beta(2, 2), mid-heavy | 28.6% (1.40x) | 10.9% (1.12x) | 3.5% (1.04x) | 0.3% (1.00x) |
| Beta(1, 1), uniform | 40.0% (1.67x) | 22.2% (1.29x) | 11.8% (1.13x) | 3.1% (1.03x) |
| Beta(0.5, 0.5), U-shaped | 54.7% (2.21x) | 39.3% (1.65x) | 28.0% (1.39x) | 14.1% (1.16x) |
| Beta(0.3, 0.3), strongly U-shaped | 65.9% (2.93x) | 53.4% (2.15x) | 43.4% (1.77x) | 28.6% (1.40x) |
The one measured trace in the source set drifts toward the solved end. In DAPO's run the share of prompts with avg@32 equal to 100% rises from near zero to roughly 0.55 to 0.6 by step 8,000 (read off its Figure 3b; approximate)3. With more than half the prompts dead from that one side alone, refilling needs at least \(1/(1-0.55)\approx 2.2\) to \(2.5\) times as many prompts as the batch holds (derived; a lower bound, since accuracy-0 prompts are dropped too). Olmo 3 describes DAPO's dynamic sampling as generating "three times the number of prompts used in each training batch"10 (§4.4.3), the same order.
A larger \(G\) is not a free fix. Informative prompts per rollout fall as \(G\) grows: for the U-shaped Beta(0.5, 0.5) population, 11.3, 7.6, 4.5 and 1.3 informative prompts per 100 rollouts at \(G=4,8,16,64\) (derived from Table 4). Group size buys a better baseline and rarer dead groups at the price of fewer prompts. ScaleRL's one test at fixed total batch, sweeping 8, 16, 24 and 32 generations per prompt, left the fitted scaling curves "essentially unchanged"9 (§5).
Dead groups and the cost of refilling a batch, for one pass rate
Formula: \(P_0=p^G+(1-p)^G\); prompts to fill \(=B/(1-P_0)\). Assumes every prompt has the same pass rate \(p\); real batches mix rates (Table 4).
4.2 What a dead group costs
Two different bills arrive. The first is generation. Olmo 3's 32B reasoner used 8 H100 nodes for training and 20 for inference; its learner waits for data 75% of the time, about five times as many GPUs serve inference as training, and in one representative run a step averaged 1,000 s of which 125 s was training10 (§4.4.3, footnote 41). A dead group is paid for entirely in that dominant phase and returns nothing. The second is statistical. Under batch-level token averaging the dead group's tokens stay in the denominator \(N\), so the gradient is the informative groups' sum divided by a count that includes the dead ones: the nominal batch of 512 prompts becomes an effective batch of roughly \(512\times(1-\text{dead fraction})\), and gradient noise grows as one over the effective count (derived). Adam, which divides by a running scale of the gradient, largely undoes the smaller magnitude that DAPO points to (derived), so what survives is the noise.
4.3 Four remedies
| Remedy | Mechanism | Extra rollouts | Reported effect | Source |
|---|---|---|---|---|
| Dynamic sampling | Oversample, drop groups with accuracy 0 or 1, repeat until the buffer holds N informative groups | Yes, about 2 to 3 times the prompts late in training (Table 4) | Last step of the ladder, 42 to 50 on AIME24 avg@32; reaches the same score in fewer steps and, the authors say, not much more wall-clock time | DAPO §3.2, Alg. 1, Fig. 63 |
| Zero-gradient filtering with active sampling | Filter dead groups; the asynchronous actors keep pulling completions and prompts until the batch is full of non-zero-advantage samples | Yes, but continuous batching leaves no idle slots | Keeps the batch size from decaying; "greatly reduced loss variance" in an RL-Zero math run | Olmo 3 §4.4.1, §4.4.3, Fig. 2610 |
| Zero-variance filtering | Drop dead groups from the loss; the batch simply shrinks | No | Higher asymptote in the forward test; LOO without it: A 0.595 against 0.610, B 1.69 against 2.01 | ScaleRL §3.2, Fig. 6a, Fig. 79 |
| No-Positive-Resampling | Remove a prompt for good once its historical pass rate reaches 0.9 | Fewer: dead groups are never generated | Higher asymptote in the forward test; LOO without it: A 0.605, B 1.62, the lowest B of the nine variants | ScaleRL §3.2, Fig. 6b, Fig. 79 |
The two filters are different in kind. Dynamic sampling and active sampling keep the number of informative prompts constant and pay in generation. Zero-variance filtering accepts a smaller effective batch. ScaleRL notes the distinction (filtering "merely drop[s] the prompts" while DAPO "resamples more prompts until the batch is full") and its DAPO baseline filters rather than refills, because refilling stalled generators in its pre-assigned PipelineRL layout9 (§3.2, App. A.16). Hugging Face TRL's GRPOTrainer implements none of these; it logs frac_reward_zero_std, the fraction of samples whose group has zero reward spread, and leaves the choice to the user45. A zero-variance filter keyed on the total reward fires only when the whole reward is constant, which Section 6 shows is a weaker condition than "accuracy is constant".
5. Truncated samples: overlong filtering and soft punishment
A generation cap \(L_{\max}\) makes an unfinished chain of thought score as wrong; DAPO's default assigns truncated samples a punitive reward. DAPO's objection is that this is noise: "a sound reasoning process can be penalized solely due to its excessive length"3 (§3.4). The loss aggregation makes it worse (derived). Truncated samples are the longest samples in the batch. Under sample-level averaging each of their tokens gets the smallest weight in the group; under token-level averaging they hold the largest share, as the 7,800-token response holds 34.5% of the weight in Table 2. A scheme that fixes the length bias therefore sends most of the negative push to the samples most likely to be unfinished rather than wrong, which is why DAPO ships the fix together with truncation handling.
DAPO has two remedies. Overlong filtering masks the loss of truncated samples. Soft overlong punishment keeps them but adds a graded length reward to the correctness reward3 (§3.4, Eq. 13):
$$R_{\text{length}}(y)=\begin{cases}0,&|y|\le L_{\max}-L_{\text{cache}}\\[2pt] \dfrac{(L_{\max}-L_{\text{cache}})-|y|}{L_{\text{cache}}},&L_{\max}-L_{\text{cache}}<|y|\le L_{\max}\\[6pt] -1,&L_{\max}<|y|.\end{cases}$$The paper sets an expected maximum of 16,384 tokens and a punishment cache of 4,096, so the hard cap is \(L_{\max}=16{,}384+4{,}096=20{,}480\)3 (§4.1). A response of 18,432 tokens, halfway through the ramp, receives \((16{,}384-18{,}432)/4{,}096=-0.5\), so a correct answer there scores 0.5 where a correct answer under 16,384 tokens scores 1 and a wrong one scores 0 (derived). TRL ships the same function as get_soft_overlong_punishment(max_completion_len, soft_punish_cache) and a mask_truncated_completions option for the filtering variant, default False54.
The evidence is mixed. In DAPO's ladder, overlong filtering is the first and, after dynamic sampling, the second largest step (30 to 36), and soft punishment adds three more (38 to 41)3 (Table 1). Olmo 3 tried "additional changes (e.g., overlong filtering)" and "did not find these gave consistent improvements"10 (footnote 34). ScaleRL compared two ways to bound length under its recipe, forced interruption (inserting an end-of-thinking phrase at a random point between 10,000 and 12,000 tokens) and a DAPO-style penalty applied to correct traces only, and found that replacing interruption with the penalty "does not improve performance"; the penalty's leave-one-out run has \(B=1.75\) against 2.01 for the full recipe at an unchanged \(A=0.610\)9 (App. A.10, Fig. 7). ScaleRL also treats truncation rate as a health signal: at batch size 768, truncation of 10% to 15% "typically destabilized training", while its own runs stayed below 5% for over 90% of training9 (App. A.15). DeepSeek-V3.2 uses a length penalty inside the reward for reasoning and agent tasks and trains its Speciale variant "with a reduced length penalty"8 (§3), so length control by reward is alive at the frontier next to the loss-side fixes of Section 2.
6. Advantage without the standard deviation
DeepSeek-V3.2 sets the advantage to "subtracting the average reward of the group from the reward of output", \(\hat A_{i,t}=R_i-\mathrm{mean}(\boldsymbol R)\), with no division8 (§3.1). The objective printed beside it still averages with \(1/|o_i|\) and carries the KL term, as a review of GRPO, and the report does not say whether its implementation aggregates at sample or token level or why the standard deviation was dropped8. Olmo 3 states its reason: "no standard deviation normalization [...] removes a difficulty bias", following Dr. GRPO, alongside a token-level loss normalized over the batch's tokens and no KL loss10 (§4.4.1). Two consequences follow, both derived here.
Update size now follows the reward scale. TRL's documentation says so directly: turning off std scaling "removes variance normalization, so update magnitudes depend directly on the raw reward scale and batch composition"4. For a binary reward the centered advantage has standard deviation \(\sqrt{\hat p(1-\hat p)}\le 0.5\), so a balanced prompt's advantages are at most half the size they would be after division (a quarter of the energy \(\sum_iA_i^2\)), and those of extreme prompts are far smaller (Figure 3). A global factor like that is absorbed by Adam; the relative scale between prompts and between reward components is what remains, and with reward terms of different magnitude that is a design decision rather than a side effect.
Composite rewards change what "dead" means. Suppose the reward is accuracy (0 or 1) plus \(0.1\,c\), where \(c\in[0,1]\) is a continuous auxiliary term such as a language-consistency fraction, and take an all-correct group whose \(c\) values have a standard deviation of 0.015. The reward spread is 0.0015. Centered, the advantages are about \(\pm 0.0015\), roughly 300 times smaller than the \(\pm0.5\) of a balanced accuracy group. Divided by the group standard deviation they become unit-variance and the jitter in a 10%-weight bonus drives the gradient of a prompt the verifier calls solved with the full strength of an informative one (illustrative numbers; Module 10, §2.4 works the language-consistency reward in detail). The same composite reward also defeats a filter keyed on total reward: groups are "all identical" only when every component ties, so dead-in-accuracy groups pass through Olmo 3's filter ("rewards are all identical")10 or TRL's frac_reward_zero_std counter5. DAPO's constraint is written on accuracy, \(0<|\{o_i:\text{is\_equivalent}(a,o_i)\}|\lt G\)3 (Eq. 8), which ignores the auxiliary term by construction. Which condition to filter on is a choice that interacts with whether you divide by the standard deviation.
7. What the controlled comparisons show
7.1 DAPO's ladder
DAPO's headline is 50 AIME 2024 points on Qwen2.5-32B against 30 for "naive GRPO" on the same base and 47 for DeepSeek-R1-Zero-Qwen-32B3 (Fig. 1, Table 1). Figure 6 plots the ladder. Of its five rungs, two concern this module (token-level loss, dynamic sampling), two are truncation fixes (Section 5) and one belongs to Module 12 (Clip-Higher).
Three limits apply before using the ladder to rank techniques. It is cumulative, so each gain is conditional on the rows above it and would differ in another order; dynamic sampling's +8 comes last, when more than half the prompts are dead (Section 4.1), which is when it has the most to recover (derived reading). It is one base model and one benchmark: AIME 2024 has 30 problems, and avg@32 removes most sampling noise but not problem-set noise. Even so, sampling alone leaves a standard error of up to \(\sqrt{30\times0.25/32}/30=1.6\) points when each problem's accuracy is near one half (derived), so the +1 of the token-level loss and the +2 of Clip-Higher cannot be separated from zero without repeated runs, and the table reports one number per row. The authors themselves describe the token-level loss as bringing "less performance improvement" but better stability3 (§4.2).
7.2 ScaleRL's two tests
ScaleRL (Khatri et al.) is the only study in the source set that varies loss aggregation and advantage normalization in a controlled way and fits a compute curve to each: pass rate \(R\) as a saturating function of GPU-hours with an asymptote \(A\) and an efficiency exponent \(B\)9 (Eq. 1). Its forward ablations start from a PipelineRL baseline on an 8B dense model, with 48 prompts of 16 generations (batch 768) from the Polaris-53K math set. Loss aggregation: prompt average and token average both reach \(A=0.535\) (\(B=2.98\) and \(2.04\)), sample average \(A=0.520\) (\(B=4.55\)). Advantage normalization: none \(A=0.525\) (\(B=3.69\)), batch-level 0.530 (3.15), prompt-level 0.520 (4.55)9 (App. A.9, Fig. 14; values read from the printed legends). The paper adopts prompt average because it "achieves the highest asymptotic performance", and batch-level normalization because it is "theoretically sound and marginally better" while "all three methods are observed to yield similar performance"9 (§3.2).
The spreads are 0.015 and 0.010 in \(A\), smaller than the \(\pm0.02\) seen across three independent runs of the final recipe9 (Fig. 8a). \(B\) cannot be compared across runs with different \(A\), because the two fitted parameters trade off: sample average and prompt-level normalization both fit \(B=4.55\) at the lowest \(A\). The paper therefore pins \(A=0.605\) in its leave-one-out table (Figure 7), where each choice is reverted from the full recipe at 16,000 GPU-hours per run.
Every reverted choice here lands within 0.02 of the full recipe's \(A=0.610\); the largest gap, prompt-level standard-deviation division at 0.590, sits on the noise margin. Pinned-\(A\) efficiency falls from 2.01 to 1.84 (sample-average loss), 1.82 (prompt-level division), 1.69 (no zero-variance filtering) and 1.62 (uniform prompt sampling), that is by 8%, 9%, 16% and 19% (derived), though the paper gives no margin for \(B\) and does not translate the differences into GPU-hours. This is the evidence behind the claim that these choices "primarily modulate compute efficiency without materially shifting the asymptote"9 (abstract). Two reservations. The asymptote is a fitted parameter extrapolated from runs of at most 16,000 GPU-hours (fits up to 8,000), and the 100,000 GPU-hour validation run is the full recipe only. And the same study finds that the loss type (CISPO or GSPO against DAPO) and FP32 logits do move the fitted asymptote in the forward tests, from 0.52 to 0.61 for the latter9 (§3.2, Fig. 5), so "mostly efficiency" describes aggregation and normalization specifically (Modules 12 and 14 cover the others). One model family, one data domain, one benchmark protocol.
7.3 The ledger
| Evidence | Setting | Contrast | Result | Limit |
|---|---|---|---|---|
| DAPO Table 13 | Qwen2.5-32B base, DAPO-Math-17K, AIME24 avg@32 | six cumulative rows | 30, 36, 38, 41, 42, 50 | one run per row, order-dependent, 30 problems; sampling s.e. up to 1.6 (derived) |
| Dr. GRPO Figs. 8 and 92 | Qwen2.5-1.5B, 3K questions, 8 samples, 3,000 tokens | GRPO, without length norm, without std norm, Dr. GRPO | both single removals beat vanilla; length term drives length; Dr. GRPO better across 3 seeds | small model, short cap, curves rather than a table |
| Dr. GRPO Table 42 | Qwen2.5-Math-7B, MATH levels 3 to 5, greedy, 3,000-token budget | Oat-Zero-7B against other 7B recipes | AIME24 43.3 (13 of 30 problems), five-benchmark average 51.4 | different data, templates and frameworks across rows: not a loss ablation |
| ScaleRL Fig. 149 | 8B dense, math, batch 768, PipelineRL baseline | 3 aggregations, 3 advantage scalings | A spread 0.015 and 0.010 | inside the 0.02 run-to-run margin; B not comparable across A |
| ScaleRL Fig. 79 | same, 16,000 GPU-hours per run | revert one choice from the full recipe | A 0.590 to 0.610; pinned-A B 1.62 to 2.01 | one run per variant; no margin on B; fitted asymptote extrapolated |
| Olmo 3 Fig. 26, footnote 3410 | RL-Zero math, 7B | active sampling on and off; overlong filtering tried | active sampling keeps the batch full and lowers loss variance; overlong filtering no consistent gain | figure and one sentence; no table |
| MiniMax-M1 §513 | M1 at 40K to 80K output length | token-level loss with GRPO advantage normalization at long length | negative samples lengthen faster, "disproportionately large negative gradients" in sequence tails | narrative; no ablation numbers for the fix |
The last row is the counterweight to Section 2. MiniMax-M1 scaled generation from 40K to 80K tokens and observed pattern collapse late in each length window, with the tails of generations turning into incoherent text and perplexity rising. Its diagnosis: negative samples "increase in length substantially faster than positive samples", so "disproportionately large negative gradients accumulate in the latter segments of generation sequences", an imbalance it traces to "the inherently unequal nature of GRPO's advantage normalization and the token-level loss we adopt"13 (§5). That is the 4.19-fold negative-to-positive imbalance of Table 2, seen in a production run. Its remedies were early stopping on repetitive patterns, "combined sample-level loss and token-level normalization", and lower gradient-clipping and \(\epsilon_{\text{high}}\) thresholds13. The report does not publish an ablation of any of the three.
8. The choices side by side
| Recipe | Loss aggregation | Advantage scale | KL | Dead groups | G | Source |
|---|---|---|---|---|---|---|
| DeepSeekMath-RL (original GRPO) | sample-level | group std | in the loss, 0.04 | not discussed | 64 | §4.1 to §4.21 |
| DAPO | prompt-level token mean (Eq. 8) | group std (Eq. 9) | none | dynamic sampling on accuracy | 16 | §2.3, §3, §4.13 |
| Dr. GRPO | constant (generation budget, 3,000 tokens in its runs) | none | none | not discussed | 8 | §3.2, Table 62 |
| DeepSeek-V3.2 | sample-level as printed; implementation not stated | \(R_i-\mathrm{mean}(R)\) | unbiased K3 estimate; weak or none for math | not stated | not stated | §3.18 |
| OlmoRL | batch-level token mean | none | none | zero-gradient filtering, active sampling | 8 | §4.4.1, Table 4910 |
| ScaleRL | prompt-level averaging, CISPO loss | batch std of the advantages | none in the printed loss | zero-variance filtering, no-positive-resampling | 16 | §3.2, §49 |
| GLM-5 reasoning RL | sample-level (Eq. 1) | group std | removed | not stated | 32 | §3.27 |
| MiniMax-M1 | token-level (Eq. 4); sample-plus-token hybrid at 80K | group std (Eq. 2) | none | dynamic sampling from DAPO | not stated | §3.1, §513 |
| TRL defaults | dapo: token mean over the global accumulated batch | group: Bessel std plus 1e-4 | 0.0 | none; logs frac_reward_zero_std | 8 | config and trainer source54 |
No two rows are the same recipe. The three with controlled experiments behind them (DAPO, Olmo 3, ScaleRL) share a direction: aggregate by tokens or by prompt rather than by response, handle dead groups explicitly, and drop the KL term for verifiable rewards. They disagree on the standard deviation: group-level in DAPO, absent in Olmo 3, batch-level in ScaleRL. The library default is a mixture, DAPO's loss with GRPO's group division, a combination whose std half Dr. GRPO argues against and Olmo 3 and V3.2 do not use.
Where experts disagree
The question is how the GRPO loss and advantage should be normalized. Everyone who has measured it agrees that original GRPO's per-response length average is unsatisfactory in long-chain-of-thought runs. They disagree on the replacement, on whether the standard deviation belongs in the advantage, and on whether token-level weighting is safe when responses reach tens of thousands of tokens.
A. Remove both normalizers
Dr. GRPO2; Olmo 3 (token-level loss over the batch, no std)10; DeepSeek-V3.2 on the advantage8; TRL's documentation recommends no scaling on the Dr. GRPO authority4
Dividing by \(|o_i|\) changes the objective inside each prompt and dividing by the standard deviation changes the weight of each prompt. Take both out and the estimator is the plain Monte Carlo policy gradient with a leave-one-out-equivalent baseline, equal to RLOO up to a factor \(G/(G-1)\)2 (App. A).
- Unbiased by construction; fewer knobs; the length term moves length and the std term does not (Dr. GRPO ablation).
- Adopted independently by a fully open pipeline (Olmo 3) and a frontier report (V3.2, advantage only).
B. Token-level loss fixes length; keep the group std
DAPO3; TRL's defaults5; MiniMax-M1's CISPO loss13
Sample-level averaging underweights long samples and cannot penalize gibberish in them; a token-level loss fixes that and leaves the advantage as in GRPO, with dynamic sampling removing dead groups.
- The best-documented full recipe on a hard benchmark: 30 to 50 AIME24 points on one base model.
- Keeps every non-dead prompt at unit advantage scale.
- Eq. 8's per-prompt denominator brings back a prompt-to-prompt weighting; the std division is the difficulty bias of Section 3.
- The token-level gain is +1 on the ladder, inside sampling noise.
C. These are efficiency knobs; look elsewhere for the asymptote
ScaleRL (Khatri et al.)9
Under fitted compute curves, loss aggregation, advantage normalization, curriculum and off-policy algorithm "primarily modulate compute efficiency without materially shifting the asymptote"; use prompt-level averaging and batch-level normalization and spend effort on loss type and numerics.
- The only study that fits curves, reports a noise margin and reverts one choice at a time at 16,000 GPU-hours.
- Consistent with the small A spreads in Table 6.
- One 8B dense model, one math domain, one protocol; the asymptote is extrapolated.
- An 8% to 19% change in the efficiency exponent is not nothing, and B has no stated margin.
D. Token-level weighting is unsafe at extreme length; keep sample-level or a hybrid
MiniMax-M1 at 80K tokens13; GLM-5 reasoning RL keeps \(1/|y_i|\) and group std7; V3.2's printed objective8
When negative samples are far longer than positive ones, token-level weighting lets long, degenerate tails dominate the negative gradient; sample-level averaging keeps positive and negative mass balanced (1.0 against 4.19 in Table 2).
- Observed failure mode in a real 80K run, with a stated mechanism.
- Balanced push per response regardless of length.
- Reintroduces the length preference of Section 2.1 and its accuracy-dependent drift.
- No published ablation for M1's hybrid or GLM-5's choice; V3.2's implementation is unstated.
On the length term, A and B are right and the toy of Section 2.2 shows why: the 1/|o| factor changes each prompt's optimum with a sign that depends on accuracy, and every controlled comparison that touched it points the same way. D is right about a different problem. The token-level estimator is unbiased, but unbiasedness does not make a very long repetition loop a good thing to weight in proportion to its length. At 80K the remedy M1 reports for that (repetition detection and early stopping, with truncation handling) addresses the cause; reintroducing \(1/|o|\) trades one known bias for another, and no one has measured that trade. On the standard deviation, the evidence is thin and says second-order. Use a batch-level scale or none; if you keep the group division, filter dead groups on accuracy and look at the spread of each reward component before trusting the normalized advantage (Section 6). C is the best-calibrated reading of the data, with the scope warning that it is one model family. The experiment that would settle this is a matched grid of \(\{\)sample, prompt, token, constant\(\}\times\{\)group, batch, none\(\}\) with at least three seeds and error bars at both a short and a very long generation cap, logging the wrong-to-correct length ratio and the dead-group fraction. It does not exist yet.
A second, smaller disagreement is how to control response length. Dr. GRPO argues the unbiased loss itself curbs overthinking2; DAPO, V3.2 and others add a length penalty38; ScaleRL prefers forced interruption9. No source reverts the loss fix and the penalty jointly (derived from the evidence ledger), so the defensible order is to fix the loss first, since it costs nothing, and enforce a length budget with a penalty or interruption separately.
What is changing
Hugging Face TRL's loss_type now lists variants beyond the papers read here, among them luspo, "Length-Unbiased Sequence Policy Optimization", a sequence-level loss that scales each sequence's loss by its length and modifies GSPO, and vespo and sapo (pointer only: described in the TRL configuration and documentation, not read at the paper level)54. That a length-unbiased loss was written for sequence-level ratios shows the length question returns whenever a method averages at the response level (Module 12 covers GSPO and its relatives). TRL also offers multi_objective_aggregation="normalize_then_sum", which normalizes each reward component within the group before summing, aimed at the composite-reward problem of Section 65. Open as of 2026-10-27: whether the aggregation results hold at 32B and above and for mixture-of-experts policies, since the ScaleRL ablations are dense 8B, and which aggregation DeepSeek-V3.2 actually runs.
References
- paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the GRPO objective (§4.1.1, Eq. 3) and the outcome-supervision advantage written with std(r) (§4.1.2); DeepSeekMath-RL settings (§4.2: KL 0.04, 64 outputs per question, one policy update per exploration stage).
- paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the two biases (§3.1), Dr. GRPO and Listing 1 (§3.2), Table 2 (PPO implementations), Figs. 5 and 7, Table 4 (App. B), Figs. 8 and 9 (App. C), RLOO equivalence (App. A), hyperparameters (App. G, Table 6).
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. (ByteDance Seed, Tsinghua AIR). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: Eqs. 8, 9 and 13; dynamic sampling and Algorithm 1 (§3.2); token-level loss (§3.3); overlong reward shaping (§3.4); training details (§4.1); Table 1 ladder (§4.2); Fig. 3b read off the rendered plot.
- officialHugging Face. TRL documentation: GRPO Trainer (main branch; no version stated on the page). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the loss-type formulas and the DAPO and Dr. GRPO remarks ("Loss Types"), the scale_rewards note on raw reward scale, the frac_reward_zero_std metric, defaults.
- repoHugging Face. TRL source: trl/trainer/grpo_trainer.py (last changed in commit 63d9074e, 2026-10-21), grpo_config.py, utils.py (nanstd) and rewards/other_rewards.py, main branch. https://github.com/huggingface/trl/tree/main/trl. Accessed 2026-10-27. Used for: per-loss-type normalizers in compute_loss, nanstd with Bessel’s correction and the +1e-4 epsilon in advantage scaling, the scale_rewards="batch" computation, get_soft_overlong_punishment, mask_truncated_completions default, loss_type and multi_objective_aggregation options. Files fetched from raw.githubusercontent.com/huggingface/trl/main/.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: the trade-off of removing std normalization (pp. 73–74) and the three loss-aggregation strategies (§6.3.2).
- paperGLM-5 Team (Zeng, A., Lv, X. et al.; Zhipu AI and Tsinghua University). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the reasoning-RL loss with 1/|y_i|, group-std advantage, KL removed, group size 32 (§3.2, Eq. 1).
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: Mixed RL rewards and the Speciale variant (§3); the printed GRPO objective and the advantage \(R_i-\mathrm{mean}(R)\) (§3.1, Eqs. 5 and 6).
- paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R. et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: loss-aggregation and advantage-normalization ablations (§3.2, Fig. 14), the ScaleRL recipe and leave-one-out table (§4, Figs. 7 and 8), length control (App. A.10), truncation rates (App. A.15), recipe descriptions (App. A.2, A.16). Values in Figs. 7 and 14 read from the rendered PDF.
- paperTeam Olmo (Ettinger, A., Bertsch, A. et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL algorithm details and footnote 34 (§4.4.1), active sampling and rollout cost (§4.4.3, footnote 41), Table 49 hyperparameters, Fig. 26 (RL-Zero active sampling).
- paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (expanded Nature version), 4 Jan 2026. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the group-normalized advantage (Eq. 3) and the thinking-time reading of the response-length curve (Fig. 1b).
- paperShao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., Du, S. S., Lambert, N., Min, S., Krishna, R., Tsvetkov, Y., Hajishirzi, H., Koh, P. W., Zettlemoyer, L. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947v2, 25 Feb 2026. https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: the random-reward result (abstract: +21.4 against +29.1 points on MATH-500) and its clipping-bias explanation.
- paperMiniMax (Chen, A., Li, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: the CISPO objective with token-level loss and group-std advantage (§3.1, Eqs. 2–4); the 80K length-scaling instability and its remedies (§5).