Reinforcement Learning for LLM Post-Training · Module 9
Critic-Free Policy Gradient: REINFORCE with Group Baselines, RLOO, GRPO, and Kimi's Mirror-Descent Objective
PPO's value model is a second policy-sized network that must predict a final reward from a half-written answer. This module covers what replaced it: baselines built from several samples of the same prompt (RLOO's leave-one-out mean, GRPO's group mean and standard deviation, Kimi k1.5's sampled mean inside a mirror-descent loss), the DeepSeekMath settings that introduced GRPO, and the controlled experiments for and against dropping the critic. Afterwards you can write GRPO and RLOO from the equations, say what each assumes about the reward and the sampling budget, and read the critic-versus-no-critic literature without mistaking a tuned baseline for a verdict.
1. What the critic bought, and what replaces it
PPO-RLHF (Module 5) trains a value model \(V_\psi(s_t)\) and feeds it to GAE to get a per-token advantage \(A_t\). The value model has one job: it is the baseline, a prefix-dependent estimate of the final reward that is subtracted to cut variance (Module 2). What it buys is token-level credit and, at GAE \(\lambda\lt 1\), bootstrapped variance reduction at the price of bias (Module 3). Every method in this module declines to pay for it, and the reasons are different enough that they should be kept apart.
1.1 Five arguments against a per-token value model
- Memory and compute. The value function "is typically another model of comparable size as the policy model", which "brings a substantial memory and computational burden"2 (§4.1.1). ReMax measured it for a 7B model: the value model and its training state use 46% of GPU memory, the reward model 4%1 (footnote 1). Module 1 (Table 6) does the model-state arithmetic.
- Last-token reward. "Usually only the last token is assigned a reward score by the reward model, which may complicate the training of a value function that is accurate at each token"2 (§4.1.1). For long chains of thought, R1's appendix adds that reflection and revision mean "the content initially generated may later be revised or contradicted, which makes it even less feasible to predict the final reward based on a partial response"5 (Supp. A.3).
- Variance is already low. Ahmadian et al. argue that a pretrained, prompt-conditioned policy concentrates probability on few tokens, so bootstrapping's bias is "not worth it"3 (§3.1). On their Llama SFT model, about 60% of the mass sits on the single most probable token at each step and over 90% on the top 163 (App. A).
- One action per completion. Rewards "are only attributed to full generations", so the completion can be modeled as a single action3 (§1); Module 2 §1 draws the two views.
- Credit assignment can punish exploration. Kimi's example: a reasoning step containing an error that the model later repairs would get a negative advantage from an oracle value function, yet "exploring [it] is extremely valuable for training the model to generate long CoT"4 (§2.3.2).
These are claims about cost and learnability. None says a perfect value function would hurt; Section 8 reads the experiments that test whether a practical one helps.
1.2 Paying for a baseline in samples instead of parameters
| Method | Baseline for sample \(i\) | Extra samples per prompt | Extra trained model | Credit granularity |
|---|---|---|---|---|
| PPO67 | \(V_\psi(s_t)\), through GAE | 0 | value model | per token |
| REINFORCE, moving average3 | mean reward over all steps so far | 0 | none | per sequence |
| ReMax1 | reward of the greedy decode | 1 | none | per sequence |
| RLOO3 | mean reward of the other \(k-1\) samples | \(k-1\) | none | per sequence |
| GRPO2 | group mean (including \(i\)), result divided by group std | \(G-1\) | none | per sequence; per step with a process reward |
| Kimi k1.54 | empirical mean of \(k\) sampled rewards | \(k-1\) | none | per sequence |
| REINFORCE++ with baseline8 | group mean, then divided by the global-batch std | \(G-1\) | none | per sequence |
| VinePPO9 | mean return of \(K=9\) rollouts (default) from each step's state | \(K\) per step state | none | per step |
The price is not extra decoding, it is the shape of the batch. At a fixed budget of \(N\) sequences per step, a group method trains on \(N/G\) distinct prompts instead of \(N\) (derived). Module 1 reports that ScaleRL found this trade neutral over the range it tested, and VAPO's ablation found fewer prompts with more repetitions marginally better even with a critic10 (§4.3). What you do give up is per-token credit: every token of a completion inherits one scalar. The last row of Table 1, and GRPO's process-supervised variant, are the attempts to get credit back without a value network.
2. REINFORCE and RLOO: the leave-one-out baseline
2.1 The estimator
Draw \(k\) completions \(y_{(1)},\dots,y_{(k)}\) from \(\pi_\theta(\cdot\mid x)\) and score each with a sequence-level reward \(R\). RLOO treats each sample in turn as the one being credited and uses the other \(k-1\) as its baseline:
$$\hat g=\frac1k\sum_{i=1}^{k}\Big[R(y_{(i)},x)-\frac1{k-1}\sum_{j\neq i}R(y_{(j)},x)\Big]\nabla_\theta\log\pi_\theta(y_{(i)}\mid x).$$Ahmadian et al. call the baseline "akin to a parameter-free value-function, but estimated at each training step", and credit the estimator to Kool et al. (2019)3 (§2.3). It is unbiased because \(b_i\) does not depend on \(y_{(i)}\); Module 2 §4.2 proves it and shows what happens when the sample is included in its own baseline. The two forms differ by a constant, \(R_i-b_i=\tfrac{k}{k-1}(R_i-\bar R)\)11 (Eq. 51), and Lambert shows that Dr. GRPO's advantage equals RLOO's up to that factor11 (§6.2.8, Eq. 62). Table 3 checks the factor numerically.
Two structural choices separate RLOO from GRPO. The KL penalty is folded into the sequence reward, \(R=r_\phi-\beta\log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}\)3 (Eq. 3), so it passes through the baseline and is broadcast to every token11 (§6.2.4); GRPO puts it in the loss (Module 6). And there is no importance ratio or clip in the estimator11 (§6.2.11), so it is a plain on-policy gradient.
2.2 What the paper measured
Setup: Pythia-6.9B on TL;DR Summarize and Anthropic-HH, Llama-7B on HH; the reward model and policy start from the SFT checkpoint; constant learning rate \(10^{-6}\) with 3% warm-up; \(\beta=0.03\) on TL;DR and 0.10 on HH; for the Pythia runs a rollout batch of 512 and a step batch of 256, and "for all algorithms, we take 2 gradient steps for each batch"3 (App. C). Win-rates are GPT-4 judgments against reference completions, at the checkpoint with the highest test reward3 (§4.2, Table 1).
| Method | TL;DR, Pythia-6.9B | HH, Pythia-6.9B | HH, Llama-7B | Length, HH Llama (tokens) |
|---|---|---|---|---|
| RLOO, \(k=4\) | 77.9 | 43.7 | 64.1 | 60.6 |
| RAFT, \(k=4\) | 73.2 | 42.1 | 63.3 | 62.4 |
| RLOO, \(k=2\) | 74.2 | 47.6 | 62.2 | 58.6 |
| RAFT, \(k=2\) | 72.1 | 37.7 | 58.4 | 52.8 |
| REINFORCE with baseline | 70.7 | 37.9 | 55.3 | 47.2 |
| Vanilla policy gradient | 70.4 | 36.4 | 52.3 | 39.1 |
| PPO | 67.6 | 29.2 | 32.0 | 16.5 |
| DPO | 66.6 | 39.0 | 61.9 | 104.4 |
Four readings. First, REINFORCE with a moving-average baseline already beats PPO by 3.1, 8.7 and 23.3 points (derived from Table 2). The paper's introduction states "3.2% to 20.3%"; I reproduce the upper end as vanilla policy gradient against PPO on HH-Llama (52.3 − 32.0), not the lower end exactly, so quote the table, not the range. Second, RLOO at \(k=4\) beats PPO by 10.3, 14.5 and 32.1 points, as the text says (derived check: 77.9 − 67.6, 43.7 − 29.2, 64.1 − 32.0). Third, RLOO beats RAFT at equal \(k\), and RLOO at \(k=2\) matches or beats RAFT at \(k=4\) on two of three columns (1.1 points behind on HH-Llama): using all samples through a baseline beats keeping only the best one. Fourth, RAFT degrades faster than RLOO as the KL weight rises and as Gaussian noise (\(\sigma\) of 1, 3 and 5) is added to the reward, because RAFT trains only on the top-ranked sample and anything that scrambles the ranking hurts it3 (§5.2.2, Figures 5-6).
2.3 Why PPO lost in this setup
The paper takes PPO apart. Sweeping the GAE parameter over \(\lambda\in\{0,0.5,0.95,1.0\}\) gave "a monotonically decreasing reward with decreasing \(\lambda\)", so the best PPO is the one where the critic serves only as a baseline, with no bootstrapping3 (§3.1, Figure 1). That is the estimator GAE(\(\gamma,1\)): the empirical return minus a value baseline, unbiased "regardless of the accuracy of \(V\)"7 (§3, Eq. 18). The loss was clipped "on average < 5% of the time per batch", and turning off clipping and then the ratio slightly helped3 (§3.2).
3. GRPO as DeepSeekMath defines it
3.1 The objective
For each question \(q\), sample a group \(\{o_1,\dots,o_G\}\) from \(\pi_{old}\) and maximize2 (§4.1.1, Eq. 3)
$$\mathcal J_{GRPO}(\theta)=\mathbb E\Big[\frac1G\sum_{i=1}^{G}\frac1{|o_i|}\sum_{t=1}^{|o_i|}\Big(\min\big(\rho_{i,t}\hat A_{i,t},\ \mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\hat A_{i,t}\big)-\beta\,\mathbb D_{KL}\big[\pi_\theta\,\Vert\,\pi_{ref}\big]_{i,t}\Big)\Big],\qquad \rho_{i,t}=\frac{\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})}{\pi_{old}(o_{i,t}\mid q,o_{i,\lt t})}.$$The per-token KL estimator is \(\frac{\pi_{ref}}{\pi_\theta}-\log\frac{\pi_{ref}}{\pi_\theta}-1\), "guaranteed to be positive"2 (Eq. 4); it is Schulman's \(k_3\)12, whose value and gradient Module 6 separates. With outcome supervision every token of \(o_i\) receives the same advantage, the normalized reward2 (§4.1.2):
$$\hat A_{i,t}=\tilde r_i=\frac{r_i-\operatorname{mean}(\mathbf r)}{\operatorname{std}(\mathbf r)}.$$With process supervision, a process reward model scores the end of each reasoning step, all step rewards in the group are normalized by the same mean and standard deviation, and a token's advantage is the sum of the normalized rewards of the steps that end at or after it, \(\hat A_{i,t}=\sum_{index(j)\ge t}\tilde r_i^{index(j)}\)2 (§4.1.3). That is a return without discounting, which gives dense credit without a value network. The authors' stated reason for the group form is that it "aligns well with the comparative nature of rewards models", which are trained on comparisons between outputs for the same question2 (§4.1.1).
3.2 What changes relative to PPO
Four things differ. The advantage comes from the group, not from \(V_\psi\) and GAE. The KL term sits in the loss, so it does not enter the advantage2 (§4.1.1); R1's appendix argues that PPO's per-token KL reward penalizes the cumulative KL and "may implicitly penalize the length of the response", which GRPO's placement avoids5 (Supp. A.3). The batch has \(G\) completions per prompt, 64 in DeepSeekMath2 (§4.2). And the clipped ratio, the reference model and the reward source stay. Resident models drop from four to three; with \(\beta=0\) and a verifier, the reference and the reward model both go (Module 1, Table 6), which is why TRL defaults to \(\beta=0\), "reducing memory usage and improving training speed"13.
3.3 The gradient when there is one update per rollout
DeepSeekMath's RL stage makes "a single update following each exploration stage"2 (§4.2), so at the moment of the gradient \(\pi_\theta=\pi_{old}\). Then \(\rho_{i,t}=1\) sits inside \((1-\varepsilon,1+\varepsilon)\), the \(\min\) selects the unclipped branch, and \(\nabla\rho=\rho\nabla\log\pi_\theta=\nabla\log\pi_\theta\). For the KL term write \(\varrho=\pi_{ref}/\pi_\theta\), so \(\nabla\varrho=-\varrho\,\nabla\log\pi_\theta\) and \(\nabla k_3=\nabla(\varrho-\log\varrho-1)=(1-\varrho)\nabla\log\pi_\theta\) (derived). Ascending \(\mathcal J\) therefore follows
$$\nabla_\theta\mathcal J=\mathbb E\Big[\frac1G\sum_{i=1}^{G}\frac1{|o_i|}\sum_{t=1}^{|o_i|}\Big(\hat A_{i,t}+\beta\Big(\frac{\pi_{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)}-1\Big)\Big)\nabla_\theta\log\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})\Big],$$which is DeepSeekMath's Eq. 20, with "gradient coefficient" \(GC=\hat A_{i,t}+\beta(\pi_{ref}/\pi_\theta-1)\) in Eq. 212 (App. A.1.6; the derivation there also assumes \(\pi_{\theta_{old}}=\pi_\theta\)). At one update per rollout, GRPO is REINFORCE with group-normalized, length-normalized sequence weights plus a KL pull toward the reference; Module 2 (Table 2) lists the row next to SFT, RFT, PPO and DPO. The clip only becomes active when updates are reused: R1 splits an 8,192-output rollout into 16 minibatches of 512 and trains one inner epoch5 (§2.1), so minibatch 1 is on-policy and minibatches 2 to 16 see parameters that have moved 1 to 15 optimizer steps (derived). Module 12 covers what the ratio and clip do there. Module 6 (§3.3) shows that, on-policy, the expected gradient of this \(k_3\) term is the gradient of \(\mathrm{KL}(\pi_{ref}\Vert\pi_\theta)\), the opposite direction from the one the estimator is meant to track.
3.4 One step on one prompt
Figure 3 runs the arithmetic on four completions with continuous reward-model scores, the setting of the original paper; Section 4 covers the binary-reward case.
- Sample. The old policy writes four completions of lengths 6, 9, 5 and 7 tokens for one prompt (illustrative lengths).
- Score. A reward model or verifier returns one scalar per completion: 2.1, −0.4, 0.9, 1.4 (illustrative values).
- Group statistics. Mean 1.000 and sample standard deviation 1.055, computed over these four numbers only.
- Advantage. Each reward becomes a z-score within its group: +1.04, −1.33, −0.09, +0.38. They sum to zero by construction.
- Broadcast. The scalar is copied to every token of its completion; the loss multiplies it by the ratio and clip of Section 3.1 and adds the KL term.
3.5 As code
The loss is short enough to write out in full. This version follows Eq. 3 (per-sequence \(1/|o_i|\) normalization, symmetric clip, \(k_3\) in the loss). The advantage uses the sample standard deviation and adds \(10^{-4}\) to the denominator, which is what TRL does14 (lines 2812-2834 of grpo_trainer.py, with the Bessel correction in nanstd); neither detail appears in the paper.
import torch
def group_advantage(rewards, eps=1e-4):
# rewards: (P, G), G samples for each of P prompts
mean = rewards.mean(dim=1, keepdim=True)
std = rewards.std(dim=1, keepdim=True) # unbiased (n - 1), as in TRL
return (rewards - mean) / (std + eps) # (P, G); zero when all rewards in a group are equal
def grpo_loss(logp, logp_old, logp_ref, adv, mask, eps=0.2, beta=0.04):
# logp, logp_old, logp_ref: (N, T) log-probs of the sampled tokens, N = P * G sequences
# adv: (N,) one scalar per sequence; mask: (N, T) 1.0 on completion tokens
ratio = torch.exp(logp - logp_old) # pi_theta / pi_old
a = adv.unsqueeze(1)
surr = torch.minimum(ratio * a, torch.clamp(ratio, 1 - eps, 1 + eps) * a)
d = logp_ref - logp # log(pi_ref / pi_theta)
kl = torch.exp(d) - d - 1 # k3, >= 0
per_token = surr - beta * kl
per_seq = (per_token * mask).sum(1) / mask.sum(1) # 1/|o_i| sum_t
return -per_seq.mean() # 1/G sum_i, negated for descent
I checked the gradient of grpo_loss against the closed form of Eq. 20 on a toy model (eight sequences of 2 to 6 tokens, vocabulary 5, \(\pi_{old}\) a detached copy of \(\pi_\theta\)): the maximum absolute difference was \(1.9\times10^{-9}\) in fp32 (PyTorch 2.10), and the ratio was exactly 1 everywhere, as Section 3.3 predicts.
4. What the group advantage does to the gradient
4.1 Binary rewards, group of eight
With a 0/1 verifier and \(k\) correct samples out of \(G\), the group pass rate is \(\hat p=k/G\) and every correct sample shares one advantage, every incorrect sample another. Table 3 gives both for the four forms in use. All values are derived from the definitions; the "÷ sample std" column is what TRL computes14.
| \(k\) | \(\hat p\) | R − mean | Leave-one-out | ÷ sample std | ÷ pop. std | \(\sum A^2\) |
|---|---|---|---|---|---|---|
| 1 | 0.125 | +0.875 / −0.125 | +1.000 / −0.143 | +2.475 / −0.354 | +2.646 / −0.378 | 0.875 |
| 2 | 0.250 | +0.750 / −0.250 | +0.857 / −0.286 | +1.620 / −0.540 | +1.732 / −0.577 | 1.500 |
| 4 | 0.500 | +0.500 / −0.500 | +0.571 / −0.571 | +0.935 / −0.935 | +1.000 / −1.000 | 2.000 |
| 6 | 0.750 | +0.250 / −0.750 | +0.286 / −0.857 | +0.540 / −1.620 | +0.577 / −1.732 | 1.500 |
| 7 | 0.875 | +0.125 / −0.875 | +0.143 / −1.000 | +0.354 / −2.475 | +0.378 / −2.646 | 0.875 |
Group advantage under a binary reward
Formulas for \(\hat p=\min(k,G)/G\): R − mean gives \(1-\hat p\) and \(-\hat p\); leave-one-out multiplies by \(G/(G-1)\); dividing by the sample std \(\sqrt{G\hat p(1-\hat p)/(G-1)}\) gives the third form; \(\sum A^2=G\hat p(1-\hat p)\) for R − mean, and exactly \(G-1\) after dividing by the sample std whenever the group is mixed. At \(k=0\) there are no correct samples and at \(k=G\) no wrong ones, so those outputs are vacuous; the group's advantage is zero.
4.2 What the table implies
- Zero sum. Every column sums to zero over the group: the three mean-centered forms by definition, leave-one-out because \(\sum_i(R_i-b_i)=\sum R-\frac{(G-1)\sum R}{G-1}=0\) (derived).
- The factor \(G/(G-1)\). Leave-one-out over R − mean is 1.000/0.875 = 0.857/0.750 = 8/7 in every row, as Section 2.1 states.
- Equalized pull. For a 0/1 reward, \(\sum_i(r_i-\bar r)^2=G\hat p(1-\hat p)\), so the unnormalized group carries 2.3 times the squared weight at \(k=4\) that it does at \(k=1\) (2.000 against 0.875). Dividing by the sample std makes \(\sum A^2\) equal to \(G-1=7\) for every mixed group. Standard-deviation normalization therefore gives every informative prompt the same total weight, which up-weights very easy and very hard prompts relative to the unnormalized form: the "question-level difficulty bias" of Liu et al.17 (§3.1). Lambert's counterpoint is that removing the division down-weights prompts where one sample in the group succeeds, "valuable learning signal" on the hardest cases11 (§6.2.8). REINFORCE++ proposes dividing by the std of the whole batch instead (Table 1) and argues the per-prompt form is a biased estimator8 (abstract, App. A). Module 11 weighs the options.
- Scale invariance. Dividing by the std makes the advantage invariant to \(r\mapsto ar+b\) for \(a>0\) (derived). The other forms scale with the reward; TRL's docs warn that without it "update magnitudes depend directly on the raw reward scale and batch composition"13. That matters when a reward is a sum of components such as correctness and format (Module 10).
- Uniform groups carry nothing. At \(k=0\) or \(k=G\) the numerator is zero. With the \(10^{-4}\) added to the denominator the advantage is exactly 0 rather than NaN14; how often that happens as a function of \(\hat p\) and \(G\) is Module 2's Table 6.
4.3 Online RFT against GRPO
DeepSeekMath's own comparison is between GRPO and Online RFT, which samples from the live policy and trains on correct answers only. Its coefficient is the indicator of correctness (1 or 0), whatever the rest of the group did2 (App. A.1.2-A.1.3, Eqs. 10-11). GRPO's coefficient adapts to the group; for a 0/1 reward with population std, a correct sample gets \(\sqrt{(1-\hat p)/\hat p}\) and an incorrect one \(-\sqrt{\hat p/(1-\hat p)}\) (derived). Figure 4 draws both.
DeepSeekMath reads its result as evidence for "altering positive and negative gradient coefficients": GRPO beat Online RFT on GSM8K and MATH, and GRPO with process supervision beat GRPO with outcome supervision, "indicating the benefits of using fine-grained, step-aware gradient coefficients"2 (§5.2.1, Figure 5). The figure shows what the negative side costs. A negative coefficient lowers a completion's likelihood without saying where the mass goes (Module 2 §3.2), and the std-normalized form makes that push largest when failures are rare.
5. DeepSeekMath-RL, and what later recipes changed
5.1 The original settings, and the same knobs in later reports
DeepSeekMath-RL starts from DeepSeekMath-Instruct 7B and trains on about 144K chain-of-thought-format questions drawn from the GSM8K and MATH portions of the SFT data, excluding all other SFT questions so that the other benchmarks stay out of domain2 (§4.2). The reward model is initialized from DeepSeekMath-Base 7B with learning rate \(2\times10^{-5}\) and trained on rule-based correctness judgments2 (§4.2, §5.2.1), so even here the reward is a learned model of a rule, not the rule itself; Module 10 returns to why R1 dropped the model. Table 4 sets those settings beside the same knobs in later critic-free recipes.
| Recipe | \(G\) | Outputs per rollout | Updates per rollout | Learning rate | KL \(\beta\) | Clip \(\varepsilon\) | Max response (tokens) |
|---|---|---|---|---|---|---|---|
| DeepSeekMath-RL 7B2 | 64 | 1,024 ? | 1 | 1e-6 | 0.04 | — | 1,024 |
| R1-Zero5 | 16 | 8,192 | 16 | 3e-6 | 0.001 | — | 32,768; 65,536 after step 8.2k |
| DAPO18 | 16 | 8,192 | 16 | 1e-6 | none | 0.2 / 0.28 | 20,480 |
| Olmo 3 7B Think15 | 8 | 512 | 1 | 1e-6 | none | 0.2 / 0.272 | 32,768 |
| Olmo 3 32B Think15 | 8 | 1,024 | 1 | 2e-6 | none | 0.2 / 0.272 | 32,768 |
| Nemotron 3 Ultra19 | 16 | global batch 8,192 ? | — | — | — | — | 48K, later 64K |
| TRL defaults13 | 8 | — | 1 | — | 0.0 | 0.2 | — |
Three things stand out. Group size fell from 64 to 8 or 16 while responses grew from 1,024 to 20,480 to 65,536 tokens. The arithmetic of group size shows what that costs: a prompt the policy solves 10% of the time yields a mixed group with probability \(1-0.9^G-0.1^G\), which is 57% at \(G=8\), 81% at \(G=16\) and 99.9% at \(G=64\) (derived; Module 2, Table 6, has the general form), so a smaller group wastes more prompts; whether the saved decode time repays that is the budget trade Module 15 measures. The KL coefficient went from 0.04 to 0.001 to none; Module 13 treats whether that is safe. And updates per rollout went from 1 to 16 in R1-Zero and DAPO, which is where the clip starts to matter (Module 12). At the upper bounds, and reading 1,024 as outputs, DeepSeekMath's rollout is at most \(1{,}024\times1{,}024\approx1.05\) M generated tokens, R1-Zero's at most \(8{,}192\times32{,}768\approx268\) M, a factor of 256 before R1-Zero's later 65,536-token limit (derived; real lengths are far below the cap, and Module 1 uses measured means).
The "?" for DeepSeekMath is a real ambiguity. The report states "we sample 64 outputs" per question, "the training batch size is 1024" and a single update per exploration stage, and does not say whether 1,024 counts outputs or questions2 (§4.2). R1 counts outputs ("32 unique questions, resulting in a training batch size of 512")5 (§2.1). On that convention a step is 16 questions of 64 outputs, and 144K questions are 9,000 steps per epoch, which fits the 5,300- to 8,000-step axes of Figures 5 and 6 in the report; if 1,024 counted questions, a step would hold 65,536 outputs and an epoch would be about 141 steps, which does not (derived; the writer's reading of the axes).
5.2 What it bought
| Benchmark | Instruct 7B | RL 7B | Gain (points) |
|---|---|---|---|
| GSM8K | 82.9 | 88.2 | +5.3 |
| MATH | 46.8 | 51.7 | +4.9 |
| MGSM-zh (out of domain) | 73.2 | 79.6 | +6.4 |
| CMATH (out of domain) | 84.6 | 88.8 | +4.2 |
The out-of-domain gains are the paper's point about generality: the RL data came only from GSM8K- and MATH-related SFT questions2 (§4.2). The paper's own diagnosis is more cautious. Maj@K improves with RL and Pass@K does not (temperature 0.7, K up to 64), which the authors read as RL "rendering the output distribution more robust" and "boosting the correct response from TopK rather than the enhancement of fundamental capabilities"2 (§5.2.2, Figure 7). That sentence is the seed of the sharpen-or-expand debate; Module 20 collects the later evidence.
5.3 Iterative GRPO
Algorithm 1 wraps GRPO in an outer loop: at the start of each iteration the reference model is set to the current policy; an inner loop samples a batch, sets \(\pi_{old}\leftarrow\pi_\theta\), draws \(G\) outputs per question, scores them, computes group advantages and takes \(\mu\) GRPO updates; at the end of the iteration the reward model is trained further on fresh policy samples with "a replay mechanism that incorporates 10% of historical data"2 (§4.1.4, Algorithm 1). The motivation is that "the old reward model may not be sufficient to supervise the current policy model"2 (§4.1.4). Two iterations were run and "the iterative RL significantly improves the performance, especially at the first iteration"2 (§5.2.1, Figure 6). Two choices in that loop, resetting the reference and refreshing the scorer, reappear in R1 and Kimi in changed form; Section 6 draws the first.
6. Kimi k1.5: online policy mirror descent
6.1 From a KL-regularized objective to a squared loss
Kimi k1.5 was posted on the same day as DeepSeek-R1 (22 Jan 2025) and does not use GRPO45. At iteration \(i\) it takes the current model \(\pi_{\theta_i}\) as the reference and solves a relative-entropy-regularized problem (§2.3.2, Eq. 2):
$$\max_\theta\ \mathbb E_{x}\Big[\mathbb E_{(y,z)\sim\pi_\theta}\big[r(x,y,y^*)\big]-\tau\,\mathrm{KL}\big(\pi_\theta(x)\,\Vert\,\pi_{\theta_i}(x)\big)\Big],\qquad \pi^*(y,z\mid x)=\frac{\pi_{\theta_i}(y,z\mid x)\,e^{r/\tau}}{Z}.$$This is the optimum Module 6 derives with the reference in the role of \(\pi_{ref}\). Taking logarithms gives, for any sampled \((y,z)\), \(r-\tau\log Z=\tau\log\frac{\pi^*}{\pi_{\theta_i}}\), "which allows us to leverage off-policy data". That identity is turned into a regression of the log-ratio on the centered reward:
$$L(\theta)=\mathbb E_{(y,z)\sim\pi_{\theta_i}}\Big[\Big(r-\tau\log Z-\tau\log\frac{\pi_\theta(y,z\mid x)}{\pi_{\theta_i}(y,z\mid x)}\Big)^2\Big].$$The expectation is over the fixed distribution \(\pi_{\theta_i}\), not over \(\pi_\theta\). The data distribution does not move while \(\theta\) does, so several gradient steps on one batch need no importance ratio and no clip; that is the structural difference from GRPO (derived from the form of \(L\); the paper says the formulation lets it use off-policy data). \(\tau\log Z\) is replaced by a sample estimate: \(\tau\log\frac1k\sum_j e^{r_j/\tau}\), or the plain mean \(\bar r\) of the \(k\) sampled rewards, which "yields effective practical results" and is justified because \(\tau\log Z\to\mathbb E_{\pi_{\theta_i}}[r]\) as \(\tau\to\infty\)4 (§2.3.2). The resulting gradient is
$$\frac1k\sum_{j=1}^{k}\Big[\nabla_\theta\log\pi_\theta(y_j,z_j\mid x)\,(r_j-\bar r)-\frac\tau2\nabla_\theta\Big(\log\frac{\pi_\theta(y_j,z_j\mid x)}{\pi_{\theta_i}(y_j,z_j\mid x)}\Big)^2\Big].$$Write \(\ell_j=\log(\pi_\theta/\pi_{\theta_i})\) for sample \(j\). Since \(\nabla\ell_j^2=2\ell_j\nabla\log\pi_\theta\), the update is a weighted score with weight \(r_j-\bar r-\tau\ell_j\) (derived). At the start of an iteration \(\ell_j=0\) and the weight is \(r_j-\bar r\), the "R − mean" column of Table 3 with the sample included in its own mean. As the policy moves, \(-\tau\ell_j\) cancels the weight of samples whose likelihood has already risen and boosts those that have fallen, which is a restoring force with strength \(\tau\) per nat; it does the job the clip does in PPO (the writer's reading). The fixed point is \(\ell_j=(r_j-\tau\log Z)/\tau\). Because the reference changes every iteration, "we also reset the optimizer at the start of each iteration"4 (§2.3.2).
6.2 How good is the sampled mean as \(\tau\log Z\)?
| \(\tau\) | \(\tau\log\frac1k\sum e^{r_j/\tau}\) | Gap to mean 0.375 |
|---|---|---|
| 0.1 | 0.902 | 0.527 |
| 0.3 | 0.723 | 0.348 |
| 1 | 0.497 | 0.122 |
| 3 | 0.415 | 0.040 |
| 10 | 0.387 | 0.012 |
| 100 | 0.376 | 0.001 |
The two baselines agree only when \(\tau\) is large relative to the reward range. At small \(\tau\) the exact estimate approaches the best sampled reward and would give nearly every sample a negative weight, while the mean gives weights that sum to zero. Which regime Kimi runs in is not disclosed; the sentence that the mean works well is a report of practice, not a derivation (the writer's reading).
6.3 Against GRPO, and where the reference sits
The authors note that the gradient "resembles the policy gradient of (2) using the mean of sampled rewards as the baseline (Kool et al. 2019; Ahmadian et al. 2024)", with two differences: responses come from \(\pi_{\theta_i}\) rather than on-policy, and an \(\ell_2\) regularizer is applied4 (§2.3.2). Relative to GRPO there is no division by the group std, no ratio and no clip, and the KL-like term is a quadratic on the log-ratio anchored at the current iterate. The lineage continues: K2 "adopt[s] the policy optimization algorithm introduced in K1.5 [...] as the foundation"20 (§3.2.3); K2.5 departs from it with a token-level mask on the log-ratio that ignores the sign of the advantage21; K3 delegates tolerance of stale rollouts to "a per-token regularization"22 (§4.1.2). Modules 12 and 28 follow that thread.
The three recipes in this module also differ in where the KL anchor sits, which Figure 5 draws.
7. Side by side: what each objective assumes
| Property | GRPO2 | RLOO3 | Kimi k1.54 | ReMax1 |
|---|---|---|---|---|
| Advantage of sample \(i\) | \((r_i-\bar r)/\operatorname{std}\) | \(r_i-\) mean of the other \(k-1\) | \(r_i-\bar r\), then \(-\tau\ell_i\) as the policy moves | \(r_i-r(\text{greedy})\) |
| Scales with the reward? | no (invariant, up to \(\epsilon\)) | yes | yes | yes |
| Ratio and clip | per-token ratio, clip \(\varepsilon\) | none | none; quadratic anchor of strength \(\tau\) | none |
| KL to a reference | \(\beta k_3\) in the loss | sequence log-ratio inside the reward | reference is the current iterate, reset each iteration | reference kept for the KL penalty |
| Meant for several updates per rollout? | yes (\(\mu\ge1\), clip) | no ratio; the paper still takes 2 steps per batch | yes (loss is off-policy by construction) | no |
| Smallest useful group | 2 | 2 | 2 (at \(k=1\), \(r_1-\bar r=0\)) | 1 plus one greedy decode |
| Reward assumption | comparable within a prompt | comparable and scaled across prompts | comparable and scaled across prompts | comparable and scaled across prompts |
Four assumptions follow from the table and Section 4, and they are what to check before choosing among the methods.
- Mixed groups. GRPO, RLOO and Kimi's loss give every sample in a group whose rewards are all equal an advantage of zero. For a prompt solved with probability \(p=0.1\), seeing at least one success with probability 0.95 takes \(G\ge\ln0.05/\ln0.9\approx28.4\), so 29 samples (derived); at the \(G=8\) to 16 of Table 4 such a prompt gives a mixed group 57% to 81% of the time. A binary verifier therefore needs a base model that already solves a useful fraction of the prompts (Module 2 §7, Modules 10 and 11).
- Exchangeable samples. The unbiasedness argument needs the other samples to be draws from the same policy as the one being credited (Module 2 §4.2). With partial rollouts a trajectory can span several policy versions22 (§4.1.2), so the assumption holds only approximately in asynchronous systems (the writer's reading; Modules 14 and 18).
- One scalar per completion. Outcome-supervised GRPO gives a 30,000-token chain of thought the same advantage on every token. The process-supervised variant is the only credit refinement GRPO itself defines; the rest of the literature either accepts the bandit view or replaces the baseline (Section 8).
- Reward scale. Only GRPO's form ignores the reward's units. RLOO and Kimi's loss need a reward whose scale is controlled, which is easy for 0/1 verifiers and not for learned reward models or sums of components.
TRL's GRPO trainer ships a version that differs from the paper in five ways: the group std (scale_rewards="group", the default) is computed with Bessel's correction and has \(10^{-4}\) added14; loss_type="dapo" (token-level normalization over the batch rather than \(1/|o_i|\) per sequence); \(\beta=0\); an optional \(\varepsilon_{high}\) (DAPO recommends 0.28); and an importance_sampling_level of "token", with "sequence" giving GSPO behavior. Its default of one iteration per batch matches DeepSeekMath's setting13. Check the version you run; Module 11 covers the normalization choice and Module 12 the ratio.
8. The critic question: the experiments that exist
8.1 Who runs what
Among the reports that state their RL algorithm, the large reasoning runs are critic-free: R1 (GRPO)5, Qwen3 (GRPO, 3,995 query-verifier pairs)23, DeepSeek-V3.2 (GRPO, advantage \(R-\bar R\))16, DeepSeek-V4's specialists (GRPO)24, GLM-5 (GRPO with IcePop, KL removed)25, Olmo 3 (OlmoRL: GRPO plus zero-gradient filtering, token-level loss, no KL, clip-higher)15, Nemotron 3 Ultra (asynchronous GRPO)19, DAPO18 and the Kimi line20. The exceptions are Tulu 3's RLVR stage, which runs PPO with a value function initialized from a general reward model and lists value-free GRPO as future work26, and VAPO10. Adoption shows what teams made work at scale on a deadline, not where the ceiling is. I found no report in this course's source map that ablates a critic against a group baseline at its own scale, so the evidence comes from smaller controlled studies.
8.2 Four controlled comparisons
Ahmadian et al. (Section 2) found RLOO ahead of PPO, with PPO's best setting the one where the critic is only a baseline. R1's appendix compares PPO and GRPO on MATH with DeepSeek-Coder-V2-Lite: at the common default \(\lambda=0.95\) "PPO performs considerably worse than GRPO", at \(\lambda=1.0\) its performance "improves substantially, nearing that of GRPO", and the authors conclude PPO "can achieve comparable performance when appropriately tuned" but costs hyperparameter search and a second model5 (Supp. A.3). Figure 6 (right) reads off the plot.
VinePPO's controlled comparison points the other way. From one SFT checkpoint and one episode budget on DeepSeekMath 7B, tuned PPO scores 42.8 on MATH against 36.4 for GRPO and 36.8 for RLOO, and 78.9 on GSM8K against 74.8 and 75.3 (gaps to GRPO of 6.4 and 4.1 points, derived); VinePPO, which replaces the network with Monte Carlo values, reaches 46.0 and 80.19 (Figure 4). Two details matter. The authors' hyperparameters were tuned for PPO, and GRPO and RLOO were initialized from them and given "additional tuning to stabilize training" under the same episode budget (§5). And the critic was not undertrained, with explained variance of 0.7 to 0.9, yet when asked to rank five candidate next steps from one state it "performs near chance levels for most of the training" (§6.1, §7, Figure 8). A value network can be accurate on average and still useless at telling siblings apart, which is the comparison credit assignment needs. VinePPO's price is per-step cost, up to 5x PPO's on a 1.1B model and 2x on 7B, repaid by reaching PPO's peak in about 3.0x and 1.51x less wall-clock time (§6.2).
VAPO tests the long-chain-of-thought regime on Qwen2.5-32B with no SFT data. Plain PPO scores 5 on AIME 2024: the value model collapses and responses shrink until the model answers without reasoning10 (§5.2). The authors name three failure modes of value models in this regime. A value model initialized from a reward model starts with a positive bias. A fixed GAE \(\lambda=0.95\) multiplies the terminal reward by \(0.95^{100}\approx0.006\) for a token 100 steps from the end, so values become almost entirely bootstrapped (§3.1, §4.2). And binary verifier rewards are sparse. Their seven modifications include value pretraining, decoupled GAE (\(\lambda=1.0\) for the critic's targets), a length-adaptive policy \(\lambda=1-1/(\alpha l)\) with \(\alpha=0.05\), Clip-Higher and a token-level loss (§4, §5.1). The result is 60.4 at best against DAPO's 50 and R1-Zero-Qwen-32B's 47 (Figure 7).
8.3 Reading the four together
The results do not conflict as much as they look. First, "PPO" is not one method: on one base model its score ranges from 5 to 60 depending on seven choices (Figure 7), while the critic-free arms cluster (RLOO 36.8 and GRPO 36.4 in VinePPO's study). Group baselines are robust by default; critics reward engineering. Second, horizon: VinePPO caps responses at 1,024 tokens (App. C) and R1's appendix does not state the cap for its experiment, while VAPO documents failure modes that appear when the response is long and the reward sparse. Third, accounting: VAPO reports steps, VinePPO equal episodes, and no study charges the critic's forward and backward passes against the group method's extra samples. Fourth, each paper tunes the arm it favors. The Monte Carlo result also shows that the useful ingredient may be credit assignment itself, not the network: GiGPO gets step-level groups from repeated environment states with no critic and no extra rollouts, reporting gains of more than 12% on ALFWorld and 9% on WebShop over GRPO27 (abstract; Module 18).
Where experts disagree
The question is whether PPO's learned value model earns its cost on LLM post-training. The sources agree on the arithmetic of what a critic costs and disagree about what it buys once it is built well, and the experiments do not share a protocol.
A. A critic is unnecessary; spend the memory on samples
Ahmadian et al.3, Shao et al.2, the DeepSeek-R1 appendix5, Kimi k1.54, Li et al. (ReMax)1
Rewards arrive at the last token, transitions are deterministic and the initial policy is strong, so a per-token value model adds cost and bias for little variance reduction. A group baseline is the same estimate computed on the fly.
- One fewer policy-sized model: 46% of GPU memory at 7B in ReMax's accounting1.
- Robust defaults. PPO needed \(\lambda=1.0\) to approach GRPO in R1's appendix5, and nearly every large open reasoning recipe in Section 8.1 chose it.
- Credit is per sequence. A tuned critic beat GRPO and RLOO by 3.6 to 6.4 points on 7B MATH and GSM8K9.
- Uniform groups give zero gradient, so it needs a base model with a useful pass rate and several samples per prompt (Section 7).
B. A well-built value model has the higher ceiling
ByteDance Seed (VAPO)10; Tulu 3's RLVR ablation26; Xu et al.28
Value models give precise credit assignment, lower-variance per-token estimates and generalization across samples; VAPO argues critic-based methods "possess a higher performance ceiling if the challenges in training value models can be addressed"10 (§1). Tulu 3 found a value function initialized from a general reward model worked best and left GRPO untested.
- 60.4 against DAPO's 50 on the same base, with no crashes across three runs10.
- Xu et al. find a carefully configured PPO ahead of DPO and other alignment methods on every testbed they tried, though against DPO rather than against group baselines28.
- It took seven modifications; plain PPO scored 5, and dropping value pretraining left 11 (Figure 7).
- The comparison counts steps, not GPU-hours, and I found no independent replication in the source map.
C. Credit assignment matters, but estimate it by sampling
Kazemnejad et al. (VinePPO)9; Feng et al. (GiGPO)27; DeepSeekMath's process supervision2
The value network is the weak part, not the idea of per-step credit: it ranked sibling steps near chance for most of training, while Monte Carlo values from the same states are unbiased.
- 46.0 against PPO's 42.8 on 7B MATH, with no extra model9.
- GiGPO's anchor-state groups add step-level credit with no critic and no extra rollouts27.
- VinePPO's steps cost up to 5x PPO's, and its evidence is 1.1B and 7B models with responses capped at 1,024 tokens9 (App. C).
- Process supervision needs step labels, which R1 reported as hard to define and easy to hack (Module 4).
These positions answer different questions, and the conditions decide. With a verifier, eight or more samples per prompt, responses up to a few thousand tokens and a limited engineering budget, a group baseline is the right default: it was about 2 points ahead of a \(\lambda\)-tuned critic in R1's test (reading the plot) and 3.6 to 6.4 points behind a tuned critic in VinePPO's, and it has fewer ways to fail. If you can afford a policy-sized critic and replicate VAPO's seven fixes for long chain of thought, there is one documented 10-point advantage over DAPO, unreplicated and counted in steps. Where the bottleneck is credit across many steps of an agent episode, sampling-based credit is the more promising direction (Module 18). What would settle it: one base model, one dataset and DAPO- and Dr. GRPO-grade fixes applied to the group arm and VAPO-grade fixes to the critic arm, at equal GPU-hours including the critic, at two response lengths, over several seeds, reporting pass@1 and pass@k. Nothing in the sources does this. The conclusion that critic-free methods dominate practice is about what teams shipped, not about what is optimal. A second dispute, whether to divide the group advantage by its standard deviation, is Module 11's.
What is changing
The group baseline is staying while its surroundings are stripped. DeepSeek-V3.2 and Olmo 3 dropped the division by the standard deviation1615, and DAPO, GLM-5 and Olmo 3 dropped the KL term182515; the clip is being replaced or reshaped (Module 12). Several 2026 recipes use RL to train specialists and consolidate them afterwards: DeepSeek-V4 runs GRPO per domain and replaces the mixed RL stage with on-policy distillation24, and Lambert records Kimi K3 consolidating nine experts while noting that labs "are far from unanimous" on the step11 (§3.2.4; Modules 21 and 22). The critic question has moved to agentic and multi-turn settings, where group baselines have the least to offer per step and credit assignment is the live problem (Module 18). TRL's defaults have moved from the paper's form to a DAPO-style loss with no KL, so a "GRPO" result is only as specific as its version and flags.
References
- paperLi, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., Luo, Z.-Q. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML 2024; arXiv v4, 16 May 2024. https://arxiv.org/abs/2310.10505. Accessed 2026-10-27. Used for: the greedy-decode baseline (§4.2, Eq. 9, Algorithm 1), the 46% share of GPU memory taken by the value model and its training state at 7B and 4% by the reward model (footnote 1, Figure 2), the 1.6x per-iteration speed-up. Added beyond the source map.
- paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the GRPO objective, the KL estimator and the PPO comparison (§4.1.1, Eqs. 1-4, Figure 4); outcome and process supervision (§4.1.2-§4.1.3); iterative GRPO and Algorithm 1 (§4.1.4); training setup and Table 5 (§4.2); the unified paradigm, Online RFT versus GRPO, iterative RL and Maj@K versus Pass@K (§5.2, Figures 5-7); gradient coefficients (App. A.1, Eqs. 5-21).
- paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: the single-action argument (§1, §3.3), REINFORCE and RLOO (§2.2-§2.3), the GAE-λ and clipping ablations (§3.1-§3.2, Figure 1), win-rates and fluency (§5.2, Tables 1-2), KL and noise robustness (§5.2.2, Figures 5-6), probability-mass concentration (App. A), hyperparameters (App. C).
- paperKimi Team (Du, A., Gao, B., Xing, B. et al.). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: the online policy mirror descent objective, its surrogate loss, gradient and sampled baseline (§2.3.2, Eqs. 2-3), the argument against value functions (§2.3.2), the length reward (§2.3.3).
- paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026 (expanded Nature version, Nature 645, 633-638, 2025). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the GRPO objective as printed (§2.1, Eqs. 1-3), R1-Zero's batch, group size, learning rate, KL coefficient, reference reset and minibatch split (§2.1), and the PPO-versus-GRPO comparison, its argument about value models and KL placement, and Figures 3-4 (Supplementary A.3); v1 dated 22 Jan 2025.
- paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the actor-critic advantage built from a learned value function (§5, Eqs. 10-11) and the clipped surrogate that GRPO keeps (§3, Eq. 7).
- paperSchulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016; arXiv v6, 20 Oct 2018. https://arxiv.org/pdf/1506.02438v6. Accessed 2026-10-26. Used for: GAE(γ, 1) as the empirical return minus a value baseline, unbiased whatever the accuracy of V (§3, Eqs. 17-18).
- paperHu, J., Liu, J. K., Xu, H., Shen, W. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Normalization. arXiv v9, 10 Nov 2025. https://arxiv.org/abs/2501.03262. Accessed 2026-10-27. Used for: the "with baseline" variant (group mean, then division by the global-batch standard deviation) and its argument that per-prompt standard-deviation normalization is a biased estimator (abstract, §1, App. A). Added beyond the source map.
- paperKazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., Le Roux, N. VinePPO: Refining Credit Assignment in RL Training of LLMs. ICML 2025; arXiv v2, 3 Jun 2025. https://arxiv.org/abs/2410.01679. Accessed 2026-10-27. Used for: Monte Carlo value estimates in place of a value network (§4, Eqs. 5-6), the PPO, GRPO and RLOO baselines and their tuning (§5), Pass@1 results (Figure 4), per-iteration cost and wall-clock (§6.2), value-network failure analysis (§7, Figure 8). Added beyond the source map.
- paperByteDance Seed (Yue, Y. et al.). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: the case for value-model-based methods (§1), the three failure modes of value models in long CoT (§3), the seven modifications and training details (§4-§5.1), and the ablation ladder (Table 1). Added beyond the source map.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: RLOO and its relation to group-relative advantages (§6.2.4, Eqs. 49-51), GRPO, Dr. GRPO and their algebraic equivalence to RLOO (§6.2.8, Eqs. 57-62), the comparison of algorithms and their importance-ratio granularity (§6.2.11, Table 3), the K3 expert count (§3.2.4), KL placement across methods (§6.2.4).
- blogSchulman, J. Approximating KL Divergence. Personal site, posted 2020-03-07. http://joschu.net/blog/kl-approx.html. Accessed 2026-10-26. Used for: the k3 estimator that GRPO uses for its KL term.
- officialHugging Face TRL maintainers. TRL documentation: GRPO Trainer (main branch). docs/trl/main; version not stated on the page. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the documented algorithm, the GRPOConfig defaults (num_generations 8, beta 0.0, epsilon 0.2, scale_rewards group, loss_type dapo, num_iterations 1) and the loss types.
- repoHugging Face. TRL, trl/trainer/grpo_trainer.py and trl/trainer/utils.py (main branch; last commit touching grpo_trainer.py: 63d9074, 2026-10-21). https://raw.githubusercontent.com/huggingface/trl/main/trl/trainer/grpo_trainer.py. Accessed 2026-10-27. Used for: the advantage computation: group mean, Bessel-corrected standard deviation (nanstd in utils.py) and the 1e-4 added to the standard deviation (lines 2812-2834). Added beyond the source map.
- paperTeam Olmo (Ettinger, A., Bertsch, A. et al.). Olmo 3. arXiv v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL as GRPO plus modifications (§4.4.1) and the RL hyperparameters of the 7B Think, 32B Think, 7B Instruct and 7B RL-Zero runs (Table 49, p. 109).
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the advantage computed as the reward minus the group mean, with no division by the standard deviation, and mixed RL with GRPO (§3, §3.1).
- paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the response-level length bias from 1/|o_i| and the question-level difficulty bias from the standard deviation (§3.1), pointed to here and treated in Module 11.
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: training details (§4.1: 512 prompts x 16 responses, 16 gradient updates per rollout, learning rate, clip range, length cap), the removal of the KL term (§2.3), the naive-GRPO baseline of 30 AIME points (§1, Table 1).
- paperNVIDIA (Blakeman, A., Thomas, A. et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: asynchronous GRPO, global batch 8192 with 16 rollouts per sample, 48K to 64K generation limit (§3.2).
- paperKimi Team (Bai, Y., Bao, Y., Charles, Y. et al.). Kimi K2: Open Agentic Intelligence. arXiv v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: K2's adoption of the K1.5 policy-optimization algorithm as its foundation (§3.2.3).
- paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.). Kimi K2.5: Visual Agentic Intelligence. arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: the departure from the K1.5 loss by a token-level log-ratio mask (RL algorithm section).
- paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.). Kimi K3: Open Frontier Intelligence. arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: staleness tolerance delegated to a per-token regularization in the loss (§4.1.2).
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: reasoning RL with GRPO on 3,995 query-verifier pairs (§4.2).
- paperDeepSeek-AI (Xu, A., Lin, B., Xue, B. et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv v1, id 2606.19348. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: GRPO for the specialist RL stage and on-policy distillation replacing mixed RL (§5.1, §5.1.1).
- paperGLM-5 Team (Zeng, A., Lv, X. et al.). GLM-5: from Vibe Coding to Agentic Engineering. arXiv v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: Reasoning RL as GRPO with IcePop and the KL term removed (§3.2).
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the finding that initializing the RLVR value function from a general reward model works best (§6.2.1) and the statement that value-free methods such as GRPO were left as future work (Limitations).
- paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025; arXiv v3, 28 Oct 2025. https://arxiv.org/abs/2505.10978. Accessed 2026-10-26. Used for: step-level groups built from repeated environment states, with no critic (abstract).
- paperXu, S., Fu, W., Gao, J., Ye, W. et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: the abstract's claim that a carefully configured PPO surpasses DPO and other alignment methods on all testbeds.