Reinforcement Learning for LLM Post-Training · Module 9

Critic-Free Policy Gradient: REINFORCE with Group Baselines, RLOO, GRPO, and Kimi's Mirror-Descent Objective

PPO's value model is a second policy-sized network that must predict a final reward from a half-written answer. This module covers what replaced it: baselines built from several samples of the same prompt (RLOO's leave-one-out mean, GRPO's group mean and standard deviation, Kimi k1.5's sampled mean inside a mirror-descent loss), the DeepSeekMath settings that introduced GRPO, and the controlled experiments for and against dropping the critic. Afterwards you can write GRPO and RLOO from the equations, say what each assumes about the reward and the sampling budget, and read the critic-versus-no-critic literature without mistaking a tuned baseline for a verdict.

Course 45 min read 28 sources As of 2026-10-27 Prerequisites: Modules 3, 5, 6
One policy-gradient update with the baseline step highlighted, and five ways to compute the baseline: a learned value model, a greedy decode, a leave-one-out mean, a group mean divided by the group standard deviation, and a sampled mean inside a squared-error loss Prompt qfrom the dataset G samplesfrom the policy Score rverifier or RM Baseline bthe design choice A = r − badvantage Σ A ∇log πweighted update PPOlearned V(s)per-token A +1 model Modules 3 and 5 ReMaxgreedy decodeb = r(greedy) +1 decode Li et al., 2024 RLOOother k−1 samplesmean of the rest k−1 samples Ahmadian et al., 2024 GRPOgroup mean ÷ stdall G samples G−1 samples Shao et al., 2024 Kimi k1.5sampled meansquared-error loss k−1 samples Kimi Team, 2025
Figure 1. The shared update and the five baselines this module compares. PPO buys its baseline with a trained network; the other four buy it with extra decodes, one greedy decode for ReMax and extra samples of the same prompt for the rest1. The "+" costs are relative to a single sample per prompt, and in RLOO, GRPO and Kimi's method the extra samples are also trained on. Sources: GRPO2, RLOO3, Kimi k1.54.

1. What the critic bought, and what replaces it

PPO-RLHF (Module 5) trains a value model \(V_\psi(s_t)\) and feeds it to GAE to get a per-token advantage \(A_t\). The value model has one job: it is the baseline, a prefix-dependent estimate of the final reward that is subtracted to cut variance (Module 2). What it buys is token-level credit and, at GAE \(\lambda\lt 1\), bootstrapped variance reduction at the price of bias (Module 3). Every method in this module declines to pay for it, and the reasons are different enough that they should be kept apart.

1.1 Five arguments against a per-token value model

  • Memory and compute. The value function "is typically another model of comparable size as the policy model", which "brings a substantial memory and computational burden"2 (§4.1.1). ReMax measured it for a 7B model: the value model and its training state use 46% of GPU memory, the reward model 4%1 (footnote 1). Module 1 (Table 6) does the model-state arithmetic.
  • Last-token reward. "Usually only the last token is assigned a reward score by the reward model, which may complicate the training of a value function that is accurate at each token"2 (§4.1.1). For long chains of thought, R1's appendix adds that reflection and revision mean "the content initially generated may later be revised or contradicted, which makes it even less feasible to predict the final reward based on a partial response"5 (Supp. A.3).
  • Variance is already low. Ahmadian et al. argue that a pretrained, prompt-conditioned policy concentrates probability on few tokens, so bootstrapping's bias is "not worth it"3 (§3.1). On their Llama SFT model, about 60% of the mass sits on the single most probable token at each step and over 90% on the top 163 (App. A).
  • One action per completion. Rewards "are only attributed to full generations", so the completion can be modeled as a single action3 (§1); Module 2 §1 draws the two views.
  • Credit assignment can punish exploration. Kimi's example: a reasoning step containing an error that the model later repairs would get a negative advantage from an oracle value function, yet "exploring [it] is extremely valuable for training the model to generate long CoT"4 (§2.3.2).

These are claims about cost and learnability. None says a perfect value function would hurt; Section 8 reads the experiments that test whether a practical one helps.

1.2 Paying for a baseline in samples instead of parameters

Table 1. Where the baseline \(b\) comes from, and what it costs. "Extra samples" counts completions beyond one per prompt. Method definitions from the cited papers; the cost columns are the writer's reading of them.
MethodBaseline for sample \(i\)Extra samples per promptExtra trained modelCredit granularity
PPO67\(V_\psi(s_t)\), through GAE0value modelper token
REINFORCE, moving average3mean reward over all steps so far0noneper sequence
ReMax1reward of the greedy decode1noneper sequence
RLOO3mean reward of the other \(k-1\) samples\(k-1\)noneper sequence
GRPO2group mean (including \(i\)), result divided by group std\(G-1\)noneper sequence; per step with a process reward
Kimi k1.54empirical mean of \(k\) sampled rewards\(k-1\)noneper sequence
REINFORCE++ with baseline8group mean, then divided by the global-batch std\(G-1\)noneper sequence
VinePPO9mean return of \(K=9\) rollouts (default) from each step's state\(K\) per step statenoneper step

The price is not extra decoding, it is the shape of the batch. At a fixed budget of \(N\) sequences per step, a group method trains on \(N/G\) distinct prompts instead of \(N\) (derived). Module 1 reports that ScaleRL found this trade neutral over the range it tested, and VAPO's ablation found fewer prompts with more repetitions marginally better even with a critic10 (§4.3). What you do give up is per-token credit: every token of a completion inherits one scalar. The last row of Table 1, and GRPO's process-supervised variant, are the attempts to get credit back without a value network.

2. REINFORCE and RLOO: the leave-one-out baseline

2.1 The estimator

Draw \(k\) completions \(y_{(1)},\dots,y_{(k)}\) from \(\pi_\theta(\cdot\mid x)\) and score each with a sequence-level reward \(R\). RLOO treats each sample in turn as the one being credited and uses the other \(k-1\) as its baseline:

$$\hat g=\frac1k\sum_{i=1}^{k}\Big[R(y_{(i)},x)-\frac1{k-1}\sum_{j\neq i}R(y_{(j)},x)\Big]\nabla_\theta\log\pi_\theta(y_{(i)}\mid x).$$

Ahmadian et al. call the baseline "akin to a parameter-free value-function, but estimated at each training step", and credit the estimator to Kool et al. (2019)3 (§2.3). It is unbiased because \(b_i\) does not depend on \(y_{(i)}\); Module 2 §4.2 proves it and shows what happens when the sample is included in its own baseline. The two forms differ by a constant, \(R_i-b_i=\tfrac{k}{k-1}(R_i-\bar R)\)11 (Eq. 51), and Lambert shows that Dr. GRPO's advantage equals RLOO's up to that factor11 (§6.2.8, Eq. 62). Table 3 checks the factor numerically.

Two structural choices separate RLOO from GRPO. The KL penalty is folded into the sequence reward, \(R=r_\phi-\beta\log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}\)3 (Eq. 3), so it passes through the baseline and is broadcast to every token11 (§6.2.4); GRPO puts it in the loss (Module 6). And there is no importance ratio or clip in the estimator11 (§6.2.11), so it is a plain on-policy gradient.

2.2 What the paper measured

Setup: Pythia-6.9B on TL;DR Summarize and Anthropic-HH, Llama-7B on HH; the reward model and policy start from the SFT checkpoint; constant learning rate \(10^{-6}\) with 3% warm-up; \(\beta=0.03\) on TL;DR and 0.10 on HH; for the Pythia runs a rollout batch of 512 and a step batch of 256, and "for all algorithms, we take 2 gradient steps for each batch"3 (App. C). Win-rates are GPT-4 judgments against reference completions, at the checkpoint with the highest test reward3 (§4.2, Table 1).

Table 2. Win-rates (%) and mean output length from Ahmadian et al.3 (Tables 1 and 2). Length is for the Llama HH models. RAFT fine-tunes on the highest-reward of \(k\) samples. Best win-rate per column marked, and the worst win-rate on HH-Llama.
MethodTL;DR, Pythia-6.9BHH, Pythia-6.9BHH, Llama-7BLength, HH Llama (tokens)
RLOO, \(k=4\)77.943.764.160.6
RAFT, \(k=4\)73.242.163.362.4
RLOO, \(k=2\)74.247.662.258.6
RAFT, \(k=2\)72.137.758.452.8
REINFORCE with baseline70.737.955.347.2
Vanilla policy gradient70.436.452.339.1
PPO67.629.232.016.5
DPO66.639.061.9104.4

Four readings. First, REINFORCE with a moving-average baseline already beats PPO by 3.1, 8.7 and 23.3 points (derived from Table 2). The paper's introduction states "3.2% to 20.3%"; I reproduce the upper end as vanilla policy gradient against PPO on HH-Llama (52.3 − 32.0), not the lower end exactly, so quote the table, not the range. Second, RLOO at \(k=4\) beats PPO by 10.3, 14.5 and 32.1 points, as the text says (derived check: 77.9 − 67.6, 43.7 − 29.2, 64.1 − 32.0). Third, RLOO beats RAFT at equal \(k\), and RLOO at \(k=2\) matches or beats RAFT at \(k=4\) on two of three columns (1.1 points behind on HH-Llama): using all samples through a baseline beats keeping only the best one. Fourth, RAFT degrades faster than RLOO as the KL weight rises and as Gaussian noise (\(\sigma\) of 1, 3 and 5) is added to the reward, because RAFT trains only on the top-ranked sample and anything that scrambles the ranking hurts it3 (§5.2.2, Figures 5-6).

2.3 Why PPO lost in this setup

The paper takes PPO apart. Sweeping the GAE parameter over \(\lambda\in\{0,0.5,0.95,1.0\}\) gave "a monotonically decreasing reward with decreasing \(\lambda\)", so the best PPO is the one where the critic serves only as a baseline, with no bootstrapping3 (§3.1, Figure 1). That is the estimator GAE(\(\gamma,1\)): the empirical return minus a value baseline, unbiased "regardless of the accuracy of \(V\)"7 (§3, Eq. 18). The loss was clipped "on average < 5% of the time per batch", and turning off clipping and then the ratio slightly helped3 (§3.2).

3. GRPO as DeepSeekMath defines it

3.1 The objective

For each question \(q\), sample a group \(\{o_1,\dots,o_G\}\) from \(\pi_{old}\) and maximize2 (§4.1.1, Eq. 3)

$$\mathcal J_{GRPO}(\theta)=\mathbb E\Big[\frac1G\sum_{i=1}^{G}\frac1{|o_i|}\sum_{t=1}^{|o_i|}\Big(\min\big(\rho_{i,t}\hat A_{i,t},\ \mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\hat A_{i,t}\big)-\beta\,\mathbb D_{KL}\big[\pi_\theta\,\Vert\,\pi_{ref}\big]_{i,t}\Big)\Big],\qquad \rho_{i,t}=\frac{\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})}{\pi_{old}(o_{i,t}\mid q,o_{i,\lt t})}.$$

The per-token KL estimator is \(\frac{\pi_{ref}}{\pi_\theta}-\log\frac{\pi_{ref}}{\pi_\theta}-1\), "guaranteed to be positive"2 (Eq. 4); it is Schulman's \(k_3\)12, whose value and gradient Module 6 separates. With outcome supervision every token of \(o_i\) receives the same advantage, the normalized reward2 (§4.1.2):

$$\hat A_{i,t}=\tilde r_i=\frac{r_i-\operatorname{mean}(\mathbf r)}{\operatorname{std}(\mathbf r)}.$$

With process supervision, a process reward model scores the end of each reasoning step, all step rewards in the group are normalized by the same mean and standard deviation, and a token's advantage is the sum of the normalized rewards of the steps that end at or after it, \(\hat A_{i,t}=\sum_{index(j)\ge t}\tilde r_i^{index(j)}\)2 (§4.1.3). That is a return without discounting, which gives dense credit without a value network. The authors' stated reason for the group form is that it "aligns well with the comparative nature of rewards models", which are trained on comparisons between outputs for the same question2 (§4.1.1).

3.2 What changes relative to PPO

Component diagram contrasting PPO-RLHF and GRPO: PPO trains a policy and a value model, freezes a reference and a reward model, puts the KL penalty inside a per-token reward and computes the advantage with GAE; GRPO has no value model, adds the KL term to the loss, and computes the advantage from the mean and standard deviation of the group's rewards PPO-RLHF Policy Value model Reference Reward model r − β·KL per tokenKL inside the reward GAE → A_tone A per token V(s_t) at every token GRPO PolicyG samples per q no value model Reference RM or verifier r_1 … r_GG scores, one prompt (r − mean) ÷ stdsame A on every token β·KL(π ‖ π_ref) is added to the loss, not to r
Figure 2. PPO against GRPO. Blue and orange blocks are trained, grey blocks are frozen, the dashed block is what GRPO removes. GRPO keeps the ratio and the clip; it replaces the critic and GAE with group statistics, and moves the KL term from the reward into the loss. Redrawn from DeepSeekMath Figure 42 and R1's Supplementary Figure 35.

Four things differ. The advantage comes from the group, not from \(V_\psi\) and GAE. The KL term sits in the loss, so it does not enter the advantage2 (§4.1.1); R1's appendix argues that PPO's per-token KL reward penalizes the cumulative KL and "may implicitly penalize the length of the response", which GRPO's placement avoids5 (Supp. A.3). The batch has \(G\) completions per prompt, 64 in DeepSeekMath2 (§4.2). And the clipped ratio, the reference model and the reward source stay. Resident models drop from four to three; with \(\beta=0\) and a verifier, the reference and the reward model both go (Module 1, Table 6), which is why TRL defaults to \(\beta=0\), "reducing memory usage and improving training speed"13.

3.3 The gradient when there is one update per rollout

DeepSeekMath's RL stage makes "a single update following each exploration stage"2 (§4.2), so at the moment of the gradient \(\pi_\theta=\pi_{old}\). Then \(\rho_{i,t}=1\) sits inside \((1-\varepsilon,1+\varepsilon)\), the \(\min\) selects the unclipped branch, and \(\nabla\rho=\rho\nabla\log\pi_\theta=\nabla\log\pi_\theta\). For the KL term write \(\varrho=\pi_{ref}/\pi_\theta\), so \(\nabla\varrho=-\varrho\,\nabla\log\pi_\theta\) and \(\nabla k_3=\nabla(\varrho-\log\varrho-1)=(1-\varrho)\nabla\log\pi_\theta\) (derived). Ascending \(\mathcal J\) therefore follows

$$\nabla_\theta\mathcal J=\mathbb E\Big[\frac1G\sum_{i=1}^{G}\frac1{|o_i|}\sum_{t=1}^{|o_i|}\Big(\hat A_{i,t}+\beta\Big(\frac{\pi_{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)}-1\Big)\Big)\nabla_\theta\log\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})\Big],$$

which is DeepSeekMath's Eq. 20, with "gradient coefficient" \(GC=\hat A_{i,t}+\beta(\pi_{ref}/\pi_\theta-1)\) in Eq. 212 (App. A.1.6; the derivation there also assumes \(\pi_{\theta_{old}}=\pi_\theta\)). At one update per rollout, GRPO is REINFORCE with group-normalized, length-normalized sequence weights plus a KL pull toward the reference; Module 2 (Table 2) lists the row next to SFT, RFT, PPO and DPO. The clip only becomes active when updates are reused: R1 splits an 8,192-output rollout into 16 minibatches of 512 and trains one inner epoch5 (§2.1), so minibatch 1 is on-policy and minibatches 2 to 16 see parameters that have moved 1 to 15 optimizer steps (derived). Module 12 covers what the ratio and clip do there. Module 6 (§3.3) shows that, on-policy, the expected gradient of this \(k_3\) term is the gradient of \(\mathrm{KL}(\pi_{ref}\Vert\pi_\theta)\), the opposite direction from the one the estimator is meant to track.

3.4 One step on one prompt

Figure 3 runs the arithmetic on four completions with continuous reward-model scores, the setting of the original paper; Section 4 covers the binary-reward case.

One prompt q, G = 4 completions from the old policy o1 o2 o3 o4 r_i 2.1 −0.4 0.9 1.4 mean = (2.1 − 0.4 + 0.9 + 1.4) ÷ 4 = 1.000 Σ (r − mean)² = 3.34 std = √(3.34 ÷ 3) = 1.055 sample std (n − 1), as TRL computes it A_i +1.04 −1.33 −0.09 +0.38 A_i = (r_i − 1.000) ÷ 1.055 every token of o_i carries the same A_i
  1. Sample. The old policy writes four completions of lengths 6, 9, 5 and 7 tokens for one prompt (illustrative lengths).
  2. Score. A reward model or verifier returns one scalar per completion: 2.1, −0.4, 0.9, 1.4 (illustrative values).
  3. Group statistics. Mean 1.000 and sample standard deviation 1.055, computed over these four numbers only.
  4. Advantage. Each reward becomes a z-score within its group: +1.04, −1.33, −0.09, +0.38. They sum to zero by construction.
  5. Broadcast. The scalar is copied to every token of its completion; the loss multiplies it by the ratio and clip of Section 3.1 and adds the KL term.

3.5 As code

The loss is short enough to write out in full. This version follows Eq. 3 (per-sequence \(1/|o_i|\) normalization, symmetric clip, \(k_3\) in the loss). The advantage uses the sample standard deviation and adds \(10^{-4}\) to the denominator, which is what TRL does14 (lines 2812-2834 of grpo_trainer.py, with the Bessel correction in nanstd); neither detail appears in the paper.

import torch

def group_advantage(rewards, eps=1e-4):
    # rewards: (P, G), G samples for each of P prompts
    mean = rewards.mean(dim=1, keepdim=True)
    std = rewards.std(dim=1, keepdim=True)          # unbiased (n - 1), as in TRL
    return (rewards - mean) / (std + eps)            # (P, G); zero when all rewards in a group are equal

def grpo_loss(logp, logp_old, logp_ref, adv, mask, eps=0.2, beta=0.04):
    # logp, logp_old, logp_ref: (N, T) log-probs of the sampled tokens, N = P * G sequences
    # adv: (N,) one scalar per sequence; mask: (N, T) 1.0 on completion tokens
    ratio = torch.exp(logp - logp_old)                                    # pi_theta / pi_old
    a = adv.unsqueeze(1)
    surr = torch.minimum(ratio * a, torch.clamp(ratio, 1 - eps, 1 + eps) * a)
    d = logp_ref - logp                                                   # log(pi_ref / pi_theta)
    kl = torch.exp(d) - d - 1                                             # k3, >= 0
    per_token = surr - beta * kl
    per_seq = (per_token * mask).sum(1) / mask.sum(1)                     # 1/|o_i| sum_t
    return -per_seq.mean()                                                # 1/G sum_i, negated for descent

I checked the gradient of grpo_loss against the closed form of Eq. 20 on a toy model (eight sequences of 2 to 6 tokens, vocabulary 5, \(\pi_{old}\) a detached copy of \(\pi_\theta\)): the maximum absolute difference was \(1.9\times10^{-9}\) in fp32 (PyTorch 2.10), and the ratio was exactly 1 everywhere, as Section 3.3 predicts.

4. What the group advantage does to the gradient

4.1 Binary rewards, group of eight

With a 0/1 verifier and \(k\) correct samples out of \(G\), the group pass rate is \(\hat p=k/G\) and every correct sample shares one advantage, every incorrect sample another. Table 3 gives both for the four forms in use. All values are derived from the definitions; the "÷ sample std" column is what TRL computes14.

Table 3. Advantage of a correct / incorrect sample, \(G=8\), binary reward (derived). "R − mean" is the form of Dr. GRPO and DeepSeek-V3.21116; "leave-one-out" is RLOO; "÷ sample std" is GRPO as shipped in TRL; "÷ pop. std" is the same with \(n\) instead of \(n-1\) in the variance. Groups with \(k=0\) or \(k=8\) give zero in every column. The last column is \(\sum_i A_i^2\) for the "R − mean" form.
\(k\)\(\hat p\)R − meanLeave-one-out÷ sample std÷ pop. std\(\sum A^2\)
10.125+0.875 / −0.125+1.000 / −0.143+2.475 / −0.354+2.646 / −0.3780.875
20.250+0.750 / −0.250+0.857 / −0.286+1.620 / −0.540+1.732 / −0.5771.500
40.500+0.500 / −0.500+0.571 / −0.571+0.935 / −0.935+1.000 / −1.0002.000
60.750+0.250 / −0.750+0.286 / −0.857+0.540 / −1.620+0.577 / −1.7321.500
70.875+0.125 / −0.875+0.143 / −1.000+0.354 / −2.475+0.378 / −2.6460.875

Group advantage under a binary reward

Pass rate
R − mean, correct
R − mean, wrong
Leave-one-out, correct
Leave-one-out, wrong
÷ sample std, correct
÷ sample std, wrong
Σ A², R − mean

Formulas for \(\hat p=\min(k,G)/G\): R − mean gives \(1-\hat p\) and \(-\hat p\); leave-one-out multiplies by \(G/(G-1)\); dividing by the sample std \(\sqrt{G\hat p(1-\hat p)/(G-1)}\) gives the third form; \(\sum A^2=G\hat p(1-\hat p)\) for R − mean, and exactly \(G-1\) after dividing by the sample std whenever the group is mixed. At \(k=0\) there are no correct samples and at \(k=G\) no wrong ones, so those outputs are vacuous; the group's advantage is zero.

4.2 What the table implies

  1. Zero sum. Every column sums to zero over the group: the three mean-centered forms by definition, leave-one-out because \(\sum_i(R_i-b_i)=\sum R-\frac{(G-1)\sum R}{G-1}=0\) (derived).
  2. The factor \(G/(G-1)\). Leave-one-out over R − mean is 1.000/0.875 = 0.857/0.750 = 8/7 in every row, as Section 2.1 states.
  3. Equalized pull. For a 0/1 reward, \(\sum_i(r_i-\bar r)^2=G\hat p(1-\hat p)\), so the unnormalized group carries 2.3 times the squared weight at \(k=4\) that it does at \(k=1\) (2.000 against 0.875). Dividing by the sample std makes \(\sum A^2\) equal to \(G-1=7\) for every mixed group. Standard-deviation normalization therefore gives every informative prompt the same total weight, which up-weights very easy and very hard prompts relative to the unnormalized form: the "question-level difficulty bias" of Liu et al.17 (§3.1). Lambert's counterpoint is that removing the division down-weights prompts where one sample in the group succeeds, "valuable learning signal" on the hardest cases11 (§6.2.8). REINFORCE++ proposes dividing by the std of the whole batch instead (Table 1) and argues the per-prompt form is a biased estimator8 (abstract, App. A). Module 11 weighs the options.
  4. Scale invariance. Dividing by the std makes the advantage invariant to \(r\mapsto ar+b\) for \(a>0\) (derived). The other forms scale with the reward; TRL's docs warn that without it "update magnitudes depend directly on the raw reward scale and batch composition"13. That matters when a reward is a sum of components such as correctness and format (Module 10).
  5. Uniform groups carry nothing. At \(k=0\) or \(k=G\) the numerator is zero. With the \(10^{-4}\) added to the denominator the advantage is exactly 0 rather than NaN14; how often that happens as a function of \(\hat p\) and \(G\) is Module 2's Table 6.

4.3 Online RFT against GRPO

DeepSeekMath's own comparison is between GRPO and Online RFT, which samples from the live policy and trains on correct answers only. Its coefficient is the indicator of correctness (1 or 0), whatever the rest of the group did2 (App. A.1.2-A.1.3, Eqs. 10-11). GRPO's coefficient adapts to the group; for a 0/1 reward with population std, a correct sample gets \(\sqrt{(1-\hat p)/\hat p}\) and an incorrect one \(-\sqrt{\hat p/(1-\hat p)}\) (derived). Figure 4 draws both.

Gradient coefficient per sample against group pass rate for Online RFT, the unnormalized group-mean advantage, and GRPO with standard-deviation normalization, for correct and incorrect samples coefficient on ∇log π, per sample 3210−1−2−3 0.10.30.50.70.9 group pass rate (fraction of the G samples that are correct) +2.65 at k = 1 of 8 −2.65 at k = 7 of 8 Online RFT: 1 (correct), 0 (wrong) R − mean GRPO, ÷ population std solid: correctdashed: wrong
Figure 4. Per-sample coefficient on \(\nabla\log\pi\) for a 0/1 reward as the group's pass rate varies (derived from the definitions, with the population standard deviation; for \(G=8\) these advantages are 6.9% larger than the sample-std values of Table 3). Online RFT never penalizes a failure; the group forms always do. The std-normalized form pushes the minority sample hard, up to ±2.65 when one of eight differs, against ±0.875 for R − mean. DeepSeekMath's Figure 5 reports GRPO ahead of Online RFT and process supervision ahead of outcome supervision on DeepSeekMath-Instruct 1.3B2 (§5.2.1).

DeepSeekMath reads its result as evidence for "altering positive and negative gradient coefficients": GRPO beat Online RFT on GSM8K and MATH, and GRPO with process supervision beat GRPO with outcome supervision, "indicating the benefits of using fine-grained, step-aware gradient coefficients"2 (§5.2.1, Figure 5). The figure shows what the negative side costs. A negative coefficient lowers a completion's likelihood without saying where the mass goes (Module 2 §3.2), and the std-normalized form makes that push largest when failures are rare.

5. DeepSeekMath-RL, and what later recipes changed

5.1 The original settings, and the same knobs in later reports

DeepSeekMath-RL starts from DeepSeekMath-Instruct 7B and trains on about 144K chain-of-thought-format questions drawn from the GSM8K and MATH portions of the SFT data, excluding all other SFT questions so that the other benchmarks stay out of domain2 (§4.2). The reward model is initialized from DeepSeekMath-Base 7B with learning rate \(2\times10^{-5}\) and trained on rule-based correctness judgments2 (§4.2, §5.2.1), so even here the reward is a learned model of a rule, not the rule itself; Module 10 returns to why R1 dropped the model. Table 4 sets those settings beside the same knobs in later critic-free recipes.

Table 4. Reported settings of critic-free recipes. "Outputs per rollout" is the number of completions generated before the optimizer runs; "updates per rollout" is how many optimizer steps consume them. "?" means the report leaves the unit open (see text); "—" means not stated. Olmo 3 removes the KL loss in §4.4.1 and DAPO in §2.3. Derived products are marked in the text.
Recipe\(G\)Outputs per rolloutUpdates per rolloutLearning rateKL \(\beta\)Clip \(\varepsilon\)Max response (tokens)
DeepSeekMath-RL 7B2641,024 ?11e-60.04—1,024
R1-Zero5168,192163e-60.001—32,768; 65,536 after step 8.2k
DAPO18168,192161e-6none0.2 / 0.2820,480
Olmo 3 7B Think15851211e-6none0.2 / 0.27232,768
Olmo 3 32B Think1581,02412e-6none0.2 / 0.27232,768
Nemotron 3 Ultra1916global batch 8,192 ?————48K, later 64K
TRL defaults138—1—0.00.2—

Three things stand out. Group size fell from 64 to 8 or 16 while responses grew from 1,024 to 20,480 to 65,536 tokens. The arithmetic of group size shows what that costs: a prompt the policy solves 10% of the time yields a mixed group with probability \(1-0.9^G-0.1^G\), which is 57% at \(G=8\), 81% at \(G=16\) and 99.9% at \(G=64\) (derived; Module 2, Table 6, has the general form), so a smaller group wastes more prompts; whether the saved decode time repays that is the budget trade Module 15 measures. The KL coefficient went from 0.04 to 0.001 to none; Module 13 treats whether that is safe. And updates per rollout went from 1 to 16 in R1-Zero and DAPO, which is where the clip starts to matter (Module 12). At the upper bounds, and reading 1,024 as outputs, DeepSeekMath's rollout is at most \(1{,}024\times1{,}024\approx1.05\) M generated tokens, R1-Zero's at most \(8{,}192\times32{,}768\approx268\) M, a factor of 256 before R1-Zero's later 65,536-token limit (derived; real lengths are far below the cap, and Module 1 uses measured means).

The "?" for DeepSeekMath is a real ambiguity. The report states "we sample 64 outputs" per question, "the training batch size is 1024" and a single update per exploration stage, and does not say whether 1,024 counts outputs or questions2 (§4.2). R1 counts outputs ("32 unique questions, resulting in a training batch size of 512")5 (§2.1). On that convention a step is 16 questions of 64 outputs, and 144K questions are 9,000 steps per epoch, which fits the 5,300- to 8,000-step axes of Figures 5 and 6 in the report; if 1,024 counted questions, a step would hold 65,536 outputs and an epoch would be about 141 steps, which does not (derived; the writer's reading of the axes).

5.2 What it bought

Table 5. DeepSeekMath-Instruct 7B before and after RL, chain-of-thought top-1 accuracy (%), from Table 5 of the report2. GSM8K and MATH are in-domain for the RL data; the other two are out of domain. Differences derived.
BenchmarkInstruct 7BRL 7BGain (points)
GSM8K82.988.2+5.3
MATH46.851.7+4.9
MGSM-zh (out of domain)73.279.6+6.4
CMATH (out of domain)84.688.8+4.2

The out-of-domain gains are the paper's point about generality: the RL data came only from GSM8K- and MATH-related SFT questions2 (§4.2). The paper's own diagnosis is more cautious. Maj@K improves with RL and Pass@K does not (temperature 0.7, K up to 64), which the authors read as RL "rendering the output distribution more robust" and "boosting the correct response from TopK rather than the enhancement of fundamental capabilities"2 (§5.2.2, Figure 7). That sentence is the seed of the sharpen-or-expand debate; Module 20 collects the later evidence.

5.3 Iterative GRPO

Algorithm 1 wraps GRPO in an outer loop: at the start of each iteration the reference model is set to the current policy; an inner loop samples a batch, sets \(\pi_{old}\leftarrow\pi_\theta\), draws \(G\) outputs per question, scores them, computes group advantages and takes \(\mu\) GRPO updates; at the end of the iteration the reward model is trained further on fresh policy samples with "a replay mechanism that incorporates 10% of historical data"2 (§4.1.4, Algorithm 1). The motivation is that "the old reward model may not be sufficient to supervise the current policy model"2 (§4.1.4). Two iterations were run and "the iterative RL significantly improves the performance, especially at the first iteration"2 (§5.2.1, Figure 6). Two choices in that loop, resetting the reference and refreshing the scorer, reappear in R1 and Kimi in changed form; Section 6 draws the first.

6. Kimi k1.5: online policy mirror descent

6.1 From a KL-regularized objective to a squared loss

Kimi k1.5 was posted on the same day as DeepSeek-R1 (22 Jan 2025) and does not use GRPO45. At iteration \(i\) it takes the current model \(\pi_{\theta_i}\) as the reference and solves a relative-entropy-regularized problem (§2.3.2, Eq. 2):

$$\max_\theta\ \mathbb E_{x}\Big[\mathbb E_{(y,z)\sim\pi_\theta}\big[r(x,y,y^*)\big]-\tau\,\mathrm{KL}\big(\pi_\theta(x)\,\Vert\,\pi_{\theta_i}(x)\big)\Big],\qquad \pi^*(y,z\mid x)=\frac{\pi_{\theta_i}(y,z\mid x)\,e^{r/\tau}}{Z}.$$

This is the optimum Module 6 derives with the reference in the role of \(\pi_{ref}\). Taking logarithms gives, for any sampled \((y,z)\), \(r-\tau\log Z=\tau\log\frac{\pi^*}{\pi_{\theta_i}}\), "which allows us to leverage off-policy data". That identity is turned into a regression of the log-ratio on the centered reward:

$$L(\theta)=\mathbb E_{(y,z)\sim\pi_{\theta_i}}\Big[\Big(r-\tau\log Z-\tau\log\frac{\pi_\theta(y,z\mid x)}{\pi_{\theta_i}(y,z\mid x)}\Big)^2\Big].$$

The expectation is over the fixed distribution \(\pi_{\theta_i}\), not over \(\pi_\theta\). The data distribution does not move while \(\theta\) does, so several gradient steps on one batch need no importance ratio and no clip; that is the structural difference from GRPO (derived from the form of \(L\); the paper says the formulation lets it use off-policy data). \(\tau\log Z\) is replaced by a sample estimate: \(\tau\log\frac1k\sum_j e^{r_j/\tau}\), or the plain mean \(\bar r\) of the \(k\) sampled rewards, which "yields effective practical results" and is justified because \(\tau\log Z\to\mathbb E_{\pi_{\theta_i}}[r]\) as \(\tau\to\infty\)4 (§2.3.2). The resulting gradient is

$$\frac1k\sum_{j=1}^{k}\Big[\nabla_\theta\log\pi_\theta(y_j,z_j\mid x)\,(r_j-\bar r)-\frac\tau2\nabla_\theta\Big(\log\frac{\pi_\theta(y_j,z_j\mid x)}{\pi_{\theta_i}(y_j,z_j\mid x)}\Big)^2\Big].$$

Write \(\ell_j=\log(\pi_\theta/\pi_{\theta_i})\) for sample \(j\). Since \(\nabla\ell_j^2=2\ell_j\nabla\log\pi_\theta\), the update is a weighted score with weight \(r_j-\bar r-\tau\ell_j\) (derived). At the start of an iteration \(\ell_j=0\) and the weight is \(r_j-\bar r\), the "R − mean" column of Table 3 with the sample included in its own mean. As the policy moves, \(-\tau\ell_j\) cancels the weight of samples whose likelihood has already risen and boosts those that have fallen, which is a restoring force with strength \(\tau\) per nat; it does the job the clip does in PPO (the writer's reading). The fixed point is \(\ell_j=(r_j-\tau\log Z)/\tau\). Because the reference changes every iteration, "we also reset the optimizer at the start of each iteration"4 (§2.3.2).

6.2 How good is the sampled mean as \(\tau\log Z\)?

Table 6. The sample estimate \(\tau\log\frac1k\sum_j e^{r_j/\tau}\) against the plain mean, for \(k=8\) samples of which 3 are correct (rewards 1,1,1,0,0,0,0,0; mean 0.375). Derived by the writer from the formulas in Kimi's §2.3.2; the report does not give \(\tau\) or \(k\).
\(\tau\)\(\tau\log\frac1k\sum e^{r_j/\tau}\)Gap to mean 0.375
0.10.9020.527
0.30.7230.348
10.4970.122
30.4150.040
100.3870.012
1000.3760.001

The two baselines agree only when \(\tau\) is large relative to the reward range. At small \(\tau\) the exact estimate approaches the best sampled reward and would give nearly every sample a negative weight, while the mean gives weights that sum to zero. Which regime Kimi runs in is not disclosed; the sentence that the mean works well is a report of practice, not a derivation (the writer's reading).

6.3 Against GRPO, and where the reference sits

The authors note that the gradient "resembles the policy gradient of (2) using the mean of sampled rewards as the baseline (Kool et al. 2019; Ahmadian et al. 2024)", with two differences: responses come from \(\pi_{\theta_i}\) rather than on-policy, and an \(\ell_2\) regularizer is applied4 (§2.3.2). Relative to GRPO there is no division by the group std, no ratio and no clip, and the KL-like term is a quadratic on the log-ratio anchored at the current iterate. The lineage continues: K2 "adopt[s] the policy optimization algorithm introduced in K1.5 [...] as the foundation"20 (§3.2.3); K2.5 departs from it with a token-level mask on the log-ratio that ignores the sign of the advantage21; K3 delegates tolerance of stale rollouts to "a per-token regularization"22 (§4.1.2). Modules 12 and 28 follow that thread.

The three recipes in this module also differ in where the KL anchor sits, which Figure 5 draws.

Three placements of the reference policy over training: fixed at the SFT model, reset every 400 steps as in R1-Zero, and reset at the start of every iteration as in iterative GRPO and Kimi k1.5 Fixed referenceπ_ref = the SFT model for the whole run Periodic resetR1-Zero: π_ref ← latest policy every 400 steps; 10,400 steps = 26 intervals 400 Per-iteration resetiterative GRPO and Kimi k1.5: π_ref ← π_θ at each iteration start (3 iterations drawn) training step
Figure 5. Where the reference policy sits. A fixed reference bounds total drift from the SFT model; a reset reference bounds drift only from the last snapshot, so the KL term stops limiting how far the run can travel. DeepSeekMath calls the reference "usually the initial SFT model"2 (§4.1.1) and resets it per iteration in Algorithm 1; R1 replaces it with the latest policy every 400 steps and explains the aim as balancing "the scope that the training policy can explore and the stability of the training"5 (Supp. A.3); Kimi uses the current iterate and resets the optimizer too4 (§2.3.2). The three iterations in the bottom lane are illustrative; the 26 intervals are 10,400 ÷ 400 (derived).

7. Side by side: what each objective assumes

Table 7. The three critic-free objectives of this module, with ReMax for reference. Entries are from the equations in the cited papers; the last two rows are the writer's reading of them.
PropertyGRPO2RLOO3Kimi k1.54ReMax1
Advantage of sample \(i\)\((r_i-\bar r)/\operatorname{std}\)\(r_i-\) mean of the other \(k-1\)\(r_i-\bar r\), then \(-\tau\ell_i\) as the policy moves\(r_i-r(\text{greedy})\)
Scales with the reward?no (invariant, up to \(\epsilon\))yesyesyes
Ratio and clipper-token ratio, clip \(\varepsilon\)nonenone; quadratic anchor of strength \(\tau\)none
KL to a reference\(\beta k_3\) in the losssequence log-ratio inside the rewardreference is the current iterate, reset each iterationreference kept for the KL penalty
Meant for several updates per rollout?yes (\(\mu\ge1\), clip)no ratio; the paper still takes 2 steps per batchyes (loss is off-policy by construction)no
Smallest useful group222 (at \(k=1\), \(r_1-\bar r=0\))1 plus one greedy decode
Reward assumptioncomparable within a promptcomparable and scaled across promptscomparable and scaled across promptscomparable and scaled across prompts

Four assumptions follow from the table and Section 4, and they are what to check before choosing among the methods.

  1. Mixed groups. GRPO, RLOO and Kimi's loss give every sample in a group whose rewards are all equal an advantage of zero. For a prompt solved with probability \(p=0.1\), seeing at least one success with probability 0.95 takes \(G\ge\ln0.05/\ln0.9\approx28.4\), so 29 samples (derived); at the \(G=8\) to 16 of Table 4 such a prompt gives a mixed group 57% to 81% of the time. A binary verifier therefore needs a base model that already solves a useful fraction of the prompts (Module 2 §7, Modules 10 and 11).
  2. Exchangeable samples. The unbiasedness argument needs the other samples to be draws from the same policy as the one being credited (Module 2 §4.2). With partial rollouts a trajectory can span several policy versions22 (§4.1.2), so the assumption holds only approximately in asynchronous systems (the writer's reading; Modules 14 and 18).
  3. One scalar per completion. Outcome-supervised GRPO gives a 30,000-token chain of thought the same advantage on every token. The process-supervised variant is the only credit refinement GRPO itself defines; the rest of the literature either accepts the bandit view or replaces the baseline (Section 8).
  4. Reward scale. Only GRPO's form ignores the reward's units. RLOO and Kimi's loss need a reward whose scale is controlled, which is easy for 0/1 verifiers and not for learned reward models or sums of components.

TRL's GRPO trainer ships a version that differs from the paper in five ways: the group std (scale_rewards="group", the default) is computed with Bessel's correction and has \(10^{-4}\) added14; loss_type="dapo" (token-level normalization over the batch rather than \(1/|o_i|\) per sequence); \(\beta=0\); an optional \(\varepsilon_{high}\) (DAPO recommends 0.28); and an importance_sampling_level of "token", with "sequence" giving GSPO behavior. Its default of one iteration per batch matches DeepSeekMath's setting13. Check the version you run; Module 11 covers the normalization choice and Module 12 the ratio.

8. The critic question: the experiments that exist

8.1 Who runs what

Among the reports that state their RL algorithm, the large reasoning runs are critic-free: R1 (GRPO)5, Qwen3 (GRPO, 3,995 query-verifier pairs)23, DeepSeek-V3.2 (GRPO, advantage \(R-\bar R\))16, DeepSeek-V4's specialists (GRPO)24, GLM-5 (GRPO with IcePop, KL removed)25, Olmo 3 (OlmoRL: GRPO plus zero-gradient filtering, token-level loss, no KL, clip-higher)15, Nemotron 3 Ultra (asynchronous GRPO)19, DAPO18 and the Kimi line20. The exceptions are Tulu 3's RLVR stage, which runs PPO with a value function initialized from a general reward model and lists value-free GRPO as future work26, and VAPO10. Adoption shows what teams made work at scale on a deadline, not where the ceiling is. I found no report in this course's source map that ablates a critic against a group baseline at its own scale, so the evidence comes from smaller controlled studies.

8.2 Four controlled comparisons

Ahmadian et al. (Section 2) found RLOO ahead of PPO, with PPO's best setting the one where the critic is only a baseline. R1's appendix compares PPO and GRPO on MATH with DeepSeek-Coder-V2-Lite: at the common default \(\lambda=0.95\) "PPO performs considerably worse than GRPO", at \(\lambda=1.0\) its performance "improves substantially, nearing that of GRPO", and the authors conclude PPO "can achieve comparable performance when appropriately tuned" but costs hyperparameter search and a second model5 (Supp. A.3). Figure 6 (right) reads off the plot.

VinePPO's controlled comparison points the other way. From one SFT checkpoint and one episode budget on DeepSeekMath 7B, tuned PPO scores 42.8 on MATH against 36.4 for GRPO and 36.8 for RLOO, and 78.9 on GSM8K against 74.8 and 75.3 (gaps to GRPO of 6.4 and 4.1 points, derived); VinePPO, which replaces the network with Monte Carlo values, reaches 46.0 and 80.19 (Figure 4). Two details matter. The authors' hyperparameters were tuned for PPO, and GRPO and RLOO were initialized from them and given "additional tuning to stabilize training" under the same episode budget (§5). And the critic was not undertrained, with explained variance of 0.7 to 0.9, yet when asked to rank five candidate next steps from one state it "performs near chance levels for most of the training" (§6.1, §7, Figure 8). A value network can be accurate on average and still useless at telling siblings apart, which is the comparison credit assignment needs. VinePPO's price is per-step cost, up to 5x PPO's on a 1.1B model and 2x on 7B, repaid by reaching PPO's peak in about 3.0x and 1.51x less wall-clock time (§6.2).

Two bar charts of MATH accuracy. Left: DeepSeekMath 7B from VinePPO, where PPO with a tuned critic reaches 42.8, above GRPO at 36.4 and RLOO at 36.8, and VinePPO reaches 46.0. Right: DeepSeek-Coder-V2-Lite from R1, where GRPO reaches about 55.4, PPO with lambda 1.0 about 53.5, and PPO with lambda 0.95 about 50.1 VinePPO: DeepSeekMath 7B, MATH Pass@1 (%) 30 35 40 45 32.8 SFT 34.9 RestEM 36.8 RLOO 36.4 GRPO 37.6 DPO+ 42.8 PPO 46.0 VinePPO axis starts at 30; same SFT start, same episode budget R1 Supp. A.3: MATH accuracy (%) Coder-V2-Lite, step about 2,400, read off the plot 48 50 52 54 56 ≈50.1 PPO λ=0.95 ≈53.5 PPO λ=1.0 ≈55.4 GRPO axis starts at 48
Figure 6. Two controlled comparisons of a critic against group baselines, on different bases. Left: Pass@1 on MATH after RL from the same SFT checkpoint of DeepSeekMath 7B (orange: critic-free; blue and cyan: a value network and Monte Carlo values; grey: other baselines)9 (Figure 4). Right: PPO at the common GAE \(\lambda=0.95\) and at \(\lambda=1.0\) against GRPO5 (Supp. Figure 4); the values are my readings of the plotted markers at the last step, accurate to about ±0.3. Both axes are truncated. The two papers tune different arms, so the panels do not agree on which method is ahead.

VAPO tests the long-chain-of-thought regime on Qwen2.5-32B with no SFT data. Plain PPO scores 5 on AIME 2024: the value model collapses and responses shrink until the model answers without reasoning10 (§5.2). The authors name three failure modes of value models in this regime. A value model initialized from a reward model starts with a positive bias. A fixed GAE \(\lambda=0.95\) multiplies the terminal reward by \(0.95^{100}\approx0.006\) for a token 100 steps from the end, so values become almost entirely bootstrapped (§3.1, §4.2). And binary verifier rewards are sparse. Their seven modifications include value pretraining, decoupled GAE (\(\lambda=1.0\) for the critic's targets), a length-adaptive policy \(\lambda=1-1/(\alpha l)\) with \(\alpha=0.05\), Clip-Higher and a token-level loss (§4, §5.1). The result is 60.4 at best against DAPO's 50 and R1-Zero-Qwen-32B's 47 (Figure 7).

Horizontal bars of AIME 2024 avg@32 on Qwen2.5-32B: vanilla PPO 5, VAPO ablations from 11 to 55, DeepSeek-R1-Zero-Qwen-32B 47, DAPO 50 and full VAPO 60 AIME 2024 avg@32 (points), Qwen2.5-32B base 0 10 20 30 40 50 60 Vanilla PPO 5 VAPO without value pretraining 11 VAPO without decoupled GAE 33 VAPO without length-adaptive GAE 45 VAPO without Clip-Higher 46 DeepSeek-R1-Zero-Qwen-32B (GRPO) 47 DAPO (critic-free) 50 VAPO without token-level loss 53 VAPO without positive-example LM loss 54 VAPO without group sampling 55 VAPO (full) 60 blue: value-model-based; orange: critic-free; green: full VAPO
Figure 7. What a critic needed to win on long chain of thought. VAPO Table 1: removing any one of its seven modifications costs 5 to 49 points, and plain PPO collapses to 510. The R1-Zero-Qwen-32B and DAPO bars are the numbers VAPO quotes for the critic-free methods; DAPO reached 50 with 50% of R1-Zero-Qwen-32B's update steps, and VAPO reports 60.4 at its best, within 5,000 steps10 (§5.2). The paper counts steps, not GPU-hours, so the critic's per-step cost is not in the comparison.

8.3 Reading the four together

The results do not conflict as much as they look. First, "PPO" is not one method: on one base model its score ranges from 5 to 60 depending on seven choices (Figure 7), while the critic-free arms cluster (RLOO 36.8 and GRPO 36.4 in VinePPO's study). Group baselines are robust by default; critics reward engineering. Second, horizon: VinePPO caps responses at 1,024 tokens (App. C) and R1's appendix does not state the cap for its experiment, while VAPO documents failure modes that appear when the response is long and the reward sparse. Third, accounting: VAPO reports steps, VinePPO equal episodes, and no study charges the critic's forward and backward passes against the group method's extra samples. Fourth, each paper tunes the arm it favors. The Monte Carlo result also shows that the useful ingredient may be credit assignment itself, not the network: GiGPO gets step-level groups from repeated environment states with no critic and no extra rollouts, reporting gains of more than 12% on ALFWorld and 9% on WebShop over GRPO27 (abstract; Module 18).

Where experts disagree

The question is whether PPO's learned value model earns its cost on LLM post-training. The sources agree on the arithmetic of what a critic costs and disagree about what it buys once it is built well, and the experiments do not share a protocol.

A. A critic is unnecessary; spend the memory on samples

Ahmadian et al.3, Shao et al.2, the DeepSeek-R1 appendix5, Kimi k1.54, Li et al. (ReMax)1

Rewards arrive at the last token, transitions are deterministic and the initial policy is strong, so a per-token value model adds cost and bias for little variance reduction. A group baseline is the same estimate computed on the fly.

  • One fewer policy-sized model: 46% of GPU memory at 7B in ReMax's accounting1.
  • Robust defaults. PPO needed \(\lambda=1.0\) to approach GRPO in R1's appendix5, and nearly every large open reasoning recipe in Section 8.1 chose it.
  • Credit is per sequence. A tuned critic beat GRPO and RLOO by 3.6 to 6.4 points on 7B MATH and GSM8K9.
  • Uniform groups give zero gradient, so it needs a base model with a useful pass rate and several samples per prompt (Section 7).

B. A well-built value model has the higher ceiling

ByteDance Seed (VAPO)10; Tulu 3's RLVR ablation26; Xu et al.28

Value models give precise credit assignment, lower-variance per-token estimates and generalization across samples; VAPO argues critic-based methods "possess a higher performance ceiling if the challenges in training value models can be addressed"10 (§1). Tulu 3 found a value function initialized from a general reward model worked best and left GRPO untested.

  • 60.4 against DAPO's 50 on the same base, with no crashes across three runs10.
  • Xu et al. find a carefully configured PPO ahead of DPO and other alignment methods on every testbed they tried, though against DPO rather than against group baselines28.
  • It took seven modifications; plain PPO scored 5, and dropping value pretraining left 11 (Figure 7).
  • The comparison counts steps, not GPU-hours, and I found no independent replication in the source map.

C. Credit assignment matters, but estimate it by sampling

Kazemnejad et al. (VinePPO)9; Feng et al. (GiGPO)27; DeepSeekMath's process supervision2

The value network is the weak part, not the idea of per-step credit: it ranked sibling steps near chance for most of training, while Monte Carlo values from the same states are unbiased.

  • 46.0 against PPO's 42.8 on 7B MATH, with no extra model9.
  • GiGPO's anchor-state groups add step-level credit with no critic and no extra rollouts27.
  • VinePPO's steps cost up to 5x PPO's, and its evidence is 1.1B and 7B models with responses capped at 1,024 tokens9 (App. C).
  • Process supervision needs step labels, which R1 reported as hard to define and easy to hack (Module 4).
Assessment

These positions answer different questions, and the conditions decide. With a verifier, eight or more samples per prompt, responses up to a few thousand tokens and a limited engineering budget, a group baseline is the right default: it was about 2 points ahead of a \(\lambda\)-tuned critic in R1's test (reading the plot) and 3.6 to 6.4 points behind a tuned critic in VinePPO's, and it has fewer ways to fail. If you can afford a policy-sized critic and replicate VAPO's seven fixes for long chain of thought, there is one documented 10-point advantage over DAPO, unreplicated and counted in steps. Where the bottleneck is credit across many steps of an agent episode, sampling-based credit is the more promising direction (Module 18). What would settle it: one base model, one dataset and DAPO- and Dr. GRPO-grade fixes applied to the group arm and VAPO-grade fixes to the critic arm, at equal GPU-hours including the critic, at two response lengths, over several seeds, reporting pass@1 and pass@k. Nothing in the sources does this. The conclusion that critic-free methods dominate practice is about what teams shipped, not about what is optimal. A second dispute, whether to divide the group advantage by its standard deviation, is Module 11's.

What is changing

The group baseline is staying while its surroundings are stripped. DeepSeek-V3.2 and Olmo 3 dropped the division by the standard deviation1615, and DAPO, GLM-5 and Olmo 3 dropped the KL term182515; the clip is being replaced or reshaped (Module 12). Several 2026 recipes use RL to train specialists and consolidate them afterwards: DeepSeek-V4 runs GRPO per domain and replaces the mixed RL stage with on-policy distillation24, and Lambert records Kimi K3 consolidating nine experts while noting that labs "are far from unanimous" on the step11 (§3.2.4; Modules 21 and 22). The critic question has moved to agentic and multi-turn settings, where group baselines have the least to offer per step and credit assignment is the live problem (Module 18). TRL's defaults have moved from the paper's form to a DAPO-style loss with no KL, so a "GRPO" result is only as specific as its version and flags.

References

  1. paperLi, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., Luo, Z.-Q. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML 2024; arXiv v4, 16 May 2024. https://arxiv.org/abs/2310.10505. Accessed 2026-10-27. Used for: the greedy-decode baseline (§4.2, Eq. 9, Algorithm 1), the 46% share of GPU memory taken by the value model and its training state at 7B and 4% by the reward model (footnote 1, Figure 2), the 1.6x per-iteration speed-up. Added beyond the source map.
  2. paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the GRPO objective, the KL estimator and the PPO comparison (§4.1.1, Eqs. 1-4, Figure 4); outcome and process supervision (§4.1.2-§4.1.3); iterative GRPO and Algorithm 1 (§4.1.4); training setup and Table 5 (§4.2); the unified paradigm, Online RFT versus GRPO, iterative RL and Maj@K versus Pass@K (§5.2, Figures 5-7); gradient coefficients (App. A.1, Eqs. 5-21).
  3. paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: the single-action argument (§1, §3.3), REINFORCE and RLOO (§2.2-§2.3), the GAE-λ and clipping ablations (§3.1-§3.2, Figure 1), win-rates and fluency (§5.2, Tables 1-2), KL and noise robustness (§5.2.2, Figures 5-6), probability-mass concentration (App. A), hyperparameters (App. C).
  4. paperKimi Team (Du, A., Gao, B., Xing, B. et al.). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: the online policy mirror descent objective, its surrogate loss, gradient and sampled baseline (§2.3.2, Eqs. 2-3), the argument against value functions (§2.3.2), the length reward (§2.3.3).
  5. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H. et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026 (expanded Nature version, Nature 645, 633-638, 2025). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the GRPO objective as printed (§2.1, Eqs. 1-3), R1-Zero's batch, group size, learning rate, KL coefficient, reference reset and minibatch split (§2.1), and the PPO-versus-GRPO comparison, its argument about value models and KL placement, and Figures 3-4 (Supplementary A.3); v1 dated 22 Jan 2025.
  6. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the actor-critic advantage built from a learned value function (§5, Eqs. 10-11) and the clipped surrogate that GRPO keeps (§3, Eq. 7).
  7. paperSchulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016; arXiv v6, 20 Oct 2018. https://arxiv.org/pdf/1506.02438v6. Accessed 2026-10-26. Used for: GAE(γ, 1) as the empirical return minus a value baseline, unbiased whatever the accuracy of V (§3, Eqs. 17-18).
  8. paperHu, J., Liu, J. K., Xu, H., Shen, W. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Normalization. arXiv v9, 10 Nov 2025. https://arxiv.org/abs/2501.03262. Accessed 2026-10-27. Used for: the "with baseline" variant (group mean, then division by the global-batch standard deviation) and its argument that per-prompt standard-deviation normalization is a biased estimator (abstract, §1, App. A). Added beyond the source map.
  9. paperKazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., Le Roux, N. VinePPO: Refining Credit Assignment in RL Training of LLMs. ICML 2025; arXiv v2, 3 Jun 2025. https://arxiv.org/abs/2410.01679. Accessed 2026-10-27. Used for: Monte Carlo value estimates in place of a value network (§4, Eqs. 5-6), the PPO, GRPO and RLOO baselines and their tuning (§5), Pass@1 results (Figure 4), per-iteration cost and wall-clock (§6.2), value-network failure analysis (§7, Figure 8). Added beyond the source map.
  10. paperByteDance Seed (Yue, Y. et al.). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: the case for value-model-based methods (§1), the three failure modes of value models in long CoT (§3), the seven modifications and training details (§4-§5.1), and the ablation ladder (Table 1). Added beyond the source map.
  11. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: RLOO and its relation to group-relative advantages (§6.2.4, Eqs. 49-51), GRPO, Dr. GRPO and their algebraic equivalence to RLOO (§6.2.8, Eqs. 57-62), the comparison of algorithms and their importance-ratio granularity (§6.2.11, Table 3), the K3 expert count (§3.2.4), KL placement across methods (§6.2.4).
  12. blogSchulman, J. Approximating KL Divergence. Personal site, posted 2020-03-07. http://joschu.net/blog/kl-approx.html. Accessed 2026-10-26. Used for: the k3 estimator that GRPO uses for its KL term.
  13. officialHugging Face TRL maintainers. TRL documentation: GRPO Trainer (main branch). docs/trl/main; version not stated on the page. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the documented algorithm, the GRPOConfig defaults (num_generations 8, beta 0.0, epsilon 0.2, scale_rewards group, loss_type dapo, num_iterations 1) and the loss types.
  14. repoHugging Face. TRL, trl/trainer/grpo_trainer.py and trl/trainer/utils.py (main branch; last commit touching grpo_trainer.py: 63d9074, 2026-10-21). https://raw.githubusercontent.com/huggingface/trl/main/trl/trainer/grpo_trainer.py. Accessed 2026-10-27. Used for: the advantage computation: group mean, Bessel-corrected standard deviation (nanstd in utils.py) and the 1e-4 added to the standard deviation (lines 2812-2834). Added beyond the source map.
  15. paperTeam Olmo (Ettinger, A., Bertsch, A. et al.). Olmo 3. arXiv v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL as GRPO plus modifications (§4.4.1) and the RL hyperparameters of the 7B Think, 32B Think, 7B Instruct and 7B RL-Zero runs (Table 49, p. 109).
  16. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the advantage computed as the reward minus the group mean, with no division by the standard deviation, and mixed RL with GRPO (§3, §3.1).
  17. paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the response-level length bias from 1/|o_i| and the question-level difficulty bias from the standard deviation (§3.1), pointed to here and treated in Module 11.
  18. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: training details (§4.1: 512 prompts x 16 responses, 16 gradient updates per rollout, learning rate, clip range, length cap), the removal of the KL term (§2.3), the naive-GRPO baseline of 30 AIME points (§1, Table 1).
  19. paperNVIDIA (Blakeman, A., Thomas, A. et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: asynchronous GRPO, global batch 8192 with 16 rollouts per sample, 48K to 64K generation limit (§3.2).
  20. paperKimi Team (Bai, Y., Bao, Y., Charles, Y. et al.). Kimi K2: Open Agentic Intelligence. arXiv v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: K2's adoption of the K1.5 policy-optimization algorithm as its foundation (§3.2.3).
  21. paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.). Kimi K2.5: Visual Agentic Intelligence. arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: the departure from the K1.5 loss by a token-level log-ratio mask (RL algorithm section).
  22. paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.). Kimi K3: Open Frontier Intelligence. arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: staleness tolerance delegated to a per-token regularization in the loss (§4.1.2).
  23. paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: reasoning RL with GRPO on 3,995 query-verifier pairs (§4.2).
  24. paperDeepSeek-AI (Xu, A., Lin, B., Xue, B. et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv v1, id 2606.19348. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: GRPO for the specialist RL stage and on-policy distillation replacing mixed RL (§5.1, §5.1.1).
  25. paperGLM-5 Team (Zeng, A., Lv, X. et al.). GLM-5: from Vibe Coding to Agentic Engineering. arXiv v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: Reasoning RL as GRPO with IcePop and the KL term removed (§3.2).
  26. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the finding that initializing the RLVR value function from a general reward model works best (§6.2.1) and the statement that value-free methods such as GRPO were left as future work (Limitations).
  27. paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025; arXiv v3, 28 Oct 2025. https://arxiv.org/abs/2505.10978. Accessed 2026-10-26. Used for: step-level groups built from repeated environment states, with no critic (abstract).
  28. paperXu, S., Fu, W., Gao, J., Ye, W. et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: the abstract's claim that a carefully configured PPO surpasses DPO and other alignment methods on all testbeds.