Reinforcement Learning for LLM Post-Training · Module 2

Policy Gradients for Token Sequences: the Log-Derivative Trick, Baselines, and Advantages

Every policy-gradient method in this course, from PPO to GRPO and its 2025-2026 descendants, is one gradient estimator with a different answer to two questions: where does the baseline come from, and what limits the step. This module derives that estimator for an autoregressive policy, shows that it is reward-weighted SFT on the model's own samples, and works the variance arithmetic exactly. Afterwards you can write REINFORCE with a baseline from the equations and place any later method in the (baseline, step control) grid.

Course 45 min read 18 sources As of 2026-10-27 Prerequisite: Module 1
The policy-gradient family as three slots: where samples come from, what the advantage subtracts, and how the step is limited gradient = E [ Ψ · ∇θ log πθ( yₜ | x, y<ₜ ) ] Ψ = the credit each sampled token receives (a reward minus a baseline, in most methods) 1 Samples come from 2 Ψ = reward − baseline 3 Step is limited by fixed dataset (SFT, RFT) own policy, one update own policy, reused × N Sections 3 and 6 none: Ψ = R mean or leave-one-out of R group mean ÷ group std learned V(sₜ), then GAE Sections 4 and 5, Modules 3 and 9 nothing (vanilla step) one update per rollout ratio clip (PPO family) KL constraint (TRPO) Section 6 and Module 3
Figure 1. The grid used throughout the course. DeepSeekMath writes SFT, RFT, DPO, PPO and GRPO as one gradient whose pieces are a data source, a reward function and a gradient coefficient1; Schulman et al. list the six interchangeable choices of Ψ (Eq. 1)2. The row labels in each column are the options this module and Modules 3 and 9 develop.

1. Generation as a decision process

A policy-gradient method needs three things fixed before any gradient exists: what the agent observes, what it chooses, and when the reward arrives. For an autoregressive LM there are two defensible answers. The literature moves between them without announcing it, so the first job is to see exactly what they share.

1.1 The token-level MDP

Write \(x\) for the prompt, \(y=(y_1,\dots,y_T)\) for the completion, and \(\pi_\theta\) for the LLM. The state at step \(t\) is \(s_t=(x,y_{\lt t})\): the prompt plus every token generated so far. The action is the next token \(a_t=y_t\). The transition is deterministic, since \(s_{t+1}\) is \(s_t\) with the sampled token appended. The episode ends at EOS or at the length cap. The reward model's score lands on the EOS token; every earlier token carries only the per-token KL term, and the discount is \(\gamma=1\)34.

The probability of a completion is a product of softmax outputs:

$$\pi_\theta(y\mid x)=\prod_{t=1}^{T}\pi_\theta(y_t\mid x,y_{\lt t}).$$

Two things follow. The policy network is the LLM itself and the "state" is its context window; nothing is added to the architecture. And because the transition is deterministic and the prompt distribution does not depend on \(\theta\), the environment drops out of \(\nabla_\theta\log p(\tau)\): only the policy's factors survive differentiation3. That is why a unit test, a human click, or an exact-match checker can serve as a reward without being differentiable.

If you want the software picture, Gymnasium's Env.step(action) returns (observation, reward, terminated, truncated, info) and either flag ends the episode5. The mapping is the writer's, not Gymnasium's: observation is the token ids so far, action is the next token id, terminated is EOS sampled, truncated is max_new_tokens reached. The two cases are not interchangeable. A reward model scoring a truncated answer produces out-of-distribution scores, and Lambert's remedy is to score only at EOS and to penalize overlong generations3. A bootstrapping critic also has a reason to treat the two flags differently (the writer's reasoning: a truncated episode has a future the critic can still estimate, a terminated one does not). Module 17 builds the full environment interface.

1.2 The bandit view

InstructGPT states the second answer directly: the RL environment "is a bandit environment which presents a random customer prompt and expects a response to the prompt. Given the prompt and response, it produces a reward determined by the reward model and ends the episode"6. Ahmadian et al. argue this is the right abstraction for a pretrained LLM. Because "rewards are only attributed to full generations", they write, "it is more appropriate and efficient to model the entire generation as a single action"4 (§1), and their §3.3 observes that deterministic dynamics let the MDP collapse to a bandit with one initial state and one terminal state4. Lambert notes the same convention: many RLHF implementations treat the whole completion as one action, so the \((x,y)\) notation is as valid as \((s,a)\)3 (§6.1).

The action space of the bandit is every possible completion, which is astronomically large. You never enumerate it. You sample \(y\) and evaluate \(\log\pi_\theta(y\mid x)=\sum_t\log\pi_\theta(y_t\mid s_t)\), which the factorization above makes cheap.

A two-token generation tree with reward only at the end-of-sequence leaves, and the same tree collapsed into a single bandit arm Token MDP: one decision per token s₁ = x The 0.6 Let 0.4 s₂ = x, The s₂ = x, Let sum 0.5 answer 0.5 x 0.75 n 0.25 The sum ⟨eos⟩p = 0.30R = 1 The answer ⟨eos⟩p = 0.30R = 0 Let x ⟨eos⟩p = 0.30R = 0 Let n ⟨eos⟩p = 0.10R = 1 reward is zero at every interior state; p(y) is the product of edge probabilities J = 0.30·1 + 0.30·0 + 0.30·0 + 0.10·1 = 0.40 collapse Bandit: one decision state: prompt x action: whole yy ~ π(y | x) reward R(x, y) episode ends
Figure 2. The same generation problem in the two views. Left: states are prefixes, actions are tokens, the reward appears only at the ⟨eos⟩ leaf (illustrative probabilities; the structure follows InstructGPT §3.56 and Ahmadian et al. §2.1 and §3.34). Right: the whole completion is one action. The objective is identical, \(J=\sum_y p(y)R(y)\), and both views sample from the same \(\pi_\theta\).

1.3 When the two views give the same gradient

The two views are not rival algorithms. The token-level reward-to-go form of the policy gradient weights each token's score by the return from that token onward, \(G_t=\sum_{t'\ge t}r_{t'}\)2. With reward only at the end and \(\gamma=1\), \(G_t=R\) for every \(t\), so

$$\mathbb{E}\Big[\sum_{t=1}^{T}G_t\,\nabla_\theta\log\pi_\theta(y_t\mid s_t)\Big]=\mathbb{E}\Big[R\sum_{t=1}^{T}\nabla_\theta\log\pi_\theta(y_t\mid s_t)\Big]=\mathbb{E}\big[R\,\nabla_\theta\log\pi_\theta(y\mid x)\big],$$

which is the bandit form, term for term. The views differ only where something is allowed to vary across tokens. There are three such places:

  1. Per-token rewards. If KL to the reference is folded into the reward, \(G_t\) depends on \(t\). Lambert records the three conventions: PPO subtracts a per-token KL from the per-token reward before computing the advantage, RLOO sums the KL into the sequence reward, and GRPO keeps a sequence-level advantage and adds a separate per-token KL term to the loss3 (§6.2.4).
  2. A baseline that depends on the prefix. A value function \(V(s_t)\) gives each token its own advantage (Section 5).
  3. Bootstrapping. A TD target built from \(V(s_{t+1})\) instead of the observed return, for example GAE with \(\lambda\lt1\) (Section 5 and Module 3).
Table 1. The two framings of LLM generation and what each lets you vary. Sources: Lambert §3.1.3 Table 1 and §6.3.2.23; Ahmadian et al. §2.1-§3.34; InstructGPT §3.56.
AspectToken-level MDPBandit (one action per completion)
State\((x,y_{\lt t})\), changes every token\(x\) only
ActionOne vocabulary tokenThe whole completion \(y\)
RewardZero until EOS; per-token KL optionalOne scalar \(R(x,y)\), KL summed in if used
Discount\(\gamma=1\) in practiceNot applicable
Advantage granularityPer token (needs \(V(s_t)\) or per-token rewards)One scalar, broadcast to all tokens
Typical usersPPO with a learned value modelREINFORCE, RLOO, GRPO-style advantages

2. The policy gradient for a sequence

2.1 The log-derivative trick

The objective is the expected reward of completions sampled from the current policy:

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\,\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\big[R(x,y)\big]=\sum_x p(x)\sum_y \pi_\theta(y\mid x)\,R(x,y).$$

Differentiating, the reward is a constant with respect to \(\theta\); only the sampling distribution moves. The result \(\sum_y R\,\nabla_\theta\pi_\theta\) is not an expectation under \(\pi_\theta\) because it has no \(\pi_\theta\) factor to sample from. The identity \(\nabla\log u=\nabla u/u\) supplies one. Sutton and Barto introduce the factor by "multiplying and then dividing" by \(\pi(a\mid S_t,\theta)\)7 (§13.3, p. 327), and Lambert derives the same line in trajectory form3 (§6.2.1, Eqs. 36-38):

$$\nabla_\theta J=\sum_x p(x)\sum_y \pi_\theta(y\mid x)\,R(x,y)\,\nabla_\theta\log\pi_\theta(y\mid x)=\mathbb{E}\big[R(x,y)\,\nabla_\theta\log\pi_\theta(y\mid x)\big].$$

Expanding the log of the product from Section 1 turns the sequence score into a sum of token scores:

$$\nabla_\theta J=\mathbb{E}_{x,\;y\sim\pi_\theta}\Big[R(x,y)\sum_{t=1}^{T}\nabla_\theta\log\pi_\theta(y_t\mid x,y_{\lt t})\Big].\tag{PG}$$

This is Williams' REINFORCE result. His Theorem 1 states that for an update of the form \(\Delta w=\alpha\,(r-b)\,e\), where \(e=\partial\ln g/\partial w\) is the "characteristic eligibility", the expected update lies along the gradient of expected reinforcement8 (§4, Theorem 1). His §5 handles the case where "a single reinforcement value \(r\) is delivered" at the end of an episode of \(k\) steps by summing the eligibilities over time8 (§5). That is the LLM setting verbatim: one reward per completion, one eligibility term per token.

The Monte Carlo estimator draws \(N\) prompt-completion pairs and averages:

$$\hat g=\frac1N\sum_{i=1}^{N}R_i\sum_{t=1}^{T_i}\nabla_\theta\log\pi_\theta(y_{i,t}\mid s_{i,t}).$$

2.2 The score has zero mean

One identity carries most of the later arguments. For any prompt,

$$\mathbb{E}_{y\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(y\mid x)\big]=\sum_y\nabla_\theta\pi_\theta(y\mid x)=\nabla_\theta\sum_y\pi_\theta(y\mid x)=\nabla_\theta 1=0.$$

The same holds per token, conditioned on the prefix: \(\mathbb{E}_{y_t\sim\pi_\theta(\cdot\mid s_t)}[\nabla_\theta\log\pi_\theta(y_t\mid s_t)]=0\). Sutton and Barto use exactly this step to show a baseline leaves the update unchanged7 (§13.4, p. 329). Section 4 uses it twice: once to license baselines, once to bound the variance of the sequence score.

2.3 The loss that autodiff differentiates

Nobody codes (PG) as an explicit gradient. PPO's paper describes the standard construction: build an objective whose gradient is the estimator and let autodiff differentiate it, \(L^{PG}(\theta)=\hat{\mathbb{E}}_t[\log\pi_\theta(a_t\mid s_t)\hat A_t]\)9 (§2.1, Eqs. 1-2). For a bandit-style advantage, the snippet is short enough to write in full; Lambert gives the same two lines in §6.2.13.

import torch
import torch.nn.functional as F

def reinforce_loss(logits, tokens, mask, adv):
    # logits: (B, L, V) policy logits, aligned so logits[:, t] scores tokens[:, t]
    # tokens: (B, L) sampled ids; mask: (B, L) 1.0 on completion tokens, 0.0 on prompt/padding
    # adv:    (B,)   one scalar per completion (reward minus baseline)
    logp = F.log_softmax(logits.float(), dim=-1).gather(-1, tokens.unsqueeze(-1)).squeeze(-1)
    seq_logp = (logp * mask).sum(dim=-1)          # log pi(y | x) = sum of token log-probs
    return -(adv.detach() * seq_logp).mean()      # descending this ascends E[A * log pi]

def loo_advantage(rewards):                        # rewards: (P, K), K samples per prompt
    K = rewards.shape[1]
    baseline = (rewards.sum(dim=1, keepdim=True) - rewards) / (K - 1)
    return rewards - baseline

I checked the gradient of reinforce_loss against a token-by-token autograd sum on a toy table model; the maximum absolute difference was \(3\times10^{-8}\) in fp32.

Computation graph of the sequence-level policy gradient: token log-probabilities are summed and multiplied by one detached scalar advantage per completion sampled x, y(B, L) ids πθ forward(B, L, V) log π(yₜ|sₜ)(B, L) × mask Σₜ log πlog π(y|x), (B,) × A(B,) −meanscalar loss backward: ∇θ flows only through log πθ reward Rverifier or RM baseline bother samples A = R − bdetached stop-gradient on A no ∇ through the reward B = completions per batch, L = padded length, V = vocabulary; bottom row uses no model parameters
Figure 3. One REINFORCE step as a computation graph. The top row is the only differentiable path; the advantage is built from rewards alone and enters as a per-sequence constant, as in PPO's surrogate construction (§2.1)9 and Lambert's loss snippet (§6.2.1)3.

3. REINFORCE is SFT with a signed, per-sequence weight

You already know the SFT gradient: average the gradient of each token's log-likelihood over human-written data. DeepSeekMath writes it as \(\nabla_\theta\mathcal{J}_{SFT}=\mathbb{E}_{(q,o)\sim P_{sft}}\big[\tfrac1{|o|}\sum_t\nabla_\theta\log\pi_\theta(o_t\mid q,o_{\lt t})\big]\), with a "gradient coefficient" that is "always set to 1"1 (App. A.1.1, Eq. 7). Equation (PG) has the same form. Three things change: the sequences come from the current policy instead of a dataset, each sequence carries a scalar weight, and the weight can be negative.

Table 2. One gradient, \(\nabla_\theta\mathcal{J}=\mathbb{E}\big[\tfrac1{|o|}\sum_t GC(q,o,t)\,\nabla_\theta\log\pi_\theta(o_t\mid q,o_{\lt t})\big]\), with different data sources and coefficients. Rows SFT, RFT, Online RFT, PPO, GRPO and DPO are from DeepSeekMath Table 10 and App. A.1 (Eqs. 7, 10, 11, 14, 18, 21)1; the two REINFORCE rows follow the same template and are derived here from (PG).
MethodSamples come fromRewardGradient coefficient \(GC\)
SFTHuman data, fixednone (human selection)\(1\)
RFTSFT model, sampled oncerule\(\mathbb{1}[o\text{ correct}]\)
Online RFTlive policy \(\pi_\theta\)rule\(\mathbb{1}[o\text{ correct}]\)
REINFORCE (derived)live policyany scalar\(R(q,o)\)
REINFORCE + baseline, RLOO (derived)live policyany scalar\(R(q,o)-b(q)\)
PPO\(\pi_{\theta_{old}}\)reward model\(A_t\) from GAE with a learned \(V_\psi\)
GRPOgroup \(\{o_i\}\sim\pi_{\theta_{old}}\)reward model\(\hat A_{i,t}+\beta\big(\tfrac{\pi_{ref}}{\pi_\theta}-1\big)\)
DPOSFT model, preference pairspreference or rule\(\sigma\big(\beta\log\tfrac{\pi_\theta(o^-_t)}{\pi_{ref}(o^-_t)}-\beta\log\tfrac{\pi_\theta(o^+_t)}{\pi_{ref}(o^+_t)}\big)\)

3.1 Binary-reward REINFORCE is online rejection-sampling SFT

Take \(R\in\{0,1\}\), no baseline, samples from the live policy. Then \(\hat g=\frac1N\sum_i R_i\nabla_\theta\log\pi_\theta(y_i\mid x_i)=\frac1N\sum_{i:R_i=1}\nabla_\theta\log\pi_\theta(y_i\mid x_i)\): the failures contribute exactly zero and the successes are an SFT batch (derived). RFT trains once on a frozen set of such filtered samples; Online RFT resamples from the current policy every step. DeepSeekMath found Online RFT comparable to RFT early and ahead later, which they attribute to the actor drifting away from the SFT model so that fresh samples differ more from the frozen ones1 (§5.2.1, Fig. 5). The only structural difference between rejection-sampling fine-tuning and the simplest policy gradient is how often the sampler sees new weights.

3.2 What a baseline adds: negative weights

Subtract the group's mean reward and the coefficients change sign. With binary rewards and group pass rate \(\hat p\), each correct sample gets \(1-\hat p\), each incorrect one gets \(-\hat p\), and the coefficients sum to zero over the group (derived). At \(\hat p=0.75\) the three successes each get \(+0.25\) and the single failure gets \(-0.75\). When most samples succeed the update is dominated by pushing down the few failures; when most fail it is dominated by pulling up the few successes.

A negative coefficient is gradient descent on that sample's log-likelihood. For a softmax the score with respect to the logits is \(\nabla_z\log\pi_a=e_a-\pi\), so pushing token \(a\) down raises every other token in proportion to its current probability: it removes mass from a completion without saying where the mass should go (derived). DeepSeekMath's Online RFT "does not penalize incorrect responses and uniformly reinforces all responses with correct answers at the same level of intensity", while GRPO "uniquely adjusts its gradient coefficient based on the reward value"; GRPO won the comparison on DeepSeekMath-Instruct 1.3B, which the authors read as evidence for "altering positive and negative gradient coefficients"1 (§5.2.1, Fig. 5).

For someone who has written a fine-tuning loop, this is the practical reading of the whole section: the RL trainer reuses the SFT loss, adds a sampler running the latest weights, and attaches a signed scalar to each sequence. The policy-gradient algorithms in Modules 3 to 12 differ from each other mainly in that scalar and in how the step is limited; choosing the scalar well is the variance problem of Section 4.

4. Baselines: what may be subtracted, and what it buys

4.1 What a baseline may depend on

Replace \(R\) in (PG) with \(R-b(x)\). The extra term has zero expectation by the identity in Section 2.2: \(\mathbb{E}_y[b(x)\nabla_\theta\log\pi_\theta(y\mid x)]=b(x)\cdot 0\). Sutton and Barto state the general rule: the baseline "can be any function, even a random variable, as long as it does not vary with \(a\)"7 (§13.4, Eq. 13.10, p. 329). Williams' definition of a REINFORCE algorithm requires the baseline to be "conditionally independent" of the unit's sampled output8 (§4), and Schulman et al. give the sequential version: \(\hat A_t=Q_t-b_t\) is unbiased when \(b_t\) is "an arbitrary function of the states and actions sampled before \(a_t\)"2 (Proposition 1).

In LLM terms, the baseline may depend on the prompt \(x\) and, token by token, on the prefix \(y_{\lt t}\). That is what a value function \(V(x,y_{\lt t})\) is. It may not depend on the token being credited, on later tokens, or on this completion's own reward. The last restriction is the one that bites in group methods (Section 4.2).

These results prove unbiasedness only. Williams is explicit that the analysis "offer[s] no basis for comparing various choices of reinforcement baseline"; the superiority of an adaptive baseline is, in his words, "generally believed"8 (§4, p. 236). The variance claim rests on later analysis and on experiments, such as Sutton and Barto's short-corridor gridworld where adding a baseline "can make it learn much faster"7 (Fig. 13.2, p. 330), and on the exact calculation in Section 4.4.

4.2 Why the group mean is biased and leave-one-out is not

Draw \(G\) completions \(y_1,\dots,y_G\) i.i.d. from \(\pi_\theta(\cdot\mid x)\) with scores \(u_i=\nabla_\theta\log\pi_\theta(y_i\mid x)\) and rewards \(R_i\). Use the plain batch mean \(\bar R=\frac1G\sum_jR_j\) as the baseline for every sample, including sample \(i\) itself. The mean depends on \(R_i\), which depends on \(y_i\), so the legality condition fails. The damage is computable. Writing \(R_i-\bar R=(1-\tfrac1G)R_i-\tfrac1G\sum_{j\ne i}R_j\) and using independence and \(\mathbb{E}[u_i]=0\):

$$\mathbb{E}\big[(R_i-\bar R)\,u_i\big]=\Big(1-\frac1G\Big)\mathbb{E}[R_i u_i]-\frac1G\sum_{j\ne i}\mathbb{E}[R_j]\,\underbrace{\mathbb{E}[u_i]}_{0}=\frac{G-1}{G}\,\nabla_\theta J.$$

The estimator points the right way and is short by a factor \((G-1)/G\): 75% of the true gradient at \(G=4\), 50% at \(G=2\), 94% at \(G=16\) (derived; the enumeration in Section 4.4 reproduces 0.75 exactly). The leave-one-out baseline \(b_i=\frac1{G-1}\sum_{j\ne i}R_j\) does not depend on \(y_i\), so it is unbiased; Ahmadian et al. describe it as using the other \(k-1\) samples "to create an unbiased estimate of the expected return for the prompt, akin to a parameter-free value-function, but estimated at each training step"4 (§2.3). The two advantages differ by a constant:

$$R_i-b_i=\frac{G}{G-1}\,\big(R_i-\bar R\big),$$

which is Lambert's Eq. 513 (§6.2.4). For fixed \(G\) the bias is a learning-rate rescaling, not a change of direction; it matters when \(G\) is small or differs between prompts. Lambert makes the related observation that Dr. GRPO's advantage equals RLOO's up to a constant factor3 (§6.2.8). Modules 9 and 11 take this thread further.

4.3 Where the variance comes from

Two sources: the spread of \(R\), which the baseline attacks, and the noise of the score itself, which it does not. For the second, write \(S=\sum_{t=1}^{T}u_t\) with \(u_t=\nabla_\theta\log\pi_\theta(y_t\mid s_t)\). Each \(u_t\) has zero mean given the prefix (Section 2.2), and \(u_v\) for \(v\lt t\) is a function of the prefix, so the cross terms vanish: \(\mathbb{E}[u_v^\top u_t]=\mathbb{E}\big[u_v^\top\,\mathbb{E}[u_t\mid y_{\lt t}]\big]=0\). Hence

$$\mathbb{E}\|S\|^2=\sum_{t=1}^{T}\mathbb{E}\|u_t\|^2,$$

which grows linearly in \(T\) (derived). I checked it by Monte Carlo on a toy bigram table (vocabulary 8, 2,000 samples per length); the ratio of \(\mathbb{E}\|S\|^2\) to \(\sum_t\mathbb{E}\|u_t\|^2\) came out between 0.988 and 1.007 for \(T=4,8,16,32\). For a fixed advantage magnitude, a completion's gradient norm therefore grows roughly like \(\sqrt T\): a 10,000-token chain of thought is a noisier sample than a 500-token answer. A per-token critic is one response to this, and the length-normalization choices of Module 11 are another.

4.4 A worked example, exactly

Figure 4 walks one prompt through a group update. The numbers are illustrative; the arithmetic is what matters.

y₁ y₂ y₃ y₄ completion tokens R 1 0 1 1 LOO baseline bᵢ 0.67 1.00 0.67 0.67 group mean R̄ = (1 + 0 + 1 + 1) / 4 = 0.75 A = R − R̄A = R − bᵢ +0.25+0.33 −0.75−1.00 +0.25+0.33 +0.25+0.33 both columns sum to zero; the LOO column is 4/3 times the mean column green: log-prob of the token is pushed upred: log-prob of the token is pushed down
Figure 4. One group update on one prompt, step by step (illustrative lengths and rewards). The advantages are \(R-\bar R\) and the leave-one-out form of Lambert Eqs. 49-513; broadcasting one sequence-level scalar over every token is the bandit-style advantage of Lambert §6.3.2.23.
  1. Sample. One prompt, \(G=4\) completions from the current policy, of lengths 5, 7, 4 and 6 tokens (illustrative).
  2. Score. A verifier returns \(R=1,0,1,1\): pass rate 0.75 in this group.
  3. Baseline. For \(y_1\) the other three rewards are \((0,1,1)\), so \(b_1=2/3\); for \(y_2\) they are \((1,1,1)\), so \(b_2=1\). The plain group mean is 0.75 for everyone.
  4. Advantage. Subtracting gives \((+0.25,-0.75,+0.25,+0.25)\) with the group mean and \((+0.33,-1.00,+0.33,+0.33)\) leave-one-out. Without any baseline the weights would be \((1,0,1,1)\): the failure would be ignored instead of pushed down.
  5. Broadcast. Every token of a completion receives that completion's scalar. The weighted sum of token scores is the gradient estimate; the optimizer moves the weights along it.

The stepper shows the advantages but not their effect on variance. For that, take a policy small enough to enumerate. Prompt A has three possible completions with probabilities \((0.2,0.3,0.5)\); the first two are correct, so the pass rate is 0.5. Prompt B has probabilities \((0.1,0.4,0.5)\) and only the first is correct: pass rate 0.1. The policy is a softmax over three logits, so the score of completion \(a\) is \(e_a-\pi\). An estimator averages \(G=4\) i.i.d. single-sample terms. All \(3^4=81\) outcomes are enumerated with their probabilities, so every mean and variance below is exact (derived, not simulated). I report the signal-normalized noise \(\operatorname{tr}\operatorname{Cov}(\hat g)/\lVert\mathbb{E}\hat g\rVert^2\), which is comparable across estimators with different expected step lengths.

Table 3. Four-sample gradient estimators on the toy softmax policy (exact enumeration, derived by the writer). "Step" is \(\mathbb{E}\hat g\) projected on the true gradient, as a fraction of it; "tr Cov" is the trace of the covariance in logit space; "N/S" is tr Cov divided by \(\lVert\mathbb{E}\hat g\rVert^2\). Oracle \(V(x)\) uses the true expected reward and is not available in practice.
EstimatorStep (fraction of \(\nabla J\))tr Cov, A (0.5)tr Cov, B (0.1)N/S, AN/S, B
No baseline, \(b=0\)1.000.08380.02740.882.25
Group mean incl. self0.750.02020.01580.382.31
Leave-one-out1.000.03580.02820.382.31
Oracle \(V(x)\)1.000.01500.02280.161.87
Noise-to-signal ratio of four-sample policy-gradient estimators on a toy softmax policy, for a prompt with pass rate 0.5 and one with pass rate 0.1 0 0.5 1 1.5 2 2.5 tr Cov(ĝ) ÷ ‖E ĝ‖² (lower is better; G = 4 samples, exact enumeration) 0.88 none 0.38 mean/LOO 0.16 oracle V prompt A: pass rate 0.5 2.25 none 2.31 mean/LOO 1.87 oracle V prompt B: pass rate 0.1
Figure 5. The noise-to-signal column of Table 3 for the unbiased estimators (the group mean and leave-one-out coincide after rescaling by \(G/(G-1)\)). At pass rate 0.5 a baseline cuts relative noise by 2.3x (leave-one-out) to 5.6x (oracle). At pass rate 0.1 nothing helps much: even the oracle only moves 2.25 to 1.87. Derived by exhaustive enumeration of the toy policy described in the text.

Three readings of Table 3 matter for everything that follows.

  • The bias is real and exact. The group mean including the sample itself delivers 0.75 of the true gradient at \(G=4\), as Section 4.2 predicted. After multiplying by \(G/(G-1)=4/3\) its covariance becomes the leave-one-out covariance: \(0.0202\times(4/3)^2=0.0359\), against 0.0358 (rounding).
  • A baseline helps most when the mean reward is far from zero relative to its spread. At pass rate 0.5, leave-one-out cuts noise-to-signal from 0.88 to 0.38. At pass rate 0.1, the no-baseline estimator is already close to centered, because most samples have \(R=0\) and contribute nothing; leave-one-out at \(G=4\) is marginally worse than no baseline (2.31 against 2.25); with only three other samples, its baseline is itself a noisy estimate (the writer's reading).
  • Noise-to-signal above 1 means a single step is mostly noise. With rare successes, a group of four usually contains none: the chance that prompt B's group has a success is \(1-0.9^4=34\%\). Averaging \(N\) independent groups divides noise-to-signal by \(N\) (derived), which is the arithmetic behind large prompt batches and larger \(G\) on hard prompts. Section 7 returns to this.

4.5 The menu of credit signals, and why advantage is the target

Schulman et al. list six interchangeable weights for the score term in the policy gradient, all with the same expectation when exact2 (Eq. 1). Table 4 instantiates each for an LLM with deterministic transitions and \(\gamma=1\). For a deterministic policy-transition pair, \(Q(s_t,a_t)=r_t+V(s_{t+1})\), so the advantage equals the TD residual in the last row3 (§6.2.1, Eq. 43).

Table 4. The six choices of \(\Psi_t\) in \(g=\mathbb{E}[\sum_t\Psi_t\nabla_\theta\log\pi_\theta(a_t\mid s_t)]\) (Schulman et al., Eq. 1)2 with the LLM instantiation (writer's mapping; Lambert §6.2.1 lists the same set3).
#\(\Psi_t\)LLM instantiationNeeds a learned critic
1\(\sum_{t=0}^{\infty}r_t\), total reward\(R(x,y)\)no
2\(\sum_{t'\ge t}r_{t'}\), reward after \(a_t\)\(R\) if only EOS pays; differs once per-token KL rewards existno
3\(\sum_{t'\ge t}r_{t'}-b(s_t)\)\(R-b(x)\) (bandit baselines) or \(R-V(s_t)\)only if \(b=V(s_t)\)
4\(Q^\pi(s_t,a_t)\)\(r_t+V^\pi(s_{t+1})\)yes
5\(A^\pi(s_t,a_t)\)\(Q^\pi-V^\pi\)yes
6\(r_t+V^\pi(s_{t+1})-V^\pi(s_t)\), TD residual\(V(s_{t+1})-V(s_t)\) for interior tokensyes

Of the six, advantage is the target. The GAE paper says the choice \(\Psi_t=A^\pi(s_t,a_t)\) "yields almost the lowest possible variance" and gives the intuition: a gradient step "should increase the probability of better-than-average actions and decrease the probability of worse-than-average actions"2 (§2). The same quantity is what makes a policy update provably useful. TRPO's performance-difference identity, \(\eta(\tilde\pi)=\eta(\pi)+\mathbb{E}_{\tilde\pi}[\sum_t\gamma^tA_\pi(s_t,a_t)]\), says any update with non-negative expected advantage at every state cannot reduce performance10 (§2, Eq. 1). Module 3 builds the trust-region machinery on that identity. The practical question for LLM RL is how cheaply a usable estimate of \(A\) can be had, and that is where the algorithms diverge: a learned \(V\) (Section 5), or other samples from the same prompt (Modules 9 and 11).

5. Value functions when the reward arrives at the last token

5.1 What \(V\) means for an LLM, and what it redistributes

With \(\gamma=1\) and a reward only at EOS, the value of a prefix is the expected final reward from it under the current policy: \(V^\pi(x,y_{\lt t})=\mathbb{E}[R\mid x,y_{\lt t}]\). For a 0/1 verifier this is the probability that the finished answer will be correct. Transitions are deterministic and interior rewards are zero, so \(Q(s_t,a_t)=V(s_{t+1})\) and the advantage of a token is the change in predicted success probability that the token caused3 (§6.2.1, Eq. 43):

$$\delta_t=r_t+V(s_{t+1})-V(s_t),\qquad \delta_t=V(s_{t+1})-V(s_t)\ \ (t\lt T),\qquad \delta_T=R-V(s_T).$$

Summing over the sequence telescopes: \(\sum_{t=1}^{T}\delta_t=R-V(x)\) for any function \(V\) with \(V=0\) at the terminal state (derived). A critic therefore does not create or destroy credit. It redistributes the same total, \(R-V(x)\), along the sequence. If \(V\) is accurate, the tokens where the success probability jumped receive most of it and filler tokens receive almost none. A critic-free method gives every token the same weight, \(R-b(x)\); the filler tokens' updates are then zero-mean noise that averages out only over many samples (Section 4.3).

This is also the line between a baseline and a critic. Sutton and Barto: a learned state-value function used only as a baseline, with Monte Carlo targets, is REINFORCE with baseline and is unbiased; it becomes an actor-critic only when the value function is used for bootstrapping, and "only through bootstrapping do we introduce bias and an asymptotic dependence on the quality of the function approximation"7 (§13.5, p. 331). GAE with \(\lambda=1\) is the first case: "simply the empirical returns minus the value function baseline", unbiased "regardless of the accuracy of \(V\)" but high variance2 (§3). Any \(\lambda\lt1\) is the second. Lambert's simplest PPO critic uses Monte Carlo return targets with no GAE, which makes it the first kind3 (§6.2.7).

5.2 Why an accurate per-token \(V\) is hard to get

DeepSeekMath gives two objections. The value function "is typically another model of comparable size as the policy model", a "substantial memory and computational burden"; and because "usually only the last token is assigned a reward score", training a value function "that is accurate at each token" may be complicated1 (§4.1.1). Lambert adds that learning a value function from an LM backbone is a setting "where research hasn't established best practices"3 (§6.2.8), and that initialization is an open design choice: InstructGPT starts the value network from the reward model, Tulu 3 follows it for RLVR, and others append a randomly initialized value head to the SFT checkpoint3 (§6.3). The initialization matters empirically: in Tulu 3's RLVR ablations, "Initializing RLVR's Value Function from a General RM Works Best"11 (§6.2.1). InstructGPT's own constraint was size: a 175B reward model trained unstably and was "less suitable to be used as the value function", so all RMs were 6B6 (§3.5, App. C).

The critic is also a second large training job inside the RL loop, with its own loss (Lambert's Figure 24 trains it by mean-squared error against return targets)3. Module 5 covers the four-model PPO-RLHF system and Module 1 its memory cost. What matters here is the trade: the critic buys token-level credit and lower variance at the price of a second model that must itself be learned from the same sparse signal.

6. On-policy sampling and the importance-sampling correction

6.1 Why the samples must come from the policy being differentiated

The expectation in (PG) is taken under \(\pi_\theta\) itself. If the completions were drawn by an older policy \(\pi_{old}\), averaging them estimates the wrong quantity. The standard repair is importance sampling:

$$\mathbb{E}_{y\sim\pi_\theta}[f(y)]=\mathbb{E}_{y\sim\pi_{old}}\big[\rho(y)f(y)\big],\qquad \rho(y)=\frac{\pi_\theta(y\mid x)}{\pi_{old}(y\mid x)}.$$

Differentiate the surrogate \(\mathbb{E}_{\pi_{old}}[\rho\,A]\) with \(A\) held fixed. Since \(\nabla_\theta\rho=\rho\,\nabla_\theta\log\pi_\theta\) and \(\rho=1\) at \(\theta=\theta_{old}\), the gradient at that point is \(\mathbb{E}[A\,\nabla_\theta\log\pi_\theta]\): the policy gradient (derived). This is the surrogate in TRPO's practical algorithm, where the sum over actions is replaced "by an importance sampling estimator"10 (§5, Eq. 14), and PPO's \(L^{CPI}(\theta)=\hat{\mathbb{E}}_t[r_t(\theta)\hat A_t]\) with \(r_t=\pi_\theta/\pi_{\theta_{old}}\)9 (§3, Eq. 6). The first optimizer step on a rollout is exactly on-policy. Every further step moves \(\theta\) away from \(\theta_{old}\), the ratio departs from 1, and the estimator is only as good as the ratio stays near 1. PPO's authors warn that taking "multiple steps of optimization on this loss \(L^{PG}\) using the same trajectory" is "not well-justified, and empirically it often leads to destructively large policy updates"9 (§2.1). Clipping the ratio or constraining the KL is how Module 3's algorithms keep that departure small.

6.2 How stale the reported recipes let the data get

Rollouts are the expensive part of an RL iteration (Module 1), so most recipes reuse each rollout for several optimizer steps and accept some staleness. The reported settings span a wide range.

Table 5. Optimizer steps fed by one rollout in three reports. "Staleness at last step" is the number of earlier optimizer steps taken on the same rollout, derived from the reported sizes. DeepSeekMath: §4.21; R1-Zero: §2.112; DAPO: §4.113.
ReportOutputs per rolloutMini-batch (outputs)Optimizer steps per rolloutStaleness at last step
DeepSeekMath-RL (7B)64 per question; batch size 1,024 (unit not stated)n/a10
DeepSeek-R1-Zero8,1925121615
DAPO (Qwen2.5-32B)8,192 (512 prompts x 16)5121615

DeepSeekMath-RL took "a single update following each exploration stage", which makes \(\rho\equiv1\) and the clip inert1 (§4.2, App. A.1.5). R1-Zero's rollout "generates 8,192 outputs, which are randomly split into 16 mini-batches and trained for only a single inner epoch"12 (§2.1), so the last mini-batch is fifteen optimizer steps stale. DAPO's configuration gives the same count: 512 prompts with 16 responses each, a mini-batch of 512, "16 gradient updates for each rollout step"13 (§4.1). How much this costs is empirical. Ahmadian et al. measured the PPO loss being clipped "on average < 5% of the time per batch" across their datasets and base models and read it as a regime "close to being 'on-policy'"; removing clipping and the ratio altogether gave "a slight boost"4 (§3.2). That result is for short RLHF generations; Module 12 returns to ratios and clipping for long reasoning rollouts, and Modules 14 and 18 cover staleness from asynchronous pipelines and the mismatch between the sampler's and the trainer's probabilities, which makes even the first mini-batch not quite on-policy.

6.3 Why the ratio is per token

The sequence-level ratio is a product of per-token ratios, \(\rho(y)=\prod_t\rho_t\), so \(\log\rho=\sum_t\log\rho_t\). If the per-token log-ratios are roughly independent with standard deviation \(\sigma\), the standard deviation of \(\log\rho\) is \(\sigma\sqrt T\) (derived, with that independence assumption). An illustrative \(\sigma=0.01\) gives 0.32 at \(T=1{,}000\) tokens, so sequence weights range over a factor of about \(e^{\pm0.32}=1.4\), and 1.0 at \(T=10{,}000\), a factor of about \(e^{\pm1}=2.7\). The per-token ratio keeps each term near 1; PPO defines \(r_t(\theta)\) per timestep9 (§3), while the sequence-level alternative that Lambert's Table 3 lists for GSPO takes the ratio to the power \(1/|a_i|\)3 (§6.2.11), a length root that offsets this growth (the writer's reading). Whether token-level or sequence-level is the better trade is one of the open questions of Module 12.

7. Exploration and sparse rewards

7.1 Exploration lives in the sampler

A policy-gradient method explores by sampling from its own stochastic policy. Sutton and Barto list among the advantages of parameterized policies that they "can learn appropriate levels of exploration and approach deterministic policies asymptotically"7 (§13.8, p. 337). An LLM is stochastic by construction, so no \(\epsilon\)-greedy rule or injected noise is needed. The knobs are the ones an inference engineer already owns:

  • Sampling temperature. R1-Zero rolls out at temperature 112 (§2.1). Kimi K2 treats it as a schedule: a high temperature early "is crucial" for exploration and reduces "the risk of premature convergence", and is decayed because a high temperature late "compromises the reliability and consistency of the model's outputs"14 (§3.2.3, Temperature Decay).
  • The policy's own entropy. Across RL runs without an entropy intervention, policy entropy "dropped sharply at the early training stage, leading to an overly confident policy model", and performance saturates15 (Abstract). Module 13 treats the controls.
  • Group size and prompt selection, which decide how often a rare success is seen at all (Section 7.2).

One consequence for the sampler is easy to miss. Estimator (PG) is unbiased only when \(y\) is drawn from the distribution whose log-probability is differentiated. A rollout engine that samples from \(\operatorname{softmax}(z/\tau)\) with \(\tau\ne1\), or with top-\(p\) or top-\(k\) truncation, draws from a different distribution than the \(\operatorname{softmax}(z)\) the trainer differentiates (derived). Either the trainer applies the same transform to its logits, or the sampler runs at \(\tau=1\) with no truncation. Olmo 3 states its RL sampler as temperature 1.0 and top-\(p\) 1.016 (§4.4.2). Module 14 covers the remaining ways the two engines disagree.

7.2 A binary reward teaches only on prompts the policy sometimes solves

Suppose a prompt has pass rate \(p\) under the current policy and a group has \(G\) independent samples. The group is all-fail with probability \((1-p)^G\) and all-pass with probability \(p^G\). With any baseline built from the group, those groups give every sample the same reward and therefore zero advantage; with no baseline, all-fail groups still give exactly zero. Only mixed groups push the policy anywhere:

$$P(\text{informative})=1-p^{G}-(1-p)^{G},\qquad G_{95}(p)=\Big\lceil\frac{\ln 0.05}{\ln(1-p)}\Big\rceil,$$

where \(G_{95}\) is the group size that sees at least one success with probability 0.95 (derived). DAPO states the all-correct case plainly: "if all outputs [...] of a particular prompt are correct and receive the same reward, the resulting advantage for this group is zero. A zero advantage results in zero policy gradients"13 (§3.2). They also observe that the number of prompts with accuracy 1 "continues to increase" during training, so the informative share of a fixed prompt set shrinks as the policy improves13 (§3.2, Fig. 3b).

Table 6. How many samples per prompt a binary reward needs, and how often a group is informative (derived from \(1-(1-p)^G\) and \(1-p^G-(1-p)^G\); no simulation).
Pass rate \(p\)\(G\) for 50% chance of a success\(G_{95}\)Informative, \(G=8\)Informative, \(G=16\)Informative, \(G=64\)
0.0016932,9950.0080.0160.062
0.01692990.0770.1490.474
0.0514590.3370.5600.963
0.17290.5700.8150.999
0.253110.9000.9901.000
0.5150.9921.0001.000
Probability that a group of G samples contains both a success and a failure, versus the prompt pass rate, for G of 4, 8, 16 and 64 0.00 0.25 0.50 0.75 1.00 0.001 0.01 0.1 0.5 prompt pass rate p under the current policy (log scale; the curve mirrors about p = 0.5) P(group has at least one success and one failure) = 1 − p^G − (1−p)^G 0.15 0.47 G = 4 G = 8 G = 16 G = 64
Figure 6. Table 6 as curves. A prompt the policy solves 1% of the time yields an informative group only 15% of the time at G = 16 and 47% at G = 64 (marked). Derived from the formula in the text.

Reports respond to this arithmetic in three ways. Kimi k1.5 observes that "spending a restricted computation budget on very hard problems often yields few correct samples, resulting in lower training efficiency", and answers with a curriculum from easy to hard plus prioritized sampling proportional to \(1-s_i\), where \(s_i\) is the tracked success rate of problem \(i\)17 (§2.3.4). DAPO's Dynamic Sampling over-samples and discards prompts whose accuracy is exactly 0 or 1 until the batch is full of informative prompts13 (§3.2). Olmo 3 filters offline: it draws eight rollouts per prompt from the starting checkpoint and removes prompts above a 62.5% pass rate (that is, more than 5 of 8 correct), and for the 32B model relies on active sampling that "fills RL batches only on samples with a non-zero GRPO group gradient"16 (§4.4.2). Larger \(G\) is the fourth response: R1-Zero uses 16 samples per question12 (§2.1) and DeepSeekMath 641 (§4.2). Modules 10 and 11 treat prompt selection and zero-variance groups in depth; the point here is that the prerequisite for learning from a verifier is a base policy that already solves a useful fraction of the training prompts.

When does a group carry a gradient?

At least one success
Informative group (mixed outcomes)
G for 95% chance of one success
Group-mean baseline: fraction of true step

Formulas: \(1-(1-p)^G\); \(1-p^G-(1-p)^G\); \(\lceil\ln0.05/\ln(1-p)\rceil\); \((G-1)/G\) from Section 4.2. Assumes independent samples with a fixed per-prompt pass rate.

8. Placing any algorithm: baseline source and step control

Sections 3 to 7 reduce to four questions that classify a method from its paper's first page. Where does the baseline come from: nothing, a running mean, the other samples for the same prompt, or a learned critic? Is the resulting advantage one scalar broadcast over the sequence or a separate number per token? How are samples reused: not at all, with a ratio and a clip, or under a KL constraint? And where does the KL to a reference sit: in the reward, in the loss, or absent? Table 7 answers them for the methods whose primary sources this module read.

Table 7. Policy-gradient methods classified by baseline, credit granularity, step control and KL placement. Sources: RLOO paper §2.1-§2.34; InstructGPT §3.56; PPO §39; DeepSeekMath §4.1 and §5.2.11; Kimi k1.5 §2.3.217; Lambert §6.2.4, §6.2.11 and Table 33. The last column follows from each baseline's definition (derived); group sizes are from R1-Zero12 (§2.1) and DeepSeekMath1 (§4.2).
MethodBaseline \(b\)CreditStep controlKL to referenceExtra learned modelSamples per prompt needed
Online RFTnone (0/1 indicator)per sequencenonenonenone1 or more
REINFORCE, moving-average baselinemean of all past rewardsper sequencenone (no ratio)in the rewardnone1
RLOOmean of the other \(k-1\) samplesper sequencenone (no ratio)in the rewardnone\(k\ge2\)
PPO (InstructGPT)learned \(V(s_t)\), GAEper tokenratio clip, several epochsper token, in the rewardvalue model1 (the critic is the baseline)
GRPOgroup mean, divided by group stdper sequence, broadcasttoken ratio clipin the lossnone\(G\ge2\); 16 in R1-Zero, 64 in DeepSeekMath
Kimi k1.5mean of \(k\) sampled rewardsper sequenceonline mirror descent, squared-error surrogaterelative entropy to the current modelnone\(k\ge2\)

Two entries need a gloss. Lambert's own comparison table makes the same point for the larger family: REINFORCE and RLOO "have no importance-sampling ratio", while PPO, GRPO, GSPO and CISPO each introduce one "to enable multiple gradient steps per batch of rollouts, differing in granularity and clipping strategy", and "only PPO requires a learned value function"3 (§6.2.11). And the KL column is where reasoning recipes have drifted: with the move to verifiable rewards, Lambert notes the "prevalence of KL penalties has decreased overall, with many reasoning adaptations of RLHF code turning them off entirely"3 (§6.2.4), and DAPO removes the term because a long-CoT policy "can diverge significantly from the initial model"13 (§2.3). Module 6 returns to KL; Module 9 develops the group-baseline rows and Kimi's objective.

One lineage point is easy to get wrong. DeepSeek-R1's RL stage uses GRPO, which DeepSeekMath introduced in 2024112 (§2.1). TRPO10 is the 2015 trust-region ancestor of PPO, not the algorithm R1 trains with. Module 3 draws the TRPO, PPO and GRPO lineage, and Module 27 reads R1 stage by stage.

8.1 Where this sits in the textbook

Everything above is Chapter 13 of Sutton and Barto, specialized to a softmax policy whose state is its own prefix. Table 8 maps the sections to the places they appear in an LLM trainer, so you can read the textbook derivations when you need them.

Table 8. Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., Chapter 13 (section and page numbers read from the book's table of contents and text7) mapped to this module (mapping by the writer).
SectionPageContentIn an LLM trainer
13.1 Policy approximation and its advantages322Parameterized stochastic policy, soft-max action preferencesThe LM head; the policy is a softmax over the vocabulary
13.2 The policy gradient theorem324\(\nabla J\propto\sum_s\mu(s)\sum_a q_\pi(s,a)\nabla\pi(a\mid s)\) (Eq. 13.5, p. 326; proof p. 325)Equation (PG), Section 2
13.3 REINFORCE: Monte Carlo policy gradient326\(\theta\leftarrow\theta+\alpha G_t\nabla\ln\pi(A_t\mid S_t,\theta)\) (Eq. 13.8, p. 327)The reward-weighted token-score sum; Section 3
13.4 REINFORCE with baseline329Subtract \(b(s)\) that does not vary with \(a\) (Eqs. 13.10-13.11)Section 4; RLOO and GRPO baselines; PPO's \(V\) at \(\lambda=1\)
13.5 Actor-critic methods331Bootstrapping adds bias and cuts varianceGAE with \(\lambda\lt1\), Section 5 and Module 3

Where experts disagree

The question this module frames is whether an LLM policy needs a learned critic, and how fine its credit assignment should be. The answer is not settled by a single controlled comparison; the positions below rest on different tasks and compute scales. Module 9 takes up the critic-free side in full.

Drop the critic: sequence-level credit with a sample baseline

Ahmadian et al. (RLOO)4; Shao et al. (GRPO)1; Kimi k1.517

Rewards arrive only for the full generation, so the whole completion is one action; a pretrained policy's concentrated distribution makes REINFORCE variance "not a practical concern", so extra variance reduction at the cost of bias "is not worth it"4 (§3.1). DeepSeekMath adds that the critic is policy-sized and hard to train from a last-token reward1. Kimi k1.5 states the design stance as no "value functions" in its abstract17.

  • No second model to hold, train or initialize; the estimator stays unbiased.
  • Ahmadian et al. report a 3.2% to 20.3% win-rate margin over PPO, and 10.3, 14.5 and 32.1 points over their PPO for RLOO \(k=4\) on TL;DR and the two HH settings4 (§1, §5.2).
  • Needs several samples per prompt, which costs sampling time4 (§2.3).
  • No token-level credit; at low pass rates and small \(G\) the baseline barely reduces noise (Table 3, prompt B).

Keep a learned per-token critic, initialized from a reward model

InstructGPT6; Lambert et al. (Tulu 3)11; Xu et al.18

Tulu 3's RLVR ablations found that initializing the value function from a general reward model gave the best GSM8K and average scores, and the report lists value-free methods such as GRPO as future work rather than a tested alternative11 (§6.2.1, Limitations). Xu et al. report that PPO "is able to surpass other alignment methods in all cases"18 (Abstract); their abstract frames the comparison as PPO against DPO, so it bounds what a well-tuned actor-critic can reach, not what a critic adds over a group baseline.

  • Token-level advantages that can localize credit, and variance reduction without needing many samples per prompt.
  • Xu et al. report state-of-the-art results in challenging code competitions with PPO18 (Abstract).
  • A second policy-sized model and a value-learning problem with, per Lambert, no established best practice3 (§6.2.8).
  • Bootstrapping (\(\lambda\lt1\)) reduced reward monotonically in Ahmadian et al.'s RLHF ablation4 (§3.1), though GAE's authors found \(\lambda\in[0.92,0.99]\) best on continuous control2 (§6).

Finer credit from step labels instead of a critic

Shao et al. (GRPO with process supervision)1, against DeepSeek-R1's rejection of neural reward models12

DeepSeekMath's GRPO with process supervision "shows superior performance compared to GRPO+OS, indicating the benefits of using fine-grained, step-aware gradient coefficients"1 (§5.2.1, Fig. 5): token credit without a value network. R1 then chose rule-based rewards and "abstain[ed] from applying neural reward models—whether outcome-based or process-based—to reasoning tasks", reasoning that neural reward models are susceptible to reward hacking12 (§2.2).

  • Measured gain over outcome supervision in the DeepSeekMath 1.3B comparison.
  • Needs step labels or a process reward model, which R1 judged hackable at scale; Modules 4 and 18 treat both sides.
Assessment

No source runs the controlled comparison that would settle this: the same base model, data and sampling budget, with a tuned RM-initialized critic, a \(\lambda=1\) learned baseline and a group baseline, reporting both gradient variance and final pass@k. What exists is consistent with a conditional answer, and the conditions are the writer's reading. For short RLHF generations from a strong initialization, the sample baseline matches or beats a critic at lower cost4. For verifiable-reward reasoning with 8 or more cheap samples per prompt, the large recipes (R1 with GRPO12, Kimi k1.517, and DAPO13) went critic-free, and the extra model buys little when the signal is one bit at the end. A critic is worth testing again where samples are expensive (long agentic episodes, costly environments) so that \(G\) must stay small, or where reliable step-level labels exist. Table 3 is the quantitative reason: when pass rates are low and \(G\) is small, no baseline of any kind removes most of the noise, so the choice is really between more samples and a better value estimate.

What is changing

The theory in this module is settled; what moves is convention. Lambert observes that discounting at \(\gamma=1\) is justified because the reward scores the whole completion, and that as agentic RL matures, with tool calls and web actions as genuinely distinct sequential decisions, discounting "may become relevant again"3 (§6.3.2.2); Module 18 takes up credit assignment across turns. The KL term, the critic and the ratio are each present in some published recipes and absent in others: DAPO drops the KL term13, GRPO drops the critic1, and REINFORCE and RLOO have no ratio3. And methods that supply a dense per-token teacher signal instead of one end-of-sequence scalar change what \(\Psi_t\) is altogether; Modules 21 and 22 cover them.

References

  1. paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: GRPO definition and the PPO comparison (§4.1.1, Eqs. 1-4), outcome supervision (§4.1.2), training settings (§4.2), the unified gradient and Table 10 (§5.2.1, App. A.1 Eqs. 6-21), Online RFT versus GRPO and process versus outcome supervision (Fig. 5).
  2. paperSchulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016; arXiv v6, 20 Oct 2018. https://arxiv.org/pdf/1506.02438v6. Accessed 2026-10-26. Used for: the six choices of \(\Psi_t\) (Eq. 1), Proposition 1 on admissible baselines, GAE(\(\gamma,1\)) (Eqs. 15 and 18), the best \(\lambda\) range (§6).
  3. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: RLHF as a modified RL setup (§3.1.3, Table 1), the policy-gradient derivation and the Ψ list (§6.2.1), REINFORCE and RLOO (§6.2.3-§6.2.4, Eq. 51), value functions (§6.2.7), GRPO and Dr. GRPO (§6.2.8), algorithm comparison (§6.2.11, Table 3), implementation notes and MDP versus bandit framing (§6.3, §6.3.2.2).
  4. paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: the single-action argument (§1, §3.3), the token-level PPO setup (§2.1), REINFORCE and RLOO (§2.2-§2.3), the GAE-\(\lambda\) ablation (§3.1), the clipping rate (§3.2), win-rates (§5.2, Table 1).
  5. officialFarama Foundation. Gymnasium documentation: Basic Usage. Live documentation, 2026. https://gymnasium.farama.org/introduction/basic_usage/. Accessed 2026-10-26. Used for: the Env.step return signature and the terminated versus truncated flags.
  6. paperOuyang, L., Wu, J., Jiang, X., Almeida, D. et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: the bandit-environment description, per-token KL penalty, value function initialized from the RM, and the 6B RM choice (§3.5, Eq. 2).
  7. textbookSutton, R. S., Barto, A. G. Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018. Read in the PDF copy at https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf (publisher page: https://mitpress.ublish.com/book/reinforcement-learning-an-introduction-2). Accessed 2026-10-27. Used for: Chapter 13, pp. 321-338: the policy gradient theorem (§13.2, Eq. 13.5), REINFORCE (§13.3, Eq. 13.8), baselines (§13.4, Eqs. 13.10-13.11, Fig. 13.2), the baseline versus actor-critic distinction (§13.5, p. 331), the summary (§13.8, p. 337).
  8. paperWilliams, R. J. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning 8, 229-256, 1992. https://link.springer.com/content/pdf/10.1007/BF00992696.pdf. Accessed 2026-10-27. Used for: the REINFORCE update form and Theorem 1 (§4, pp. 234-235), the statement on baselines (p. 236), episodic REINFORCE with one reward per episode (§5, pp. 236-237).
  9. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the policy-gradient estimator, surrogate-loss construction and the multi-step warning (§2.1, Eqs. 1-2), the ratio and \(L^{CPI}\) (§3, Eq. 6).
  10. paperSchulman, J., Levine, S., Moritz, P., Jordan, M., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv v5, 20 Apr 2017. https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Used for: the performance-difference identity (§2, Eq. 1) and the importance-sampling surrogate (§5, Eq. 14).
  11. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the finding that initializing the RLVR value function from a general RM works best (§6.2.1) and GRPO named as future work (Limitations).
  12. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026 (expanded Nature version, Nature 645, 633-638, 2025). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: GRPO use, 16 samples per question, rollout temperature 1, the 8,192-output rollout split into 16 mini-batches (§2.1), and the rejection of neural reward models (§2.2).
  13. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: zero-advantage groups and Dynamic Sampling (§3.2, Fig. 3b), KL removal (§2.3), the 512-prompt by 16-response rollout with 16 updates (§4.1).
  14. paperKimi Team (Bai, Y., Bao, Y., Charles, Y. et al.). Kimi K2: Open Agentic Intelligence. arXiv v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: the temperature-decay schedule from exploration to exploitation (§3.2.3).
  15. paperCui, G., Zhang, Y., Chen, J., Yuan, L. et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv v1, 28 May 2025. https://arxiv.org/pdf/2505.22617v1. Accessed 2026-10-26. Used for: the early entropy drop and performance saturation (Abstract).
  16. paperTeam Olmo (Ettinger, A., Bertsch, A. et al.). Olmo 3. arXiv v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: sampler temperature 1.0 and top-p 1.0, offline difficulty filtering above a 62.5% pass rate, active sampling for non-zero group gradients (§4.4.2).
  17. paperKimi Team (Du, A., Gao, B., Xing, B. et al.). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: the no-value-function design stance (Abstract), the mean-reward baseline and mirror-descent objective (§2.3.2), curriculum and prioritized sampling (§2.3.4).
  18. paperXu, S., Fu, W., Gao, J., Ye, W. et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: the claim that PPO surpasses other alignment methods in all cases (Abstract; only the abstract was read).