Reinforcement Learning for LLM Post-Training · Module 3

From TRPO to PPO: Trust Regions, the Clipped Surrogate, and GAE

PPO is the algorithm InstructGPT used and the one GRPO modifies; TRPO is the 2015 trust-region method PPO simplified, and GAE is the advantage estimator inside both. This module derives each constraint, shows what the clip does and does not guarantee, and computes a GAE advantage by hand. Afterwards you can read any PPO-family loss and name what it keeps from TRPO, what it drops, and where DeepSeek-R1 sits (it trains with GRPO, not TRPO).

Course 45 min read 19 sources As of 2026-10-27 Prerequisite: Module 2
Lineage from TRPO (2015) through GAE, PPO and InstructGPT to GRPO (2024) and DeepSeek-R1 (2025), with what each step keeps and drops TRPO2015 GAE2016 PPO2017 InstructGPT2022 GRPOFeb 2024 DeepSeek-R1Jan 2025 avg KL ≤ 0.01CG + line searchsecond-order TD residualsλ-weighted sumbias vs variance ratio clip 0.2Adam, K epochsfirst-order PPO on an RMKL in the rewardvalue init: RM PPO clip keptno critic, no GAEKL in the loss trains with GRPOclip ε = 1016 minibatches TRPO to PPO keeps: ratio × advantage surrogate drops: hard KL bound, Fisher step, line search, improvement guarantee Sections 2 to 5 PPO to GRPO keeps: ratio, clip, minibatch reuse drops: value model and GAE moves: KL to loss; adds group baseline Sections 6 and 7; GRPO itself is Module 9
Figure 1. The lineage this module covers. TRPO appeared at ICML 20151, GAE at ICLR 2016 and was run inside TRPO2, PPO in 20173. InstructGPT's RL stage used PPO4. GRPO was introduced in DeepSeekMath, Feb 20245; DeepSeek-R1 adopts it and reports a first-stage clip ratio of 10 and 16 minibatches per rollout6 (§3.2.1).

1. Why a plain policy-gradient step is unsafe to reuse

Module 2 ended with the estimator \(\hat g=\hat{\mathbb E}_t[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,\hat A_t]\) and the remark that it is valid only at the \(\theta\) that generated the samples. Three facts turn step size into the central problem of every algorithm in this module: the surrogate is local, parameter distance is not policy distance, and reuse of rollouts is what makes both bite.

1.1 The gradient is a first-order statement

TRPO replaces the true return \(\eta(\pi)\) with a local surrogate that weights advantages by the old policy's state-visitation frequencies \(\rho_\pi\):

$$L_\pi(\tilde\pi)=\eta(\pi)+\sum_s\rho_\pi(s)\sum_a\tilde\pi(a\mid s)\,A_\pi(s,a).$$

The surrogate agrees with \(\eta\) in value and gradient at \(\theta_{old}\) (Eq. 4 of the paper), "but does not give us any guidance on how big of a step to take"1 (§2). It ignores what changes as soon as the policy moves: the states the new policy would visit, and everything beyond first order. Every method below is a different way to pay for that ignorance.

1.2 Parameter distance is not policy distance

A step of fixed length in \(\theta\) moves the policy by very different amounts in different places. The smallest example is a Bernoulli policy with logit \(\theta\) and \(p=\sigma(\theta)\). Solving exactly for the logit shift that produces \(\mathrm{KL}(p\,\|\,q)=0.01\) nats (derived):

Table 1. Logit shift \(\Delta\theta\) that moves a Bernoulli policy by exactly 0.01 nats of KL, for four starting probabilities (exact root of \(\mathrm{KL}(p\Vert q)=0.01\), derived by the writer).
Start \(p\)Logit shift \(\Delta\theta\)New \(q\)Ratio \(q/p\)
0.50.2830.5701.14
0.10.4440.1481.48
0.011.1570.03113.11
0.0012.6170.013513.5

Two lessons. A trust region has to be measured in policy space, because the same logit shift buys very different KL depending on where the policy sits. And an average-KL budget and a per-token ratio bound constrain different things: the same 0.01 nats lets a rare token (\(p=0.01\)) triple while a common one (\(p=0.5\)) moves 14%. Section 5.2 returns to this mismatch.

1.3 Reuse is what makes it matter

If every optimizer step used a fresh rollout, a small enough learning rate would be the only safeguard needed. It is not affordable: rollouts are the dominant cost of an RL iteration (Module 1), so recipes feed one rollout to several optimizer steps. PPO's authors state the consequence for the plain objective \(L^{PG}=\hat{\mathbb E}_t[\log\pi_\theta(a_t\mid s_t)\hat A_t]\): taking multiple steps on the same trajectory "is not well-justified, and empirically it often leads to destructively large policy updates"3 (§2.1). Their ablation on seven MuJoCo tasks shows the size of the effect: with neither clipping nor a KL penalty the average normalized score is \(-0.39\), worse than the random policy at 0, because one environment (HalfCheetah) collapses3 (§6.1, Table 1). TRPO's authors report the same failure mode at the single-step level: without a line search on the final step, "the algorithm occasionally computes large steps that cause a catastrophic degradation of performance"1 (App. C).

2. TRPO's theory: a lower bound on improvement

TRPO starts from a guarantee and then approximates it. PPO inherits its surrogate from that guarantee, and GRPO inherits PPO's.

2.1 The identity the surrogate comes from

Module 2 used the performance-difference identity of Kakade and Langford. In TRPO's notation, for any two policies,

$$\eta(\tilde\pi)=\eta(\pi)+\mathbb{E}_{\tau\sim\tilde\pi}\Big[\sum_{t=0}^{\infty}\gamma^{t}A_\pi(s_t,a_t)\Big]=\eta(\pi)+\sum_s\rho_{\tilde\pi}(s)\sum_a\tilde\pi(a\mid s)A_\pi(s,a),$$

where \(\rho_\pi(s)=\sum_t\gamma^tP(s_t=s\mid\pi)\) is the discounted visitation frequency1 (§2, Eqs. 1-2). Any update whose expected advantage is nonnegative at every state cannot reduce \(\eta\). The catch is in the weights: \(\rho_{\tilde\pi}\) belongs to the new policy, so the right-hand side cannot be maximized directly. The surrogate \(L_\pi\) of Section 1.1 swaps in \(\rho_\pi\), which is computable from old-policy samples.

2.2 Theorem 1 and the penalty form

Kakade and Langford bounded the error of that swap for a mixture update \(\pi_{new}=(1-\alpha)\pi_{old}+\alpha\pi'\). TRPO's main theoretical result extends the bound to arbitrary stochastic policies by replacing \(\alpha\) with a distance. With the total-variation distance \(D_{TV}^{\max}(\pi,\tilde\pi)=\max_sD_{TV}(\pi(\cdot\mid s)\,\|\,\tilde\pi(\cdot\mid s))\) and \(\alpha=D_{TV}^{\max}(\pi_{old},\pi_{new})\):

$$\eta(\pi_{new})\ \ge\ L_{\pi_{old}}(\pi_{new})-\frac{4\epsilon\gamma}{(1-\gamma)^2}\,\alpha^2,\qquad \epsilon=\max_{s,a}|A_\pi(s,a)|.$$

Because \(D_{TV}(p\|q)^2\le D_{KL}(p\|q)\), the bound becomes a KL penalty1 (§3, Theorem 1 and Eq. 9):

$$\eta(\tilde\pi)\ \ge\ L_\pi(\tilde\pi)-C\,D_{KL}^{\max}(\pi,\tilde\pi),\qquad C=\frac{4\epsilon\gamma}{(1-\gamma)^2},\quad D_{KL}^{\max}=\max_sD_{KL}(\pi(\cdot\mid s)\,\|\,\tilde\pi(\cdot\mid s)).$$

The right-hand side equals \(\eta\) at \(\tilde\pi=\pi\) and lies below it everywhere else, so it is a minorizer of \(\eta\). Maximizing it at each iteration is a minorization-maximization step: writing \(M_i(\pi)=L_{\pi_i}(\pi)-CD_{KL}^{\max}(\pi_i,\pi)\), the chain \(\eta(\pi_{i+1})-\eta(\pi_i)\ge M_i(\pi_{i+1})-M_i(\pi_i)\ge0\) gives a monotonically non-decreasing sequence of returns. The paper notes the algorithm is also "reminiscent of proximal gradient methods and mirror descent"1 (§3).

2.3 What the theory does not give you

The authors say plainly where it fails as an algorithm: "if we used the penalty coefficient \(C\) recommended by the theory above, the step sizes would be very small"1 (§4). The size of \(C\) is a function of the discount alone:

Table 2. The theory's penalty coefficient per unit of \(\epsilon\), \(C/\epsilon=4\gamma/(1-\gamma)^2\), against the effective horizon \(1/(1-\gamma)\) that the GAE paper associates with the discount2 (§4). Values derived from the formula in Theorem 1 / Eq. 91.
Discount \(\gamma\)Horizon \(1/(1-\gamma)\) (steps)\(C/\epsilon\)
0.910360
0.9910039,600
0.9991,0003,996,000
1 (LLM default)unboundedundefined

\(C\) grows with the square of the horizon, so a theory-sized penalty pins the policy to \(\pi_{old}\). Three more gaps matter for an LLM practitioner.

  • Worst case over states. \(D_{KL}^{\max}\) is "impractical to solve due to the large number of constraints"1 (§4); for a language model the states are every possible prefix.
  • Discounting. The theorem is stated for an infinite-horizon MDP with \(\gamma\in(0,1)\)1 (§2). The LLM setup of Module 2 is finite-horizon with \(\gamma=1\), the common choice in RLHF code7 (§6.4.1), where \(C\) as written is undefined. I did not find a finite-horizon version in the sources read for this course, so treat the theorem as motivation for a KL-shaped trust region, not as a guarantee for LLM RL (the writer's reading).
  • Exact advantages. The practical algorithm "ignores estimation error for the advantage function"1 (§5); in LLM RL the advantage is a noisy sample estimate (Module 2).

3. Practical TRPO: an average-KL constraint and a natural-gradient step

3.1 The constrained problem, and where PPO's ratio comes from

The practical algorithm makes three substitutions. The penalty becomes a hard constraint, because "it is hard to choose a single value of \(\beta\) that performs well across different problems—or even within a single problem"3 (§2.2). The max over states becomes an average over states visited by the old policy. And the sum over actions becomes an importance-sampling estimate. The result is1 (§4-§5, Eqs. 12 and 14):

$$\max_\theta\ \mathbb{E}_{s\sim\rho_{\theta_{old}},\,a\sim q}\Big[\frac{\pi_\theta(a\mid s)}{q(a\mid s)}\,Q_{\theta_{old}}(s,a)\Big]\quad\text{s.t.}\quad \mathbb{E}_{s\sim\rho_{\theta_{old}}}\big[D_{KL}\big(\pi_{\theta_{old}}(\cdot\mid s)\,\|\,\pi_\theta(\cdot\mid s)\big)\big]\le\delta.$$

In the single-path scheme \(q=\pi_{\theta_{old}}\), so the importance weight is \(\pi_\theta/\pi_{\theta_{old}}\): exactly the ratio \(r_t(\theta)\) that PPO writes down two years later1 (§5.1)3 (§3). Two details are worth fixing in memory because they survive into every PPO-family loss. The KL has the old policy first, \(D_{KL}(\pi_{old}\|\pi_\theta)\), and PPO's restatement of the constraint uses the same order3 (Eq. 4). Replacing \(A\) by \(Q\) "only changes the objective by a constant", so a baseline is a variance device, not part of the trust-region problem1 (§5). The step size was \(\delta=0.01\) "for all experiments" (§8.1).

3.2 Solving it: Fisher matrix, conjugate gradient, line search

The paper solves the constrained problem approximately, with a linear model of the objective and a quadratic model of the constraint. The quadratic model is \(D_{KL}(\theta_{old},\theta)\approx\tfrac12(\theta-\theta_{old})^\top A(\theta-\theta_{old})\), where \(A\) (written \(F\) in Figure 2) is the Fisher information matrix, built "by analytically computing the Hessian of the KL divergence" rather than from gradient covariances1 (§6). With \(g=\nabla_\theta L\), maximizing \(g^\top\Delta\) subject to \(\tfrac12\Delta^\top A\Delta\le\delta\) gives (derived by a Lagrangian; it is the paper's \(\beta s\) with \(s=A^{-1}g\)):

$$\Delta\theta=\sqrt{\frac{2\delta}{g^\top A^{-1}g}}\;A^{-1}g,\qquad \beta=\sqrt{\frac{2\delta}{s^\top As}}\ \text{ for }\ s\approx A^{-1}g\ \ \text{(App. C)}.$$

Three engineering pieces make this tractable for a neural policy1 (App. C, C.1). \(A^{-1}g\) is found by conjugate gradient from Fisher-vector products alone, never forming \(A\); a product costs "about as expensive as computing the gradient of an objective", and \(k=10\) iterations were "quite effective". Because \(A\) only acts as a metric, it can be computed on 10% of the data, making "the total cost of Hessian-vector products ... about the same as computing the gradient". And a line search shrinks \(\beta\) exponentially until the true surrogate improves and the true KL constraint holds, since both depart from the quadratic model; without it "the algorithm occasionally computes large steps that cause a catastrophic degradation of performance".

A two-parameter toy: the KL ellipse, a vanilla gradient step that overshoots it, the same step shrunk to the KL budget, and the TRPO natural-gradient step on the boundary KL ≤ 0.01 region θ_old vanilla, same length KL 0.125 scaled to budget TRPO step θ₁: KL-flat direction (F = 1) θ₂: KL-sensitive (F = 25) Same toy, three steps KL (nats) gain gᵀΔθ vanilla, same length 0.1250.196 vanilla, scaled to δ 0.0100.055 TRPO natural step 0.0100.144 illustrative toy, derived by the writer: F = diag(1, 25), g = (1, 1), δ = 0.01 at equal KL the natural step gains 2.6x more first-order improvement than the gradient step
Figure 2. Why the metric matters, on a toy with an assumed Fisher matrix \(F=\mathrm{diag}(1,25)\) and gradient \(g=(1,1)\) (illustrative; arrows to scale, 1,500 px per unit of \(\theta\)). A gradient step as long as the TRPO step lands at 12.5x the KL budget; scaled to the budget it gains 0.055. The natural step \(\beta F^{-1}g\)1 (App. C) reaches the boundary and gains 0.144, 2.6x more. All numbers derived from the formulas in the text.

The toy also fixes the relationship to simpler methods. The TRPO paper shows that its update contains the natural policy gradient as a special case (linear \(L\), quadratic KL, Eq. 17) and the standard gradient as another (an \(\ell_2\) constraint, Eq. 18); what TRPO adds over natural gradient is that it enforces the KL bound at every update, which "significantly improves the algorithm's performance on larger problems"1 (§7). On hopper and walker, natural gradient with a fixed penalty "was unable to generate hopping and walking gaits that made forward progress" while TRPO solved all problems1 (§8.1).

3.3 Single path and vine, and why vine matters again for LLMs

TRPO's two sampling schemes estimate \(Q_{\theta_{old}}\) differently1 (§5). Single path simulates the policy, collects trajectories, and sets \(\hat Q\) to the discounted sum of rewards that followed each state-action pair; it needs no resets. Vine picks a rollout set of states along "trunk" trajectories, samples several actions at each, and runs a short rollout after each action with common random numbers to cut the variance of the \(Q\)-differences. Vine gives "much better estimates of the advantage values" at the price of far more simulator calls and the requirement that the system can be restored to arbitrary states, "which is typically only possible in simulation" (§1).

A language model satisfies the reset requirement for free: a state is a token prefix, so any prefix can be re-fed and continued. VinePPO uses exactly this. For each state in a sampled completion it draws \(K\) continuations from the current policy, sets \(\hat V_{MC}(s_t)=\frac1K\sum_{k=1}^{K}R(\eta_k)\) and forms the advantage \(r_t+\gamma\hat V_{MC}(s_{t+1})-\hat V_{MC}(s_t)\), with \(K=9\) by default; the paper cites TRPO's vine variant as the precedent8 (§3, §4, §5). Its authors report that this unbiased estimate beats a learned value network on MATH and GSM8K, up to 3.0x faster in wall-clock (their claim, on DeepSeekMath 7B and RhoMath 1.1B)8 (abstract, §6.1-§6.2). For an inference engineer the cost model is the interesting part: vine trades a policy-sized critic for extra decode work that shares prefixes (the writer's reading). Module 18 returns to credit assignment by rollout.

3.4 Why LLM recipes do not run TRPO

The sources give one direct reason. PPO's authors say TRPO "is relatively complicated, and is not compatible with architectures that include noise (such as dropout) or parameter sharing (between the policy and value function, or with auxiliary tasks)"3 (§1). No source in this course's map trains an LLM with TRPO, so the rest is the writer's reasoning, not a documented failure:

  • The paper's accounting is favourable (about one extra gradient per update). What it omits is engineering: a Fisher-vector product is a Jacobian-vector product through the whole network and the line search needs extra forward passes, neither of which fits cleanly into sharded, mixed-precision trainers.
  • A first-order clip needs only the per-token log-probabilities the trainer computes anyway (Section 4).
  • RLHF already has a second regularizer, the KL to the reference model; Lambert argues this makes much of the difference between PPO and plain REINFORCE "far less meaningful for fine-tuning language models than training agents from scratch"7 (§6.4.2).

The trust-region idea itself survives in LLM recipes as a clip (PPO, GRPO), as a KL-regularized proximal objective (Kimi k1.5's online mirror descent9 (§2.3.2)), and as a KL term to a reference policy (Module 6).

4. PPO: the clipped surrogate

4.1 The objective and its gradient, case by case

PPO keeps TRPO's surrogate and drops the constraint. Define the probability ratio \(r_t(\theta)=\pi_\theta(a_t\mid s_t)/\pi_{\theta_{old}}(a_t\mid s_t)\), so \(r(\theta_{old})=1\). TRPO's objective is \(L^{CPI}(\theta)=\hat{\mathbb E}_t[r_t(\theta)\hat A_t]\), named after conservative policy iteration, and PPO states that "without a constraint, maximization of \(L^{CPI}\) would lead to an excessively large policy update"3 (§3, Eq. 6). The clipped objective is

$$L^{CLIP}(\theta)=\hat{\mathbb E}_t\Big[\min\big(r_t(\theta)\hat A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t\big)\Big],\qquad \epsilon=0.2\ \text{suggested}$$

(Eq. 7)3. The second term removes the incentive to push \(r_t\) outside \([1-\epsilon,1+\epsilon]\); the \(\min\) keeps the objective a lower bound on \(L^{CPI}\). The gradient of one token's term follows from \(\nabla_\theta r_t=r_t\nabla_\theta\log\pi_\theta\) and is (derived)

$$\nabla_\theta\,\ell_t=\mathbf 1[\text{active}]\;\hat A_t\,r_t(\theta)\,\nabla_\theta\log\pi_\theta(a_t\mid s_t),\qquad \text{active}\iff (\hat A_t>0\wedge r_t\le1+\epsilon)\ \vee\ (\hat A_t<0\wedge r_t\ge1-\epsilon).$$

Where the term is active, the gradient is the Module 2 policy-gradient term multiplied by the ratio, which is the importance correction; where it is inactive, it is exactly zero. Table 3 gives the cases; Lambert's chapter enumerates the same regions7 (§6.2.5-§6.2.6).

Table 3. Gradient of one token's \(L^{CLIP}\) term by sign of advantage and region of the ratio. Derived from Eq. 7; the same regions appear in Lambert's Figure 237 and in the PPO paper's Figure 13.
\(r\lt1-\epsilon\)\(1-\epsilon\le r\le1+\epsilon\)\(r\gt1+\epsilon\)
\(\hat A\gt0\)active: \(\hat A\,r\,\nabla\log\pi\), still pushes upactivezero: objective flat at \((1+\epsilon)\hat A\)
\(\hat A\lt0\)zero: objective flat at \((1-\epsilon)\hat A\)activeactive: still pushes down (the pessimistic branch)
Two-panel plot of one token's clipped surrogate against the probability ratio, for positive and negative advantage, with epsilon 0.2 A > 0: a better-than-baseline token A < 0: a worse-than-baseline token 0 0.8 1 1.2 2 start r = 1 gradient = 0 flat at 1.2 A dashed: r · A active, even below 0.8 0 0.8 1 1.2 2 gradient = 0 flat at 0.8 A dashed: r · A active: keeps pushing down horizontal axis: r = π_θ / π_old
Figure 3. One token's term of \(L^{CLIP}\) (solid) against \(r\), with \(\epsilon=0.2\) and \(|A|=1\); dashed is the unclipped \(rA\) that the \(\min\) discards; the orange dot is the start, \(r=1\). Left: above 1.2 a good token's gradient vanishes, but below 0.8 it stays active. Right: a bad token whose probability has risen is still pushed down. Redrawn from Eq. 7 and the paper's Figure 13 (§3); regions as in Lambert §6.2.67.

4.2 The min as a pessimistic bound

Because \(\min(a,b)\le a\), \(L^{CLIP}\le L^{CPI}\) pointwise, and the two agree to first order at \(r=1\)3 (§3). The authors describe the asymmetry: "we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse". In Table 3 that is the two flagged cells (the clip) against the two off-diagonal active cells (the pessimism). A good token that has already become more likely stops contributing; a good token whose probability has fallen below \(1-\epsilon\) still pushes up; a bad token whose probability has risen above \(1+\epsilon\) is still pushed down.

One qualification (the writer's reading). The paper's introduction calls the clipped objective "a pessimistic estimate (i.e., lower bound) of the performance of the policy"3 (§1), but what \(\min\) bounds is \(L^{CPI}\), the surrogate. TRPO's penalty form bounds the true return \(\eta\) (Section 2.2). PPO's pessimism makes the optimizer unwilling to exploit surrogate errors in the favourable direction; it does not carry TRPO's improvement guarantee.

4.3 How big is \(\epsilon=0.2\) next to \(\delta=0.01\)?

For small policy changes, \(\mathrm{KL}(\pi_{old}\|\pi_\theta)=\mathbb{E}_{old}[-\log r]\approx\tfrac12\mathbb{E}_{old}[(r-1)^2]\), because \(\mathbb{E}_{old}[r]=1\) kills the linear term (derived). The root-mean-square deviation of the ratio is therefore \(\sqrt{2\,\mathrm{KL}}\): 0.141 for TRPO's \(\delta=0.01\), and a KL of \(\tfrac12\epsilon^2=0.02\) for a typical ratio sitting exactly at \(1\pm0.2\). On random 50-way softmax pairs with logit noise up to 0.2 the approximation is within 2% of the exact KL (derived by script). The PPO paper's Figure 2 lands in the same place: the updated policy at the maximum of \(L^{CLIP}\), for the first Hopper update, "has a KL divergence of about 0.02 from the initial policy"3 (§3, Fig. 2).

That is an equivalence of scale, not of guarantee. TRPO bounds the mean KL across states; PPO's clip acts on individual samples in the tails of the ratio distribution and says nothing about the mean (Section 5).

4.4 The loop: sample, then several optimizer passes over the same data

PPO's Algorithm 1 is a plain loop3 (§5): run \(\pi_{\theta_{old}}\), compute advantage estimates, optimize the surrogate "with \(K\) epochs and minibatch size \(M\le NT\)", set \(\theta_{old}\leftarrow\theta\), repeat. With shared policy and value parameters the objective adds a value-regression term and an entropy bonus, \(\hat{\mathbb E}_t[L^{CLIP}_t-c_1L^{VF}_t+c_2S[\pi_\theta](s_t)]\) (Eq. 9). In an LLM trainer the "timesteps" \(NT\) are the response tokens of a rollout batch, and \(K\) and the minibatch count set how stale the last optimizer step is (Module 2, Table 5). The clipped term is the part found in every PPO-family trainer (the code is the writer's, written from Eq. 7; the token-mean normalization is one of several in use, see Module 11):

def ppo_clip_loss(logp_new, logp_old, adv, mask, eps=0.2):
    # (B, L) tensors. logp_old and adv are constants from the rollout phase.
    ratio = torch.exp(logp_new - logp_old)                 # r_t(theta)
    unclipped = ratio * adv                                # L^CPI term
    clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * adv
    per_token = torch.minimum(unclipped, clipped)          # Eq. 7
    loss = -(per_token * mask).sum() / mask.sum()          # token-mean over the batch
    # fraction of tokens outside the interval (a monitor, not all have zero gradient)
    outside = ((ratio - 1).abs() > eps) & (mask > 0)
    return loss, outside.float().sum() / mask.sum()

Two properties of this loop decide how much the clip matters in practice.

  • The first optimizer step is unclipped. The ratios start at exactly 1, so the first step "is identical to a maximization step over the unclipped surrogate objective"10 (§4); the clip can only act on later passes. Tulu 3 makes the dependency explicit: rollout-phase and learning-phase log-probabilities must match so that the ratio is 1 on the first pass, and if they differ "drastically due to dropout, all ratios could potentially be clipped, resulting in zero gradient", so dropout is disabled11 (§6.2). A mismatch between sampling engine and trainer has the same effect (Module 14).
  • Many LLM recipes barely reuse data. PPO and GRPO are often run with one gradient step per batch, so "the PPO-native regularization is never applied" and the KL to the reference dominates; DAPO uses 16 gradient steps per batch, and Tulu 3 uses \(K=4\) for 8B and 70B and \(K=1\) for 405B7 (§6.4.2)11 (Tables 21, 35). Lambert's practical monitor is the clip fraction, with the caveat that only the flat cells of Table 3 have zero gradient7 (§6.4.2).

4.5 Why the clip beat the KL penalty

PPO's paper includes the alternative that keeps TRPO's penalty form: add \(-\beta\,\mathrm{KL}[\pi_{\theta_{old}},\pi_\theta]\) to the surrogate and adapt \(\beta\) so each update lands near a target \(d_{targ}\). After optimizing, compute \(d=\hat{\mathbb E}_t[\mathrm{KL}]\); if \(d\lt d_{targ}/1.5\), halve \(\beta\); if \(d\gt1.5\,d_{targ}\), double it3 (§4, Eq. 8). The authors report that "the KL penalty performed worse than the clipped surrogate objective" and kept it "because it's an important baseline". Figure 4 plots their ablation (7 MuJoCo tasks, 3 seeds, 21 runs per bar).

Average normalized score on seven MuJoCo tasks for PPO surrogate variants: no clipping or penalty, clipping at three epsilons, adaptive KL at three targets, fixed KL at four coefficients -0.500.51.0 average normalized score (random policy = 0, best result = 1) No clipping or penalty -0.39 Clipping, ε = 0.1 0.76 Clipping, ε = 0.2 0.82 Clipping, ε = 0.3 0.70 Adaptive KL, d_targ = 0.003 0.68 Adaptive KL, d_targ = 0.01 0.74 Adaptive KL, d_targ = 0.03 0.71 Fixed KL, β = 0.3 0.62 Fixed KL, β = 1 0.71 Fixed KL, β = 3 (β = 10: 0.69) 0.72
Figure 4. PPO's comparison of surrogate objectives: mean normalized score over 21 runs (7 MuJoCo tasks, 3 seeds), 1M timesteps, \(\beta\) initialized at 1 for the adaptive variants. Clipping at \(\epsilon=0.2\) scores highest (0.82); best adaptive KL 0.74; best fixed KL 0.72 (\(\beta=10\), 0.69, not drawn). The paper reports no spread, so do not over-read differences of a few hundredths. Data: Table 13 (§6.1).

The comparison supports a narrow claim: on small continuous-control tasks with the other hyperparameters fixed (the paper's Table 3), the best clipped setting beat the best KL-penalty settings (0.82 against 0.74 and 0.72). The spread across \(\epsilon\) (0.70 to 0.82) is as wide as the spread across KL-penalty settings (0.62 to 0.74), so it does not show that a KL penalty is worse in general. Clipping in log space was "no better"3 (§6.1). For LLM RL the record is thinner and mixed (Where experts disagree).

5. What the clip does not guarantee

The clip is a cheap heuristic that became the default. Four of its limits are documented, and each returns in a later module.

5.1 There is no KL bound; the optimizer sets the step

Engstrom et al. measured what TRPO and PPO actually enforce on MuJoCo. TRPO "precisely enforces" its mean-KL region, "nearly by construction". PPO does not enforce the ratio region it is trained on: the maximum ratio "consistently" violates \(1+\epsilon\), and "all three algorithms fail to maintain a ratio-based trust region, despite PPO and PPO-M being trained directly with a ratio-clipping objective"10 (§4, Fig. 2). Their explanation is the one from Section 4.4: the first step is unclipped, and after that the step is "determined solely by the steepness of the surrogate landscape", so the trust region PPO realizes depends on the optimizer and its learning rate, not on the objective10 (§4).

The same study found that "code-level optimizations" present in the PPO implementation but not in the TRPO baseline (value-function clipping, reward scaling, learning-rate annealing, global gradient clipping and others) "are responsible for most of PPO's gain in cumulative reward over TRPO"10 (abstract, §3). The LLM analogue is Tulu 3's PPO recipe: advantage whitening, a gradient-norm threshold of 1.0, a value-function coefficient of 0.1, a \(-10\) penalty for responses without EOS, and dropout disabled11 (§6.2, Table 21). When two reports both say "PPO" and differ by a few points, details like these are the first suspects (Module 5).

5.2 The probability headroom is relative, so rare tokens barely move

The clip bounds the ratio, not the probability change. A token with \(\pi_{old}=p\) can gain at most \(\epsilon p\) in probability before its gradient dies, and can lose at most \(\epsilon p\). DAPO states the consequence with two numbers: for \(\epsilon=0.2\) and \(\hat A>0\), tokens at \(0.01\) and \(0.9\) have upper bounds of \(0.012\) and \(1.08\); the high-probability token is "not constrained to get even extremely larger probabilities like 0.999", while the rare one needs a large relative rise to matter12 (§3.1). They also measure that the mean probability of up-clipped tokens is below 0.212 (§3.1, Fig. 3a). Table 4 generalizes (derived from the clip definition).

Table 4. Probability reachable within the clip for a token with \(\hat A>0\) (upper limits) or \(\hat A<0\) (lower limit), by prior probability \(p\). "Gain" is limit minus \(p\). Derived by the writer from Eq. 7 with \(\epsilon=0.2\) and DAPO's asymmetric \(\epsilon_{high}=0.28\)12 (§3.1, §4.1). A limit of 1 or more means the up-clip cannot fire.
Prior \(p\)Upper limit, \(\epsilon=0.2\)GainUpper limit, \(\epsilon_{high}=0.28\)GainLower limit, \(\epsilon=0.2\)
0.010.012+0.0020.0128+0.00280.008
0.050.06+0.0100.064+0.0140.04
0.10.12+0.0200.128+0.0280.08
0.30.36+0.0600.384+0.0840.24
0.50.60+0.1000.64+0.1400.40
0.91.08 (never fires)to 1.01.152 (never fires)to 1.00.72

Two thresholds follow (derived): with \(\epsilon=0.2\) the up-clip can never fire for \(p\ge1/1.2=0.833\), and with \(\epsilon_{high}=0.28\) not for \(p\ge0.781\). Compare Table 1: a KL budget of 0.01 let a \(p=0.01\) token triple, while the clip stops it at \(1.2\times\). DAPO's response is Clip-Higher: \(\epsilon_{high}=0.28\), \(\epsilon_{low}=0.2\), keeping the lower bound because raising it "will suppress the probability of these tokens to 0, resulting in the collapse of the sampling space"12 (§3.1, §4.1). MiniMax-M1 reaches the same diagnosis from the other side: rare reflective "fork" tokens ("However", "Recheck", "Wait", "Aha") have low base-model probability, receive a large ratio after the first update, and are clipped out of every later off-policy update13 (§3.1). Its fix, CISPO, clips the importance weight and keeps every token's gradient (Module 12).

5.3 The clip is biased even when the reward carries no information

The cleanest demonstration is the Spurious Rewards study. Train Qwen2.5-Math-7B with GRPO on random rewards, independent of the rollout, and MATH-500 still rises by 21.4 points absolute, against 29.1 with ground-truth rewards14 (abstract). The authors' account is a gradient bias from the clip term. With random rewards the group-centered advantage has zero mean, so without clipping the expected gradient is zero. With clipping, the expected token gradient (their Eq. 2, up to a positive constant) is \(+\nabla\log\pi\) when \(r\lt1-\epsilon\), 0 inside the interval, and \(-\nabla\log\pi\) when \(r\gt1+\epsilon\): in each outer region the half of the advantage distribution that would push further out is clipped away, leaving a net push back toward the interval (the sign-by-region reading follows from Table 3)14 (§4, App. B). Since \(r\le1/p_{old}\), a token with \(p_{old}=0.85\) can never reach the upper clip at 1.2, while one at 0.02 is out of range once its probability passes 0.024; the net effect "asymmetrically suppresses low-probability tokens and reinforces high-probability ones"14 (§4). With clipping disabled (three variants, including one update per rollout) random rewards give no consistent gains14 (§4, Fig. 4). The behaviour amplified in Qwen2.5-Math is code-style reasoning, up from 65.0% of MATH-500 solutions to about 90% within 15 steps14 (§5.2), and the effect is model-dependent: spurious rewards stay effective on Qwen2.5 models but generally fail on other families such as OLMo2 and Llama314 (§1, §3, Fig. 3).

Read this as a documented mechanism with a hypothesis attached: the paper presents the clipping-bias account as a hypothesis that its ablations support, on one model family. What it settles here is that the clip term is not a neutral stability device; it carries a prior-favouring bias that interacts with whatever the pretrained model already does well (Module 20).

5.4 How loose the clip is allowed to be in reasoning recipes

DeepSeek-R1's report lists, for the first RL stage, "the GRPO clip ratio \(\epsilon\) to 10", with 16 minibatches of 512 outputs from each 8,192-output rollout and one inner epoch6 (§3.2.1). With \(\epsilon=10\) the interval is \([-9,11]\); a ratio is never negative, so the lower clip cannot fire, and the upper clip fires only when a token's probability rises elevenfold within one rollout (derived). I read that as a clip close to switched off, with step control carried by the learning rate (3e-6), the few optimizer steps per rollout and the KL coefficient of 0.001; the report does not say why it chose 10, so that reading is the writer's. The R1-Zero hyperparameters in the main text (§2.1) do not state \(\epsilon\) at all6.

The R1 report also writes its GRPO objective with a sequence-level ratio and no \(1/|o_i|\) factor6 (§2.1, Eq. 1), where DeepSeekMath's Eq. 3 had a token-level ratio and \(1/|o_i|\)5 (§4.1.1); the report does not say whether that is shorthand or an implementation change. When you read a PPO-family loss, check where the ratio and the normalization live (Modules 11 and 12).

6. Generalized advantage estimation

PPO needs an advantage \(\hat A_t\) per timestep. Module 2 showed that a learned \(V\) redistributes the sequence-level credit \(R-V(x)\) along the tokens and listed the TD residual as one choice of \(\Psi_t\). GAE interpolates between that residual and the Monte Carlo return, and it is the advantage inside every critic-based recipe in this course.

6.1 Definition and recursion

With a value function \(V\) and discount \(\gamma\), the TD residual is \(\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)\). Summing \(k\) residuals gives the \(k\)-step estimator \(\hat A_t^{(k)}=\sum_{l=0}^{k-1}\gamma^l\delta_{t+l}=-V(s_t)+r_t+\gamma r_{t+1}+\dots+\gamma^{k-1}r_{t+k-1}+\gamma^kV(s_{t+k})\), which is a \(k\)-step return minus a baseline. GAE is the exponentially weighted average of these2 (§3, Eqs. 11-16):

$$\hat A_t^{GAE(\gamma,\lambda)}=(1-\lambda)\big(\hat A_t^{(1)}+\lambda\hat A_t^{(2)}+\lambda^2\hat A_t^{(3)}+\dots\big)=\sum_{l=0}^{\infty}(\gamma\lambda)^l\,\delta_{t+l}.$$

Computed backward over a finished trajectory, the sum is a one-line recursion, \(\hat A_t=\delta_t+\gamma\lambda\hat A_{t+1}\), with \(\hat A_{T+1}=0\); that is the loop in Lambert's reference implementation7 (§6.4.1), which also resets the accumulator at terminal tokens so that packed sequences do not leak into each other. The two endpoints are2 (Eqs. 17-18):

  • \(\lambda=0\): \(\hat A_t=\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)\). One real reward, then trust the critic.
  • \(\lambda=1\): \(\hat A_t=\sum_l\gamma^lr_{t+l}-V(s_t)\). The empirical return minus the critic as a baseline, with no bootstrapping.
GAE as a spectrum from one-step TD at lambda 0 to Monte Carlo at lambda 1, with the weight placed on each TD residual for three lambdas λ = 0: one-step TDλ = 1: Monte Carlo return minus V more bias if V is wrong more variance from rewards λ = 0 Â = δ_t λ = 0.9 Â = Σ 0.9^l δ_(t+l) λ = 1 Â = return − V(s_t) weight on δ_(t+l), l = 0 to 9 steps ahead (γ = 1)
Figure 5. The GAE spectrum: the weight \((\gamma\lambda)^l\) that the advantage at token \(t\) puts on the TD residual \(l\) steps ahead, for \(\gamma=1\). At \(\lambda=0\) the estimate leans entirely on \(V\); at \(\lambda=1\) all residuals count equally and the sum telescopes to the empirical return. Weights from Eq. 162; annotations follow the paper's discussion (§3).

6.2 Two knobs, two different kinds of bias

The paper's bias-variance statements are specific, and the distinction between \(\gamma\) and \(\lambda\) is the useful part2 (§3). An advantage estimator is \(\gamma\)-just if it does not bias the discounted policy gradient; a sufficient condition (Proposition 1) is that it equals an unbiased estimate of \(Q^{\pi,\gamma}\) minus any baseline that depends only on earlier states and actions. Then:

  • \(\mathrm{GAE}(\gamma,1)\) is \(\gamma\)-just "regardless of the accuracy of \(V\)", but has high variance from the long sum of rewards.
  • \(\mathrm{GAE}(\gamma,0)\) is \(\gamma\)-just only if \(V=V^{\pi,\gamma}\); otherwise it is biased, but has much lower variance.
  • "Taking \(\gamma\lt1\) introduces bias into the policy gradient estimate, regardless of the value function's accuracy. On the other hand, \(\lambda\lt1\) introduces bias only when the value function is inaccurate."

That asymmetry is the paper's explanation for why the best \(\lambda\) is lower than the best \(\gamma\): \(\lambda\) is the cheaper knob. In their cart-pole sweep "the best results are obtained at intermediate values of the parameters: \(\gamma\in[0.96,0.99]\) and \(\lambda\in[0.92,0.99]\)"2 (§6.3.1). The discount also has a horizon reading: \(\gamma\lt1\) drops terms with \(l\gg1/(1-\gamma)\), and \(\gamma\lambda\) ignores residuals with \(l\gg1/(1-\gamma\lambda)\)2 (§4). For PPO's MuJoCo setting (\(\gamma=0.99\), \(\lambda=0.95\))3 that is \(1/(1-0.9405)=16.8\) steps (derived). With potential \(\Phi=V\) the shaped reward is the residual \(\delta^V\), and a good \(V\) turns a temporally extended response of reward to action into an immediate one2 (§4, Eqs. 20-25).

One more TRPO echo: the GAE paper fits the value function under its own average-KL trust region, solved with conjugate gradient2 (§5, Eqs. 29-30). PPO's reference implementation (OpenAI baselines) instead fits the value network with a clipped, PPO-like objective, listed first among its code-level optimizations10 (§3); Lambert's critic code carries the same optional clip7 (§6.2.7).

6.3 The LLM case in closed form, and a hand computation

Take \(\gamma=1\), a reward only at the EOS token, \(r_T=R\), and \(V=0\) after the terminal state (per-token KL terms are ignored here; Module 5 adds them). Then \(\delta_t=V(s_{t+1})-V(s_t)\) for \(t\lt T\) and \(\delta_T=R-V(s_T)\), as in Module 2 Section 5.1. Summing \(\sum_l\lambda^l\delta_{t+l}\) and rearranging gives (derived; the derivation follows below):

$$\hat A_t=\underbrace{\lambda^{T-t}R+(1-\lambda)\sum_{m=1}^{T-t}\lambda^{m-1}V(s_{t+m})}_{\text{weights sum to }1}\;-\;V(s_t).$$

The advantage is a convex mixture of what actually happened (\(R\)) and what the critic predicts at later positions, minus what it predicts now. At \(\lambda=1\) the mixture is all \(R\) and \(\hat A_t=R-V(s_t)\); at \(\lambda=0\) it is all \(V(s_{t+1})\). The part that matters for LLMs is the weight \(\lambda^{T-t}\): the final reward reaches a token \(k=T-t\) positions before the end directly only with weight \(\lambda^k\); everything else the token sees is the critic talking to itself.

Derivation by summation by parts

Split \(\hat A_t=\sum_{l=0}^{n-1}\lambda^l\big(V(s_{t+l+1})-V(s_{t+l})\big)+\lambda^{n}\big(R-V(s_T)\big)\) with \(n=T-t\). The first sum telescopes with weights: \(\sum_{l=0}^{n-1}\lambda^lV_{t+l+1}-\sum_{l=0}^{n-1}\lambda^lV_{t+l}=-V_t+(1-\lambda)\sum_{m=1}^{n-1}\lambda^{m-1}V_{t+m}+\lambda^{n-1}V_{t+n}\). Since \(V_{t+n}=V_T\), the terms in \(V_T\) combine as \(\lambda^{n-1}V_T-\lambda^nV_T=(1-\lambda)\lambda^{n-1}V_T\), which extends the weighted sum to \(m=n\). What remains is \(\lambda^nR+(1-\lambda)\sum_{m=1}^{n}\lambda^{m-1}V_{t+m}-V_t\). The weights \(\lambda^n+(1-\lambda)\sum_{m=1}^{n}\lambda^{m-1}=\lambda^n+(1-\lambda^n)=1\). I verified the closed form against the backward recursion numerically on the example below for \(\lambda\in\{0,0.5,0.9,0.95,1\}\).

Table 5 computes it by hand for a five-token response. The critic values \(V(s_1),\dots,V(s_5)\) are invented for illustration (\(V(s_5)=0.30\) is the predicted success probability just before the EOS token), the verifier returns \(R=1\), and \(\gamma=1\). The residuals are \(\delta=(-0.10,\,0.15,\,0.05,\,-0.30,\,0.70)\), whose sum \(0.50\) equals \(R-V(s_1)\), the telescoping identity of Module 2. For \(\lambda=0.9\) the backward recursion \(\hat A_t=\delta_t+0.9\,\hat A_{t+1}\) runs: \(\hat A_5=0.70\); \(\hat A_4=-0.30+0.9(0.70)=0.33\); \(\hat A_3=0.05+0.9(0.33)=0.347\); \(\hat A_2=0.15+0.9(0.347)=0.4623\); \(\hat A_1=-0.10+0.9(0.4623)=0.3161\).

Table 5. GAE advantages for one five-token response with a final reward \(R=1\), \(\gamma=1\), and invented critic values (illustrative; all numbers derived by the writer from Eq. 162). The \(\lambda=1\) column is \(R-V(s_t)\); the \(\lambda=0\) column is the raw residual \(\delta_t\).
Token \(t\)\(V(s_t)\)\(\delta_t\)\(\hat A\), \(\lambda=0\)\(\hat A\), \(\lambda=0.5\)\(\hat A\), \(\lambda=0.9\)\(\hat A\), \(\lambda=1\)
10.50-0.10-0.1000-0.00630.31610.5000
20.400.150.15000.18750.46230.6000
30.550.050.05000.07500.34700.4500
40.60-0.30-0.30000.05000.33000.4000
5 (EOS)0.300.700.70000.70000.70000.7000

Read down the token-4 row. The critic says the fourth token cut the predicted success probability from 0.60 to 0.30. At \(\lambda=0\) the advantage records that (\(-0.30\)) and the token is discouraged although the response succeeded; at \(\lambda=1\) it is \(+0.40\) and the token is reinforced, because the only information used is that the whole response was correct. If the critic is right, \(\lambda=0\) is the sharper signal; if it is wrong, \(\lambda=0\) bakes the error into the gradient. That is the bias-variance trade in one cell.

6.4 What \(\lambda\) does at reasoning-model lengths

The effective horizon in Section 6.2 was about 17 steps. A reasoning trace is thousands of tokens, and \(\lambda^k\) decays fast: Figure 6 plots the direct weight of the final reward against distance from the end.

Weight that the final reward receives at a token k positions before the end, lambda to the power k, for lambda 0.95, 0.99 and 0.999, on a log axis 0.25 0.50 0.75 1.00 0 1 10 100 1,000 k = tokens between this token and the end of the response (log scale) λ^k: share of the final reward that reaches the token through its own TD residual (γ = 1) 0.95^100 = 0.0059 λ = 0.95 λ = 0.99 λ = 0.999 dot: k = 1/(1−λ)
Figure 6. The weight \(\lambda^k\) that the final reward receives at a token \(k\) positions before EOS (\(\gamma=1\); Section 6.3). Dots mark \(k=1/(1-\lambda)\), where each curve has fallen to \(e^{-1}\approx0.37\): 20, 100 and 1,000 tokens. At \(k=100\), \(\lambda=0.95\) gives 0.0059, the "\(\approx0.006\)" VAPO quotes15 (§4.2). Derived from \(\lambda^k\).

At \(\lambda=0.95\), a token 100 positions from the end sees 0.6% of the reward directly, and one 1,000 positions away about \(5\times10^{-23}\) (derived). For \(\lambda=0.99\) the 1,000-token weight is \(4.3\times10^{-5}\); only \(\lambda=0.999\) leaves 0.37. The VAPO authors draw the practical conclusion: with \(\lambda=0.95\) value updates "become almost entirely bootstrapped, relying on highly biased estimates", because the end reward "propagates backward as \(\lambda^{T-t}R\)"15 (§3.1). In the closed form of Section 6.3 the early tokens' advantages are then dominated by the critic's own predictions, which makes critic quality decisive. Three reports bear on it, and they agree on direction.

Table 6. Evidence on the GAE \(\lambda\) for LLM policy optimization, as reported by each paper's authors. RLOO paper §3.1, Fig. 116; DeepSeek-R1 App. A.36; VAPO §4-§5, Table 115. Tulu 3 fixes \(\lambda=0.95\), \(\gamma=1\) with responses capped at 1,024 to 2,048 tokens and reports no \(\lambda\) sweep11 (Table 21); Lambert calls 0.95 "a common default for LLM fine-tuning"7 (§6.4.1).
ReportSetting\(\lambda\) comparedReported result
Ahmadian et al. (RLOO paper)PPO, Llama-7B-based models, HH dataset1.0, 0.95, 0.5, 0"monotonically decreasing reward with decreasing \(\lambda\)"; \(\lambda=1\) best
DeepSeek-R1, Supplement A.3PPO vs GRPO on MATH, DeepSeekCoder-V2-Lite (16B MoE, 2.4B active)0.95 (the open-source default), 1.00.95: "considerably worse than GRPO"; 1.0: improves "substantially, nearing" GRPO
VAPO (ByteDance Seed)Qwen-32B, AIME24 avg@320.95 with RM-initialized value; decoupled; length-adaptivevanilla PPO 5; without decoupled GAE 33; without length-adaptive GAE 45; full VAPO 60

The RLOO paper's argument is structural: a pretrained LLM starts with probability mass concentrated on a few tokens, so variance is less of a problem than in MuJoCo and bias is the avoidable cost16 (§3.1). R1 adds a mechanism for why a critic is hard on long reasoning: reflection and revision mean early content "may later be revised or contradicted, which makes it even less feasible to predict the final reward based on a partial response"6 (App. A.3). VAPO is the constructive answer from the PPO side, with three changes15 (§4.1, §4.2, §5.1, Table 1): decoupled GAE (value target with \(\lambda_{critic}=1.0\), policy advantage with a separate \(\lambda\)); value pretraining (a 50-step value warmup, without which VAPO reaches 11 on AIME24); and length-adaptive GAE, \(\lambda_{policy}=1-1/(\alpha\ell)\) with \(\alpha=0.05\) and \(\ell\) the response length, so that \(1/(1-\lambda)\) is proportional to length. For \(\ell=100,1{,}000,10{,}000\) that gives \(\lambda=0.8,0.98,0.998\) (derived). One consequence the paper does not state: \(\lambda^\ell\to e^{-1/\alpha}=e^{-20}\approx2\times10^{-9}\) for long responses, so the design does not make the final reward reach the first token directly; it fixes the bootstrapping horizon at 5% of the response and relies on the \(\lambda=1\)-trained critic for the rest (the writer's derivation).

Two cautions. The VAPO and R1 comparisons are ablations by the proposers, on their own recipes. And the field has not converged on critic initialization: Tulu 3 reports "Initializing RLVR's Value Function from a General RM Works Best"11 (§6.2.1), while VAPO attributes a positive initialization bias to RM-initialized value models (following VC-PPO) and pretrains the value function separately15 (§3.1). GRPO removes the \(\lambda\) problem by giving every token the same group-relative advantage, which corresponds to the \(\lambda=1\) end of the spectrum with an unlearned baseline; Module 9 takes up that comparison.

7. The lineage in one table, and what GRPO keeps

Everything so far can be compared on the same rows. Table 7 places TRPO, PPO as published, PPO as run in LLM post-training, and GRPO side by side; Table 8 puts the reference hyperparameters of the original papers next to the values that later LLM recipes report.

Table 7. TRPO, PPO and GRPO compared. TRPO: §4-§6, App. C1; GAE: §32; PPO: §3, §5, App. A3; LLM PPO: InstructGPT §3.54 and Tulu 3 Table 2111; GRPO: DeepSeekMath §4.15 and R1 §2.1, §3.2.16. The last row is the writer's summary of Sections 2 and 5.
TRPO (2015)PPO-clip (2017)PPO for LLMsGRPO
Step controlmean KL(old, new) \(\le\delta=0.01\), true KL checked by line searchper-sample ratio clip, \(\epsilon=0.2\); no KL checksame clip, \(\epsilon=0.2\); KL to reference folded into the rewardsame clip; KL to reference added to the loss
Optimizerconjugate gradient on Fisher-vector products, then line searchfirst-order (Adam), several epochs of minibatchesfirst-order; \(K=4\) (8B, 70B) or 1 (405B) in Tulu 3first-order; 1 update (DeepSeekMath) or 16 minibatches (R1)
Criticnone in the algorithm; GAE paper adds a value net with its own trust regionvalue net, separate or shared with the policypolicy-sized value model, initialized from an RMnone
Advantage\(\hat Q\) from single-path or vine rollouts (GAE paper: GAE)GAE, \(\gamma=0.99\), \(\lambda=0.95\)per-token GAE, \(\gamma=1\), \(\lambda=0.95\)group-normalized reward, one value per sequence
Improvement guaranteetheory only, for \(\gamma\lt1\) and exact advantagesnonenonenone
Table 8. Reference hyperparameters: the original papers and later reports. Dashes mean the report does not state the value (or the method has none). PPO MuJoCo: its Appendix A Table 3 and §6.1; PPO Atari: its Appendix A Table 53; TRPO: §8 and App. E1; Tulu 3 RLVR 8B: Table 2111; DeepSeekMath-RL: §4.25; R1 first RL stage: §3.2.16; DAPO: §3.1, §4.112; VAPO: §4.2, §5.115.
Setting\(\gamma\)\(\lambda\)Clip \(\epsilon\) or KL boundReuse per rolloutLearning rate
TRPO, locomotion0.99-\(\delta=0.01\)one constrained step per batch-
PPO, MuJoCo (1M steps)0.990.950.210 epochs, minibatch 64, horizon 2,048\(3\times10^{-4}\)
PPO, Atari0.990.95\(0.1\alpha\), \(\alpha\) annealed 1 to 03 epochs, minibatch \(32\times8\), horizon 128, 8 actors\(2.5\times10^{-4}\alpha\)
Tulu 3 RLVR, 8B (PPO)1.00.950.2; KL coefficient 0.05\(K=4\), 1 minibatch\(3\times10^{-7}\)
DeepSeekMath-RL (GRPO)--not stated; KL coefficient 0.04single update per exploration\(1\times10^{-6}\)
R1, first RL stage (GRPO)--10; KL coefficient 0.00116 minibatches of 512, 1 epoch\(3\times10^{-6}\)
DAPO (GRPO variant)--0.2 low, 0.28 high; no KL16 updates (minibatch 512 of 8,192)\(1\times10^{-6}\)
VAPO (PPO, Qwen-32B)1.0critic 1.0; policy \(1-1/(0.05\ell)\)0.2 low, 0.28 high16 updates (minibatch 512 of 8,192)actor \(1\times10^{-6}\), critic \(2\times10^{-6}\)

7.1 Kept, dropped, moved

DeepSeekMath writes PPO and GRPO with the same shape: a per-token ratio, the same \(\min\) and clip, and a \(1/|o|\) average over tokens5 (§4.1.1, Eqs. 1 and 3). What changes is the advantage and the KL.

  • Kept from TRPO via PPO: the ratio, the surrogate \(rA\), the pessimistic min, the clip interval, and reuse of one rollout for several optimizer steps. The 1/|o| length averaging was already in open-source PPO implementations17 (§3.1); Lambert notes it "comes from common implementation practices, but is not in a formal derivation of the loss"7 (§6.2.5).
  • Dropped: the value model and GAE, for the memory of a policy-sized critic and the difficulty of an accurate per-token value when only the last token is rewarded5 (§4.1.1), plus R1's reflection argument (Section 6.4)6.
  • Moved: the KL to the reference leaves the reward and enters the loss, "avoiding complicating the calculation of \(\hat A_{i,t}\)"5 (§4.1.1). R1 adds that PPO's per-token KL reward penalizes cumulative KL, "which may implicitly penalize the length of the response"6 (App. A.3).
  • Added: a group baseline, \(\hat A_i=(r_i-\mathrm{mean})/\mathrm{std}\) over the \(G\) samples of a prompt6 (§2.1, Eq. 3). Module 9 develops it; Modules 11 and 12 take apart the length weighting and the ratio.

One branch left the PPO family. Kimi k1.5 trains with a variant of online policy mirror descent (a relative-entropy-regularized objective whose reference is the current model, a mean-of-\(k\)-samples baseline and a squared-error surrogate) and uses no value function9 (abstract, §2.3.2). That is the mirror-descent reading of Section 2.2 taken literally (Module 9).

Where experts disagree

For LLM RL the live question about this lineage is not whether to use PPO's surrogate but what, if anything, should limit the step. The four positions differ on whether a per-token ratio clip is the right device, and each rests on a different kind of evidence. Module 12 develops the variants.

A. Keep the token-level clip and retune it

PPO's authors3; DeepSeekMath and R156; DAPO's Clip-Higher12; Tulu 311

The clip is the cheapest device that makes multi-step reuse of a rollout safe, and it needs only per-token log-probabilities. Where it hurts (entropy collapse from suppressed rare tokens) the fix is a different \(\epsilon_{high}\), not a different mechanism: DAPO's ladder goes from 36 to 38 AIME24 points when Clip-Higher is added to overlong filtering12 (Table 1).

  • Implemented everywhere; one scalar to tune.
  • Beat adaptive and fixed KL penalties on MuJoCo3 (Table 1).
  • No KL bound; the realized trust region is set by the optimizer10.
  • Biased under random rewards; relative headroom starves rare tokens (Table 4)14.

B. The trust region is unnecessary in LLM RL

Ahmadian et al. (Cohere)16; Lambert7

A pretrained LLM already concentrates probability on a few tokens, so destructive steps are rare: the PPO loss was clipped "on average < 5% of the time per batch", and removing clipping and the ratio "gives a slight boost"16 (§3.2). Lambert adds that with one gradient step per batch the PPO-native regularization "is never applied"7 (§6.4.2).

  • No ratio, no clip bias, no second set of log-probabilities to keep consistent.
  • With one update per exploration the clip is inert even by DeepSeekMath's analysis5 (App. A.1.5).
  • Needs fresh rollouts per step; DAPO and R1 reuse a rollout 16 times126.
  • Evidence is short-response RLHF on 7B-class models, not long-CoT RLVR.

C. Move the ratio to the sequence

Qwen team, GSPO18

The authors trace GRPO's instability to "the fundamental misapplication and invalidation of importance sampling weights": a token-level weight rests on a single sample per next-token distribution, so it cannot correct the distribution and injects noise that grows with length and is amplified by clipping. GSPO uses the length-normalized sequence likelihood ratio with clip ranges of 3e-4 and 4e-4 (GRPO tuned to 0.2 and 0.27), and drops the need for Routing Replay on their MoE18 (§1, §3-§5.3).

  • Matches a sequence-level reward; large MoE-stability gain.
  • The SAPO authors argue that one sequence-level ratio suppresses a whole sequence when only a few tokens are off-policy.
  • Evidence is from the proposing team's own models.

D. Keep every token's gradient; clip the weight or gate softly

MiniMax-M1, CISPO13; SAPO per Lambert and TRL719

PPO-style clipping removes rare "fork" tokens from later updates. CISPO "clips importance sampling weights rather than token updates" and matches DAPO's score with 50% of the steps on Qwen2.5-32B13 (abstract, §3.1). SAPO replaces the hard clip with a smooth temperature-controlled gate7 (§6.4.3)19.

  • Addresses the rare-token problem of Table 4 directly.
  • Gives up the pessimistic min; an extreme weight is bounded only by the chosen cap, and comparisons are by the proposers.
Assessment

This is the writer's reasoning, not a sourced result. Where rollouts are used once, A and B are the same algorithm, because the ratio is 1 and the clip never fires; the choice only matters once data is reused or stale (asynchronous pipelines, 16-minibatch regimes, MoE routing noise). Then the evidence is uneven: B rests on short-response RLHF at 7B, C and D on the proposers' ablations on their own long-CoT and MoE models. A with a raised \(\epsilon_{high}\) is the lowest-risk default today, because it is the baseline everything else is measured against, and R1's \(\epsilon=10\) suggests the exact range matters less than the learning rate, the reuse count and the KL. What would settle it: matched-compute runs on one base model and verifier set that vary only the ratio treatment at fixed staleness, reporting clip fraction, mean KL to the sampling policy and entropy with the score. No such study is in this course's source map.

What is changing

Three things are moving as of October 2026. The clip is becoming a configuration surface: the Hugging Face TRL GRPO trainer exposes \(\epsilon\) (default 0.2), a separate \(\epsilon_{high}\) (DAPO recommends 0.28), an importance-sampling level that defaults to "token" with "sequence" giving GSPO's behaviour, and DAPO, CISPO and SAPO loss types19; the page is main-branch documentation without a version number, so read the defaults as a snapshot. The critic is being re-argued, not retired: VAPO is a 2025 attempt to make a value model work at reasoning lengths, while Tulu 3 listed value-free GRPO as untried future work11 (Limitations). And the trust-region idea is migrating from the step to the data: the ratio is increasingly a mask or weight against sampler-trainer mismatch and staleness (TRL turns sequence-level masked importance sampling on by default for vLLM generation19; Modules 12, 14 and 18).

References

  1. paperSchulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv v5, 20 Apr 2017. https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Used for: Theorem 1 and Eq. 9 (§3); average-KL constraint and importance-sampling form, Eqs. 12 and 14 (§4-§5); single path and vine (§5); Fisher matrix, natural-gradient relation and Eqs. 17-18 (§6-§7); delta = 0.01 and the natural-gradient comparison (§8.1); conjugate gradient, Fisher-vector products and line search (App. C, C.1); Tables 2-3 (App. E).
  2. paperSchulman, J., Moritz, P., Levine, S., Jordan, M. I., Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016; arXiv v6, 20 Oct 2018. https://arxiv.org/pdf/1506.02438v6. Accessed 2026-10-26. Used for: GAE definition and endpoints, Eqs. 11-18 (§3); gamma-just estimators and Proposition 1; gamma versus lambda bias; horizon and reward-shaping reading (§4); value-function trust region, Eqs. 29-30 (§5); cart-pole ranges (§6.3.1); TRPO as the policy optimizer (§6.1).
  3. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: L^CPI and L^CLIP, Eqs. 6-7 and Figs. 1-2 (§3); adaptive KL penalty (§4); Algorithm 1 and combined objective, Eq. 9 (§5); the surrogate comparison, Table 1 (§6.1); hyperparameters, Tables 3 and 5 (App. A).
  4. paperOuyang, L. et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: PPO as the RL stage, per-token KL penalty from the SFT model, value function initialized from the RM (§3.5).
  5. paperShao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: introduction of GRPO; PPO and GRPO objectives, Eqs. 1 and 3, and the critic and KL arguments (§4.1.1); DeepSeekMath-RL settings (§4.2); the single-update simplification (App. A.1.5).
  6. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: GRPO as R1's algorithm and its displayed objective (§2.1, Eqs. 1-3); first-stage hyperparameters including clip ratio 10 (§3.2.1); PPO versus GRPO with lambda 0.95 and 1.0, and the value-model and KL arguments (App. A.3).
  7. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: PPO objective and the sign-by-region analysis (§6.2.5-§6.2.6, Fig. 23); value function and optional value clip (§6.2.7); GAE and gamma = 1 (§6.4.1); double regularization, clip fraction and update counts (§6.4.2); SAPO and VAPO summaries (§6.4.3).
  8. paperKazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., Le Roux, N. VinePPO: Refining Credit Assignment in RL Training of LLMs. ICML 2025; arXiv v2, 3 Jun 2025. https://arxiv.org/abs/2410.01679. Accessed 2026-10-27. Used for: the vine variant of TRPO transplanted to LLMs, Monte Carlo value estimates with K auxiliary rollouts, K = 9 default (§4, §5); the authors' MATH and GSM8K results and the 3.0x wall-clock claim (abstract, §6.1-§6.2). Source added by the writer beyond the source map.
  9. paperKimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: online policy mirror descent variant with a relative-entropy term and no value function (abstract, §2.3.2).
  10. paperEngstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., Madry, A. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. ICLR 2020; arXiv v1, 25 May 2020. https://arxiv.org/abs/2005.12729. Accessed 2026-10-27. Used for: code-level optimizations list (§3), PPO's first step equal to the unclipped step, ratio violations and KL enforcement of TRPO versus PPO (§4, Fig. 2), attribution of PPO's gain over TRPO (abstract). Source added by the writer beyond the source map.
  11. paperLambert, N. et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: PPO hyperparameters for RM and RLVR runs (Table 21; the 405B table, labelled Table 35); implementation details, dropout and advantage whitening, value initialization (§6.2, §6.2.1); the value-free GRPO limitation (§8.1, Limitations).
  12. paperYu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: Clip-Higher and the 0.01 / 0.9 example (§3.1, Fig. 3a); epsilon_low 0.2, epsilon_high 0.28 and 16 updates per rollout (§4.1); ablation ladder (Table 1).
  13. paperMiniMax. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: clipping of rare fork tokens and the CISPO weight clip (§3.1); the comparison with DAPO (abstract, Fig. 2).
  14. paperShao, R. et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv v2, 25 Feb 2026. https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: random-reward gains on Qwen2.5-Math-7B (abstract); clipping-bias account, Eq. 2, the 0.85 / 0.02 token example and the clipping ablation (§4, App. B, Fig. 4); code-reasoning frequency (§5.2); model dependence (§3, Fig. 3).
  15. paperYue, Y. et al. (ByteDance Seed). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: reward decay lambda^(T-t) and value bias with lambda 0.95 (§3.1); decoupled GAE, value pretraining and length-adaptive GAE (§4.1-§4.2); settings and ablation, Table 1 (§5). Source added by the writer beyond the source map.
  16. paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: the GAE lambda ablation (§3.1, Fig. 1); clipping rarely active, under 5% (§3.2).
  17. paperLiu, Z. et al. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: 1/|o| length normalization already present in open-source PPO implementations (§3.1).
  18. paperZheng, C. et al. (Qwen Team, Alibaba). Group Sequence Policy Optimization. arXiv v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: the single-sample argument against token-level importance weights (§1), sequence-level ratio and clip ranges (§3, §4), Routing Replay (§5.3).
  19. officialHugging Face. TRL documentation: GRPO Trainer (main branch; version not stated on the page). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-26. Used for: epsilon, epsilon_high, importance_sampling_level defaults and the DAPO, CISPO and SAPO loss types; default masked importance sampling for vLLM generation.