Reinforcement Learning for LLM Post-Training · Module 5

PPO-RLHF as a System: Policy, Reference, Reward and Value Models, Per-Token KL Reward, GAE, and the Pretraining Mix

InstructGPT-style RLHF runs four language models inside one loop: an actor that generates and is trained, a frozen reference that prices drift, a frozen reward model that scores the finished text, and a value model that spreads that one score over the tokens. This module follows one iteration tensor by tensor (the per-token KL reward, GAE over response tokens, the minibatch loop, the pretraining-gradient mix), then reads the measured evidence on cost, on quality against DPO, and on how much of PPO's result is implementation detail. Afterwards you can draw the dataflow, list its tensors, and say what a critic-free method removes from it.

Course 50 min read 16 sources As of 2026-10-27 Prerequisites: Module 3, Module 4
Dataflow of one PPO-RLHF iteration: the actor generates responses; the reference, reward, value and actor models each emit a tensor; reward shaping and GAE turn them into advantages and value targets; the learner updates the actor and the value model prompt x Actor π_θ trained y: ids (B,T) Reference π_ref frozen SFT copy Reward model r_ψ frozen Value model V_φ trained, init = RM Actor (scoring) same θ, no grad log π_ref (B,T) r_RM at EOS (B) V_t (B,T) log π_old (B,T) Reward shaping −β log(π_old/π_ref) + r_RM on the EOS token GAE (γ, λ) δ = r + γV′ − V Â_t = Σ (γλ)^l δ_t+l emits Â_t (B,T) R̂_t = Â_t + V_t (B,T) PPO update: minibatches × epochs Actor: −min(ρÂ, clip(ρ, 1±ε)Â) + γ_ptx · (−log p_pretrain) Value: ½ (V_φ − R̂)² ρ = π_θ / π_old pretraining text D_pretrain new θ trained frozen B = sequences per batch, T = response tokens. The orange arrows are the only edges that change weights.
Figure 1. One PPO-RLHF iteration. The same actor weights generate, re-score and receive the update. The reference, reward and value models and the actor's own scoring pass each emit one tensor; reward shaping and GAE turn them into advantages and value targets; the learner updates the actor with the clipped loss (plus the pretraining term of PPO-ptx, section 6) and the value model with a regression loss. Structure from InstructGPT §3.51, Lambert Fig. 19 and §6.2.72, and DeepSeekMath Fig. 4 and Eq. 23.

1. The recipe and its environment

1.1 Three stages, three datasets

InstructGPT fine-tunes GPT-3 in three steps1: supervised fine-tuning on labeler demonstrations, a reward model trained on labeler rankings, and PPO against that reward model. The datasets differ in size: the SFT set has about 13k training prompts, the RM set 33k, and the PPO set 31k, the last drawn only from the API (the SFT and RM prompts mix API and labeler-written ones)1 (§3.2). About 40 contractors produced the labels1 (§1). The policies come in three sizes (1.3B, 6B, 175B), but one 6B reward model and one 6B value function serve all of them, so that policy size is the only variable1 (Appendix C.4). Models run in fp16 with fp32 master weights, a 2k-token context, prompts filtered to at most 1k tokens, and responses capped at 1k1 (Appendix C).

No single piece was new in 2022. Ziegler et al. fine-tuned a language model with PPO against a learned reward, with the KL penalty to the starting model written into the reward4 (§2.1). Stiennon et al. moved the recipe to summarization with human comparisons5, and InstructGPT says it follows them1 (§3.5). InstructGPT adds scale, real API prompts and the pretraining mix of section 6. Module 4 covered the reward model; one fact from it matters here. The Bradley-Terry loss is shift-invariant, so InstructGPT adds a bias to the RM so that labeler demonstrations score a mean of 0 before RL starts1 (§3.5). KL coefficients are therefore relative to a reward scale someone chose: Ivison et al. lower β from 0.05 to 0.0325 when they switch to 70B reward models, and note that reward models trained under different settings "can have very different output ranges"6 (Appendix F).

The three-stage InstructGPT recipe with dataset and model sizes, and the bandit environment that stage three presents to the policy 1 · SFT ~13k prompts labeler demonstrations policy sizes 1.3B · 6B · 175B 16 epochs, cosine LR 2 · Reward model 33k prompts K = 4 to 9 ranked responses each one 6B model, 1 epoch pairwise loss (Module 4) 3 · PPO 31k prompts, API only 256k episodes policy 1.3B · 6B · 175B value model 6B, init = RM What stage 3 sees: a bandit environment reset() random prompt x Policy π_θ samples response y y step(y) → reward r_RM(x,y) and KL terms reward; episode ends (done) one step per episode
Figure 2. Top: the three stages with the dataset and model sizes InstructGPT reports (§3.2, §3.5, Appendix C.1-C.4)1. Bottom: the environment stage 3 presents. An episode is one prompt and one response; the reward model's score arrives when the response ends, and the KL terms are added token by token (section 3).

1.2 The environment is a bandit

InstructGPT's description of stage 3 is literal1 (§3.5): "The environment is a bandit environment which presents a random customer prompt and expects a response to the prompt. Given the prompt and response, it produces a reward determined by the reward model and ends the episode." In the usual reset/step interface, reset draws a prompt, step receives the whole response and returns the reward with done set, and there is no second step.

Two things follow. First, the bandit label describes the environment, not the learner. PPO still treats every BPE token as a time step, as Stiennon et al. state5 (§3.1), and the only reward before the last token is the KL term of section 3. Ahmadian et al. argue that the learner should follow the environment and treat the completion as one action7 (§1); Module 2 (Section 1) derives when the two views give the same gradient, and section 9 reports their results. Second, the unit of progress is the episode, not the epoch. InstructGPT trains for 256k episodes over about 31k unique prompts1 (Appendix C.4), about 8.3 passes over the prompt set (derived). The Tulu 3 RLVR ablations make about 13 passes over the 7,473 GSM8K prompts8 (§6.2), and Huang et al. reshuffle and reuse the SFT prompts until 1M episodes are reached9 (Detail 21). Reuse is cheap here because each pass resamples the response, and the response is what carries the training signal.

The same loop with the reward model replaced by a checker is RLVR. Tulu 3 defines it as leveraging "the existing RLHF objective but replaces the reward model with a verification function"8 (§6), and its RLVR runs use PPO unchanged (section 8). So this module also describes how the first RLVR models were trained; Part 3 then removes the value model.

2. Four models: two trained, two frozen

2.1 What each model is for

Table 1 lists the four roles with InstructGPT's choices. The actor and the value model are trained; the reference and the reward model are only run forward. Initialization is the quiet design decision: the actor and the reference start as the same checkpoint, so at step 0 the KL term is zero, and the value model starts as the reward model, so that its representations already encode what the reward scores.

Table 1. The four models of PPO-RLHF. Sizes and initializations are InstructGPT's1 (Appendix C.2-C.4); the last row is Tulu 3's RLVR variant8 (§6.3, §6.4, §8.1).
ModelTrained in RL?Initialized fromSizeEmits per step
Actor π_θyesSFT model, 2 epochs with 10% pretraining data mixed in1.3B, 6B, 175Btoken ids; log π_old per token
Reference π_refnothe same SFT-plus-pretraining-mix checkpoint, used "to compute the KL reward"= actorlog π_ref per token
Reward model r_ψno6B GPT-3 fine-tuned on public NLP datasets; GPT-3 or SFT starts gave "similar results"6B (all policies)one scalar at the EOS position
Value model V_φyesthe reward model6B (all policies)V_t at every response token
Tulu 3 RLVR: no reward modelpolicy and valuevalue from a general RM trained on the Tulu 3 8B preference mixvalue 8B at 405B policyverifier outcome replaces the RM scalar

2.2 One iteration, stage by stage

Actor: generate y: token ids (B, P+T) Actor: scoring pass log π_old (B, T) Reference log π_ref (B, T) Value model V_t (B, T) Reward model r_RM (B) one scalar per response, read at the EOS position Reward shaping r_t = −β(log π_old − log π_ref) + r_RM · 1[t = EOS] (B, T) GAE (backward) Â_t and R̂_t = Â_t + V_t (B, T) Learner mb 1mb 2mb 3mb 4mb 5mb 6mb 7mb 8 new θ for the next rollout
  1. Generate. The actor samples one response per prompt for a batch of 512 prompts at temperature 1, each response capped at 1k tokens1 (Appendix C.4). This is autoregressive decoding, the serving workload. In Ivison et al.'s implementation it takes "more than 95%" of the training time6 (Appendix G).
  2. Score with fixed weights. Four forward passes over the finished sequences, no gradients: the actor records log π_old, the reference records log π_ref, the value model emits V_t at every response token, and the reward model emits one scalar at the EOS position. Each is a single parallel pass (prefill-shaped, in the writer's terms) and returns per-token floats, not logits (section 2.3).
  3. Shape the reward, run GAE. The reward at each token is the KL term; the last token also gets the reward model's score (section 3). A backward recursion over the tokens turns rewards and values into advantages Â_t and value targets R̂_t (section 4).
  4. Update in minibatches. InstructGPT splits the 512 sequences into 8 minibatches of 64 and makes a single pass1 (Appendix C.4). Each step applies the clipped loss to the actor and a squared-error loss to the value model, with the pretraining gradients of section 6 accumulated into the same step.
  5. Hand over the weights. The updated θ becomes the generator for the next rollout; the reference and the reward model never change. Moving θ into the sampling engine is the infrastructure problem of System Design Modules 36 and 37. Tulu 3 does it with an NCCL broadcast into vLLM8 (§8.1).

2.3 What moves between the models

Module 1 (Table 6, Section 7) counts what stays resident: 36 bytes per policy parameter of model state for this system, 252 GB at 7B and 2,520 GB at 70B with policy-sized models, and 18N FLOPs per sampled token (all derived there). The other half of the cost model is what crosses between the models in one iteration. Table 2 lists the tensors (writer's accounting) at InstructGPT-like shapes: 512 sequences, 1,024-token prompts and 1,024-token responses.

Table 2. Tensors exchanged in one PPO-RLHF iteration at B = 512, P = T = 1,024 (1 MB = 106 bytes; ids assumed int32, int64 would double that row). Derived by the writer from the dataflow of Figure 1; shapes and the reward and loss definitions are from InstructGPT Eq. 21 and Lambert §6.3.52.
TensorShapedtypeProduced byConsumed bySize (MB)
token ids(B, P+T)int32generatorall four models4.19
log π_old(B, T)fp32actorratio ρ, reward shaping2.10
log π_ref(B, T)fp32referencereward shaping2.10
V_t(B, T)fp32value modelGAE, value clip2.10
r_RM(B)fp32reward modelreward shaping0.002
r_t(B, T)fp32reward shapingGAE2.10
Â_t(B, T)fp32GAEactor loss2.10
R̂_t(B, T)fp32GAEvalue loss2.10
completion mask(B, T)boolgeneratorall losses0.52
Total17.30

The whole data plane is 17.30 MB per iteration. One bf16 copy of a 7B policy is 14 GB, about 810 times more (derived), and a 175B fp16 copy is 350 GB. The data that must move is tiny; the weights that must move are not. There is also a way to inflate the data plane by a factor of about 7,800: shipping logits instead of the log-probability of the sampled token. With a 128,000-token vocabulary (Llama 3's, §3.2)10, fp16 logits for 512 × 1,024 response positions are 134 GB (derived), more than the 7B weights. The practical consequence (the writer's inference) is that each scoring pass should gather the sampled token's log-probability inside the forward pass and return one float per token.

What moves between the models in one iteration

Token ids
Six fp32 per-token tensors
Mask and RM scores
All per-iteration data
One bf16 weight sync
Weights divided by data
Full fp16 logits instead (V = 128,000)

Formulas: ids \(4B(P+T)\) bytes; six fp32 tensors \(24BT\); mask \(BT\); RM scores \(4B\); weights \(2\Psi\) with \(\Psi\) in parameters; logits \(2BTV\). Derived by the writer; the vocabulary size is Llama 3's10. Activation memory and the generator's KV cache are not included.

3. The reward PPO sees: per-token KL plus one score at the end

3.1 The formula

Set the pretraining term aside until section 6. InstructGPT's RL objective (Eq. 2) is the reward model's score minus a KL-style penalty to the SFT model1. Here policy parameters are θ and the reward model's are ψ (the paper uses φ and θ):

$$\mathbb{E}_{(x,y)\sim\pi_\theta}\Big[\,r_\psi(x,y)\;-\;\beta\,\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{SFT}}(y\mid x)}\Big]$$

The text says the penalty is applied "at each token". DeepSeekMath's Eq. 23 and Ivison et al.'s Eq. 46 write the per-token reward that implementations feed to GAE, with \(T\) the position of the EOS token and \(\pi_{\mathrm{old}}\) the policy that generated the batch:

$$r_t \;=\; -\beta\,\log\frac{\pi_{\mathrm{old}}(y_t\mid x,y_{<t})}{\pi_{\mathrm{ref}}(y_t\mid x,y_{<t})}\;+\;r_\psi(x,y)\,\mathbb{1}[t=T]$$

Two consequences. The per-token terms sum to the sequence-level log-ratio, \(\sum_t \log(\pi_{\mathrm{old}}/\pi_{\mathrm{ref}})=\log(\pi_{\mathrm{old}}(y\mid x)/\pi_{\mathrm{ref}}(y\mid x))\), so Eq. 2 and the per-token reward have the same total return at \(\gamma=1\). What changes is when the penalty is paid, which is what a critic and GAE can use for credit assignment. And the sampled log-ratio is a one-sample estimate of the KL between the policy and the reference (the estimator Module 6 calls k1, with its variance and alternatives). A single term can be negative, a bonus for a token the policy has made less likely than the reference did. Only the expectation is non-negative.

3.2 A six-token example, by hand

Take β = 0.05, the value in Stiennon et al., Huang et al. and Ivison et al.596, and a response whose sixth token is EOS. The log-probabilities are invented: \(\log\pi_{\mathrm{old}}=(-0.10,-0.60,-0.05,-0.90,-0.20,-0.30)\) and \(\log\pi_{\mathrm{ref}}=(-0.30,-0.90,-0.15,-3.70,-0.45,-2.10)\). The reward model, normalized as in section 1, scores the response 1.20. The critic values are invented too: negative everywhere but the last position, which is what Huang et al. observe for a value model initialized from a reward model, because the reward logits at non-EOS positions are almost always negative9 (Details 13 and 22). With \(\gamma=1\), the value InstructGPT uses ("no discount")1, and the GAE recursion of Module 3 gives every column of Table 3; Figure 4 plots them.

Table 3. Reward and advantage for one six-token response. Illustrative: log-probabilities, the RM score (1.20) and the critic values \(V(s_t)\) are invented by the writer; every other column is derived from them with β = 0.05, γ = 1 and \(\delta_t=r_t+V(s_{t+1})-V(s_t)\), \(V(s_7)=0\). The λ = 1 column is \(G_t-V(s_t)\), the Monte Carlo return minus the critic.
Tokenℓt = log π_old − log π_ref−βℓtrtV(st)δt (= Â, λ = 0)Â, λ = 0.95Â, λ = 1
1 The0.20−0.0100−0.0100−0.500.04001.15121.4275
2 answer0.30−0.0150−0.0150−0.450.03501.16971.3875
3 is0.10−0.0050−0.0050−0.40−0.15501.19441.3525
4 422.80−0.1400−0.1400−0.550.06001.42041.5075
5 .0.25−0.0125−0.0125−0.351.13751.43201.4475
6 EOS1.80−0.09001.11000.800.31000.31000.3100
Sum5.45−0.27250.92751.4275
Left: per-token reward for the six-token example, small negative KL terms on tokens one to five and the reward-model score plus a KL term on the EOS token. Right: advantages at lambda 0, 0.95 and 1 Reward r_t (β = 0.05) 0.0 0.5 1.0 The answer is 42 . EOS −0.140 +1.20 −0.09 net 1.110 Advantage Â_t 0.0 0.5 1.0 1.5 2.0 1.15 The 1.17 answer 1.19 is 1.42 42 1.43 . 0.31 EOS λ = 0 (= δ_t) λ = 0.95 (labelled) λ = 1 (G_t − V_t)
Figure 4. The data of Table 3 (invented log-probabilities, RM score and critic values; arithmetic derived by the writer). Left: tokens 1 to 5 receive only a small negative KL term, and the EOS token receives the RM score (green) minus its own KL term (red), net 1.110. Right: advantages for three values of λ. Method: InstructGPT Eq. 21 and the GAE definition11 (§3).

Four things to read off. First, the KL terms are small and signed per token, but their sum is exactly the sequence-level penalty: \(-0.2725=-\beta\times 5.45\), 23% of the RM score here. The largest single term falls on token 4, where the policy makes a token \(e^{2.80}\approx 16.4\) times likelier than the SFT model did, followed by the EOS (about 6.05 times). Huang et al. find that fine-tuned models change their behavior mostly at the start and the end of a response9 (§7.2), which is where a real log-ratio profile would put its mass. Second, causality: \(G_t\) sums only later rewards, so token 4's penalty lowers the \(\lambda=1\) advantage of tokens 1 to 4 and not of tokens 5 and 6. A penalty cannot blame a token that came after it. At \(\lambda=0\), \(\delta_4=-0.140+(-0.35+0.55)=0.060\): the critic's rise from \(-0.55\) to \(-0.35\) outweighs the penalty. Third, the advantages are all positive at \(\lambda\ge0.95\), between 1.15 and 1.51 except the EOS token's 0.31, because the critic's negative values sit below the realized return of 0.9275. Every token looks better than expected. This offset is why implementations whiten the advantages, subtracting the batch mean and dividing by the batch standard deviation9 (Detail 25)8 (§6.2). Fourth, the value targets \(\hat R_t=\hat A_t+V_t\) at \(\lambda=0.95\) are \((0.65,0.72,0.79,0.87,1.08,1.11)\) (derived), so early updates pull the critic up by about 1.15 at token 1.

3.3 What the KL term is for, and how β is set

Stiennon et al. give the penalty two jobs5 (§3.1): it "acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode", and it keeps the policy from producing text "too different from those that the reward model has seen during training". Ziegler et al. also tune β with a controller, \(\beta_{t+1}=\beta_t(1+K_\beta e_t)\) with \(e_t=\mathrm{clip}\big((\mathrm{KL}_t-\mathrm{KL}_{\mathrm{target}})/\mathrm{KL}_{\mathrm{target}},-0.2,0.2\big)\) and \(K_\beta=0.1\), because seeds with the same β ended at "quite different values" of KL4 (§2.2). Ivison et al. report that most recent implementations have moved to a fixed β, and in their comparison of five open implementations only one keeps the adaptive controller6 (Appendix F, Table 10).

The coefficient is small and found by sweeping. InstructGPT uses β = 0.02, and its sweep gives poor Likert scores at both 0 and 2, with the optimum "around 0.01 and 0.02"1 (Appendix C.4, E.7). The KL term does not protect capabilities, though. When InstructGPT raised β to 2.0, 100 times the default (with the pretrained GPT-3 as the KL reference in that sweep), the regressions on public NLP datasets remained, which is what motivated the pretraining mix of section 61 (Appendix E.6). Tulu 3 sees the same limit from the verifiable side: more KL budget "does not necessarily" improve verifiable reward on GSM8K and MATH, and large KL lowers the average score across evaluations8 (§6.2.1).

The placement is a design choice. PPO puts the penalty in the reward, so it flows through the critic into every token's advantage. DeepSeekMath's GRPO instead adds the KL to the loss, "avoiding complicating the calculation" of the advantage3 (§4.1.1). Module 6 derives what each placement does to the gradient and compares KL coefficients across reports. The scale of the reward is the other lever, and recipes disagree: Stiennon et al. and InstructGPT normalize the reward model so demonstrations average 0, Huang et al. optionally whiten rewards within a batch and find that this shortens responses9 (Detail 24), and Ivison et al. use no reward normalization or whitening at all but whiten advantages6 (Appendix F).

4. The value model: initialization, GAE over tokens, and its cost

4.1 Initialize from the reward model, train as a separate network

The value model is a language model with a scalar head, like the reward model, and every PPO-RLHF recipe described here initializes it from the reward model: InstructGPT does1 (§3.5), Stiennon et al. do5 (§3.1), and so do Huang et al., Ivison et al. and Tulu 3. Huang et al. note, citing earlier work, that this warm start improves the initial gradients to the policy and reduces drift9 (Detail 22). The alternatives exist: Ivison et al. list implementations that initialize the value head from the SFT model or the base model, and in their comparison table only AlpacaFarm, among the other implementations, starts the value model from the reward model6 (Appendix F, Table 10). Tulu 3 tested it directly in its RLVR setting, starting the value model from a general reward model or from the anchored DPO checkpoint, and found that the reward-model start gave "the highest GSM8K test score and also higher average scores", from which it concludes that "the value function plays an important role in RLVR's training"8 (§6.2.1).

Stiennon et al. also settled whether the value network should share the policy's parameters. It should not: a separate Transformer "prevents updates to the value function from partially destroying the pretrained policy early in training", it outperformed a shared network in their ablation, and it can be initialized from the reward model; the price is memory5 (§3.1, Appendix G.1). Size is a free choice. InstructGPT paired 1.3B, 6B and 175B policies with one 6B value function, in part because a 175B reward model "could be unstable", made a poor value initialization, and would have greatly raised the compute of PPO1 (§3.5, Appendix C.2), and Tulu 3 paired a 405B policy with an 8B value model, about 50 times smaller (derived)8 (§8.1).

4.2 GAE over response tokens

Module 3 derives GAE, its closed form for a last-token reward, and a five-token hand computation; Table 3 above applies it to a reward vector with KL terms. The settings are consistent across recipes: \(\gamma=1\) in every recipe in Table 4 (InstructGPT: "No discount is applied when estimating the generalized advantage"1), and \(\lambda=0.95\) in Stiennon et al., Huang et al., Ivison et al. and Tulu 35968. PPO's MuJoCo defaults are \(\gamma=0.99\) and \(\lambda=0.95\)12 (Table 3), so the LLM recipes kept \(\lambda\) and dropped the discount. No report states why; discounting a finite text episode has no obvious purpose (the writer's reading).

What \(\lambda=0.95\) does depends on length, by the \(\lambda^{T-t}\) weight of Module 3 (Section 6.4). In Huang et al.'s TL;DR setup, responses are sampled to a fixed 53 tokens9 (Detail 23), so the first token receives \(0.95^{52}\approx0.07\) of the final reward directly (derived). In Tulu 3's RLVR runs, responses can reach 2,048 tokens8 (Table 21), and the same weight at a 1,024-token response is about \(10^{-23}\) (derived). The value of \(\lambda\) was set in the first regime and carried into the second unchanged. Nothing in the reports shows it being re-tuned, which means that at reasoning lengths the critic's own predictions supply almost all of the early tokens' advantage.

The critic is trained on \(\hat R_t=\hat A_t+V_t\), the \(\lambda\)-return, with a squared-error loss weighted by 0.1 in Huang et al., Ivison et al. and Tulu 3, and a value-clip range of 0.2 in the first two9 (Table 7)6 (Table 9)8 (Table 21). The value clip is a "code-level optimization" inherited from the MuJoCo implementation of PPO, and Engstrom et al. list it first among nine code-level optimizations whose effect, they find, accounts for most of PPO's gain in cumulative reward over TRPO13. Ahmadian et al. removed both the policy clip and the value clip in an RLHF setting and saw no meaningful change7 (§3.2).

4.3 Why the critic is the weak part

The last-token reward is the root of it. DeepSeekMath states the objection in two parts: the value function "is typically another model of comparable size as the policy model", a "substantial memory and computational burden", and because "usually only the last token is assigned a reward score", training a value function "that is accurate at each token" may be complicated3 (§4.1.1). Module 2 (Section 5) derives why the critic can only redistribute the sequence-level credit \(R-V(x)\) and not create it. What Huang et al.'s logs show is the starting point: the RM-initialized value network behaves like a per-token reward model that is negative until the EOS position, and in training "effectively acting as a per-token RM" for the end-of-episode return9 (Detail 22).

Kimi k1.5 adds an argument that is not about cost. With a long chain of thought, an oracle value function would mark a reasoning step that leads into an error as worse than a correct one, and standard credit assignment would penalize it, yet "exploring" such a step "is extremely valuable" if the model recovers and reaches the right answer. Kimi excluded the value network for this reason as well as for efficiency14 (§2.3.2). Table 5 of Module 3 shows the same effect at \(\lambda=0\), where a successful response still gives its fourth token a negative advantage. Part 3 picks up both threads: Module 9 for the critic-free estimators, Module 11 for the advantage without a critic.

5. The inner loop: minibatches, epochs and the clip

5.1 The settings, side by side

Table 4 collects the PPO settings that the primary sources report. Against the MuJoCo defaults of Module 3 (horizon 2,048, Adam at 3e-4, 10 epochs over minibatches of 64)12, three things change. The learning rate is 20 to 3,000 times smaller (3e-4 against 1.5e-5 down to 1e-7, derived), presumably because the policy starts from a trained SFT model. The number of passes over a batch falls from 10 to between 1 and 4. And the sampling temperature is a real exploration setting (1.0 or 0.7), because the policy's own sampling is the only source of exploration. A batch is 64 to 1,856 complete responses of up to 1k or 2k tokens, not a horizon of environment steps.

Table 4. PPO settings in the RLHF-era reports. "Minibatches × epochs" is the number of minibatches per rollout batch times the passes over it; n/s = not stated in the text read; Ivison's episode count is derived from their 60,908-example datasets. γ = 1 in all; λ = 0.95 where stated (InstructGPT states no λ). Sources: InstructGPT App. C.41; Stiennon App. B5; Huang Table 79; Ivison Tables 9 and 11, App. F6; Tulu 3 Table 21, §6.4 and §8.1, and for the 405B row the table labelled Table 35, whose caption describes the general-RM run while its column header reads 405B RLVR (the header is taken as the 405B RLVR configuration; its printed 300,000-episode budget was not run)8.
Recipe (policy)EpisodesBatch (responses)Minibatches × epochsβLearning rateTemp.
InstructGPT (1.3B, 6B, 175B)256k5128 × 10.02n/s (swept 2.55e-6 to 2.55e-5)1
Stiennon et al. (1.3B, 6.7B)1M512 (1.3B), 256 (6.7B)n/s × 40.051.5e-5 (1.3B), 7e-6 (6.7B)n/s
Huang et al. (Pythia 1B to 6.9B)1M5121 × 40.053e-60.7
Ivison et al. (Tulu 2 13B)about 61k (1 epoch)641 × 10.05 (0.0325 with 70B RMs)1e-60.7
Tulu 3 PPO against a general RM300k2241 × 10.05 to 0.01 swept3e-71.0
Tulu 3 RLVR (8B / 70B)100k / 400k224 / 6401 × 40.05 / 0.073e-7 / 1e-71.0
Tulu 3 RLVR (405B)300k printed; 75 steps run1,8561 × 10.051e-71.0

5.2 Where the clip can fire

The clipped loss only acts when the ratio \(\rho=\pi_\theta/\pi_{\mathrm{old}}\) differs from 1, and the ratio is exactly 1 at the first optimizer step on a batch if the old log-probabilities come from the same weights with dropout off. Tulu 3 and Huang et al. both disable dropout for this reason: if the two passes disagree, "all ratios could potentially be clipped, resulting in zero gradient"8 (§6.2)9 (Detail 7). After that first step the policy has moved, so only steps 2 to \(n\) can clip, where \(n\) is minibatches times epochs. Figure 5 counts them.

For each recipe, a row of squares for the optimizer steps taken on one rollout batch; the first square is grey because the importance ratio is exactly one, the rest are blue because the clip can bind Optimizer steps taken on one rollout batch (minibatches × epochs) InstructGPT (8 minibatches × 1 epoch) 7 of 8 steps can clip Ziegler et al. (1 × 4) 3 of 4 steps can clip Huang et al. reproduction (1 × 4) 3 of 4 steps can clip Tulu 3 RLVR, 8B and 70B (1 × 4) 3 of 4 steps can clip Tulu 3 PPO against a general RM (1 × 1) 0 of 1 steps can clip Tulu 3 405B RLVR (1 × 1) 0 of 1 steps can clip Ivison et al. (1 × 1) 0 of 1 steps can clip ρ = 1 at the first step: old log-probs come from the same weights, with dropout off ρ ≠ 1 is possible after earlier updates, so the clip can bind
Figure 5. Optimizer steps per rollout batch, derived as minibatches × epochs from Table 4 (InstructGPT 8 × 1; Ziegler et al. one minibatch for four epochs4 (§2.2); Huang et al. 1 × 4; Tulu 3 RLVR 1 × 4, with 1 × 1 for the RM runs and the 405B run). The ratio is 1 at the first step only when the old log-probabilities are recomputed with the same weights and dropout off. Three of the seven rows have no step at which the clip can bind.

Three of the seven rows, Tulu 3's RM runs and its 405B run and Ivison et al.'s, take one step per batch, so the clip never binds and the update is a policy gradient weighted by the GAE advantage. Lambert states this directly: with one gradient step per sample, \(\pi_\theta=\pi_{\theta_{\mathrm{old}}}\) and the update "reduces to" standard policy gradient2 (§6.3.5.1). He also notes that DAPO uses 16 gradient steps per batch while Tulu 3 uses 4 for 8B and 70B and drops to 1 at 405B "to maintain training stability"2 (§6.4.2). Where the clip can bind, it rarely does. Ahmadian et al. measured the loss as clipped "on average < 5% of the time per batch" across their datasets and models, and found that removing both the policy and value clips "does not impact learning meaningfully"7 (§3.2). The regularization that does the work in RLHF is then the small learning rate, the few steps and the KL term in the reward (the writer's reading, consistent with Lambert's §6.4.2). Module 3 (Section 5) shows why the clip is not a KL bound even when it does bind.

5.3 Staleness inside a batch

With several minibatches per rollout, later minibatches train on data from weights that have already moved, as in InstructGPT's 8 minibatches per batch. Ivison et al. hit the cost of this when they raised the prompt batch to 512 with a minibatch of 64: computing log-probabilities and values once per batch, as most implementations do, "severely destabilized training", and the value loss "did not converge fast enough to supply policy training with reliable advantage signals". Their fix was to run the forward passes per minibatch, so the log-probabilities and values come from the most current models, which they note deviates from most PPO implementations6 (Appendix G). The same report found that raising the prompt batch from 64 to 512 cut training time from 60 h to 12 h at a cost of 0.6 points of average score (section 9). A "single pass" over 8 minibatches is already off-policy by up to 7 optimizer steps; Modules 12 and 14 cover the lag between sampler and learner.

5.4 Sections 3 to 5 as tensor code

The functions below are the writer's, in the shape of Lambert's reference PPO code2 (§6.2.7, §6.3.5). Run on the inputs of Table 3 they return its reward, advantage and value-target columns exactly. Three details carry the weight. Every stage is masked, so a KL term on a padded position cannot leak into a return, a hazard Lambert's code comments flag. The value after the last token is zero. And PPO-ptx adds \(\gamma_{\mathrm{ptx}}\) times a language-modeling loss on a separate pretraining batch to this loss before the backward pass.

import torch

def shape_rewards(logp_old, logp_ref, rm_score, mask, beta):
    # logp_*: (B,T) log-prob of each sampled token; rm_score: (B,); mask: (B,T), 1 on response tokens
    r = -beta * (logp_old - logp_ref) * mask                  # per-token KL term, zero on padding
    eos = mask.sum(1).long() - 1                               # index of the last response token
    r[torch.arange(r.size(0)), eos] += rm_score                # the RM score lands on the EOS position
    return r

def gae(r, v, mask, gamma=1.0, lam=0.95):
    B, T = r.shape
    adv, run = torch.zeros_like(r), torch.zeros(B)
    for t in reversed(range(T)):
        v_next = v[:, t + 1] * mask[:, t + 1] if t + 1 < T else 0.0   # V = 0 after the last token
        delta = r[:, t] + gamma * v_next - v[:, t]
        run = (delta + gamma * lam * run) * mask[:, t]          # padding zeroes the recursion
        adv[:, t] = run
    return adv, adv + v * mask                                   # advantages, value targets R_hat

def ppo_loss(logp_new, logp_old, v_new, v_old, adv, ret, mask, eps=0.2, eps_v=0.2, c_v=0.1):
    a = adv[mask > 0]
    adv = (adv - a.mean()) / (a.std() + 1e-8)                    # whiten over the batch
    ratio = (logp_new - logp_old).exp()
    pg = torch.max(-adv * ratio, -adv * ratio.clamp(1 - eps, 1 + eps))
    v_clip = v_old + (v_new - v_old).clamp(-eps_v, eps_v)
    vf = 0.5 * torch.max((v_new - ret) ** 2, (v_clip - ret) ** 2)
    return (((pg + c_v * vf) * mask).sum(1) / mask.sum(1)).mean()   # sequence-mean aggregation

6. PPO-ptx: pretraining gradients and the alignment tax

6.1 The objective and what it costs

InstructGPT's Eq. 2 has a second term, \(\gamma_{\mathrm{ptx}}\,\mathbb{E}_{x\sim D_{\mathrm{pretrain}}}[\log\pi_\theta(x)]\): the ordinary language-modeling log-likelihood on pretraining text, added to the RL objective1 (§3.5). The paper calls the coefficient γ; this module writes \(\gamma_{\mathrm{ptx}}\) because γ is the discount. "For 'PPO' models, γ is set to 0", so plain PPO is the \(\gamma_{\mathrm{ptx}}=0\) special case, and "InstructGPT" in the paper means the PPO-ptx models. The mechanics are in Appendix C.4: the run uses 8 times more pretraining examples than RL episodes (2,048,000 for 256k episodes, derived), "for each minibatch, we compute the PPO gradients and pretraining gradients in consecutive steps and accumulate them both into the gradient buffers", and the pretraining gradient is multiplied by \(\gamma_{\mathrm{ptx}}=27.8\). One value of 27.8 "seems to work well across model sizes, from 1.3B to 175B"1 (Appendix E.6). On the 1.3B model, any \(\gamma_{\mathrm{ptx}}\ge20\) recovered the regressions.

The cost the paper reports is wall-clock; the pretraining batches run through the same actor. With a ratio of 8 the training time "doubles that of the corresponding experiment without using pretraining mix"; at a ratio of 4 the pretraining loss would often increase during training, and a ratio of 32 gave better Likert scores but "the training time also increases by a few fold"1 (Appendix E.11). It also needs a data path the other three models do not: a stream of pretraining text feeding the learner. The same appendix reports that the SFT initialization itself was trained with 10% pretraining data mixed in, "the only setting [that] stands out" among the 0%, 10% and 50% mixes tested (Appendix E.8).

6.2 What the mix buys, and what it does not

The tax is real. At 175B with few-shot prompting, plain PPO falls to 74.5% of GPT-3's SQuADv2 F1, 78.8% of its DROP F1 and 66.6% of its French-to-English BLEU, and PPO-ptx recovers most of it (Figure 6). The tax starts before RL: the SFT model is already at 87.8% to 94.5% of GPT-3 on three of the four, and PPO-ptx ends above SFT on three of four, so the mix repairs some SFT damage as well as RL damage.

Grouped bars of SFT, PPO and PPO-ptx scores at 175B few-shot as a percentage of GPT-3 175B on SQuADv2, DROP, HellaSwag and WMT15 French to English; PPO is lowest on three of four and PPO-ptx recovers most of the loss % of GPT-3 175B few-shot score (GPT-3 = 100) SFT PPO PPO-ptx 60 70 80 90 100 110 94.5 74.5 100.3 SQuADv2 (F1) 101.6 78.8 94.5 DROP (F1) 93.7 96.0 103.7 HellaSwag (acc.) 87.8 66.6 92.1 WMT15 fr-en (BLEU)
Figure 6. The alignment tax at 175B, few-shot, as a percentage of the GPT-3 175B score on the same task (dashed line = 100). Percentages are derived by the writer from InstructGPT's Table 14 (Appendix E): SQuADv2 F1 69.75 / 65.90 / 51.95 / 69.93, DROP F1 35.27 / 35.85 / 27.78 / 33.34, HellaSwag accuracy 0.791 / 0.741 / 0.759 / 0.820, fr-en BLEU 39.93 / 35.07 / 26.58 / 36.76 for GPT-3 / SFT / PPO / PPO-ptx1.

Three caveats from the same paper keep the claim honest. First, the repair is partial. In zero-shot SQuADv2, PPO-ptx reaches 59.85 F1 against GPT-3's 64.30, about 93% (derived from Table 14), and the paper says its PPO-ptx model "still lags behind GPT-3 on DROP, SQuADv2, and translation"1 (§4.2). Second, the KL term cannot do this job. Even at \(\beta=2.0\), regressions on DROP and SQuAD remained, which the authors read as showing that "pretraining data distribution is critical" (Appendix E.6). The writer's reading of why: the KL penalty is evaluated only on the prompts and responses being sampled, so it can leave the model's behavior on unrelated text unconstrained, while the pretraining term trains on that text directly. Third, the effect decays with training. Running the 1.3B model for 512k episodes instead of 256k brought DROP and SQuADv2 back below the GPT-3 baseline, after starting above it (Appendix E.6, Figure 35).

The mix has a second, less advertised effect on stability. Without it, "all runs with learning rate greater than 8.05e-6 diverged" on the 1.3B and 6B models, and PPO with the mix "appears to be less sensitive to change of the learning rate" (Appendix E.9)1. Later open recipes mostly omit the term: Ivison et al.'s PPO setup lists none, Xu et al. drop the auxiliary SFT loss "because of the limited amount of data"15 (§5), and Tulu 3's RLVR stage leaves other benchmarks nearly flat without it (section 8.2).

7. What InstructGPT reported, and the caveats

The headline is that labelers prefer the outputs of the 1.3B InstructGPT model to those of the 175B GPT-3, "despite having over 100x fewer parameters"1 (§1). The remaining numbers the paper leads with:

85 ± 3%175B InstructGPT preferred to 175B GPT-31
71 ± 4%preferred to few-shot 175B GPT-3
21% vs 41%closed-domain hallucination rate, InstructGPT vs GPT-3
60 vs 3,640petaflops/s-days: 175B PPO-ptx vs GPT-3 pretraining (1.6%, derived)

The paper is candid about the limits. The preference measure is human ratings on the API prompt distribution by about 40 contractors; held-out labelers prefer InstructGPT at about the same rate, but the authors call this "preliminary" and say more work is needed on broader groups and on inputs where humans disagree1 (§1). The 85% is against base GPT-3, which is not instruction-tuned. Against fine-tunes on the public T0 and FLAN collections the paper reports a winrate for InstructGPT of 73.4 ± 2% against its baseline, versus 26.8 ± 2% for T0 and 29.8 ± 2% for FLAN1 (§1). Toxicity falls by about 25% when the model is prompted to be respectful, but "InstructGPT does not significantly improve over GPT-3" on Winogender or CrowS-Pairs, the bias benchmarks. The model also "still makes simple mistakes". The alignment tax of section 6 is the capability cost that the headline does not show.

The compute comparison is the part that survives: 4.9 petaflops/s-days for the 175B SFT model and 60 for the 175B PPO-ptx model, against 3,640 for GPT-3, which the authors read as RLHF being "very effective at making language models more helpful to users, more so than a 100x model size increase"1 (§5). Post-training added about one-sixtieth of the pretraining compute. Module 1 puts RL at about 3.75% of pretraining compute for R1-Zero and above 10% for DeepSeek-V3.2, 2 to 6 times that share (derived), on recipes that had dropped the value model.

8. Tulu 3: PPO against a verifier

8.1 The recipe

Tulu 3 is the clearest public account of PPO as the engine of RLVR. The objective is the RLHF objective with the reward model replaced by a verification function \(v\): maximize \(\mathbb{E}[v(x,y)-\beta\,\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})]\), where \(v=\alpha\) if the answer is verified and 0 otherwise, with \(\alpha=10\) "based on pilot experiments" and not tuned further8 (§6, Eq. 7-8). The prompts are 7,473 GSM8K training questions, 7,500 MATH questions and 14,973 instructions with verifiable constraints, 29,946 in all (Table 22). RLVR runs after preference tuning, starting from the best DPO checkpoint. The adopted implementation details are the ones Huang et al. enumerate9: a value model initialized from a general reward model, dropout off, repeated passes over the prompts with shuffling (about 13 over GSM8K in the ablations), a −10 reward for responses without an EOS token, and advantage whitening8 (§6.2). The rest of the PPO configuration is in Table 4: γ = 1, λ = 0.95, ε = 0.2, temperature 1.0, gradient-norm clip 1.0.

8.2 What the ablations found

Five findings, each from the report's own sweeps8 (§6.2.1, Appendix B.4):

  • A reward-model start for the value function is best. It beat starting from the DPO checkpoint on GSM8K and on the average score (section 4.1).
  • KL budget does not buy verifiable reward. On GSM8K and MATH, "incurring more KL budget does not necessarily lead to improvements in verifiable rewards", and larger KL lowered the average score across evaluations: "overoptimization happens". The clearest failure is on constraint prompts at β = 0.01, where the policy satisfied the checker and dropped the content: for "the letter e should appear 14 times" it answered "e, e, e, …" fourteen times, and for "5 paragraphs" it printed bare "Paragraph 1 *** Paragraph 2 …" separators (Appendix B.4, Figure 28).
  • Do not add reward-model scores on top of the verifier. Verifier-only reward outperformed verifier plus RM score, which added noise to the average.
  • A stronger start learns with less drift. Starting from the weaker SFT checkpoint reached the same verifiable reward as the DPO checkpoint but at a larger KL for the same β, and the stronger start gave better test performance.
  • Selection matters. Checkpoints were evaluated every 100 steps (40 at 70B) and the final 8B model was the one with the best overall MATH and IFEval performance; some 8B runs reached 89.4% on GSM8K and 84.8% on IFEval but did worse elsewhere and lowered the average (§6.4).
Table 5. Tulu 3 RLVR runs and results. GPU-hours are derived as GPUs × wall-clock hours; the report notes that an earlier-than-final checkpoint was taken from each run, so the delivered models used fewer. Scores are DPO checkpoint → RLVR checkpoint from the report's Table 23; the 405B run is described only as MATH gaining "over 5 points" within 25 RLVR steps, of 75 run8 (§6.3, §6.4, §8.1).
RunGPUsWall-clock (h)GPU-hoursAverageGSM8KMATHIFEval
8B reward model (value init)8972n/an/an/an/a
8B RLVR86552064.4 → 64.884.3 → 87.642.0 → 43.781.1 → 82.4
70B RLVR48602,88075.9 → 76.093.5 → 93.562.3 → 63.082.6 → 83.2
405B RLVR2564611,776n/snot usedover +5 in 25 stepsnot used

Read with the average column in view, the result is modest and targeted. At 8B, GSM8K gains 3.3 points, MATH 1.7 and IFEval 1.3, while MMLU drops 0.5 (68.7 to 68.2), TruthfulQA 1.1 and the safety average 1.7 (87.2 to 85.5); the average moves 0.4. At 70B, GSM8K was already saturated at 93.5 and the average moves 0.18 (Table 23). The other benchmarks stay within about two points, without any pretraining mix, which supports Lambert's description of RL as something that "can be done surgically" when the prompt distribution is narrow2 (§3.1.6). It does not show a free lunch: the safety and knowledge numbers move the wrong way.

8.3 Infrastructure

The system holds three models, since the reward model is gone: policy and value are trained, and the reference only runs inference. ZeRO stage 3 fits them, and Ray places dedicated GPUs running vLLM with PagedAttention for generation, because "often inference time is a bottleneck in RLHF infrastructure". Training is asynchronous: inference runs alongside the learner, and the policy trains on the second-latest inference batch for reproducibility, at the price of stale data when the sampler outruns the learner8 (§6.3).

At 405B the split is 16 GPUs for vLLM at 16-way tensor parallelism and 240 for training, with an NCCL broadcast to move weights. Per iteration the report gives about 550 s of inference, about 25 s of weight transfer and about 1,500 s of training, an 8B value model standing in for a 405B critic, 75 steps run, and NCCL timeouts that needed manual intervention8 (§8.1). Module 1 (Section 5.2) turns those timings into a utilization estimate; System Design Modules 36 and 37 cover how the placement and weight transfer are engineered. The report's closing suggestion is "larger value models or alternate value model-free RL algorithms such as GRPO" (§8.1), which is where Part 3 begins.

9. PPO versus DPO: cost, quality, and implementation details

9.1 Cost

The cost side is not in dispute. In Tulu 3's controlled, untuned setup PPO took about 28 hours on two nodes and DPO about 4 hours on one8 (§5.4.1), 14 times the node-hours (derived; Module 1, Section 7.3). Llama 3's team "also explored on-policy algorithms such as PPO" and found "DPO required less compute for large-scale models"10 (§4.1.4). Ivison et al. give the reason: DPO needs no reward-model stage, no online decoding and no value model6 (§2.1). In their PPO, rollouts took "more than 95%" of training time; a prompt batch of 512 instead of 64 cut a 60-hour run to 12 hours for 0.6 points of average score (61.7 to 61.1), and four rollouts per prompt at batch 256 recovered 61.4 at 32 hours6 (Appendix G, Table 11).

9.2 Quality: a ledger, not a verdict

Table 6. Head-to-head PPO and DPO-family results in the primary sources. Setups differ, so numbers are comparable only within a row. Sources: Ivison et al. Table 26; Xu et al. Tables 3-815; Ahmadian et al. Table 17; Tulu 3 Table 18 (an early-checkpoint exploration) and §5.4.1 with Figure 16 (the controlled comparison, a separate experiment)8; Llama 3 §4.1.410.
StudySetting and metricPPODPO familyReading
Ivison et al.Tulu 2 13B, same preference data, 11-benchmark average (UltraFeedback)62.261.0PPO ahead on all 5 datasets, mean +0.7; truthfulness −2.5
Xu et al.CodeLlama 34B, APPS Introductory pass@5; CodeContest 10@1k test44.4%; 22.4%DPO-Iter 34.2%; DPO 0.0%, DPO-Iter 3.2%DPO-Iter below SFT (38.6%) on APPS even with test-case labels
Xu et al.Llama 2 7B, HH-RLHF, OpenAssistant reward0.718DPO 0.611; DPO-Iter 0.678GPT-4 judged PPO over DPO 42 to 30 (28 ties)
Ahmadian et al.GPT-4 win-rate (%): TL;DR, HH Pythia, HH Llama67.6, 29.2, 32.0DPO 66.6, 39.0, 61.9DPO ahead on both HH runs; RLOO (k = 4) 77.9, 43.7, 64.1
Tulu 3, Table 18early Tulu 3 SFT checkpoint, UltraFeedback; Tulu average (hyperparameter exploration)54.5, 55.5 (β = 0.0325, 0.05)DPO 55.2; length-normalized DPO 46.8 to 57.3 over 6 settings; SFT base 55.7PPO 0.2 and 1.2 below the SFT base; length-normalized DPO best at 57.3
Tulu 3, §5.4.1 and Figure 16later controlled, untuned comparison: DPO preference mix, RM trained on it; Tulu average"similar (though slightly lower)" than DPO (Figure 16 caption); values plotted onlyvalues plotted onlyPPO about 28 h on 2 nodes; DPO about 4 h on 1 node
Llama 38B to 405B product models"explored"chosen"performed better, especially on instruction following benchmarks like IFEval"; no numbers

Three rows (Xu et al. twice, Ivison et al.) have PPO ahead, one (Ahmadian et al.) has it far behind on two of three datasets, and Tulu 3 and Llama 3 judge it not worth its cost. The rows differ in the reward (test cases in Xu et al.'s code tasks, learned reward models elsewhere), in PPO tuning (Xu et al. and Ivison et al. tuned; Ahmadian et al. ran a plain PPO to argue it is over-built), and in DPO setup (Xu et al. show DPO degrading under distribution shift; Tulu 3's Table 18 shows length-normalized DPO spanning 10.5 points across six hyperparameter settings). One label covers all of these runs, which is the first reason to distrust a single "PPO versus DPO" number.

9.3 Implementation details move the result more than the algorithm label

Engstrom et al. found for MuJoCo that "code-level optimizations" account for most of PPO's gain in cumulative reward over TRPO13. RLHF shows the same pattern, and Figure 7 is its cleanest demonstration. Xu et al. hold the algorithm, data and reward fixed and add three details to a PPO with batch size 64. On APPS Introductory, the baseline PPO scores 18.0%, 20.6 points below its own SFT starting point (38.6%); advantage normalization alone brings it back to 38.1%, a larger batch to 42.3%, and an exponentially averaged reference model to 44.4%, 5.8 points above SFT (derived)15 (Table 3). On the dialogue task the same three changes move the reward only from 0.706 to 0.718. In this one study the details mattered most on the hard, sparse-reward tasks.

Two panels of bars showing SFT, baseline PPO at batch size 64, and PPO with three cumulative changes on APPS Introductory pass@5 and CodeContest pass@1k APPS Introductory, pass@5 (%) 0 10 20 30 40 50 38.6 SFT 18.0 PPO 38.1 +adv norm 42.3 +batch 44.4 +ref EMA CodeContest, pass@1k (%) 0 5 10 15 20 25 12.0 SFT 7.7 PPO 15.4 +adv norm 19.6 +batch 21.4 +ref EMA
Figure 7. Cumulative PPO ablation of Xu et al. (CodeLlama-34B). Each bar adds one change to the one before: the baseline uses batch size 64, then advantage normalization, a larger batch, and an exponential moving average of the reference model. The dashed line is the SFT score. Data: Table 3 of Xu et al.15.
Table 7. Further implementation details with evidence in the sources. Huang et al. enumerate 25 details for the TL;DR reproduction9; the rows pick those with a measured or stated effect.
DetailReported effectSource
Dropout offlog-probs reproducible; ρ = 1 at the first step; reliable KLHuang Detail 7; Tulu 3 §6.2
Separate value networkoutperformed a shared network; costs memoryStiennon App. G.1
Value init from the RMbest GSM8K and average in Tulu 3 RLVRTulu 3 §6.2.1
EOS penaltydefines a reward for truncated responses (−10 in Tulu 3 and Ivison et al.)Huang Detail 23; Tulu 3; Ivison
Reward whiteningshorter responses; similar quality at matched lengthHuang Detail 24, §7.1
Forward passes per minibatchavoided destabilization at batch 512, minibatch 64Ivison App. G
RM benchmark scoreRewardBench 61.0, 73.6, 73.9, 79.8 gave PPO averages 62.2, 62.8, 61.8, 61.6Ivison Table 3

The last row is the other half of the lesson. The reward models ranked 79.8, 73.9, 73.6 and 61.0 on RewardBench produced PPO policies scoring 61.6, 61.8, 62.8 and 62.2 on the 11-benchmark average, close to the reverse order, with the best RewardBench model giving the lowest average and only GSM8K moving clearly (53.0 to 58.0 with the 70B UltraFeedback RM)6 (Table 3). Tulu 3 draws the same caution about reward-model benchmarks8 (§5.4.1). A caution on one detail: Xu et al. attribute the reference EMA to InstructGPT, whose Appendix C.4 describes an exponential moving average of the weights (decay 0.992) and uses the SFT model for the KL term, without saying the reference itself was averaged1. Treat the reference EMA as Xu et al.'s own result.

Where experts disagree

Two questions are tangled in this module's evidence: whether online RL against a reward model beats offline preference optimization, and whether PPO's critic earns its place. The sources answer them differently depending on the reward, the budget and how carefully each side was tuned.

A. For preference data, a DPO-family method is enough

Llama 3 team10 (§1, §4.1.4); Tulu 38 (§5.4.1)

Llama 3 chose SFT, rejection sampling and DPO over "more complex reinforcement learning algorithms" that "tend to be less stable and harder to scale". Tulu 3 did preference iteration with DPO and reserved PPO "primarily for RLVR".

  • About 14 times fewer node-hours in Tulu 3's controlled test, and no reward-model stage, decode loop or value model.
  • Used at 405B in Llama 3.
  • Tulu 3 calls its PPO untuned and says more budget could raise it.
  • Xu et al. find DPO sensitive to distribution shift and failing on code even with test-case labels15.

B. Tuned online PPO wins when the reward is good

Xu et al.15; Ivison et al.6; Lambert2 (§3.1.6)

Xu et al. report PPO ahead on every testbed, including code competitions. Ivison et al. find PPO ahead by up to 2.5% in math and 1.2% in general domains. Lambert writes that "RL-based methods continue to win out", at much higher infrastructure cost.

  • 22.4% against 0.0% to 3.2% on CodeContest, where the reward is a test suite.
  • Online samples track the policy's own distribution.
  • Margins on learned-reward tasks are small next to preference-data quality (up to 8 points in Ivison et al.).
  • The wins depend on details a plain PPO lacks (Figure 7), and Ahmadian et al. find PPO well behind DPO on HH7.

C. The critic is dead weight

Ahmadian et al.7 (§1); DeepSeekMath3 (§4.1.1); Kimi k1.514 (§2.3.2)

The reward arrives for the whole completion, so model the completion as one action and baseline it with other samples. Ahmadian et al. report REINFORCE ahead of PPO by 3.2% to 20.3% win-rate and call PPO "unnecessarily complicated for a pre-trained LLM environment".

  • Drops a policy-sized model and GAE; RLOO at k = 4 scored 77.9 on TL;DR against PPO's 67.6.
  • Kimi's credit-assignment argument applies to long reasoning.
  • The PPO baselines are the proposers' own, run without the Xu et al. details.
  • RLOO pays in extra sampling per prompt, and Tulu 3 lists GRPO only as untested future work.

D. A well-built critic still pays

Tulu 38 (§6.2.1); Stiennon et al.5 (App. G.1); VAPO16 (Abstract, Table 1)

Tulu 3 concludes the value function "plays an important role" in RLVR, and VAPO reports 60.4 on AIME24 with a value-based recipe on Qwen 32B, against 47 for R1-Zero with GRPO and 50 for DAPO as it cites them.

  • Token-level credit and lower variance than a bare outcome baseline.
  • A separate value network beat one sharing the policy's parameters.
  • VAPO's own table gives 5 for vanilla PPO: the win needs value pretraining and length-adaptive λ.
  • It costs a second large model, and Tulu 3 used an 8B stand-in at 405B.
Assessment

On the first question, A is the default for preference tuning with learned reward models: the measured gap is at most a few points, smaller than the effect of data quality and sensitive to implementation details, while the cost gap is large. B is right when the reward is a trustworthy checker and the task is hard enough that offline pairs lose coverage, as in the code results; there the reward, not the algorithm label, is the variable. On the second question the evidence does not settle it. C's evidence compares against untuned PPO, and D's is a few recipes that needed substantial engineering. No source compares a tuned PPO (Xu et al.'s three details, a reward-model-initialized value network, a tuned λ) against RLOO or GRPO in one codebase at equal generated tokens, with a learned reward and with a verifier. That experiment would settle both. The reasoning-RL reports of Part 3 behave as if C were right for long chains of thought, which is evidence about practice, not about the optimum.

What is changing

PPO with a critic is now mostly a baseline. Lambert notes that REINFORCE-style methods without a value model drew interest for their simplicity and that GRPO is particularly popular for reasoning tasks2 (§6.1); DeepSeekMath introduced GRPO as a PPO variant that "foregoes the critic"3, and Tulu 3 closed by naming value-free methods as the next step8. What persists is the scaffolding: dropout off, EOS handling, advantage normalization, the placement of the KL term, and one gradient step per batch, which Lambert notes leaves PPO's own clip inactive (§6.4.2). The pretraining-gradient term is absent from the open recipes read here. Modules 9 to 12 cover the critic-free successors and Module 33 the choice between them. No primary source used here describes the closed labs' current practice.

References

  1. paperOuyang, L., Wu, J., Jiang, X., Almeida, D. et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: dataset sizes and the labeler team (§1, §3.2), the bandit environment, per-token KL, RM-initialized value function and the PPO-ptx objective (§3.5, Eq. 2), PPO hyperparameters and the 6B RM and value choice (Appendix C.2-C.4), the alignment-tax, KL-coefficient and learning-rate ablations (Appendix E.6-E.11, Table 14), headline results, limitations and compute (§1, §4, §5).
  2. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: the RLHF loop figure (Fig. 19, §6.1), PPO value-function and loss code (§6.2.7, §6.3.5), the one-gradient-step simplification (§6.3.5.1), update-iteration counts across recipes (§6.4.2), RL "surgical" and "continue to win out" (§3.1.6).
  3. paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the PPO objective and per-token KL reward (§4.1.1, Eq. 1-2), the case against the value model and KL placed in the loss (§4.1.1, Fig. 4).
  4. paperZiegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., Irving, G. Fine-Tuning Language Models from Human Preferences. arXiv 1909.08593 v2, 8 Jan 2020. https://arxiv.org/abs/1909.08593. Accessed 2026-10-27. Used for: the KL-penalized reward and its purposes (§2.1, Eq. 2), the adaptive KL controller and the PPO settings (§2.2).
  5. paperStiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P. Learning to summarize from human feedback. NeurIPS 2020; arXiv 2009.01325 v3, 15 Feb 2022. https://arxiv.org/pdf/2009.01325. Accessed 2026-10-27. Used for: the two purposes of the KL term, the token-level time step and the separate value network initialized from the reward model (§3.1), PPO settings (Appendix B), the shared-versus-separate value ablation (Appendix G.1).
  6. paperIvison, H., Wang, Y., Liu, J., Wu, Z. et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv v2, 7 Oct 2024. https://arxiv.org/abs/2406.09279. Accessed 2026-10-27. Used for: DPO versus PPO (§2.1, §3.2, Table 2), reward-model scale versus downstream PPO quality (§3.3, Table 3), PPO equations, implementation details and hyperparameters (Appendix F, Tables 9-10), the rollout share of training time and batch-size trade-offs (Appendix G, Table 11).
  7. paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: the single-action argument and the claim that PPO is over-built (§1), clipping rarely active and the effect of removing clips (§3.2), win-rates for PPO, REINFORCE, RLOO and DPO (Table 1).
  8. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. (Allen Institute for AI). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025 (v1, 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the early-checkpoint DPO and PPO exploration (Table 18) and, separately, the controlled PPO-versus-DPO comparison, Figure 16 and the 28 h and 4 h run times (§5.4.1), the RLVR objective, data, implementation details and ablations (§6.1-6.2, Tables 21-22), infrastructure and run costs (§6.3), final results (§6.4, Table 23), the 405B run (§8.1, and the 405B hyperparameter table labelled Table 35, whose caption describes the general-RM run while its column header reads 405B RLVR), IFEval over-optimization (Appendix B.4), the 70B KL coefficient discrepancy (§6.4 against Table 21 caption).
  9. paperHuang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., Tunstall, L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. arXiv 2403.17031 v1, 24 Mar 2024 (published at COLM 2024, per Tulu 3's bibliography). https://arxiv.org/abs/2403.17031. Accessed 2026-10-27. Used for: dropout off (Detail 7), RM-initialized value behaviour (Details 13, 22), prompt reuse (Detail 21), the EOS trick (Detail 23), reward and advantage whitening (Details 24-25), PPO settings (Table 7), observations on over-optimization and token shift (§7.1-7.2).
  10. paperGrattafiori, A., Dubey, A., Jauhri, A. et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: the choice of DPO over PPO (§1, §4.1.4) and the 128,000-token vocabulary (§3.2, Table 3).
  11. paperSchulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016; arXiv v6, 20 Oct 2018. https://arxiv.org/pdf/1506.02438v6. Accessed 2026-10-26. Used for: the GAE estimator and the TD residual (§3).
  12. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the MuJoCo reference hyperparameters (Appendix A, Table 3).
  13. paperEngstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., Madry, A. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. ICLR 2020; arXiv 2005.12729 v1, 25 May 2020. https://arxiv.org/abs/2005.12729. Accessed 2026-10-27. Used for: code-level optimizations and their share of PPO's gain over TRPO (abstract, §2 list, value-function clipping first).
  14. paperKimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv v4, 3 Jun 2025 (v1, 22 Jan 2025). https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: excluding the value network, and the credit-assignment argument against value functions for long chain of thought (§2.3.2).
  15. paperXu, S., Fu, W., Gao, J., Ye, W. et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-27. Used for: the three key PPO factors and the Table 3 ablation (§5), implementation notes (§5), benchmark results on HH-RLHF, SafeRLHF, APPS and CodeContest (§6, Tables 4-8), DPO under distribution shift (§4).
  16. paperByteDance Seed (Yue, Y. et al.). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv 2504.05118 v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: AIME24 avg@32 of the value-based recipe against vanilla PPO, R1-Zero GRPO and DAPO (Abstract, Table 1).