Reinforcement Learning for LLM Post-Training · Module 5
PPO-RLHF as a System: Policy, Reference, Reward and Value Models, Per-Token KL Reward, GAE, and the Pretraining Mix
InstructGPT-style RLHF runs four language models inside one loop: an actor that generates and is trained, a frozen reference that prices drift, a frozen reward model that scores the finished text, and a value model that spreads that one score over the tokens. This module follows one iteration tensor by tensor (the per-token KL reward, GAE over response tokens, the minibatch loop, the pretraining-gradient mix), then reads the measured evidence on cost, on quality against DPO, and on how much of PPO's result is implementation detail. Afterwards you can draw the dataflow, list its tensors, and say what a critic-free method removes from it.
1. The recipe and its environment
1.1 Three stages, three datasets
InstructGPT fine-tunes GPT-3 in three steps1: supervised fine-tuning on labeler demonstrations, a reward model trained on labeler rankings, and PPO against that reward model. The datasets differ in size: the SFT set has about 13k training prompts, the RM set 33k, and the PPO set 31k, the last drawn only from the API (the SFT and RM prompts mix API and labeler-written ones)1 (§3.2). About 40 contractors produced the labels1 (§1). The policies come in three sizes (1.3B, 6B, 175B), but one 6B reward model and one 6B value function serve all of them, so that policy size is the only variable1 (Appendix C.4). Models run in fp16 with fp32 master weights, a 2k-token context, prompts filtered to at most 1k tokens, and responses capped at 1k1 (Appendix C).
No single piece was new in 2022. Ziegler et al. fine-tuned a language model with PPO against a learned reward, with the KL penalty to the starting model written into the reward4 (§2.1). Stiennon et al. moved the recipe to summarization with human comparisons5, and InstructGPT says it follows them1 (§3.5). InstructGPT adds scale, real API prompts and the pretraining mix of section 6. Module 4 covered the reward model; one fact from it matters here. The Bradley-Terry loss is shift-invariant, so InstructGPT adds a bias to the RM so that labeler demonstrations score a mean of 0 before RL starts1 (§3.5). KL coefficients are therefore relative to a reward scale someone chose: Ivison et al. lower β from 0.05 to 0.0325 when they switch to 70B reward models, and note that reward models trained under different settings "can have very different output ranges"6 (Appendix F).
1.2 The environment is a bandit
InstructGPT's description of stage 3 is literal1 (§3.5): "The environment is a bandit environment which presents a random customer prompt and expects a response to the prompt. Given the prompt and response, it produces a reward determined by the reward model and ends the episode." In the usual reset/step interface, reset draws a prompt, step receives the whole response and returns the reward with done set, and there is no second step.
Two things follow. First, the bandit label describes the environment, not the learner. PPO still treats every BPE token as a time step, as Stiennon et al. state5 (§3.1), and the only reward before the last token is the KL term of section 3. Ahmadian et al. argue that the learner should follow the environment and treat the completion as one action7 (§1); Module 2 (Section 1) derives when the two views give the same gradient, and section 9 reports their results. Second, the unit of progress is the episode, not the epoch. InstructGPT trains for 256k episodes over about 31k unique prompts1 (Appendix C.4), about 8.3 passes over the prompt set (derived). The Tulu 3 RLVR ablations make about 13 passes over the 7,473 GSM8K prompts8 (§6.2), and Huang et al. reshuffle and reuse the SFT prompts until 1M episodes are reached9 (Detail 21). Reuse is cheap here because each pass resamples the response, and the response is what carries the training signal.
The same loop with the reward model replaced by a checker is RLVR. Tulu 3 defines it as leveraging "the existing RLHF objective but replaces the reward model with a verification function"8 (§6), and its RLVR runs use PPO unchanged (section 8). So this module also describes how the first RLVR models were trained; Part 3 then removes the value model.
2. Four models: two trained, two frozen
2.1 What each model is for
Table 1 lists the four roles with InstructGPT's choices. The actor and the value model are trained; the reference and the reward model are only run forward. Initialization is the quiet design decision: the actor and the reference start as the same checkpoint, so at step 0 the KL term is zero, and the value model starts as the reward model, so that its representations already encode what the reward scores.
| Model | Trained in RL? | Initialized from | Size | Emits per step |
|---|---|---|---|---|
| Actor π_θ | yes | SFT model, 2 epochs with 10% pretraining data mixed in | 1.3B, 6B, 175B | token ids; log π_old per token |
| Reference π_ref | no | the same SFT-plus-pretraining-mix checkpoint, used "to compute the KL reward" | = actor | log π_ref per token |
| Reward model r_ψ | no | 6B GPT-3 fine-tuned on public NLP datasets; GPT-3 or SFT starts gave "similar results" | 6B (all policies) | one scalar at the EOS position |
| Value model V_φ | yes | the reward model | 6B (all policies) | V_t at every response token |
| Tulu 3 RLVR: no reward model | policy and value | value from a general RM trained on the Tulu 3 8B preference mix | value 8B at 405B policy | verifier outcome replaces the RM scalar |
2.2 One iteration, stage by stage
- Generate. The actor samples one response per prompt for a batch of 512 prompts at temperature 1, each response capped at 1k tokens1 (Appendix C.4). This is autoregressive decoding, the serving workload. In Ivison et al.'s implementation it takes "more than 95%" of the training time6 (Appendix G).
- Score with fixed weights. Four forward passes over the finished sequences, no gradients: the actor records log π_old, the reference records log π_ref, the value model emits V_t at every response token, and the reward model emits one scalar at the EOS position. Each is a single parallel pass (prefill-shaped, in the writer's terms) and returns per-token floats, not logits (section 2.3).
- Shape the reward, run GAE. The reward at each token is the KL term; the last token also gets the reward model's score (section 3). A backward recursion over the tokens turns rewards and values into advantages Â_t and value targets R̂_t (section 4).
- Update in minibatches. InstructGPT splits the 512 sequences into 8 minibatches of 64 and makes a single pass1 (Appendix C.4). Each step applies the clipped loss to the actor and a squared-error loss to the value model, with the pretraining gradients of section 6 accumulated into the same step.
- Hand over the weights. The updated θ becomes the generator for the next rollout; the reference and the reward model never change. Moving θ into the sampling engine is the infrastructure problem of System Design Modules 36 and 37. Tulu 3 does it with an NCCL broadcast into vLLM8 (§8.1).
2.3 What moves between the models
Module 1 (Table 6, Section 7) counts what stays resident: 36 bytes per policy parameter of model state for this system, 252 GB at 7B and 2,520 GB at 70B with policy-sized models, and 18N FLOPs per sampled token (all derived there). The other half of the cost model is what crosses between the models in one iteration. Table 2 lists the tensors (writer's accounting) at InstructGPT-like shapes: 512 sequences, 1,024-token prompts and 1,024-token responses.
| Tensor | Shape | dtype | Produced by | Consumed by | Size (MB) |
|---|---|---|---|---|---|
| token ids | (B, P+T) | int32 | generator | all four models | 4.19 |
| log π_old | (B, T) | fp32 | actor | ratio ρ, reward shaping | 2.10 |
| log π_ref | (B, T) | fp32 | reference | reward shaping | 2.10 |
| V_t | (B, T) | fp32 | value model | GAE, value clip | 2.10 |
| r_RM | (B) | fp32 | reward model | reward shaping | 0.002 |
| r_t | (B, T) | fp32 | reward shaping | GAE | 2.10 |
| Â_t | (B, T) | fp32 | GAE | actor loss | 2.10 |
| R̂_t | (B, T) | fp32 | GAE | value loss | 2.10 |
| completion mask | (B, T) | bool | generator | all losses | 0.52 |
| Total | 17.30 |
The whole data plane is 17.30 MB per iteration. One bf16 copy of a 7B policy is 14 GB, about 810 times more (derived), and a 175B fp16 copy is 350 GB. The data that must move is tiny; the weights that must move are not. There is also a way to inflate the data plane by a factor of about 7,800: shipping logits instead of the log-probability of the sampled token. With a 128,000-token vocabulary (Llama 3's, §3.2)10, fp16 logits for 512 × 1,024 response positions are 134 GB (derived), more than the 7B weights. The practical consequence (the writer's inference) is that each scoring pass should gather the sampled token's log-probability inside the forward pass and return one float per token.
What moves between the models in one iteration
Formulas: ids \(4B(P+T)\) bytes; six fp32 tensors \(24BT\); mask \(BT\); RM scores \(4B\); weights \(2\Psi\) with \(\Psi\) in parameters; logits \(2BTV\). Derived by the writer; the vocabulary size is Llama 3's10. Activation memory and the generator's KV cache are not included.
3. The reward PPO sees: per-token KL plus one score at the end
3.1 The formula
Set the pretraining term aside until section 6. InstructGPT's RL objective (Eq. 2) is the reward model's score minus a KL-style penalty to the SFT model1. Here policy parameters are θ and the reward model's are ψ (the paper uses φ and θ):
$$\mathbb{E}_{(x,y)\sim\pi_\theta}\Big[\,r_\psi(x,y)\;-\;\beta\,\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{SFT}}(y\mid x)}\Big]$$The text says the penalty is applied "at each token". DeepSeekMath's Eq. 23 and Ivison et al.'s Eq. 46 write the per-token reward that implementations feed to GAE, with \(T\) the position of the EOS token and \(\pi_{\mathrm{old}}\) the policy that generated the batch:
$$r_t \;=\; -\beta\,\log\frac{\pi_{\mathrm{old}}(y_t\mid x,y_{<t})}{\pi_{\mathrm{ref}}(y_t\mid x,y_{<t})}\;+\;r_\psi(x,y)\,\mathbb{1}[t=T]$$Two consequences. The per-token terms sum to the sequence-level log-ratio, \(\sum_t \log(\pi_{\mathrm{old}}/\pi_{\mathrm{ref}})=\log(\pi_{\mathrm{old}}(y\mid x)/\pi_{\mathrm{ref}}(y\mid x))\), so Eq. 2 and the per-token reward have the same total return at \(\gamma=1\). What changes is when the penalty is paid, which is what a critic and GAE can use for credit assignment. And the sampled log-ratio is a one-sample estimate of the KL between the policy and the reference (the estimator Module 6 calls k1, with its variance and alternatives). A single term can be negative, a bonus for a token the policy has made less likely than the reference did. Only the expectation is non-negative.
3.2 A six-token example, by hand
Take β = 0.05, the value in Stiennon et al., Huang et al. and Ivison et al.596, and a response whose sixth token is EOS. The log-probabilities are invented: \(\log\pi_{\mathrm{old}}=(-0.10,-0.60,-0.05,-0.90,-0.20,-0.30)\) and \(\log\pi_{\mathrm{ref}}=(-0.30,-0.90,-0.15,-3.70,-0.45,-2.10)\). The reward model, normalized as in section 1, scores the response 1.20. The critic values are invented too: negative everywhere but the last position, which is what Huang et al. observe for a value model initialized from a reward model, because the reward logits at non-EOS positions are almost always negative9 (Details 13 and 22). With \(\gamma=1\), the value InstructGPT uses ("no discount")1, and the GAE recursion of Module 3 gives every column of Table 3; Figure 4 plots them.
| Token | ℓt = log π_old − log π_ref | −βℓt | rt | V(st) | δt (= Â, λ = 0) | Â, λ = 0.95 | Â, λ = 1 |
|---|---|---|---|---|---|---|---|
| 1 The | 0.20 | −0.0100 | −0.0100 | −0.50 | 0.0400 | 1.1512 | 1.4275 |
| 2 answer | 0.30 | −0.0150 | −0.0150 | −0.45 | 0.0350 | 1.1697 | 1.3875 |
| 3 is | 0.10 | −0.0050 | −0.0050 | −0.40 | −0.1550 | 1.1944 | 1.3525 |
| 4 42 | 2.80 | −0.1400 | −0.1400 | −0.55 | 0.0600 | 1.4204 | 1.5075 |
| 5 . | 0.25 | −0.0125 | −0.0125 | −0.35 | 1.1375 | 1.4320 | 1.4475 |
| 6 EOS | 1.80 | −0.0900 | 1.1100 | 0.80 | 0.3100 | 0.3100 | 0.3100 |
| Sum | 5.45 | −0.2725 | 0.9275 | 1.4275 |
Four things to read off. First, the KL terms are small and signed per token, but their sum is exactly the sequence-level penalty: \(-0.2725=-\beta\times 5.45\), 23% of the RM score here. The largest single term falls on token 4, where the policy makes a token \(e^{2.80}\approx 16.4\) times likelier than the SFT model did, followed by the EOS (about 6.05 times). Huang et al. find that fine-tuned models change their behavior mostly at the start and the end of a response9 (§7.2), which is where a real log-ratio profile would put its mass. Second, causality: \(G_t\) sums only later rewards, so token 4's penalty lowers the \(\lambda=1\) advantage of tokens 1 to 4 and not of tokens 5 and 6. A penalty cannot blame a token that came after it. At \(\lambda=0\), \(\delta_4=-0.140+(-0.35+0.55)=0.060\): the critic's rise from \(-0.55\) to \(-0.35\) outweighs the penalty. Third, the advantages are all positive at \(\lambda\ge0.95\), between 1.15 and 1.51 except the EOS token's 0.31, because the critic's negative values sit below the realized return of 0.9275. Every token looks better than expected. This offset is why implementations whiten the advantages, subtracting the batch mean and dividing by the batch standard deviation9 (Detail 25)8 (§6.2). Fourth, the value targets \(\hat R_t=\hat A_t+V_t\) at \(\lambda=0.95\) are \((0.65,0.72,0.79,0.87,1.08,1.11)\) (derived), so early updates pull the critic up by about 1.15 at token 1.
3.3 What the KL term is for, and how β is set
Stiennon et al. give the penalty two jobs5 (§3.1): it "acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode", and it keeps the policy from producing text "too different from those that the reward model has seen during training". Ziegler et al. also tune β with a controller, \(\beta_{t+1}=\beta_t(1+K_\beta e_t)\) with \(e_t=\mathrm{clip}\big((\mathrm{KL}_t-\mathrm{KL}_{\mathrm{target}})/\mathrm{KL}_{\mathrm{target}},-0.2,0.2\big)\) and \(K_\beta=0.1\), because seeds with the same β ended at "quite different values" of KL4 (§2.2). Ivison et al. report that most recent implementations have moved to a fixed β, and in their comparison of five open implementations only one keeps the adaptive controller6 (Appendix F, Table 10).
The coefficient is small and found by sweeping. InstructGPT uses β = 0.02, and its sweep gives poor Likert scores at both 0 and 2, with the optimum "around 0.01 and 0.02"1 (Appendix C.4, E.7). The KL term does not protect capabilities, though. When InstructGPT raised β to 2.0, 100 times the default (with the pretrained GPT-3 as the KL reference in that sweep), the regressions on public NLP datasets remained, which is what motivated the pretraining mix of section 61 (Appendix E.6). Tulu 3 sees the same limit from the verifiable side: more KL budget "does not necessarily" improve verifiable reward on GSM8K and MATH, and large KL lowers the average score across evaluations8 (§6.2.1).
The placement is a design choice. PPO puts the penalty in the reward, so it flows through the critic into every token's advantage. DeepSeekMath's GRPO instead adds the KL to the loss, "avoiding complicating the calculation" of the advantage3 (§4.1.1). Module 6 derives what each placement does to the gradient and compares KL coefficients across reports. The scale of the reward is the other lever, and recipes disagree: Stiennon et al. and InstructGPT normalize the reward model so demonstrations average 0, Huang et al. optionally whiten rewards within a batch and find that this shortens responses9 (Detail 24), and Ivison et al. use no reward normalization or whitening at all but whiten advantages6 (Appendix F).
4. The value model: initialization, GAE over tokens, and its cost
4.1 Initialize from the reward model, train as a separate network
The value model is a language model with a scalar head, like the reward model, and every PPO-RLHF recipe described here initializes it from the reward model: InstructGPT does1 (§3.5), Stiennon et al. do5 (§3.1), and so do Huang et al., Ivison et al. and Tulu 3. Huang et al. note, citing earlier work, that this warm start improves the initial gradients to the policy and reduces drift9 (Detail 22). The alternatives exist: Ivison et al. list implementations that initialize the value head from the SFT model or the base model, and in their comparison table only AlpacaFarm, among the other implementations, starts the value model from the reward model6 (Appendix F, Table 10). Tulu 3 tested it directly in its RLVR setting, starting the value model from a general reward model or from the anchored DPO checkpoint, and found that the reward-model start gave "the highest GSM8K test score and also higher average scores", from which it concludes that "the value function plays an important role in RLVR's training"8 (§6.2.1).
Stiennon et al. also settled whether the value network should share the policy's parameters. It should not: a separate Transformer "prevents updates to the value function from partially destroying the pretrained policy early in training", it outperformed a shared network in their ablation, and it can be initialized from the reward model; the price is memory5 (§3.1, Appendix G.1). Size is a free choice. InstructGPT paired 1.3B, 6B and 175B policies with one 6B value function, in part because a 175B reward model "could be unstable", made a poor value initialization, and would have greatly raised the compute of PPO1 (§3.5, Appendix C.2), and Tulu 3 paired a 405B policy with an 8B value model, about 50 times smaller (derived)8 (§8.1).
4.2 GAE over response tokens
Module 3 derives GAE, its closed form for a last-token reward, and a five-token hand computation; Table 3 above applies it to a reward vector with KL terms. The settings are consistent across recipes: \(\gamma=1\) in every recipe in Table 4 (InstructGPT: "No discount is applied when estimating the generalized advantage"1), and \(\lambda=0.95\) in Stiennon et al., Huang et al., Ivison et al. and Tulu 35968. PPO's MuJoCo defaults are \(\gamma=0.99\) and \(\lambda=0.95\)12 (Table 3), so the LLM recipes kept \(\lambda\) and dropped the discount. No report states why; discounting a finite text episode has no obvious purpose (the writer's reading).
What \(\lambda=0.95\) does depends on length, by the \(\lambda^{T-t}\) weight of Module 3 (Section 6.4). In Huang et al.'s TL;DR setup, responses are sampled to a fixed 53 tokens9 (Detail 23), so the first token receives \(0.95^{52}\approx0.07\) of the final reward directly (derived). In Tulu 3's RLVR runs, responses can reach 2,048 tokens8 (Table 21), and the same weight at a 1,024-token response is about \(10^{-23}\) (derived). The value of \(\lambda\) was set in the first regime and carried into the second unchanged. Nothing in the reports shows it being re-tuned, which means that at reasoning lengths the critic's own predictions supply almost all of the early tokens' advantage.
The critic is trained on \(\hat R_t=\hat A_t+V_t\), the \(\lambda\)-return, with a squared-error loss weighted by 0.1 in Huang et al., Ivison et al. and Tulu 3, and a value-clip range of 0.2 in the first two9 (Table 7)6 (Table 9)8 (Table 21). The value clip is a "code-level optimization" inherited from the MuJoCo implementation of PPO, and Engstrom et al. list it first among nine code-level optimizations whose effect, they find, accounts for most of PPO's gain in cumulative reward over TRPO13. Ahmadian et al. removed both the policy clip and the value clip in an RLHF setting and saw no meaningful change7 (§3.2).
4.3 Why the critic is the weak part
The last-token reward is the root of it. DeepSeekMath states the objection in two parts: the value function "is typically another model of comparable size as the policy model", a "substantial memory and computational burden", and because "usually only the last token is assigned a reward score", training a value function "that is accurate at each token" may be complicated3 (§4.1.1). Module 2 (Section 5) derives why the critic can only redistribute the sequence-level credit \(R-V(x)\) and not create it. What Huang et al.'s logs show is the starting point: the RM-initialized value network behaves like a per-token reward model that is negative until the EOS position, and in training "effectively acting as a per-token RM" for the end-of-episode return9 (Detail 22).
Kimi k1.5 adds an argument that is not about cost. With a long chain of thought, an oracle value function would mark a reasoning step that leads into an error as worse than a correct one, and standard credit assignment would penalize it, yet "exploring" such a step "is extremely valuable" if the model recovers and reaches the right answer. Kimi excluded the value network for this reason as well as for efficiency14 (§2.3.2). Table 5 of Module 3 shows the same effect at \(\lambda=0\), where a successful response still gives its fourth token a negative advantage. Part 3 picks up both threads: Module 9 for the critic-free estimators, Module 11 for the advantage without a critic.
5. The inner loop: minibatches, epochs and the clip
5.1 The settings, side by side
Table 4 collects the PPO settings that the primary sources report. Against the MuJoCo defaults of Module 3 (horizon 2,048, Adam at 3e-4, 10 epochs over minibatches of 64)12, three things change. The learning rate is 20 to 3,000 times smaller (3e-4 against 1.5e-5 down to 1e-7, derived), presumably because the policy starts from a trained SFT model. The number of passes over a batch falls from 10 to between 1 and 4. And the sampling temperature is a real exploration setting (1.0 or 0.7), because the policy's own sampling is the only source of exploration. A batch is 64 to 1,856 complete responses of up to 1k or 2k tokens, not a horizon of environment steps.
| Recipe (policy) | Episodes | Batch (responses) | Minibatches × epochs | β | Learning rate | Temp. |
|---|---|---|---|---|---|---|
| InstructGPT (1.3B, 6B, 175B) | 256k | 512 | 8 × 1 | 0.02 | n/s (swept 2.55e-6 to 2.55e-5) | 1 |
| Stiennon et al. (1.3B, 6.7B) | 1M | 512 (1.3B), 256 (6.7B) | n/s × 4 | 0.05 | 1.5e-5 (1.3B), 7e-6 (6.7B) | n/s |
| Huang et al. (Pythia 1B to 6.9B) | 1M | 512 | 1 × 4 | 0.05 | 3e-6 | 0.7 |
| Ivison et al. (Tulu 2 13B) | about 61k (1 epoch) | 64 | 1 × 1 | 0.05 (0.0325 with 70B RMs) | 1e-6 | 0.7 |
| Tulu 3 PPO against a general RM | 300k | 224 | 1 × 1 | 0.05 to 0.01 swept | 3e-7 | 1.0 |
| Tulu 3 RLVR (8B / 70B) | 100k / 400k | 224 / 640 | 1 × 4 | 0.05 / 0.07 | 3e-7 / 1e-7 | 1.0 |
| Tulu 3 RLVR (405B) | 300k printed; 75 steps run | 1,856 | 1 × 1 | 0.05 | 1e-7 | 1.0 |
5.2 Where the clip can fire
The clipped loss only acts when the ratio \(\rho=\pi_\theta/\pi_{\mathrm{old}}\) differs from 1, and the ratio is exactly 1 at the first optimizer step on a batch if the old log-probabilities come from the same weights with dropout off. Tulu 3 and Huang et al. both disable dropout for this reason: if the two passes disagree, "all ratios could potentially be clipped, resulting in zero gradient"8 (§6.2)9 (Detail 7). After that first step the policy has moved, so only steps 2 to \(n\) can clip, where \(n\) is minibatches times epochs. Figure 5 counts them.
Three of the seven rows, Tulu 3's RM runs and its 405B run and Ivison et al.'s, take one step per batch, so the clip never binds and the update is a policy gradient weighted by the GAE advantage. Lambert states this directly: with one gradient step per sample, \(\pi_\theta=\pi_{\theta_{\mathrm{old}}}\) and the update "reduces to" standard policy gradient2 (§6.3.5.1). He also notes that DAPO uses 16 gradient steps per batch while Tulu 3 uses 4 for 8B and 70B and drops to 1 at 405B "to maintain training stability"2 (§6.4.2). Where the clip can bind, it rarely does. Ahmadian et al. measured the loss as clipped "on average < 5% of the time per batch" across their datasets and models, and found that removing both the policy and value clips "does not impact learning meaningfully"7 (§3.2). The regularization that does the work in RLHF is then the small learning rate, the few steps and the KL term in the reward (the writer's reading, consistent with Lambert's §6.4.2). Module 3 (Section 5) shows why the clip is not a KL bound even when it does bind.
5.3 Staleness inside a batch
With several minibatches per rollout, later minibatches train on data from weights that have already moved, as in InstructGPT's 8 minibatches per batch. Ivison et al. hit the cost of this when they raised the prompt batch to 512 with a minibatch of 64: computing log-probabilities and values once per batch, as most implementations do, "severely destabilized training", and the value loss "did not converge fast enough to supply policy training with reliable advantage signals". Their fix was to run the forward passes per minibatch, so the log-probabilities and values come from the most current models, which they note deviates from most PPO implementations6 (Appendix G). The same report found that raising the prompt batch from 64 to 512 cut training time from 60 h to 12 h at a cost of 0.6 points of average score (section 9). A "single pass" over 8 minibatches is already off-policy by up to 7 optimizer steps; Modules 12 and 14 cover the lag between sampler and learner.
5.4 Sections 3 to 5 as tensor code
The functions below are the writer's, in the shape of Lambert's reference PPO code2 (§6.2.7, §6.3.5). Run on the inputs of Table 3 they return its reward, advantage and value-target columns exactly. Three details carry the weight. Every stage is masked, so a KL term on a padded position cannot leak into a return, a hazard Lambert's code comments flag. The value after the last token is zero. And PPO-ptx adds \(\gamma_{\mathrm{ptx}}\) times a language-modeling loss on a separate pretraining batch to this loss before the backward pass.
import torch
def shape_rewards(logp_old, logp_ref, rm_score, mask, beta):
# logp_*: (B,T) log-prob of each sampled token; rm_score: (B,); mask: (B,T), 1 on response tokens
r = -beta * (logp_old - logp_ref) * mask # per-token KL term, zero on padding
eos = mask.sum(1).long() - 1 # index of the last response token
r[torch.arange(r.size(0)), eos] += rm_score # the RM score lands on the EOS position
return r
def gae(r, v, mask, gamma=1.0, lam=0.95):
B, T = r.shape
adv, run = torch.zeros_like(r), torch.zeros(B)
for t in reversed(range(T)):
v_next = v[:, t + 1] * mask[:, t + 1] if t + 1 < T else 0.0 # V = 0 after the last token
delta = r[:, t] + gamma * v_next - v[:, t]
run = (delta + gamma * lam * run) * mask[:, t] # padding zeroes the recursion
adv[:, t] = run
return adv, adv + v * mask # advantages, value targets R_hat
def ppo_loss(logp_new, logp_old, v_new, v_old, adv, ret, mask, eps=0.2, eps_v=0.2, c_v=0.1):
a = adv[mask > 0]
adv = (adv - a.mean()) / (a.std() + 1e-8) # whiten over the batch
ratio = (logp_new - logp_old).exp()
pg = torch.max(-adv * ratio, -adv * ratio.clamp(1 - eps, 1 + eps))
v_clip = v_old + (v_new - v_old).clamp(-eps_v, eps_v)
vf = 0.5 * torch.max((v_new - ret) ** 2, (v_clip - ret) ** 2)
return (((pg + c_v * vf) * mask).sum(1) / mask.sum(1)).mean() # sequence-mean aggregation
6. PPO-ptx: pretraining gradients and the alignment tax
6.1 The objective and what it costs
InstructGPT's Eq. 2 has a second term, \(\gamma_{\mathrm{ptx}}\,\mathbb{E}_{x\sim D_{\mathrm{pretrain}}}[\log\pi_\theta(x)]\): the ordinary language-modeling log-likelihood on pretraining text, added to the RL objective1 (§3.5). The paper calls the coefficient γ; this module writes \(\gamma_{\mathrm{ptx}}\) because γ is the discount. "For 'PPO' models, γ is set to 0", so plain PPO is the \(\gamma_{\mathrm{ptx}}=0\) special case, and "InstructGPT" in the paper means the PPO-ptx models. The mechanics are in Appendix C.4: the run uses 8 times more pretraining examples than RL episodes (2,048,000 for 256k episodes, derived), "for each minibatch, we compute the PPO gradients and pretraining gradients in consecutive steps and accumulate them both into the gradient buffers", and the pretraining gradient is multiplied by \(\gamma_{\mathrm{ptx}}=27.8\). One value of 27.8 "seems to work well across model sizes, from 1.3B to 175B"1 (Appendix E.6). On the 1.3B model, any \(\gamma_{\mathrm{ptx}}\ge20\) recovered the regressions.
The cost the paper reports is wall-clock; the pretraining batches run through the same actor. With a ratio of 8 the training time "doubles that of the corresponding experiment without using pretraining mix"; at a ratio of 4 the pretraining loss would often increase during training, and a ratio of 32 gave better Likert scores but "the training time also increases by a few fold"1 (Appendix E.11). It also needs a data path the other three models do not: a stream of pretraining text feeding the learner. The same appendix reports that the SFT initialization itself was trained with 10% pretraining data mixed in, "the only setting [that] stands out" among the 0%, 10% and 50% mixes tested (Appendix E.8).
6.2 What the mix buys, and what it does not
The tax is real. At 175B with few-shot prompting, plain PPO falls to 74.5% of GPT-3's SQuADv2 F1, 78.8% of its DROP F1 and 66.6% of its French-to-English BLEU, and PPO-ptx recovers most of it (Figure 6). The tax starts before RL: the SFT model is already at 87.8% to 94.5% of GPT-3 on three of the four, and PPO-ptx ends above SFT on three of four, so the mix repairs some SFT damage as well as RL damage.
Three caveats from the same paper keep the claim honest. First, the repair is partial. In zero-shot SQuADv2, PPO-ptx reaches 59.85 F1 against GPT-3's 64.30, about 93% (derived from Table 14), and the paper says its PPO-ptx model "still lags behind GPT-3 on DROP, SQuADv2, and translation"1 (§4.2). Second, the KL term cannot do this job. Even at \(\beta=2.0\), regressions on DROP and SQuAD remained, which the authors read as showing that "pretraining data distribution is critical" (Appendix E.6). The writer's reading of why: the KL penalty is evaluated only on the prompts and responses being sampled, so it can leave the model's behavior on unrelated text unconstrained, while the pretraining term trains on that text directly. Third, the effect decays with training. Running the 1.3B model for 512k episodes instead of 256k brought DROP and SQuADv2 back below the GPT-3 baseline, after starting above it (Appendix E.6, Figure 35).
The mix has a second, less advertised effect on stability. Without it, "all runs with learning rate greater than 8.05e-6 diverged" on the 1.3B and 6B models, and PPO with the mix "appears to be less sensitive to change of the learning rate" (Appendix E.9)1. Later open recipes mostly omit the term: Ivison et al.'s PPO setup lists none, Xu et al. drop the auxiliary SFT loss "because of the limited amount of data"15 (§5), and Tulu 3's RLVR stage leaves other benchmarks nearly flat without it (section 8.2).
7. What InstructGPT reported, and the caveats
The headline is that labelers prefer the outputs of the 1.3B InstructGPT model to those of the 175B GPT-3, "despite having over 100x fewer parameters"1 (§1). The remaining numbers the paper leads with:
The paper is candid about the limits. The preference measure is human ratings on the API prompt distribution by about 40 contractors; held-out labelers prefer InstructGPT at about the same rate, but the authors call this "preliminary" and say more work is needed on broader groups and on inputs where humans disagree1 (§1). The 85% is against base GPT-3, which is not instruction-tuned. Against fine-tunes on the public T0 and FLAN collections the paper reports a winrate for InstructGPT of 73.4 ± 2% against its baseline, versus 26.8 ± 2% for T0 and 29.8 ± 2% for FLAN1 (§1). Toxicity falls by about 25% when the model is prompted to be respectful, but "InstructGPT does not significantly improve over GPT-3" on Winogender or CrowS-Pairs, the bias benchmarks. The model also "still makes simple mistakes". The alignment tax of section 6 is the capability cost that the headline does not show.
The compute comparison is the part that survives: 4.9 petaflops/s-days for the 175B SFT model and 60 for the 175B PPO-ptx model, against 3,640 for GPT-3, which the authors read as RLHF being "very effective at making language models more helpful to users, more so than a 100x model size increase"1 (§5). Post-training added about one-sixtieth of the pretraining compute. Module 1 puts RL at about 3.75% of pretraining compute for R1-Zero and above 10% for DeepSeek-V3.2, 2 to 6 times that share (derived), on recipes that had dropped the value model.
8. Tulu 3: PPO against a verifier
8.1 The recipe
Tulu 3 is the clearest public account of PPO as the engine of RLVR. The objective is the RLHF objective with the reward model replaced by a verification function \(v\): maximize \(\mathbb{E}[v(x,y)-\beta\,\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})]\), where \(v=\alpha\) if the answer is verified and 0 otherwise, with \(\alpha=10\) "based on pilot experiments" and not tuned further8 (§6, Eq. 7-8). The prompts are 7,473 GSM8K training questions, 7,500 MATH questions and 14,973 instructions with verifiable constraints, 29,946 in all (Table 22). RLVR runs after preference tuning, starting from the best DPO checkpoint. The adopted implementation details are the ones Huang et al. enumerate9: a value model initialized from a general reward model, dropout off, repeated passes over the prompts with shuffling (about 13 over GSM8K in the ablations), a −10 reward for responses without an EOS token, and advantage whitening8 (§6.2). The rest of the PPO configuration is in Table 4: γ = 1, λ = 0.95, ε = 0.2, temperature 1.0, gradient-norm clip 1.0.
8.2 What the ablations found
Five findings, each from the report's own sweeps8 (§6.2.1, Appendix B.4):
- A reward-model start for the value function is best. It beat starting from the DPO checkpoint on GSM8K and on the average score (section 4.1).
- KL budget does not buy verifiable reward. On GSM8K and MATH, "incurring more KL budget does not necessarily lead to improvements in verifiable rewards", and larger KL lowered the average score across evaluations: "overoptimization happens". The clearest failure is on constraint prompts at β = 0.01, where the policy satisfied the checker and dropped the content: for "the letter e should appear 14 times" it answered "e, e, e, …" fourteen times, and for "5 paragraphs" it printed bare "Paragraph 1 *** Paragraph 2 …" separators (Appendix B.4, Figure 28).
- Do not add reward-model scores on top of the verifier. Verifier-only reward outperformed verifier plus RM score, which added noise to the average.
- A stronger start learns with less drift. Starting from the weaker SFT checkpoint reached the same verifiable reward as the DPO checkpoint but at a larger KL for the same β, and the stronger start gave better test performance.
- Selection matters. Checkpoints were evaluated every 100 steps (40 at 70B) and the final 8B model was the one with the best overall MATH and IFEval performance; some 8B runs reached 89.4% on GSM8K and 84.8% on IFEval but did worse elsewhere and lowered the average (§6.4).
| Run | GPUs | Wall-clock (h) | GPU-hours | Average | GSM8K | MATH | IFEval |
|---|---|---|---|---|---|---|---|
| 8B reward model (value init) | 8 | 9 | 72 | n/a | n/a | n/a | n/a |
| 8B RLVR | 8 | 65 | 520 | 64.4 → 64.8 | 84.3 → 87.6 | 42.0 → 43.7 | 81.1 → 82.4 |
| 70B RLVR | 48 | 60 | 2,880 | 75.9 → 76.0 | 93.5 → 93.5 | 62.3 → 63.0 | 82.6 → 83.2 |
| 405B RLVR | 256 | 46 | 11,776 | n/s | not used | over +5 in 25 steps | not used |
Read with the average column in view, the result is modest and targeted. At 8B, GSM8K gains 3.3 points, MATH 1.7 and IFEval 1.3, while MMLU drops 0.5 (68.7 to 68.2), TruthfulQA 1.1 and the safety average 1.7 (87.2 to 85.5); the average moves 0.4. At 70B, GSM8K was already saturated at 93.5 and the average moves 0.18 (Table 23). The other benchmarks stay within about two points, without any pretraining mix, which supports Lambert's description of RL as something that "can be done surgically" when the prompt distribution is narrow2 (§3.1.6). It does not show a free lunch: the safety and knowledge numbers move the wrong way.
8.3 Infrastructure
The system holds three models, since the reward model is gone: policy and value are trained, and the reference only runs inference. ZeRO stage 3 fits them, and Ray places dedicated GPUs running vLLM with PagedAttention for generation, because "often inference time is a bottleneck in RLHF infrastructure". Training is asynchronous: inference runs alongside the learner, and the policy trains on the second-latest inference batch for reproducibility, at the price of stale data when the sampler outruns the learner8 (§6.3).
At 405B the split is 16 GPUs for vLLM at 16-way tensor parallelism and 240 for training, with an NCCL broadcast to move weights. Per iteration the report gives about 550 s of inference, about 25 s of weight transfer and about 1,500 s of training, an 8B value model standing in for a 405B critic, 75 steps run, and NCCL timeouts that needed manual intervention8 (§8.1). Module 1 (Section 5.2) turns those timings into a utilization estimate; System Design Modules 36 and 37 cover how the placement and weight transfer are engineered. The report's closing suggestion is "larger value models or alternate value model-free RL algorithms such as GRPO" (§8.1), which is where Part 3 begins.
9. PPO versus DPO: cost, quality, and implementation details
9.1 Cost
The cost side is not in dispute. In Tulu 3's controlled, untuned setup PPO took about 28 hours on two nodes and DPO about 4 hours on one8 (§5.4.1), 14 times the node-hours (derived; Module 1, Section 7.3). Llama 3's team "also explored on-policy algorithms such as PPO" and found "DPO required less compute for large-scale models"10 (§4.1.4). Ivison et al. give the reason: DPO needs no reward-model stage, no online decoding and no value model6 (§2.1). In their PPO, rollouts took "more than 95%" of training time; a prompt batch of 512 instead of 64 cut a 60-hour run to 12 hours for 0.6 points of average score (61.7 to 61.1), and four rollouts per prompt at batch 256 recovered 61.4 at 32 hours6 (Appendix G, Table 11).
9.2 Quality: a ledger, not a verdict
| Study | Setting and metric | PPO | DPO family | Reading |
|---|---|---|---|---|
| Ivison et al. | Tulu 2 13B, same preference data, 11-benchmark average (UltraFeedback) | 62.2 | 61.0 | PPO ahead on all 5 datasets, mean +0.7; truthfulness −2.5 |
| Xu et al. | CodeLlama 34B, APPS Introductory pass@5; CodeContest 10@1k test | 44.4%; 22.4% | DPO-Iter 34.2%; DPO 0.0%, DPO-Iter 3.2% | DPO-Iter below SFT (38.6%) on APPS even with test-case labels |
| Xu et al. | Llama 2 7B, HH-RLHF, OpenAssistant reward | 0.718 | DPO 0.611; DPO-Iter 0.678 | GPT-4 judged PPO over DPO 42 to 30 (28 ties) |
| Ahmadian et al. | GPT-4 win-rate (%): TL;DR, HH Pythia, HH Llama | 67.6, 29.2, 32.0 | DPO 66.6, 39.0, 61.9 | DPO ahead on both HH runs; RLOO (k = 4) 77.9, 43.7, 64.1 |
| Tulu 3, Table 18 | early Tulu 3 SFT checkpoint, UltraFeedback; Tulu average (hyperparameter exploration) | 54.5, 55.5 (β = 0.0325, 0.05) | DPO 55.2; length-normalized DPO 46.8 to 57.3 over 6 settings; SFT base 55.7 | PPO 0.2 and 1.2 below the SFT base; length-normalized DPO best at 57.3 |
| Tulu 3, §5.4.1 and Figure 16 | later controlled, untuned comparison: DPO preference mix, RM trained on it; Tulu average | "similar (though slightly lower)" than DPO (Figure 16 caption); values plotted only | values plotted only | PPO about 28 h on 2 nodes; DPO about 4 h on 1 node |
| Llama 3 | 8B to 405B product models | "explored" | chosen | "performed better, especially on instruction following benchmarks like IFEval"; no numbers |
Three rows (Xu et al. twice, Ivison et al.) have PPO ahead, one (Ahmadian et al.) has it far behind on two of three datasets, and Tulu 3 and Llama 3 judge it not worth its cost. The rows differ in the reward (test cases in Xu et al.'s code tasks, learned reward models elsewhere), in PPO tuning (Xu et al. and Ivison et al. tuned; Ahmadian et al. ran a plain PPO to argue it is over-built), and in DPO setup (Xu et al. show DPO degrading under distribution shift; Tulu 3's Table 18 shows length-normalized DPO spanning 10.5 points across six hyperparameter settings). One label covers all of these runs, which is the first reason to distrust a single "PPO versus DPO" number.
9.3 Implementation details move the result more than the algorithm label
Engstrom et al. found for MuJoCo that "code-level optimizations" account for most of PPO's gain in cumulative reward over TRPO13. RLHF shows the same pattern, and Figure 7 is its cleanest demonstration. Xu et al. hold the algorithm, data and reward fixed and add three details to a PPO with batch size 64. On APPS Introductory, the baseline PPO scores 18.0%, 20.6 points below its own SFT starting point (38.6%); advantage normalization alone brings it back to 38.1%, a larger batch to 42.3%, and an exponentially averaged reference model to 44.4%, 5.8 points above SFT (derived)15 (Table 3). On the dialogue task the same three changes move the reward only from 0.706 to 0.718. In this one study the details mattered most on the hard, sparse-reward tasks.
| Detail | Reported effect | Source |
|---|---|---|
| Dropout off | log-probs reproducible; ρ = 1 at the first step; reliable KL | Huang Detail 7; Tulu 3 §6.2 |
| Separate value network | outperformed a shared network; costs memory | Stiennon App. G.1 |
| Value init from the RM | best GSM8K and average in Tulu 3 RLVR | Tulu 3 §6.2.1 |
| EOS penalty | defines a reward for truncated responses (−10 in Tulu 3 and Ivison et al.) | Huang Detail 23; Tulu 3; Ivison |
| Reward whitening | shorter responses; similar quality at matched length | Huang Detail 24, §7.1 |
| Forward passes per minibatch | avoided destabilization at batch 512, minibatch 64 | Ivison App. G |
| RM benchmark score | RewardBench 61.0, 73.6, 73.9, 79.8 gave PPO averages 62.2, 62.8, 61.8, 61.6 | Ivison Table 3 |
The last row is the other half of the lesson. The reward models ranked 79.8, 73.9, 73.6 and 61.0 on RewardBench produced PPO policies scoring 61.6, 61.8, 62.8 and 62.2 on the 11-benchmark average, close to the reverse order, with the best RewardBench model giving the lowest average and only GSM8K moving clearly (53.0 to 58.0 with the 70B UltraFeedback RM)6 (Table 3). Tulu 3 draws the same caution about reward-model benchmarks8 (§5.4.1). A caution on one detail: Xu et al. attribute the reference EMA to InstructGPT, whose Appendix C.4 describes an exponential moving average of the weights (decay 0.992) and uses the SFT model for the KL term, without saying the reference itself was averaged1. Treat the reference EMA as Xu et al.'s own result.
Where experts disagree
Two questions are tangled in this module's evidence: whether online RL against a reward model beats offline preference optimization, and whether PPO's critic earns its place. The sources answer them differently depending on the reward, the budget and how carefully each side was tuned.
A. For preference data, a DPO-family method is enough
Llama 3 team10 (§1, §4.1.4); Tulu 38 (§5.4.1)
Llama 3 chose SFT, rejection sampling and DPO over "more complex reinforcement learning algorithms" that "tend to be less stable and harder to scale". Tulu 3 did preference iteration with DPO and reserved PPO "primarily for RLVR".
- About 14 times fewer node-hours in Tulu 3's controlled test, and no reward-model stage, decode loop or value model.
- Used at 405B in Llama 3.
- Tulu 3 calls its PPO untuned and says more budget could raise it.
- Xu et al. find DPO sensitive to distribution shift and failing on code even with test-case labels15.
B. Tuned online PPO wins when the reward is good
Xu et al.15; Ivison et al.6; Lambert2 (§3.1.6)
Xu et al. report PPO ahead on every testbed, including code competitions. Ivison et al. find PPO ahead by up to 2.5% in math and 1.2% in general domains. Lambert writes that "RL-based methods continue to win out", at much higher infrastructure cost.
- 22.4% against 0.0% to 3.2% on CodeContest, where the reward is a test suite.
- Online samples track the policy's own distribution.
- Margins on learned-reward tasks are small next to preference-data quality (up to 8 points in Ivison et al.).
- The wins depend on details a plain PPO lacks (Figure 7), and Ahmadian et al. find PPO well behind DPO on HH7.
C. The critic is dead weight
Ahmadian et al.7 (§1); DeepSeekMath3 (§4.1.1); Kimi k1.514 (§2.3.2)
The reward arrives for the whole completion, so model the completion as one action and baseline it with other samples. Ahmadian et al. report REINFORCE ahead of PPO by 3.2% to 20.3% win-rate and call PPO "unnecessarily complicated for a pre-trained LLM environment".
- Drops a policy-sized model and GAE; RLOO at k = 4 scored 77.9 on TL;DR against PPO's 67.6.
- Kimi's credit-assignment argument applies to long reasoning.
- The PPO baselines are the proposers' own, run without the Xu et al. details.
- RLOO pays in extra sampling per prompt, and Tulu 3 lists GRPO only as untested future work.
D. A well-built critic still pays
Tulu 38 (§6.2.1); Stiennon et al.5 (App. G.1); VAPO16 (Abstract, Table 1)
Tulu 3 concludes the value function "plays an important role" in RLVR, and VAPO reports 60.4 on AIME24 with a value-based recipe on Qwen 32B, against 47 for R1-Zero with GRPO and 50 for DAPO as it cites them.
- Token-level credit and lower variance than a bare outcome baseline.
- A separate value network beat one sharing the policy's parameters.
- VAPO's own table gives 5 for vanilla PPO: the win needs value pretraining and length-adaptive λ.
- It costs a second large model, and Tulu 3 used an 8B stand-in at 405B.
On the first question, A is the default for preference tuning with learned reward models: the measured gap is at most a few points, smaller than the effect of data quality and sensitive to implementation details, while the cost gap is large. B is right when the reward is a trustworthy checker and the task is hard enough that offline pairs lose coverage, as in the code results; there the reward, not the algorithm label, is the variable. On the second question the evidence does not settle it. C's evidence compares against untuned PPO, and D's is a few recipes that needed substantial engineering. No source compares a tuned PPO (Xu et al.'s three details, a reward-model-initialized value network, a tuned λ) against RLOO or GRPO in one codebase at equal generated tokens, with a learned reward and with a verifier. That experiment would settle both. The reasoning-RL reports of Part 3 behave as if C were right for long chains of thought, which is evidence about practice, not about the optimum.
What is changing
PPO with a critic is now mostly a baseline. Lambert notes that REINFORCE-style methods without a value model drew interest for their simplicity and that GRPO is particularly popular for reasoning tasks2 (§6.1); DeepSeekMath introduced GRPO as a PPO variant that "foregoes the critic"3, and Tulu 3 closed by naming value-free methods as the next step8. What persists is the scaffolding: dropout off, EOS handling, advantage normalization, the placement of the KL term, and one gradient step per batch, which Lambert notes leaves PPO's own clip inactive (§6.4.2). The pretraining-gradient term is absent from the open recipes read here. Modules 9 to 12 cover the critic-free successors and Module 33 the choice between them. No primary source used here describes the closed labs' current practice.
References
- paperOuyang, L., Wu, J., Jiang, X., Almeida, D. et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: dataset sizes and the labeler team (§1, §3.2), the bandit environment, per-token KL, RM-initialized value function and the PPO-ptx objective (§3.5, Eq. 2), PPO hyperparameters and the 6B RM and value choice (Appendix C.2-C.4), the alignment-tax, KL-coefficient and learning-rate ablations (Appendix E.6-E.11, Table 14), headline results, limitations and compute (§1, §4, §5).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: the RLHF loop figure (Fig. 19, §6.1), PPO value-function and loss code (§6.2.7, §6.3.5), the one-gradient-step simplification (§6.3.5.1), update-iteration counts across recipes (§6.4.2), RL "surgical" and "continue to win out" (§3.1.6).
- paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the PPO objective and per-token KL reward (§4.1.1, Eq. 1-2), the case against the value model and KL placed in the loss (§4.1.1, Fig. 4).
- paperZiegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., Irving, G. Fine-Tuning Language Models from Human Preferences. arXiv 1909.08593 v2, 8 Jan 2020. https://arxiv.org/abs/1909.08593. Accessed 2026-10-27. Used for: the KL-penalized reward and its purposes (§2.1, Eq. 2), the adaptive KL controller and the PPO settings (§2.2).
- paperStiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P. Learning to summarize from human feedback. NeurIPS 2020; arXiv 2009.01325 v3, 15 Feb 2022. https://arxiv.org/pdf/2009.01325. Accessed 2026-10-27. Used for: the two purposes of the KL term, the token-level time step and the separate value network initialized from the reward model (§3.1), PPO settings (Appendix B), the shared-versus-separate value ablation (Appendix G.1).
- paperIvison, H., Wang, Y., Liu, J., Wu, Z. et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv v2, 7 Oct 2024. https://arxiv.org/abs/2406.09279. Accessed 2026-10-27. Used for: DPO versus PPO (§2.1, §3.2, Table 2), reward-model scale versus downstream PPO quality (§3.3, Table 3), PPO equations, implementation details and hyperparameters (Appendix F, Tables 9-10), the rollout share of training time and batch-size trade-offs (Appendix G, Table 11).
- paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: the single-action argument and the claim that PPO is over-built (§1), clipping rarely active and the effect of removing clips (§3.2), win-rates for PPO, REINFORCE, RLOO and DPO (Table 1).
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. (Allen Institute for AI). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025 (v1, 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the early-checkpoint DPO and PPO exploration (Table 18) and, separately, the controlled PPO-versus-DPO comparison, Figure 16 and the 28 h and 4 h run times (§5.4.1), the RLVR objective, data, implementation details and ablations (§6.1-6.2, Tables 21-22), infrastructure and run costs (§6.3), final results (§6.4, Table 23), the 405B run (§8.1, and the 405B hyperparameter table labelled Table 35, whose caption describes the general-RM run while its column header reads 405B RLVR), IFEval over-optimization (Appendix B.4), the 70B KL coefficient discrepancy (§6.4 against Table 21 caption).
- paperHuang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., Tunstall, L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. arXiv 2403.17031 v1, 24 Mar 2024 (published at COLM 2024, per Tulu 3's bibliography). https://arxiv.org/abs/2403.17031. Accessed 2026-10-27. Used for: dropout off (Detail 7), RM-initialized value behaviour (Details 13, 22), prompt reuse (Detail 21), the EOS trick (Detail 23), reward and advantage whitening (Details 24-25), PPO settings (Table 7), observations on over-optimization and token shift (§7.1-7.2).
- paperGrattafiori, A., Dubey, A., Jauhri, A. et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: the choice of DPO over PPO (§1, §4.1.4) and the 128,000-token vocabulary (§3.2, Table 3).
- paperSchulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016; arXiv v6, 20 Oct 2018. https://arxiv.org/pdf/1506.02438v6. Accessed 2026-10-26. Used for: the GAE estimator and the TD residual (§3).
- paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the MuJoCo reference hyperparameters (Appendix A, Table 3).
- paperEngstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., Madry, A. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. ICLR 2020; arXiv 2005.12729 v1, 25 May 2020. https://arxiv.org/abs/2005.12729. Accessed 2026-10-27. Used for: code-level optimizations and their share of PPO's gain over TRPO (abstract, §2 list, value-function clipping first).
- paperKimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv v4, 3 Jun 2025 (v1, 22 Jan 2025). https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: excluding the value network, and the credit-assignment argument against value functions for long chain of thought (§2.3.2).
- paperXu, S., Fu, W., Gao, J., Ye, W. et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-27. Used for: the three key PPO factors and the Table 3 ablation (§5), implementation notes (§5), benchmark results on HH-RLHF, SafeRLHF, APPS and CodeContest (§6, Tables 4-8), DPO under distribution shift (§4).
- paperByteDance Seed (Yue, Y. et al.). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv 2504.05118 v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Used for: AIME24 avg@32 of the value-based recipe against vanilla PPO, R1-Zero GRPO and DAPO (Abstract, Table 1).