Reinforcement Learning for LLM Post-Training · Module 6
KL Regularization and Reward-Model Over-Optimization: Reference Policies, the k1/k2/k3 Estimators, and Gold-versus-Proxy Scaling Laws
Almost every RLHF-style recipe keeps the policy on a KL leash to a reference model, and then spends compute finding out how far the leash can be loosened. This module derives what the leash does (a closed-form optimum), how it is estimated from sampled tokens and where it is placed (reward or loss, k1/k2/k3), what gradient each choice really produces, and what the gold-versus-proxy scaling laws say about how much optimization a learned reward survives. Afterwards you can pick and implement a KL estimator for a given placement, read a KL curve, and say what a KL budget buys.
1. Why the policy is kept on a leash
1.1 The objective and the collapse it prevents
Maximizing expected reward alone has a degenerate optimum. If the reward is a function of the sequence, the optimal policy is a point mass on the highest-scoring sequence, and Korbak et al. point out that this holds even with perfect exploration and a reward that perfectly captures preferences2. Empirically the collapse shows up as lost fluency and diversity: higher perplexity, lower entropy, repetition2. Ziegler et al., who first ran RL against a learned reward on language, saw it directly. Without a KL term their mock-sentiment policy reached a score of about +8.0 (99.97% positive) while emitting gibberish built from repeated words such as "instantly" and "tightly"1 (Table 10).
Their remedy, still the template, is to subtract a log-ratio from the reward: \(R(x,y)=r(x,y)-\beta\log\frac{\pi(y|x)}{\rho(y|x)}\), with \(\rho\) the initial language model (their Eq. 2). They list three roles for the term: it acts as an entropy bonus, it keeps the policy in the range where the reward model is valid, and on style tasks it is part of the task definition1. Stiennon et al. restate the first two3, and InstructGPT adds a per-token KL penalty from the SFT model "to mitigate over-optimization of the reward model"4 (§3.5). In the notation of the DPO paper (Eq. 3)5 the objective is
$$ J(\theta)\;=\;\mathbb{E}_{x\sim\mathcal D,\;y\sim\pi_\theta(\cdot|x)}\big[r_\phi(x,y)\big]\;-\;\beta\,\mathbb{D}_{\mathrm{KL}}\big[\pi_\theta(\cdot|x)\,\|\,\pi_{\mathrm{ref}}(\cdot|x)\big]. $$The entropy-bonus role is literal. Since \(\mathrm{KL}(\pi\|\pi_{\mathrm{ref}})=-H(\pi)-\mathbb{E}_{\pi}[\log\pi_{\mathrm{ref}}]\), the objective equals \(\mathbb{E}[r_\phi+\beta\log\pi_{\mathrm{ref}}]+\beta H(\pi_\theta)\) (derived): an entropy bonus plus a reward for reference likelihood.
1.2 The optimum in closed form
For a fixed prompt define \(Z(x)=\sum_y \pi_{\mathrm{ref}}(y|x)\,e^{r(x,y)/\beta}\) and \(\pi^*(y|x)=\pi_{\mathrm{ref}}(y|x)\,e^{r(x,y)/\beta}/Z(x)\). Then, following the DPO paper's Appendix A.15,
$$ J=\mathbb{E}_{y\sim\pi}\Big[r-\beta\log\tfrac{\pi}{\pi_{\mathrm{ref}}}\Big]=-\beta\,\mathbb{E}_{y\sim\pi}\Big[\log\tfrac{\pi}{\pi^*}-\log Z\Big]=-\beta\,\mathrm{KL}(\pi\|\pi^*)+\beta\log Z(x). $$\(Z\) does not depend on \(\pi\), so maximizing \(J\) is minimizing \(\mathrm{KL}(\pi\|\pi^*)\): the optimum is \(\pi=\pi^*\), and the best attainable objective is \(\beta\log Z(x)\) (the last step is derived here). The same cancellation of \(Z\) is what lets DPO skip the RL loop; Module 7 reuses this result. Korbak et al. read \(\pi^*\) as a Bayesian posterior with prior \(\pi_{\mathrm{ref}}\) and evidence \(e^{r/\beta}\), so that RL with a KL penalty is variational inference toward it2 (§4).
Table 1 evaluates this on four candidate responses with rewards 0 to 3 and reference probabilities (0.50, 0.30, 0.15, 0.05), so the reference expects \(\mathbb{E}_{\mathrm{ref}}[r]=0.75\). The rare, best response is amplified by \(e^{3/\beta}/Z\): from 0.05 to 0.29 at \(\beta=1\) and 0.65 at \(\beta=0.5\), for 0.53 and 1.65 nats.
| \(\beta\) | \(\pi^*(r{=}0)\) | \(\pi^*(r{=}1)\) | \(\pi^*(r{=}2)\) | \(\pi^*(r{=}3)\) | \(\mathbb{E}_{\pi^*}[r]\) | KL(\(\pi^*\|\pi_{\mathrm{ref}}\)) (nats) |
|---|---|---|---|---|---|---|
| \(\infty\) (= reference) | 0.5000 | 0.3000 | 0.1500 | 0.0500 | 0.7500 | 0 |
| 3 | 0.3713 | 0.3109 | 0.2169 | 0.1009 | 1.0475 | 0.0515 |
| 1 | 0.1459 | 0.2379 | 0.3233 | 0.2930 | 1.7634 | 0.5314 |
| 0.5 | 0.0161 | 0.0713 | 0.2635 | 0.6491 | 2.5456 | 1.6546 |
Derivation of the slope identity
Let \(s=1/\beta\), \(\pi_s\propto\pi_{\mathrm{ref}}e^{sr}\) and \(\Lambda(s)=\log\sum_y\pi_{\mathrm{ref}}(y)e^{sr(y)}\). Then \(\mathbb{E}_{\pi_s}[r]=\Lambda'(s)\), \(\mathrm{Var}_{\pi_s}[r]=\Lambda''(s)\), and \(\mathrm{KL}(\pi_s\|\pi_{\mathrm{ref}})=s\Lambda'(s)-\Lambda(s)\). Differentiating, \(d\mathrm{KL}/ds=s\Lambda''(s)\) and \(d\mathbb{E}[r]/ds=\Lambda''(s)\), so \(d\mathbb{E}[r]/d\mathrm{KL}=1/s=\beta\).
1.3 Which direction of KL, and what it fails to protect
RLHF penalizes the reverse KL, \(\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\), an expectation under the policy being trained, so it can be estimated from the policy's own rollouts. Lambert describes the consequence: it applies a large penalty when the new model puts substantial mass where the reference assigns little, whereas the other direction behaves like distillation or behavioral cloning7 (§15.1). Figure 3 shows the geometry on a constructed example: a single Gaussian fitted to a two-mode target. Minimizing the forward KL gives the moment-matched fit, centered between the modes. Minimizing the reverse KL locks onto one mode, at a cost of \(\ln 2\).
The general version of the green fit's cost is a one-line identity (derived): if the policy equals the reference restricted to a set of reference mass \(m\), then \(\mathrm{KL}(\pi\|\pi_{\mathrm{ref}})=\ln(1/m)\). Discarding 90% of the reference's mass costs \(\ln 10=2.30\) nats, a bounded price that a coefficient of order 0.01 to 0.1 turns into a negligible penalty. This is my reading, not a result from the cited papers: the reverse-KL leash restrains drift into regions the reference finds unlikely far better than it restrains loss of diversity, which is one reason entropy collapse (Module 13) can proceed with the penalty switched on.
2. Estimating KL from sampled tokens
2.1 What a training loop can compute
Fix one token position. Samples come from the policy, \(x\sim q=\pi_\theta(\cdot|\text{prefix})\); the other distribution is \(p=\pi_{\mathrm{ref}}\); write \(r=p(x)/q(x)=\pi_{\mathrm{ref}}/\pi_\theta\). Schulman's setting is the right one for LLMs: probabilities are computable for any \(x\), the sum over the vocabulary is not wanted, and "we can simplify code by just storing the log-prob, not the whole distribution"8. At sequence level the KL is the expected sum of per-token KLs along sequences sampled from the policy9 (§4), so a sampled-token estimator summed over positions estimates the sequence KL.
The systems reason is memory. An exact per-token KL needs the full next-token distribution from both models. With a 128K-token vocabulary10 (§3.2), fp32 logits are about 128,000 × 4 B = 500 KiB per token per model, so a 1M-token rollout batch is about 512 GB per model if materialized (derived; real implementations chunk the computation). A sampled estimator needs one gathered log-probability per token per model, 4 MB per million tokens. The reference model itself is one forward pass, with no gradients, over each rollout. When the coefficient is zero TRL does not load it at all: "reducing memory usage and improving training speed"11. In bf16 (2 B per parameter, derived) that avoids 16 GB of reference weights at 8B parameters, 140 GB at 70B and 810 GB at 405B. Module 1 counts the models in an RL step; System Design Modules 36 to 38 cover where they are placed.
2.2 The three estimators
Schulman's post builds three single-sample estimators of \(\mathrm{KL}[q,p]=\mathbb{E}_q[\log(q/p)]\) from samples of \(q\)8:
- \(k_1=-\log r\). Unbiased but noisy: "it's negative for half of the samples, whereas KL is always positive".
- \(k_2=\tfrac12(\log r)^2\). Lower variance, biased. Its expectation is an f-divergence with \(f(x)=\tfrac12(\log x)^2\), and every f-divergence with differentiable \(f\) looks like KL to second order when \(q\approx p\) (both have \(f''(1)=1\)), which is why the bias is small near the reference.
- \(k_3=(r-1)-\log r\). Take \(k_1\) and add \(\lambda(r-1)\), which has zero mean because \(\mathbb{E}_q[r]=1\). With \(\lambda=1\) the result is non-negative, since \(\log x\le x-1\): it is the vertical distance between the log curve and its tangent.
The same construction gives an estimator for the opposite direction, \(\mathrm{KL}[p,q]\): use \(f(r)-f'(1)(r-1)\) with \(f(x)=x\log x\), giving \(r\log r-(r-1)\)8.
| Estimator | Expression | Unbiased value? | Always \(\ge 0\)? | std/KL at 0.005 | std/KL at 0.5 | bias/KL at 0.005 | bias/KL at 0.5 |
|---|---|---|---|---|---|---|---|
| \(k_1\) | \(-\log r\) | yes | no | 20 | 2 | 0 | 0 |
| \(k_2\) | \(\tfrac12(\log r)^2\) | no | yes | 1.42 | 1.73 | 0.002 | 0.25 |
| \(k_3\) | \((r-1)-\log r\) | yes | yes | 1.42 | 1.7 | 0 | 0 |
Schulman's verdict is "k3 has even lower standard deviation than k2 while being unbiased, so it appears to be a strictly better estimator"8. That is a statement about estimating the value of the KL. §3 shows it answers a different question from the one a training loop asks.
2.3 Reproducing the numbers
For equal-variance Gaussians with mean shift \(\mu\) (true KL \(=\mu^2/2\)), the table entries have closed forms (derived here):
$$ \frac{\mathrm{std}(k_1)}{\mathrm{KL}}=\frac{2}{\mu},\qquad \frac{\mathrm{bias}(k_2)}{\mathrm{KL}}=\frac{\mu^2}{4}=\frac{\mathrm{KL}}{2},\qquad \frac{\mathrm{std}(k_2)}{\mathrm{KL}}=2\sqrt{\tfrac12+\tfrac{\mu^2}{4}},\qquad \frac{\mathrm{std}(k_3)}{\mathrm{KL}}=\frac{\sqrt{e^{\mu^2}-1-\mu^2}}{\mu^2/2}. $$At \(\mu=0.1\) these give 20, 0.0025, 1.4177 and 1.4166; at \(\mu=1\) they give 2, 0.25, 1.7321 and 1.6950. They agree with the post's tables. The one visible difference is the first \(k_2\) bias, which the post prints as 0.002 from a Monte Carlo run; my rerun with \(2\times10^7\) samples gave 0.0020 against a standard error of about 0.0003, so the exact 0.0025 is consistent with it. The practical reading of the 20 in the first row (derived): to estimate a KL of 0.005 to 10% relative error, \(k_1\) needs about \((20/0.1)^2=40{,}000\) tokens and \(k_3\) about \((1.42/0.1)^2\approx 200\).
3. Where the term enters: reward or loss, and the gradient each produces
3.1 KL in the reward
In the PPO-RLHF lineage the log-ratio is attached to each token's reward and treated as data. DeepSeekMath writes the standard form (its Eq. 2)12 (§4.1.1):
$$ r_t=r_\phi(q,o_{\le t})-\beta\log\frac{\pi_\theta(o_t\mid q,o_{<t})}{\pi_{\mathrm{ref}}(o_t\mid q,o_{<t})}, $$with the reward-model score typically present only at the last token. This is \(k_1\), one sample per position; summed over positions the log-ratios give the sequence log-ratio. Because it sits in the reward it passes through the return and GAE into the advantage \(\hat A_t\), and the policy gradient then uses it as a coefficient on \(\nabla\log\pi_\theta\). InstructGPT applies no discount when estimating the advantage4 (Appendix C.4), so the return at token \(t\) contains the KL terms of all later tokens. Module 5 draws this dataflow.
3.2 KL in the loss
GRPO moves the term out of the reward: "instead of adding KL penalty in the reward, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of \(\hat A_{i,t}\)"12 (§4.1.1). The estimator is \(k_3\) (their Eq. 4), described as "guaranteed to be positive", and DeepSeek-R1 uses the same form in its Eqs. 1 and 213 (§2.1). TRL's GRPO trainer applies the same per-token estimate inside the loss, with the coefficient defaulting to 0.011. Here nothing is detached: automatic differentiation reaches the KL term through \(\log\pi_\theta\) directly.
3.3 Value is not gradient
A training loop never uses the estimator's value; it differentiates it. The KL is an expectation under \(\pi_\theta\), so its gradient has two parts: a pathwise term from the integrand and a score-function term from the sampling distribution. Differentiating a sampled loss with the sample held fixed yields only the first9 (§3), and Shah et al. write the full gradient as the sum of both14 (Eq. 4). Write \(s=\nabla_\theta\log\pi_\theta(x)\), so \(\mathbb{E}_{x\sim\pi_\theta}[s]=0\), and apply the chain rule to \(r=\pi_{\mathrm{ref}}/\pi_\theta\), for which \(\nabla r=-r\,s\). The per-sample loss gradients are (derived):
$$ \nabla k_1=s,\qquad \nabla k_2=\log\tfrac{\pi_\theta}{\pi_{\mathrm{ref}}}\,s,\qquad \nabla k_3=\Big(1-\tfrac{\pi_{\mathrm{ref}}}{\pi_\theta}\Big)s. $$Taking expectations under \(x\sim\pi_\theta\): \(\mathbb{E}[\nabla k_1]=\mathbb{E}[s]=0\). \(\mathbb{E}[\nabla k_2]=\mathbb{E}[\log\frac{\pi_\theta}{\pi_{\mathrm{ref}}}\,s]=\nabla\,\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\), which is the score-function form of the reverse-KL gradient. And \(\mathbb{E}[\nabla k_3]=0-\sum_x\pi_{\mathrm{ref}}(x)\nabla\log\pi_\theta(x)=\nabla\,\mathrm{KL}(\pi_{\mathrm{ref}}\|\pi_\theta)\), the gradient of the forward KL. So differentiating \(k_1\) adds only zero-mean noise9 (Eq. 2), differentiating \(k_2\) gives the right gradient although \(k_2\) is a biased value estimate, and differentiating \(k_3\), the estimator everyone uses, regularizes toward the wrong direction of KL9 (Eq. 3). Liu et al. reach the same conclusions: \(k_1\) in the reward and \(k_2\) as a loss are gradient-equivalent on-policy, and \(k_3\) as a loss is a biased first-order approximation15. Shah et al. add that the \(k_3\)-in-loss gradient is a sum of per-token forward-KL gradients14 (§4.1).
| \(\pi_{\mathrm{ref}}\) | \(\pi_\theta\) | \(r\) | \(k_1\) | \(k_2\) | \(k_3\) | coef \(k_1\) | coef \(k_2\) | coef \(k_3\) |
|---|---|---|---|---|---|---|---|---|
| 0.30 | 0.30 | 1 | 0.000 | 0.000 | 0.000 | 1 | 0.000 | 0.000 |
| 0.30 | 0.60 | 0.5 | 0.693 | 0.240 | 0.193 | 1 | 0.693 | 0.500 |
| 0.60 | 0.30 | 2 | -0.693 | 0.240 | 0.307 | 1 | -0.693 | -1.000 |
| 0.02 | 0.40 | 0.05 | 2.996 | 4.487 | 2.046 | 1 | 2.996 | 0.950 |
| 0.50 | 0.001 | 500 | -6.215 | 19.311 | 492.785 | 1 | -6.215 | -499.000 |
The last row is the case DeepSeek-V3.2 warns about (§3.6): the policy has learned to avoid a token the reference likes. Sampled once, it contributes 492.8 to the \(k_3\) sum, which is about 1.0 to the per-token mean of a 500-token response, and its gradient coefficient is -499, against -6.2 for \(k_2\). Note also that \(k_1\) is negative on the rows where the policy has moved away from a token the reference favors; it is an unbiased average, not a per-token distance.
| Quantity | token a | token b | token c | token d | Equals |
|---|---|---|---|---|---|
| true \(\nabla\,\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\) | +0.2474 | -0.0846 | -0.1001 | -0.0627 | the target (reverse KL) |
| true \(\nabla\,\mathrm{KL}(\pi_{\mathrm{ref}}\|\pi_\theta)\) | +0.2500 | -0.0500 | -0.1000 | -0.1000 | \(\pi_\theta-\pi_{\mathrm{ref}}\) (forward KL) |
| \(\mathbb{E}[\nabla k_1]\) | 0 | 0 | 0 | 0 | nothing |
| \(\mathbb{E}[\nabla k_2]\) | +0.2474 | -0.0846 | -0.1001 | -0.0627 | reverse KL |
| \(\mathbb{E}[\nabla k_3]\) | +0.2500 | -0.0500 | -0.1000 | -0.1000 | forward KL |
| \(\mathbb{E}[\nabla(\tfrac{\pi_\theta}{\pi_{\mathrm{old}}}k_3)]\), \(\pi_{\mathrm{old}}=\pi_\theta\) | +0.2474 | -0.0846 | -0.1001 | -0.0627 | reverse KL (DeepSeek-V3.2 form, §3.6) |
3.4 Token-level losses give a partial gradient
A correct per-token estimator still does not give the gradient of the sequence KL. The sequence KL is \(\mathbb{E}\big[\sum_t \mathrm{KL}_t\big]\) over sequences drawn from \(\pi_\theta\), and a token \(y_t\) also changes the distribution of every later token, hence their KL terms. A sum of token-level losses omits that dependence; Tang and Munos: "token-level losses only produce a partial derivative at best"9 (§4.1). Their sequence-level estimator multiplies the summed log-ratios by the summed score, and dropping the terms with zero mean gives the cumulative form \(\sum_t\big(\sum_{s\ge t}\log\rho_s\big)\nabla\log\pi(y_t|y_{<t})\) with \(\rho=\pi_\theta/\pi_{\mathrm{ref}}\)9 (§4.2). KL-in-reward with no discount has this structure by construction, since the return at token \(t\) is \(-\beta\sum_{s\ge t}\log\rho_s\) (derived). My reading is that PPO-RLHF's placement is therefore sequence-correct up to GAE's \(\lambda\) and critic error, while GRPO's placement inherited the token-level shortcut.
How much it matters depends on how far the policy travels. In Tang and Munos's low-KL experiment (Llama 3 8B on MATH training prompts with a string-match reward, \(\beta=0.001\)) the \(k_3\)-style loss and the proper estimator trace similar performance-versus-KL curves; they suggest that when KL is small the two directions coincide, \(\mathrm{KL}(\pi,\pi_{\mathrm{ref}})\approx\mathrm{KL}(\pi_{\mathrm{ref}},\pi)\)9 (§5.1). In their high-KL experiment, on-policy distillation of a 70B model into an 8B model by minimizing the KL itself, the \(k_3\)-style loss "does not make much progress", the vanilla token-level estimate does, and proper sequence-level estimators with variance reduction do best9 (§5.2, Figure 8). That second setting is the subject of Module 21.
3.5 The empirical grid
Shah et al. (COLM 2026) tested the 2×2 of estimator by placement with everything else fixed: an RLOO baseline, batch 256, 5 samples per prompt, 1,024 maximum tokens, learning rate \(10^{-6}\), on MATH training prompts, with \(\beta\in\{0.05,0.1,0.3,1\}\)14 (§4.1). Only one cell has an unbiased sequence-level gradient (Table 5).
| Configuration | Gradient of the sequence reverse KL | Training outcome |
|---|---|---|
| K1 in reward | unbiased | stable; ahead of K3-in-loss on every task in both models. Gain over K3-in-loss: Qwen2.5-7B +6.21% in-domain, +19.06% out-of-domain; Llama-3.1-8B-Instruct +15.94% in-domain, +3.65% out-of-domain |
| K1 in loss | biased (zero in expectation) | training instabilities in both models at several \(\beta\); where stable, it matches training with no KL |
| K3 in reward | biased | complete or partial collapse for every \(\beta\) tried |
| K3 in loss (GRPO, TRL) | biased: per-token forward KL | stable, but behind K1-in-reward |
Two further results from the same paper. In an asynchronous setup (Qwen3-4B-Instruct-2507 on Countdown, staleness levels 5, 10 and 20), K1-in-reward and K3-in-loss stayed stable while no KL, K1-in-loss and K3-in-reward did not, so the KL term helps with staleness as well as drift14 (§4.2, Figure 5). And adding the estimator to both the reward and the loss recovers the correct gradient up to a normalizing factor, on-policy14 (§2.2). The paper's own text is slightly inconsistent about which \(\beta\) values were stable for K1-in-loss (the Figure 2 caption and the body disagree for Qwen2.5-7B at \(\beta=0.1\)), so Table 5 states only what both agree on.
3.6 DeepSeek-V3.2's importance-ratio fix
Rollouts come from \(\pi_{\mathrm{old}}\), not \(\pi_\theta\). DeepSeek-V3.2 therefore weights the \(k_3\) term by \(\pi_\theta/\pi_{\mathrm{old}}\) (its Eq. 7) and states that the gradient of the original \(k_3\) "assigns disproportionately large, unbounded weights to maximize the likelihood of these tokens" when sampled tokens are much less probable under the current policy than under the reference, \(\pi_\theta\ll\pi_{\mathrm{ref}}\); it also reports that for domains such as mathematics a weak KL penalty, or none, can work better16 (§3.1). Zhang et al. (ICLR 2026) independently identify "an off-policy importance-weighting mismatch in GRPO's KL term"17.
What the fix does to the gradient is not printed in the report, but it follows in two lines (derived). With \(w=\pi_\theta/\pi_{\mathrm{old}}\) differentiated through, \(\nabla(w\,k_3)=w\,(k_3+1-r)\,s=w\,\log\tfrac{\pi_\theta}{\pi_{\mathrm{ref}}}\,s\), because \(k_3+1-r=-\log r\). Under \(\pi_{\mathrm{old}}\) the expectation is \(\mathbb{E}_{\pi_\theta}[\log\tfrac{\pi_\theta}{\pi_{\mathrm{ref}}}\,s]=\nabla\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\), the correct reverse-KL gradient (last row of Table 4), and the per-sample coefficient grows like a logarithm (-6.2 in Table 3's last row) rather than like \(1/\pi_\theta\) (-499). This is still a token-level loss, so §3.4 applies.
# logp: log pi_theta of sampled tokens (requires grad); old_logp, ref_logp: detached; one row per token
log_ratio = logp - ref_logp # log(pi_theta / pi_ref) = -log r
k1_loss = log_ratio # value is an unbiased KL estimate; E[grad] = 0 on-policy
k3_loss = (-log_ratio).exp() - 1 + log_ratio # GRPO/TRL: E[grad] = grad of forward KL(pi_ref || pi_theta)
k2_surrogate = log_ratio.detach() * logp # grad = log_ratio * grad(logp) = reverse-KL gradient
k3_is_loss = (logp - old_logp).exp() * k3_loss # DeepSeek-V3.2 Eq. 7: weight pi_theta/pi_old, differentiated through
# KL in the reward instead (PPO-RLHF): no gradient path, folded into per-token rewards
token_reward = -beta * log_ratio.detach() # add r_phi at the last token, then GAE
4. What a KL budget buys: the coefficient, the reference policy and the schedule
4.1 The coefficient is a price in reward units per nat
By the identity of §1.2, at the optimum one more nat of KL must buy \(\beta\) more reward. A coefficient therefore carries no meaning without two other facts: the scale of the reward, and how the KL is summed. Tulu 3's verifiable reward is 0 or \(\alpha=10\), "based on pilot experiments"18 (§6, Eq. 8), with \(\beta=0.05\). R1's reward is accuracy plus format, converted to a group-standardized advantage13 (Eq. 3), with \(\beta=0.001\). InstructGPT normalizes the reward model so labeler demonstrations score 0 on average and uses \(\beta=0.02\) on a token-summed log-ratio4 (§3.5, Appendix C.4). GRPO averages the KL over tokens, but it averages the surrogate the same way (the common \(1/|o_i|\) in DeepSeekMath's Eq. 3), so relative to the advantage the KL still enters at \(\beta\) per nat of sequence KL (derived). The numbers in Table 6 are priced in different currencies; they are not a menu.
InstructGPT's own sweep shows the same trade: the human Likert score is poor at coefficients 0 and 2, with the best region around 0.01 to 0.024 (Appendix E.7), and raising the coefficient to 2.0, 100 times the default, did not repair the NLP-benchmark regressions, which needed the pretraining mix of Module 5 instead4 (Appendix E).
| Recipe | Placement | Estimator | Coefficient \(\beta\) | Reference policy | Source |
|---|---|---|---|---|---|
| Ziegler et al. 2019 | reward | log-ratio (\(k_1\)) | constant, or adapted to a target KL | initial LM \(\rho\) | 1 (§2, §2.2) |
| Stiennon et al. 2020 | reward | log-ratio (\(k_1\)) | 0.05 (both main runs) | SFT model | 3 (§3.4, App. B.1) |
| InstructGPT 2022 | reward, per token | log-ratio (\(k_1\)) | 0.02; best Likert near 0.01 to 0.02 | SFT model (with pretraining mix) | 4 (App. C.4, E.7) |
| DeepSeekMath-RL 2024 | loss | \(k_3\) | 0.04 | the instruction-tuned model; iterative GRPO (Algorithm 1, explored in §5.2) resets it to the current policy each iteration | 12 (§4.2, Alg. 1) |
| DeepSeek-R1-Zero, R1 stage 1 2025 | loss | \(k_3\) | 0.001 | replaced by the latest policy every 400 steps | 13 (§2.1, §3.2.1) |
| Tulu 3 RLVR 2024 | in the objective (Eq. 7) | not stated | 0.05 (8B), 0.07 (70B); sweep 0.1, 0.05, 0.03, 0.01 | start model (best DPO checkpoint) | 18 (§6, Table 21) |
| DeepSeek-V3.2 2025 | loss | \(k_3\times\pi_\theta/\pi_{\mathrm{old}}\) | domain-dependent; weak or none for math | not stated | 16 (§3.1) |
| ProRL 2025 | loss | not stated | not stated | hard reset to a recent policy snapshot, optimizer re-initialized | 19 (§2.3.1) |
| DAPO 2025 | none | none | removed | none | 20 (§2.3) |
| Olmo 3 (OlmoRL) 2025 | none | none | "No KL loss" | none | 21 (§4.4.1) |
| GLM-5 reasoning RL 2026 | none | none | removed relative to IcePop | none | 22 (§3.2) |
| TRL GRPOTrainer (docs) | loss | \(k_3\) | default 0.0, so the reference is not loaded; docs cite R1's 0.001 | not stated; sync_ref_model defaults to False | 11 |
4.2 The reference policy: anchored, started-from, moving, or absent
The early RLHF papers anchor to the SFT model for the whole run34; InstructGPT also reports that using pretrained GPT-3 instead of the PPO initialization as the KL model "gives similar results"4 (§4.2). Later recipes anchor to whatever the stage started from, which makes the coefficient depend on the starting checkpoint: Tulu 3 found that RLVR from an SFT model and from the DPO model reached the same verifiable reward, but the SFT start incurred a larger KL at the same \(\beta\) on GSM8K, because the SFT model was further from good at the task18 (§6.2.1, Figure 20). KL measures distance travelled, not capability gained.
A third pattern moves the anchor. DeepSeekMath's iterative GRPO sets the reference to the current policy at the start of every outer iteration12 (Algorithm 1), R1 replaces it every 400 steps13, and ProRL hard-resets it, with the optimizer state, because "the KL term may increasingly dominate the loss, leading to diminishing policy updates", and also when validation metrics degrade or plateau19 (§2.3.1, Appendix E). The fourth pattern, no reference, is Table 6's bottom block; Module 13 weighs it for long-CoT runs.
Resets change what the leash means. Suppose each stage converges to the optimum of §1.2 for a fixed reward \(r\). After the first stage \(\pi_1\propto\pi_0e^{r/\beta}\); resetting the reference to \(\pi_1\) and optimizing again gives \(\pi_2\propto\pi_1e^{r/\beta}\propto\pi_0e^{2r/\beta}\), and after \(k\) stages \(\pi_k\propto\pi_0e^{kr/\beta}\) (derived). Relative to the original SFT model the effective coefficient is \(\beta/k\) and the cumulative KL is unbounded in the number of resets. Real stages do not converge and the reward drifts, so the identity is an idealization, but the direction is right: a reset turns "stay near the SFT model" into "stay near the recent past", which is a trust region (Module 3) more than a leash.
4.3 Fixed coefficient or controlled KL
Ziegler et al. noticed that runs with the same \(\beta\) but different seeds ended at quite different KL values, "making them hard to compare", and for some experiments adapted \(\beta\) to hit a target KL with a log-space proportional controller1 (§2.2):
$$ e_t=\mathrm{clip}\Big(\frac{\mathrm{KL}(\pi_t,\rho)-\mathrm{KL}_{\mathrm{target}}}{\mathrm{KL}_{\mathrm{target}}},\,-0.2,\,0.2\Big),\qquad \beta_{t+1}=\beta_t\,(1+K_\beta e_t),\quad K_\beta=0.1. $$The clip bounds each update to a 2% change in \(\beta\) (\(0.1\times0.2\)), so moving \(\beta\) by a factor of 10 takes \(\ln10/\ln1.02\approx116\) updates (derived). Every other recipe in Table 6 that keeps a KL term reports a fixed \(\beta\). The controller converts a price into a target: it equalizes the KL budget across runs, which is what you want when comparing algorithms, and it is the one place where the question "how many nats should this run spend" becomes an input rather than a result. The scaling laws below say what that input should be.
5. Gold versus proxy: scaling laws for over-optimization
5.1 Measuring Goodhart's law with a synthetic human
A learned reward is a proxy, and optimizing it hard eventually lowers the true objective. Measuring that needs ground truth, which means human labels, too expensive to sweep. Gao et al. replace the human with a fixed "gold" reward model, the 6B InstructGPT RM, and train proxy RMs of 3M to 3B parameters on comparisons the gold model labels: 100,000 synthetic comparisons, 10% held out, the higher gold score always marked preferred6 (§2.1). Policies are GPT-3-series models fine-tuned for two epochs on InstructGPT demonstrations. Optimization is PPO (batch 256, clip 0.2, GAE \(\lambda=0.95\), Appendix C) or best-of-\(n\) (BoN); the KL penalty is zero in every RL run except §3.6, and scores are recentered so the initial policy averages 06 (§2.2).
The x-axis of everything that follows is KL, and for BoN it is known in closed form: \(\mathrm{KL}_{\mathrm{bon}}=\ln n-\frac{n-1}{n}\)6 (§2), taken from Stiennon et al.3 (Appendix G.3). This gives 1.40 nats at \(n=10\), 3.62 at 100, 5.91 at 1,000, 8.21 at 10,000 and 10.00 at 60,000 (derived from the formula). The authors hypothesized their functional forms from BoN data up to \(n=1{,}000\) (about 6 nats), then ran BoN to \(n=60{,}000\) (about 10 nats) as a true advance prediction6 (§3.1).
Best-of-n: how many nats does sampling n candidates spend?
Formula: \(\mathrm{KL}_{\mathrm{bon}}=\ln n-\frac{n-1}{n}\), Gao et al. §2 and Stiennon et al. Appendix G.3. The slider is continuous in \(\log_{10}n\); \(n\) itself need not be an integer here.
5.2 Two functional forms, and where they peak
Write \(d=\sqrt{\mathrm{KL}(\pi\|\pi_{\mathrm{init}})}\), chosen because KL behaves as a quadratic distance6 (§1). The gold reward as a function of \(d\) follows, with \(R(0):=0\),
$$ R_{\mathrm{bon}}(d)=d\,(\alpha_{\mathrm{bon}}-\beta_{\mathrm{bon}}\,d),\qquad R_{\mathrm{RL}}(d)=d\,(\alpha_{\mathrm{RL}}-\beta_{\mathrm{RL}}\log d). $$The RL form has infinite slope at the origin, so it likely fails near \(d=0\); the two finite-slope alternatives they tried, \(d(\alpha-\beta\log(1+d))\) and a power law, fit or extrapolated worse (the first) or needed an extra free exponent whose best values approached the log form (the second)6 (footnote 1, Appendix B). The proxy score is roughly linear in \(d\) for both methods6 (§3.2). Setting the derivative to zero gives the peak (derived):
$$ \text{BoN: } d^*=\frac{\alpha}{2\beta},\;\; R^*=\frac{\alpha^2}{4\beta};\qquad \text{RL: } d^*=e^{\alpha/\beta-1},\;\; R^*=\beta\,d^*. $$(For RL, \(R'(d)=\alpha-\beta\log d-\beta=0\), and at that point \(\alpha-\beta\log d^*=\beta\).) Before the peak both rise together; after it the policy is still buying proxy reward but losing gold reward, and the gap is the shortfall. Gao et al. map the two coefficients onto Goodhart's taxonomy: the \(\alpha\) term, the difference between the proxy's slope and the gold curve's linear part, measures regressional Goodhart (optimization selecting on the proxy's noise), while the \(\beta\) term, mostly responsible for the non-monotonic gold curve, reflects extremal Goodhart (the policy leaving the distribution the proxy was trained on)6 (§4.2). Causal Goodhart behaves like regressional; adversarial Goodhart is not captured because the models are not capable enough to manipulate the reward model, and the authors advise caution when extrapolating to systems that can6 (§4.2.4).
Peak of the fitted forms for chosen coefficients
Formulas (derived): BoN \(d^*=\alpha/2\beta\), \(R^*=\alpha^2/4\beta\); RL \(d^*=e^{\alpha/\beta-1}\), \(R^*=\beta d^*\); KL\(^*=d^{*2}\). Defaults are the illustrative coefficients of Figure 6, not fitted values. The RL peak KL is exponential in \(\alpha/\beta\), so small changes in the ratio move it by orders of magnitude.
5.3 What moves the coefficients
Gao et al. vary one factor at a time and read the effect off \(\alpha\) and \(\beta\) (Table 7). The result that matters for planning is the first row: the coefficients move smoothly with reward-model size, so peak gold reward for a larger proxy can be predicted from smaller ones. The result that matters for expectations is the third: a bigger policy gains less from the same reward model but over-optimizes about as much, peaking at nearly the same KL.
| Factor | Range tested | Finding | Where |
|---|---|---|---|
| Proxy RM parameters | 3M to 3B (policy 1.2B, 90,000 RM examples) | \(\alpha_{\mathrm{bon}},\beta_{\mathrm{bon}}\) change smoothly, about logarithmically; for RL, \(\alpha_{\mathrm{RL}}\) can be held fixed and \(\beta_{\mathrm{RL}}\) carries the scaling; peak gold scores are predictable | §1, §3.2, Figs. 3, 12 |
| Proxy RM data | RM fixed at 12M; up to 90,000 comparisons | near-chance below about 2,000 comparisons; more data then lowers Goodharting. Four epochs on 2,000 examples gave no gain, one epoch on 8,000 did; larger RMs show the threshold no earlier | §3.3, Figs. 6, 13 |
| Policy size | 1.2B versus 6B (RM 12M; 3B RM repeated) | larger policy gains less from optimization but gold peaks at almost the same KL and the proxy-gold gap is almost the same | §3.4, Figs. 7, 24 |
| RM validation loss | across RM size and data size | weak evidence that equal validation loss means equal robustness to optimization | §3.3, Fig. 5 |
| KL penalty in RL | several coefficients | gold score depends only on the KL reached; the penalty only makes it converge earlier | §3.6, Fig. 9 |
For the policy-size result the authors offer one hypothesis, from the Bayesian reading of §1.2: if RLHF is inference from the prior of the initial policy, a larger policy only improves the model of the demonstration distribution6 (§4.4). They do not test it.
5.4 RL versus best-of-n, and the penalty as early stopping
The same gold reward costs different amounts of KL under different optimizers. BoN searches close to the initial policy, and its KL grows like \(\ln n\); RL moves the policy a little every step, and without a penalty its KL grows roughly quadratically in the number of steps6 (§3.5). Stiennon et al. had seen the same ordering: at a given average reward the two had similar human-judged quality, but the PPO policy was farther from the supervised model in KL3 (Appendix G.3). Measured against the proxy score instead of KL, BoN and RL look much more alike, although RL begins with a larger proxy-gold gap and then peaks at a higher gold score6 (§3.5, Figure 8). The authors conclude that KL "should not be used to compare the amount of optimization between different optimization algorithms": perturbations orthogonal to the reward raise KL without raising any reward, and tiny, well-aimed perturbations can change behavior within a small KL budget6 (§4.1).
The same paper's finding about the penalty is the least comfortable one for §1. Varying the KL coefficient in RL changed only how soon the gold score converged, not the gold-versus-KL frontier: "akin to early stopping". A nonzero penalty also produced a strictly larger proxy-gold gap, so the authors set it to zero elsewhere, while noting the result "could be particularly sensitive to hyperparameters"6 (§3.6). They also observe that PPO's own clipped surrogate acts as an implicit penalty toward recent policies that slows KL growth and appears to over-optimize less than an explicit penalty, for reasons they do not know6 (§3.6). One reconciliation with InstructGPT's tuned coefficient, which is poor at 0 and at 2 (this module's §4.1): if the penalty is a stopping rule, the best \(\beta\) is the one whose equilibrium KL lands near the gold peak. That is my reading, not a result of either paper.
5.5 Iterating the reward model
Bai et al. recommended refreshing the reward model with new preference data during RLHF; Gao et al. analyze it under two assumptions, constant \(\alpha_{\mathrm{RL}},\beta_{\mathrm{RL}}\) across iterations and additive \(d\)6 (§4.3). With \(k\) iterations each covering \(d/k\), the gold reward becomes \(d(\alpha_{\mathrm{RL}}-\beta_{\mathrm{RL}}\log d+\beta_{\mathrm{RL}}\log k)\). The \(\alpha\) term, regressional Goodhart, is untouched; the gain is \(\beta_{\mathrm{RL}}\,d\,\ln k\). At fixed total distance, 2, 4 and 8 iterations are worth 0.69, 1.39 and 2.08 times \(\beta_{\mathrm{RL}}d\) (derived), and the authors state the law can hold only up to some maximum \(k\). This is different from the reference-policy resets of §4.2, where the reward stays fixed: here the proxy is replaced, so the extremal-Goodhart term is renewed.
5.6 Over-optimization outside the synthetic setup
With real human judgments the curve has the same shape. Stiennon et al. optimized against an earlier reward model at a range of KL coefficients; labelers preferred the policies under light optimization, then preference fell off relative to the reward model's prediction, and "eventually the reward model becomes anti-correlated with human preferences"3 (§4.3, Figure 5). Lambert's textbook describes the same shape with a train and test preference model, where gains on the training model stop transferring at roughly 150K training samples, and states that over-optimization is "fundamental and unavoidable" for a learned reward7 (§14.2). The mitigations it lists are larger policies, reward-model ensembles and changing the optimizer, and it adds that direct alignment algorithms are still prone to it.
Verifiable rewards remove the learned proxy but not the KL curve. In Tulu 3's RLVR runs on GSM8K and MATH, spending more KL did not necessarily raise the verifiable reward; lowering \(\beta\) raised the KL and typically lowered the average score over evaluations, with one exception; some higher-KL IFEval runs produced visibly over-optimized outputs; and a higher learning rate made the KL explode early18 (§6.2.1, Appendix B.4, §6.4). The final 70B run, at a lower learning rate, stayed well below 1. The operating point in these reports is chosen by held-out evaluation, not by the coefficient: Tulu 3 evaluated every 100 steps (40 for 70B) and took the 8B checkpoint with the best overall MATH and IFEval performance18 (§6.4). DeepSeek-R1 cites Gao et al. when explaining why it avoided model-based process rewards at scale13 (Supplement G.2). Module 19 covers the exploits that remain with rule-based rewards.
| Signal | What it indicates | Where it appears |
|---|---|---|
| KL to the reference, per token and per sequence | distance spent; a "very large" value "can indicate a potential bug or broken model"; early explosions followed a higher learning rate | 7 (§14.2); 18 (§6.4, footnote 18) |
| Proxy reward versus a held-out judge, RM or human preference | the shortfall between predicted and actual reward is the exploitation | 6 (§3.4); 3 (Figure 5) |
| Response length | very long answers when short ones would be preferred; a recurring extremal or causal exploit | 6 (footnote 10); 18 (Figure 19, length plotted beside reward and KL) |
| Average over held-out evaluations, not only the optimized metric | narrow gains with broad loss; more KL typically meant a lower average | 18 (§6.2.1, Figure 21) |
Where experts disagree
Two questions are tangled in the reports: whether the KL-to-reference term should be there at all, and, if it is, which estimator in which place. The RLHF-era papers answered the first with "yes" and the second by habit. The reasoning-RL era split on both. Module 13 treats the long-chain-of-thought case in depth; this section sets out the four positions the sources support.
Keep the term and make its gradient the reverse-KL gradient
Tang and Munos (Meta)9; Shah et al. (Mila, COLM 2026)14; Liu et al.15; the PPO-RLHF tradition14
The term exists to be a specific gradient, so the estimator is a gradient choice: \(k_1\) in the reward, \(k_2\) as a loss, or an importance-weighted \(k_3\). Differentiating \(k_3\) or \(k_1\) as a loss optimizes something else.
- The derivations are exact and checkable (Table 4); three independent groups agree on them.
- K1-in-reward beat K3-in-loss on every reported task in both models, and KL of the right kind stabilized asynchronous runs14.
- In the high-KL regime, such as distillation, the proper sequence-level gradient minimized KL fastest9.
- The evidence is 7B to 8B models, math tasks, 250 steps; at \(\beta=0.001\) Tang and Munos saw similar curves for the "incorrect" and correct losses.
- Gao et al. found the penalty equivalent to early stopping in their setup6, which weakens the case for caring about its exact form.
Keep it as GRPO ships it: \(k_3\) in the loss
DeepSeekMath and DeepSeek-R11213; TRL's implementation11; DeepSeek-V3.2 with an importance-ratio correction16
Keeping the KL out of the reward leaves the advantage computation untouched, and \(k_3\) is low-variance and non-negative. At coefficients of 0.001 to 0.04 it has trained models at scale.
Drop the term for long-chain-of-thought RL with verifiers
DAPO20; Olmo 321; GLM-522; TRL's default of 0.011
A reasoning policy is expected to diverge a lot from its initialization, so a restriction toward it is unnecessary20. Olmo 3 reports that removing it "allows less-restricted policy updates" and does not lead to over-optimization or destabilized training21. With a rule-based verifier there is no learned proxy to exploit.
- Frees the update and saves a policy-size reference model and one forward pass per rollout.
- Consistent with Gao et al.'s finding that the penalty did not improve the gold-versus-KL frontier6.
- The sources read give the decision, not an ablation: DAPO's ablation ladder (Table 1) has no KL row.
- It leans on the verifier not being hackable (Module 19), and V3.2 says KL strength is domain-dependent, not absent everywhere16.
Keep it, but reset the anchor
ProRL19; R1's 400-step replacement13; DeepSeekMath's iterative GRPO12
Starting from an SFT or distilled checkpoint rather than a raw base model, an explicit KL term holds entropy and stability better than other tricks, and when the term starts to dominate the loss the reference and optimizer are reset. ProRL says the "remove KL" advice mostly applies to base models before any supervised fine-tuning19.
- Keeps a restoring force while allowing unbounded cumulative progress.
- Resets also gave a point to retune hyperparameters and data in ProRL's long run.
- By §4.2 the leash becomes a trust region; it no longer limits distance from the SFT model.
- Evidence is a 1.5B model on one recipe, and the reset schedules (every 400 steps; on plateau) are heuristics.
The positions answer different questions, and the evidence supports a split rather than a winner. Whether: with a learned proxy (a reward model or a generative judge) keep a KL term, or an equivalent held-out early-stopping rule, because the over-optimization curve of §5 is real, appears with human judges as well, and the cited recipes pick their stopping point from held-out evaluation. With a rule-based verifier on a long-chain-of-thought run, removal is the convergent practice of three independent open recipes; I would still treat it as unablated, since none of the reports read isolates it. How: if you keep the term at a coefficient above roughly 0.01 or expect large KL (distillation, long runs), use a gradient-correct form (k1 in the reward, the k2 surrogate, or the importance-weighted k3). At 0.001, Tang and Munos suggest the choice barely matters; that is their conjecture from one experiment, not an established result. What would settle it: a matched ablation of {no KL, k3 in loss, k1 in reward, importance-weighted k3, k3 with resets} on one long-chain-of-thought recipe for thousands of steps, reporting held-out evaluations, entropy and KL; Shah et al.'s grid is the closest existing test and covers 250 steps on 7B to 8B math.
What is changing
Estimator correctness has gone from folklore to published bug since mid-2025: Tang and Munos (June 2025), V3.2's corrected \(k_3\) (December 2025) and Shah et al. (v4, August 2026) all postdate DeepSeekMath's original choice (February 2024)91614. TRL's default is still \(\beta=0\) with a \(k_3\) loss available11, so check the version you run. The scaling-law evidence is 2022-vintage: GPT-3-family policies, PPO, a 6B gold model6. The sources read contain no comparable gold-versus-proxy law for generative-judge or rubric rewards (Module 16), or for on-policy distillation, where the reverse KL is the objective itself and gradient correctness matters most (Module 21).
References
- paperZiegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., Irving, G. Fine-Tuning Language Models from Human Preferences. arXiv 1909.08593 v2, 8 Jan 2020. https://arxiv.org/abs/1909.08593. Accessed 2026-10-27. Used for: the KL-penalized reward and its stated purposes (§2, Eq. 2), the adaptive KL controller and its constants (§2.2), the no-KL gibberish samples and +8.0 mock-sentiment score (Table 10).
- paperKorbak, T., Perez, E., Buckley, C. L. RL with KL penalties is better viewed as Bayesian inference. arXiv 2205.11275 v2, 21 Oct 2022. https://arxiv.org/abs/2205.11275. Accessed 2026-10-27. Used for: distribution collapse of the plain RL objective (§2), the KL-regularised objective (§3, Eq. 3) and its reading as a Bayesian posterior (§4, Eq. 5).
- paperStiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P. Learning to summarize from human feedback. NeurIPS 2020; arXiv 2009.01325 v3, 15 Feb 2022. https://arxiv.org/pdf/2009.01325. Accessed 2026-10-27. Used for: the KL-penalized reward and its two purposes (§3.4), the 0.05 coefficient (Appendix B.1), over-optimization against the reward model judged by humans (§4.3, Figure 5), the best-of-N KL formula and the PPO-versus-BoN comparison (Appendix G.3).
- paperOuyang, L., Wu, J., Jiang, X., Almeida, D. et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: the per-token KL penalty and the PPO-ptx objective (§3.5, Eq. 2), \(\beta=0.02\) and no discounting (Appendix C.4), the pretraining mix versus a larger KL coefficient and the KL-model remark (§4.2, Appendix E.6), the optimal-coefficient sweep (Appendix E.7).
- paperRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023; arXiv v3, 29 Jul 2024. https://arxiv.org/pdf/2305.18290v3. Accessed 2026-10-26. Used for: the KL-constrained reward-maximization objective (§3, Eq. 3), the closed-form optimal policy and partition function (§4, Eq. 4) and their derivation (Appendix A.1).
- paperGao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023; arXiv v1, 19 Oct 2022. https://arxiv.org/pdf/2210.10760v1. Accessed 2026-10-26. Used for: the synthetic gold/proxy setup (§2), the functional forms (§1, Appendix B), scaling with RM size, RM data and policy size (§3.2-3.4), RL versus best-of-n (§3.5), the KL penalty as early stopping (§3.6), KL as a yardstick and the Goodhart mapping (§4.1-4.2), iterated RLHF (§4.3), hyperparameters (Appendix C).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: the reverse-KL penalty and its direction (§15.1), over-optimization, the train/test preference-model example, KL monitoring and mitigations (§14.2).
- blogSchulman, J. Approximating KL Divergence. Personal site, posted 2020-03-07. http://joschu.net/blog/kl-approx.html. Accessed 2026-10-26. Used for: the definitions of \(k_1\), \(k_2\), \(k_3\) and the reverse-direction estimator, the two numerical tables (true KL 0.005 and 0.5) and the experiment code.
- paperTang, Y., Munos, R. On a few pitfalls in KL divergence gradient estimation for RL. arXiv 2506.09477 v1, 11 Jun 2025. https://arxiv.org/abs/2506.09477. Accessed 2026-10-27. Used for: the expected gradients of the \(k_1\), \(k_2\) and \(k_3\) losses (§3, Eqs. 2-4), the sequence-level partial-gradient pitfall and cumulative estimator (§4), the low-KL and on-policy-distillation experiments (§5, Figures 7-8).
- paperGrattafiori, A., Dubey, A., Jauhri, A. et al. (Llama Team, AI @ Meta) The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: the 128K-token vocabulary (§3.2).
- officialHugging Face TRL maintainers TRL documentation: GRPO Trainer (main branch). docs/trl/main; version not stated on the page. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the per-token \(k_3\) estimator inside the loss, the default \(\beta=0.0\) with the reference model not loaded, the R1 value of 0.001, and the
sync_ref_modeldefault in the config signature. Quotes are WebFetch extractions of a 114,487-character page, of which the first 100,000 characters were read. - paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: KL in the reward (§4.1.1, Eq. 2), KL in the loss and the \(k_3\) estimator (§4.1.1, Eqs. 3-4), the 0.04 coefficient (§4.2), the reference reset in iterative GRPO (Algorithm 1).
- paperDeepSeek-AI DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026 (expanded version; Nature 645:633-638, 2025); v1, 22 Jan 2025. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the GRPO objective with the \(k_3\) KL term and group-standardized advantage (§2.1, Eqs. 1-3), \(\beta=0.001\) and the 400-step reference replacement (§2.1, §3.2.1), the reward-hacking argument against model-based process rewards citing Gao et al. (Supplement G.2).
- paperShah, V., Obando-Ceron, J., Jain, V. et al. A Comedy of Estimators: On KL Regularization in RL Training of LLMs. COLM 2026; arXiv 2512.21852 v4, 25 Aug 2026. https://arxiv.org/abs/2512.21852. Accessed 2026-10-27. Used for: the gradient bias of \(K_1\) and \(K_3\) in the reward and in the loss (§2, Table 1, Eq. 4), the on-policy grid on Qwen2.5-7B and Llama-3.1-8B-Instruct (§4.1, Figures 2-4), the asynchronous experiments (§4.2, Figure 5).
- paperLiu, K., Liu, J. K., Chen, M., Liu, Y. Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization. arXiv 2510.01555 v2, 6 Oct 2025. https://arxiv.org/abs/2510.01555. Accessed 2026-10-27. Used for: gradient equivalence of \(k_1\) in the reward and \(k_2\) as a loss, and \(k_3\) as a loss as a biased first-order approximation (Abstract, §1).
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.) DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the importance-ratio-corrected \(K_3\) estimator, the unbounded-gradient argument and the domain-dependent KL strength (§3.1, Eq. 7).
- paperZhang, Y., Liu, Y., Yuan, H., Yuan, Y., Gu, Q., Yao, A. C.-C. On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning. ICLR 2026; arXiv 2505.17508 v4, 19 Feb 2026. https://arxiv.org/abs/2505.17508. Accessed 2026-10-27. Used for: the off-policy importance-weighting mismatch in GRPO's KL term (Abstract). Only the abstract and introduction were read.
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the RLVR objective and reward of 10 (§6, Eqs. 7-8), the \(\beta\) sweeps and over-optimization findings (§6.2.1, Figures 19-21), Table 21 coefficients, final-run selection and the low 70B KL (§6.4), IFEval over-optimized outputs (Appendix B.4).
- paperLiu, M., Diao, S., Lu, X., Hu, J. et al. (NVIDIA) ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: the KL penalty, its dominance of the loss over time, the reference-policy and optimizer reset, and the argument that KL removal applies to raw base models (§2.3.1; Appendix E).
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: the removal of the KL term for long-CoT reasoning (§2.3) and the ablation ladder (Table 1).
- paperTeam Olmo (Ettinger, A., Bertsch, A. et al.) Olmo 3. arXiv v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: OlmoRL's omission of the KL loss and its stated reason (§4.4.1).
- paperGLM-5 Team (Zeng, A., Lv, X. et al.) GLM-5: from Vibe Coding to Agentic Engineering. arXiv v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the removal of the KL regularization term from the IcePop-based reasoning-RL loss (§3.2).