Reinforcement Learning for LLM Post-Training · Module 7
Direct Alignment: Deriving DPO, the Implicit Reward, Its Failure Modes, and the KTO, Length-Normalized and NLL-Regularized Variants
Direct Preference Optimization replaces the reward model and the PPO loop with one classification loss over preference pairs. This module derives that loss from the KL-regularized objective of Module 6, reads its gradient and its implicit reward, and catalogues what goes wrong in practice (unconstrained probability mass, falling chosen likelihood, length drift, over-optimization) together with the fixes Llama 3, Tulu 3 and Olmo 3 shipped. Afterwards you can derive and implement DPO, pick among its variants, and say precisely which evidence supports and which contradicts "DPO is enough".
1. What DPO removes from the RLHF pipeline
1.1 Three stages become one loss
The classical recipe you met in Modules 4 and 5 has three stages: supervised fine-tuning to get \(\pi_{\mathrm{SFT}}\), a reward model fit to preference pairs with a Bradley-Terry loss, and an RL stage that maximizes the learned reward minus a KL penalty to the SFT model1 (§3). DPO keeps the first stage and the pairs, and replaces the other two with one loss computed on stored responses. The abstract states the consequence directly: no sampling from the language model during fine-tuning and no significant hyperparameter tuning1.
The cost difference is measured. In Tulu 3's controlled comparison PPO took about 28 hours on two nodes and DPO about 4 hours on one, with roughly equal average scores2 (§5.4.1); Module 5 §9.1 converts this to 14 times the node-hours. Tulu 3's production DPO runs took 10 hours on 8 H100 GPUs (8B) and 19 hours on 64 H100 GPUs (70B)2, which is 80 and 1,216 H100-hours (derived).
What remains is a training loop of supervised-learning shape: per pair, the policy runs forward and backward on two sequences and the frozen reference runs forward on the same two. Tulu 3 takes the reference out of GPU memory by precomputing its log-probabilities once over the dataset, and runs chosen and rejected sequences in separate forward passes instead of one concatenated batch; the two changes gave near-identical training losses and lower peak memory2 (§5.4.2, Figure 17). Lambert reports that caching alone cuts peak memory by 50%3 (§8.3). The reference costs one extra forward pass over the dataset, not a standing second copy of the weights.
1.2 What the loss still needs, and what it never sees
Three things survive from the RLHF recipe. The pairs \((x, y_w, y_l)\) are still required, and they should come from or near the reference policy: the paper sets \(\pi_{\mathrm{ref}}=\pi_{\mathrm{SFT}}\) when the pairs were sampled from the SFT model, and otherwise fits \(\pi_{\mathrm{ref}}\) by maximum likelihood on the chosen responses to reduce the distribution shift1 (§4). The reference policy is still the anchor of the KL term. And \(\beta\) is still the price of moving away from it (Module 6, Section 4.1); in DPO it is the scale inside the sigmoid, not a penalty added to a reward.
What DPO never sees is any response the policy generates itself during training. Every probability the loss touches belongs to a response that was written down before training began. Sections 4 and 6 show that this one fact explains most of the method's failure modes and most of the evidence for online RL.
2. Deriving the loss
2.1 Five equations, in order
The derivation is a change of variables from rewards to policies. Start from the objective of Module 6, \(\max_\pi \mathbb E_{y\sim\pi}[r(x,y)]-\beta\,\mathrm{KL}(\pi\|\pi_{\mathrm{ref}})\) (Eq. 3 in the paper)1. Its maximizer, derived in Module 6 Section 1.2 and in Appendix A.1 of the paper, is \(\pi_r(y|x)=\pi_{\mathrm{ref}}(y|x)\,e^{r(x,y)/\beta}/Z(x)\) (Eq. 4), with \(Z(x)=\sum_y\pi_{\mathrm{ref}}(y|x)e^{r(x,y)/\beta}\). Taking logarithms and solving for the reward gives Eq. 5:
$$ r(x,y)=\beta\log\frac{\pi_r(y|x)}{\pi_{\mathrm{ref}}(y|x)}+\beta\log Z(x). $$Eq. 5 is useless for a reward model by itself, because \(Z(x)\) sums over every possible response. The step that makes it usable is that Bradley-Terry depends only on the difference of two rewards for the same prompt (Module 4, Section 2.3). Substituting Eq. 5 into \(p^*(y_1\succ y_2|x)=\sigma(r^*(x,y_1)-r^*(x,y_2))\), the two \(\beta\log Z(x)\) terms are identical and cancel (Eq. 6; Appendix A.2)1. What is left is a preference probability written entirely in policies. Maximum likelihood over a parametrized policy \(\pi_\theta\) in place of the unknown optimal one gives Eq. 7, the DPO loss:
$$ \mathcal L_{\mathrm{DPO}}(\pi_\theta;\pi_{\mathrm{ref}})=-\mathbb E_{(x,y_w,y_l)\sim\mathcal D}\Big[\log\sigma\Big(\beta\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)}-\beta\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\Big)\Big]. $$- Objective. The KL-regularized reward maximization every RLHF recipe uses, for any reward function.
- Optimum. The maximizer is the reference policy exponentially tilted by the reward.
- Invert. Solve the tilt for the reward: a scaled log-ratio plus a prompt-only normalizer.
- Cancel. Substituted into Bradley-Terry, the normalizer drops out because both responses share the prompt.
- Fit. Maximize the likelihood of the observed preferences over the policy itself.
Figure 2. The derivation chain, one equation per step. The numbering is the paper's1 (§4, Appendix A.1-A.2).
2.2 Why cancelling \(Z\) loses nothing
Two reward functions that differ by \(f(x)\) are equivalent: they induce the same Bradley-Terry preference distribution (Lemma 1) and the same optimal policy (Lemma 2). Theorem 1 then states that every equivalence class contains exactly one reward of the form \(\beta\log\frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)}\), namely the one whose normalizer satisfies \(\sum_y\pi_{\mathrm{ref}}(y|x)e^{r(x,y)/\beta}=1\)1 (§5.1, Appendix A.5-A.6). The reparameterization therefore does not restrict which preference distributions can be represented. It selects one representative per class, and because \(\pi_\theta\) is a normalized distribution, the constraint that picks the representative holds automatically.
The guarantee has conditions: a Bradley-Terry data-generating process, a reference policy with \(\pi_{\mathrm{ref}}(y|x)>0\) everywhere, and a policy class rich enough to hit the optimum1 (§5.1, Theorem 1); the paper states consistency only "under suitable assumption of the preference data distribution"1 (§4). A finite offline dataset covers a vanishing fraction of the response space, which is where Section 4 begins.
2.3 The implicit reward is a sum of token log-ratios
The paper's title claim is that the language model is secretly a reward model: \(\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}\)1. By the chain rule this is \(\beta\sum_t\log\frac{\pi_\theta(y_t|x,y_{<t})}{\pi_{\mathrm{ref}}(y_t|x,y_{<t})}\) (derived), a sum over tokens of the same log-ratio that is the k1 sample of \(\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\) in Module 6, here evaluated on a stored response rather than a sampled one. Lambert stresses the reading: DPO is, at its core, learning a reward model, and the policy is how that reward is parametrized3 (§8.1.1). The reference implementation returns these detached implicit rewards for logging1 (Appendix B), and the fraction of pairs with \(\hat r_w>\hat r_l\) is the usual "reward accuracy" curve.
2.4 The gradient: a per-pair weight on a contrast
Differentiating Eq. 7 (Appendix A.4)1 gives
$$ \nabla_\theta\mathcal L_{\mathrm{DPO}}=-\beta\,\mathbb E\Big[\underbrace{\sigma\big(\hat r_\theta(x,y_l)-\hat r_\theta(x,y_w)\big)}_{\text{larger when the implicit ordering is wrong}}\big(\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)\big)\Big]. $$The structure is the one Module 4 derived for the Bradley-Terry reward-model gradient, with \(\nabla\hat r_\theta=\beta\nabla\log\pi_\theta\) in place of \(\nabla r_\theta\). Each pair raises the likelihood of \(y_w\) and lowers that of \(y_l\), scaled by \(\beta\) and by a weight in \((0,1)\) that is 0.5 at a tie. The paper reports that the weight matters: an unweighted likelihood-up, likelihood-down baseline produced meaningless text on summarization1 (Appendix C.3). With \(\Delta=\log\frac{\pi_\theta(y_w)}{\pi_{\mathrm{ref}}(y_w)}-\log\frac{\pi_\theta(y_l)}{\pi_{\mathrm{ref}}(y_l)}\) in nats, the weight is \(\sigma(-\beta\Delta)\), and \(\beta\) sets how many nats of log-ratio gap a pair can absorb before it stops teaching.
Worked pair (derived). Take \(\beta=0.1\), a chosen response whose log-ratio is \(+2.0\) nats and a rejected one at \(-3.0\) nats. Then \(\Delta=5.0\), the margin inside the sigmoid is \(\beta\Delta=0.5\), the loss is \(-\log\sigma(0.5)=0.474\), and the weight is \(\sigma(-0.5)=0.378\). The step on \(\nabla\log\pi_\theta(y_w)\) is \(\beta\cdot0.378=0.0378\) times the learning rate. The weight falls to 0.1 when \(\beta\Delta=\ln 9=2.197\), i.e. at \(\Delta=22.0\) nats for \(\beta=0.1\) and \(\Delta=0.44\) nats for \(\beta=5\).
2.5 The loss as code
The implementation is short enough that the code is the clearest statement of what is computed. This version follows the paper's Appendix B1, with a token mask (Section 4.4) and the optional NLL term of Section 4.2 added, and with the reference log-probabilities passed in as cached values.
import torch
import torch.nn.functional as F
def seq_logp(logits, labels, mask):
# logits (B,T,V); labels, mask (B,T). mask is 1 on response tokens that count in the loss
lp = torch.log_softmax(logits.float(), dim=-1).gather(-1, labels.unsqueeze(-1)).squeeze(-1)
return (lp * mask).sum(-1) # log pi(y|x), summed over tokens
def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1, n_w=None, alpha=0.0):
# pi_*: (B,) policy sequence log-probs; ref_*: (B,) cached reference log-probs
delta = (pi_w - ref_w) - (pi_l - ref_l) # log-ratio difference, nats
loss = -F.logsigmoid(beta * delta) # Eq. 7; gradient weight is sigmoid(-beta * delta)
if alpha > 0: # NLL on the chosen sequence (Section 4.2)
loss = loss + alpha * (-pi_w / n_w) # n_w = number of unmasked chosen tokens
r_w = (beta * (pi_w - ref_w)).detach() # implicit rewards, for logging
r_l = (beta * (pi_l - ref_l)).detach()
return loss.mean(), r_w, r_l
Two details are easy to get wrong. The log-probabilities are sums over response tokens, not means, unless you are deliberately running the length-normalized variant (Section 3.2). And the reference pass must use the same tokenization, template and mask as the policy pass, or every \(\Delta\) shifts by a constant the loss will then try to undo.
3. What you set: β, learning rate, the reference model and the data
3.1 Reported settings span two orders of magnitude in learning rate
The original paper's defaults are \(\beta=0.1\), batch size 64, RMSprop and a learning rate of \(10^{-6}\) with a 150-step linear warmup, with \(\beta=0.5\) for TL;DR summarization1 (Appendix B). Lambert notes that the method only worked for the community after it found the right data and "surprisingly low learning rates"3 (§8). Table 1 collects what the primary reports used since. Two patterns: the learning rate falls by more than two orders of magnitude from Llama 3 to Olmo 3 Think, and \(\beta\) jumps from 0.1 to 5 exactly when the loss changes from summed to length-normalized log-ratios (Section 3.2).
| Recipe | Loss form | β | Learning rate | Batch | Other settings |
|---|---|---|---|---|---|
| DPO paper, models ≤ 6B1 (App. B) | sum | 0.1 (TL;DR 0.5) | 1e-6, RMSprop | 64 | 150-step warmup |
| Llama 3, up to 405B4 (§4.1.4) | sum + NLL | 0.1 | 1e-5 | n/s | NLL coefficient 0.2 on chosen; formatting tokens masked; six rounds |
| Tulu 3 8B2 (Table 20) | norm | 5 | 5e-7 | 128 | 1 epoch, warmup ratio 0.1, max length 2,048 |
| Tulu 3 70B2 (Table 20) | norm | 5 | 2e-7 | 128 | as 8B; 19 h on 64 H100 |
| Olmo 3 Think 7B5 (Table 48) | norm | 5 | 8e-8 | 128 | 150K pairs, 1 epoch, max length 16K, 32 GPUs |
| Olmo 3 Think 32B5 (Table 48) | norm | 5 | 7e-8 | 128 | 200K pairs, 1 epoch, max length 8K, 64-128 GPUs |
| Olmo 3 Instruct 7B5 (Table 48) | norm | 5 | 1e-6 | 128 | 260K pairs, 1 epoch, max length 16K, 16 GPUs |
| Iterative RPO (GSM8K)6 (§2, §3.1) | sum + NLL | 0.1 | 7e-7, AdamW | 16 | NLL coefficient 1; about 55-60k pairs per iteration; four iterations |
3.2 What β means once log-ratios are divided by length
The length-normalized loss (Tulu 3 Eq. 62; the Olmo 3 appendix writes the same form5, A.6.2) replaces each log-ratio by its per-token mean:
$$ \mathcal L_{\mathrm{DPO\text{-}norm}}=-\mathbb E\Big[\log\sigma\Big(\tfrac{\beta}{|y_w|}\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)}-\tfrac{\beta}{|y_l|}\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\Big)\Big]. $$Two consequences follow (derived). First, \(\beta\) is only comparable across loss forms at a given length: \(\beta_{\mathrm{norm}}\) acts like a summed-form \(\beta_{\mathrm{sum}}=\beta_{\mathrm{norm}}/L\), so \(\beta_{\mathrm{norm}}=5\) equals the paper's 0.1 at \(L=50\) tokens. As an illustration only, Llama 3's preference data average 284.0 response tokens4 (Table 6), which puts \(\beta_{\mathrm{norm}}=5\) at \(5/284=0.018\) on the summed scale; a pair then saturates (weight 0.1) at a mean per-token gap of \(\ln 9/5=0.44\) nats, about 125 nats of summed gap, against 22 nats at \(\beta_{\mathrm{sum}}=0.1\). Second, the per-token gradient on each response is \(\beta/|y|\) times the pair weight, so a rejected response twice as long as the chosen one is pushed down half as hard per token; in the summed form every token gets \(\beta\) times the weight regardless of length, and longer responses absorb more total gradient. Tulu 3 states the intent as mitigating the length bias common in human and model preferences2 (§5.1).
3.3 The reference model and the SFT stage
DPO keeps the reference fixed at the SFT checkpoint, which makes it the anchor of the implicit KL (Module 6, Section 4.2 lists the alternatives). The evidence that this stage matters comes from the KTO paper: SFT "is always needed for best results with DPO", and DPO run without SFT on a base model rambles and hallucinates whole conversations, while KTO at 13B and 30B does not need SFT7 (§1, §4.3, Figure 4). When training is iterated, the reference moves: Iterative RPO uses the previous iteration's model \(M_t\) as the reference for iteration \(t+1\)6 (§2), and Xu et al.'s DPO-Iter sets the reference to the latest DPO model each round8 (§4.3). With a cached reference pass (Section 1.1) that costs one forward pass over the new data per round.
3.4 Pair construction: strength filters and synthetic judges
Llama 3's annotators rate preference strength on four levels (significantly better, better, slightly better, marginally better). For both reward modeling and DPO the team trains on pairs labeled significantly better or better and discards pairs with similar responses; reward modeling uses all preference data available, while DPO uses only the most recent batches4 (§4.2.1). Tulu 3 builds 354,192 preference instances with an UltraFeedback-style pipeline: four responses per prompt from a pool of 22 models plus on-policy samples from the Tulu SFT model, rated 1 to 5 by GPT-4o-2024-08-06 on helpfulness, instruction following, truthfulness and honesty, then binarized2 (§1, §5.2.1, Figure 7). Ivison et al. rank the factors: preference data quality gave the largest improvements, then the learning algorithm, then a better reward model, then extra unlabeled prompts9 (abstract). Section 4 turns to what the loss itself does not control.
4. Failure modes, in the order they bite
The failures below share one cause: the loss sees only differences of log-ratios on stored responses, so anything it cannot see is left to the network's inductive bias. They are listed roughly by how early a practitioner meets them.
4.1 The loss is flat along directions it cannot see
For one prompt with stored responses \(y_1\succ y_2\) and \(\pi_{\mathrm{ref}}(y_1)=\pi_{\mathrm{ref}}(y_2)\), Eq. 7 reduces to \(\log\big(1+(\pi(y_2)/\pi(y_1))^{\beta}\big)\) (derived): any change that keeps the ratio fixed leaves the loss untouched, whatever it does to every other response. Rafailov et al. give the general form. Written as logistic regression on vectors \(q_i=\mathbb 1_{(x,y_w)}-\mathbb 1_{(x,y_l)}\) over the prompt-response space, the data matrix has one row per pair and one column per possible (prompt, response), so its null space is enormous, and infinitely many minimizers put weight on responses absent from the data (Proposition 1, adapted from Hejna et al.)10 (§4). Their minimal example, two conflicting pairs \(y_1\succ y_2\) and \(y_2\succ y_1\) plus an unseen \(y_3\), is minimized by both \((0.5,0.5,0)\) and \((0,0,1)\). Xu et al. give a one-pair version with \(\pi_{\mathrm{ref}}=(0.5,0.5,0)\), where \(\pi_{\mathrm{DPO}}=(0.1,0.0,0.9)\) attains the minimum but a PPO policy cannot leave the reference's support8 (Table 1, §4.1).
RLHF does not have this hole because its KL term is evaluated on responses sampled from the policy, so mass that moves to \(y_3\) is charged for; PPO can also use prompt-only data, and the KL regularizes those samples8 (§4.1). What keeps the drift small in practice is a small learning rate, a preference set covering diverse domains, and the robustness of large models to new data, as SimPO's authors argue for a method with no reference at all11 (§2.3). Where those fail, mass leaks: in Rafailov et al.'s tree-structured toy MDP with one pair, DPO, IPO and SLiC all put substantial mass on out-of-distribution trajectories while in-distribution probability falls10 (§4, Figure 7).
4.2 Likelihood displacement: the chosen response gets less likely
The most-cited symptom is that \(\log\pi_\theta(y_w|x)\) decreases during DPO training while the margin grows. Pal et al. prove that the DPO loss can reduce the preferred response's likelihood as long as the preferred-to-dispreferred ratio increases, and show it empirically, especially when completions differ in few tokens12 (abstract, §3): a shared prefix contributes nothing to \(\nabla\log\pi(y_w)-\nabla\log\pi(y_l)\), the differing token is pushed up, and the tokens after it can be pushed down (Figure 1). Tajwar et al. see the chosen response's induced reward fall as prompts grow relative to model capacity: Pythia-1.4B on AlpacaFarm lowers both responses, while Mistral-7B on UltraFeedback raises the chosen one, which they attribute to more distinct pairs13 (§5.2.2, Figures 16-17). Razin et al. name it likelihood displacement, tie it to preferences with similar hidden embeddings (a centered hidden embedding similarity, CHES, score), and show it can be harmful: mass moves to responses with a different meaning, and refusal training lowered Llama-3-8B-Instruct's refusal rate from 74.4% to 33.4%14 (abstract).
Figure 5 reproduces the mechanism in a model small enough to inspect. Three responses have feature vectors \(\phi_w=(1,0)\), \(\phi_l=(0.8,0.3)\), \(\phi_o=(0.4,-1)\) and logits \(z_y=\theta\cdot\phi_y\), starting from \(\theta=0\) (uniform). The DPO gradient moves \(\theta\) along \(\phi_w-\phi_l\), which raises the logit of the "other" response \(o\) faster than that of \(y_w\), so \(o\) absorbs mass even though the margin grows.
Three fixes have shipped. An NLL term on the chosen sequence. Llama 3 adds one with coefficient 0.2; the report says the term "helps further stabilize DPO training by maintaining desired formatting for generation and preventing the decrease of log probability of chosen responses"4 (§4.1.4). Pang et al. use coefficient 1 with the term normalized by response length, and measure its value on GSM8K from Llama-2-70B-Chat: DPO on the iteration-1 pairs reaches 61.8%, DPO plus NLL after one iteration reaches 73.1% (Table 1), and the chosen log-probability falls under plain DPO but rises with the NLL term (Figure 3)6 (§3.1). Their Figure 2 also shows why SFT on the chosen responses alone is not a substitute: it raises the rejected responses' probability along with the chosen ones. A hinge on the chosen log-ratio. DPO-Positive adds \(-\lambda\max\!\big(0,\log\frac{\pi_{\mathrm{ref}}(y_w)}{\pi_\theta(y_w)}\big)\) inside the sigmoid, which is zero until the chosen likelihood drops below the reference; the paper uses \(\beta=0.3,\ \lambda=50\)12 (§4, §5). Data filtering. Razin et al. drop the pairs with the highest CHES scores and report that this mitigated the unalignment in their experiments14 (abstract).
Whether the falling likelihood is a defect at all is disputed. If \(\pi_{\mathrm{ref}}\) was fit by SFT on the chosen responses, the expected implicit reward of the chosen response is approximately \(-\mathrm{KL}(\pi_{\mathrm{ref}}\|\pi_\theta)\), which is non-positive by construction, and Rafailov et al. find win rate rising while it falls, until the over-optimization hump of Section 4.610 (§3.5, Eq. 7, Figure 5). The disagreement section returns to this.
4.3 Length and verbosity
The DPO paper's own judge shows the hazard: with the simple prompt GPT-4 preferred longer, more repetitive summaries than humans did, so the headline numbers use a prompt that also asks for concision1 (§6.4). Rafailov et al. relay that DPO-trained models grow longer during training without a matching rise in ground-truth win rate10 (§2, §3.3, citing Park et al.), and the KTO authors see response length climb sharply when DPO is run without SFT7 (Figure 4). SimPO adds a diagnosis: the DPO reward and the likelihood that decoding uses can disagree. On UltraFeedback training pairs with \(\hat r_w>\hat r_l\), about half have a lower average log-likelihood for the chosen response, and the Spearman correlation between average log-likelihood and length is 0.82 for SimPO without length normalization, 0.59 for DPO and 0.34 for SimPO11 (§2.2, §4.4, Table 6). Applied to DPO itself, length normalization raised AlpacaEval 2 length-controlled win rate from 15.1% to 21.0% on Mistral-Base, where the data have strong length bias, and lowered it from 26.8% to 21.7% on Mistral-Instruct, where winners and losers have similar lengths11 (Appendix I, Table 15).
4.4 Formatting tokens and tail repetition
Llama 3 masks header and termination tokens out of the DPO loss for both chosen and rejected responses, because letting them contribute produced tail repetition and abruptly generated termination tokens4 (§4.1.4). The team's hypothesis is that common tokens in both responses give a conflicting objective, since the model must raise and lower their likelihood at once. The gradient makes this concrete (derived): a token with the same prefix in both responses cancels exactly in \(\nabla\log\pi(y_w)-\nabla\log\pi(y_l)\), but an end-of-turn token follows two different prefixes, so it is pushed up in \(y_w\) and down in \(y_l\) with nothing to cancel it. Masking the formatting tokens, as in the mask argument of Section 2.5, removes that term.
4.5 Deterministic preferences and the case for IPO
Azar et al. show that when a pair is always ordered the same way, Bradley-Terry requires the reward gap to diverge and the optimal policy gives the loser zero probability for any \(\beta\); with finite data an empirical preference of 1 can arise even when the true one is 0.8. RLHF is protected in practice because its reward model is under-fit, which regularizes the policy; DPO skips the reward model and loses that protection15 (§4.2). IPO regresses the gap instead of maximizing a likelihood:
$$ \mathcal L_{\mathrm{IPO}}=\mathbb E\Big[\Big(\log\tfrac{\pi_\theta(y_w)\,\pi_{\mathrm{ref}}(y_l)}{\pi_\theta(y_l)\,\pi_{\mathrm{ref}}(y_w)}-\tfrac{1}{2\tau}\Big)^{2}\Big], $$pulling the log-ratio gap to a finite target \(1/(2\tau)\) (Eq. 17)15; with \(\tau=0.1\) the target is 5 nats. The support is tabular, small bandit examples with two or three actions, not language models15 (§5.3-§5.4). At LLM scale the evidence is mixed: Rafailov et al. find IPO less prone to over-optimization, with the 6.9B model showing little or none10 (§3.1), while in SimPO's tuned comparison IPO trails DPO's AlpacaEval 2 length-controlled win rate in all four settings (11.8% against 15.1% on Mistral-Base)11 (Table 4). Lambert lists conservative DPO (a label-noise assumption) with IPO as overfitting remedies3 (§8.2).
4.6 Over-optimization and a narrow stable region
DPO has its own version of reward over-optimization (Module 6, Section 5). Rafailov et al. trained Pythia 1B, 2.8B and 6.9B on Reddit TL;DR with DPO, IPO and SLiC at seven values of \(\beta\), scoring by GPT-4 win rate against the dataset summary. Every objective shows a hump: win rate rises and then falls as the KL budget grows, and the Gao et al. functional form \(R(d)=d(\alpha-\beta_{\mathrm{fit}}\log d)\) in \(d=\sqrt{\mathrm{KL}}\) fits the points10 (§3.1-§3.2, Figure 1). The 1B model over-optimizes almost at once, and in the intra-epoch curves the best result arrives within the first 25% of an epoch for most settings, after which win rate declines while KL keeps rising (Figure 2).
Tulu 3's hyperparameter table is the practitioner's view of the same hump. Figure 6 plots every row: one algorithm, length-normalized DPO, spans 46.8 to 57.3 across six settings, 10.5 points, and only two settings (56.1 and 57.3) beat the SFT checkpoint's 55.7. Moving from three epochs to one at \(\beta=5\) was worth 3.9 points (53.4 to 57.3)2 (§5.4.1, Table 18).
5. Variants: what each one changes
5.1 One table, with the same benchmark where it exists
Most variants change one of four things: the form of the loss on the log-ratio gap, whether a reference model is needed, how length enters, or what the data must look like. Table 2 lines them up. The two numeric columns come from the one paper that tuned DPO, IPO, KTO, ORPO and SimPO on the same data and judge (SimPO's Table 4). They are chat-benchmark numbers (AlpacaEval 2, length-controlled win rate) and say nothing about reasoning or safety.
| Method | What changes | Data | Reference | Length | LC Mistral | LC Llama-3 |
|---|---|---|---|---|---|---|
| SFT start | none | demonstrations | n/a | n/a | 8.4 | 6.2 |
| DPO | logistic loss on summed log-ratio gap | pairs | yes (cacheable) | none; sums grow with length | 15.1 | 18.2 |
| DPO, length-normalized | divide each log-ratio by \(|y|\) | pairs | yes | per-token mean | 21.0 | n/r |
| R-DPO | adds a length term \(\alpha(|y_w|-|y_l|)\) to the margin | pairs | yes | explicit penalty | 17.4 | 17.6 |
| DPO + NLL | adds \(\alpha\cdot\mathrm{NLL}(y_w)\) | pairs | yes | none | n/r | n/r |
| DPOP | penalizes chosen log-ratio below 0 (\(\lambda=50\)) | pairs | yes | none | n/r | n/r |
| IPO | squared loss to gap \(1/(2\tau)\) | pairs | yes | none | 11.8 | 14.4 |
| KTO | sigmoid utility of the reward against a KL reference point | binary labels | yes | none | 13.1 | 14.2 |
| ORPO | SFT loss + log-odds-ratio term on per-token mean probabilities | pairs | no | per-token mean | 14.7 | 12.2 |
| SimPO | mean log-prob as reward, margin \(\gamma\) | pairs | no | per-token mean | 21.5 | 22.0 |
The loss changes that act on the gradient weight (IPO, DPOP, the NLL term) target the failure modes of Section 4 and are mostly evaluated on those, not on chat win rates; the changes that act on the reward definition (length normalization, SimPO's reference-free reward, ORPO) are the ones that move AlpacaEval 2; and KTO changes the data contract.
5.2 KTO: learning from a binary signal
KTO starts from a different question: does the feedback have to be a pair? Ethayarajh et al. recast alignment losses as human-aware losses (HALOs), prove that DPO and PPO-Clip belong to the family, and derive a member from the Kahneman-Tversky value function7 (§3, Theorem 3.5, §4.1). With \(r_\theta=\log\frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}\) and a reference point \(z_0=\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\), the loss is \(\mathbb E[\lambda_y-v(x,y)]\) with
$$ v(x,y)=\begin{cases}\lambda_D\,\sigma\big(\beta(r_\theta-z_0)\big)&y\text{ desirable}\\ \lambda_U\,\sigma\big(\beta(z_0-r_\theta)\big)&y\text{ undesirable.}\end{cases} $$Every example is a single (prompt, response, label) triple, so n preference pairs become 2n examples, and a 1:10 imbalance of desirable to undesirable examples is handled by setting \(\lambda_D n_D/\lambda_U n_U\in[1,3/2]\). Because sampling from \(\pi_\theta\) is slow, \(z_0\) is estimated from mismatched (prompt, response) pairs within a microbatch of at least two, clamped at zero, with no gradient through it7 (§4.1-§4.2). The recommended start is \(\beta=0.1\), batch size 8 to 128 and a learning rate of \(5\times10^{-6}\) with AdamW, 2 to 10 times DPO's, because the reference-adjusted reward is smaller7 (§4.2).
The reported results are strong on the authors' benchmarks: SFT+KTO matches SFT+DPO from 1B to 30B on GPT-4-judged win rate, matches DPO with up to 90% fewer desirable examples, and on Zephyr-β-SFT with UltraFeedback raises GSM8K from 40.0 to 53.5 and BBH from 44.1 to 52.6 over DPO (Table 2)7 (§4.3). The authors' explanations: an example's gradient vanishes as its implied reward goes to either extreme (Proposition 4.1), so mislabeled and intransitive feedback is effectively ignored, and under contradictory preferences DPO can favor the minority answer when the reference is unaligned (Theorem 4.3). With low-noise, transitive preferences they expect DPO to do better, since KTO risks underfitting7 (§4.4-§4.5). The v5 revision of 8 September 2026 changed proofs and wording, not the empirical results7 (Acknowledgements).
5.3 Reference-free variants, and what independent comparisons found
SimPO defines the reward as the length-normalized average log-probability, \(r(x,y)=\frac{\beta}{|y|}\log\pi_\theta(y|x)\), and trains \(-\log\sigma(r_w-r_l-\gamma)\) with a target margin \(\gamma\) and no reference model11 (§2.2-§2.3, Eq. 6), with tuned \(\beta\in\{2,2.5\}\) and \(\gamma\) between 0.3 and 1.6 (Table 8). It reports gains over DPO of up to 6.4 points on AlpacaEval 2 and 7.5 on Arena-Hard, notes that a higher learning rate lifted chat scores at the cost of forgetting on GSM8K and MMLU, and shows every baseline other than R-DPO trailing DPO's length-controlled win rate in all four settings (abstract, Table 4, Appendix J). ORPO folds SFT and an odds-ratio term into one reference-free stage, \(\mathcal L=\mathcal L_{\mathrm{SFT}}+\lambda\,\mathcal L_{\mathrm{OR}}\) with \(\mathcal L_{\mathrm{OR}}=-\log\sigma\big(\log\frac{\mathrm{odds}_\theta(y_w)}{\mathrm{odds}_\theta(y_l)}\big)\)16 (§4.1-§4.2), and reports 11.33% and 12.20% on AlpacaEval 2.0 for two Mistral-7B models after one epoch on UltraFeedback (§5-§6.1).
Independent runs are less generous. Tulu 3 tried SimPO at two settings and got 51.8 and 52.9 against an SFT base of 55.7, and found that only length-normalized DPO beat the base overall2 (§5.4.1, Table 18). The KTO paper's own comparison puts ORPO (\(\lambda=0.1\)) below the SFT starting point on GSM8K, 36.5 against 39.07 (Table 2). Lambert's reading is that the algorithm is a second-order choice: "the choice of algorithm is far less important than the initial model and the data used"3 (§8.2). What survives these sources: start with DPO; use the length-normalized form when chosen responses are systematically longer; add the NLL term when chosen likelihood falls or the task is reasoning-like; reach for KTO when labels are binary; treat SimPO and ORPO as methods that need their own hyperparameter search, not drop-in upgrades.
6. Offline versus on-policy preference data
6.1 When the data stop describing the policy
DPO's pairs are written before training starts, by the SFT model or a pool of other models; as the policy moves, they describe a distribution it no longer samples from. Xu et al. isolate the effect on SafeRLHF with Llama-2-7B trained on Alpaca: DPO lifts the safe-response rate from 41.6% to only 55.4% where PPO reaches 99.5%; SFT on the safe responses first, so the reference sits closer to the data, lifts DPO to 71.8%; and regenerating responses, labeling them with a learned reward model, and repeating with the latest DPO model as reference ("DPO-Iter") reaches 86.7% after one round and 99.9% after three8 (§4.3, Table 2). Table 3 gives all rows, including the cost: DPO-Iter's helpfulness change stays near −3 while PPO's is +1.69.
| Setting | Δ helpfulness | Δ harmfulness | Safe rate (%) |
|---|---|---|---|
| SFT (Alpaca) | −2.62 | 1.50 | 41.6 |
| PPO | 1.69 | −12.08 | 99.5 |
| DPO | −4.19 | −0.97 | 55.4 |
| DPO, + SFT (Safe) as reference | −1.62 | −3.50 | 71.8 |
| DPO, + filter dual-unsafe pairs | 2.46 | −4.88 | 80.8 |
| DPO, + filter dual-safe pairs | −2.86 | −6.82 | 95.8 |
| DPO-Iter, round 1 | −3.22 | −5.23 | 86.7 |
| DPO-Iter, round 2 | −3.27 | −8.83 | 99.7 |
| DPO-Iter, round 3 | −3.26 | −10.21 | 99.9 |
| DPO-Iter, round 4 | −2.96 | −11.07 | 99.9 |
Two further studies agree. Tajwar et al. find across bandit, synthetic and full-scale problems that on-policy sampling and a negative gradient (pushing probability down on bad responses, as DPO's contrastive term does) each beat offline maximum likelihood and are complementary; on-policy DPO and IPO converge faster and to better solutions than offline DPO and on-policy RL in their synthetic length tasks, though they stop short of saying on-policy DPO always beats PPO13 (abstract, §5.1-§5.3). The benefit concentrates where the reward peak lies in a low-probability region of the reference; where it lies in a high-probability region, offline supervised methods can suffice. Tang et al. measure at equal KL budget in a Gao-style gold-reward setup (Module 6) and find online algorithms a Pareto improvement over offline ones, with the gap surviving contrastive and non-contrastive losses and not closed by scaling the policy; offline policies were better pairwise classifiers but worse generators, and the one working fix was generating the data close to the starting policy17 (abstract, §1, §6).
6.2 Llama 3: iterate, and use only the newest batches
Llama 3 is the large-scale answer. Each of six rounds collects fresh preference annotations by sampling two responses from two different models, with the best models of the previous round doing the sampling; reward-model training uses all preference data collected so far, while DPO uses only the newest batches. The report's reasoning: "As a result, our training data conforms better to the distribution of the policy model that is being optimized in each round"4 (§4.1.4, §4.1.6, §4.2.1). Models are also averaged within each stage (§4.1.5). The team also tried PPO and found that DPO "required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval"4 (§4.1.4). The report gives no PPO numbers.
6.3 The contrast matters more than either response
On-policy pairs are one theory of what makes DPO data work. Olmo 3 and SmolLM3 support another, delta learning: the quality of a pair depends mainly on the gap between chosen and rejected, not on either response alone. Both took chosen responses from Qwen3-32B and rejected ones from Qwen3-0.6B, developed concurrently and independently3 (§8.4). Olmo 3 shows why the framing is useful. Continued SFT of a development 7B SFT checkpoint on the Qwen3-32B chosen responses lowered the 10-benchmark average from 70.3 to 64.5; DPO preferring those same responses over Qwen3-0.6B's raised it to 72.9, 8.4 points above SFT on the same chosen data (derived)5 (§4.5, Table 21). The authors conjecture that the 32B responses are weaker than what the model has already seen, so imitation no longer helps while the contrast still does. A single 7B run then found DPO a better start for RLVR than SFT alone: 74.1 for SFT+DPO+RLVR against 71.9 for SFT+RLVR (Table 22)5 (§4.5).
The same logic, with a verifier choosing the pairs, is Iterative Reasoning Preference Optimization: sample 30 chain-of-thought solutions per problem, form ten winner-loser pairs per problem with correct answers as winners and incorrect ones as losers, train DPO plus NLL, and repeat from the new model. On GSM8K from Llama-2-70B-Chat the four iterations score 73.1%, 78.0%, 81.1% and 81.6%, against 55.6% zero-shot and 63.5% for SFT on gold solutions; the per-iteration gains of 17.5, 4.9, 3.1 and 0.5 points show saturation on a fixed prompt set6 (§3.1, Table 1). This is offline-in-rounds learning from a binary outcome signal. Modules 9 and 10 treat the online version, in which the policy is updated on every batch of its own samples.
7. DeepSeekMath's view: one gradient, different data and coefficients
DeepSeekMath writes SFT, rejection-sampling fine-tuning (RFT), DPO, PPO and GRPO as one gradient, \(\nabla_\theta\mathcal J_{\mathcal A}=\mathbb E_{(q,o)\sim\mathcal D}\big[\tfrac1{|o|}\sum_t GC_{\mathcal A}(q,o,t,\pi_{rf})\,\nabla_\theta\log\pi_\theta(o_t|q,o_{<t})\big]\), and says the methods differ in three things: the data source \(\mathcal D\), the reward function \(\pi_{rf}\), and the algorithm that turns reward into the gradient coefficient \(GC\)18 (§5.2.1, Eq. 5). Table 4 reproduces their summary for the rows relevant here. DPO's coefficient is the pair weight of Section 2.4, \(\sigma(\hat r_l-\hat r_w)\) (their Eq. 14), and their Eq. 12 writes the DPO objective with \(1/|o|\) averaging of the log-ratios, i.e. in the length-normalized form of Section 3.2.
| Method | Data source | Reward | Gradient coefficient | Data | Sign |
|---|---|---|---|---|---|
| SFT | \(q,o\sim P_{sft}\) | none (human selection) | 1 | offline | positive |
| RFT | \(q\sim P_{sft},\ o\sim\pi_{sft}\) | rule | \(\mathbb 1[\text{correct}]\) | offline | positive |
| DPO | \(q\sim P_{sft},\ o^+,o^-\sim\pi_{sft}\) | preference (rule in math) | \(\sigma(\hat r_l-\hat r_w)\), Eq. 14 | offline | signed pair |
| Online RFT | \(q\sim P_{sft},\ o\sim\pi_\theta\) | rule | \(\mathbb 1[\text{correct}]\) | online | positive |
| PPO | \(q\sim P_{sft},\ o\sim\pi_\theta\) | reward model | advantage, Eq. 18 | online | signed |
| GRPO | \(q\sim P_{sft},\ \{o_i\}\sim\pi_\theta\) | reward model | group-relative advantage, Eq. 21 | online | signed |
The last two columns are added here, and they organize Section 6: the two ingredients Tajwar et al. found complementary, on-policy sampling and a negative gradient, are the two axes. RFT is offline and non-negative, Online RFT online and non-negative, DPO offline and signed, PPO and GRPO online and signed. DeepSeekMath's measurements fall on the same axes: Online RFT matches RFT early, when "the actor and the SFT model exhibit close resemblance", and pulls ahead as the policy's samples diverge; GRPO beats Online RFT because its coefficient penalizes incorrect answers while Online RFT "does not penalize incorrect responses"18 (§5.2.1, Figure 5, DeepSeekMath-Instruct 1.3B). "Online DPO", which samples pairs from the current policy and labels them, fills the remaining cell, online and signed with a pairwise coefficient3 (§8.2). GRPO itself is Module 9.
8. The evidence ledger: what supports "DPO is enough"
Module 5 Section 9.2 (Table 6) lists the head-to-head PPO and DPO-family studies one row each. Table 5 is organized by claim, so that the evidence on both sides can be read together. Cost has the least dissent: 28 hours against 4 hours in Tulu 3's controlled setup is 7 times (derived), or 14 times in node-hours2 (§5.4.1), and Llama 3 reports less compute for DPO at its scale4 (§4.1.4).
| Claim | Evidence for | Evidence against or limiting |
|---|---|---|
| DPO matches or beats PPO-RLHF on preference tuning | TL;DR with GPT-J: DPO about 61% win rate at temperature 0.0 against PPO's 57% at its best temperature; human raters preferred DPO samples 58% of the time1 (§6.2). Llama 3 found DPO better on IFEval4. Tulu 3: PPO reached a comparable but slightly lower average in an untuned setup2 (§5.4.1). | Xu et al.: PPO ahead on every testbed, from dialogue to code8. Ivison et al.: PPO ahead by up to 2.5% on math and 1.2% in general domains9. The paper's own authors note that β was not meaningfully tuned and models were ≤ 6B1 (§6.2, §7). |
| DPO generalizes out of distribution as well as PPO | Summarization policies moved to CNN/DailyMail: GPT-4 win rate against ground truth 0.36 (DPO) against 0.26 (PPO) at temperature 0, and 0.31 against 0.23 at 0.251 (§6.3, Table 1). | Distribution shift between reference and pairs is DPO's weak point: 55.4% against PPO's 99.5% safe rate until the mismatch is repaired8 (Table 2); online algorithms are a Pareto improvement at equal KL17. |
| DPO suffices for tasks with verifiable outcomes | DPO plus NLL on verifier-labeled pairs lifts GSM8K from 55.6% to 81.6% over four iterations6 (Table 1). Olmo 3 reports DPO before RLVR as a better start than SFT alone5 (Table 22). | CodeContests test 10@1k with CodeLlama-34B: SFT 15.2%, DPO 0.0%, DPO-Iter 3.2%, PPO with test-case rewards 22.4%8 (Table 8). Tulu 3 kept DPO for preference iteration and used PPO for RLVR2 (§5.4.1). Iterative RPO's gains shrink to 0.5 points by round four6. |
| Online or on-policy data are necessary | Tajwar et al., Tang et al. and Xu et al. (Section 6.1); DPO-Iter closes most of the safety gap in Table 3. | Delta learning gains came from fixed pairs generated by other models5; Llama 3's rounds are batch-collected, not online4. Tajwar et al. say offline supervised methods can suffice when the reward peak lies in a high-probability region of the reference13. |
| A better variant loss improves results | SimPO: up to +6.4 AlpacaEval 2 and +7.5 Arena-Hard over DPO11. KTO: +13.5 GSM8K over DPO on Zephyr-β-SFT7. | Tulu 3: only length-normalized DPO beat the base2. In SimPO's own table most variants trail DPO. Data quality ranks above the algorithm9. |
The ledger has a shape. DPO's advantages are cost and simplicity, and they are real. The evidence against it is concentrated in two regimes: when the reference and the pairs are mismatched, and when the objective is a hard verifiable outcome. In the regime where most DPO is actually used, chat-style preference tuning from a strong SFT checkpoint with well-built pairs, the differences between DPO and tuned PPO are a few points, and Ivison et al. rank them below the effect of the preference data9.
Where experts disagree
The primary disagreement is whether offline DPO is enough or online RL is needed for preference tuning. Three more sit under it: whether the loss or the data is the lever, whether a falling chosen likelihood is a defect, and whether the variants beat DPO at all (Section 5.3). The studies below use different stacks, which is part of the problem.
DPO-family methods are the right default
Llama 3 team, Tulu 3 authors, Lambert's practical advice423
Llama 3 chose a simple post-training loop over RL algorithms that "tend to be less stable and harder to scale", tried PPO, and kept DPO for compute and IFEval reasons4 (§1, §4.1.4). Tulu 3 measured PPO at about 7 times the wall-clock for a comparable average and spent the savings on data iteration2 (§5.4.1). Lambert advises starting with direct alignment on synthetic preferences3 (§8.4).
- Seven times cheaper in wall-clock in the one controlled measurement; stable at 405B.
- Fast data iteration, with cacheable reference log-probs.
- Llama 3 publishes no PPO numbers and Tulu 3's PPO was untuned, so neither shows a ceiling.
- Both recipes carry patches (NLL term, token masking, length normalization, one epoch), so "simple" describes the loss, not the recipe.
Online RL wins, and the gap is structural
Xu et al., Tang et al., Tajwar et al.; Lambert's book817133
The loss is flat off the data's support (Section 4.1), the pairs go stale as the policy moves (Section 6.1), and hard verifiable tasks expose both: DPO scored 0.0% on CodeContests where PPO scored 22.4%8 (Table 8). Tang et al. find a Pareto gap at equal KL that scaling the policy does not close17. Lambert writes that "despite the simplicity of alternatives, RL-based methods continue to win out"3 (§3.1.6).
- Theory (a proper-subset result in Xu et al.) and experiments agree across three groups.
- Fits why Tulu 3 and Olmo 3 end their pipelines with RLVR.
- The strongest numbers pair a heavily tuned PPO with a DPO lacking an NLL term, length control or iteration.
- Tang et al. use a gold-reward synthetic setup; no source here has a matched comparison at 70B and up on human preferences.
The loss is second-order; the pairs are the lever
Ivison et al., Olmo 3 authors, Lambert953
Ivison et al. find preference data quality gives the largest gains, ahead of the algorithm, the reward model and extra prompts9. Olmo 3 turns a response set that hurts under SFT into a gain under DPO by choosing the rejected side5 (Table 21). Lambert writes that the algorithm matters far less than the initial model and the data3 (§8.2).
- Consistent across independent groups and the variant comparisons of Section 5.3.
- Actionable: build pairs with a large capability delta or from on-policy samples.
- "Good data" has two competing definitions, on-policy (Tulu 3) and large delta (Olmo 3, SmolLM3), which Lambert presents as separate theories3 (§8.4).
- No study here varies data and algorithm jointly at scale.
A falling chosen likelihood is a bug to patch, or it is expected
Pal et al., Razin et al., Pang et al., Llama 3 against Rafailov et al.1214610
The first group patches it (NLL term, DPOP, CHES filtering) and shows harm: a refusal rate falling from 74.4% to 33.4%14, GSM8K at 61.8% without the NLL term against 73.1% with it6. Rafailov et al. argue that after SFT on the chosen responses a negative implied reward is the expected sign, and that win rate rises while it falls, until the over-optimization hump10 (§3.5).
- The patch is cheap, and Llama 3 and Pang et al. report it helps.
- The skeptical view fits the DPO objectives of Tulu 3 and Olmo 3, which carry no NLL term in the cited sections.
- The harm is shown for near-duplicate pairs (Pal et al.) and refusal data (Razin et al.).
- Rafailov et al. measured TL;DR summarization at 1B to 6.9B; the NLL gains are on a reasoning task.
Both camps are right inside their regimes. For chat, instruction-following and safety tuning from a strong SFT start with well-built pairs, a patched DPO is the efficient default: length normalization when chosen responses run long, token masking, one epoch, early stopping judged on generations, and an NLL term when the chosen log-ratio falls and the task is reasoning-like. The measured gap to tuned PPO there is a few points and ranks below the effect of the data9. Online or verifier-driven RL wins where the target is a hard verifiable outcome, where reference and pairs cannot be kept matched by iteration, or where the goal is to move beyond the support of the data. Chosen-likelihood decline is a monitor, not a verdict: log it, and patch when downstream behavior degrades in the way Razin et al. describe. What would settle the primary question is a matched-compute, matched-data comparison with both sides tuned, at 70B or larger, scored on held-out human preferences and on verifiable tasks, reporting KL and the chosen-likelihood trace. No source here contains one. Module 33 turns this into a decision procedure.
What is changing
DPO's role has moved from "the alignment method" to one stage in a longer pipeline. Olmo 3 runs SFT, then length-normalized DPO on delta-learning pairs, then RLVR, and reports DPO as the better start for RL5 (§4.5). Lambert expects that as reasoning models are trained mainly with RL, investment will return to RL for preference tuning, widening the margin over direct alignment3 (§8.5). Whether a DPO stage expands what the model can solve, as Olmo 3 claims from improved pass@k on AIME5 (§4.5, Figure 20), or only sharpens it, is the question of Module 20. The critic-free online methods of Modules 9 to 12 occupy the cell Section 7 left open: online data with a signed coefficient and no value model. The evidence here is dated: the controlled comparisons are from 2023 to 2024, the production reports from 2024 to 2025, and the KTO paper was revised as late as September 2026, so treat any single number as a snapshot of one stack.
References
- paperRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023; arXiv v3, 29 Jul 2024. https://arxiv.org/pdf/2305.18290v3. Accessed 2026-10-26. Used for: Eqs. 3-7 and the derivations in Appendix A.1-A.6; Lemmas 1-2 and Theorem 1 (§5.1); gradient (Appendix A.4); defaults and the PyTorch loss (Appendix B); TL;DR 61% against 57% and the human study (§6.2, §6.4); CNN/DailyMail transfer (Table 1); the unlikelihood baseline (Appendix C.3).
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025 (v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: Length-normalized DPO (Eq. 6); the algorithm and hyperparameter search (§5.4.1, Table 18); final settings (Table 20); PPO about 28 h on two nodes against DPO about 4 h on one; reference log-prob caching and separate forward passes (§5.4.2, Figure 17); the 354,192-instance preference pipeline (§1, §5.2.1).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026; web edition built 14 Oct 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: Chapter 8: the implicit-reward reading (§8.1.1), variants and preference displacement (§8.2), reference caching (§8.3), delta learning and synthetic preferences (§8.4), DAAs against RL (§8.5); the RL stance in §3.1.6.
- paperGrattafiori, A., Dubey, A., Jauhri, A., et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024 (v1 31 Jul 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: Design stance (§1); DPO settings, token masking, NLL coefficient 0.2 and the PPO comparison (§4.1.4); six rounds (§4.1.6); preference-data strength levels and filtering (§4.2.1); response-length statistics (Table 6).
- paperOlmo Team (Ettinger, A., Bertsch, A., et al.). Olmo 3. arXiv v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: Delta learning and the Dolci Think DPO data (§4.3, §4.5); Tables 21 and 22; length-normalized DPO loss and hyperparameters (Appendix A.6.2, Table 48).
- paperPang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., Weston, J. Iterative Reasoning Preference Optimization. arXiv v3, 26 Jun 2024. https://arxiv.org/abs/2404.19733. Accessed 2026-10-27. Used for: The DPO plus NLL loss and iteration scheme (§2); hyperparameters and GSM8K results (§3.1, Table 1, Figures 2-3).
- paperEthayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D. KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024; arXiv v5, 8 Sep 2026. https://arxiv.org/abs/2402.01306. Accessed 2026-10-27. Used for: HALO definition and Theorem 3.5 (§3); the KTO loss and KL estimate (§4.1); hyperparameters (§4.2); results, Table 2 and Figure 4 (§4.3); Proposition 4.1, Theorem 4.3 and the DPO-or-KTO guidance (§4.4-§4.5). Full text read; the source map had the abstract only.
- paperXu, S., Fu, W., Gao, J., Ye, W., et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: Three-action counterexample and the proper-subset theorem (Table 1, §4.1); SafeRLHF distribution-shift and DPO-Iter results (Table 2, §4.3); APPS and CodeContests results (Tables 7-8).
- paperIvison, H., Wang, Y., Liu, J., Wu, Z., et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv v2, 7 Oct 2024. https://arxiv.org/abs/2406.09279. Accessed 2026-10-26. Used for: Ranking of factors (data, algorithm, reward model, prompts) and the PPO margins of 2.5% (math) and 1.2% (general), from the abstract.
- paperRafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., Niekum, S. Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS 2024; arXiv v2, 5 Nov 2024. https://arxiv.org/abs/2406.02900. Accessed 2026-10-27. Used for: Hump-shaped over-optimization in DPO, IPO and SLiC (§3.1-§3.2, Figures 1-2); implicit-reward accuracy against win rate (§3.4); decreasing-likelihood argument (§3.5); null-space argument, Proposition 1 and Figure 7 (§4).
- paperMeng, Y., Xia, M., Chen, D. SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS 2024; arXiv v3, 1 Nov 2024. https://arxiv.org/abs/2405.14734. Accessed 2026-10-27. Used for: The reward and objective (§2.2-§2.3, Eq. 6); Table 4 (tuned baselines); Table 6 and Figure 4 (length and likelihood mismatch); Appendices B, I (Table 15, DPO with length normalization) and J.
- paperPal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., White, C. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv v2, 3 Jul 2024. https://arxiv.org/abs/2402.13228. Accessed 2026-10-27. Used for: Preferred-likelihood reduction result and the token-level account (§3, Figure 1); the DPOP loss (Eq. 3) and settings beta 0.3, lambda 50 (§4-§5).
- paperTajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., Kumar, A. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. arXiv v3, 2 Jun 2024. https://arxiv.org/abs/2404.14367. Accessed 2026-10-27. Used for: On-policy sampling and negative-gradient takeaways (§5.1-§5.3); induced-reward behavior under DPO (Figures 16-17).
- paperRazin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., Hanin, B. Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. ICLR 2025; arXiv v4, 27 Apr 2025. https://arxiv.org/abs/2410.08847. Accessed 2026-10-27. Used for: Definition of likelihood displacement, the CHES score, the refusal-rate drop from 74.4% to 33.4% and the filtering result (abstract).
- paperAzar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., Munos, R. A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv v2, 22 Nov 2023. https://arxiv.org/abs/2310.12036. Accessed 2026-10-27. Used for: Overfitting under deterministic preferences (§4.2); the IPO loss (Eq. 17); bandit experiments (§5.3-§5.4).
- paperHong, J., Lee, N., Thorne, J. ORPO: Monolithic Preference Optimization without Reference Model. arXiv v2, 14 Mar 2024. https://arxiv.org/abs/2403.07691. Accessed 2026-10-27. Used for: The odds-ratio objective and per-token mean probability (§4.1-§4.2); Mistral-ORPO AlpacaEval 2.0 results.
- paperTang, Y., Guo, D., Zheng, Z., Calandriello, D., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv v1, 14 May 2024. https://arxiv.org/abs/2405.08448. Accessed 2026-10-27. Used for: Online against offline at equal KL budget; hypotheses on coverage, data quality, classification, loss form and scale (abstract, §1, §3, §5, §6).
- paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: Unified gradient form and Table 10 (§5.2.1, Eq. 5); the DPO objective and coefficient (Appendix A.1.4, Eqs. 12-14); Online RFT against RFT and GRPO (Figure 5).