Reinforcement Learning for LLM Post-Training · Module 7

Direct Alignment: Deriving DPO, the Implicit Reward, Its Failure Modes, and the KTO, Length-Normalized and NLL-Regularized Variants

Direct Preference Optimization replaces the reward model and the PPO loop with one classification loss over preference pairs. This module derives that loss from the KL-regularized objective of Module 6, reads its gradient and its implicit reward, and catalogues what goes wrong in practice (unconstrained probability mass, falling chosen likelihood, length drift, over-optimization) together with the fixes Llama 3, Tulu 3 and Olmo 3 shipped. Afterwards you can derive and implement DPO, pick among its variants, and say precisely which evidence supports and which contradicts "DPO is enough".

Course 45 min read 18 sources As of 2026-10-27 Prerequisites: Modules 4, 6
Two pipelines from preference pairs to a policy: RLHF with PPO needs a reward model, online rollouts and a value model; DPO needs only log-probabilities of the stored responses under the policy and a frozen reference RLHF with PPO: reward model, online rollouts, critic preference pairs reward model trained, Module 4 rollouts sample from π_θ value model GAE, KL reward PPO update clipped surrogate, Module 3 DPO: stored pairs, policy and frozen reference, one loss preference pairs log π_θ and log π_ref of y_w and y_l (reference can be cached) −log σ(β Δ) Eq. 7 gradient step supervised-style training deleted by DPO: an explicit reward model, sampling from the policy during training, a value model kept: preference pairs, a reference policy, and β, which carries the KL strength
Figure 1. The RLHF pipeline of Modules 4 and 5 (top) against DPO (bottom). The three orange boxes are what DPO removes. The pairs, the reference policy and β remain; the reference log-probabilities can be precomputed once. Sources: DPO Figure 1 and §3-§41; reference-log-prob caching, Tulu 3 §5.4.22.

1. What DPO removes from the RLHF pipeline

1.1 Three stages become one loss

The classical recipe you met in Modules 4 and 5 has three stages: supervised fine-tuning to get \(\pi_{\mathrm{SFT}}\), a reward model fit to preference pairs with a Bradley-Terry loss, and an RL stage that maximizes the learned reward minus a KL penalty to the SFT model1 (§3). DPO keeps the first stage and the pairs, and replaces the other two with one loss computed on stored responses. The abstract states the consequence directly: no sampling from the language model during fine-tuning and no significant hyperparameter tuning1.

The cost difference is measured. In Tulu 3's controlled comparison PPO took about 28 hours on two nodes and DPO about 4 hours on one, with roughly equal average scores2 (§5.4.1); Module 5 §9.1 converts this to 14 times the node-hours. Tulu 3's production DPO runs took 10 hours on 8 H100 GPUs (8B) and 19 hours on 64 H100 GPUs (70B)2, which is 80 and 1,216 H100-hours (derived).

What remains is a training loop of supervised-learning shape: per pair, the policy runs forward and backward on two sequences and the frozen reference runs forward on the same two. Tulu 3 takes the reference out of GPU memory by precomputing its log-probabilities once over the dataset, and runs chosen and rejected sequences in separate forward passes instead of one concatenated batch; the two changes gave near-identical training losses and lower peak memory2 (§5.4.2, Figure 17). Lambert reports that caching alone cuts peak memory by 50%3 (§8.3). The reference costs one extra forward pass over the dataset, not a standing second copy of the weights.

1.2 What the loss still needs, and what it never sees

Three things survive from the RLHF recipe. The pairs \((x, y_w, y_l)\) are still required, and they should come from or near the reference policy: the paper sets \(\pi_{\mathrm{ref}}=\pi_{\mathrm{SFT}}\) when the pairs were sampled from the SFT model, and otherwise fits \(\pi_{\mathrm{ref}}\) by maximum likelihood on the chosen responses to reduce the distribution shift1 (§4). The reference policy is still the anchor of the KL term. And \(\beta\) is still the price of moving away from it (Module 6, Section 4.1); in DPO it is the scale inside the sigmoid, not a penalty added to a reward.

What DPO never sees is any response the policy generates itself during training. Every probability the loss touches belongs to a response that was written down before training began. Sections 4 and 6 show that this one fact explains most of the method's failure modes and most of the evidence for online RL.

2. Deriving the loss

2.1 Five equations, in order

The derivation is a change of variables from rewards to policies. Start from the objective of Module 6, \(\max_\pi \mathbb E_{y\sim\pi}[r(x,y)]-\beta\,\mathrm{KL}(\pi\|\pi_{\mathrm{ref}})\) (Eq. 3 in the paper)1. Its maximizer, derived in Module 6 Section 1.2 and in Appendix A.1 of the paper, is \(\pi_r(y|x)=\pi_{\mathrm{ref}}(y|x)\,e^{r(x,y)/\beta}/Z(x)\) (Eq. 4), with \(Z(x)=\sum_y\pi_{\mathrm{ref}}(y|x)e^{r(x,y)/\beta}\). Taking logarithms and solving for the reward gives Eq. 5:

$$ r(x,y)=\beta\log\frac{\pi_r(y|x)}{\pi_{\mathrm{ref}}(y|x)}+\beta\log Z(x). $$

Eq. 5 is useless for a reward model by itself, because \(Z(x)\) sums over every possible response. The step that makes it usable is that Bradley-Terry depends only on the difference of two rewards for the same prompt (Module 4, Section 2.3). Substituting Eq. 5 into \(p^*(y_1\succ y_2|x)=\sigma(r^*(x,y_1)-r^*(x,y_2))\), the two \(\beta\log Z(x)\) terms are identical and cancel (Eq. 6; Appendix A.2)1. What is left is a preference probability written entirely in policies. Maximum likelihood over a parametrized policy \(\pi_\theta\) in place of the unknown optimal one gives Eq. 7, the DPO loss:

$$ \mathcal L_{\mathrm{DPO}}(\pi_\theta;\pi_{\mathrm{ref}})=-\mathbb E_{(x,y_w,y_l)\sim\mathcal D}\Big[\log\sigma\Big(\beta\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)}-\beta\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\Big)\Big]. $$
Eq. 3 max_π E[ r(x,y) ] − β · KL( π(·|x) ‖ π_ref(·|x) ) the RLHF objective for a reward r (Module 6, Section 1) Eq. 4 π_r(y|x) = π_ref(y|x) · exp( r(x,y)/β ) / Z(x) closed-form optimum; Z(x) = Σ_y π_ref · exp(r/β) depends on the prompt only Eq. 5 r(x,y) = β · log[ π_r(y|x) / π_ref(y|x) ] + β · log Z(x) invert: the reward is a log-ratio plus a term that depends on x alone Eq. 6 p*(y1 ≻ y2) = σ( β log π*(y1)/π_ref(y1) − β log π*(y2)/π_ref(y2) ) Bradley-Terry subtracts two rewards at the same x, so β · log Z(x) cancels Eq. 7 L_DPO = −E log σ( β log π_θ(y_w)/π_ref(y_w) − β log π_θ(y_l)/π_ref(y_l) ) maximum likelihood over the policy replaces fitting a separate reward model
  1. Objective. The KL-regularized reward maximization every RLHF recipe uses, for any reward function.
  2. Optimum. The maximizer is the reference policy exponentially tilted by the reward.
  3. Invert. Solve the tilt for the reward: a scaled log-ratio plus a prompt-only normalizer.
  4. Cancel. Substituted into Bradley-Terry, the normalizer drops out because both responses share the prompt.
  5. Fit. Maximize the likelihood of the observed preferences over the policy itself.

Figure 2. The derivation chain, one equation per step. The numbering is the paper's1 (§4, Appendix A.1-A.2).

2.2 Why cancelling \(Z\) loses nothing

Two reward functions that differ by \(f(x)\) are equivalent: they induce the same Bradley-Terry preference distribution (Lemma 1) and the same optimal policy (Lemma 2). Theorem 1 then states that every equivalence class contains exactly one reward of the form \(\beta\log\frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)}\), namely the one whose normalizer satisfies \(\sum_y\pi_{\mathrm{ref}}(y|x)e^{r(x,y)/\beta}=1\)1 (§5.1, Appendix A.5-A.6). The reparameterization therefore does not restrict which preference distributions can be represented. It selects one representative per class, and because \(\pi_\theta\) is a normalized distribution, the constraint that picks the representative holds automatically.

The guarantee has conditions: a Bradley-Terry data-generating process, a reference policy with \(\pi_{\mathrm{ref}}(y|x)>0\) everywhere, and a policy class rich enough to hit the optimum1 (§5.1, Theorem 1); the paper states consistency only "under suitable assumption of the preference data distribution"1 (§4). A finite offline dataset covers a vanishing fraction of the response space, which is where Section 4 begins.

2.3 The implicit reward is a sum of token log-ratios

The paper's title claim is that the language model is secretly a reward model: \(\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}\)1. By the chain rule this is \(\beta\sum_t\log\frac{\pi_\theta(y_t|x,y_{<t})}{\pi_{\mathrm{ref}}(y_t|x,y_{<t})}\) (derived), a sum over tokens of the same log-ratio that is the k1 sample of \(\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\) in Module 6, here evaluated on a stored response rather than a sampled one. Lambert stresses the reading: DPO is, at its core, learning a reward model, and the policy is how that reward is parametrized3 (§8.1.1). The reference implementation returns these detached implicit rewards for logging1 (Appendix B), and the fraction of pairs with \(\hat r_w>\hat r_l\) is the usual "reward accuracy" curve.

2.4 The gradient: a per-pair weight on a contrast

Differentiating Eq. 7 (Appendix A.4)1 gives

$$ \nabla_\theta\mathcal L_{\mathrm{DPO}}=-\beta\,\mathbb E\Big[\underbrace{\sigma\big(\hat r_\theta(x,y_l)-\hat r_\theta(x,y_w)\big)}_{\text{larger when the implicit ordering is wrong}}\big(\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)\big)\Big]. $$

The structure is the one Module 4 derived for the Bradley-Terry reward-model gradient, with \(\nabla\hat r_\theta=\beta\nabla\log\pi_\theta\) in place of \(\nabla r_\theta\). Each pair raises the likelihood of \(y_w\) and lowers that of \(y_l\), scaled by \(\beta\) and by a weight in \((0,1)\) that is 0.5 at a tie. The paper reports that the weight matters: an unweighted likelihood-up, likelihood-down baseline produced meaningless text on summarization1 (Appendix C.3). With \(\Delta=\log\frac{\pi_\theta(y_w)}{\pi_{\mathrm{ref}}(y_w)}-\log\frac{\pi_\theta(y_l)}{\pi_{\mathrm{ref}}(y_l)}\) in nats, the weight is \(\sigma(-\beta\Delta)\), and \(\beta\) sets how many nats of log-ratio gap a pair can absorb before it stops teaching.

Worked pair (derived). Take \(\beta=0.1\), a chosen response whose log-ratio is \(+2.0\) nats and a rejected one at \(-3.0\) nats. Then \(\Delta=5.0\), the margin inside the sigmoid is \(\beta\Delta=0.5\), the loss is \(-\log\sigma(0.5)=0.474\), and the weight is \(\sigma(-0.5)=0.378\). The step on \(\nabla\log\pi_\theta(y_w)\) is \(\beta\cdot0.378=0.0378\) times the learning rate. The weight falls to 0.1 when \(\beta\Delta=\ln 9=2.197\), i.e. at \(\Delta=22.0\) nats for \(\beta=0.1\) and \(\Delta=0.44\) nats for \(\beta=5\).

DPO per-pair gradient weight sigma(-beta times Delta) against the log-ratio difference Delta in nats for beta 0.1, 0.5 and 5 0 0.25 0.5 0.75 1 -20 -10 0 10 20 Δ = chosen log-ratio − rejected log-ratio (nats) gradient weight σ(−βΔ) β = 0.1 β = 0.5 β = 5 β=0.1 reaches weight 0.1 at Δ = 22.0 β=5 reaches weight 0.1 at Δ = 0.44 worked pair: Δ = 5.0, weight 0.378
Figure 3. The per-pair weight \(\sigma(-\beta\Delta)\) on the DPO gradient against \(\Delta\), the chosen-minus-rejected change in log-ratio, in nats (derived from the gradient in Appendix A.41). At \(\beta=0.1\) a pair keeps teaching until the gap reaches tens of nats; at \(\beta=5\) it saturates within a fraction of a nat. Length normalization (Section 3.2) moves a recipe along this axis.

2.5 The loss as code

The implementation is short enough that the code is the clearest statement of what is computed. This version follows the paper's Appendix B1, with a token mask (Section 4.4) and the optional NLL term of Section 4.2 added, and with the reference log-probabilities passed in as cached values.

import torch
import torch.nn.functional as F

def seq_logp(logits, labels, mask):
    # logits (B,T,V); labels, mask (B,T). mask is 1 on response tokens that count in the loss
    lp = torch.log_softmax(logits.float(), dim=-1).gather(-1, labels.unsqueeze(-1)).squeeze(-1)
    return (lp * mask).sum(-1)                      # log pi(y|x), summed over tokens

def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1, n_w=None, alpha=0.0):
    # pi_*: (B,) policy sequence log-probs; ref_*: (B,) cached reference log-probs
    delta = (pi_w - ref_w) - (pi_l - ref_l)         # log-ratio difference, nats
    loss = -F.logsigmoid(beta * delta)              # Eq. 7; gradient weight is sigmoid(-beta * delta)
    if alpha > 0:                                   # NLL on the chosen sequence (Section 4.2)
        loss = loss + alpha * (-pi_w / n_w)         # n_w = number of unmasked chosen tokens
    r_w = (beta * (pi_w - ref_w)).detach()          # implicit rewards, for logging
    r_l = (beta * (pi_l - ref_l)).detach()
    return loss.mean(), r_w, r_l

Two details are easy to get wrong. The log-probabilities are sums over response tokens, not means, unless you are deliberately running the length-normalized variant (Section 3.2). And the reference pass must use the same tokenization, template and mask as the policy pass, or every \(\Delta\) shifts by a constant the loss will then try to undo.

3. What you set: β, learning rate, the reference model and the data

3.1 Reported settings span two orders of magnitude in learning rate

The original paper's defaults are \(\beta=0.1\), batch size 64, RMSprop and a learning rate of \(10^{-6}\) with a 150-step linear warmup, with \(\beta=0.5\) for TL;DR summarization1 (Appendix B). Lambert notes that the method only worked for the community after it found the right data and "surprisingly low learning rates"3 (§8). Table 1 collects what the primary reports used since. Two patterns: the learning rate falls by more than two orders of magnitude from Llama 3 to Olmo 3 Think, and \(\beta\) jumps from 0.1 to 5 exactly when the loss changes from summed to length-normalized log-ratios (Section 3.2).

Table 1. Reported DPO settings, by recipe. "Sum" is the original loss on summed token log-ratios; "norm" divides each log-ratio by its response length. "n/s" means not stated in the cited section. Sources in the first column.
RecipeLoss formβLearning rateBatchOther settings
DPO paper, models ≤ 6B1 (App. B)sum0.1 (TL;DR 0.5)1e-6, RMSprop64150-step warmup
Llama 3, up to 405B4 (§4.1.4)sum + NLL0.11e-5n/sNLL coefficient 0.2 on chosen; formatting tokens masked; six rounds
Tulu 3 8B2 (Table 20)norm55e-71281 epoch, warmup ratio 0.1, max length 2,048
Tulu 3 70B2 (Table 20)norm52e-7128as 8B; 19 h on 64 H100
Olmo 3 Think 7B5 (Table 48)norm58e-8128150K pairs, 1 epoch, max length 16K, 32 GPUs
Olmo 3 Think 32B5 (Table 48)norm57e-8128200K pairs, 1 epoch, max length 8K, 64-128 GPUs
Olmo 3 Instruct 7B5 (Table 48)norm51e-6128260K pairs, 1 epoch, max length 16K, 16 GPUs
Iterative RPO (GSM8K)6 (§2, §3.1)sum + NLL0.17e-7, AdamW16NLL coefficient 1; about 55-60k pairs per iteration; four iterations

3.2 What β means once log-ratios are divided by length

The length-normalized loss (Tulu 3 Eq. 62; the Olmo 3 appendix writes the same form5, A.6.2) replaces each log-ratio by its per-token mean:

$$ \mathcal L_{\mathrm{DPO\text{-}norm}}=-\mathbb E\Big[\log\sigma\Big(\tfrac{\beta}{|y_w|}\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)}-\tfrac{\beta}{|y_l|}\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\Big)\Big]. $$

Two consequences follow (derived). First, \(\beta\) is only comparable across loss forms at a given length: \(\beta_{\mathrm{norm}}\) acts like a summed-form \(\beta_{\mathrm{sum}}=\beta_{\mathrm{norm}}/L\), so \(\beta_{\mathrm{norm}}=5\) equals the paper's 0.1 at \(L=50\) tokens. As an illustration only, Llama 3's preference data average 284.0 response tokens4 (Table 6), which puts \(\beta_{\mathrm{norm}}=5\) at \(5/284=0.018\) on the summed scale; a pair then saturates (weight 0.1) at a mean per-token gap of \(\ln 9/5=0.44\) nats, about 125 nats of summed gap, against 22 nats at \(\beta_{\mathrm{sum}}=0.1\). Second, the per-token gradient on each response is \(\beta/|y|\) times the pair weight, so a rejected response twice as long as the chosen one is pushed down half as hard per token; in the summed form every token gets \(\beta\) times the weight regardless of length, and longer responses absorb more total gradient. Tulu 3 states the intent as mitigating the length bias common in human and model preferences2 (§5.1).

3.3 The reference model and the SFT stage

DPO keeps the reference fixed at the SFT checkpoint, which makes it the anchor of the implicit KL (Module 6, Section 4.2 lists the alternatives). The evidence that this stage matters comes from the KTO paper: SFT "is always needed for best results with DPO", and DPO run without SFT on a base model rambles and hallucinates whole conversations, while KTO at 13B and 30B does not need SFT7 (§1, §4.3, Figure 4). When training is iterated, the reference moves: Iterative RPO uses the previous iteration's model \(M_t\) as the reference for iteration \(t+1\)6 (§2), and Xu et al.'s DPO-Iter sets the reference to the latest DPO model each round8 (§4.3). With a cached reference pass (Section 1.1) that costs one forward pass over the new data per round.

3.4 Pair construction: strength filters and synthetic judges

Llama 3's annotators rate preference strength on four levels (significantly better, better, slightly better, marginally better). For both reward modeling and DPO the team trains on pairs labeled significantly better or better and discards pairs with similar responses; reward modeling uses all preference data available, while DPO uses only the most recent batches4 (§4.2.1). Tulu 3 builds 354,192 preference instances with an UltraFeedback-style pipeline: four responses per prompt from a pool of 22 models plus on-policy samples from the Tulu SFT model, rated 1 to 5 by GPT-4o-2024-08-06 on helpfulness, instruction following, truthfulness and honesty, then binarized2 (§1, §5.2.1, Figure 7). Ivison et al. rank the factors: preference data quality gave the largest improvements, then the learning algorithm, then a better reward model, then extra unlabeled prompts9 (abstract). Section 4 turns to what the loss itself does not control.

4. Failure modes, in the order they bite

The failures below share one cause: the loss sees only differences of log-ratios on stored responses, so anything it cannot see is left to the network's inductive bias. They are listed roughly by how early a practitioner meets them.

4.1 The loss is flat along directions it cannot see

For one prompt with stored responses \(y_1\succ y_2\) and \(\pi_{\mathrm{ref}}(y_1)=\pi_{\mathrm{ref}}(y_2)\), Eq. 7 reduces to \(\log\big(1+(\pi(y_2)/\pi(y_1))^{\beta}\big)\) (derived): any change that keeps the ratio fixed leaves the loss untouched, whatever it does to every other response. Rafailov et al. give the general form. Written as logistic regression on vectors \(q_i=\mathbb 1_{(x,y_w)}-\mathbb 1_{(x,y_l)}\) over the prompt-response space, the data matrix has one row per pair and one column per possible (prompt, response), so its null space is enormous, and infinitely many minimizers put weight on responses absent from the data (Proposition 1, adapted from Hejna et al.)10 (§4). Their minimal example, two conflicting pairs \(y_1\succ y_2\) and \(y_2\succ y_1\) plus an unseen \(y_3\), is minimized by both \((0.5,0.5,0)\) and \((0,0,1)\). Xu et al. give a one-pair version with \(\pi_{\mathrm{ref}}=(0.5,0.5,0)\), where \(\pi_{\mathrm{DPO}}=(0.1,0.0,0.9)\) attains the minimum but a PPO policy cannot leave the reference's support8 (Table 1, §4.1).

Three-response example: two policies with the same probability ratio between the chosen and rejected responses have identical DPO loss, but one places 0.901 on a response that appears in no pair 0 0.25 0.5 0.75 1 probability 0.45 0.9 0.09 y1 (chosen) 0.45 0.09 0.009 y2 (rejected) 0.1 0.01 0.901 y3 (not in data) π_ref P1: KL 0.456 nats P2: KL 1.801 nats π(y1)/π(y2) = 10 for P1 and P2 DPO loss, β = 0.1: 0.5846 DPO loss, β = 1: 0.0953 identical for both policies P2 puts 0.901 on y3, a response no pair contains
Figure 4. A three-response example (constructed and computed here). The reference is (0.45, 0.45, 0.10) and the only pair is \(y_1\succ y_2\). P1 and P2 both have \(\pi(y_1)/\pi(y_2)=10\), so \(\mathcal L=\log(1+10^{-\beta})\) is identical for them (0.5846 at \(\beta=0.1\)); their KL to the reference differs by a factor of 3.9. Nothing in Eq. 7 prefers P1. The mechanism is the one in the null-space argument of Rafailov et al.10 and the counterexample of Xu et al.8.

RLHF does not have this hole because its KL term is evaluated on responses sampled from the policy, so mass that moves to \(y_3\) is charged for; PPO can also use prompt-only data, and the KL regularizes those samples8 (§4.1). What keeps the drift small in practice is a small learning rate, a preference set covering diverse domains, and the robustness of large models to new data, as SimPO's authors argue for a method with no reference at all11 (§2.3). Where those fail, mass leaks: in Rafailov et al.'s tree-structured toy MDP with one pair, DPO, IPO and SLiC all put substantial mass on out-of-distribution trajectories while in-distribution probability falls10 (§4, Figure 7).

4.2 Likelihood displacement: the chosen response gets less likely

The most-cited symptom is that \(\log\pi_\theta(y_w|x)\) decreases during DPO training while the margin grows. Pal et al. prove that the DPO loss can reduce the preferred response's likelihood as long as the preferred-to-dispreferred ratio increases, and show it empirically, especially when completions differ in few tokens12 (abstract, §3): a shared prefix contributes nothing to \(\nabla\log\pi(y_w)-\nabla\log\pi(y_l)\), the differing token is pushed up, and the tokens after it can be pushed down (Figure 1). Tajwar et al. see the chosen response's induced reward fall as prompts grow relative to model capacity: Pythia-1.4B on AlpacaFarm lowers both responses, while Mistral-7B on UltraFeedback raises the chosen one, which they attribute to more distinct pairs13 (§5.2.2, Figures 16-17). Razin et al. name it likelihood displacement, tie it to preferences with similar hidden embeddings (a centered hidden embedding similarity, CHES, score), and show it can be harmful: mass moves to responses with a different meaning, and refusal training lowered Llama-3-8B-Instruct's refusal rate from 74.4% to 33.4%14 (abstract).

Figure 5 reproduces the mechanism in a model small enough to inspect. Three responses have feature vectors \(\phi_w=(1,0)\), \(\phi_l=(0.8,0.3)\), \(\phi_o=(0.4,-1)\) and logits \(z_y=\theta\cdot\phi_y\), starting from \(\theta=0\) (uniform). The DPO gradient moves \(\theta\) along \(\phi_w-\phi_l\), which raises the logit of the "other" response \(o\) faster than that of \(y_w\), so \(o\) absorbs mass even though the margin grows.

Toy log-linear model: under DPO alone the chosen log-probability falls from -1.10 to -4.00 while the margin grows to 2.87; adding a 0.1 NLL term on the chosen response keeps it rising to -0.25 0 -2 -4 -6 0 100 200 300 gradient step log probability (nats) chosen -0.25 rejected -2.85 chosen -4.00 rejected -6.87 chosen, DPO rejected, DPO chosen, DPO + NLL rejected, DPO + NLL reference, ln(1/3) = −1.099
Figure 5. Likelihood displacement in a constructed two-parameter model (computed here; \(\beta=1\), learning rate 0.5, 300 steps). Without the NLL term the chosen log-probability falls from −1.10 to −4.00 while the margin \(\beta\Delta\) rises to 2.87 and 98.1% of the mass moves to the response in neither slot. With a 0.1-weighted NLL term it rises to −0.25, the margin reaches 2.61 and the other response holds 16.0%. The toy illustrates the mechanism of Pal et al.12 and Razin et al.14; it measures no LLM.

Three fixes have shipped. An NLL term on the chosen sequence. Llama 3 adds one with coefficient 0.2; the report says the term "helps further stabilize DPO training by maintaining desired formatting for generation and preventing the decrease of log probability of chosen responses"4 (§4.1.4). Pang et al. use coefficient 1 with the term normalized by response length, and measure its value on GSM8K from Llama-2-70B-Chat: DPO on the iteration-1 pairs reaches 61.8%, DPO plus NLL after one iteration reaches 73.1% (Table 1), and the chosen log-probability falls under plain DPO but rises with the NLL term (Figure 3)6 (§3.1). Their Figure 2 also shows why SFT on the chosen responses alone is not a substitute: it raises the rejected responses' probability along with the chosen ones. A hinge on the chosen log-ratio. DPO-Positive adds \(-\lambda\max\!\big(0,\log\frac{\pi_{\mathrm{ref}}(y_w)}{\pi_\theta(y_w)}\big)\) inside the sigmoid, which is zero until the chosen likelihood drops below the reference; the paper uses \(\beta=0.3,\ \lambda=50\)12 (§4, §5). Data filtering. Razin et al. drop the pairs with the highest CHES scores and report that this mitigated the unalignment in their experiments14 (abstract).

Whether the falling likelihood is a defect at all is disputed. If \(\pi_{\mathrm{ref}}\) was fit by SFT on the chosen responses, the expected implicit reward of the chosen response is approximately \(-\mathrm{KL}(\pi_{\mathrm{ref}}\|\pi_\theta)\), which is non-positive by construction, and Rafailov et al. find win rate rising while it falls, until the over-optimization hump of Section 4.610 (§3.5, Eq. 7, Figure 5). The disagreement section returns to this.

4.3 Length and verbosity

The DPO paper's own judge shows the hazard: with the simple prompt GPT-4 preferred longer, more repetitive summaries than humans did, so the headline numbers use a prompt that also asks for concision1 (§6.4). Rafailov et al. relay that DPO-trained models grow longer during training without a matching rise in ground-truth win rate10 (§2, §3.3, citing Park et al.), and the KTO authors see response length climb sharply when DPO is run without SFT7 (Figure 4). SimPO adds a diagnosis: the DPO reward and the likelihood that decoding uses can disagree. On UltraFeedback training pairs with \(\hat r_w>\hat r_l\), about half have a lower average log-likelihood for the chosen response, and the Spearman correlation between average log-likelihood and length is 0.82 for SimPO without length normalization, 0.59 for DPO and 0.34 for SimPO11 (§2.2, §4.4, Table 6). Applied to DPO itself, length normalization raised AlpacaEval 2 length-controlled win rate from 15.1% to 21.0% on Mistral-Base, where the data have strong length bias, and lowered it from 26.8% to 21.7% on Mistral-Instruct, where winners and losers have similar lengths11 (Appendix I, Table 15).

4.4 Formatting tokens and tail repetition

Llama 3 masks header and termination tokens out of the DPO loss for both chosen and rejected responses, because letting them contribute produced tail repetition and abruptly generated termination tokens4 (§4.1.4). The team's hypothesis is that common tokens in both responses give a conflicting objective, since the model must raise and lower their likelihood at once. The gradient makes this concrete (derived): a token with the same prefix in both responses cancels exactly in \(\nabla\log\pi(y_w)-\nabla\log\pi(y_l)\), but an end-of-turn token follows two different prefixes, so it is pushed up in \(y_w\) and down in \(y_l\) with nothing to cancel it. Masking the formatting tokens, as in the mask argument of Section 2.5, removes that term.

4.5 Deterministic preferences and the case for IPO

Azar et al. show that when a pair is always ordered the same way, Bradley-Terry requires the reward gap to diverge and the optimal policy gives the loser zero probability for any \(\beta\); with finite data an empirical preference of 1 can arise even when the true one is 0.8. RLHF is protected in practice because its reward model is under-fit, which regularizes the policy; DPO skips the reward model and loses that protection15 (§4.2). IPO regresses the gap instead of maximizing a likelihood:

$$ \mathcal L_{\mathrm{IPO}}=\mathbb E\Big[\Big(\log\tfrac{\pi_\theta(y_w)\,\pi_{\mathrm{ref}}(y_l)}{\pi_\theta(y_l)\,\pi_{\mathrm{ref}}(y_w)}-\tfrac{1}{2\tau}\Big)^{2}\Big], $$

pulling the log-ratio gap to a finite target \(1/(2\tau)\) (Eq. 17)15; with \(\tau=0.1\) the target is 5 nats. The support is tabular, small bandit examples with two or three actions, not language models15 (§5.3-§5.4). At LLM scale the evidence is mixed: Rafailov et al. find IPO less prone to over-optimization, with the 6.9B model showing little or none10 (§3.1), while in SimPO's tuned comparison IPO trails DPO's AlpacaEval 2 length-controlled win rate in all four settings (11.8% against 15.1% on Mistral-Base)11 (Table 4). Lambert lists conservative DPO (a label-noise assumption) with IPO as overfitting remedies3 (§8.2).

4.6 Over-optimization and a narrow stable region

DPO has its own version of reward over-optimization (Module 6, Section 5). Rafailov et al. trained Pythia 1B, 2.8B and 6.9B on Reddit TL;DR with DPO, IPO and SLiC at seven values of \(\beta\), scoring by GPT-4 win rate against the dataset summary. Every objective shows a hump: win rate rises and then falls as the KL budget grows, and the Gao et al. functional form \(R(d)=d(\alpha-\beta_{\mathrm{fit}}\log d)\) in \(d=\sqrt{\mathrm{KL}}\) fits the points10 (§3.1-§3.2, Figure 1). The 1B model over-optimizes almost at once, and in the intra-epoch curves the best result arrives within the first 25% of an epoch for most settings, after which win rate declines while KL keeps rising (Figure 2).

Tulu 3's hyperparameter table is the practitioner's view of the same hump. Figure 6 plots every row: one algorithm, length-normalized DPO, spans 46.8 to 57.3 across six settings, 10.5 points, and only two settings (56.1 and 57.3) beat the SFT checkpoint's 55.7. Moving from three epochs to one at \(\beta=5\) was worth 3.9 points (53.4 to 57.3)2 (§5.4.1, Table 18).

Tulu 3 hyperparameter and algorithm search on an early SFT checkpoint with UltraFeedback: average score for SimPO, DPO, PPO and length-normalized DPO settings, against the SFT base of 55.7 44 46 48 50 52 54 56 58 SFT base 55.7 SFT base 55.7 SimPO β=2, γ/β=0.5, 1 epoch 51.8 SimPO β=10, γ/β=0.3, 1 epoch 52.9 DPO β=0.1, 3 epochs 55.2 PPO β=0.0325, 1 epoch 54.5 PPO β=0.05, 1 epoch 55.5 DPO-norm β=5, lr 1e-7, 3 epochs 56.1 DPO-norm β=10, 3 epochs 55.2 DPO-norm β=15, 3 epochs 55.7 DPO-norm β=2, 3 epochs 46.8 DPO-norm β=5, 3 epochs 53.4 DPO-norm β=5, 1 epoch 57.3 Tulu 3 average score (axis starts at 44)
Figure 6. Average score for each configuration in Tulu 3's preference-tuning search (UltraFeedback, early SFT checkpoint; Table 182). The dashed line is the SFT base, 55.7. Length-normalized DPO (orange) ranges from 46.8 to 57.3 depending on \(\beta\) and epochs. DPO, SimPO and length-normalized DPO rows differ also in learning rate and batch size, as listed in the source table.

5. Variants: what each one changes

5.1 One table, with the same benchmark where it exists

Most variants change one of four things: the form of the loss on the log-ratio gap, whether a reference model is needed, how length enters, or what the data must look like. Table 2 lines them up. The two numeric columns come from the one paper that tuned DPO, IPO, KTO, ORPO and SimPO on the same data and judge (SimPO's Table 4). They are chat-benchmark numbers (AlpacaEval 2, length-controlled win rate) and say nothing about reasoning or safety.

Table 2. DPO and its variants. LC = AlpacaEval 2 length-controlled win rate (%) from the SimPO paper's tuned comparison, Mistral-7B-Base and Llama-3-8B-Base after UltraChat SFT and UltraFeedback preference tuning11 (Table 4, Table 15). "n/r" = not reported there. Loss definitions: DPO1, IPO15, ORPO16, KTO7, DPOP12, NLL term6.
MethodWhat changesDataReferenceLengthLC MistralLC Llama-3
SFT startnonedemonstrationsn/an/a8.46.2
DPOlogistic loss on summed log-ratio gappairsyes (cacheable)none; sums grow with length15.118.2
DPO, length-normalizeddivide each log-ratio by \(|y|\)pairsyesper-token mean21.0n/r
R-DPOadds a length term \(\alpha(|y_w|-|y_l|)\) to the marginpairsyesexplicit penalty17.417.6
DPO + NLLadds \(\alpha\cdot\mathrm{NLL}(y_w)\)pairsyesnonen/rn/r
DPOPpenalizes chosen log-ratio below 0 (\(\lambda=50\))pairsyesnonen/rn/r
IPOsquared loss to gap \(1/(2\tau)\)pairsyesnone11.814.4
KTOsigmoid utility of the reward against a KL reference pointbinary labelsyesnone13.114.2
ORPOSFT loss + log-odds-ratio term on per-token mean probabilitiespairsnoper-token mean14.712.2
SimPOmean log-prob as reward, margin \(\gamma\)pairsnoper-token mean21.522.0

The loss changes that act on the gradient weight (IPO, DPOP, the NLL term) target the failure modes of Section 4 and are mostly evaluated on those, not on chat win rates; the changes that act on the reward definition (length normalization, SimPO's reference-free reward, ORPO) are the ones that move AlpacaEval 2; and KTO changes the data contract.

5.2 KTO: learning from a binary signal

KTO starts from a different question: does the feedback have to be a pair? Ethayarajh et al. recast alignment losses as human-aware losses (HALOs), prove that DPO and PPO-Clip belong to the family, and derive a member from the Kahneman-Tversky value function7 (§3, Theorem 3.5, §4.1). With \(r_\theta=\log\frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}\) and a reference point \(z_0=\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\), the loss is \(\mathbb E[\lambda_y-v(x,y)]\) with

$$ v(x,y)=\begin{cases}\lambda_D\,\sigma\big(\beta(r_\theta-z_0)\big)&y\text{ desirable}\\ \lambda_U\,\sigma\big(\beta(z_0-r_\theta)\big)&y\text{ undesirable.}\end{cases} $$

Every example is a single (prompt, response, label) triple, so n preference pairs become 2n examples, and a 1:10 imbalance of desirable to undesirable examples is handled by setting \(\lambda_D n_D/\lambda_U n_U\in[1,3/2]\). Because sampling from \(\pi_\theta\) is slow, \(z_0\) is estimated from mismatched (prompt, response) pairs within a microbatch of at least two, clamped at zero, with no gradient through it7 (§4.1-§4.2). The recommended start is \(\beta=0.1\), batch size 8 to 128 and a learning rate of \(5\times10^{-6}\) with AdamW, 2 to 10 times DPO's, because the reference-adjusted reward is smaller7 (§4.2).

The reported results are strong on the authors' benchmarks: SFT+KTO matches SFT+DPO from 1B to 30B on GPT-4-judged win rate, matches DPO with up to 90% fewer desirable examples, and on Zephyr-β-SFT with UltraFeedback raises GSM8K from 40.0 to 53.5 and BBH from 44.1 to 52.6 over DPO (Table 2)7 (§4.3). The authors' explanations: an example's gradient vanishes as its implied reward goes to either extreme (Proposition 4.1), so mislabeled and intransitive feedback is effectively ignored, and under contradictory preferences DPO can favor the minority answer when the reference is unaligned (Theorem 4.3). With low-noise, transitive preferences they expect DPO to do better, since KTO risks underfitting7 (§4.4-§4.5). The v5 revision of 8 September 2026 changed proofs and wording, not the empirical results7 (Acknowledgements).

5.3 Reference-free variants, and what independent comparisons found

SimPO defines the reward as the length-normalized average log-probability, \(r(x,y)=\frac{\beta}{|y|}\log\pi_\theta(y|x)\), and trains \(-\log\sigma(r_w-r_l-\gamma)\) with a target margin \(\gamma\) and no reference model11 (§2.2-§2.3, Eq. 6), with tuned \(\beta\in\{2,2.5\}\) and \(\gamma\) between 0.3 and 1.6 (Table 8). It reports gains over DPO of up to 6.4 points on AlpacaEval 2 and 7.5 on Arena-Hard, notes that a higher learning rate lifted chat scores at the cost of forgetting on GSM8K and MMLU, and shows every baseline other than R-DPO trailing DPO's length-controlled win rate in all four settings (abstract, Table 4, Appendix J). ORPO folds SFT and an odds-ratio term into one reference-free stage, \(\mathcal L=\mathcal L_{\mathrm{SFT}}+\lambda\,\mathcal L_{\mathrm{OR}}\) with \(\mathcal L_{\mathrm{OR}}=-\log\sigma\big(\log\frac{\mathrm{odds}_\theta(y_w)}{\mathrm{odds}_\theta(y_l)}\big)\)16 (§4.1-§4.2), and reports 11.33% and 12.20% on AlpacaEval 2.0 for two Mistral-7B models after one epoch on UltraFeedback (§5-§6.1).

Independent runs are less generous. Tulu 3 tried SimPO at two settings and got 51.8 and 52.9 against an SFT base of 55.7, and found that only length-normalized DPO beat the base overall2 (§5.4.1, Table 18). The KTO paper's own comparison puts ORPO (\(\lambda=0.1\)) below the SFT starting point on GSM8K, 36.5 against 39.07 (Table 2). Lambert's reading is that the algorithm is a second-order choice: "the choice of algorithm is far less important than the initial model and the data used"3 (§8.2). What survives these sources: start with DPO; use the length-normalized form when chosen responses are systematically longer; add the NLL term when chosen likelihood falls or the task is reasoning-like; reach for KTO when labels are binary; treat SimPO and ORPO as methods that need their own hyperparameter search, not drop-in upgrades.

6. Offline versus on-policy preference data

6.1 When the data stop describing the policy

DPO's pairs are written before training starts, by the SFT model or a pool of other models; as the policy moves, they describe a distribution it no longer samples from. Xu et al. isolate the effect on SafeRLHF with Llama-2-7B trained on Alpaca: DPO lifts the safe-response rate from 41.6% to only 55.4% where PPO reaches 99.5%; SFT on the safe responses first, so the reference sits closer to the data, lifts DPO to 71.8%; and regenerating responses, labeling them with a learned reward model, and repeating with the latest DPO model as reference ("DPO-Iter") reaches 86.7% after one round and 99.9% after three8 (§4.3, Table 2). Table 3 gives all rows, including the cost: DPO-Iter's helpfulness change stays near −3 while PPO's is +1.69.

Table 3. Xu et al.'s SafeRLHF experiment, Llama-2-7B trained on Alpaca then aligned. Helpfulness and harmfulness are reward-model score changes (higher helpfulness is better, lower harmfulness is better); "+" rows are labeled as in the source8 (Table 2).
SettingΔ helpfulnessΔ harmfulnessSafe rate (%)
SFT (Alpaca)−2.621.5041.6
PPO1.69−12.0899.5
DPO−4.19−0.9755.4
DPO, + SFT (Safe) as reference−1.62−3.5071.8
DPO, + filter dual-unsafe pairs2.46−4.8880.8
DPO, + filter dual-safe pairs−2.86−6.8295.8
DPO-Iter, round 1−3.22−5.2386.7
DPO-Iter, round 2−3.27−8.8399.7
DPO-Iter, round 3−3.26−10.2199.9
DPO-Iter, round 4−2.96−11.0799.9

Two further studies agree. Tajwar et al. find across bandit, synthetic and full-scale problems that on-policy sampling and a negative gradient (pushing probability down on bad responses, as DPO's contrastive term does) each beat offline maximum likelihood and are complementary; on-policy DPO and IPO converge faster and to better solutions than offline DPO and on-policy RL in their synthetic length tasks, though they stop short of saying on-policy DPO always beats PPO13 (abstract, §5.1-§5.3). The benefit concentrates where the reward peak lies in a low-probability region of the reference; where it lies in a high-probability region, offline supervised methods can suffice. Tang et al. measure at equal KL budget in a Gao-style gold-reward setup (Module 6) and find online algorithms a Pareto improvement over offline ones, with the gap surviving contrastive and non-contrastive losses and not closed by scaling the policy; offline policies were better pairwise classifiers but worse generators, and the one working fix was generating the data close to the starting policy17 (abstract, §1, §6).

6.2 Llama 3: iterate, and use only the newest batches

Llama 3 is the large-scale answer. Each of six rounds collects fresh preference annotations by sampling two responses from two different models, with the best models of the previous round doing the sampling; reward-model training uses all preference data collected so far, while DPO uses only the newest batches. The report's reasoning: "As a result, our training data conforms better to the distribution of the policy model that is being optimized in each round"4 (§4.1.4, §4.1.6, §4.2.1). Models are also averaged within each stage (§4.1.5). The team also tried PPO and found that DPO "required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval"4 (§4.1.4). The report gives no PPO numbers.

Offline DPO on a fixed dataset against Llama 3's six-round loop in which each round's preference data are sampled from the best models of the previous round Offline DPO: one fixed dataset pairs from π_SFT or a pool of other models DPO, 1-3 epochs reference = π_SFT policy the policy never writes its own training data Llama 3: six rounds, each sampled from the latest models sample two responses from two different models annotators rate strength, then edit keep significantly better and better pairs SFT, then DPO on the newest batches average models round k+1 samples from the best round-k models: ×6 Reward-model training uses all preference data collected so far; DPO uses only the newest batches. PPO was tried and dropped: DPO needed less compute at scale and did better on IFEval.
Figure 7. Offline DPO (top) and the iterated loop Llama 3 describes (bottom). The loop is not online RL: each round's data are collected in a batch and used for many steps; what changes is that the sampling policy tracks the training policy at the granularity of a round. Sources: Llama 3 §4.1.4, §4.1.6, §4.2.14.

6.3 The contrast matters more than either response

On-policy pairs are one theory of what makes DPO data work. Olmo 3 and SmolLM3 support another, delta learning: the quality of a pair depends mainly on the gap between chosen and rejected, not on either response alone. Both took chosen responses from Qwen3-32B and rejected ones from Qwen3-0.6B, developed concurrently and independently3 (§8.4). Olmo 3 shows why the framing is useful. Continued SFT of a development 7B SFT checkpoint on the Qwen3-32B chosen responses lowered the 10-benchmark average from 70.3 to 64.5; DPO preferring those same responses over Qwen3-0.6B's raised it to 72.9, 8.4 points above SFT on the same chosen data (derived)5 (§4.5, Table 21). The authors conjecture that the 32B responses are weaker than what the model has already seen, so imitation no longer helps while the contrast still does. A single 7B run then found DPO a better start for RLVR than SFT alone: 74.1 for SFT+DPO+RLVR against 71.9 for SFT+RLVR (Table 22)5 (§4.5).

Olmo 3 delta learning: average benchmark score of the chosen and rejected generators, the 7B SFT checkpoint, continued SFT on the chosen responses, and DPO with chosen over rejected 0 20 40 60 80 100 Qwen3 32B (chosen) 83.2 Qwen3 0.6B (rejected) 35.1 Dev. 7B SFT checkpoint 70.3 continued SFT on the chosen 64.5 DPO: 32B chosen over 0.6B rejected 72.9 average over 10 reasoning, knowledge and instruction benchmarks (0-100)
Figure 8. Olmo 3 development 7B SFT checkpoint, average over ten benchmarks (MMLU, BBH, GPQA, Zebra, AGI, AIME 2025, AIME 2024, CHE, LCB, IFEval, with the source's column names). Imitating the Qwen3-32B responses hurts the 70.3 checkpoint; preferring them over Qwen3-0.6B's helps. Data: Table 215. Module 31 treats the full Olmo 3 recipe.

The same logic, with a verifier choosing the pairs, is Iterative Reasoning Preference Optimization: sample 30 chain-of-thought solutions per problem, form ten winner-loser pairs per problem with correct answers as winners and incorrect ones as losers, train DPO plus NLL, and repeat from the new model. On GSM8K from Llama-2-70B-Chat the four iterations score 73.1%, 78.0%, 81.1% and 81.6%, against 55.6% zero-shot and 63.5% for SFT on gold solutions; the per-iteration gains of 17.5, 4.9, 3.1 and 0.5 points show saturation on a fixed prompt set6 (§3.1, Table 1). This is offline-in-rounds learning from a binary outcome signal. Modules 9 and 10 treat the online version, in which the policy is updated on every batch of its own samples.

7. DeepSeekMath's view: one gradient, different data and coefficients

DeepSeekMath writes SFT, rejection-sampling fine-tuning (RFT), DPO, PPO and GRPO as one gradient, \(\nabla_\theta\mathcal J_{\mathcal A}=\mathbb E_{(q,o)\sim\mathcal D}\big[\tfrac1{|o|}\sum_t GC_{\mathcal A}(q,o,t,\pi_{rf})\,\nabla_\theta\log\pi_\theta(o_t|q,o_{<t})\big]\), and says the methods differ in three things: the data source \(\mathcal D\), the reward function \(\pi_{rf}\), and the algorithm that turns reward into the gradient coefficient \(GC\)18 (§5.2.1, Eq. 5). Table 4 reproduces their summary for the rows relevant here. DPO's coefficient is the pair weight of Section 2.4, \(\sigma(\hat r_l-\hat r_w)\) (their Eq. 14), and their Eq. 12 writes the DPO objective with \(1/|o|\) averaging of the log-ratios, i.e. in the length-normalized form of Section 3.2.

Table 4. Data source, reward and gradient coefficient for the methods of this module and its neighbours, from DeepSeekMath's Table 1018. \(P_{sft}\) is the SFT question distribution; \(\pi_{sft}\) the SFT model; \(\pi_\theta\) the policy being trained.
MethodData sourceRewardGradient coefficientDataSign
SFT\(q,o\sim P_{sft}\)none (human selection)1offlinepositive
RFT\(q\sim P_{sft},\ o\sim\pi_{sft}\)rule\(\mathbb 1[\text{correct}]\)offlinepositive
DPO\(q\sim P_{sft},\ o^+,o^-\sim\pi_{sft}\)preference (rule in math)\(\sigma(\hat r_l-\hat r_w)\), Eq. 14offlinesigned pair
Online RFT\(q\sim P_{sft},\ o\sim\pi_\theta\)rule\(\mathbb 1[\text{correct}]\)onlinepositive
PPO\(q\sim P_{sft},\ o\sim\pi_\theta\)reward modeladvantage, Eq. 18onlinesigned
GRPO\(q\sim P_{sft},\ \{o_i\}\sim\pi_\theta\)reward modelgroup-relative advantage, Eq. 21onlinesigned

The last two columns are added here, and they organize Section 6: the two ingredients Tajwar et al. found complementary, on-policy sampling and a negative gradient, are the two axes. RFT is offline and non-negative, Online RFT online and non-negative, DPO offline and signed, PPO and GRPO online and signed. DeepSeekMath's measurements fall on the same axes: Online RFT matches RFT early, when "the actor and the SFT model exhibit close resemblance", and pulls ahead as the policy's samples diverge; GRPO beats Online RFT because its coefficient penalizes incorrect answers while Online RFT "does not penalize incorrect responses"18 (§5.2.1, Figure 5, DeepSeekMath-Instruct 1.3B). "Online DPO", which samples pairs from the current policy and labels them, fills the remaining cell, online and signed with a pairwise coefficient3 (§8.2). GRPO itself is Module 9.

8. The evidence ledger: what supports "DPO is enough"

Module 5 Section 9.2 (Table 6) lists the head-to-head PPO and DPO-family studies one row each. Table 5 is organized by claim, so that the evidence on both sides can be read together. Cost has the least dissent: 28 hours against 4 hours in Tulu 3's controlled setup is 7 times (derived), or 14 times in node-hours2 (§5.4.1), and Llama 3 reports less compute for DPO at its scale4 (§4.1.4).

Table 5. Claims about DPO against the evidence for and against them. Numbers are as reported; setups differ, so compare within a row only.
ClaimEvidence forEvidence against or limiting
DPO matches or beats PPO-RLHF on preference tuningTL;DR with GPT-J: DPO about 61% win rate at temperature 0.0 against PPO's 57% at its best temperature; human raters preferred DPO samples 58% of the time1 (§6.2). Llama 3 found DPO better on IFEval4. Tulu 3: PPO reached a comparable but slightly lower average in an untuned setup2 (§5.4.1).Xu et al.: PPO ahead on every testbed, from dialogue to code8. Ivison et al.: PPO ahead by up to 2.5% on math and 1.2% in general domains9. The paper's own authors note that β was not meaningfully tuned and models were ≤ 6B1 (§6.2, §7).
DPO generalizes out of distribution as well as PPOSummarization policies moved to CNN/DailyMail: GPT-4 win rate against ground truth 0.36 (DPO) against 0.26 (PPO) at temperature 0, and 0.31 against 0.23 at 0.251 (§6.3, Table 1).Distribution shift between reference and pairs is DPO's weak point: 55.4% against PPO's 99.5% safe rate until the mismatch is repaired8 (Table 2); online algorithms are a Pareto improvement at equal KL17.
DPO suffices for tasks with verifiable outcomesDPO plus NLL on verifier-labeled pairs lifts GSM8K from 55.6% to 81.6% over four iterations6 (Table 1). Olmo 3 reports DPO before RLVR as a better start than SFT alone5 (Table 22).CodeContests test 10@1k with CodeLlama-34B: SFT 15.2%, DPO 0.0%, DPO-Iter 3.2%, PPO with test-case rewards 22.4%8 (Table 8). Tulu 3 kept DPO for preference iteration and used PPO for RLVR2 (§5.4.1). Iterative RPO's gains shrink to 0.5 points by round four6.
Online or on-policy data are necessaryTajwar et al., Tang et al. and Xu et al. (Section 6.1); DPO-Iter closes most of the safety gap in Table 3.Delta learning gains came from fixed pairs generated by other models5; Llama 3's rounds are batch-collected, not online4. Tajwar et al. say offline supervised methods can suffice when the reward peak lies in a high-probability region of the reference13.
A better variant loss improves resultsSimPO: up to +6.4 AlpacaEval 2 and +7.5 Arena-Hard over DPO11. KTO: +13.5 GSM8K over DPO on Zephyr-β-SFT7.Tulu 3: only length-normalized DPO beat the base2. In SimPO's own table most variants trail DPO. Data quality ranks above the algorithm9.

The ledger has a shape. DPO's advantages are cost and simplicity, and they are real. The evidence against it is concentrated in two regimes: when the reference and the pairs are mismatched, and when the objective is a hard verifiable outcome. In the regime where most DPO is actually used, chat-style preference tuning from a strong SFT checkpoint with well-built pairs, the differences between DPO and tuned PPO are a few points, and Ivison et al. rank them below the effect of the preference data9.

Where experts disagree

The primary disagreement is whether offline DPO is enough or online RL is needed for preference tuning. Three more sit under it: whether the loss or the data is the lever, whether a falling chosen likelihood is a defect, and whether the variants beat DPO at all (Section 5.3). The studies below use different stacks, which is part of the problem.

DPO-family methods are the right default

Llama 3 team, Tulu 3 authors, Lambert's practical advice423

Llama 3 chose a simple post-training loop over RL algorithms that "tend to be less stable and harder to scale", tried PPO, and kept DPO for compute and IFEval reasons4 (§1, §4.1.4). Tulu 3 measured PPO at about 7 times the wall-clock for a comparable average and spent the savings on data iteration2 (§5.4.1). Lambert advises starting with direct alignment on synthetic preferences3 (§8.4).

  • Seven times cheaper in wall-clock in the one controlled measurement; stable at 405B.
  • Fast data iteration, with cacheable reference log-probs.
  • Llama 3 publishes no PPO numbers and Tulu 3's PPO was untuned, so neither shows a ceiling.
  • Both recipes carry patches (NLL term, token masking, length normalization, one epoch), so "simple" describes the loss, not the recipe.

Online RL wins, and the gap is structural

Xu et al., Tang et al., Tajwar et al.; Lambert's book817133

The loss is flat off the data's support (Section 4.1), the pairs go stale as the policy moves (Section 6.1), and hard verifiable tasks expose both: DPO scored 0.0% on CodeContests where PPO scored 22.4%8 (Table 8). Tang et al. find a Pareto gap at equal KL that scaling the policy does not close17. Lambert writes that "despite the simplicity of alternatives, RL-based methods continue to win out"3 (§3.1.6).

  • Theory (a proper-subset result in Xu et al.) and experiments agree across three groups.
  • Fits why Tulu 3 and Olmo 3 end their pipelines with RLVR.
  • The strongest numbers pair a heavily tuned PPO with a DPO lacking an NLL term, length control or iteration.
  • Tang et al. use a gold-reward synthetic setup; no source here has a matched comparison at 70B and up on human preferences.

The loss is second-order; the pairs are the lever

Ivison et al., Olmo 3 authors, Lambert953

Ivison et al. find preference data quality gives the largest gains, ahead of the algorithm, the reward model and extra prompts9. Olmo 3 turns a response set that hurts under SFT into a gain under DPO by choosing the rejected side5 (Table 21). Lambert writes that the algorithm matters far less than the initial model and the data3 (§8.2).

  • Consistent across independent groups and the variant comparisons of Section 5.3.
  • Actionable: build pairs with a large capability delta or from on-policy samples.
  • "Good data" has two competing definitions, on-policy (Tulu 3) and large delta (Olmo 3, SmolLM3), which Lambert presents as separate theories3 (§8.4).
  • No study here varies data and algorithm jointly at scale.

A falling chosen likelihood is a bug to patch, or it is expected

Pal et al., Razin et al., Pang et al., Llama 3 against Rafailov et al.1214610

The first group patches it (NLL term, DPOP, CHES filtering) and shows harm: a refusal rate falling from 74.4% to 33.4%14, GSM8K at 61.8% without the NLL term against 73.1% with it6. Rafailov et al. argue that after SFT on the chosen responses a negative implied reward is the expected sign, and that win rate rises while it falls, until the over-optimization hump10 (§3.5).

  • The patch is cheap, and Llama 3 and Pang et al. report it helps.
  • The skeptical view fits the DPO objectives of Tulu 3 and Olmo 3, which carry no NLL term in the cited sections.
  • The harm is shown for near-duplicate pairs (Pal et al.) and refusal data (Razin et al.).
  • Rafailov et al. measured TL;DR summarization at 1B to 6.9B; the NLL gains are on a reasoning task.
Assessment

Both camps are right inside their regimes. For chat, instruction-following and safety tuning from a strong SFT start with well-built pairs, a patched DPO is the efficient default: length normalization when chosen responses run long, token masking, one epoch, early stopping judged on generations, and an NLL term when the chosen log-ratio falls and the task is reasoning-like. The measured gap to tuned PPO there is a few points and ranks below the effect of the data9. Online or verifier-driven RL wins where the target is a hard verifiable outcome, where reference and pairs cannot be kept matched by iteration, or where the goal is to move beyond the support of the data. Chosen-likelihood decline is a monitor, not a verdict: log it, and patch when downstream behavior degrades in the way Razin et al. describe. What would settle the primary question is a matched-compute, matched-data comparison with both sides tuned, at 70B or larger, scored on held-out human preferences and on verifiable tasks, reporting KL and the chosen-likelihood trace. No source here contains one. Module 33 turns this into a decision procedure.

What is changing

DPO's role has moved from "the alignment method" to one stage in a longer pipeline. Olmo 3 runs SFT, then length-normalized DPO on delta-learning pairs, then RLVR, and reports DPO as the better start for RL5 (§4.5). Lambert expects that as reasoning models are trained mainly with RL, investment will return to RL for preference tuning, widening the margin over direct alignment3 (§8.5). Whether a DPO stage expands what the model can solve, as Olmo 3 claims from improved pass@k on AIME5 (§4.5, Figure 20), or only sharpens it, is the question of Module 20. The critic-free online methods of Modules 9 to 12 occupy the cell Section 7 left open: online data with a signed coefficient and no value model. The evidence here is dated: the controlled comparisons are from 2023 to 2024, the production reports from 2024 to 2025, and the KTO paper was revised as late as September 2026, so treat any single number as a snapshot of one stack.

References

  1. paperRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023; arXiv v3, 29 Jul 2024. https://arxiv.org/pdf/2305.18290v3. Accessed 2026-10-26. Used for: Eqs. 3-7 and the derivations in Appendix A.1-A.6; Lemmas 1-2 and Theorem 1 (§5.1); gradient (Appendix A.4); defaults and the PyTorch loss (Appendix B); TL;DR 61% against 57% and the human study (§6.2, §6.4); CNN/DailyMail transfer (Table 1); the unlikelihood baseline (Appendix C.3).
  2. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025 (v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: Length-normalized DPO (Eq. 6); the algorithm and hyperparameter search (§5.4.1, Table 18); final settings (Table 20); PPO about 28 h on two nodes against DPO about 4 h on one; reference log-prob caching and separate forward passes (§5.4.2, Figure 17); the 354,192-instance preference pipeline (§1, §5.2.1).
  3. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026; web edition built 14 Oct 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: Chapter 8: the implicit-reward reading (§8.1.1), variants and preference displacement (§8.2), reference caching (§8.3), delta learning and synthetic preferences (§8.4), DAAs against RL (§8.5); the RL stance in §3.1.6.
  4. paperGrattafiori, A., Dubey, A., Jauhri, A., et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024 (v1 31 Jul 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: Design stance (§1); DPO settings, token masking, NLL coefficient 0.2 and the PPO comparison (§4.1.4); six rounds (§4.1.6); preference-data strength levels and filtering (§4.2.1); response-length statistics (Table 6).
  5. paperOlmo Team (Ettinger, A., Bertsch, A., et al.). Olmo 3. arXiv v2, 14 Apr 2026 (v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: Delta learning and the Dolci Think DPO data (§4.3, §4.5); Tables 21 and 22; length-normalized DPO loss and hyperparameters (Appendix A.6.2, Table 48).
  6. paperPang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., Weston, J. Iterative Reasoning Preference Optimization. arXiv v3, 26 Jun 2024. https://arxiv.org/abs/2404.19733. Accessed 2026-10-27. Used for: The DPO plus NLL loss and iteration scheme (§2); hyperparameters and GSM8K results (§3.1, Table 1, Figures 2-3).
  7. paperEthayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D. KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024; arXiv v5, 8 Sep 2026. https://arxiv.org/abs/2402.01306. Accessed 2026-10-27. Used for: HALO definition and Theorem 3.5 (§3); the KTO loss and KL estimate (§4.1); hyperparameters (§4.2); results, Table 2 and Figure 4 (§4.3); Proposition 4.1, Theorem 4.3 and the DPO-or-KTO guidance (§4.4-§4.5). Full text read; the source map had the abstract only.
  8. paperXu, S., Fu, W., Gao, J., Ye, W., et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: Three-action counterexample and the proper-subset theorem (Table 1, §4.1); SafeRLHF distribution-shift and DPO-Iter results (Table 2, §4.3); APPS and CodeContests results (Tables 7-8).
  9. paperIvison, H., Wang, Y., Liu, J., Wu, Z., et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv v2, 7 Oct 2024. https://arxiv.org/abs/2406.09279. Accessed 2026-10-26. Used for: Ranking of factors (data, algorithm, reward model, prompts) and the PPO margins of 2.5% (math) and 1.2% (general), from the abstract.
  10. paperRafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., Niekum, S. Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS 2024; arXiv v2, 5 Nov 2024. https://arxiv.org/abs/2406.02900. Accessed 2026-10-27. Used for: Hump-shaped over-optimization in DPO, IPO and SLiC (§3.1-§3.2, Figures 1-2); implicit-reward accuracy against win rate (§3.4); decreasing-likelihood argument (§3.5); null-space argument, Proposition 1 and Figure 7 (§4).
  11. paperMeng, Y., Xia, M., Chen, D. SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS 2024; arXiv v3, 1 Nov 2024. https://arxiv.org/abs/2405.14734. Accessed 2026-10-27. Used for: The reward and objective (§2.2-§2.3, Eq. 6); Table 4 (tuned baselines); Table 6 and Figure 4 (length and likelihood mismatch); Appendices B, I (Table 15, DPO with length normalization) and J.
  12. paperPal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., White, C. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv v2, 3 Jul 2024. https://arxiv.org/abs/2402.13228. Accessed 2026-10-27. Used for: Preferred-likelihood reduction result and the token-level account (§3, Figure 1); the DPOP loss (Eq. 3) and settings beta 0.3, lambda 50 (§4-§5).
  13. paperTajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., Kumar, A. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. arXiv v3, 2 Jun 2024. https://arxiv.org/abs/2404.14367. Accessed 2026-10-27. Used for: On-policy sampling and negative-gradient takeaways (§5.1-§5.3); induced-reward behavior under DPO (Figures 16-17).
  14. paperRazin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., Hanin, B. Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. ICLR 2025; arXiv v4, 27 Apr 2025. https://arxiv.org/abs/2410.08847. Accessed 2026-10-27. Used for: Definition of likelihood displacement, the CHES score, the refusal-rate drop from 74.4% to 33.4% and the filtering result (abstract).
  15. paperAzar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., Munos, R. A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv v2, 22 Nov 2023. https://arxiv.org/abs/2310.12036. Accessed 2026-10-27. Used for: Overfitting under deterministic preferences (§4.2); the IPO loss (Eq. 17); bandit experiments (§5.3-§5.4).
  16. paperHong, J., Lee, N., Thorne, J. ORPO: Monolithic Preference Optimization without Reference Model. arXiv v2, 14 Mar 2024. https://arxiv.org/abs/2403.07691. Accessed 2026-10-27. Used for: The odds-ratio objective and per-token mean probability (§4.1-§4.2); Mistral-ORPO AlpacaEval 2.0 results.
  17. paperTang, Y., Guo, D., Zheng, Z., Calandriello, D., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv v1, 14 May 2024. https://arxiv.org/abs/2405.08448. Accessed 2026-10-27. Used for: Online against offline at equal KL budget; hypotheses on coverage, data quality, classification, loss form and scale (abstract, §1, §3, §5, §6).
  18. paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: Unified gradient form and Table 10 (§5.2.1, Eq. 5); the DPO objective and coefficient (Appendix A.1.4, Eqs. 12-14); Online RFT against RFT and GRPO (Figure 5).