Reinforcement Learning for LLM Post-Training · Module 4
Reward Models from Preferences: the Bradley-Terry Loss, Architecture and Scale, and Outcome versus Process Supervision
A reward model is a language model with a scalar head, trained so that the difference of its scores on two completions is the log-odds that a human prefers the first. This module derives that loss and its gradient from the primary sources, bounds what labeler agreement lets any reward model reach, and covers how the model is built, scaled and trained. It then sets outcome against process supervision with the measured evidence, so you can write the loss, say which questions a scalar score cannot answer, and explain why 2026 recipes keep learned reward models for some jobs and drop them for others.
1. Preference data: what the model is fit to
A reward model is trained on comparisons, not on absolute scores. Christiano et al. set the template for deep RL: show a human two trajectory segments, record which one they prefer, and treat "equally good" as a uniform label and "cannot compare" as a dropped example3 (§2.2.2). Stiennon et al. carried the same format to language, collecting 64,832 summary comparisons on TL;DR and noting that they chose comparisons over Likert scores4 (§1, §3.3, footnote 6).
1.1 Pairs, rankings and strength labels
The atomic record is a prompt \(x\), a preferred completion \(y_w\) and a rejected completion \(y_l\)2 (§3). How the pairs are produced differs by lab:
- Rankings expanded to pairs (InstructGPT). Labelers rank \(K\) between 4 and 9 responses per prompt, which yields \(\binom{K}{2}\) comparisons per prompt. Ties were dropped, and the RM training set held 33,207 prompts1 (§3.5, Appendix C.2, Table 9).
- Two responses plus a strength label (Llama 3). Annotators compare two responses sampled from two different models and mark the preference as significantly better, better, slightly better or marginally better. For some prompts they also edit the chosen response, which creates a ranking edited > chosen > rejected. Both reward modeling and DPO train only on pairs labeled significantly better or better and discard pairs with similar responses5 (§4.2.1).
- Ratings turned into pairs. With 1-5 ratings per completion, a common construction pairs the highest-rated completion with a random lower-rated one, the way UltraFeedback-style datasets are built6 (§11.2.2).
The chosen response is better relative to the alternatives shown, not globally correct: both responses in a pair can be wrong, and the model still learns from a well-labeled ordering6 (§11.2).
InstructGPT's batching rule is a systems decision with an algorithmic reason. Comparisons from one labeling task are strongly correlated, and shuffling all pairs into one dataset made the RM overfit within a single pass. Training on all \(\binom{K}{2}\) comparisons of a prompt as one batch element fixed the overfitting and needed one forward pass per completion rather than \(\binom{K}{2}\) passes1 (§3.5), the factor \(K-1\) in Figure 2. Llama 3 concatenates the prompt with two or three shuffled responses into one row, which its ablations showed improved training efficiency without losing accuracy5 (§4.1.2).
Each lane is a prefill-shaped computation: one forward pass over prompt plus response, no sampling (Module 1 prices that pass inside the RL loop). The prompt is still re-encoded in every completion's row unless the implementation shares the prefix, an optimization the sources do not describe.
1.2 Labeler agreement is the ceiling on pair accuracy
Human labels on language comparisons are noisy, and the noise level is measured. InstructGPT reports that training labelers agree with each other 72.6 ± 1.5% of the time and held-out labelers 77.3 ± 1.3%, against 73 ± 4% researcher-researcher agreement in the earlier summarization work1 (§3.4). Stiennon et al. give the same picture with more structure: labelers agree with each other 72% of the time on the training corpus, researcher-researcher agreement runs from about 65% on the hardest comparisons (two high-temperature samples from one RL policy) to about 80% on the easiest and about 95% when the researchers discussed the pair, and taking the modal label of three labelers lifts agreement with researchers from 72% to 77%4 (Appendix C.2). The RewardBench authors summarize older validation sets such as Anthropic HH and TL;DR as having accuracy ceilings between 60 and 70% because of inter-annotator disagreement7 (§1).
Reward-model accuracies sit inside that band. InstructGPT's 6B RMs, cross-validated across five labeler groups, reached 72.4 ± 0.4% on preferences of labelers in their training groups and 69.6 ± 0.9% on a held-out group1 (§4.1). Stiennon's 6.7B RM agreed with labelers 66.5% of the time on out-of-distribution CNN/DM summaries against an inter-labeler agreement of 66.9%4 (§4.3). A reward model that matches inter-labeler agreement predicts a human's choice about as well as one more labeler would.
Accuracy tracks preference strength
The ceiling is not uniform across pairs. Llama 2 splits its test pairs by the annotators' four-level strength label and reports accuracy per bucket8 (Table 8):
The authors read this as expected: modeling preferences between two similar responses is hard because of annotator subjectivity, and the annotation agreement rate is also higher on distinct responses8 (§3.2.2). Two consequences follow. A single headline accuracy hides a mixture of nearly deterministic pairs and near-coin-flips. And the pairs where the RM is most reliable are the separable ones, which fits Llama 3's decision to discard similar pairs5 (§4.2.1); the report states the filter but not its rationale.
A bound on what any RM can score (derived)
Agreement between labelers pins the achievable accuracy between two numbers. Suppose each pair has a latent probability \(q\in[\tfrac12,1]\) that a labeler picks its majority side, and labelers are exchangeable and independent given the pair. Two labelers agree with probability \(a=\mathbb{E}[q^2+(1-q)^2]\). An oracle that knows \(q\) and always predicts the majority side scores \(\mathbb{E}[q]\) against a single labeler's label. Then
$$ a \;\le\; \mathbb{E}[q] \;\le\; \frac{1+\sqrt{2a-1}}{2}. $$Why the two inequalities hold
Lower bound: \(q-\big(q^2+(1-q)^2\big)=(2q-1)(1-q)\ge 0\) for \(q\in[\tfrac12,1]\), so the oracle is at least as accurate as a second labeler pointwise. Upper bound: \(a=1-2\mathbb{E}[q]+2\mathbb{E}[q^2]\ge 1-2\mathbb{E}[q]+2\mathbb{E}[q]^2\) by Jensen's inequality; the right side increases in \(\mathbb{E}[q]\) on \([\tfrac12,1]\), and solving \(2m^2-2m+1-a=0\) for \(m\) gives the stated root. Equality in the upper bound holds when every pair has the same \(q\); equality in the lower bound holds when \(q\in\{\tfrac12,1\}\) (pairs are either coin flips or certain).
For InstructGPT's training labelers, \(a=0.726\) gives a ceiling between 72.6% and 83.6%; for held-out labelers, \(a=0.773\) gives 77.3% to 86.9% (derived from the agreement rates above1). Stiennon's three-label vote is a consistency check on where in that interval real data sits. With \(a=0.72\), the homogeneous case predicts that the modal label of three labelers agrees with another rater 78.2% of the time, and the coin-flip-or-certain case predicts 72%. The measured value, agreement with researchers, is 77%4, near the homogeneous end, which suggests that on that corpus most pairs are moderately noisy rather than a mixture of certain and hopeless ones. The check is approximate because it treats researchers and labelers as exchangeable.
Accuracy ceiling from labeler agreement
Formula: \( a \le \mathbb{E}[q] \le \tfrac{1}{2}\big(1+\sqrt{2a-1}\big) \). Derived above; assumes exchangeable, conditionally independent labelers. Set 72.6% for InstructGPT's training labelers1, 77.3% for its held-out labelers, 66.9% for Stiennon's CNN/DM inter-labeler figure4.
Christiano et al. build a noise floor into the model itself: they assume a 10% chance that the human answers uniformly at random, because "human raters have a constant probability of making an error, which doesn't decay to 0" as the reward difference grows3 (§2.2.3). Under that assumption no predictor can agree with the human more than \(0.9+0.1\times0.5=95\%\) of the time (derived). Plain Bradley-Terry as used for language-model RMs has no such floor; section 2.2 derives what label noise does to the optimal score gap.
1.3 Who produces the labels in 2026
Two facts shape current practice. First, on-policy matters: preference data collected on the current model family transfers better to RLHF than datasets pooled from unrelated models, and as of 2026 no open model ships with fully open human preference data and its collection methods; the largest recent releases are NVIDIA's HelpSteer line6 (§11.2). Second, AI labelers fill the gap. DeepSeek-R1's helpfulness RM was trained on 66,000 pairs that DeepSeek-V3 labeled: each pair judged four times with randomized order, averaged, and kept only if the score difference exceeded 1, with chosen and rejected responses length-matched across the dataset9 (§3.1). Module 8 covers AI feedback; the point here is that the Bradley-Terry loss does not care who produced \((y_w,y_l)\).
2. The Bradley-Terry model, its loss and its gradient
2.1 From pairwise choice to a logistic loss
Every reward model in this module descends from one assumption, quoted here from the sources that introduced it to language. DPO states it as the standard preference model of RLHF:
"The BT model stipulates that the human preference distribution \(p^*\) can be written as" \( p^*(y_1\succ y_2\mid x)=\dfrac{\exp(r^*(x,y_1))}{\exp(r^*(x,y_1))+\exp(r^*(x,y_2))} \). "Framing the problem as a binary classification we have the negative log-likelihood loss" \( \mathcal{L}_R(r_\phi,\mathcal{D})=-\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\big[\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)\big] \)2 (§3, Eq. 1-2).
InstructGPT, following Stiennon et al., writes the same loss with the K-way batching of section 1.1 and states what the scalar means:
"They use a cross-entropy loss, with the comparisons as labels—the difference in rewards represents the log odds that one response will be preferred to the other by a human labeler." The loss is \( \mathrm{loss}(\theta)=-\tfrac{1}{\binom{K}{2}}\,\mathbb{E}_{(x,y_w,y_l)\sim D}\big[\log\sigma\big(r_\theta(x,y_w)-r_\theta(x,y_l)\big)\big] \)1 (§3.5, Eq. 1).
The model itself is older. Bradley and Terry (1952) gave each item a latent strength; Christiano et al. identify the reward-difference form with the Elo rating system, where the gap in rating points estimates the probability that one player beats the other3 (§2.2.3). Lambert's derivation makes the reparameterization explicit: with strengths \(p_i=e^{r_i}\) the choice probability becomes \(P(i>j)=e^{r_i}/(e^{r_i}+e^{r_j})=\sigma(r_i-r_j)\), and "adding the same constant \(c\) to every \(r_k\) leaves \(P(i>j)\) unchanged"6 (§5.1, Eq. 12-13). The loss is the negative log-likelihood of the observed choices, i.e. logistic regression on the score difference. The softplus form \(\log(1+e^{r_l-r_w})\) used in some codebases is the same function6 (Eq. 17-18).
2.2 The gradient, derived
Write \(\Delta=r_\theta(x,y_w)-r_\theta(x,y_l)\) and \(\mathcal{L}(\Delta)=-\log\sigma(\Delta)=\log(1+e^{-\Delta})\). Using \(\sigma'(\Delta)=\sigma(\Delta)\sigma(-\Delta)\),
$$ \frac{d\mathcal{L}}{d\Delta}=-\frac{\sigma'(\Delta)}{\sigma(\Delta)}=-\sigma(-\Delta)=-\big(1-\sigma(\Delta)\big),\qquad \nabla_\theta\mathcal{L}=-\sigma(-\Delta)\,\big(\nabla_\theta r_\theta(x,y_w)-\nabla_\theta r_\theta(x,y_l)\big). $$Three properties matter for practice. First, the per-pair weight \(\sigma(-\Delta)\) lies in \((0,1)\): it is 0.5 at a tie, approaches 1 when the model orders the pair wrongly, and approaches 0 when it orders it correctly with a wide margin (Table 1). This is the same weight DPO's gradient carries, with the implicit reward in place of \(r_\theta\): "higher weight when reward estimate is wrong"2 (§4). Second, the gradient is bounded, so one mislabeled pair can push with weight at most 1 however confident the model is; the loss grows only linearly, \(\mathcal{L}\approx-\Delta\) for \(\Delta\ll0\). Third, the two scores receive equal and opposite signals, \(\partial\mathcal{L}/\partial r_w=-\sigma(-\Delta)\) and \(\partial\mathcal{L}/\partial r_l=+\sigma(-\Delta)\), which is what Figure 1 labels.
| Margin Δ | P(y_w ≻ y_l) | Loss | Gradient weight σ(−Δ) | Reading |
|---|---|---|---|---|
| −4 | 0.0180 | 4.0181 | 0.9820 | confidently wrong; weight saturates at 1 |
| −2 | 0.1192 | 2.1269 | 0.8808 | wrong |
| −1 | 0.2689 | 1.3133 | 0.7311 | wrong |
| 0 | 0.5000 | 0.6931 | 0.5000 | tie; loss is ln 2 |
| 1 | 0.7311 | 0.3133 | 0.2689 | right |
| 2 | 0.8808 | 0.1269 | 0.1192 | right |
| 4 | 0.9820 | 0.0181 | 0.0180 | learned; pair contributes under 2% of maximum weight |
One pair under Bradley-Terry
Formulas from section 2.2. The loss gradient is \( -\sigma(-\Delta)\,(\nabla r_w - \nabla r_l) \), so descent raises \( r_w \) and lowers \( r_l \) with equal force.
What the optimum looks like under label noise (derived)
Suppose a pair's label says \(y_w\) with probability \(p\). The expected loss \(-p\log\sigma(\Delta)-(1-p)\log\sigma(-\Delta)\) has zero gradient exactly where \(\sigma(\Delta)=p\), so the optimal margin is the log-odds \(\Delta^*=\ln\frac{p}{1-p}\). Take the two ends of the interval derived in section 1.2 for InstructGPT's training labelers1: \(p=0.726\) gives \(\Delta^*=0.97\) and \(p=0.836\) gives \(\Delta^*=1.63\). A well-fit reward model on this kind of data therefore produces score gaps of order one nat, not tens. Margins far above that on training pairs are memorization: each pair carries a single label, nothing in the data opposes pushing its gap beyond \(\Delta^*\), and every extra pass lets the network do so. This is a reading of the loss, not a claim from the sources, but it is consistent with what they report: InstructGPT's RM "quickly overfit" on multiple epochs with deteriorating validation loss, and it trained for one1 (Appendix C.2).
2.3 Shift invariance, and what it forces you to do
The loss depends only on \(r_w-r_l\), so \(\mathcal{L}(r_w+c,\,r_l+c)=\mathcal{L}(r_w,r_l)\). Because comparisons are always within one prompt, the invariance is per prompt: DPO's Lemma 1 states that two reward functions with \(r'(x,y)=r(x,y)+f(x)\) induce the same Bradley-Terry and Plackett-Luce preference distribution2 (§5.1, Appendix A.5). Two practical consequences.
- The reward's level is not trained. The head's bias receives gradient \(\partial\mathcal{L}/\partial r_w+\partial\mathcal{L}/\partial r_l=0\), so absolute scores are an accident of initialization. Pipelines that describe the step normalize after training: InstructGPT adds a bias so the labeler demonstrations score a mean of 0 before RL1 (§3.5); Stiennon et al. normalize so the reference summaries average 04 (§3.4); Gao et al. recenter every RM so the initial policy averages 0 and scale the gold RM to unit variance10 (§2.2); DPO describes the same practice as normalizing so \(\mathbb{E}[r_\phi(x,y)]=0\)2 (§3).
- Drift has to be controlled by something other than the loss. Llama 3's vision reward model adds "a weighted regularization term on the square of the reward logits averaged over the batch, which prevents the reward scores from drifting"5 (§7.5.4). Llama 2 saw a related effect from its margin loss, which pushed scores into a binary split; its authors flag reward calibration as future work because PPO can be sensitive to changes in the reward distribution8 (Appendix A.3.3).
The within-prompt invariance is also why group-relative RL fits RM scores naturally: DeepSeekMath observes that the group-relative advantage "aligns well with the comparative nature of rewards models, as reward models are typically trained on datasets of comparisons between outputs on the same question"11 (§4.1.1). A per-group baseline removes exactly the \(f(x)\) the RM never constrained (the writer's reading; Module 9 develops the estimator).
2.4 What a scalar Bradley-Terry score cannot represent
- Cross-prompt comparison. Section 2.3 means \(r(x_1,y_1)>r(x_2,y_2)\) carries no information unless the RM was separately calibrated; only within-prompt gaps are identified.
- Intransitive or incomparable preferences. A scalar score induces a total order on the completions of a prompt. Lambert lists the assumptions that fail in practice: measured preferences may not be transitive or comparable, and the number and presentation of options changes the answer6 (§10). Aggregating annotators with different tastes into one score is the usual source of such cycles (the writer's reading); the model fits the aggregate and cannot express the disagreement.
- Ties and uncertainty. InstructGPT dropped ties1 (Appendix C.2) and Christiano et al. map "equally good" to a uniform label3; neither leaves the trained model with an uncertainty estimate on its output.
- Length and style preferences it was not asked about. Stiennon's 6.7B RM preferred improving edits that make a summary shorter only 62.6% of the time, against 76.4% for humans, so it is biased toward longer summaries4 (§4.3). R1's pipeline length-matches chosen and rejected responses in its RM data for this reason9 (§3.1).
- Anything off the training distribution. The RM is fit to samples from the SFT policy or a close relative. DPO's description of why the RL objective carries a KL term is that it "prevents the model from deviating too far from the distribution on which the reward model is accurate"2 (§3). Module 6 quantifies what happens when it is not enough.
2.5 Variants of the pairwise loss
| Loss | Form | Needs | Reported use and effect |
|---|---|---|---|
| Pairwise BT | \(-\log\sigma(r_c-r_r)\) | (chosen, rejected) pairs | InstructGPT, Stiennon, Llama 2, Llama 3, Tulu 3148512 |
| K-way batched BT | \(-\binom{K}{2}^{-1}\sum\log\sigma(r_i-r_j)\), one batch element per prompt | rankings of 4 to 9 | InstructGPT; avoids overfitting from correlated pairs1 |
| Margin BT | \(-\log\sigma(r_c-r_r-m)\) | graded strength labels | Llama 2: margins {1, 2/3, 1/3, 0} (small) or {3, 2, 1, 0} (large); Meta Helpful accuracy 62.5% without margin, 63.0% small, 62.9% large8 (Tables 27-28). Llama 3 removed it after "diminishing improvements after data scaling"5 |
| Plackett-Luce | \(\prod_k \dfrac{e^{r_{\tau(k)}}}{\sum_{j\ge k}e^{r_{\tau(j)}}}\), equals BT at \(K=2\) | full rankings | Used in early Starling models6 (§5.5.3); DPO derives its objective for it as well2 (Appendix A.3) |
| Point-wise label | binary cross-entropy on a single response | safe/unsafe tags | R1's safety RM: 106,000 prompts, trained point-wise, unlike its pairwise helpfulness RM9 (§3.1) |
The margin result is worth reading carefully. A 0.4 to 0.5 point average gain from encoding annotator strength is small, it is largest on the most separable bucket ("significantly better" moved from 79.1% to 80.4% and 80.7%), and the Llama 3 team dropped the margin once data volume grew85. The plain pairwise logistic loss stays the default, in the writer's reading because no variant has beaten it by enough to justify the change; Lambert writes that the modifications "have not solidified into a single best practice"6 (§5.5).
3. Architecture, scale and training recipes
3.1 A language model with a scalar head
The reports that describe the architecture build the reward model the same way: take a decoder-only LM, remove the next-token output layer, and put a one-output projection on the hidden state of the last position. InstructGPT states it for both reward models and value functions: "the unembedding layer of the original model is replaced with a projection layer to output a scalar value"1 (Appendix C). Stiennon et al. add "a randomly initialized linear head"4 (§3.4); Llama 2 replaces the classification head for next-token prediction with a regression head and initializes from the chat checkpoint on the argument that "the reward model 'knows' what the chat model knows", which avoids an information mismatch that could favor hallucinations8 (§3.2.2). Tulu 3 reads the logit at the end-of-sequence token and initializes the head with weights drawn from \(\mathcal{N}(0,1/\sqrt{d_{\text{model}}+1})\)12 (§5.4.1). Lambert's reference implementation takes the last non-padding token's hidden state through a linear layer6 (§5.1.1-5.1.2).
Reading at the last position is forced by the causal mask: only the final token has attended to the whole prompt and response, so it is the one state that summarizes both. It also means a truncated response, which has no EOS, is scored from a state the RM never saw during training (Module 2 returns to this for the RL reward). Initialization matters less than the head design suggests. InstructGPT started its RM from a 6B GPT-3 fine-tuned on public NLP datasets "mostly for historical reasons" and found similar results starting from GPT-3 or the SFT model1 (Appendix C.2).
import torch, torch.nn as nn, torch.nn.functional as F
class RewardModel(nn.Module):
def __init__(self, backbone, d_model):
super().__init__()
self.backbone = backbone # decoder-only LM without its unembedding
self.head = nn.Linear(d_model, 1) # the only new parameters
def forward(self, input_ids, attention_mask): # right-padded batch: prompt + response
h = self.backbone(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
last = attention_mask.sum(dim=1) - 1 # index of the final real token (EOS)
h_last = h[torch.arange(h.size(0)), last]
return self.head(h_last).squeeze(-1) # one scalar per sequence
def pair_loss(model, chosen, rejected): # each is a dict(input_ids, attention_mask)
r_w, r_l = model(**chosen), model(**rejected)
return -F.logsigmoid(r_w - r_l).mean() # section 2.2, batch-averaged
The pattern follows Lambert's reference code6 (§5.1.2), rewritten for right-padded batches. The code is the mechanism: the whole model is one head plus a two-line loss, and everything else in a production RM pipeline is data loading and distributed training.
3.2 Training recipes as reported
| Report | RM size and init | Preference data | Epochs, LR, batch | Reported accuracy | Role |
|---|---|---|---|---|---|
| Stiennon 20204 | 160M to 13B ablated; supervised baseline plus random linear head; RM, policy and value same size | 64,832 TL;DR comparisons released; 8k to 64k used in the size study | 1 epoch; 5e-6 (6.7B), 1.5e-5 (1.3B); 64 (Appendix B.1) | 6.7B RM 66.5% on CNN/DM vs 66.9% inter-labeler | PPO reward; value initialized from RM |
| InstructGPT 20221 | 6B only; GPT-3 6B fine-tuned on public NLP sets | 33,207 training prompts, K = 4 to 9, ties dropped | 1 epoch; 9e-6, cosine to 10%; 64 prompts, up to 2,304 comparisons | 72.4 ± 0.4% (train groups); 69.6 ± 0.9% (held-out group) | PPO reward; value initialized from RM |
| Llama 2 20238 | 7B, 13B, 70B; from chat checkpoint; separate helpfulness and safety RMs | 2,919,326 comparisons, 1,418,091 of them Meta-collected | 1 epoch; 5e-6 (70B), 1e-5 (others); 512 pairs (1,024 rows) | Helpfulness RM 63.2% (Meta Helpful); Safety RM 64.5% (Meta Safety) | PPO reward and rejection sampling |
| Llama 3 20245 | pre-trained checkpoint; size not stated | all preference data after dropping similar pairs; 2 to 3 ranked responses per row | not stated; margin term removed | not stated | rejection sampling (10 to 30 samples per prompt), quality scoring; not in the DPO loss |
| DeepSeek-R1 helpful RM9 | R1 architecture plus a reward head | 66,000 pairs labeled by DeepSeek-V3 | 1 epoch; 6e-6; 256; 8,192 tokens max | not stated | stage-2 RL reward, final 400 of 1,700 steps |
| Tulu 3 202412 | from Tulu 3 SFT; scalar head at EOS | the Tulu 3 8B preference mix (same data as the best DPO run) | 1 epoch; 3e-6, linear decay; 256; 2,048 tokens max | not tuned or reported | PPO-vs-DPO comparison; value-function init for RLVR |
Two patterns stand out. Every setup that states an epoch count uses one: InstructGPT found multiple epochs "quickly overfit the model to the training data with obvious deterioration in the validation loss"1 (Appendix C.2), Llama 2 reports the same8 (§3.2.2), and Lambert gives one epoch as the common practice6 (§5.1.2). And reward-model training is cheap next to the RL in the same pipeline. Tulu 3's final 8B reward model took 9 hours on 8 H100 GPUs, while the final 8B RL run took 65 hours on 8 GPUs12 (§6.3), which is 72 against 520 GPU-hours, a factor of 7.2 (derived).
3.3 Scale: what limits the size of a reward model
InstructGPT used one 6B reward model for policies of 1.3B, 6B and 175B parameters. The stated reasons were stability and cost: larger 175B RMs "had the potential to achieve lower validation loss, but (1) their training was more unstable which made them less suitable for use as initializations for the PPO value functions, and (2) using a 175B RM and value function greatly increase the compute requirements of PPO", while 6B RMs "were stable across a wide range of learning rates, and led to equally strong PPO models"1 (Appendix C.2). That is a systems argument: because the value function is initialized from the RM, a 175B RM would mean a 175B value function training alongside the policy (Module 5 counts the four models).
Within the sizes that are practical, accuracy keeps improving. Stiennon et al. trained seven RMs from 160M to 13B parameters on 8k to 64k comparisons and found that doubling the training data adds about 1.1% to validation accuracy while doubling the model size adds about 1.8%4 (§4.3, Figure 6). Llama 2 reports that RM accuracy rose with both more weekly data batches and model size from 7B to 70B, and had not plateaued at 1.4M Meta comparisons8 (§3.2.2, Figure 6). Gao et al.'s synthetic study, which used 3M to 3B parameter proxy RMs against a 6B gold RM, adds the small-data end: below roughly 2,000 comparisons every RM stays close to chance loss, larger RMs do not reach the threshold sooner, and four epochs on the same data give no gain while one epoch on four times the data helps substantially10 (§3.3). How these RM properties set the over-optimization curve is Module 6's subject.
Llama 2 also shows a limit that scale does not remove: helpfulness and safety trade off, so the authors trained two RMs rather than one. Each wins on its own test set (Helpfulness RM 63.2% vs Safety RM 56.2% on Meta Helpful; the reverse on Meta Safety, 62.8% vs 64.5%), which they tentatively attribute ("potentially due to") to the tension between the objectives confusing a single model8 (§3.2.2, Tables 7-8).
3.4 Pair accuracy as a proxy for the policy you get
The accepted justification for evaluating an RM on held-out pairs is Llama 2's: RM accuracy is "one of the most important proxies for the final performance", and because the ranking task has no ambiguity, an RM improvement should translate into a model improvement everything else equal8 (§3.2.2). InstructGPT used the RM score itself to choose SFT checkpoints, finding it "more predictive of human preference results compared to validation loss"1 (Appendix C.1). The evidence on how far that proxy extends is mixed:
- Frick et al. build Preference Proxy Evaluations by correlating offline RM metrics with the post-RLHF human preference ranking of nine models. Pair accuracy on a large human preference set is the best single predictor, and the full set of PPE evaluations reaches a 77% Pearson correlation with downstream performance13 (§7, §9). Their downstream training used DPO, not PPO, and they note the result may not transfer to online RL13 (§8.2). They also report that for top models the correlation between RewardBench scores and downstream RLHF performance has turned negative13 (§2).
- Tulu 3's authors write that "strong RM performance on RM-specific benchmarks does not necessarily translate to better downstream performance for PPO"12 (§5.4.1), and for that reason trained their RM once without tuning.
- Gao et al. show the other direction: a proxy RM can keep rising while the gold score falls. That is Module 6.
Pair accuracy is the best offline proxy available but an imperfect one, and it cannot see what optimization pressure will exploit. DPO drops the separate model: its loss is the Bradley-Terry loss with the reward defined as \(\beta\log\pi_\theta/\pi_{\text{ref}}\), so "your language model is secretly a reward model"2 (§4-5). Module 7 derives it; for this module the relevant fact is that every property in section 2 (shift invariance, bounded gradient weight, noise floor) carries over to the implicit reward.
4. Outcome and process supervision
A preference RM scores a whole response. For reasoning tasks there are two other ways to turn a solution into a learned score, and the choice between them is the most contested design decision in reward modeling. Outcome supervision labels a solution by its final answer. Process supervision labels each reasoning step. Lightman et al. ran the large-scale comparison on MATH14, following Uesato et al., who made the first comprehensive comparison of the two on grade-school math15.
4.1 Four learned scorers and what each predicts
| Type | Predicts | Label per training example | Loss | Used at inference as |
|---|---|---|---|---|
| Preference RM | sequence-level quality \(r_\theta(x,y)\) | a pair ordered on one prompt | \(-\log\sigma(r_c-r_r)\) | one scalar from the EOS state: rerank, or RL reward |
| Outcome RM (ORM) | probability the answer ends correct, at every token | one correct/incorrect label per sample, from a verifier or dataset, copied to every completion token | per-token binary cross-entropy; no pairs needed | score of the final token14, or mean, min or product over tokens |
| Process RM (PRM) | correctness of each step given the prefix | a label per step: positive, neutral or negative; supervised only up to the first error14 | per-step cross-entropy on a label token after each step | product (or minimum) of step probabilities; prune branches in search |
| Value function | expected remaining return from the current prefix | return from the policy's own rollouts, recomputed as the policy changes | regression to the return at each token | baseline for advantages; not a reward |
Lambert's formal versions are the per-token loss \(L_{\text{token}}=-\mathbb{E}\big[\tfrac1T\sum_t r\log p_\theta(s_t)+(1-r)\log(1-p_\theta(s_t))\big]\) for the ORM, with the sequence outcome label \(r\in\{0,1\}\) applied at every completion token and prompt tokens masked, and a per-step cross-entropy over the \(K\) annotated steps for the PRM6 (§5.2-5.3, Eq. 19-21). Three boundaries in the table are blurred in the literature and worth fixing:
- ORM versus value function. Both put a scalar head on every token. With discount \(\gamma=1\) and a terminal correctness reward, the value of a prefix is the probability the completion ends correct, which is what the ORM predicts. They differ in where targets come from: an ORM trains offline on fixed verifier labels, a value function trains on-policy returns that move with the policy6 (§5.4). Module 2 covers the value-function side.
- "ORM" is used loosely. Lightman et al. introduce ORMs and PRMs through Uesato et al.'s outcome/process distinction14 (§1), and Cobbe et al.'s GSM8K verifiers, which were also trained jointly with the language-modeling loss, are the template they follow; some papers use the name for any verifier trained to predict correctness6 (§5.2).
- A BT model on correct-versus-incorrect pairs is still a BT model. If the chosen response is a correct solution and the rejected one an incorrect solution to the same problem, the training uses the contrastive sequence-level loss and falls in the first row of Table 46 (§5.4).
4.2 Scoring one solution both ways
The difference shows on solutions that reach the right answer for the wrong reason. Lightman et al. note that outcome labels come from final-answer checking, which is "not perfectly reliable": a solution that reaches the correct answer with incorrect reasoning is misgraded as positive14 (§2.5). Figure 5 shows the mechanism on a constructed example.
4.3 What the measurements show
| Study and setting | Outcome supervision | Process supervision | What it measures |
|---|---|---|---|
| Uesato et al. 2022, GSM8K15 | final-answer error 23.5% without an RM, 16.6% with an RM; best trace error 12.4% when RL targets final-answer correctness | final-answer error 22.3% without an RM, 14.8% with an RM; trace error 3.8% for the best process-based method | error rates of the trained generator, using human trace annotation (§1). RL against a learned RM that emulates process feedback reached 5.5% trace error |
| Lightman et al. 2023, MATH, GPT-4 base14 | ORM best-of-1,860: 72.4%; majority vote 69.6%. Out-of-distribution STEM set (234 problems in Table 1; the text says 224; best-of-100): 63.8% (majority 61.3%) | PRM best-of-1,860: 78.2%. STEM set: 72.9%. Active learning: about 2.6× more data-efficient | reranking quality of a fixed generator; no RL (§2.1, §3, §5, Table 1) |
| Math-Shepherd, Mistral-7B16 | PPO against an ORM: GSM8K 81.8%, MATH 31.3% (base 77.9%, 28.6%) | step-by-step PPO against an automatically labeled PRM: 84.1%, 33.0% | greedy-decoding accuracy after RL (Table 2) |
| DeepSeekMath-Instruct 1.3B11 | GRPO with outcome supervision | GRPO with process supervision "shows superior performance" | benchmark accuracy over RL steps, GSM8K and MATH (§5.2.1, Figure 5); no numbers in text |
Four readings keep the table honest.
- The largest margin comes from selection, not training. Lightman et al. "do not attempt to improve the generator with reinforcement learning" and score RMs by best-of-N search14 (§2.1). The 78.2% against 72.4% says a PRM ranks candidates better than an ORM; it does not say that rewarding with a PRM yields a better policy.
- The Uesato-Lightman conflict is a data-scale effect. Uesato et al. found similar final-answer error for both kinds of supervision15. Lightman et al. argue that a little process supervision and a lot of outcome supervision perform alike, and that process supervision pulls ahead when scaled, even judged on outcomes alone14 (§7.1). Math-Shepherd's authors add that the PRM advantage is larger on MATH than on GSM8K, whose problems need fewer steps16 (§4.1).
- The clearest win is on the reasoning trace, not the answer. In Uesato et al., every RL run that targeted final-answer correctness left high trace error: 12.4% at best, against 3.8% for the best process-based method, and 5.5% for RL against a learned RM. Even ORMs agreed more with process labels than with their own outcome labels, a result the authors flag as possibly dataset-specific15 (§1).
- Lightman's outcome baseline is confounded. Final-answer labels give false positives on right-answer-wrong-reasoning solutions. When the large PRM supplied outcome-style labels instead, the outcome-trained RM improved noticeably, which the authors consider the more representative baseline where false positives are rarer. Process supervision still beat both at every data scale in that small-scale study14 (§4.1).
Combining step probabilities into one score (derived)
A PRM outputs one probability per step; ranking whole solutions needs a reduction. Lightman et al. tried product and minimum, with neutral steps read as positive or negative. The four settings gave 78.2%, 77.6%, 77.4% and 77.8% best-of-1,860, "minor" differences, and they chose the product with neutral as positive14 (Appendix F.2, Table 4). The product carries a length bias, which they note: if each step is correct with probability \(q\), a solution of \(n\) steps scores \(q^n\). At \(q=0.97\) that is 0.737 for ten steps and 0.544 for twenty, while the minimum stays at 0.97. For a reward in RL the same bias would favor shorter solutions regardless of correctness unless it were normalized, one reason a PRM's output is not used as a drop-in reward (the writer's reading).
4.4 Where process labels come from, and how good they are
Human step labels are the expensive part. Lightman et al. collected 1,085,590 step-level labels over 101,599 solutions and kept about 800,000 over 75,000 solutions (12,000 problems) as PRM800K. Labelers had to agree with gold labels at least 75% of the time on 30 screening questions14 (§2.4, Appendix B). They chose which solutions to label by active learning: show "convincing wrong-answer" solutions, meaning ones the current best PRM scores highly although the final answer is wrong, because each of those is known to contain a step the PRM misjudges. The resulting data is skewed by design: 14.2% of the labeled solutions end correct, against 85.1% in the smaller first collection phase14 (Appendix B, Table 3).
The labels are noisier than the headline suggests. DeepSeekMath's authors write that even PRM800K, "carefully annotated by well-trained annotators", contains approximately 20% incorrect annotations, citing a GitHub issue thread for the figure rather than a measurement11 (§5.2.3, footnote 7); treat that number as a secondhand report.
Math-Shepherd removes the humans. For a prefix ending in step \(s_i\), a "completer" model samples \(N\) continuations to a final answer, and the step label is either the hard estimate \(y^{\text{HE}}_{s_i}=1\) if any completion reaches the correct answer \(a^*\), or the soft estimate \(y^{\text{SE}}_{s_i}=\tfrac1N\sum_{j=1}^N\mathbb{1}[a_j=a^*]\)16 (§3.3, Eq. 3-4). The soft label is a Monte Carlo estimate of the probability that the completer finishes correctly from that prefix. In Module 2's terms it is the value of the prefix under the completer policy with a binary terminal reward and \(\gamma=1\): an automatic PRM is a value function trained offline on a fixed policy's rollouts (the writer's observation, resting on the definitions in sections 3.3 of that paper and 4.1 here). A hand check of 160 GSM8K steps found the hard estimate 86% accurate at \(N=4\) with a LLaMA2-70B completer, falling for larger \(N\) because of false positives; verifier quality was about the same with either estimate16 (§5.2). On MATH the automatic data beat the human-annotated PRM800K for open-source generators, which the authors attribute to a distribution gap (PRM800K steps were labeled on GPT-4 samples) and to four times as much data16 (§5.1).
That last point is the structural weakness of every learned PRM: its labels describe the policy that produced the labeled samples. After RL changes the policy, the labels are stale. Math-Shepherd saw this directly. On MATH500, reranking the PPO-trained model with the original reward models (ORM 41.3%, Math-Shepherd 41.1%) fell below plain self-consistency (42.3%), though on GSM8K the reward models still edged ahead. The authors' explanation is that "the initial reward model is not sufficient to supervise the more powerful model after PPO"16 (§4.1, Table 3).
4.5 Why PRMs did not survive RL at scale
DeepSeek-R1 records its attempt in the "unsuccessful attempts" supplement. It calls PRM "a reasonable method to guide the model toward better approaches" and then lists three limitations: "it is challenging to explicitly define a fine-grain step in general reasoning"; determining whether an intermediate step is correct is hard, because automated annotation "may not yield satisfactory results" and manual annotation does not scale; and "once a model-based PRM is introduced, it inevitably leads to reward hacking", with retraining adding cost and complexity. The conclusion is carefully scoped: a PRM "demonstrates a good ability to rerank the top-N responses generated by the model or assist in guided search," but its advantages are "limited compared to the additional computational overhead it introduces during the large-scale reinforcement learning process in our experiments"9 (Supplement G.2; §4.2 in arXiv v1). The main text applies the decision to all neural reward models for reasoning, outcome or process9 (§2.2). Kimi k1.5 states the same design stance in its abstract: no Monte Carlo tree search, value functions or process reward models17.
The three objections map onto the mechanics above. Step granularity is the label problem of section 4.4. Step correctness is the cost and noise problem. Reward hacking is the staleness problem: a model-based PRM is a frozen approximation of "correct" fit on yesterday's policy, and RL pressure finds its gaps (Module 6 shows the shape of that curve; Module 19 covers practice). The cost objection is separate: R1's own pipeline trains a helpfulness RM and a safety RM for non-reasoning data, so the objection is to using reward models in reasoning RL, not to training them9 (§3.1).
What did survive is the off-loop use. Llama 3 trains "outcome and stepwise reward models" to filter SFT data in which intermediate reasoning steps were wrong, and for harder prompts it runs Monte Carlo tree search with learned step-wise reward models to generate valid reasoning traces5 (§4.3.3). The reward is not in an RL loop that can exploit it; it removes bad samples from a fixed training set. Finer credit assignment inside RL without a learned PRM comes back by other routes, such as grouping trajectories that share intermediate states, which Module 18 treats.
5. Where learned reward models live in current recipes
Read across the reports, a learned scalar reward model has held five distinct jobs: the RL reward, the initialization of the value function, a selector in rejection sampling or best-of-N, a filter or scorer for training data, and a proxy metric for choosing checkpoints. Table 6 lists what each report documents; one model's pipeline often spans several families.
| Family | Report | What replaces or uses the RM |
|---|---|---|
| PPO-RLHF | InstructGPT, Stiennon14 | RM score ends a bandit episode and is the RL reward; value function initialized from the RM; RM score also selects SFT checkpoints |
| Llama 28 | Two RMs (helpfulness, safety) feed PPO and rejection-sampling fine-tuning | |
| Tulu 3 preference study12 | RM trained once for a PPO-versus-DPO comparison: PPO was comparable but cost about 28 hours on two nodes against about 4 hours on one node for DPO, so DPO was kept for preference tuning | |
| DPO pipelines | Llama 35 | RM on the pre-trained checkpoint serves rejection sampling (10 to 30 samples per prompt, best by RM), quality scoring (top RM quartile counts as high quality) and, with step-wise RMs, filtering of reasoning traces; DPO consumes preference pairs directly |
| Olmo 3 Think DPO18 | No RM: pairs contrast a Qwen3 32B Thinking response with a Qwen3 0.6B Thinking response ("delta learning"), plus pairs from a delta-aware LLM-judge pipeline | |
| DeepSeek-R1 SFT data9 | Reasoning data filtered by rule-based correctness, some judged by a generative reward model via DeepSeek-V3 | |
| RLVR and reasoning RL | DeepSeek-R1-Zero and stage 19 | Rule-based accuracy and format rewards only; "we abstain from applying neural reward models" to reasoning |
| DeepSeek-R1 stage 29 | Helpfulness and safety RMs enter only for the final 400 of 1,700 steps, because more steps with the model-based preference reward "may lead to reward hacking" (Figure 6 of the report: reward up while CodeForces performance falls) | |
| Tulu 3 RLVR12 | Verifier reward; the RM only initializes the value function (best GSM8K and average scores), and adding RM scores on top of verifiable rewards "seems to introduce more noise" | |
| 2026 general-RL recipes | Qwen319 | Three reward types over more than 20 tasks: rule-based; model-based with a reference answer (Qwen2.5-72B-Instruct as judge); model-based without one (a reward model trained on human preference data, producing a scalar) |
| GLM-520 | Hybrid system of rule-based rewards, outcome reward models and generative reward models, each covering the others' weakness | |
| DeepSeek-V3.221 | Rule-based outcome reward for reasoning and agent tasks; a generative reward model with per-prompt rubrics for general tasks | |
| DeepSeek-V422 | "We dispense with these conventional scalar-based reward models"; a generative reward model is trained jointly with the actor, which doubles as the judge | |
| Kimi K3, Olmo 32318 | K3: an agentic generative reward model that must read the output, write a rubric, score each candidate and record a scorepad. Olmo 3: verifiers plus an LM judge (Qwen3 32B) for chat; no learned scalar RM is described |
Four things follow. First, the pairwise-trained scalar RM is absent from the verifiable-reward stages (R1-Zero, Tulu 3 RLVR) and documented as a reward only in preference-centred or general-purpose stages. Second, its most durable jobs are those where no policy optimizes directly against its output: selection, filtering, and value-function initialization (Tulu 3's value initialized from a general RM beat the alternative). Third, where reports keep a learned reward inside RL they bound its exposure: R1 confines it to 400 steps, Tulu 3 drops it when a verifier exists, and GLM-5 pairs outcome reward models with generative ones because outcome RMs give "low-variance signals and high training efficiency, but are more susceptible to reward hacking" while generative RMs "are more robust to such exploitation, but tend to exhibit higher variance"20 (§3.4). Fourth, the replacement for non-verifiable tasks is a judge that reasons before scoring, not a smaller or better scalar head, which is Module 16's subject.
Where experts disagree
Two questions are open. The first is whether supervising reasoning steps is worth its cost, now that outcome-only RL with rule-based rewards has produced frontier reasoning models. The second is what should fill the learned-reward slot for tasks without a verifier: a scalar Bradley-Terry model, or a generative judge. The positions below are what the reports state; where no report ran the comparison, the card says so.
A. Process supervision gives better reward models, and it should be used
Lightman et al.14; Shao et al. (DeepSeekMath)11; Wang et al. (Math-Shepherd)16
Step labels localize errors, which makes credit assignment easier; Lightman et al. also argue process supervision is easier to interpret and incurs a "negative alignment tax" on MATH (§6.2). DeepSeekMath's GRPO with process supervision beat the outcome version, and Math-Shepherd's step-by-step PPO beat ORM-PPO.
B. Do not use learned reward models for reasoning RL; use rules
DeepSeek-R19; Kimi k1.517; Tulu 3's RLVR ablation12
R1 abstains from neural reward models, outcome or process, for reasoning because they are susceptible to hacking at scale and need retraining; accuracy and format rewards were enough to take R1-Zero's AIME 2024 pass@1 from 15.6% to 77.9% in the expanded version9 (§2.3). Tulu 3 found that layering RM scores on verifiable rewards added noise12.
- Precise reward with no second model to train, serve or refresh.
- Removes the hacking surface that R1 observed with its helpfulness RM (reward rising while CodeForces performance fell)9.
- Works only where a reliable verifier exists; R1 itself says that for complex tasks that "cannot be effectively evaluated by a reliable reward model", scaling pure RL "remains an open challenge"9 (Supplement G.1).
- The PRM verdict is scoped to "our experiments", and the report shows no controlled run of RL with a PRM against RL without one.
C. Keep process reward models, but outside the RL loop
Llama 35; R1's own concession9; Lightman et al.'s best-of-N framing14
Use outcome and step-wise RMs to filter SFT data, run tree search with step-wise RMs on hard prompts, and rerank at inference. R1 agrees that PRMs "rerank the top-N responses" and "assist in guided search" well; its objection is to their cost inside large-scale RL.
- No policy optimizes against the RM, so there is no exploitation pressure.
- Uses the selection gains that are actually measured.
- Does not improve the on-policy training signal.
- The Llama 3 passage read gives no measurement of the filter's effect on the final model5.
D. Replace scalar reward models with generative judges
DeepSeek-V422; Kimi K323; DeepSeek-V3.221. Retaining trained scalar or outcome RMs: Qwen319, GLM-520; Lambert's benchmark observation6
V4 drops scalar RMs for a rubric-guided generative reward model trained jointly with the actor. Lambert notes that on RM evaluations generative models "tend to be behind existing reward models"6 (§5.6), and Qwen3 and GLM-5 still train preference or outcome reward models alongside rule-based and judge rewards.
- GLM-5 describes generative RMs as more robust to exploitation20.
- V4 reports "highly robust scoring" from only a minimal set of diverse human annotations, by fusing reasoning into evaluation22.
- Higher variance than outcome RMs20; each judgment costs a decode, not a prefill (Module 1).
- None of the passages read compares a judge and a scalar RM at equal data and scale.
A and B measure different things and can both be true. A is a statement about verification quality: given a fixed generator, a PRM selects better solutions than an ORM, and the measured advantage grows with the number of candidates. B is a statement about training stability: inside a long RL run, a frozen learned scorer fit on earlier samples is the weakest component, and R1 reports it being exploited. Math-Shepherd's post-PPO staleness result and R1's hacking curve look like the same phenomenon at two scales (the writer's reading). The unmeasured quantity is what a continually refreshed PRM, retrained on on-policy samples as DeepSeekMath's iterative GRPO does for its reward model11 (§4.1.4), would add over a rule-based outcome reward at 30B+ parameters and thousands of steps, scored on pass@k and on reasoning-trace error. No source runs that experiment. Until one does, the defensible rule is: where a reliable verifier exists, use it as the RL reward (B) and keep PRMs for filtering, reranking and search (C); where none exists, steps are rarely definable (R1's first objection), and the live choice is between a scalar RM and a judge (D).
On D, the evidence is thin on both sides. The reports that drop scalar RMs for judges (V4, K3) publish no controlled comparison against a scalar RM, and Lambert's RM-benchmark gap measures pair accuracy, which section 3.4 showed predicts downstream policy quality imperfectly. A judge-versus-scalar comparison at matched data, with downstream human preference as the endpoint (the design of Preference Proxy Evaluations13), would settle it; none of the reports supplies one.
What is changing
Three movements are visible as of October 2026. The scalar head is being absorbed into the policy: DeepSeek-V4 trains the actor as its own judge, and DPO showed earlier that the policy itself defines a Bradley-Terry reward222. Reward-model evaluation is moving from raw pair accuracy toward benchmarks validated against post-RLHF outcomes, with the caution that the one downstream validation among the sources used DPO rather than online RL13. And process-level credit is leaving the learned-PRM form: automatic step labels such as Math-Shepherd's are Monte Carlo value estimates (section 4.4), and the process-level credit in later recipes comes from group-structured or critic-style estimators rather than a separately trained PRM (Modules 9 and 18). Lambert's book still describes reward modeling as a "relatively under-explored area" whose modifications "have not solidified into a single best practice"6 (§5.5).
References
- paperOuyang, L., Wu, J., Jiang, X., Almeida, D. et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: the K-way comparison batching, loss and normalization (§3.5, Eq. 1), labeler agreement (§3.4), RM accuracy across labeler groups (§4.1), 175B-versus-6B RM stability and RM hyperparameters (Appendix C.1-C.2), dataset sizes (Table 9), bandit RL setup (§3.5).
- paperRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023; arXiv v3, 29 Jul 2024. https://arxiv.org/pdf/2305.18290v3. Accessed 2026-10-26. Used for: the Bradley-Terry preference model and reward-model loss (§3, Eq. 1-2), the gradient weight (§4), Lemma 1 on per-prompt reward equivalence (§5.1, Appendix A.5), Plackett-Luce (Appendix A.3). The paper cites Bradley and Terry, Biometrika 39(3/4):324-345, 1952, as its reference [5]; the original was not fetched.
- paperChristiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., Amodei, D. Deep Reinforcement Learning from Human Preferences. arXiv 1706.03741 v4, 17 Feb 2023. https://arxiv.org/pdf/1706.03741. Accessed 2026-10-27. Used for: preference elicitation with equal and incomparable options (§2.2.2), the Bradley-Terry/Elo preference predictor and the 10% random-response assumption (§2.2.3, Eq. 1).
- paperStiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P. Learning to summarize from human feedback. NeurIPS 2020; arXiv 2009.01325 v3, 15 Feb 2022. https://arxiv.org/pdf/2009.01325. Accessed 2026-10-27. Used for: the RM loss, linear head and reward normalization (§3.4), data and model-size scaling and the CNN/DM and length-bias probes (§4.3, Figure 6), labeler agreement and the three-label vote (Appendix C.2), the 64,832-comparison release (§1).
- paperGrattafiori, A., Dubey, A., Jauhri, A. et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv v3, 23 Nov 2024. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: reward-model training objective and multi-response rows (§4.1.2), preference-data strength levels and filtering (§4.2.1), rejection sampling and RM quality scoring (§4.2.2-4.2.3), outcome and step-wise RMs for trace filtering (§4.3.3), vision RM regularizer (§7.5.4).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: Bradley-Terry derivation and equivalent loss forms (§5.1, Eq. 12-18), architecture and implementation (§5.1.1-5.1.2), ORM and PRM losses (§5.2-5.3, Eq. 19-21), the type comparison (§5.4), margin, K-wise and generative variants (§5.5-5.6), non-transitive preferences (§10), preference-data practice (§11.2).
- paperLambert, N., Pyatkin, V., Morrison, J. et al. RewardBench: Evaluating Reward Models for Language Modeling. arXiv 2403.13787 v2, 8 Jun 2024. https://arxiv.org/pdf/2403.13787. Accessed 2026-10-27. Used for: the 60-70% accuracy ceilings of older preference validation sets due to inter-annotator disagreement (§1).
- paperTouvron, H., Martin, L., Stone, K. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv 2307.09288 v2, 19 Jul 2023. https://arxiv.org/pdf/2307.09288. Accessed 2026-10-27. Used for: the two-RM design, ranking loss with margin, training settings, accuracy by preference strength and scaling (§3.2.2, Tables 6-8, Figure 6), margin ablation and reward-distribution shift (Appendix A.3.3, Tables 27-28).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v2, 4 Jan 2026 (expanded version; Nature 645:633-638, 2025); v1, 22 Jan 2025. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: no neural RMs for reasoning (§2.2), R1-Zero AIME result (§2.3), helpful and safety reward models and AI-labeled pairs (§3.1, Supplement B.3.1), preference reward in the last 400 steps (§3.2.2), reward-hacking observation and Figure 6 (Supplement B.5), open-challenge caveat (Supplement G.1), PRM and MCTS failures (Supplement G.2; §4.2 in v1).
- paperGao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023; arXiv v1, 19 Oct 2022. https://arxiv.org/pdf/2210.10760v1. Accessed 2026-10-26. Used for: translation-invariance and recentering (§2.2), synthetic setup (§2.1), RM data-size and epoch results (§3.3).
- paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: group-relative advantage and comparative reward models (§4.1.1), process-supervision GRPO (§4.1.3), iterative GRPO with reward-model replay (§4.1.4), GRPO+PS versus GRPO+OS (§5.2.1, Figure 5), the PRM800K annotation-error remark (§5.2.3, footnote 7).
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv v5, 14 Apr 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: RM objective (§5.1.1, Eq. 3), the PPO-versus-DPO comparison, RM head and the benchmark-versus-downstream caveat (§5.4.1), RLVR value-function initialization and "do not use the scores from RM" (§6.2, §6.2.1), RM versus RL cost (§6.3), RM hyperparameters (Appendix A, Table 36).
- paperFrick, E., Li, T., Chen, C., Chiang, W.-L. et al. How to Evaluate Reward Models for RLHF (Preference Proxy Evaluations). arXiv 2410.14872 v2, 22 Oct 2024. https://arxiv.org/pdf/2410.14872. Accessed 2026-10-27. Used for: correlation of RM metrics with post-RLHF human preference, accuracy as best predictor and the 77% figure (§7, §9), the DPO-only and nine-model limitation (§8.2), the negative correlation among top RewardBench models (§2).
- paperLightman, H., Kosaraju, V., Burda, Y., Edwards, H. et al. Let's Verify Step by Step. ICLR 2024; arXiv 2305.20050 v1, 31 May 2023. https://arxiv.org/abs/2305.20050. Accessed 2026-10-27. Used for: ORM and PRM definitions and PRM800K collection (§2.4-2.6, Appendix B, D-F), best-of-1,860 results (§3, Figure 3), small-scale supervision comparison and active learning (§4), STEM generalization (§5, Table 1), discussion and related work (§6, §7.1), scoring strategies (Appendix F.2, Table 4).
- paperUesato, J., Kushman, N., Kumar, R., Song, F. et al. Solving math word problems with process- and outcome-based feedback. DeepMind, arXiv 2211.14275 v1, 25 Nov 2022. https://arxiv.org/pdf/2211.14275. Accessed 2026-10-27. Used for: final-answer and trace error rates under outcome and process supervision on GSM8K, and the ORM/process-label agreement finding (§1, key results).
- paperWang, P., Li, L., Shao, Z., Xu, R. X. et al. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv 2312.08935 v3, 19 Feb 2024. https://arxiv.org/pdf/2312.08935. Accessed 2026-10-27. Used for: automatic hard and soft step labels (§3.3, Eq. 3-4), step-by-step PPO results (§4.1, Table 2), post-PPO verification (Table 3), comparison with PRM800K (§5.1), label-quality check (§5.2).
- paperKimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. Moonshot AI, arXiv v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: the abstract's design stance of no Monte Carlo tree search, value functions or process reward models.
- paperTeam Olmo. Olmo 3. Allen Institute for AI, arXiv v2, 14 Apr 2026. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: delta-learning preference pairs without an RM (§4.3, Table 32), verifier and LM-judge rewards in OlmoRL (§4.4, footnote 36).
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: the three reward types, including a preference-trained scalar reward model (§4.4).
- paperGLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. Zhipu AI, arXiv v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: the hybrid rule-based, ORM and GRM reward system and the stated strengths and weaknesses of each (§3.4, General RL).
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: rule-based rewards for reasoning and agent tasks and a rubric-based generative reward model for general tasks (§3, Mixed RL Training).
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348 v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: dispensing with scalar reward models in favor of a jointly trained generative reward model (§5.1.1). The arXiv API date for this record is inconsistent with its identifier.
- paperKimi Team. Kimi K3: Open Frontier Intelligence. Moonshot AI, arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: the agentic generative reward model and its mandatory scoring protocol (§4.1.2).