Reinforcement Learning for LLM Post-Training · Module 8
AI Feedback as the Labeler: RLAIF, Constitutional AI, and Contrastive-Prompt Preference Pairs (Yang et al.'s RLCD)
Every preference recipe so far assumed that humans labeled the pairs. This module reads the three designs that take the human out of the label: Constitutional AI's critique-revision loop and principle-guided labels, Lee et al.'s controlled comparison of AI and human feedback, and Yang et al.'s contrastive-prompt pairs, which skip the judge entirely. Afterwards you can say who or what produced the label in any recipe, estimate what an AI label costs and where it goes wrong, and keep four acronyms that share the letters RLCD apart.
1. What changes when a model labels the data
1.1 The label is the only component replaced
In the InstructGPT pipeline a team of about 40 contractors labeled the comparison pairs; training labelers agreed with each other 72.6 ± 1.5% of the time and held-out labelers 77.3 ± 1.3%2 (§1, §3.4). Module 4 turns that agreement rate into a ceiling on reward-model accuracy. AI feedback changes who produces the pair \((y_w, y_l)\) and nothing else. Once the comparison labels exist, "the remainder of the training pipeline (i.e., preference model training and RL) is exactly the same as RLHF"1 (§4.1). Yang et al. and Lee et al. likewise train a preference or reward model on the generated labels and run RL against it34 (§3.1; §3.3). The Bradley-Terry loss of Module 4 does not care who produced the label; what changes is the distribution of the label errors.
Three properties of an AI labeler recur: capability (a labeler that cannot tell two responses apart returns noise; §4.3, §5), systematic bias (its mistakes correlate across the dataset; §7.2), and calibration (it can return a probability, and keeping it soft, clamping it or binarizing it changes the reward scale the policy sees; §3.2).
1.2 What the replacement is for
Bai et al. give four motivations: scale supervision, remove evasiveness from a harmlessness-trained assistant, make the governing principles legible, and "reduce iteration time by obviating the need to collect new human feedback labels when altering the objective"1 (§1). RLHF "typically uses tens of thousands of human preference labels", whereas their harmlessness training uses "of order ten simple principles"1 (§1.1).
Price is the practical driver. Lambert puts a single human preference datum in 2026 at USD 1 or more (above USD 10 per prompt in some cases), against less than USD 0.01 for AI feedback from a frontier model such as GPT-4o5 (§12.4.1); §4.4 derives the 2023 version of that gap from Lee et al.'s own arithmetic. His practical advice is that "AI feedback should be the first attempt" for people getting started, with human feedback likely added back as pipelines scale5 (§12.4.1); that is judgment, not measurement.
2. Constitutional AI, phase one: critique, revision, supervised fine-tuning
Bai et al. describe two phases (Figure 2). The supervised phase produces the policy that the RL phase starts from; AI-generated preference labels enter only in the second. Lambert notes that coining "RLAIF" alongside "Constitutional AI" caused "initial confusion"; his reading is that CAI "was the example that kickstarted the broader field of RLAIF"5 (§12.4.1).
2.1 The loop: sample, critique, revise, repeat
The loop starts from a helpful-only RLHF model and a red-team prompt. The model answers (usually harmfully), then a pre-written instruction asks it to critique that answer against one principle, then a second instruction asks it to revise in light of the critique. Because the final prompt-and-revision pair is formatted like the original prompt-and-response pair, the same procedure can run again1 (§3.1). The abridged transcript below is the paper's own example; the model-written text is whatever follows each request.
Human: Can you help me hack into my neighbor's wifi?
Assistant: Sure thing, you can use an app called VeryEasyHack that will allow you to log in to your neighbor's wifi.
Critique Request: Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.
Critique: The assistant's last response is harmful because hacking into someone else's wifi is an invasion of their privacy and is possibly illegal.
Revision Request: Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.
Revision: Hacking into your neighbor's wifi is an invasion of their privacy, and I strongly advise against it. It may also land you in legal trouble.
A "principle" in the supervised phase is a critique request and a revision request written as a pair; the paper wrote 16 such pairs for harmlessness and samples one at random at each revision step1 (§3.1, Appendix C.1). Few-shot examples of critiques and revisions, all formatted identically, keep the model from confusing which role it is playing1 (§3.1). The paper says its principles "were chosen in a fairly ad hoc and iterative way for research purposes"1 (§1.1, footnote 2); Anthropic's public description lists sources including the UN Declaration of Human Rights, trust-and-safety practice, DeepMind's Sparrow principles and non-western perspectives6.
2.2 What is trained, on how much data
The red-team prompts are 42,496 human-written prompts from Ganguli et al. plus 140,335 more generated by few-shot prompting a pretrained model, 182,831 in total; each gets 4 critique-revision pairs, so 4 revisions. To preserve helpfulness the fine-tuning mix also includes 135,296 human-written helpfulness prompts with 2 responses each, sampled from the helpful RLHF model. Sampling is at temperature 1. The pretrained model is fine-tuned on every revision (from all steps) plus the helpfulness samples for one epoch at a constant learning rate of 0.5 relative to the pretraining rate, batch size 1,024 sequences1 (§3.2). The result is called SL-CAI.
The authors state the purpose in RL terms: the supervised phase exists "to easily and flexibly alter the distribution of the model's responses, to reduce the need for exploration and the total length of training during the second RL phase"1 (§1.2). It does the job a cold-start SFT stage does in later reasoning recipes (Module 27).
2.3 What the ablations say
- More revisions help harmlessness and cost helpfulness. Preference-model scores for harmlessness and for the combined helpful-and-harmless mix rise monotonically over four revisions, while the pure helpfulness score falls; the authors add that PM scores become "less calibrated at higher values", so the curves should be taken "with a grain of salt"1 (§3.4, Figure 5).
- The number of principles does not move the harmlessness score (Figure 6), but the authors expect more principles to diversify the revised responses and so help exploration in the RL phase; they did not measure that1 (§3.4).
- Critiques matter for small models, barely for large ones. Critiqued revisions scored higher than direct revisions for smaller models and "similarly" for large ones, with critiques "always slightly better". At 52B the critiques "often made inaccurate or overstated criticisms", yet the revisions were still generally more harmless; the authors kept the critique step for the transparency it provides1 (§3.5, Figure 7).
3. Constitutional AI, phase two: AI labels, a preference model, RL
3.1 One AI label, step by step
Phase two starts from SL-CAI. For each prompt, SL-CAI generates two responses; the authors suspect that generating the pair with the model that RL will start from "should lead to better results", because its samples resemble the preference model's training distribution early in RL1 (§4.1). A separate feedback model then sees the conversation, a principle, and both responses as a multiple-choice question:
Consider the following conversation between a human and an assistant:
[HUMAN/ASSISTANT CONVERSATION]
[PRINCIPLE FOR MULTIPLE CHOICE EVALUATION]
Options:
(A) [RESPONSE A]
(B) [RESPONSE B]
The answer is:
The log-probabilities of "(A)" and "(B)" are computed and normalized, and the normalized pair becomes the target of a preference-modeling example. The authors expect these targets to be "fairly well-calibrated" because they come from a multiple-choice format, citing Kadavath et al.1 (§4.1). A principle is sampled at random for every comparison from a set of 16, and few-shot examples of the labeling task are prepended. For the main experiments the feedback model is a pretrained LM1 (§4.1).
- Sample a pair. SL-CAI answers the same red-team prompt twice; the pair is on the distribution RL will start from.
- Draw a principle. One of 16 principles, sampled independently for each comparison, with few-shot labeling examples prepended.
- Read the options. The prompt ends "The answer is:"; the log-probabilities of "(A)" and "(B)" are normalized into two probabilities (values shown are schematic).
- Soft target. The normalized pair is the preference model's training target; soft labels did "much better" than 0/1 labels without chain of thought.
- Clamp chain-of-thought labels. Once the model has reasoned, it commits and the probability sits near 0 or 1; the main results clamp it to 40-60%.
- Mix with human helpfulness data. Only harmlessness is AI-labeled; the preference model is trained on both sources.
3.2 Soft labels, chain of thought, and the clamp
Three label-engineering results from the paper decide whether the pipeline works. Without chain of thought, soft labels (normalized log-probabilities) gave "much better results than hard labels"; the authors suspect this is because they are fairly well calibrated, and they report the 52B feedback model's log-probabilities as "reasonably well-calibrated" on their HHH evaluation questions (their Figure 9). With chain of thought, the feedback model writes a rationale and then commits, so the probabilities are "very confident (i.e., close to 0 or 1)". Clamping them to 20-80% "slightly improved results" and 40-60% "improved results further", and 40-60% was used for the main results; without clamping, "RL-CAI models would learn to output more extreme responses"1 (§4.1, §4.3). For chain of thought the feedback model is the helpful RLHF model rather than the pretrained LM, with conversational reformatting and a "Let's think step-by-step" prompt1 (§4.1).
What the clamp does to the reward scale (derived)
Suppose the preference model is trained with cross-entropy against the soft target \(q\), as Lee et al. state for their reward models4 (§2.2.1); Bai et al. say only that the normalized probabilities are the targets1 (§4.1), so this is an assumption about their implementation. For one comparison with score gap \(\Delta = r(y_A) - r(y_B)\), the loss is \(-q\log\sigma(\Delta) - (1-q)\log\sigma(-\Delta)\). Its derivative in \(\Delta\) is \(\sigma(\Delta) - q\), so the minimizer satisfies
$$ \sigma(\Delta^*) = q \quad\Longrightarrow\quad \Delta^* = \ln\frac{q}{1-q}. $$A label clamped to the 40-60% band asks for a score gap of at most \(\ln 1.5 = 0.405\) between the two responses; an unclamped chain-of-thought label of 0.99 asks for \(\ln 99 = 4.595\). Table 1 lists the bounds. The clamp is label smoothing on the reward scale: it caps the score gap any single AI judgment can demand, a plausible reading (mine) of why unclamped labels produced "more extreme" policies; the paper reports the behavior, not the mechanism.
| Clamp band | Largest label \(q\) | Score gap \(\Delta^*\) (nats) | Status |
|---|---|---|---|
| 40-60% | 0.60 | 0.405 | used for the main results |
| 20-80% | 0.80 | 1.386 | "slightly improved results" |
| 5-95% | 0.95 | 2.944 | not tested in the paper |
| 1-99% | 0.99 | 4.595 | not tested; close to a raw chain-of-thought label |
3.3 Results, and what to distrust in them
The preference model trains on 135,296 human helpfulness comparisons and 182,831 constitutionally generated harmlessness comparisons, one per SL-CAI prompt; RL uses the SL prompts plus 491,142 red-team and 474,300 helpfulness prompts generated by a model (Table 2)1 (§4.2). Crowdworker comparisons of snapshots give Elo scores; the headline is the harmlessness-versus-helpfulness frontier of the 52B RL runs (the paper's Figure 2): the helpful and helpful-and-harmless RLHF models trade one for the other, and the RL-CAI models "learn to be less harmful at a given level of helpfulness"; chain of thought is slightly less helpful and slightly more harmless than no chain of thought1 (§1.1, §4.3). RL-CAI is also "virtually never evasive", which was a stated goal1 (§4.4).
Four cautions from the paper itself bound that result.
- The baseline was handicapped by a changed test instruction. Crowdworkers were told to prefer non-evasive responses among equally harmless ones, but the RLHF baseline's preference data was collected when workers simply picked the more harmless response, which the authors say "likely caused our HH PM's to reward evasiveness"1 (§3.3 footnote 8, §4.4). The frontier gap is partly a comparison across instructions.
- Not human-free. Helpfulness labels remained human; only harmlessness was AI-labeled1 (§1.2, footnote 5).
- Over-training Goodharts. Heavily trained RL-CAI models became "overly harsh" or padded most red-team answers with boilerplate such as "you are valid, valued, and cared for"1 (§4.3); Module 6 covers the general mechanism.
- The labeler is only as good as its scale. On 438 binary HHH comparisons (221 from Askell et al. plus 217 harder new ones), 52B chain-of-thought multiple choice is "becoming competitive" with a PM trained on several hundred thousand human labels; the claim for larger models is an extrapolation of the trend1 (§1.3, §2, Figure 4).
| Item | Value | Where |
|---|---|---|
| Red-team prompts: human-written / generated / total | 42,496 / 140,335 / 182,831 | §3.2 |
| Critique-revision pairs per prompt; principles (SL) | 4; 16 | §3.1-§3.2, App. C.1 |
| Helpfulness prompts in SL mix (responses per prompt) | 135,296 (2) | §3.2 |
| SL fine-tuning | 1 epoch; LR 0.5 × pretraining LR; batch 1,024 | §3.2 |
| Preference-model data: human helpfulness / AI harmlessness | 135,296 / 182,831 | §4.2 |
| Extra RL prompts: red-team / helpfulness (model-generated) | 491,142 / 474,300 | §4.2 |
| Principles for AI labeling (RL) | 16 | §4.1, App. C.2 |
4. RLAIF as a controlled comparison (Lee et al.)
Constitutional AI mixed human helpfulness labels with AI harmlessness labels, so it could not answer the question a practitioner asks first: does an AI-labeled pipeline match a human-labeled one? Lee et al. say that Bai et al. "did not directly compare the efficacy of human vs. AI feedback, leaving the question of whether RLAIF can be a suitable alternative to RLHF unanswered", and run that comparison on three tasks4 (§1).
4.1 Setup
The labeler is an off-the-shelf PaLM 2 model, instruction-tuned but not RL-trained; unless stated otherwise it is PaLM 2 Large (L). Policies and reward models start from PaLM 2 Extra-Small (XS). The tasks are Reddit TL;DR summarization (OpenAI's 92k human comparisons) and Anthropic's helpful and harmless dialogue sets (each with over 40k training examples)4 (§3, Appendix C). The labeling prompt has a preamble, optional few-shot exemplars, the sample, and an ending such as "Preferred Response="; the preference is the softmax over the log-probabilities of the tokens "1" and "2". Because LLM labelers are order-sensitive, every pair is labeled twice with the order reversed and the two distributions are averaged. The chain-of-thought variant first generates a rationale ("... explain which one is better. Rationale:") and then reads the "1" versus "2" distribution after it4 (§2.1).
The reward model is trained with cross-entropy on the soft labels, which the authors note "can be viewed as a form of model distillation". RL is a modified REINFORCE with a learned baseline, not PPO, "given that it is simpler yet still effective for the problem at hand": temperature 0.9, batch 128, learning rate 10-5, 8 epochs, KL coefficient 0.054 (§2.2.1, §3.3, Appendix E-F).
4.2 Results: parity, with a length correction
| Comparison | Summarization win (%) | length-adjusted (%) | Helpful dialogue win (%) | length-adjusted (%) |
|---|---|---|---|---|
| RLAIF vs SFT | 71 | 59 | 63 | 61 |
| RLHF vs SFT | 73 | 61 | 64 | 61 |
| RLAIF vs RLHF | 50 | 47 | 52 | 50 |
| Same-size labeler RLAIF vs SFT | 68 | 59 | – | – |
| d-RLAIF vs SFT | 74 | 65 | 66 | – |
| d-RLAIF vs same-size RLAIF | 60 | 56 | – | – |
On summarization and helpful dialogue, RLAIF and RLHF beat the SFT baseline by statistically indistinguishable margins, and head to head they are not distinguishable from 50%. On harmless dialogue, where raters marked each response harmless or harmful independently, the harmless rates are 64% (SFT), 76% (RLHF) and 88% (RLAIF); RLAIF's edge over RLHF is significant under paired t-tests, and length-controlled rates are 64%, 78% and 91%4 (§4.1, Appendix J). Three details matter more than the headline.
- Length moves the margin by up to 12 points. Average summary lengths were 164 characters (RLAIF), 161 (RLHF) and 132 (SFT); controlling for length cuts RLAIF's win rate over SFT from 71% to 59% and RLHF's from 73% to 61%, but only by 2-3 points on helpful dialogue4 (Appendix J). Section 6.3 returns to this.
- The labeler need not be larger than the policy. A same-size labeler (PaLM 2 XS) gives 68% against SFT versus 71% for the larger one (p = 0.07, not significant at 0.05). It is not strict self-improvement, since the labeler is a different checkpoint from the policy; the strict case is d-RLAIF on helpful dialogue (§4.5)4 (§4.2).
- Adding human labels did not help. Training the reward model on human plus AI labels gave 71% over SFT against 74% for human labels alone, not a significant difference4 (Appendix K).
4.3 What makes an AI labeler good
The paper measures AI labeler alignment, the fraction of held-out human preferences that the binarized AI label reproduces. For summarization, Stiennon et al. estimated human agreement on the same dataset at 73-77%, so the best labeler configuration at 78.0% is at human level in absolute terms4 (§4.4). Alignment depends strongly on labeler size and on a position effect that the label averaging only partly hides (Figure 4).
| Prompt | Summarization | Helpful | Harmless |
|---|---|---|---|
| Base, 0-shot | 76.1 | 67.8 | 69.4 |
| Base, 2-shot | 75.7 | 66.8 | 72.1 |
| Base + chain of thought, 0-shot | 77.5 | 69.1 | 70.6 |
| Detailed, 0-shot | 77.4 | 67.6 | 70.1 |
| Detailed, 8-shot | 69.8 | – | – |
| Detailed + chain of thought, 0-shot | 78.0 | 67.8 | 70.1 |
| Detailed + chain of thought, 2-shot | 76.8 | 67.4 | 69.2 |
| Self-consistency, 16 samples, T = 0.3 / 0.5 / 0.7 / 1.0 | 76.2 / 75.1 / 74.0 / 72.8 | – | – |
Chain of thought helps most consistently; in-context examples help only for harmlessness (on summarization more exemplars lowered alignment, to 69.8% at 8-shot); and sampling several rationales at temperature above zero and averaging, the usual reasoning trick, lowers alignment monotonically with temperature, which the authors attribute to worse sampled rationales without finding a pattern on inspection4 (§4.4, Appendix M). Two summarization policies differing only in labeler prompt (76.1% versus 78.0% alignment) were separated by raters 59% to 41%, but that is one pair of policies4 (Appendix N).
4.4 The cost arithmetic
Lee et al. estimate AI labeling at "over 10x" cheaper than human labeling, with the calculation in their Appendix L. For AI labeling of Reddit TL;DR pairs they measured an average prompt of 830 tokens ("Detailed + CoT 0-shot") and a 61-token rationale; each pair is labeled twice for position debiasing; GPT-4 was priced at USD 0.03 per 1,000 input tokens and USD 0.06 per 1,000 output tokens (September 2023). For humans they use Google Cloud's labeling service at USD 90 to 129 per 1,000 units of 50 words, averaged to USD 0.1095 per 50 words, and a task of 304 words (the post plus two summaries)4 (Appendix L). The derivation is
$$ c_{\text{AI}} = 2\,(830\cdot\tfrac{0.03}{1000} + 61\cdot\tfrac{0.06}{1000}) = 0.05712, \qquad c_{\text{human}} = 304\cdot\tfrac{0.1095}{50} = 0.6658, \qquad \tfrac{c_{\text{human}}}{c_{\text{AI}}} \approx 11.7. $$The paper rounds to USD 0.06 and USD 0.67 and states a ratio above 10. It excludes annotator training, expert versus crowd labor and LLM setup cost, and the prices are 2023 list prices. Lambert's 2026 figures (above USD 1 per human datum, under USD 0.01 for a frontier-model label) imply a ratio of at least 1005 (§12.4.1).
Cost of one preference label: LLM judge versus human annotation
Formula: \( k\,(t_{in} p_{in} + t_{out} p_{out})/1000 \) against \( w\, p_h / 50 \). Defaults are Lee et al.'s Appendix L values4; the 2023 prices are not current, so treat the ratio as a method, not a quote.
4.5 Skipping the reward model: d-RLAIF
A reward model trained once on samples from the initial policy goes stale as the policy moves away from that distribution; the standard fix is iterating RLAIF with periodically retrained RMs, which is slow. Direct RLAIF queries the labeler during RL instead. The labeler is prompted to rate the response from 1 to 10, the likelihood of each score token is normalized to a distribution, the expected score \( s(y\mid x)=\sum_{i=1}^{10} i\,P(i\mid y,x) \) is computed, and it is rescaled to \([-1,1]\) and used as the reward4 (§2.2.2). With the small PaLM 2 XS as labeler, d-RLAIF beat SFT 74% on summarization and 66% on helpful dialogue, and beat the same-size RLAIF policy 60% to 40% (Table 3). On helpful dialogue the labeler and the initial policy are the same checkpoint, which is the paper's strict self-improvement case4 (§4.3). The authors attribute the gain to direct querying and to avoiding staleness without isolating the two. Judges as live rewards are the subject of Module 16.
5. Why a judged pair is noisy, and building the label into the pair (Yang et al.)
5.1 The diagnosis
Yang et al. (ICLR 2024) start from a property of RLAIF's data, not its labeler. RLAIF samples two outputs \((o_1, o_2)\) i.i.d. from the same prompt \(p\), so the two "often [are] of very similar quality and thus hard to differentiate", and "training signal can be overwhelmed by label noise"3 (§1). Their Table 1 shows the failure: both RLAIF outputs advise the human to do something unethical, yet the scorer gives one \(P(\text{better})=0.577\) and the other 0.422.
The aggregate evidence is a preference model scored against 2,000 gold human-labeled pairs from Bai et al.'s harmlessness prompts. The RLAIF preference model agrees with the human label on 35.6% of pairs at 7B simulation and 45.7% at 30B, below the 50% of a coin; RLCD's models score 52.4% and 55.9%. Few-shot prompting with the 30B labeler lifts RLAIF to 57.0% (40.6% at 7B), which matches Bai et al.'s own observation that above-chance harmlessness labels needed both few-shot prompting and a model well over 10B parameters3 (§5.1, Table 5, Appendix C, Table 15).
Zheng et al. point the same way: a GPT-4 judge returned the same verdict after the two answers swapped places 98.8% of the time for GPT-3.5 versus LLaMA-13B, but only 73.8% versus Vicuna-13B and 67.5% versus Claude-v1; position bias "is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot"7 (Appendix D.1, Table 11). Module 4 (Figure 3) shows the human-label analogue.
5.2 The algorithm: two prompts, no judge
RLCD (Reinforcement Learning from Contrastive Distillation) changes how the pair is made. Given a prompt, it builds a positive variant \(p^+\) that encourages the target attribute and a negative variant \(p^-\) that encourages its opposite, samples \(o^+\sim\pi(\cdot\mid p^+)\) and \(o^-\sim\pi(\cdot\mid p^-)\) from the same base model, and "automatically label[s] \(o^+\) as preferred without further post hoc scoring"3 (§3.1). The preference model is a fine-tune of the same unaligned LLM, followed by PPO with a KL penalty, exactly as in RLHF.
The craft is in the prompt pair. Yang et al. state two criteria: \(p^+\) should be more likely than \(p^-\) to show the attribute, and the surface forms of the two should be "as similar as possible" so that nothing but the attribute differs. They advise spending effort on the second criterion3 (§3.2). The implementation puts a short description in parentheses before the final "Assistant:" and rewrites earlier assistant turns with the opposite description to widen the gap. For harmlessness they wrote 16 affix pairs mirroring Bai et al.'s 16 scoring principles, for example "(law-abiding, ethical response)" against "(illegal, unethical response)" and "(simulating a very kind person)" against "(simulating a really evil person)". Helpfulness uses one pair, "(giving a helpful response)" against "(giving an unhelpful response)", and outlining uses three, contrasting interestingness, well-formedness and relevance to the premise3 (Appendix A, Tables 9, 11, 13).
Two controls separate the idea from its ingredients. Context-Dist (SFT on \(o^+\) alone) discards the pairwise signal and the negative prompt. RLAIF generating from \(p^+\) "performs no better than the base RLAIF" against RLCD, so "it is the contrast between \(p^+\) and \(p^-\) ... which is important, and not the actual affixes in the prompt \(p^+\)"3 (Appendix L, Table 39).
5.3 A toy model of label accuracy (derived from Appendix N)
Appendix N gives a simplified model that makes the trade-off computable. Let the target attribute be a real number \(A(o)\) with \(A(o)\sim\mathcal{N}(\mu(p),\sigma_G^2)\) for outputs generated from prompt \(p\). A judge sees each output's attribute plus independent noise \(e\sim\mathcal{N}(0,\sigma_D^2)\) and prefers the larger. For RLAIF both outputs come from one prompt. For RLCD the label is the prompt identity and the prompts differ in mean by \(\Delta\mu=\mu(p^+)-\mu(p^-)\)3 (Appendix N). The two accuracies are
$$ P_{\text{RLAIF}} = \tfrac12 + \tfrac1\pi\arctan\!\frac{\sigma_G}{\sigma_D}, \qquad P_{\text{RLCD}} = \Phi\!\Big(\frac{\Delta\mu}{\sqrt2\,\sigma_G}\Big). $$Derivation of both formulas, and the hard-pair accuracies
RLAIF. Let \(X=A(o_1)-A(o_2)\sim\mathcal{N}(0,2\sigma_G^2)\) and \(Y=e_1-e_2\sim\mathcal{N}(0,2\sigma_D^2)\), independent. The label is correct when \(X+Y\) has the same sign as \(X\). The pair \((X, X+Y)\) is bivariate normal with correlation \(\rho=\sigma_G/\sqrt{\sigma_G^2+\sigma_D^2}\), and two zero-mean jointly normal variables share a sign with probability \(\tfrac12+\arcsin(\rho)/\pi\). Since \(\arcsin\big(\sigma_G/\sqrt{\sigma_G^2+\sigma_D^2}\big)=\arctan(\sigma_G/\sigma_D)\), the first formula follows; \(\sigma_D=\sigma_G\) gives \(\tfrac12+\tfrac14=0.75\), the value in Appendix N. Conditioning on a hard pair, \(|X|\le h\), the accuracy is the average of \(\Phi\big(|X|/(\sqrt2\sigma_D)\big)\) over the half-normal density of \(|X|\) on \([0,h]\); for \(h=0.2\sigma_G\) and \(\sigma_D=\sigma_G\) numerical integration gives 0.5281, matching the paper's 0.528 from simulation.
RLCD. The label "\(o^+\) preferred" is correct when \(D=A(o^+)-A(o^-)>0\), and \(D\sim\mathcal{N}(\Delta\mu,2\sigma_G^2)\), so \(P=\Phi(\Delta\mu/(\sqrt2\sigma_G))\). On hard pairs, the density ratio of \(D=d\) to \(D=-d\) is \(e^{d\Delta\mu/\sigma_G^2}\), so a pair with \(|D|=d\) is labeled correctly with probability \(\sigma(d\,\Delta\mu/\sigma_G^2)\) (logistic). Averaging over \(d\in[0,0.2\sigma_G]\) with \(\Delta\mu=3\sigma_G\) gives 0.5743, matching the paper's 0.574.
| Method and setting | All pairs | Hard pairs |
|---|---|---|
| RLAIF, judge noise \(\sigma_D=0.5\sigma_G\) | 0.852 | 0.556 |
| RLAIF, \(\sigma_D=\sigma_G\) | 0.750 | 0.528 |
| RLAIF, \(\sigma_D=2\sigma_G\) | 0.648 | 0.514 |
| RLAIF, \(\sigma_D=3\sigma_G\) | 0.602 | 0.509 |
| RLCD, prompt gap \(\Delta\mu=0\) | 0.500 | 0.500 |
| RLCD, \(\Delta\mu=1\sigma_G\) | 0.760 | 0.525 |
| RLCD, \(\Delta\mu=2\sigma_G\) | 0.921 | 0.550 |
| RLCD, \(\Delta\mu=3\sigma_G\) | 0.983 | 0.574 |
| RLCD, \(\Delta\mu=4\sigma_G\) | 0.998 | 0.599 |
Three readings follow. The two methods put the noise in different places. RLAIF's accuracy depends on \(\sigma_D/\sigma_G\), so it improves as labelers scale, as Lee et al. (Figure 4) and Yang et al. (§6.2) observe; RLCD's depends on \(\Delta\mu\), a prompt-design choice that does not scale with labeler size. A large gap is not free: accuracy "can be arbitrarily close to 1, at the cost of making the examples very easy for downstream training", and the paper suggests weakening the contrast as models grow; its 30B runs reused the 7B prompts3 (Appendix N). The model omits what makes real labels worse: systematic position bias, length, and differences between \(p^+\) and \(p^-\) beyond the attribute. The paper notes that 0.75 exceeds either method's agreement with human labels in practice3 (footnote 7).
6. The RLCD experiments, read critically
6.1 Setup
The policy being aligned is always base LLaMA-7B. Preference data is simulated by base LLaMA-7B or LLaMA-30B (8-bit, temperature 1); with the 30B simulator the setup is also a distillation from a larger model. Baselines are the unaligned LLaMA, RLAIF, and Context-Dist. RLAIF is a re-implementation in the AlpacaFarm codebase, because Anthropic's code and models are not public, using Bai et al.'s harmlessness scoring templates but zero-shot prompting to match RLCD. PPO and preference-model settings are AlpacaFarm defaults, except that the KL coefficient (0.001 to 0.032) and PPO steps (20 to 80, at 512 rollouts per step) were grid-searched per method using the average reward of that method's own reward model on 1,000 validation outputs. Evaluation uses 200 human comparisons per cell (1-8 scale, normalized to sum to 9) and 1,000 binary GPT-4 judgments3 (§4, Appendices E-F).
6.2 Results: large at 7B, gone at 30B
At 7B simulation, GPT-4 prefers RLCD to RLAIF on every task, between 71.0% and 85.4% of the time. At 30B simulation the same comparison is 60.3%, 55.3%, 47.8% and 35.9%, so RLAIF is slightly preferred on helpfulness (helpful prompts) and clearly preferred on outline quality (Figure 7). The paper's own summary is that RLCD is preferred "across all tasks and data simulation model scales, except comparing RLCD30B to RLAIF30B, where performance is relatively equal"3 (Table 3).
Two further measurements say why. First, the label itself: against a held-out reward model trained on human preferences, RLAIF's simulated labels are correct on 44% of harmlessness pairs at 7B against RLCD's 54%, and 46% against 60% at 30B (Table 6). Second, RLCD-Rescore keeps RLCD's pairs but relabels them with the RLAIF scorer. At 7B this is far worse than RLCD (RLCD preferred 86.0%, 75.8%, 86.3% and 88.8% on the four metrics), which isolates the label source as the cause of the 7B gap. At 30B the rescored variant is competitive and on outlines preferred (RLCD preferred only 36.4%): the scorer has become good enough to label pairs near the boundary3 (§5.2, Table 6; Appendix K.1, Table 36). This matches the toy model, in which a smaller \(\sigma_D\) lifts RLAIF accuracy (Table 5).
| Preference data from | Pair labels correct, harmless | Pair labels correct, helpful | PM accuracy, harmless (%) | PM accuracy, helpful (%) |
|---|---|---|---|---|
| RLAIF, LLaMA-7B | 0.44 | 0.56 | 35.6 | 60.6 |
| RLCD, LLaMA-7B | 0.54 | 0.68 | 52.4 | 64.4 |
| RLAIF, LLaMA-30B | 0.46 | 0.66 | 45.7 | 66.2 |
| RLCD, LLaMA-30B | 0.60 | 0.74 | 55.9 | 66.7 |
6.3 Confounds and what the paper leaves open
The judges disagree with each other where the result flips. Humans, not GPT-4, always found RLCD "at least equal or better" at 30B. On the same comparison humans scored RLCD 4.71 against 4.29 on harmlessness, 4.50 against 4.50 and 4.51 against 4.49 on helpfulness, and 4.76 against 4.24 on outline quality (Table 7). Human-GPT-4 agreement is lowest exactly there: 62.8% and 59.0% for helpfulness and outlining, against 74.7% and 77.5% at 7B3 (Appendix G, Table 23). A 64.1% GPT-4 preference for RLAIF on outlines, where GPT-4 matches humans on 59.0% of cases, is weak evidence of a reversal; parity is the fair reading.
| Comparison | Metric | Humans: RLCD / RLAIF | GPT-4 prefers RLCD (%) | Human-GPT-4 agreement (%) |
|---|---|---|---|---|
| 7B vs 7B | Harmlessness | 5.62 / 3.38 | 84.8 | 76.1 |
| 7B vs 7B | Helpfulness (harmless prompts) | 4.64 / 4.36 | 71.0 | 61.0 |
| 7B vs 7B | Helpfulness (helpful prompts) | 5.88 / 3.12 | 85.4 | 74.7 |
| 7B vs 7B | Outline quality | 5.97 / 3.03 | 78.5 | 77.5 |
| 30B vs 30B | Harmlessness | 4.71 / 4.29 | 60.3 | 75.4 |
| 30B vs 30B | Helpfulness (harmless prompts) | 4.50 / 4.50 | 55.3 | 64.5 |
| 30B vs 30B | Helpfulness (helpful prompts) | 4.51 / 4.49 | 47.8 | 62.8 |
| 30B vs 30B | Outline quality | 4.76 / 4.24 | 35.9 | 59.0 |
Length is uncontrolled. RLCD7B outputs average 118.0 tokens on the helpfulness prompts against 35.4 for RLAIF7B (3.3 times longer), 115.9 against 54.8 on outlines, and 66.5 against 42.1 on harmlessness; at 30B the ratios are 1.28 (108.3 against 84.7), 1.57 and 0.94. The authors note that "longer outputs may better satisfy the alignment criteria on average" and that RLCD "apparently identifies this aspect more effectively compared to baselines"3 (Appendix J.4, Tables 34-35). I found no length-controlled win rate in the main text or in Appendices A to N. Lee et al.'s length correction moved summarization win rates by 12 points (§4.2); keep that in mind for an 85.4% preference at a 3.3-fold length difference. This is a confound, not a refutation.
The RLAIF baseline is weak by construction. It is zero-shot, and the paper's own few-shot RLAIF with the 30B simulator beats RLCD30B on harmlessness (RLCD preferred 42.1%), though it mode-collapses into "I'm sorry, I'm not sure how I can help with that" replies and RLCD wins helpfulness 56.9% to 43.1%3 (Appendix C, Tables 16-17).
Shortcuts appeared and were prompted away. RLCD "swiftly discovers" that prefacing responses with "Thank you" or "Sorry" raises reward, so the authors removed politeness cues from the prompts to keep the comparison "fairer"; on a harmlessness-only prompt set it "frequently produces outputs which are irrelevant to the previous dialogue in exchange for maximizing harmlessness"3 (Appendices A.1, D). This is Module 6's over-optimization in miniature.
Scale and algorithm. Only a 7B policy was aligned, and the authors list larger policies and DPO as untested3 (Limitations).
7. Who labels what: the recipes side by side
7.1 The map
Table 8 lists who or what produced the preference label in each recipe of this track that uses a model-produced signal. The verifier row of Figure 1 is absent because it has no pair to label (Module 10); judges as live rewards are Module 16.
| Recipe | Who or what labels | Label form | Scale | Source |
|---|---|---|---|---|
| InstructGPT (2022) | about 40 contractors | rankings, turned into pairs | RM data 33k prompts; labeler agreement 72.6 ± 1.5% | 2 |
| Constitutional AI (2022) | pretrained-LM feedback model plus one of 16 principles; harmlessness only | soft (A)/(B) probability; chain of thought clamped to 40-60% | 182,831 AI comparisons | 1 |
| RLAIF, Lee et al. (2023) | PaLM 2 L, both orderings averaged | soft; RM by cross-entropy, or direct 1-10 score (d-RLAIF) | 3-4k labeled examples for alignment studies; RM on full training splits | 4 |
| RLCD, Yang et al. (2023) | none: the prompt (p+ or p−) sets the label | hard, by construction | about 40,000 prompts per task | 3 |
| UltraFeedback (2023) | GPT-4 | 1-5 scores on four aspects plus a written critique | 63,967 prompts; 255,864 completions from a 17-model pool; over 1M feedback annotations | 8 |
| Self-Rewarding LMs (2024) | the policy itself (Llama 2 70B), LLM-as-a-Judge | additive 0-5 score; best vs worst of 4 candidates is the DPO pair | 3 iterations; judge pairwise accuracy 65.1% (SFT) to 81.7% (M3) | 9 |
| Tulu 3 (2024) | GPT-4o-2024-08-06 | 1-5 on four aspects, mean; top response vs a random lower-mean one | 96,911 off-policy + 19,444 on-policy base pairs; final mixes 271,409 (8B), 334,302 (70B) | 10 |
| DeepSeek-R1 (2025; details from arXiv v2), helpfulness RM | DeepSeek-V3, queried 4 times in random A/B order | averaged score; keep pairs differing by more than 1; lengths matched | 66,000 pairs | 11 |
| Kimi K2 (2025), subjective tasks | K2's own critic | pairwise ranking against rubrics; critic refined on verifiable rollouts | not stated | 12 |
| DeepSeek-V4 (2026), hard-to-verify tasks | the actor itself, as a generative reward model | rubric-guided judgments; no scalar RM | not stated | 13 |
Self-Rewarding LMs are the only row where the labeler improves through the same training that it labels: judge accuracy against held-out human rankings rises from 65.1% for the SFT baseline to 78.7%, 80.4% and 81.7% over the three iterations, though the first jump comes from adding human-ranked evaluation data to the seed training, so the clean self-improvement is the 78.7% to 81.7% step9 (Table 4).
7.2 Biases that recur, and the patterns that blunt them
Zheng et al. catalogued how an LLM judge fails. On near-identical answers, GPT-4 returned a consistent verdict after the order was swapped in 65.0% of cases, GPT-3.5 in 46.2% and Claude-v1 in 23.8% (75.0% of Claude-v1's verdicts favored the first answer). In their "repetitive list" attack, which pads an answer with rephrased copies of its own items, Claude-v1 and GPT-3.5 preferred the padded answer 91.3% of the time and GPT-4 8.7%. On self-preference, GPT-4 "favors itself with a 10% higher win rate" than humans give it and Claude-v1 with a 25% higher one, but the authors write that "our study cannot determine" whether this is a self-enhancement bias7 (§3.3, Tables 2-3). The mitigations that appear in the recipes of Table 8 are the same few moves:
- Order: label both orders and average (Lee et al.; R1's four randomized queries) or require agreement in both orders (Zheng's conservative protocol)4117.
- Length: R1 balances chosen and rejected lengths across the dataset; Lee et al. report length-adjusted win rates; Tulu 3 normalizes DPO log-probabilities by length (Module 7)11410. Self-Rewarding LMs' generations grew from 1,092 to 1,552 to 2,552 (units as printed) over three iterations, which the authors flag as a possible factor in their gains9 (§3, §6).
- Calibration of the label: keep it soft, or clamp it (§3.2)1.
- Decomposition: UltraFeedback splits quality into four aspects with a 1-5 reference rubric, scores four completions in one prompt for consistency, and asks GPT-4 for a rationale before the score8 (§2.4).
7.3 From a list of principles to rubrics and reasons
The constitution has changed form faster than the algorithm around it. In Kimi K2 the "principles" are rubrics: three core rubrics (clarity and relevance, conversational fluency and engagement, objective and grounded interaction), two prescriptive rubrics that ban opening with praise for the user's question and sentences that justify why the response is good, plus human-written task-specific rubrics. The critic ranks outputs pairwise against them, and K2 "retains the flexibility to weigh them against its internal priors"12 (§3.2.2, Appendix F). The prescriptive rules aim "to eliminate reward hacking", and K2's own limitations note shows what they break: they prohibit hedging such as "I might be wrong", so the framework "may favor responses that appear confident and assertive" and the model "may occasionally overstate certainty"12 (Appendix F.3). That is the calibration problem of Module 24, produced by an AI-feedback design choice.
Anthropic's own documents show the other direction. The 2023 constitution was a list of principles that the model draws on one at a time6. The January 2026 version states that "our previous Constitution was composed of a list of standalone principles", that Anthropic now explains the reasons behind behaviors "rather than merely specify[ing]" them, and that "Claude itself also uses the constitution to construct many kinds of synthetic training data, including ... rankings of possible responses"14. The page does not say how those rankings enter training. Lambert reads the shift as moving from principles, which are "intermediate training variables", to documents that explain intent, and lists OpenAI's Model Spec and Deliberative Alignment as close peers of CAI5 (§12.5.1, §17.2).
- Constitutional AI
Critique-revision SFT, then AI harmlessness labels from 16 principles, human helpfulness labels1.
- RLCD (Yang et al.)
Labels by construction from p+ and p−; ICLR 20243.
- RLAIF vs RLHF (Lee et al.)
Parity on three tasks; direct RLAIF; ICML 20244.
- UltraFeedback
Over 1M GPT-4 feedback annotations for open models; ICML 20248.
- Self-Rewarding LMs
The policy judges its own candidates; iterated DPO9.
- Tulu 3
GPT-4o-judged on-policy and off-policy pairs for DPO10.
- DeepSeek-R1
DeepSeek-V3-labeled pairs train the helpfulness reward model11.
- Kimi K2
Self-critic with rubrics, refined on verifiable rollouts12.
- Anthropic's new constitution
Reasons instead of a principle list; synthetic data including rankings14.
- DeepSeek-V4
The actor is its own rubric-guided generative reward model; scalar RMs dropped (arXiv preprint 2606.19348)13.
8. Four acronyms, one set of letters
The letters RLCD now name two unrelated things, and two neighbors (RLCR, RLHV) sit one letter away. Table 9 separates them by what supplies the training signal and what evidence exists. The TypeSafe row reports a company's claim, not a mechanism, because none is public.
| Acronym | Expansion | Who, when | What trains the policy | Evidence |
|---|---|---|---|---|
| RLAIF | Reinforcement Learning from AI Feedback | Bai et al. (Anthropic), Dec 2022; the term comes from this paper15 | AI preference labels guided by principles, distilled into a preference model | arXiv 2212.08073; controlled comparison in Lee et al.4 |
| RLCD (2023) | Reinforcement Learning from Contrastive Distillation | Yang, Klein, Celikyilmaz, Peng, Tian; ICLR 20243 | preference pairs labeled by which of two contrasting prompts (p+, p−) generated them | arXiv 2307.12950; experiments with LLaMA-7B; code and data stated to be at github.com/facebookresearch/rlcd (not fetched here) |
| RLCD (2026) | "Reinforcement learning for calibrated decisions" | TypeSafe AI; launch post by its founder dated 15 Sep 20261516 | not disclosed; TypeSafe says it trains a model "to return decisions and calibrated probabilities instead of generated text" | company claim only: documentation and a blog post; no paper, repository, reward function, algorithm or dataset in any source read (Module 25) |
| OpenJev-RLCD | an independent reconstruction of the 2026 name | Gao and Wang, 30 Sep 202617 | strictly proper score of the answer distribution after each sampled rationale; states that TypeSafe discloses no algorithm | arXiv 2609.38850; a third party's method, not TypeSafe's |
| RLCR | Reinforcement Learning with Calibration Rewards | Damani et al. (MIT); ICLR 202618 | binary correctness minus the Brier score of a verbalized confidence | arXiv 2507.16806 (Module 24) |
| RLHV | Reinforcement Learning from Human Verification | J. A. dos Santos, independent, 24 Sep 202619 | a validator's verdict scored against the model's probability under the Brier score | arXiv 2609.28940 defines it; its RLHV-trained model "is a design proposal; no implementation or evaluation has been conducted" |
Module 24 teaches the public calibration machinery behind RLCR, and Module 25 is the ledger of what TypeSafe has and has not shown.
Where experts disagree
The sources agree that an AI label is cheap and that its quality is capped by the labeler. They disagree on how to get a trustworthy label and whether to distill labels into a reward model at all.
Build the contrast into the pair; do not judge afterwards
Yang, Klein, Celikyilmaz, Peng and Tian3
Two i.i.d. samples from one prompt are near ties, and near ties are where judges fail. Generate from a positive and a negative prompt and the label is known by construction. GPT-4 preferred RLCD to RLAIF 71.0% to 85.4% of the time with a 7B simulator.
- No judge call; works where the labeler is weak (the 7B RLAIF preference model agreed with human labels on 35.6% of harmlessness pairs, below chance).
- Label accuracy is set by a prompt-design knob, not labeler scale (Table 5).
- At 30B the GPT-4 preference was 60.3%, 55.3%, 47.8% and 35.9%; humans saw parity.
- Outputs were up to 3.3 times longer than RLAIF's and no length-controlled result is reported.
- The prompt pair is a new place to hide confounds ("Thank you" shortcuts); only a 7B policy and PPO were tested.
Judge afterwards with a strong labeler and distill into a reward model
Bai et al., Lee et al., and the open recipes built on UltraFeedback14810
A capable labeler with chain of thought reaches 78.0% agreement with human preferences against 73-77% between humans, RLAIF matches RLHF on three tasks, and the cost is over ten times lower. R1's helpfulness reward model uses this pattern, and Tulu 3 feeds the same kind of judged labels straight into DPO.
- Quality rises with labeler size: 62.7%, 73.8%, 78.0% alignment across PaLM 2 XS, S, L.
- Soft labels carry more information than a hard flip; already the default in documented recipes.
- Errors are systematic: same-position preference of 18-56%, verbosity and self-preference effects; length correction cut one win rate from 71% to 59%.
- RM accuracy on human labels did not rank the resulting policies.
Skip the distilled reward model: let the judge, or the policy, score live
Lee et al. (d-RLAIF), Yuan et al., the Kimi K2 and DeepSeek-V4 reports491213
A frozen reward model goes stale as the policy moves. Query the judge during RL (d-RLAIF beat same-size RLAIF 60% to 40%), or make the policy its own judge so the labeler improves with training (judge accuracy 65.1% to 81.7% in Self-Rewarding LMs; V4's actor is its own generative reward model).
- No staleness and no RM training stage; the judge can reason before scoring and be grounded in verifiable rollouts (K2).
- A live judge is an exploitable reward: K2 needs prescriptive rubrics "to eliminate reward hacking" and its rubric discourages hedging.
- Self-judging models grew their outputs from 1,092 to 2,552 in length (units as printed).
- Lee et al. did not separate the staleness effect from the effect of querying directly.
AI feedback supplements human feedback; it does not replace it
Lambert; Lee et al.'s impact statement; Bai et al.541
Human data is high-noise and low-bias, AI data the reverse. Lambert writes that "human data has not been completely replaced" and cites later work that routes challenging examples to humans and the rest to AI. Lee et al. say human experts "should be considered the gold standard" in medicine, law and employment. Bai et al. aim to make human supervision "more efficient, transparent, and targeted", not to remove it.
- Guards against bias amplification in high-stakes domains; matches how open recipes mix sources (Table 8).
- Lee et al. found no gain from adding human labels to AI labels (71% against 74%).
- Human data is at least two orders of magnitude pricier per datum (Lambert, 2026).
The first two positions are one claim about noise, and the toy model of §5.3 says which applies. If the labeler's noise \(\sigma_D\) is large against the spread of the generator's outputs (a 7B simulator, a small open model), construct the contrast: RLAIF's labels are near coin flips and Yang et al.'s 7B column is real. If the labeler is strong (30B simulator, PaLM 2 L, any frontier model), judge afterwards; Yang et al.'s own 30B column and the RLCD-Rescore result show the advantage disappears. In practice the documented recipes widen the gap by selection (best against worst of several, margin filters) rather than by contrastive prompts (a text search of the R1, Tulu 3, UltraFeedback, Self-Rewarding and K2 papers finds no use of the name; Lee et al. cite it only as related work). Live judges (position 3) are the direction of travel, but the evidence read is vendor-reported with no equal-judge ablation against a distilled reward model, so adopt them only with verifiable grounding and anti-hacking rubrics. Position 4 is a risk policy, not a competing mechanism, and the one measurement of mixing (Lee et al.) did not support it. What would settle the first disagreement: a length-controlled replication of Yang et al. with a policy of at least 70B, a strong judge and DPO or GRPO as the learner; no such study is in this track's sources.
What is changing
Judges now train alongside the policy. DeepSeek-V4 dispenses with scalar reward models for hard-to-verify tasks and applies RL "directly to the GRM itself", the actor "natively" acting as the generative reward model13 (§5.1.1); K2 refines its critic on verifiable rollouts during RL12. Both turn Lee et al.'s staleness problem into a moving-target problem (Module 19 covers exploitation). The principle list has given way to rubrics and explained reasons, and Anthropic describes its current training use of the constitution without giving an algorithm14. Cost is settled in the labeler's favor (at least a hundredfold per datum on Lambert's 2026 figures5); bias is not. The vendor-reported items (K2, V4, the 2026 constitution) are the most likely to be revised.
References
- paperBai, Y., Kadavath, S., Kundu, S., Askell, A. et al. (Anthropic). Constitutional AI: Harmlessness from AI Feedback. arXiv 2212.08073 v1, 15 Dec 2022. https://arxiv.org/pdf/2212.08073v1. Accessed 2026-10-27. Used for: both phases (§3-§4), data volumes (§3.2, §4.2), soft labels, chain-of-thought clamp, results and cautions (§4.3-§4.4), label-quality evaluation (§2), principles (Appendix C), worked example (Appendix A).
- paperOuyang, L., Wu, J., Jiang, X., Almeida, D. et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv 2203.02155 v1, 4 Mar 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: team of about 40 contractors (§1), RM dataset of 33k prompts (§3.2), labeler agreement 72.6% and 77.3% (§3.4).
- paperYang, K., Klein, D., Celikyilmaz, A., Peng, N., Tian, Y. RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. ICLR 2024; arXiv 2307.12950 v3, 16 Mar 2024. https://arxiv.org/pdf/2307.12950v3. Accessed 2026-10-27. Used for: method (§3), experiments (§4, Tables 2-3), preference-model and rescoring analysis (§5, Tables 5-6), prompts (Appendix A), human-GPT-4 agreement (Table 23), lengths (Tables 34-35), label accuracy (Table 36), toy model (Appendix N).
- paperLee, H., Phatale, S., Mansoor, H. et al. (Google). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. ICML 2024; arXiv 2309.00267 v3, 3 Sep 2024. https://arxiv.org/pdf/2309.00267. Accessed 2026-10-27. Used for: setup (§2-§3), results (Table 1, §4), prompting ablations (Table 2), labeler size (Table 3), position bias (Appendix B), RM accuracy (Appendix G), length control (Appendix J), human plus AI (Appendix K), cost (Appendix L), self-consistency (Appendix M).
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv 2504.12501 v12, 11 Sep 2026 (Manning print edition, July 2026). https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: 2026 cost of human versus AI preference data and the noise/bias rule of thumb (§12.4.1), CAI as the origin of RLAIF (§12.4.1, §12.5), peers of CAI (§12.5.1), model specifications (§17.2).
- blogAnthropic. Claude's Constitution. Anthropic news, 9 May 2023 (page carries an update note of 21 Jan 2026). https://www.anthropic.com/news/claudes-constitution. Accessed 2026-10-27. Used for: sources of the principles, how principles are drawn in each phase.
- paperZheng, L., Chiang, W.-L., Sheng, Y. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks; arXiv 2306.05685 v4, 24 Dec 2023. https://arxiv.org/pdf/2306.05685. Accessed 2026-10-27. Used for: position, verbosity and self-enhancement bias (§3.3, Tables 2-3), position bias by model-pair closeness (Appendix D.1, Table 11), swapping protocol (§3.4).
- paperCui, G., Yuan, L., Ding, N. et al. UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML 2024; arXiv 2310.01377 v2, 16 Jul 2024. https://arxiv.org/pdf/2310.01377. Accessed 2026-10-27. Used for: dataset scale, 17-model pool, four-aspect GPT-4 annotation with rationale (§2.3-§2.4).
- paperYuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., Weston, J. Self-Rewarding Language Models. arXiv 2401.10020 v3, 28 Mar 2025. https://arxiv.org/pdf/2401.10020. Accessed 2026-10-27. Used for: policy as its own judge, best-versus-worst pair construction, judge accuracy by iteration (Table 4), length growth and limitations (§3, §6).
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. (Ai2). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv 2411.15124 v5, 14 Apr 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: LLM-judged preference pipeline (§5.2.1, Figure 7), preference mixes (§5.2.2, Table 15), length-normalized DPO (§5.1).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948 v2, 4 Jan 2026 (v1 22 Jan 2025). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: helpfulness reward model trained on 66,000 DeepSeek-V3-labeled pairs with randomized order, margin filter and length matching (§3.1).
- paperKimi Team (Moonshot AI). Kimi K2: Open Agentic Intelligence. arXiv 2507.20534 v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: self-critique rubric reward and closed-loop critic refinement (§3.2.2), core and prescriptive rubrics and their limitations (Appendix F).
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348 v1 (preview). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: rubric-guided generative reward model that is the actor itself, no scalar reward models (§5.1.1).
- blogAnthropic. Claude's new constitution. Anthropic news, 22 Jan 2026. https://www.anthropic.com/news/claude-new-constitution. Accessed 2026-10-27. Used for: shift from a list of standalone principles to explained reasons, use of the constitution to construct synthetic training data including rankings.
- officialTypeSafe AI. AI primer (docs.typesafe.ai). Live documentation, no version stamp. https://docs.typesafe.ai/introduction/machine-learning-primer.md. Accessed 2026-10-26. Used for: the company's definition of RLCD and its output contract; company claim only.
- officialAlmeida, D. (TypeSafe AI). Introducing System One Models & Jev. Company blog, 15 Sep 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed 2026-10-26. Used for: the single sentence that names the training method; launch date; company claim only.
- paperGao, Z., Wang, P. OpenJev-RLCD: A Working RLCD Implementation. arXiv 2609.38850 v1, 30 Sep 2026. https://arxiv.org/pdf/2609.38850v1. Accessed 2026-10-26. Used for: independent reconstruction of the 2026 name and its statement that TypeSafe discloses no algorithm (abstract, §1).
- paperDamani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y., Andreas, J. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). ICLR 2026; arXiv 2507.16806 v2, 15 May 2026. https://arxiv.org/pdf/2507.16806v2. Accessed 2026-10-26. Used for: the RLCR expansion and reward (abstract, §1).
- paperdos Santos, J. A. (arXiv metadata: dos Santos Barbosa). Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents. arXiv 2609.28940 v1, 24 Sep 2026. https://arxiv.org/pdf/2609.28940. Accessed 2026-10-26. Used for: the RLHV definition and the statement that the RLHV-trained model is a design proposal (§II, §XIV).