RL Post-Training · Part 2

Preference Learning and Classical RLHF

How human and AI preferences become a training signal through a reward model plus PPO or directly through DPO, and where each approach breaks.

04Reward Models from Preferences: the Bradley-Terry Loss, RM Architecture and Scale, and Outcome versus Process Supervision45 min05PPO-RLHF as a System: Policy, Reference, Reward and Value Models, Per-Token KL Reward, GAE, and the Pretraining Mix50 min06KL Regularization and Reward-Model Over-Optimization: Reference Policies, the k1/k2/k3 Estimators, and Gold-versus-Proxy Scaling Laws45 min07Direct Alignment: Deriving DPO, the Implicit Reward, Its Failure Modes, and the KTO, Length-Normalized and NLL-Regularized Variants45 min08AI Feedback as the Labeler: RLAIF, Constitutional AI, and Contrastive-Prompt Preference Pairs (Yang et al.'s RLCD)50 min