Course · Computer Science

Reinforcement Learning for LLM Post-Training

How reinforcement learning turns a pre-trained, supervised-tuned LLM into an aligned reasoning model: RLHF, PPO, DPO, GRPO and successors, reward and verifier design, RL environments, and when to use which method.

34 modules8 parts1 appendix≈ 26 h readingSources

Part 1

The RL Frame for a Language-Model Policy

What one RL iteration costs on a cluster, how a policy gradient is estimated for token sequences, and how trust-region ideas became the PPO clip that every later algorithm modifies.

01The RL Post-Training Loop as a Cost Model: Rollout, Reward, Update, and Where the GPU-Hours Go45 min02Policy Gradients for Token Sequences: the Log-Derivative Trick, Baselines, and Advantages for an LLM Policy45 min03From TRPO to PPO: Trust Regions, Importance Ratios, the Clipped Surrogate, and GAE50 min
Part 2

Preference Learning and Classical RLHF

How human and AI preferences become a training signal through a reward model plus PPO or directly through DPO, and where each approach breaks.

04Reward Models from Preferences: the Bradley-Terry Loss, RM Architecture and Scale, and Outcome versus Process Supervision45 min05PPO-RLHF as a System: Policy, Reference, Reward and Value Models, Per-Token KL Reward, GAE, and the Pretraining Mix50 min06KL Regularization and Reward-Model Over-Optimization: Reference Policies, the k1/k2/k3 Estimators, and Gold-versus-Proxy Scaling Laws45 min07Direct Alignment: Deriving DPO, the Implicit Reward, Its Failure Modes, and the KTO, Length-Normalized and NLL-Regularized Variants45 min08AI Feedback as the Labeler: RLAIF, Constitutional AI, and Contrastive-Prompt Preference Pairs (Yang et al.'s RLCD)50 min
Part 3

Critic-Free Policy Optimization and Verifiable Rewards

GRPO and its ancestors, RL with verifiable rewards, and the loss, advantage and ratio choices that define the 2024-2026 reasoning-RL recipes.

09Critic-Free Policy Gradient: REINFORCE with Group Baselines, RLOO, GRPO, and Kimi's Mirror-Descent Objective45 min10RL with Verifiable Rewards: Rule-Based Reward Design, Prompt Selection, and What R1-Zero Showed45 min11The GRPO Loss and Advantage Under the Microscope: Length Bias, Difficulty Bias, Token-Level Averaging, and Zero-Variance Groups45 min12Importance Ratios and Clipping: Token-Level, Sequence-Level (GSPO), IS-Weight Clipping (CISPO), and Log-Ratio Masks45 min
Part 4

Making RL Run: Dynamics, Numerical Mismatch, and Compute Scaling

What goes wrong during long RL runs (entropy collapse, length drift, probability mismatch between inference and training engines) and what scaling studies say about compute.

13Training Dynamics: Entropy Collapse, KL Drift, Length Growth, and the Controls That Work50 min14Training-Inference Mismatch: Why the Rollout Engine and the Trainer Disagree on Probabilities, and the Four Families of Fixes45 min15RL Compute Scaling: Sigmoidal Fits, the ScaleRL Recipe, and What Longer Runs Actually Buy45 min
Part 5

Rewards, Environments, Agents, and Their Failure Modes

Where reward signals come from when a unit test is not enough, what an RL environment is, how multi-turn agent training differs, how rewards get exploited, and what the resulting numbers do and do not show.

16Rewards Beyond Ground Truth: LLM Judges, Rubric Rewards, Generative Reward Models, and Judges Trained Inside the Loop45 min17RL Environments for LLMs: Task Datasets, Harnesses, Verifiers, State, and Reward Shaping50 min18Agentic and Multi-Turn RL: Trajectories as Episodes, Credit Assignment, Observation Tokens, and Staleness40 min19Reward Hacking in Practice: Proxy Exploits, Emergent Misalignment from Hacked Environments, and the Defenses Reports Describe50 min20Evaluating What RL Did: pass@1 versus pass@k, Spurious Rewards, Contamination, and the Sharpen-or-Expand Debate50 min
Part 6

Beyond One Scalar Reward: Distillation, Effort Control, and Calibration

The 2025-2026 paradigms that change what the training signal is: dense on-policy distillation, multi-teacher integration, length and effort as trained targets, calibration-aware rewards, and the RLCD claims.

21On-Policy Distillation as Dense-Reward RL: Per-Token Reverse KL, Cost Arithmetic, and When It Beats RL50 min22Capability Integration: Mixed RL, Cascade RL, Parameter Merging, and Multi-Teacher On-Policy Distillation (MOPD)45 min23Length and Reasoning-Effort Control: Length Rewards, Token Budgets, Toggle, and Effort-Conditioned Experts50 min24Calibration-Aware RL: Proper Scoring Rules, the RLCR Reward, and Why Binary Rewards Teach Guessing50 min25The Name RLCD: What TypeSafe AI Claims, What Is Public, and the OpenJev Reconstruction50 min
Part 7

Case Studies: Documented Post-Training Recipes, 2024-2026

Seven recipes read stage by stage from their primary technical reports, showing how the mechanisms of Parts 1-6 were composed, with numbers, orderings and what each report says went wrong.

26Case Study: Llama 3 and Tulu 3 (2024) - DPO-First Post-Training, the PPO Comparison, and the Birth of RLVR50 min27Case Study: DeepSeek-R1 - R1-Zero, the Four-Stage R1 Pipeline, Its Compute, and What the Nature Version Changed45 min28Case Study: Kimi k1.5 to K3 - Mirror-Descent Policy Optimization, Self-Critic Rewards, Token-Budget RL, and Nine-Expert MOPD45 min29Case Study: Qwen3 - Four-Stage Reasoning Pipeline, 3,995 Query-Verifier Pairs, Thinking-Mode Fusion, and Strong-to-Weak Distillation40 min30Case Study: DeepSeek-V3.2 to V4 - Scaling GRPO Past 10% of Pre-Training Compute, Then Replacing Mixed RL with On-Policy Distillation40 min31Case Study: Olmo 3 - OlmoRL, RL-Zero from a Fully Open Base, and Measured Rollout Economics45 min32Case Study: GLM-5 and Nemotron 3 Ultra - Asynchronous Agentic RL, Deterministic Indexer Top-k, Environments the Report Calls Open, and the Limits of MOPD50 min
Part 8

Choosing Methods and Reading the Frontier

A decision framework built from the evidence of the course, and an honest map of what is unsettled as of October 2026.

33Choosing a Method: PPO, DPO, GRPO-Family, On-Policy Distillation, and Hybrids Under Reward, Data, Compute, and Stability Constraints45 min34The Frontier as of October 2026: Open Problems, Moving Parts, and What Closed Labs Do and Do Not Disclose45 min

Appendices

ABibliography