RL Post-Training · Part 3
Critic-Free Policy Optimization and Verifiable Rewards
GRPO and its ancestors, RL with verifiable rewards, and the loss, advantage and ratio choices that define the 2024-2026 reasoning-RL recipes.
09Critic-Free Policy Gradient: REINFORCE with Group Baselines, RLOO, GRPO, and Kimi's Mirror-Descent Objective45 min10RL with Verifiable Rewards: Rule-Based Reward Design, Prompt Selection, and What R1-Zero Showed45 min11The GRPO Loss and Advantage Under the Microscope: Length Bias, Difficulty Bias, Token-Level Averaging, and Zero-Variance Groups45 min12Importance Ratios and Clipping: Token-Level, Sequence-Level (GSPO), IS-Weight Clipping (CISPO), and Log-Ratio Masks45 min