RL Post-Training · Part 1

The RL Frame for a Language-Model Policy

What one RL iteration costs on a cluster, how a policy gradient is estimated for token sequences, and how trust-region ideas became the PPO clip that every later algorithm modifies.

01The RL Post-Training Loop as a Cost Model: Rollout, Reward, Update, and Where the GPU-Hours Go45 min02Policy Gradients for Token Sequences: the Log-Derivative Trick, Baselines, and Advantages for an LLM Policy45 min03From TRPO to PPO: Trust Regions, Importance Ratios, the Clipped Surrogate, and GAE50 min