System design · Part 4

AI Training Infrastructure: Systems Level

Pre-training, post-training, and RL infrastructure as systems -- data, storage, scheduling, fault tolerance, and framework architecture -- with distributed-training internals deliberately excluded.

32Training Data Pipelines: Crawl/Ingest, Filtering/Dedup, Tokenization, and Sharded Datasets30 min33Storage and Checkpointing at Scale: Cluster Network Topology35 min34Cluster Scheduling and Orchestration: Slurm, the Borg/Kubernetes Lineage, Ray, and MAST35 min35Fault Tolerance and Elasticity: Checkpoint/Restart, Stragglers, Node Health, and Incident Patterns30 min36Post-Training Pipelines: SFT/Preference Data Flows, Evaluation Gates, and Model Registries35 min37RL Training Infrastructure: Environments as Services, Rollout Generation, and Trainer/Rollout Decoupling35 min38Comparing the Open-Source RL Frameworks: verl, OpenRLHF, SkyRL, AReaL, slime, and NeMo-RL40 min