Reinforcement Learning for LLM Post-Training · Module 1

The RL Post-Training Loop as a Cost Model: Rollout, Reward, Update, and Where the GPU-Hours Go

One RL iteration is a rollout, which is autoregressive decode of the kind you already serve, followed by an update of the kind pretraining does; which of the two dominates depends on response length, GPU split and engineering: generation dominates in Olmo 3's long-reasoning run and training in Tulu 3's 405B run. This module decomposes the loop, fixes the vocabulary every later module reuses (prompts per step, samples per prompt, minibatches per rollout), and works the published cost arithmetic of R1-Zero, Olmo 3, MiniMax-M1 and ScaleRL. Afterwards you can estimate from batch size, response length and engine throughput which phase bottlenecks a run and what it costs, and read any report's "steps" and "GPU-hours" critically.

Course 45 min read 21 sources As of 2026-10-27 First module: no prerequisites
One RL iteration as six phases, coloured by hardware shape, with reported phase durations drawn to scale inference-shaped (decode) training-shaped (fwd + bwd) forward pass or data movement light work CPU, dataset HBM-bound decode verifier / RM light arithmetic compute-bound network bytes 1. SampleP prompts 2. RolloutG per prompt 3. Scorereward signal 4. Advantageper sample 5. UpdateM minibatches 6. Weight synctrainer to engine the next rollout samples from the updated weights Reported phase durations, bars to scale (0.25 px per second) Olmo 3 generation1,000 s Olmo 3 training125 s Tulu 3 inference~550 s Tulu 3 training~1,500 s Tulu 3 weight sync~25 s
Figure 1. The loop every later module modifies. Phase structure follows DeepSeek-R1's four-module RL framework (rollout, inference, rule-based reward, training)1. Lower panel: Olmo 3's footnote gives 1,000 s per batched generation and 125 s of training for "one of our main RL runs"2; Tulu 3's 405B RLVR run reports about 550 s inference, 25 s weight transfer and 1,500 s training per iteration3. The bottleneck is not the same phase in the two runs (Section 5).

1. Where RL sits in post-training

Lambert's textbook splits modern post-training into three optimization methods: instruction or supervised fine-tuning (SFT), preference fine-tuning (RLHF and related methods), and reinforcement learning with verifiable rewards (RLVR)4. Real recipes interleave them over many rounds. Tulu 3, the fully open example, runs SFT on about 1M examples, then about 1M on-policy preference pairs, then RLVR on about 10K prompts4. DeepSeek-R1 first trains R1-Zero with RL directly on the base model; R1 itself adds a cold-start SFT stage, a large RL stage, rejection sampling into an SFT set of about 600k reasoning and 200k non-reasoning samples, and a second RL stage14. The 2026 recipes add specialist training followed by multi-teacher on-policy distillation4; Modules 21 and 22 cover that. The data-flow and evaluation-gate plumbing around these stages is System Design Module 36, and rollout infrastructure and framework choice are Modules 37 and 38; this course does not repeat them.

Post-training pipeline map: pretraining, SFT, preference tuning (RM plus PPO, or DPO), RLVR, distillation, with stages that sample from the current policy marked Pretrainingfixed corpus SFTfixed demos Preference tuning RM + PPO4 models, sampling DPO2 models, no sampling RLVRGRPO or PPOverifier reward Distillation OfflineSFT on teacher On-policystudent samples samples from the current policy inside the training loop: needs a generation engine trains on fixed data: no rollout phase (preference pairs may be near-on-policy)
Figure 2. The common ordering. Orange stages need a rollout engine in the loop; grey stages do not. Stage structure from Lambert's canonical recipes (InstructGPT, Tulu 3, DeepSeek-R1, MOPD-style recipes)4; model counts are developed in Section 7 and Table 6. Recipes reorder and repeat stages: Llama 3 repeats preference tuning for six rounds5.

1.1 Which stages sample from the current policy

The test that matters for cost is simple: does the data for this update come from the model being updated, at roughly its current weights? SFT does not; its demonstrations come from humans or from other models. Reward-model training does not; it fits fixed comparisons. DPO does not sample inside its loop either, although the preference pairs may be collected from the previous round's best model: Llama 3 trains DPO mainly on "the most recent batches of preference data collected using the best performing models from the previous alignment rounds", so that the data "conforms better to the distribution of the policy model that is being optimized in each round"5. PPO-style RLHF does sample: InstructGPT draws a prompt, generates a response from the current policy, and lets the reward model's score end the episode6. RLVR does the same with a verifier in place of the reward model3.

Rejection sampling sits between the two: it samples from a checkpoint, keeps the high-reward outputs, then trains on them as ordinary SFT. R1's final-stage SFT data is of this kind, drawn from the stage-1 RL checkpoint with only correct reasoning samples kept1. A stage that samples from the current policy must keep a generation engine running inside the training loop, and that is the whole difference in cost profile. Lambert puts it directly: RL "requires a far larger infrastructure investment relative to instruction tuning or DPO-like algorithms", yet the author argues RL keeps winning anyway4.

Table 1. Post-training stages by where their training data comes from and what is resident during the update. "Resident" counts models whose weights are in accelerator memory during that stage's update, before any rollout-engine copy (Section 7). Sources: InstructGPT6, Tulu 33, Llama 35, R11, Lambert4.
StageTraining data comes fromSamples from current policy?Models residentReported example
SFTfixed demonstrationsno1 (policy)InstructGPT: about 13k prompts; Tulu 3: about 1M examples
Reward-model trainingfixed preference comparisonsno1 (RM)InstructGPT: 33k prompts, 6B RM
DPOfixed preference pairs, per roundno (data may be near-on-policy)2 (policy, reference), or 1 with cached reference log-probsLlama 3: six rounds; Tulu 3 caches reference log-probs
PPO-RLHFdataset prompts; responses from the policy; RM scoreyes4 (policy, reference, RM, value)InstructGPT: 31k PPO prompts
RLVR (PPO or GRPO)dataset prompts; responses from the policy; verifier scoreyes3 for Tulu 3's PPO; fewer for critic-free GRPOTulu 3 PPO; R1-Zero GRPO
Rejection sampling, then SFTsamples from an earlier checkpoint, filtered by rewardsampled once, then fixed1 (policy)R1: about 600k reasoning plus 200k non-reasoning samples
Distillationteacher outputs, or student samples scored by a teacheroffline: no; on-policy: yesstudent, plus a teacher forward pass if on-policyModules 21 and 22

1.2 Where this course starts, and a correction to the lineage

You already know pretraining and SFT, so the course starts at the first stage that samples from its own policy. One correction to the request that shaped this track belongs here because it changes what you read next. DeepSeek-R1 trains with GRPO, introduced in DeepSeekMath (Shao et al., February 2024)71. TRPO (Schulman et al., 2015) is the ancestor of PPO, not of R1's algorithm directly89. Module 3 teaches TRPO to PPO and Module 9 teaches GRPO; Module 27 reads R1's pipeline stage by stage. Nor did R1 coin the term RLVR; Tulu 3 did, in November 20243.

2. One iteration, phase by phase

An iteration of RL post-training is the loop in Figure 1. The notation follows PPO's Algorithm 1: sample data with the current policy, compute advantages, optimize a surrogate for several minibatches, then repeat with the new weights9. What changes for an LLM is the cost of each step, so each phase gets its resource profile here and its mathematics in later modules.

2.1 The six phases

Sample prompts. A batch of \(P\) prompts is drawn from the training set; R1-Zero uses 32 unique questions per step1. This is dataloader work and is never the bottleneck.

Rollout. The policy generates \(G\) completions per prompt, at temperature 1 in R1-Zero, Olmo 3 and Tulu 3123. This phase runs on a serving engine. R1's framework dispatches prompts across vLLM workers holding the actor, uses expert parallelism across nodes for the MoE model with redundant copies of hot experts, and uses multi-token-prediction self-speculative decoding, which its authors say minimizes "the completion time for the longest samples"1. Olmo 3 runs one independent vLLM instance per actor, with a single GPU per actor for 7B and a node per actor for 32B2.

Score. Three kinds of scorer exist, with different hardware needs. A rule-based verifier (answer matcher, format checker, code executor) needs no model in GPU memory, but R1's authors note its "execution tends to be time-consuming" and schedule it asynchronously to overlap with rollout and inference1. A scalar reward model costs one forward pass per sample; R1's inference module loads the reward model and the reference and runs exactly that on the rollout samples1. A generative reward model costs another decode: DeepSeek-V3.2 scores general tasks with a generative reward model and a per-prompt rubric10. Module 16 covers judges.

Advantage. For GRPO the advantage is the reward minus the group mean, divided by the group standard deviation in the original formulation1; the arithmetic is negligible. For PPO it is GAE over a learned value model's predictions, which costs a value-model forward pass over every response token7. The phase is cheap in FLOPs; what differs between algorithms is what has to be resident for it (Section 7).

Update. Forward and backward passes over the sampled tokens, split into \(M\) minibatches. R1 sorts the global batch by length, distributes it across data-parallel ranks, then best-fit packs sequences into fixed-length chunks to minimize padding, and uses DualPipe pipeline parallelism1. This is pretraining-shaped work: large token batches, compute-bound.

Weight sync. The rollout engine needs the new weights before the next rollout. Tulu 3's 405B run broadcasts them over NCCL in about 25 seconds3. Kimi k1.5 reports under one minute to switch from training to inference and about ten seconds back in its colocated design11. The engineering of this step is covered in System Design Modules 37 and 38, and is not repeated here.

Table 2. Resource profile of each phase. "Bound" is the writer's classification from the cited descriptions; the measured evidence is the utilization figure from Olmo 3's infrastructure benchmark (Table 23 there: MFU 1.01% and MBU 43.21% for the final configuration, measured on the inference side)2.
PhaseRuns onBoundScales withEvidence
SampleCPUnone\(P\)R1 loads prompts and dispatches them to workers1
Rolloutinference engineHBM bandwidth, KV cache\(P \cdot G \cdot \bar L\)Olmo 3: MFU about 1%, MBU about 43%2
ScoreCPU or sandbox; RM forward; judge decodedepends on scorer\(P \cdot G\) samplesR1: rule-based reward overlapped asynchronously1
AdvantageCPU or acceleratornegligible; value forward for PPO\(P \cdot G \cdot \bar L\) for PPO onlyGRPO group statistics1
Updatetraining clustercompute\(P \cdot G \cdot \bar L \cdot N_{\text{params}}\)R1 length-sorted best-fit packing1
Weight syncnetworkbytes moved\(N_{\text{params}}\)Tulu 3 405B: about 25 s3

2.2 Placement: colocated, disaggregated, asynchronous

Because the rollout engine and the trainer want different memory layouts and different parallelism, how you place them decides how much hardware idles. Three arrangements appear in the reports.

Colocated. One pool of GPUs alternates between training and inference. Kimi k1.5 runs Megatron and vLLM in sidecar containers that share all GPUs: training finishes, Megatron offloads its memory, vLLM starts with dummy weights and receives the new ones, rolls out, is halted, and training resumes11. Hugging Face TRL's default vllm_mode="colocate" runs vLLM inside the trainer process, sharing GPU memory with the training model, and its docs warn of memory contention12. R1 offloads each module's models from VRAM to host memory or disk when the module finishes1.

Disaggregated, synchronous. Dedicated inference GPUs and training GPUs, one side waiting while the other works. Tulu 3's report states the cost plainly: synchronous training "means inference GPUs could be idling while the learner GPUs run, and vice versa"3.

Disaggregated, asynchronous. Generation overlaps training. Tulu 3 always trains on the second-latest inference batch3. Olmo 3 goes further with inflight updates: weights are updated without pausing the engine and without invalidating the KV cache, which the report credits with up to 4x throughput at the same resources2. ScaleRL's PipelineRL-\(k\) has trainers wait only if they get \(k\) steps ahead of the generators; \(k = 8\) was optimal in their ablation13. The price of overlap is staleness: samples come from weights older than the ones being updated, which is the subject of Modules 12, 14 and 18.

Three placements of the same two phases over two iterations: colocated, disaggregated synchronous, disaggregated asynchronous time axis, 0.24 px per second; Olmo 3 durations: generation 1,000 s, training 125 s (footnote 41) 0 s5001,0001,5002,000 (a) Colocated: one GPU pool time-slices all GPUs rollout rollout (b) Disaggregated, synchronous inference GPUs rollout rollout training GPUs (c) Disaggregated, asynchronous (Olmo 3) inference GPUs rollout k rollout k+1 training GPUs update k: 125 s (a) pays two switches per step: under 60 s to inference, about 10 s back (Kimi k1.5, slivers drawn to scale) (b) serialized step = 1,125 s: inference GPUs busy 88.9%, training GPUs 11.1% (derived) (c) generation never waits; training GPUs busy 125 of 1,000 s = 12.5% (derived); staleness is the price
Figure 3. The same phase durations rearranged. Lane (c) is Olmo 3's regime: the footnote reports 1,000 s per batched generation and 125 s of training per step, with generation overlapped and "entirely" the bottleneck2. Lane (a) ignores the speed-up from giving rollout all GPUs and draws Kimi k1.5's reported switch times as slivers11. Lane (b) is the serialization of the same two durations and is derived, not measured.

2.3 Minibatches per rollout are a built-in off-policy lag

PPO's Algorithm 1 collects \(NT\) timesteps, then optimizes with \(K\) epochs of minibatches of size \(M \le NT\)9. In the MuJoCo setting of its Table 3 (horizon 2048, 10 epochs, minibatch 64), one iteration performs \(2048/64 \times 10 = 320\) minibatch updates, assuming a single actor (derived; the table does not list \(N\))9. LLM recipes reuse each rollout far less. InstructGPT splits each batch of 512 into 8 minibatches of 64 and trains a single inner epoch6. R1-Zero splits each rollout of 8,192 outputs into 16 minibatches, again one inner epoch1. DAPO samples 512 prompts times 16 responses and takes 16 gradient updates per rollout step14.

The consequence is easy to miss. Only the first minibatch update of a rollout is exactly on-policy. The \(j\)-th minibatch sees samples generated by weights \(j-1\) optimizer steps old, so even with one epoch the importance ratio is not identically 1 for \(M-1\) of the \(M\) updates; on average the lag is \((M-1)/2\) steps, which is 7.5 for R1-Zero (derived). More minibatches per rollout amortize the expensive rollout over more updates, and make the data staler. ScaleRL names the same knob \(k = B/\hat B\), the number of gradient updates per generated batch, and contrasts PPO-off-policy-\(k\) with PipelineRL-\(k\)13. Modules 3 and 12 explain what the ratio and its clipping do with that lag.

3. The run-size vocabulary

Every later module uses the same six quantities, and every report defines them slightly differently. A rollout is one generation phase over \(P\) prompts with \(G\) samples each, giving \(B = P \cdot G\) completions and \(B\,\bar L\) generated tokens, where \(\bar L\) is the mean response length and \(L_{\max}\) the cap. The update then makes \(M\) optimizer steps per rollout (minibatches per rollout) for \(E\) inner epochs. A run is \(S\) optimizer steps long. TRL exposes the same knobs under other names: num_generations is \(G\) (default 8), num_iterations is \(E\) (default 1), max_completion_length is \(L_{\max}\) (default 512), and steps_per_generation or generation_batch_size sets how many training steps one generation feeds; by default "there is one generation batch processed per optimization step", that is \(M = 1\)12.

Table 3. Run sizes as reported. "Completions" are samples per rollout, \(B = P \cdot G\). Dashes mean the cited section does not state the value. Entries marked † are derived by the writer from the cited figures; DeepSeekMath-RL's \(P = 16\) assumes its "training batch size" of 1,024 counts outputs, and R1-Zero's and DAPO's \(P\) is prompts per rollout (a step uses 32).
RunPrompts \(P\)\(G\)Completions per rolloutMinibatches \(M\)Max response (tokens)Steps reportedSource
InstructGPT PPO, all sizes51215128—500 rollouts, 4,000 minibatch steps†App. C.46
DeepSeekMath-RL 7B16†641,02411,024—§4.27
R1-Zero512†168,1921632,768, then 65,53610,400 steps (1.6 epochs)§2.11
DAPO, Qwen2.5-32B512168,192†1620,480—§4.114
ScaleRL (8B, main setup)48 per update16768 per updateasync, max lag \(k=8\)14,336 (12k + 2k)about 1,000 steps per epoch§2, App. A.313
Olmo 3 Think 32B12881,024†132,768750 stepsTable 492
Olmo 3 Think 7B648512†132,7681,400 stepsTable 492

Three details of the table matter later. First, \(M\) ranges from 1 (Olmo 3, DeepSeekMath) to 16 (R1-Zero, DAPO), so the share of updates that are off-policy by construction differs by recipe. Second, \(G\) is 8 or 16 in every reasoning-RL row, versus 1 for InstructGPT's bandit setup; the group is what lets a critic-free method compute a baseline (Module 9). Third, ScaleRL found that for a fixed total batch, trading prompts for generations per prompt (8, 16, 24, 32) left the fitted scaling curves essentially unchanged13, so \(P\) and \(G\) are not independent budget knobs.

3.1 R1-Zero worked through

R1-Zero's reported setup is enough to reconstruct its whole sampling budget1. Each step uses 32 unique questions with 16 samples each, a training batch of 512. A rollout generates 8,192 outputs, split at random into 16 minibatches and trained for one inner epoch. The run lasts 10,400 steps, or 1.6 epochs.

Derivation: what 10,400 steps means in rollouts and samples (derived)

Minibatches per rollout: \(8{,}192 / 512 = 16\). Rollouts: \(10{,}400 / 16 = 650\). Prompts per rollout: \(8{,}192 / 16 = 512\). Total completions: \(10{,}400 \times 512 = 5{,}324{,}800\), equivalently \(650 \times 8{,}192\). Prompt draws: \(10{,}400 \times 32 = 332{,}800\); if 1.6 epochs is exact this implies a pool of about \(332{,}800 / 1.6 = 208{,}000\) prompts, a number the report does not state in the passage cited. Compare Olmo 3 Think 32B: \(750 \times 1{,}024 = 768{,}000\) completions, about \(1/7\) of R1-Zero's sample count.

With the reported 198 hours1, the average optimizer step took \(198 \times 3{,}600 / 10{,}400 = 68.5\) s of wall-clock on 512 GPUs (derived), so one 8,192-output rollout plus its 16 updates averaged about \(16 \times 68.5 = 1{,}096\) s, or 18.3 minutes. That is an average over a run in which response length grew many-fold (Section 5.3), so early iterations were much faster and late ones slower.

4. GPU-hour arithmetic

Five reports state what their RL phase cost, in four different units. The figures below are the reported ones; anything marked derived is the writer's arithmetic on them, shown so you can redo it.

4.1 DeepSeek-R1: Table 7 and one inconsistency

R1's supplementary Table 7 prices compute at $2 per H800 GPU-hour: R1-Zero 101K GPU-hours ($202K), SFT-data creation 5K ($10K), R1 41K ($82K), total 147K ($294K)1. The text gives the hardware: 64×8 H800 GPUs for about 198 hours for R1-Zero, and "about 4 days, or roughly 80 hours" for R11. The first reconciles: \(512 \times 198 = 101{,}376\) GPU-hours, at \(\$2\) about \(\$202{,}752\). The second does not as written. Four days is 96 hours, which would be \(512 \times 96 = 49{,}152\) GPU-hours, while \(512 \times 80 = 40{,}960\) matches the table's 41K. Table 7 and "roughly 80 hours" agree with each other, so treat 80 hours as the figure of record and "4 days" as loose wording (derived). The table has no line for the A100 pilot experiments on a 30B model that the report mentions separately1. DeepSeek-V3's own cost table states that kind of exclusion explicitly: its figures cover only the official training run and exclude "prior research and ablation experiments"15.

4.2 MiniMax-M1: a dollar figure with no stated rate

MiniMax reports that M1's full RL run completed in three weeks on 512 H800 GPUs at a rental cost of $534,70016. The passages I read do not state the hourly rate. Three weeks on 512 GPUs is \(512 \times 21 \times 24 = 258{,}048\) GPU-hours, which implies \(\$534{,}700 / 258{,}048 \approx \$2.07\) per GPU-hour (derived). At R1's \(\$2\) rate the same dollar figure is 267,350 GPU-hours, or 522 hours (21.8 days) on 512 GPUs, so the two descriptions agree to within a day. By this arithmetic M1's RL run used about 2.5 times the GPU-hours of R1-Zero (\(258{,}048 / 101{,}376 = 2.55\), derived).

4.3 Olmo 3: wall-clock by design

Olmo 3 declines the single dollar figure. Its cost section notes that costs are "often reported as a single dollar figure", gives DeepSeek-V3's $5.576M as the example, and reports elapsed time instead: about 56 days on 1,024 H100 GPUs from start of training to evaluation of the 32B Think checkpoint, which at \(\$2\) per H100-hour would be $2.75M2. Pretraining took about 47 days of that. Post-training took about 9 days across SFT, DPO and RL, because "we run each stage multiple times, sweeping over learning rates and other hyperparameters"2. The final RL runs for the initial 32B Think spanned about 5 days with at least a day lost to instability, and the best run was later continued for another 21 days on 224 GPUs to produce Olmo 3.1 Think 32B2.

Its Table 49 gives the RL allocation for 32B Think: 64 learner GPUs plus 160 actor GPUs, 224 in all2. From those two sources, and assuming all 224 GPUs were held for the whole period (derived):

  • The initial 5-day run: \(224 \times 5 \times 24 = 26{,}880\) GPU-hours.
  • The 21-day continuation: \(224 \times 21 \times 24 = 112{,}896\) GPU-hours, about \$226K at \$2 per hour.
  • The whole project window: \(1{,}024 \times 56 \times 24 = 1{,}376{,}256\) GPU-hours, and \(1{,}376{,}256 \times \$2 = \$2{,}752{,}512\), which matches the reported $2.75M.

The sweeps matter for reading anyone's RL figure. Olmo 3 says its recipe was developed on 7B or smaller and applied to 32B, and "The theory for post-training, particularly, RL, is less developed, so we have to run multiple experiments to identify the optimal hyperparameters for a given base model"2. ScaleRL makes the same point numerically: its whole study took more than 400,000 GPU-hours, its design-choice runs used 3.5K to 4K GPU-hours each, its leave-one-out ablations 16K each, and its largest single run 100K GPU-hours on an 8B dense model (the study ran on NVIDIA GB200 GPUs)13. A headline "RL run" figure is one run, not one recipe's development cost.

4.4 RL as a share of pretraining

Three reported shares, from three eras. InstructGPT: the 175B PPO-ptx model took 60 petaflops/s-days, against 4.9 for the 175B SFT model and 3,640 for GPT-36, so RLHF was \(60/3{,}640 = 1.65\%\) of pretraining compute (derived). R1-Zero: the ScaleRL authors write that R1-Zero "used 100,000 H800 GPU hours for RL training – 3.75% of its pre-training compute"13. The DeepSeek-V3 cost table lets you check that: pretraining was 2,664K H800 GPU-hours15, and \(100{,}000 / 2{,}664{,}000 = 3.75\%\); with R1's own 101,376 it is 3.81% (derived). The same table puts all of V3's post-training, SFT and RL, at 5K GPU-hours15, so R1-Zero's RL alone cost about 20 times V3's whole post-training budget (derived). DeepSeek-V3.2: the report says its RL protocol "allocates a post-training computational budget exceeding 10% of the pre-training cost"10. It gives no GPU-hours and does not say whether "pre-training cost" means V3's, V3.1-Terminus's or its own continued pre-training. If the denominator were V3's 2,664K GPU-hours, 10% would be 266,400 GPU-hours, about 2.6 times R1-Zero's RL (derived, and conditional on that reading).

For Olmo 3 the shares depend on the denominator. Its main pretraining phase was about 9.5 days on 512 GPUs plus 35 days on 1,024 GPUs2, which is \(9.5 \times 512 \times 24 + 35 \times 1{,}024 \times 24 = 976{,}896\) GPU-hours (derived; excludes midtraining and context extension). Against it the 5-day run is 2.8% and the 21-day continuation 11.6%; against the full 1,376,256 GPU-hour project window they are 2.0% and 8.2%.

Two bar panels: reported or derived RL GPU-hours per run, and RL compute as a percentage of pretraining compute, with comparability caveats A. RL GPU-hours per run (hardware differs per bar) Olmo 3 Think 32B, 5 d (H100)26,880 DeepSeek-R1 (H800)41K ScaleRL 8B largest run (GB200)100,000 R1-Zero (H800)101K Olmo 3.1 continuation (H100)112,896 MiniMax-M1 (H800)258,048 ScaleRL whole study (GB200)more than 400,000 0100K200K300K400K GPU-hours B. RL compute as % of pretraining compute (denominators differ) InstructGPT 175B PPO-ptx1.65% Olmo 3 Think 32B, 5 d2.75% R1-Zero vs V3 pretraining3.75% DeepSeek-V3.2 (reported)more than 10% Olmo 3.1 continuation, 21 d11.6% 05%10%15% as reported derived by the writer from reported figures Caveat: bars mix hardware generations, unit definitions and denominators (Table 4). Do not rank runs by bar length.
Figure 4. Reported and derived RL cost. Panel A: R1 and R1-Zero from Table 7 (41K and 101K GPU-hours)1; ScaleRL's 100,000 GPU-hour run and more than 400,000 GPU-hour study13; MiniMax-M1 from 512 GPUs for three weeks16; the two Olmo 3 bars from 224 GPUs times the reported durations2. Panel B: InstructGPT's 60 versus 3,640 petaflops/s-days6; R1-Zero's 3.75% as stated by ScaleRL13 and reproduced from V3's 2,664K GPU-hours15; Olmo 3 against its 976,896 GPU-hour main pretraining phase (derived), for both the 5-day run and the 21-day continuation2; V3.2's reported budget above 10% of pre-training cost, with no denominator10.
Table 4. How each report counts compute, and what a reader cannot recover from it. Sources as cited in each row.
ReportUnitHardwareCoversLeaves out or leaves ambiguous
DeepSeek-R1, Table 71GPU-hours; USD at $2/hH800R1-Zero (198 h), SFT-data creation, R1 (about 80 h)A100 pilots on a 30B model; "4 days" in the text versus 80 h in the table
DeepSeek-V3, Table 115GPU-hours; USD at $2/hH800pre-training, context extension, post-trainingprior research and ablations (stated)
MiniMax-M116USD rental; wall-clockH800one full RL run, three weeks on 512 GPUshourly rate not stated in the passages read
ScaleRL13GPU-hoursGB200whole study; single runs of 3.5K to 100Knot convertible to H800 or H100 hours without a factor the paper does not give
Olmo 32wall-clock days by GPU countH1005-day initial RL run; 21-day continuation; project total priced at $2/hreports wall-clock because it is "more representative" than a single dollar figure; sweeps and lost days are inside the elapsed time
DeepSeek-V3.210percent of "pre-training cost"not statedRL budgetdenominator undefined; no GPU-hours
InstructGPT6petaflops/s-daysnot stated in the passage cited175B SFT (4.9) and PPO-ptx (60)FLOP-days are not GPU-hours; efficiency is hidden

5. Where the wall-clock goes

The previous section counted GPU-hours; this one asks what the GPUs were doing. The answer differs sharply between the two runs that report a per-phase breakdown, and the difference is the useful lesson.

5.1 Olmo 3: generation is the bottleneck

Olmo 3's RL rollouts are capped at 32K tokens and average more than 10K for the reasoner models; "Inference dominated our computational costs", with 8 H100 nodes for training and 20 for inference on the 32B run2. The report says the learner "spends 75% of the time waiting for data", so GPU utilization for inference is "approximately 5x" that for training, and 14x on the 7B run, which used 7 inference nodes against 2 learner nodes2. It adds that the learner uses "the minimal possible sharding configuration to fit the learner in memory" and does not optimize for speed, unlike supervised training2. That sentence matters for cost modelling: the learner's GPU count is set by memory, not by throughput, so its utilization is whatever remains.

Footnote 41 gives the step-level numbers. For one main run, "each training step averaged 1000 seconds, of which 125 seconds was spent running training", each batched generation took 1,000 seconds, generation and training overlap, and generation would need to be faster by more than 8x before training became the bottleneck2. The 8x is \(1{,}000/125\) (derived check). Three caveats apply, and a careful reader should notice them.

  • 125 s of training in a 1,000 s step means the learner was busy 12.5% of the time, so idle 87.5%, not the 75% in the main text. The footnote says only "one of our main RL runs".
  • For the 7B figure, 7 inference nodes against 2 learner nodes at 25% learner utilization gives \(7/(2 \times 0.25) = 14\), matching the report. For 32B, 20 inference nodes against 8 learner nodes at 25% gives \(20/(8 \times 0.25) = 10\), not 5 (derived). The report does not give its accounting.
  • The footnote does not say which model size the 1,000 s refers to, so the ledger below applies it to both Table 49 configurations2.
Allocated GPU-seconds per step split into inference busy, training busy, weight sync and idle, for two Olmo 3 configurations and Tulu 3 405B Allocated GPU-seconds per step, by what the GPUs were doing (derived) Olmo 3 32B config224 GPUs, 1,000 s inference busy 71.4%learner idle 25.0% training busy 3.6% (8,000 of 224,000 GPU-s) Olmo 3 7B config72 GPUs, 1,000 s inference busy 77.8%idle 19.4% training busy 2.8% (2,000 of 72,000 GPU-s) Tulu 3 405B256 GPUs, 1,525 s training busy 92.2% inference busy 2.3% (left sliver) · weight sync 1.5% · inference idle 4.0% inference GPUs busy training GPUs busy weight sync idle Olmo rows: footnote 41's 1,000 s and 125 s applied to each Table 49 configuration. Tulu row: inference overlaps training.
Figure 5. Where one step's allocated GPU-seconds go. Olmo 3: 160 actor plus 64 learner GPUs (32B) and 56 plus 16 (7B Think), 1,000 s per step with 125 s of training2. Tulu 3 405B: 16 inference GPUs and 240 training GPUs, about 550 s inference, 25 s weight transfer and 1,500 s training; the step is taken as 1,525 s with inference overlapped3. Percentages are derived: Olmo 32B is 160,000 / 8,000 / 56,000 of 224,000 GPU-s; Tulu is 8,800 / 360,000 / 6,000 / 15,600 of 390,400.

5.2 Tulu 3 405B: the bottleneck moves to the learner

Tulu 3's 405B RLVR run is the opposite regime. Inference used vLLM with 16-way tensor parallelism, and the remaining 240 of the 256 GPUs trained. Per iteration, inference took about 550 s, weight transfer about 25 s, and training about 1,500 s3. Responses were capped at 1,024 tokens and the effective batch was 1,8563. Training is about 72% of the 2,075 s sum, and with inference overlapped the 16 inference GPUs idle for about 975 s per step (derived). Figure 5 shows 92% of allocated GPU-seconds spent training.

Two different things changed relative to Olmo 3, and they are easy to conflate. Responses are at least 14 times shorter (a 1,024-token cap against Olmo 3's 14,628-token mean), so rollouts are cheap. And the learner is a 405B policy with a reference model on 240 GPUs. A generous bound shows how slow that learner was. Even if all 1,856 sequences hit the 3,072-token cap (2,048 prompt plus 1,024 response), training the policy at \(6N\) FLOPs per token17 and running a \(2N\) reference pass is \(8 \times 405 \times 10^9 \times 1{,}856 \times 3{,}072 \approx 1.8 \times 10^{19}\) FLOPs. Against \(240 \times 989\) TFLOPS \(\times 1{,}500\) s \(= 3.6 \times 10^{20}\) FLOPs of peak (989 TFLOPS is half of NVIDIA's with-sparsity figure of 1,979 TFLOPS for an H100 SXM18), that is at most about 5% utilization (derived; assumes "effective batch size" counts sequences and ignores the 8B value model)3. The report does not say why; it does say it hit NCCL timeouts and synchronization issues and trained only 75 steps because of infrastructure scaling difficulty3. So "which phase dominates" is a property of response length, GPU split and how well each side is engineered, not a law of RL.

5.3 A per-token cost model

Every sampled token is processed twice: once on the engine at decode time and once on the learner at update time. For a dense model with \(N\) non-embedding parameters, a forward pass costs about \(2N\) FLOPs per token and forward plus backward about \(6N\)17. With \(F\) the peak FLOP/s of a GPU and \(u\) its model-FLOPs utilization, the GPU-seconds per token are

$$ c_{\text{dec}} = \frac{2N}{F\,u_{\text{dec}}}, \qquad c_{\text{train}} = \frac{6N}{F\,u_{\text{train}}}, \qquad \frac{c_{\text{dec}}}{c_{\text{train}}} = \frac{u_{\text{train}}}{3\,u_{\text{dec}}}. $$

Decode runs at very low utilization because each step reads all the weights and the KV cache to emit one token per sequence. Olmo 3's infrastructure benchmark measures MFU 1.01% and MBU 43.21% on the inference side for its final configuration2. That is the bandwidth-bound signature you know from serving; the RL-specific point is that this regime sets the bill. Training utilization is not reported directly, but the footnote's 125 s lets you back it out.

Derivation: the 125 s training segment as a 6ND estimate (derived, 32B configuration assumed)

Tokens per step: \(128 \times 8 = 1{,}024\) completions at the 14,628-token mean, \(1{,}024 \times 14{,}628 = 1.498 \times 10^{7}\) (prompt tokens ignored). Training FLOPs: \(6 \times 32 \times 10^{9} \times 1.498 \times 10^{7} = 2.88 \times 10^{18}\). Peak of 64 learner H100s at 989 TFLOPS dense: \(64 \times 989 \times 10^{12} = 6.33 \times 10^{16}\) FLOP/s18. Time at full utilization: \(2.88 \times 10^{18} / 6.33 \times 10^{16} = 45.4\) s. A 125 s step therefore corresponds to \(u_{\text{train}} = 45.4/125 = 36\%\). With the 10K-token lower bound on the mean (\(1.02 \times 10^{7}\) tokens) it is 25%. Both are ordinary large-scale training utilizations. If the footnote's run was the 7B configuration (16 learner GPUs, 512 completions) the same arithmetic gives about 16%, so this is a consistency check on the order of magnitude, not a measurement.

Taking \(u_{\text{train}} \approx 0.25\) to \(0.36\) and \(u_{\text{dec}} = 0.0101\), decode costs \(0.25/(3 \times 0.0101) = 8.3\) to \(11.9\) times the GPU-seconds of training for the same tokens (derived). The ledger in Figure 5 shows 160,000 inference GPU-seconds against 8,000 of training busy time for the 32B configuration, a factor of 20, which is the same order. Two consequences follow, both specific to this long-response regime.

  • Extra models barely move the FLOP bill, but they move the learner's size. A policy-sized critic adds another \(6N\) per token on the training side, which with these utilizations is about 7% of total GPU-seconds per token (derived: \(16.7\) units on top of \(215\)). The cost shows up elsewhere, as more resident parameters per learner GPU (Section 7), and in Olmo's setup the learner is already sized by memory with most of its GPU-seconds idle.
  • A rule of thumb from two runs. R1-Zero's 101,376 GPU-hours over \(5{,}324{,}800\) completions of about 6,000 tokens on average is about 3.2 GPU-hours per million sampled tokens, all-in (derived). Olmo 3's 32B configuration, \(224 \times 1{,}000 / 3{,}600 = 62.2\) GPU-hours for \(1.498 \times 10^{7}\) tokens, is about 4.2 (derived). Different models, hardware and accounting, but both land at 3 to 4 GPU-hours per million sampled tokens, which is a usable first estimate for a long-chain-of-thought run.

One RL step: generation time against training time (first-order model)

Sampled tokens per step
Generation time, continuous batching
Training time
Generation time over training time
Step time, overlapped
Learner busy fraction, overlapped
GPU-hours per step, overlapped
GPU-hours per million sampled tokens
Generation time under static batching

Model: \(T_{\text{gen}} = PG\bar L/(N_g r)\), \(T_{\text{train}} = 6N\,PG\bar L/(N_t F u)\) with \(F = 989\) TFLOPS dense H100 BF1618 and \(6N\) FLOPs per trained token17. The defaults are Olmo 3 32B Think's Table 49 batch (128 prompts, 8 samples, 160 actor plus 64 learner GPUs) and its 14,628-token mean2; \(r = 94\) and \(u = 36\%\) are back-solved from footnote 41's 1,000 s and 125 s, so they are effective figures that include tail idling, not engine peaks. Static batching multiplies generation time by \(L_{\max}/\bar L\), the idealized memory-bound case of Section 6. Prompt tokens, the reference pass, weight sync and scoring are ignored.

Play with it and one property is visible at once: the ratio of generation to training time does not depend on \(P\), \(G\) or \(\bar L\). Both phases scale linearly with sampled tokens, so the ratio is set by the GPU split, \(N\), \(r\) and \(u\). Length enters through \(r\): decode throughput per GPU falls as contexts grow because each token reads more KV bytes. That is the writer's reasoning, and no reported curve of \(r\) against length was found, which is why the calculator takes \(r\) as an input rather than modelling it.

6. Long tails and batching waste

RL response lengths are long-tailed against a hard cap. Olmo 3's reasoner runs have a 32K cap, a mean generation length of 14,628 tokens, and some sequences that reach the cap2. Under static batching, the batch of prompts is split over the actors, each actor generates its whole share, returns the responses, and only then receives new work; when a sequence finishes, its slot stays empty until the longest one in the batch is done2. The report gives the waste as the maximum length minus the mean, over the maximum:

$$ \text{waste} = \frac{L_{\max} - \bar L}{L_{\max}} = \frac{32{,}000 - 14{,}628}{32{,}000} = 54.3\% $$

which matches its statement that "up to 54% of our compute would have been wasted with static batching"2. Figure 6 draws the difference. With the cap at 32,768 the figure is 55.4% (derived); the report does not say which it used. "Up to" is honest: the formula is exact only if every batch contains a sequence that reaches the cap. Slot-time waste translates to wall-clock waste to the extent that decode is memory-bound, where a step with few live sequences costs nearly as much as a full one because the weight reads dominate; that is the writer's reasoning, and Table 2's MBU figure describes that regime.

Gantt chart of eight decode slots under static batching, where finished slots idle, and under continuous batching, where they backfill A. Static batching: finished slots wait for the longest sequence slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 busy 46%, idle 54% (max - mean) / max (32,000 - 14,628) / 32,000 = 54.3% B. Continuous batching: a finished slot is refilled at once slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 no idle slots same slots, same horizon more sequences finished 0L_max / 2L_max decode steps (time); lengths illustrative, mean/max = 0.46
Figure 6. Eight decode slots over one batch horizon. Panel A: each slot idles after its sequence ends, so busy time is the mean length over the longest. Panel B: finished slots backfill, so the same slots complete more sequences. Sequence lengths are illustrative, chosen so mean over maximum is 0.46, about the ratio of Olmo 3's 14,628-token mean to its 32K cap2; they are not measured data.

6.1 What each fix bought

Olmo 3 benchmarked its infrastructure changes one at a time, for two hours on two 8-GPU A100 nodes, one for training and one for inference, and reports MFU and MBU on the inference side2.

Table 5. Effect of Olmo 3's RL infrastructure changes, cumulative from the OLMo 2 baseline (Table 23 of the report: 2-hour benchmark, one A100 node for training and one for inference; MFU and MBU computed on the inference side)2. The speed-up column is derived.
ConfigurationTotal tokens (Mtok)Tokens/svs OLMo 2 (derived)MFU (%)MBU (%)
OLMo 2 infrastructure6.348811.00x0.3012.90
+ continuous batching7.029751.11x0.3314.29
+ better threading9.771,3581.54x0.4619.89
+ inflight updates (Olmo 3)21.232,9493.35x1.0143.21

Continuous batching alone bought 11%, because the CPU side became the limit: the report says its first continuous-batching implementation "was slower than static batching before adding a prefetch thread to our actors that constantly refilled the inference queue"2. Threading added 39% and inflight updates 117% on top (\(2{,}949/1{,}358 = 2.17\), derived). Inflight updates replace the usual synchronization, in which every actor finishes its ongoing generations, dumps the KV cache and updates its weights, with an immediate weight update that keeps generating and keeps the KV cache2. The report's end-to-end claims are "two times faster training on half as many GPUs", the 7B Think run dropping from about 15 days to about 6, and one epoch moving from 14 days on 9 nodes to 7 days on 5 nodes, which is \(126\) node-days against \(35\) (3.6 times fewer, derived)2.

The price is algorithmic. After an inflight update, a long sequence is partly generated under older weights, with a KV cache computed by the old policy. ScaleRL's PipelineRL states this directly: generators continue "with the updated weights but a stale KV cache from the old policy"13. What that does to the importance ratio is the subject of Modules 12 and 14; the staleness taxonomy is System Design Module 37.

6.2 Attacking the tail from the algorithm side

Three more levers appear in the reports, and none comes with a speed-up figure in the passages read. Kimi k1.5's partial rollouts set a fixed output-token budget per iteration; a trajectory that exceeds it is saved to a replay buffer and continued in the next iteration, so "no single lengthy trajectory monopolizes the system's resources", and only the current iteration's segment needs on-policy computation11. Modules 18 and 28 treat this properly. R1 uses multi-token-prediction self-speculative decoding to cut "the completion time for the longest samples"1. And truncation is a cost lever with a stability edge: ScaleRL's baseline destabilized beyond about 3,500 GPU-hours because overlong generations exceeded 10% of all generations, which shrinks the effective batch, and its recipe adds interruption-based length control13.

7. Models held in memory per method

Time is half the cost model; memory is the other half, because in Olmo 3 the learner's GPU count is set by what must fit, and because the methods differ in how many models they keep resident by a factor of more than two.

7.1 What each method keeps

R1's appendix draws the comparison as a figure with trained and frozen models marked. PPO has four: policy and value model trained, reference and reward model frozen. GRPO has three: policy trained, reference and reward model frozen, with the value model replaced by a group computation1. DeepSeekMath's motivation is memory and a modelling difficulty: the PPO value function "is typically another model of comparable size as the policy model", and because the reward arrives at the last token a per-token value is hard to train7.

The "comparable size" is a design choice, not a requirement. InstructGPT used a 6B reward model and a 6B value function, the latter initialized from the former, with policies of 1.3B, 6B and 175B6; in the 175B run the critic is 3.4% of the policy (derived). Tulu 3's 405B RLVR run used an 8B value model3. When the reward is a verifier, the reward model disappears: Tulu 3 states "In RLVR, we have 3 models: the policy model, the reference policy model, and the value model", of which the policy and value are trained and the reference only does inference3.

GRPO's reference is optional. TRL's default KL coefficient is 0.0, and the docs state that at 0.0 "the reference model is not loaded, reducing memory usage and improving training speed"12. R1-Zero used 0.001 and DeepSeekMath 0.04, so those runs held a reference17. DPO keeps the policy and, nominally, a reference, but never samples. Tulu 3 removes the resident reference by precomputing and caching its log-probabilities, and runs chosen and rejected sequences in separate forward passes to save activation memory3.

7.2 The memory arithmetic

For mixed-precision Adam, a trained model with \(\Psi\) parameters holds 2\(\Psi\) bytes of half-precision weights, 2\(\Psi\) of gradients and 12\(\Psi\) of fp32 optimizer state (master weights, momentum, variance), 16\(\Psi\) in total19. A frozen model holds weights only, 2\(\Psi\) in bf16. Treating every model as policy-sized, the resident model state per method is below. It excludes activations, which at 32K-token sequences are large, and the rollout engine's own copy of the policy weights (another 2\(\Psi\), plus its KV cache, in every online row).

Table 6. Resident model state per method, all models policy-sized unless noted, weights and optimizer state only, 1 GB = 109 bytes. Bytes per parameter from the 2+2+12 accounting of ZeRO19. Every memory value is derived by the writer; the model sets are from R1's Figure 31, Tulu 33, TRL12 and InstructGPT6.
MethodTrained (16 B/param)Frozen (2 B/param)Samples?State, \(\Psi\) units7B (GB)70B (GB)
PPO-RLHFpolicy, valuereference, reward modelyes362522,520
PPO-RLVR (Tulu 3)policy, valuereferenceyes342382,380
GRPO, RM and KLpolicyreference, reward modelyes201401,400
GRPO, verifier, KL 0 (TRL default)policynoneyes161121,120
DPO, reference residentpolicyreferenceno181261,260
DPO, cached reference log-probs (Tulu 3)policynoneno161121,120
InstructGPT 175B with 6B RM and 6B value175B policy, 6B value175B reference, 6B RMyesn/an/a3,258 GB at 175B (6,300 GB if the RM and value were 175B)

Worked check for the first row at 70B: \(16\Psi + 16\Psi + 2\Psi + 2\Psi = 36\Psi = 36 \times 70 = 2{,}520\) GB. On 80 GB H100s18 that is 31.5, so at least 32 GPUs for model state alone; GRPO with a verifier and no KL is \(1{,}120/80 = 14\). For the InstructGPT row: \(16 \times 175 + 16 \times 6 + 2 \times 175 + 2 \times 6 = 3{,}258\) GB, against \(36 \times 175 = 6{,}300\) GB if the critic and reward model were policy-sized, a factor of 1.9 (derived).

Resident is not peak. R1's framework offloads each module's models from VRAM to host memory or disk when the module finishes1, and Kimi k1.5 offloads Megatron's GPU memory before vLLM starts11. Frozen models are the cheapest to time-multiplex. The states also understate the learner: Olmo 3's 32B learner holds about \(16 \times 32 = 512\) GB of states (derived), under 7 H100s' worth of HBM, yet uses 64 learner GPUs; the report calls its sharding "minimal" for fitting the learner and suspects it is suboptimal2. At long sequence lengths, activations and layout, not weights, decide the learner's size.

7.3 Compute per sampled token, and what the methods cost in practice

Counting FLOPs per sampled token with equal-sized models and the \(2N\) and \(6N\) rules of Section 5.3: PPO-RLHF is \(2\) (decode) \(+\ 6\) (policy) \(+\ 6\) (value) \(+\ 2\) (reference) \(+\ 2\) (reward model) \(= 18N\); GRPO with a reward model and KL is \(12N\); GRPO with a verifier and no KL is \(8N\) (derived). The ratio of 2.25 between the extremes understates the wall-clock gap, since in Olmo's regime decode's low utilization dominates the time and the extra models mostly cost resident memory.

DPO has no decode phase at all, which is where it saves most. Tulu 3 measured both in a controlled, untuned setup: PPO reached comparable but slightly lower average scores, at about 28 hours on two nodes against about 4 hours on one for DPO3. That is 7 times the wall-clock and 14 times the node-hours (\(56\) against \(4\), derived). Llama 3's team reports the same direction at scale: DPO "required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval"5. These are cost measurements of one stage, not a verdict on methods; the quality side is in Modules 7 and 33.

Where experts disagree

The cost model sharpens two disagreements that the reports leave open. The first is whether online RL repays its extra cost for preference tuning, where a cheaper offline method exists. The second is how large a share of the total compute budget RL should be, which the reports answer with figures that look conflicting (3.75% for R1-Zero, more than 10% for DeepSeek-V3.2) until their denominators are examined. The first question is decided on quality in Modules 7 and 33; only its cost side is weighed here.

A. Offline preference optimization is enough; online RL does not repay its cost

Llama 3 team5; Tulu 3 team3

Llama 3 chose simple post-training (SFT, rejection sampling, DPO) over RL algorithms that "tend to be less stable and harder to scale", tried PPO, and found DPO "required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval"5. Tulu 3 measured PPO at comparable but slightly lower scores for about 28 hours on two nodes against about 4 hours on one for DPO, and kept DPO for preference iteration and PPO for RLVR3.

  • 7 times the wall-clock and 14 times the node-hours in Tulu 3's measurement (derived); no rollout engine, no weight sync, no staleness.
  • Preference data can be refreshed each round from the latest model, as Llama 3 does over six rounds, which recovers much of the on-policy benefit.
  • Tulu 3's comparison was in a "Non-Tuned Setup", and its authors say more compute or tuning could plausibly raise PPO's score3.
  • Neither report shows an offline method matching RL where the reward is a verifier; Tulu 3 used PPO there.

B. Tuned online RL beats DPO, so the cost is worth paying

Xu et al.20; Ivison et al.21; Lambert4

Xu et al. report that PPO "is able to surpass other alignment methods in all cases" tested, including code competitions20. Ivison et al. find PPO ahead of DPO by up to 2.5% in math and 1.2% in general domains, while ranking the factors as preference data quality first, then algorithm, then reward model, then extra prompts21. Lambert argues that RL-based methods "continue to win out" despite needing "a far larger infrastructure investment", and that RL can be surgical: Tulu 3's math-only RL kept general capability4.

  • Measured gains in controlled comparisons; and reasoning models come from either a large RL run or distillation from one4.
  • RL's effect stays confined to the prompt distribution, so a narrow RL stage does not squash other skills4.
  • The Ivison margins (up to 2.5 and 1.2 points) are small next to the data-quality effect they rank higher.
  • "Tuned" carries the cost: PPO needs hyperparameter search that DPO largely avoids (Table 6 and Section 7.3 give the resource side).

C. RL is a modest slice of the budget; cost is not the constraint

InstructGPT authors6; R1-Zero's share as computed by the ScaleRL authors13

InstructGPT's lesson 1 is that "the cost of increasing model alignment is modest relative to pretraining": 60 against 3,640 petaflops/s-days, 1.65% (derived)6. R1-Zero's RL is 3.75% of its pretraining compute by the ScaleRL authors' figure, which reproduces from V3's table1315.

  • The figures are small and, in two cases, checkable from primary tables.
  • Olmo 3's 5-day 32B run is about 2.75% of its main pretraining GPU-hours (derived)2.
  • They count one final run; Olmo 3 states that RL needed many experiments2, and ScaleRL's study was over 400,000 GPU-hours13.
  • The share has been rising since 2022, so a 2022 figure is not a forecast.

D. RL is becoming a first-class budget line and will keep growing

DeepSeek-V3.2 authors10; ScaleRL authors13; Olmo 3 team2

V3.2 states an RL budget exceeding 10% of pre-training cost, observes "consistent performance improvements correlating with extended RL training budget", and hypothesizes that more budget would help further10. ScaleRL fits sigmoidal compute curves and notes, citing others, that RL compute grew by more than 10 times from o1 to o3 (a secondary claim here)13. Olmo 3 continued its best RL run for 21 more days after release2.

  • Reported gains keep following compute in three separate reports.
  • ScaleRL's extrapolations from 50K GPU-hours held at 100K on an 8B model13.
  • V3.2 gives no denominator, so ">10%" cannot be converted to GPU-hours.
  • R1's own distillation experiment found that RL on Qwen2.5-32B for over 10K steps only matched QwQ-32B-Preview while distilling R1 did significantly better1, so for small models compute spent on RL is not obviously the best use.
Assessment

A and B answer a question about preference tuning and get compatible answers: online RL buys at most a few points in the comparisons that favor it (Ivison: up to 2.5 and 1.2), and cost 7 to 14 times as much in the one measurement that reports cost (Tulu 3, where it did not win). DPO is the sensible default when preference data is good and no verifier exists, and PPO or GRPO earn their cost where the reward is verifiable. That is my reading, and the evidence is thin: one untuned PPO baseline and one team's statement about Llama 3. A matched-compute comparison on identical data, with GPU-hours defined, would settle it. C and D are not in conflict. They are 1.65% (2022), 3.75% (2025) and more than 10% (late 2025) on three different denominators, a rising trend rather than a dispute; the one claim that needs the denominator to be stated is V3.2's. For planning, budget by sampled tokens, about 3 to 4 GPU-hours per million in the two long-response runs worked above (derived), not by step count, which Section 3 showed is not comparable across reports.

What is changing

The cost structure this module describes is a snapshot of rollout-heavy, verifier-reward RL as of late 2026. Two shifts are visible. Lambert's textbook records that multi-teacher on-policy distillation (MOPD) became a popular final stage in 2026 recipes, with Nemotron 3 Ultra at more than ten experts, Kimi K3 at nine and DeepSeek V4 at more than ten, while labs "are far from unanimous in its usage"4; a final stage that distills several specialist teachers into one student (Modules 21, 22 and 30). And reporting is moving toward phase times, as Olmo 3's wall-clock accounting and Tulu 3's per-phase seconds show, which would make the ledgers in Figure 5 measurable instead of derived. Treat the "more than 10%" and "3.75%" figures as dated: they are likely to move within a year.

References

  1. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (4 Jan 2026; expanded version of Nature 645, 633-638, 2025). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: Fig. 1 caption and §2.1 (10,400 steps, 32 questions x 16 outputs, 8,192-output rollouts in 16 minibatches, one inner epoch); Supp. A.3 and Figure 3 (PPO and GRPO model sets); Supp. B.1 (RL framework modules); Supp. B.3.3 (SFT data sizes); Supp. B.4.4 and Table 7 (GPU-hours); Supp. F.1 (distillation versus RL).
  2. paperTeam Olmo (Ettinger, A., Bertsch, A., et al.). Olmo 3. Allen Institute for AI, arXiv:2512.13961v2 (14 Apr 2026). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: §2.4 (wall-clock and cost), §4.4.3 (OlmoRL infrastructure, footnotes 40, 41 and 43), Table 23 (infrastructure ablation), Table 49 (RL hyperparameters and GPU allocation).
  3. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124v5 (14 Apr 2025; v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: §5.4.1 (PPO versus DPO cost), §5.4.2 (cached DPO log-probs), §6.3 (RLVR infrastructure, three models), §8.1 (405B phase times), Table 35 (405B hyperparameters), RLVR coinage.
  4. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12 (11 Sep 2026); Manning print edition 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: §1 (p. 7, three optimization methods), §3.1.6 (RL's advantages and cost), §3.2.1 to §3.2.4 (canonical recipes, MOPD expert counts), pp. 27 to 31.
  5. paperGrattafiori, A., Dubey, A., Jauhri, A., et al. (Llama Team, AI Meta). The Llama 3 Herd of Models. arXiv:2407.21783v3 (23 Nov 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: §1 (design stance), §4.1.4 (DPO versus PPO compute, on-policy preference data), §4.1.6 (six rounds).
  6. paperOuyang, L., Wu, J., Jiang, X., Almeida, D., et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv:2203.02155v1. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: §3.2 (dataset sizes), §3.5 (RL as a bandit environment), Appendix C.4 (batch 512, minibatch 64, 256k episodes, 6B RM and value function), §6 lesson 1 (4.9 and 60 versus 3,640 petaflops/s-days).
  7. paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3 (27 Apr 2024). https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: GRPO's origin (Feb 2024), §4.1.1 (value-model cost and last-token reward), §4.2 (64 outputs, batch 1,024, KL 0.04, single update per exploration stage).
  8. paperSchulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv:1502.05477v5. https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Used for: the TRPO-to-PPO lineage pointer in Section 1.2 (treated in Module 3).
  9. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv:1707.06347v2 (28 Aug 2017). https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: §5 Algorithm 1 (N actors, T timesteps, K epochs, minibatch M), Appendix A Table 3 (MuJoCo hyperparameters), TRPO relationship (§1).
  10. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1 (2 Dec 2025). https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: §1 and §4 (RL budget above 10% of pre-training cost), §3 (generative reward model with per-prompt rubrics), §2.1.1 (continued pre-training from V3.1-Terminus).
  11. paperKimi Team (Du, A., Gao, B., Xing, B., et al.). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599v4 (3 Jun 2025). https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: §2.6.2 (partial rollouts), §2.6.3 (hybrid colocated deployment, under one minute to inference and about ten seconds back).
  12. officialHugging Face TRL maintainers. TRL documentation: GRPO Trainer. docs/trl/main (unversioned snapshot). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: GRPOConfig parameters num_generations, generation_batch_size, steps_per_generation, num_iterations, max_completion_length, beta (reference not loaded at 0.0), vllm_mode colocate and server.
  13. paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R., et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1 (15 Oct 2025). https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: §1 (R1-Zero 3.75% of pre-training compute; o1 to o3 growth as cited), §2 and App. A.3 (batch 768, 80 GB200 GPUs, 64 generators and 16 trainers, GPU-hour budgets), §3.1 and App. A.11 (PPO-off-policy-k, PipelineRL-k), App. A.6 (truncation instability).
  14. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2 (20 May 2025). https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: §4.1 (prompt batch 512, 16 responses, minibatch 512 giving 16 gradient updates per rollout step, maximum 20,480 tokens).
  15. paperDeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437v2 (18 Feb 2025). https://arxiv.org/abs/2412.19437. Accessed 2026-10-27. Used for: Table 1 (2,664K pre-training, 119K context extension, 5K post-training H800 GPU-hours at $2 per hour) and the exclusion of prior research and ablation costs (§1). Added beyond the source map.
  16. paperMiniMax (Chen, A., Li, A., et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1 (16 Jun 2025). https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: abstract and §3 (full RL run on 512 H800 GPUs in three weeks, rental cost $534,700).
  17. paperKaplan, J., McCandlish, S., Henighan, T., Brown, T. B., et al. Scaling Laws for Neural Language Models. arXiv:2001.08361v1 (23 Jan 2020). https://arxiv.org/abs/2001.08361. Accessed 2026-10-27. Used for: §2.1 (non-embedding compute of about 6N FLOPs per training token, forward plus backward). Added beyond the source map.
  18. officialNVIDIA. NVIDIA H100 Tensor Core GPU (product page, specifications table). https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-10-27. Used for: H100 SXM BF16 Tensor Core 1,979 TFLOPS with sparsity (dense taken as half, 989 TFLOPS, derived), 80 GB memory. Added beyond the source map.
  19. paperRajbhandari, S., Rasley, J., Ruwase, O., He, Y. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020; arXiv:1910.02054v3 (13 May 2020). https://arxiv.org/abs/1910.02054. Accessed 2026-10-27. Used for: §3.1 (mixed-precision Adam memory, 2 + 2 + K with K = 12, 16 bytes per parameter). Added beyond the source map.
  20. paperXu, S., Fu, W., Gao, J., Ye, W., et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv:2404.10719v3 (10 Oct 2024). https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: abstract (PPO surpasses other alignment methods in all cases tested, including code competitions).
  21. paperIvison, H., Wang, Y., Liu, J., Wu, Z., et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv:2406.09279v2 (7 Oct 2024). https://arxiv.org/abs/2406.09279. Accessed 2026-10-26. Used for: abstract (PPO ahead of DPO by up to 2.5% in math and 1.2% in general domains; data quality, algorithm, reward model, extra prompts ranking).