Reinforcement Learning for LLM Post-Training · Module 31

Case Study: Olmo 3: OlmoRL, RL-Zero from a Fully Open Base, and Measured Rollout Economics

Olmo 3 is the one recipe in this course's source map that publishes its data, code, logs and rollout-infrastructure numbers together. This module reads its post-training stage by stage: what SFT, DPO and RL each bought at 7B and 32B, the seven changes OlmoRL makes to GRPO and which of them the report actually tests, and the rollout economics of a learner that idles most of a 1,000 s step. Afterwards you can reproduce the economics of a rollout-bound run from the published numbers, spot the places where those numbers disagree with each other, and decide which OlmoRL choices to copy.

Course 45 min read 14 sources As of 2026-10-27 Prerequisites: Modules 1, 11, 14, 20
Olmo 3 post-training as three lanes from one base model: Think (SFT, DPO, OlmoRL), Instruct (SFT, DPO with length control, OlmoRL) and RL-Zero (OlmoRL directly from the base), each annotated with prompt counts Olmo 3 Base 7B · 32B Think SFT 2,268,468 prompts 2 epochs DPO 200,000 prompts 1 epoch OlmoRL 104,869 prompts 750 / 1,400 steps Think 7B · 32B 3.1: 2,300 Instruct SFT function calling added DPO length control OlmoRL 171,950 prompts 450 steps (7B) Instruct 7B · 32B RL-Zero OlmoRL 13,314 math prompts decontaminated RL-Zero 5 variants no SFT, no DPO imitation preference RL prompt counts: Tables 17, 19, 20, 49
Figure 1. The three post-training lanes of Olmo 3. Think is SFT on 2,268,468 prompts (the 7B count), DPO on 200,000 prompts and RL on 104,869 prompts1 (Tables 17, 19, 20); step counts 750 (32B) and 1,400 (7B) are from Table 49, and 2,300 is the extended 32B run behind Olmo 3.1 Think1 (§4.1.2, footnote 30). Instruct reuses the pipeline with a length-controlled DPO and 171,950 RL prompts1 (§5.3-5.4, Tables 20 and 49). RL-Zero skips both earlier stages and starts RL from the base1 (§6).

1. The recipe: three stages and a base-only branch

Olmo 3 is a pair of dense models, 7B and 32B, from the Allen Institute for AI, released with "every stage, checkpoint, data point, and dependency" used to build them1 (abstract). Its reasoning variant, Olmo 3 Think, is trained in three stages, SFT, DPO and RL with verifiable rewards, and the report says it observed gains at each1 (§4). It titles the RL section "The Cherry on Top" (§4.4), and the 32B wall-clock reads the same way: about two days for SFT and about five for the final RL runs, inside a post-training window of about nine days (Module 1, Section 4.3, has the ledger)1 (§2.4). The course's source map lists Olmo 3 as the only fully open end-to-end recipe (data, code, logs) with a consolidated list of GRPO changes and measured RL-infrastructure numbers. That is the map's finding, not a survey of every open model: Tulu 3 also publishes per-phase seconds for its 405B RL run2 (§8.1). It is why this case study can check the arithmetic of Modules 1 and 14 against a primary table instead of a headline.

Table 1. The Think pipeline, its 3.1 continuation and RL-Zero, as reported1. Sizes are prompts. The 7B SFT count is Table 17's total; the 32B count is 2,253,916. The Instruct lane is in Figure 1 and Section 7.
StageDataSize (prompts)TrainingStated purpose
SFTDolci Think SFT: traces from QwQ-32B and DeepSeek-R1 over math, code, chat, safety, precise instruction following, science2,268,468 (7B)two epochs, learning-rate sweep, linear merge of two checkpoints (§4.2.2)explicit thinking traces that support accurate answers
DPODolci Think DPO: chosen from Qwen3 32B thinking, rejected from Qwen3 0.6B thinking200,000one epoch, sweep of learning rate and dataset size (§4.3.2)capability gains beyond SFT; "primes the model for effective reinforcement learning" (§2.2)
RLDolci Think RL: math, code, instruction following, general chat; verifiers plus an LM judge104,869OlmoRL, 750 steps (32B), 1,400 steps (7B) (Table 49)reasoning gains across verifiable and non-verifiable domains
RL, Olmo 3.1the same RL data, further epochssamecontinued for 21 days on 224 GPUs, to 2,300 steps (§2.4, footnote 30)shows extended RL still helps
RL-ZeroDolci RL-Zero: math, code, instruction following, general, a mix13,314 (math)OlmoRL from the base, 16K responses, 2,000 steps (Table 49)a benchmark for RLVR free of data leakage (§6)

Two stages before RL carry design choices that the RL stage depends on. The DPO stage is the delta-learning construction: the chosen response comes from a stronger model and the rejected one from a weaker model, because continued SFT on the Qwen3-32B responses lowered the report's development average while DPO that paired them with Qwen3-0.6B responses raised it1 (Table 21). Module 7 covers the mechanism. And the report's ordering evidence, that RL run from a DPO checkpoint ends above RL run from the SFT checkpoint, is Module 13's DPO-start observation. OlmoRL also succeeds the RLVR stage of Tulu 3, the same lab's earlier recipe, which ran PPO with an 8B value model and a KL coefficient of 0.05 at 8B2 (§8.1, Table 21 caption); Module 26 covers it. This module keeps to the RL stage and the systems around it, and points to Modules 1, 7, 13, 14, 16 and 20 where they already hold a slice of the same report, and to Module 19, whose reward-hacking replication used the released 7B and 32B checkpoints.

2. What RL bought, stage by stage

Table 2 gives the 32B Think model at four checkpoints and the two differences that matter: what RL added on top of DPO, and what the extra 1,550 steps of Olmo 3.1 added on top of that. The report's Table 15 does the same for 7B, with one RL column; Figure 2 plots the DPO-to-RL change for all 17 benchmarks at both sizes.

Table 2. Olmo 3 32B Think across stages, benchmark points1 (Table 14). "RL 3.0" is the initial 750-step model and "RL 3.1" the 2,300-step continuation. Differences are derived. The last column is the mean run-to-run standard deviation the report measured for that benchmark across 3 runs of 14 models (§4.1.1); a dash means the report lists none.
BenchmarkSFTDPORL 3.0RL 3.13.0 − DPO3.1 − 3.0Run std
AIME 202473.576.076.880.6+0.8+3.80.54
AIME 202566.270.772.578.1+1.8+5.6—
OMEGA43.145.250.653.4+5.4+2.80.56
MATH95.695.996.196.2+0.2+0.10.25
ZebraLogic70.574.576.080.1+1.5+4.10.56
LiveCodeBench v375.881.983.583.3+1.6−0.20.29
IFEval83.980.689.093.8+8.4+4.80.88
IFBench37.034.447.668.1+13.2+20.5—
GPQA55.757.658.156.7+0.5−1.41.48
PopQA33.137.031.930.9−5.1−1.00.16
AlpacaEval 2 LC69.178.674.269.1−4.4−5.1—
Safety64.865.368.883.6+3.5+14.8—
Change in 17 benchmark scores from the DPO checkpoint to the final RL checkpoint, for Olmo 3 32B Think and 7B Think; instruction following gains most, some benchmarks lose RL stage minus DPO stage, points (derived) 32B Think (Table 14) 7B Think (Table 15) −5 0 +5 +10 +15 MATH +0.2 +2.7 AIME 2024 +0.8 −3.0 AIME 2025 +1.8 +1.9 OMEGA +5.4 +4.5 BigBenchHard +0.7 +2.9 ZebraLogic +1.5 +5.9 AGI Eval English +0.4 +2.4 HumanEvalPlus −0.2 −1.5 MBPP+ +0.8 +1.7 LiveCodeBench v3 +1.6 +0.1 IFEval +8.4 +12.3 IFBench +13.2 +13.3 MMLU +0.2 +3.0 PopQA −5.1 −1.0 GPQA +0.5 −2.4 AlpacaEval 2 LC −4.4 +1.5 Safety +3.5 +3.0 score change, benchmark points (each benchmark has its own scale)
Figure 2. RL minus DPO for the 17 benchmarks of Tables 14 and 15, in each benchmark's own points (derived)1. Instruction following (IFEval, IFBench) and OMEGA gain at both sizes. Some benchmarks lose: PopQA and AlpacaEval at 32B; AIME 2024, GPQA and HumanEvalPlus at 7B. The 7B column is a mean of three runs (Table 15 caption); Table 14 does not say.

RL's largest gains sit on the benchmarks closest to its verifiers. IFBench rises 13.2 points at 32B and 13.3 at 7B, IFEval 8.4 and 12.3, OMEGA 5.4 and 4.5 (derived). The instruction-following reward is built from IFEval-style and IFBench-train constraints1 (§4.4.2), so IFEval is partly in-distribution; the report calls IFBench the unseen evaluation1 (§4.5), which makes the IFBench gain the cleaner evidence. Math and code move little at 32B. MATH gains 0.2, AIME 2024 gains 0.8, and that 0.8 is about 1.5 times the 0.5437 run-to-run standard deviation the report measures for AIME 2024 (Avg@32)1 (§4.1.1), so one run does not separate it from noise.

The losses are as informative as the gains. At 32B, PopQA falls 5.1 points against a 0.16 run-to-run deviation, and AlpacaEval 2 LC falls 4.4. At 7B, AIME 2024 falls 3.0 and HumanEvalPlus 1.5 (derived). The AlpacaEval drop follows the report's own finding that RL on IFEval data alone raised IFEval and lowered AlpacaEval, which mixing domains and an LM-judge reward reduced1 (§4.5; Modules 13 and 16 take that up). The report comments on the AlpacaEval drop only for the 3.1 step1 (§4.1.2). Read together (my reading), RL here improves what its verifiers measure and gives some back elsewhere; the chat reward is an LM judge and it did not stop AlpacaEval from falling at 32B.

Olmo 3.1 is the same run continued. From 750 to 2,300 steps it gains 3.8 points on AIME 2024, 5.6 on AIME 2025, 4.1 on ZebraLogic, 4.8 on IFEval and 20.5 on IFBench, and loses 5.1 on AlpacaEval1 (Table 14; the report writes "gains of 4+ points on AIME, 4 points on ZebraLogic, 4 points on IFEval, and 20 points on IFBench" and a "5-point drop" on AlpacaEval, §4.1.2). The Safety column rises from 68.8 to 83.6, which the passage I read does not discuss. The AIME 2025 gain of the extra 1,550 steps is about three times the gain of the first 750 over DPO (5.6 against 1.8, derived), and the report notes that "performance had not yet fully saturated" when compute ran out1 (footnote 30). The report shows one continuation, not repeats, so this shows that the 32B model was still improving at 2,300 steps and says nothing about where it would stop.

For the ordering of the stages, the report's development table on 7B checkpoints shows an average over a ten-benchmark subset of 70.1 for SFT, 72.7 for SFT plus DPO, 71.9 for SFT plus 1,000 RL steps and 74.1 for all three1 (Table 22, one run). RL adds 1.8 on top of SFT and 1.4 on top of DPO (derived); Module 13 reads why a DPO start is better.

3. OlmoRL as a diff against GRPO

GRPO samples \(G\) completions per prompt, takes the group mean as the baseline, normalizes each completion's loss by its own length, divides advantages by the group's standard deviation, and puts a KL penalty to a reference policy in the loss3; Dr. GRPO names the second and third of these as a length bias and a difficulty bias4 (§3.1). OlmoRL "builds on Group Relative Policy Optimization" and adopts changes from DAPO and Dr. GRPO "among others"1 (§4.4.1). The final objective is Equation 1 of the report, with the advantage of Equation 2:

$$\mathcal{J}(\theta)=\frac{1}{\sum_{i=1}^{G}|y_i|}\sum_{i=1}^{G}\sum_{t=1}^{|y_i|}\min\!\left(\frac{\pi(y_{i,t}\mid x,y_{i,<t};\theta_{\text{old}})}{\pi_{\text{vllm}}(y_{i,t}\mid x,y_{i,<t};\theta_{\text{old}})},\ \rho\right)\min\!\Big(r_{i,t}A_{i,t},\ \mathrm{clip}\big(r_{i,t},\,1-\varepsilon_{\text{low}},\,1+\varepsilon_{\text{high}}\big)A_{i,t}\Big)$$ $$r_{i,t}=\frac{\pi(y_{i,t}\mid x,y_{i,<t};\theta)}{\pi(y_{i,t}\mid x,y_{i,<t};\theta_{\text{old}})},\qquad A_{i,t}=r(x,y_i)-\operatorname{mean}\big(\{r(x,y_j)\}_{j=1}^{G}\big)$$

Read left to right: the prefactor divides by the total token count of the group, not by each completion's length; the first \(\min\) is the truncated mismatch weight, the trainer's probability over the one vLLM returned, capped at \(\rho\); the second \(\min\) is PPO's clip with separate bounds \(\varepsilon_{\text{low}}\) and \(\varepsilon_{\text{high}}\); and the advantage is a centered reward with no division by the standard deviation1 (Eq. 1-2). Module 12 Section 1.2 factors the two ratios; Module 14 covers where the mismatch comes from. Table 3 lists the seven changes with the reason the report gives for each.

Table 3. OlmoRL's seven changes to GRPO, with the reason as the report words it, the paper it takes the idea from, and whether this report tests it1 (§4.4.1, §4.4.3, §6.2). "Tested" refers to evidence inside the Olmo 3 report only.
ChangeWhat it doesReason givenTaken fromTested here?Module
Zero-gradient filteringdrops groups whose rewards are all identicalavoid "training on samples that provide zero gradient"DAPO5no separate ablation11
Active samplingrefills the batch from the result queue until it holds the full count of non-zero-gradient groups"maintain a consistent batch size"; stabilizes traininga cheaper form of DAPO's dynamic sampling, which "would oversample" by three times1yes: with and without, RL-Zero math (Fig. 26)11; Section 8
Token-level lossdivides by total tokens in the batch"to avoid a length bias"DAPO5no11
No KL lossremoves the KL penalty to the reference"allows less-restricted policy updates"; removal "does not lead to over-optimization or destabilized training"GLM-4.5, DAPO, Dr. GRPOno ablation; KL is plotted (Fig. 41)6, 13
Clip-higher\(\varepsilon_{\text{high}}=0.272\) against \(\varepsilon_{\text{low}}=0.2\)"enable larger updates on tokens"DAPO5no12
Truncated importance samplingweights the loss by the trainer-over-vLLM ratio, capped at 2.0 where usedcorrect "differences between log probabilities from the inference and training engines"Yao et al., second-hand here6no ablation; 7B Think ran without it14
No std normalizationadvantage is reward minus group meanremoves a "difficulty bias"Dr. GRPO4no11

3.1 What the report tests and what it inherits

One of the seven changes has an ablation in this report. Active sampling does: on the RL-Zero math run the share of each batch with non-zero advantage is held at 1.0, while the standard loop's share mostly sits between about 0.4 and 0.8 (read by eye from Figure 26), and the training loss variance is "greatly reduced"1 (§6.2, Figure 26). The other six are adopted on the strength of other papers. DAPO's own ladder, on a Qwen2.5-32B base, goes from 30 AIME 2024 points for naive GRPO to 36 with overlong filtering, 38 with clip-higher, 41 with soft overlong punishment, 42 with token-level loss and 50 with dynamic sampling5 (Table 1), so the evidence for items 1, 3 and 5 comes from a different model, a different data set and a different length cap. The report does not claim otherwise.

The closest thing to a whole-recipe test is a replication. The first 7B Think run used an earlier infrastructure "without pipelineRL or truncation importance sampling" and took about 15 days; the same run repeated on the newer infrastructure "achiev[ed] similar performance in just 6 days"1 (§4.4.2). That changed several things at once, so it says that the combination works and not that truncated importance sampling is needed. The report also records two things it tried and dropped: overlong filtering, which it "did not find ... gave consistent improvements"1 (footnote 34), and, for RL-Zero, masking truncated completions, which Section 8 returns to.

3.2 Where the changes enter one step

Figure 3 places the seven changes in one step. Three act between rollout and loss, and four are terms of the objective. The three on the data path sit at different points. The offline difficulty filter runs once before training, using the starting checkpoint: it drops prompts whose pass rate over eight rollouts at temperature 1.0 and top-p 1.0 exceeds 62.5%1 (§4.4.2). The zero-gradient filter runs when results arrive, and active sampling is the refill loop that keeps the learner's batch full. RL-Zero's offline rule is looser and removes only prompts "fully solved in 8 out of 8"1 (§6.1). Module 10 Section 5.3 works out how soft the 62.5% rule is: with eight rollouts it is exactly 5/8, so every prompt with at least six correct is dropped, including groups of six and seven that still carry gradient. Module 11 Sections 3 and 4 derive the standard-deviation factor and the dead-group probability, and the 32B launch script's data mix names its filtered instruction-following set IF_multi_constraints_upto5_filtered_dpo_0625_filter, a 0.625 filter against the DPO checkpoint7 (32b_think_rl.sh).

One OlmoRL step as a pipeline from prompt queue through vLLM actors, verifiers, a group filter and advantage computation to a one-minibatch learner, with the seven changes of Table 3 numbered where they act Prompt queue vLLM actors Verifiers Group filter Advantage refill until the batch is full of non-zero-gradient groups 1 2 7 offline filter, once: pass rate > 62.5% out continuous batching SymPy, tests, judge drop groups withequal rewards Learner (DeepSpeed): one minibatch, one epoch 3 divide by batch tokens 6 truncated IS, cap 2.0 5 clip 0.2 / 0.272 4 no KL term A for every token weights, in flight Numbers are the rows of Table 3: 1 zero-gradient filter, 2 active sampling, 3 token-level loss, 4 no KL, 5 clip-higher, 6 truncated IS, 7 no std.
Figure 3. One OlmoRL step with the seven changes of Table 3 numbered where they act. Items 1, 2 and 7 sit between the actors and the loss; items 3 to 6 are terms of the learner's objective. The actor and learner split follows §4.4.3 and Figure 17 of the report, and the one-minibatch learner follows Table 491.

Dropping the standard deviation has one consequence the report does not discuss. With \(A=r-\operatorname{mean}(r)\) the reward's scale multiplies the update directly, where a std-normalized advantage would cancel it (Module 10 Section 4.1). The text describes a reward of 1 or 0 for math and for instruction following, either a pass fraction or 1 or 0 for code, and a judge score in [0, 1]1 (§4.4.1). Its Figure 16 draws instruction following as the fraction of satisfied constraints (0.75), which the text's "1 if all constraints are satisfied, and 0 otherwise" does not match, and code as 0.25 under the pass-fraction variant and 0.00 under the binary one. Yet the reward axes of its Figures 18, 21 and 24 reach about 6, 8 and 10 (by eye), and the text I read does not state the scaling that produces them. If you reproduce OlmoRL, the reward scale is a hyperparameter you have to recover from the code, not a constant of the method.

3.3 With one minibatch the clip is inert and the mismatch weight is the only correction

Table 49 lists one minibatch for 7B Think, 32B Think and RL-Zero and four for 7B Instruct1. With one minibatch and one inner epoch, the learner's weights equal \(\theta_{\text{old}}\) when the loss is evaluated, so \(r_{i,t}=1\) in value and the clip cannot bind; the gradient reduces to \(A_{i,t}\nabla\log\pi\) (derived from Equation 1; Module 12 Section 1.1 gives the argument and its Table 1 carries the Olmo rows). The released trainer does this explicitly: with a single minibatch the old log-probabilities are the current ones, detached, and with several they are computed before the update7 (resolve_old_logprob in grpo_utils.py, main branch, which post-dates the report). The consequence is that \(\varepsilon_{\text{high}}=0.272\), against DAPO's 0.285 (§3.1) and for a reason the report does not give, shapes the Instruct runs and not the Think or RL-Zero runs (my reading of Equation 1, Table 49 and the code).

The mismatch weight is then the only importance correction acting in the Think and RL-Zero runs, and Table 49 sets its cap \(\rho\) at 2.0 for 32B Think and RL-Zero and leaves it unset for 7B Think and 7B Instruct1. In the trainer the vLLM log-probabilities are those of the weights that generated the tokens, so in the asynchronous runs the weight also absorbs policy lag of up to the maximum asynchrony, which the report's equation does not separate from engine mismatch7 (compute_rho_correction; my reading). Because the trainer's \(\pi(\theta_{\text{old}})\) is the detached output of the loss's own forward pass, the weight costs no extra learner pass in this configuration; Module 14's extra-pass accounting applies to setups that keep a separate old-policy pass, so its Figure 5, which adds that pass to Olmo's 125 s, is an upper bound for these one-minibatch runs. One reproduction trap: the trainer's default KL coefficient is 0.05 with the reference policy loaded7 (grpo_utils.py), and the launch scripts set it to 0.0 explicitly (--beta 0.0 in 32b_think_rl.sh and 7b_think_rl_no_pipeline.sh). Table 49 lists no KL coefficient, so "no KL" is a flag you must set, not a default you inherit.

4. Verifiers, data and mixing

Four reward types cover the four domains. Math is a rule-based SymPy equivalence check against a reference answer; code runs generated programs against test cases on AWS Lambda so that verification "does not block the trainer"; instruction following checks a set of constraint functions; general chat sends the response to an LM judge, with or without a reference answer1 (§4.4.1, footnote 35). The judge is Qwen3 32B with thinking mode off, hosted on vLLM with a 32,768-token input limit and a 2,048-token response limit1 (footnote 36). The report also tried puzzle verifiers and length-control verifiers and found neither useful downstream (footnote 36). Module 10 Section 3 covers the verifiers and Module 16 the judge that anchors the mix; the part specific to Olmo is how the prompt pool was assembled.

Table 4. Prompts per domain in Dolci-Think-RL and in the Instruct RL pool1 (Table 20). Category sums and shares are derived from the per-dataset counts, and they reproduce the stated totals of 104,869 and 171,950.
DomainRewardThink RL (prompts)Share (%)Instruct RL (prompts)Share (%)
Precise instruction followingconstraint checker30,18628.838,00022.1
MathSymPy equivalence30,18628.864,00037.2
Codeunit tests23,11022.020,00011.6
General chatLM judge21,38720.449,95029.0
Total104,869100171,950100

Four construction details are worth copying or avoiding. Code prompts that came without tests go through problem rewriting, solution generation and test-case generation, and a prompt is kept only if its solution passes more than 80% of the generated tests, after which the failing tests are removed1 (§4.4.2). Chat prompts from Tulu 3's SFT set are rewritten with GPT-4.1 to extract reference answers, sampled eight times from a Qwen2.5 7B model fine-tuned on OpenThoughts 2, and dropped if the mean F1 against the reference is below 0.1 or above 0.8, which removes both noisy and overly hard items; this is the same filter-by-pass-rate idea as Section 3.2, with F1 as the score1 (§4.4.2). WildChat items are limited to ten per role-play character, because the pool was heavily weighted toward a few characters1 (footnote 38). Eight OMEGA subtasks the model struggled with are downsampled by 50%1 (footnote 39). The mix comes out near the report's stated aim of "roughly equal amounts of data" per domain "with slightly more focus on math and instruction following" (Table 4, 28.8% each against 22.0% and 20.4%).

The mixing procedure is admitted to be partial. The authors write that RL experiments "were both long and compute-expensive, preventing us from ablating the full space of datasets and algorithmic choices"1 (§4.4.2). They ran dataset-specific runs on an intermediate SFT checkpoint and watched the first 500 to 1,000 RL steps, tested algorithm changes on math alone, and periodically ran mixture experiments to check that a mix was stable. They built the mix on the 7B model and reused it for 32B "due to compute and time constraints"1 (§4.4.2). The consequence for 32B is a pool filtered by a different model than the one being trained: the 7B model's DPO checkpoint did the offline filtering, and the 32B run "rel[ies] on active sampling", which fills batches only with groups that have a non-zero gradient for the model actually training1 (§4.4.2). Active sampling is therefore doing the 32B-specific part of the difficulty filter at run time, which is a reason to read its ablation in Section 8 as more than a stability trick.

5. The rollout system

5.1 Layout

The learner is a single DeepSpeed-distributed trainer. The actors are a pool of independent vLLM instances: the learner puts prompts on a queue, the queue dispatches them to actors, and completions come back through a results queue1 (§4.4.3, Figure 17a). The report describes this as an off-policy asynchronous design after Noukhovitch et al. A 7B actor is one GPU and generation scales by data parallelism; a 32B actor is a whole node with tensor parallelism 81 (footnote 40, Table 49). Figure 4 draws the 32B layout: 8 learner nodes and 20 actor nodes, 28 nodes of 8 H100 in all, which is the same 224 GPUs the report gives for the 21-day Olmo 3.1 continuation1 (§2.4, §4.4.3). The learner is deliberately small: "we use the minimal possible sharding configuration to fit the learner in memory and do not prioritize speed at all", and the authors suspect the 32B sharding is suboptimal1 (§4.4.3).

Cluster layout of the Olmo 3 32B RL run: 8 learner nodes of 8 GPUs, 20 actor nodes each running one tensor-parallel vLLM engine, joined by a prompts queue and a results queue Learner: 8 nodes, 64 GPUs 8 GPUs 8 GPUs 8 GPUs 8 GPUs 8 GPUs 8 GPUs 8 GPUs 8 GPUs Actors: 20 nodes, 160 GPUs vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 vLLM TP 8 Prompts queue Results queue Table 49: 64 learner GPUs, 160 actor GPUs, max asynchrony 8, minimal sharding that fits the learner Footnote 41: 1,000 s per step, 125 s of it training. Main text: learner waits 75% of the time Actors decode with continuous batching and take new weights in flight; one engine owns a whole node
Figure 4. The 32B Think RL layout: 64 learner GPUs and 160 actor GPUs, each actor one 8-GPU vLLM engine1 (§4.4.3, Table 49, Figure 17a). The utilization notes are the two statements Section 6 reconciles: footnote 41 gives 1,000 s per step with 125 s of training, and the main text says the learner "spends 75% of the time waiting for data".

5.2 The four configurations

Table 5 sets Table 49's four RL runs side by side and adds what follows from them. All four use group size 8, temperature 1.0, \(\varepsilon_{\text{low}}=0.2\), \(\varepsilon_{\text{high}}=0.272\), a constant learning-rate schedule and a 2,048-token prompt limit1 (Table 49).

Table 5. RL hyperparameters and allocation of the four Olmo 3 runs1 (Table 49). Rows marked (derived) are the writer's arithmetic: completions are prompts times 8, and "prompts drawn" is steps times prompts per batch, a lower bound because filtered groups cost extra draws under active sampling.
Row7B Think32B Think7B Instruct7B RL-Zero
Training steps1,4007504502,000
Unique prompts per batch641286432
Completions per step (derived)5121,024512256
Completions in the run (derived)716,800768,000230,400512,000
Prompt pool104,869104,869171,95013,314
Prompts drawn, epochs (derived)89,600, 0.8596,000, 0.9228,800, 0.1764,000, 4.8
Max response (tokens)32,76832,7688,19216,384
Learning rate1.0e-62.0e-61.0e-61.0e-6
Minibatches1141
Max asynchrony1888
TIS capnone2.0none2.0
Learner GPUs166488
Actor GPUs561605664
GPUs per actor (TP)1811
Actor to learner GPUs (derived)3.52.578

Three readings follow. The 7B Think column reads as the earlier infrastructure: asynchrony 1 and no truncated importance sampling, consistent with the report's description of the 15-day run1 (§4.4.2); the repository's 7B Think "no pipeline" script has exactly this shape, with nine nodes, 16 learner GPUs, 56 single-GPU engines, --async_steps 1 and --inflight_updates false7. The learner's share of GPUs is 22% for 7B Think, 29% for 32B Think, 12.5% for 7B Instruct and 11% for RL-Zero (derived), so the 32B run is the least inference-heavy of the long-response runs, not the most. And a run is smaller than its step count suggests: 32B Think is 768,000 completions, about one seventh of R1-Zero's 5.3 million (Module 1, Section 3), and the extra 1,550 steps of Olmo 3.1 draw at least 294,400 prompts in total, 2.8 passes over the pool, which agrees with the report's "additional epochs" (derived).

5.3 What the engineering changes do to the engine

The report credits its improvements with a 4x speedup of RL training1 (§2.2) and names four infrastructure changes in §4.4.3. Module 1 Section 6.1 tabulates the measured effect (Table 23: 881 to 2,949 tokens/s on one inference node, MFU 0.30% to 1.01%, MBU 12.90% to 43.21%). From the engine's side the four are as follows.

  • Continuous batching. A finished sequence's slot is backfilled at once, against static batching where "that 'slot' of the batch will remain empty until we get a new batch"1 (§4.4.3, Figure 17). The waste is \((L_{\max}-\bar L)/L_{\max}\), which at a 32K cap and a 14,628-token mean is "up to 54%" (Module 1 Section 6 derives and draws it).
  • Active sampling. Prompts are resampled continuously into the queue to replace filtered groups, so the learner's batch never shrinks1 (§4.4.3). Dynamic sampling in DAPO instead oversampled three times the batch in prompts1.
  • Inflight updates. Rather than finishing every sequence, dumping the KV cache and loading new weights, actors "immediately update the weights without pausing the engine, relying on the generation framework to be thread-safe, and continue generating, without invalidating the KV cache", for up to 4x throughput at the same resources and, per the report, "without hurting accuracy"1 (§4.4.3). A sequence in flight therefore continues with cache entries computed under older weights, so one completion can span policy versions. The report shows no table for the accuracy claim in the passage I read.
  • Threading. Actors are decoupled and each syncs on its own schedule, and the CPU side was reworked. The first continuous-batching implementation "was slower than static batching before adding a prefetch thread to our actors that constantly refilled the inference queue"1 (§4.4.3).

6. Rollout economics, reproduced

Module 1 Section 5.1 builds the GPU-second ledger from footnote 41 and Module 14 uses its 125 s of 1,000 s. This section does something narrower: it checks the report's economic statements against each other and against its own day counts, because they do not all agree.

6.1 Two idle figures and two ratios

The main text says the 32B learner "spends 75% of the time waiting for data" and that "in terms of GPU utilization, we use approximately 5x as much for inference as for training"; for 7B, with 7 inference nodes and 2 learner nodes, it says about 14x1 (§4.4.3). Footnote 41 describes "one of our main RL runs, which was broadly representative": each step averaged 1,000 s, of which 125 s was training, each batched generation took 1,000 s, generation and training overlap, and generation would have to speed up "by > 8×" before training became the bottleneck1. The footnote's 8x is \(1{,}000/125\), so it is internally consistent. It implies the learner was busy 12.5% of the time and idle 87.5%, not 75%. I rely on footnote 41 for every derivation in this module, because it is the only statement that gives two durations and says the run is representative, and treat 75% as the report's rounded headline. A reconciliation is possible but is my guess: if the learner spent another 125 s per step on work that "training" excludes, such as weight broadcast, log-probability recomputation or waiting for the last actor, it would be busy 25% of the time. The report does not say.

The two ratios are harder to square with the node counts. Treat the actors as always busy and the learner as busy a fraction \(b\) of the time. Then GPU use for inference over training is \(N_{\text{inf}}/(N_{\text{learn}}\,b)\) (derived). Table 6 evaluates it.

Table 6. Inference-to-training GPU use, \(N_{\text{inf}}/(N_{\text{learn}}\,b)\), for the node counts in §4.4.3, at three learner busy fractions \(b\) (derived). The report's stated ratios are in the last column1.
RunInference : learner nodesb = 50%b = 25%b = 12.5%Report states
32B Think20 : 85.010.020.0about 5x
7B Think7 : 27.014.028.0about 14x

The 7B figure of 14x follows from the 75% statement, \(b=25\%\). The 32B figure of 5x follows only if the learner was busy half the time, which contradicts the 75% the same sentence gives for 32B; at 75% idle the 20-to-8 node split gives 10x, and at footnote 41's 87.5% it gives 20x. The report does not show its accounting, so treat "about 5x" as unreproduced, and use the node counts and a stated duration instead of the ratio.

6.2 Which run is footnote 41?

The footnote does not say whether its 1,000 s belongs to the 7B or the 32B run. The report's day counts give a check. Dividing wall-clock by the step counts of Table 49 and §4.1.2 gives an implied seconds per step for four runs (Figure 5b): about 15 days for 1,400 steps of 7B Think on the earlier infrastructure, 926 s per step; about 6 days for the same run on the newer one, assuming the same 1,400 steps, 370 s; about 5 days for the initial 32B run's 750 steps, 576 s; and 21 days for the 1,550 extra steps of Olmo 3.1, 1,171 s1 (§2.4, §4.4.2, footnote 30). These include restarts and any lost time, and the 32B initial run lost at least a day, so subtracting that day gives 461 s.

Panel a: one Olmo 3 RL step of 1,000 seconds with 125 seconds of training. Panel b: seconds per step implied by the reported days and step counts for four runs, against the 1,000 second line of footnote 41 (a) One step in footnote 41, seconds 0 250 500 750 1000 generation 1,000 s, all actors busy training learner idle, 875 s (87.5%) 125 s generation must get 8x faster (1,000 s to 125 s) before training sets the pace (b) Implied seconds per step from the reported days and steps (derived) 7B Think, earlier infra 15 days, 1,400 steps 926 s 7B Think, newer infra 6 days, 1,400 steps 370 s 32B Think, initial 5 days, 750 steps 576 s 32B Think, 3.1 extension 21 days, 1,550 steps 1,171 s footnote 41: 1,000 s
Figure 5. (a) The step of footnote 41 drawn to scale: generation fills the 1,000 s and the learner's 125 s is 12.5% of it1 (footnote 41). (b) Seconds per step implied by the reported days and step counts1 (§2.4, §4.4.2, Table 49, footnote 30); derived, including restarts. The step counts for the 3.1 extension are 2,300 minus 750.

Footnote 41's 1,000 s sits between the 7B earlier-infrastructure run (926 s) and the Olmo 3.1 extension (1,171 s), and above the initial 32B run (at most 576 s) and the newer 7B run (370 s). That is consistent with either the 7B run before the infrastructure rewrite or the 32B continuation, and it excludes the two faster runs only if the footnote's step average describes the whole run. A second check ties the 32B reading to the throughput the footnote implies. One 32B step is 1,024 completions at the 14,628-token mean, 14,979,072 tokens (derived; the report ties the mean to its 32K-cap discussion, not to a run). Over 160 actor GPUs that is 93.6 tokens/s per GPU at 1,000 s and 80.0 at the extension's 1,171 s, within 15% of each other. So the footnote's numbers are plausible for the 32B layout, and nothing in the report excludes the 7B reading.

6.3 The speedup claims, three ways

The report states the infrastructure payoff three ways. Table 23 measures 881 to 2,949 tokens/s on one inference node, 3.35x (derived)1. Footnote 43 says an initial checkpoint (nine nodes matches the 7B layout) "took 14 straight days of training across 9 nodes to achieve 1 epoch" and, with continuous batching and inflight updates, "1 epoch on 5 nodes in 7 days", which is 126 against 35 node-days, 3.6x (derived)1. The text says "two times faster training on half as many GPUs", which is 4x in GPU-time, and gives the 7B run as about 15 days before and 6 days after (§4.4.2). With footnote 43's five nodes for the 6-day run that is 135 against 30 node-days, 4.5x (derived). The first two agree within 8%, and the range 3.35x to 4.5x brackets the headline 4x. The "up to 4x" for inflight updates alone is a different claim, an upper bound on one change, and the measured step from adding it in Table 23 is 2.17x (Module 1 Section 6.1).

7. The same loop from a different start: Instruct

Olmo 3 Instruct runs the same OlmoRL code with four changes. It trains on a different pool, 171,950 prompts with easier math and code sets and no offline difficulty filtering, because the model is aimed at general instruction following and not at hard reasoning1 (§5.4, Table 20). Responses are capped at 8,192 tokens for 7B and 16,384 for 32B, and the report says it tried both lengths at both sizes and chose them after seeing "undesirable behaviors" in an internal demo with 7B at 16K and 32B at 8K1 (§5.4.1, footnote 57). It takes four minibatches per rollout (Table 49). And it starts from a length-controlled DPO model, so the RL stage cannot lean on long chains1 (§5.5). The checkpoint is chosen by average score, response length and a qualitative "vibe test", with ties broken toward benchmarks that do not improve with test-time compute1 (§5.4.1).

Table 7. What RL added on top of DPO, Think against Instruct, in benchmark points (derived)1 (Tables 14, 15, 25, 26). Think 32B is RL 3.0; Instruct 32B is Olmo 3.1 Instruct.
BenchmarkThink 7BThink 32BInstruct 7BInstruct 32B
AIME 2024−3.0+0.8+20.8+32.6
AIME 2025+1.9+1.8+12.1+34.6
MATH+2.7+0.2+7.7+6.8
IFEval+12.3+8.4+3.6+1.5
IFBench+13.3+13.2+3.0+3.4
AlpacaEval 2 LC+1.5−4.4−2.4−9.9

The pattern inverts. Think's RL stage barely moves AIME and moves instruction following by 8 to 13 points; Instruct's RL stage moves AIME 2024 by 21 and 33 points at 7B and 32B and instruction following by 1.5 to 3.6 (derived). The reason is the start. The Instruct DPO checkpoint is a short-answer model at 23.5 AIME 2024 points (7B) and 35.2 (32B), with a lot of headroom under an 8K or 16K cap, while the Think DPO checkpoint already sits at 74.6 and 76.0. RL moves a model toward whatever its verifiers reward and its start leaves room for (my reading). The report's own explanation of why the length-controlled start works at 7B is a conjecture: with a fixed 8K window, "a shorter model may be 'more intelligent per token'"; it also reports that RL "progresses more reliably" from the length-controlled policy than from a higher-scoring uncorrected one, which "tends to show earlier signs of instability or degradation"1 (§5.5). Module 23 covers length control as a trained target. The 32B Instruct AlpacaEval fall of 9.9 points is the largest drop in either family, and the report's text I read does not comment on it.

8. RL-Zero: a decontaminated testbed

RL-Zero exists because of a measurement problem. The report observes that the leading open RLVR benchmarks and algorithms train on open-weight models that do not reveal their pretraining or mid-training data, which "limits the ability for the community to study the role of pretraining data on RLVR performance" and can leave mid-training data containing the evaluation set, "which makes spurious rewards as effective as true rewards"1 (§1, §6). The cited demonstration is Spurious Rewards: random rewards lifted MATH-500 by 21.4 points on Qwen2.5-Math-7B against 29.1 for ground truth, and the effect was model-dependent, with minimal gains or degradation on the other families tested10. Module 20 holds that debate. RL-Zero is Olmo 3's answer in construction terms: RL straight from a base whose pretraining and mid-training data are public, on prompts decontaminated against them (Figure 6).

Two RLVR benchmark pipelines: a typical open-weight base with undisclosed mid-training data and possible overlap with the evaluation set, against Olmo 3 RL-Zero from a fully open base with decontaminated data, and the five uses the report makes of RL-Zero Typical open RLVR benchmark Open-weight base mid-training data hidden RLVR, public data AIME, MATH possible overlap with the evaluation data Olmo 3 RL-Zero Dolma 3 mixes pretrain + midtrain Olmo 3 Base open weights Dolci RL-Zero 13,314 prompts OlmoRL 32 x 8 per step AIME 2024, 2025 decontaminated Uses the report makes of it Fig. 27: random reward Fig. 25: midtraining mix Fig. 26: active sampling Fig. 38: vs DAPO Fig. 24: domain mix
Figure 6. The RL-Zero design against a typical open-weight RLVR benchmark, and the five uses the report makes of it1 (§6, Figures 24 to 27 and 38). Prompt count and sampling (32 prompts by 8 samples per step) are from Table 49.

8.1 Construction

The math set is 13.3K prompts from DAPO Math, Klear-Reasoner Math, Open-Reasoner-Zero and OMEGA. DAPO is deduplicated and stripped of non-English items; the three larger sets are grouped by semantic clustering and one representative question per cluster is kept, in addition to DAPO; everything is decontaminated against pretraining and evaluation data; and prompts the final base model solves 8 of 8 are dropped1 (§6.1). The decontamination procedure is the one inherited from Tulu 3: n-gram matching with 8-grams and an overlap threshold of 0.5, with heuristics to avoid false positives on common phrases and on short math tokens1 (§4.2.1)2 (Tulu 3 §3.2). Code, instruction-following and general prompts are subsampled from Dolci Think RL. Prompts are simple zero-shot templates, because "simple" templates "greatly outperform standard post-trained templates (e.g., <think></think>)" when the base was mid-trained without special formatting; the evaluation prompts are cleaned of formatting such as \boxed{} to match1 (§6.1). The algorithm is OlmoRL with a 16K response cap, evaluated at a 32K cap and temperature 1.0 to keep diversity for pass@k1 (§6.1).

8.2 What the report uses it for

Table 8. The five uses of RL-Zero and the strength of each reading1 (§6.2). Values marked "by eye" are read from the plotted curves, since the text gives no numbers for them.
UseResultLimit on the reading
Learning curve (Fig. 24)AIME 2024 pass@1 rises from about 20 to about 52 and AIME 2025 to about 39 (by eye); pass@32 shows a "decent improvement"The pass@32 result is in a figure only, and its curve is noisy, about 60 rising to about 80 (by eye)
Against DAPO (Fig. 38)Similar AIME 2024 scores to DAPO's Qwen2.5-32B; the text says DAPO trains "an order of magnitude more steps"The figure's axes read about 2.5 times on steps and 5 times on GPU hours (by eye; Module 15 reads the same); different base, size and template; DAPO's GPU hours are "taken from the DAPO reproduction on verl"
Midtraining testbed (Fig. 25)A base with an "insufficient" reasoning mix keeps a flat response length of about 800 tokens; the other grows to about 1,800 to 2,000 (by eye)Two early bases, one run each, math reward only
Algorithm ablation (Fig. 26)Active sampling holds the non-zero-advantage share at 1.0 and "greatly reduced" loss varianceOne domain, one run; the only ablation of the seven changes
Decontamination check (Fig. 27)Random binary rewards leave ten benchmarks flat or lowerA negative control; Module 20 Section 6 reads what it does and does not exclude

Two further observations belong with the table. The mixed run, in which math, code, instruction following and general chat train together, improves every domain but each less than its single-domain run, which the report offers as a harder benchmark for multi-objective RL1 (§6.2, Figure 24). And the comparison with DAPO is the report's argument that RL-Zero is cheaper to experiment on, a claim about experiment cost that the authors support with a curve rather than a controlled comparison.

8.3 RL-Zero 3.0 to 3.1: masking truncated completions

The later Olmo 3.1 RL-Zero Math improved on the first release through two changes the appendix names: a 16K completion length instead of 12K, and not masking truncated sequences, "one of the components of DAPO", plus minor loss fixes1 (§A.6.4, Figure 39). Masking looked attractive because it left fewer completions to train on and sped up the trainer. The authors "ultimately found that variations in batch size caused by some examples masked out reduced stability" and that without training on overlong negative sequences "completion lengths were higher, on average", so "any efficiency gains in training speed from masking were outweighed by slowdowns from generating longer sequence lengths"1 (§A.6.4). The new setup "improves performance more slowly to begin with but outperforms as training goes longer, plateauing at a higher score ~50%" (Figure 39). That is a datum against one of DAPO's four techniques, whose ablation ladder gains 6 AIME points from overlong filtering5 (Table 1), in a setting where the cost of generating longer samples is visible. The launch scripts keep --mask_truncated_completions False, and the later RL-Zero Math script adds --no_resampling_pass_rate 0.875, which the report text I read does not describe7.

9. Reading the report critically: discrepancies and gaps

A report this detailed is checkable, and Table 9 lists where the check fails or the evidence is absent. The ledger is the reason the sections above separate what the report states from what is derived.

Table 9. Internal inconsistencies and gaps found while reading Olmo 3 against itself and against its repository17. "Used" says which figure this module relies on.
ItemWhat the sources sayUsedSection
Learner idle share75% in §4.4.3; 87.5% implied by footnote 41's 125 s of 1,000 s87.5% for derivations6.1
"About 5x" inference to training, 32B20 inference and 8 learner nodes at 25% busy give 10x; the 7B "14x" does follownode counts and durations, not the ratio6.1
Model size of footnote 41not stated; day counts fit the 7B earlier-infrastructure run (926 s) or the 32B extension (1,171 s)both, undecided6.2
IFEval and Table 2432B Think SFT and DPO are 83.7 and 82.3 in Table 24 and 83.9 and 80.6 in Table 14; 7B Instruct RL is 85.8 against 85.6 in Table 26Tables 14, 15, 25, 262, 7
RL-Zero domains"four benchmarking domains" in §2.2; "five" in §6 (math, code, IF, general, mix)five variants1
RL-Zero steps2,000 in Table 49; the math curves in Figures 24 and 38 run to about 2,800 (by eye)Table 49 for derived sizes5.2, 8
Instruction-following rewardbinary in the text, a fraction (0.75) in Figure 16; reward axes reach about 10 with no stated scaleneither; scale unknown3.2
Number of runsTable 15 is a mean of three runs, Table 22 one run, Table 14's caption silent32B deltas treated as single-run2
DAPO comparisontext: DAPO trains "an order of magnitude more steps"; Figure 38 reads about 2.5x on steps and 5x on GPU hours (by eye)the figure8.2
Ablations of the seven changesone (active sampling); the rest adopted from DAPO, Dr. GRPO and Yao et al.stated as inherited3.1
Launch scripts against Table 497B Think (no pipeline) and 7B Instruct scripts match its learner and actor GPUs; the 32B script (12 learner nodes, 6 TP-8 engines, 64 prompts per step) and the RL-Zero Math script (16 learner GPUs, 56 engines) do not; the README says the scripts are later snapshotsTable 49 as the record5.2, 5.3

Where experts disagree

Olmo 3 makes four choices that other strong reports make differently, and its own evidence settles none of them: it has no KL, normalization or mismatch ablation, and its RL-Zero capability reading rests on a figure. Each dispute below is stated with the evidence that exists, and the cheapest experiment the released code and data would allow.

Keep or remove the KL term?

Remove it

Olmo 31 (§4.4.1); DAPO5 (§2.3)

For long chains of thought "the model distribution can diverge significantly from the initial model", so the restriction "is not necessary" (DAPO). Olmo says removal "does not lead to over-optimization or destabilized training".

  • No reference model in memory: TRL's default coefficient is 0.0 and the reference is then not loaded11.
  • Olmo's 7B Think reward curves "steadily" grow with no KL term, "suggesting smooth training" (Figure 18).
  • No with-and-without comparison in the report. AlpacaEval fell 78.6, 74.2, 69.1 and PopQA 37.0, 31.9, 30.9 across the RL stages (Table 14), which fits drift but does not show it.

Keep a small one, with resets

R112 (§2.1); ProRL13 (§2.2); Tulu 32

R1-Zero used a coefficient of 0.001 and replaced the reference with the latest policy every 400 steps. ProRL keeps KL with periodic resets and says advice to drop it "often applies to base models prior to any supervised fine-tuning".

  • ProRL reports KL as a more stable entropy control than DAPO-style tricks from a distilled start.
  • ProRL's evidence is a 1.5B distilled model, and the reference costs memory and a forward pass.
Assessment

Module 13 Section 4 holds the ledger and the argument about ProRL's boundary; Olmo 3 Think starts RL from a DPO checkpoint and drops KL, one unmatched run against ProRL's distilled 1.5B model. What Olmo adds is a cheap test and a trap. The test: the 7B Think DPO checkpoint, data and scripts are released and a run takes about 6 days on the newer infrastructure1 (§4.4.2), so a zero-KL arm against a 0.001-with-reset arm is within reach. The trap: the trainer's default coefficient is 0.05 (Section 3.3), so "no KL" needs an explicit flag, and §2.4 mentions a day lost to "stability issues" without saying whether they were training or cluster issues.

How should loss and advantage be normalized?

Batch-token loss, no std

Olmo 31; DAPO's token-level loss5 (§3.3)

Dividing by all tokens in the batch gives every token equal weight, and dropping the std removes the difficulty weighting.

  • Removes both per-response length weighting and difficulty weighting; TRL's default loss is the DAPO form11.
  • The reward scale then sets the step size (Section 3.2), and no ablation is shown.

A constant normalizer

Dr. GRPO4 (§3.2); TRL's dr_grpo option11

Divide by a constant such as the maximum length and drop the std; it reports 43.3% on AIME 2024 with a 7B base and a minimalist recipe.

  • No per-response weight at all; the simplest form.
  • The result is on Qwen2.5-Math-7B, a base whose behavior under random rewards is itself disputed10.
Assessment

Both remove the two biases Dr. GRPO names, and ScaleRL's study of loss aggregation and normalization finds that such details "primarily modulate compute efficiency without materially shifting the asymptote"8. Olmo's choice is therefore defensible and not a place to spend an ablation budget; the ranking between the two forms is not established, and Module 11 Section 7 holds the controlled comparisons. RL-Zero Math is the cheap venue for a matched run.

Fix the rollout and trainer mismatch in the loss or in the numerics?

Reweight the loss

Olmo 3's truncated importance sampling, cap 2.01 (§4.4.1)

Multiply the loss by the trainer-over-vLLM probability ratio, truncated, following Yao et al.

  • No change to either engine; in Olmo's one-minibatch setup it costs no extra learner pass (Section 3.3).
  • Not ablated; the 7B Think run without it and its repeat with it reached "similar performance" while many other things changed.

Remove the numerical gap

Qi et al., FP166 (§1)

BF16's rounding error is the root cause. Truncated importance sampling is "insufficient" because of its biased gradient, costs about 25% extra, and leaves a deployment gap.

  • One change to precision, no extra pass.
  • Needs both engines in FP16, which the Olmo report does not test.
Assessment

Olmo's configuration weakens the cost argument against reweighting, because the weight is free in learner FLOPs, but not the bias or deployment-gap arguments, which are not about cost. Olmo's evidence cannot separate the two positions. Module 14 holds the full dispute and the other fixes.

Does RLVR add capability, or sharpen?

It sharpens what the base can already do

Yue et al.14; Spurious Rewards10

RLVR models win at small k and base models win at large k; RLVR paths already lie inside the base's sampling distribution.

  • Several families and algorithms point the same way; coverage at pass@256 falls as pass@1 rises.
  • Shorter runs on contaminated or unknown bases, which RL-Zero was built to avoid.

On a clean base RL pushes past the start

Olmo 3 RL-Zero1 (§6.2)

The authors see "a decent improvement in pass@32" and conclude the run "maintains diversity and RLVR pushes the model beyond its initial capabilities".

  • Open, decontaminated base; a random-reward control stays flat.
  • The result is in a figure, at k = 32, not at the large k where Yue et al. find the crossover.
Assessment

The two do not conflict as stated: pass@32 improvement and pass@256 coverage loss can both occur. The right platform to settle it is the one Olmo built: pass@k to k = 256 or more at every checkpoint of an RL-Zero run, with a random-reward arm. Module 20 has the full ledger.

What is changing

The report stopped Olmo 3.1 Think at 2,300 steps "due to compute limitations" with performance "not yet fully saturated"1 (footnote 30), so the extended run is the only measurement past 750 steps and has no repeat. The open-instruct main branch has moved past the report: it offers a CISPO loss, a unified mismatch correction with clamp and mask bounds including a sequence-level mask its docstring calls "DeepSeek-V3.2 style", and an RL-Zero Math script that is the later 3.1 recipe7. Expect the scripts to diverge further from the report; treat Table 49, not the repository, as the record of the published runs.

References

  1. paperTeam Olmo (Ettinger, A., Bertsch, A., et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2 (14 Apr 2026; v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: §2.2-2.4 (pipeline, wall-clock), §4.1-4.5 (Tables 14, 15, 20-24, Figures 16-21), §4.4.1 (Equations 1-2, the seven changes), §4.4.2 (data, filters), §4.4.3 (infrastructure, footnotes 40, 41, 43), §5.4-5.5 (Instruct, Tables 24-26), §6 and A.6.4 (RL-Zero, Figures 24-27, 37-39), Table 49. Page images of the tables and figures were checked against the extracted text.
  2. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124v5 (14 Apr 2025). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the prompt decontamination procedure (§3.2: 8-gram matching, more than 50% of test tokens matched) and PPO-based RLVR with a KL coefficient of 0.05 (8B) and 0.07 (70B) (Table 21 caption).
  3. paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3 (27 Apr 2024). https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the GRPO definition (group baseline, no critic, KL in the loss; §4.1.1).
  4. paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783v2 (6 Oct 2025; COLM 2025). https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the length bias and difficulty bias of GRPO (§3.1), the constant-normalizer fix and the 43.3% AIME 2024 result (§3.2).
  5. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. (ByteDance Seed, Tsinghua AIR). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2 (20 May 2025). https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: the ablation ladder (Table 1), clip-higher 0.28 (§3.1), dynamic sampling (§3.2), token-level loss (§3.3), KL removal (§2.3).
  6. paperQi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., Lin, M. Defeating the Training-Inference Mismatch via FP16. arXiv:2510.26788v1 (30 Oct 2025). https://arxiv.org/pdf/2510.26788v1. Accessed 2026-10-26. Used for: the description of truncated importance sampling as a second-hand source for Yao et al. (whose post renders only "Notion" to a fetch and was not read), its bias and roughly 25% extra cost, and the BF16 rounding argument (§1).
  7. repoAllen Institute for AI. open-instruct (main branch, fetched 2026-10-27; post-dates the Olmo 3 report). https://github.com/allenai/open-instruct; raw files fetched: open_instruct/grpo_utils.py, grpo_fast.py, data_loader.py, scripts/train/olmo3/README.md, 32b_think_rl.sh, 7b_think_rl_no_pipeline.sh, 7b_rlzero_math.sh, 7b_instruct_rl.sh. Accessed 2026-10-27. Used for: resolve_old_logprob, default clip and mismatch bounds, default KL coefficient, async_steps default, active-sampling loop, launch-script settings and the README's provenance caveat. Source added beyond the researcher's map; the scripts are snapshots, not the run configurations.
  8. paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R., et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1 (15 Oct 2025). https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: PipelineRL with optimal maximum off-policyness of 8 (§3.1) and the finding that loss aggregation and normalization modulate efficiency, not the asymptote (abstract).
  9. officialvLLM project. Engine Arguments (vLLM documentation, latest). https://docs.vllm.ai/en/latest/configuration/engine_args.html. Accessed 2026-10-27. Used for: --enforce-eager ("disable CUDA graph and always execute the model in eager mode"; the page adds that it disables torch.compile too) and --enable-prefix-caching. Source added beyond the researcher's map; the last 14,000 characters of the page were not read.
  10. paperShao, R., Li, S. S., Xin, R., Geng, S., et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947v2 (25 Feb 2026). https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: random rewards giving +21.4 against +29.1 MATH-500 points on Qwen2.5-Math-7B, and the model dependence of the effect (abstract, §1, §3).
  11. officialHugging Face. TRL documentation: GRPO Trainer (main branch; version not stated on the page). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-26. Used for: the default DAPO-style loss type, the dr_grpo option, and the default KL coefficient of 0.0 under which the reference model is not loaded.
  12. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (4 Jan 2026; the expanded Nature version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: R1-Zero's KL coefficient of 0.001 and reference replacement every 400 steps (§2.1).
  13. paperLiu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., Dong, Y. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1 (30 May 2025). https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: KL with periodic reference resets and the statement that dropping KL applies to base models before SFT (§2.2).
  14. paperYue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., Huang, G. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837v5 (24 Nov 2025). https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Used for: RLVR models winning at small k and base models at large k, and pass@256 coverage falling as pass@1 rises (abstract, Figure 1).