Reinforcement Learning for LLM Post-Training · Module 15
RL Compute Scaling: Sigmoidal Fits, the ScaleRL Recipe, and What Longer Runs Actually Buy
ScaleRL's study of more than 400,000 GPU-hours fits its RL runs with a saturating sigmoid, and the fit separates what raises a recipe's ceiling from what only gets it there sooner. This module derives the fit and its limits, audits each ScaleRL design choice against the paper's own numbers, shows how little of the curve the 100,000 GPU-hour validation actually extrapolates, and gives a procedure for normalizing the compute claims in R1, DAPO, MiniMax-M1, Olmo 3, ProRL and DeepSeek-V3.2. You can fit a saturating curve to a run's early data and say whether a change moved the ceiling or only the speed.
1. What the study measured
ScaleRL (Khatri, Madaan et al.; Meta, UT Austin, UCL, Berkeley, Harvard, Periodic Labs) is the only study in this course's source set that treats RL compute as a quantity to fit rather than a budget to spend. It reports more than 400,000 GPU-hours of experiments on NVIDIA GB200 GPUs, almost all on one 8B dense model trained on a verifiable math task, and uses them to separate recipes by where their curves flatten and how quickly they get there1 (abstract, §1). Everything in Sections 2 to 6 rests on that one paper, so the setup matters.
- Model
- 8B dense (main); Llama-4 Scout 17B×16 MoE (scale check)1
- Start point
- SFT checkpoint: about 4 epochs, 32B tokens, 32 H100 nodes1
- Data and reward
- Polaris-53K math; ±1 from Sympy or Math-Verify checkers1
- Batch
- 768 completions = 48 prompts × 161
- Sequence budget
- 16,384 tokens = 12,288 thinking + 2,048 solution + 2,048 prompt1
- Optimizer
- AdamW, constant 5×10−7, ε = 10−15, 100 warmup steps1
- GPUs per run
- 80 GB200: 64 generators, 16 trainers1
- Budget tiers
- 3.5-4K GPU-hours per design-choice run; 16K per leave-one-out run; 30K-100K for large runs1
What is fitted is not a downstream benchmark. 1,000 prompts are held out from Polaris-53K, and every 100 training steps the model samples 16 generations per held-out prompt; the mean pass rate is the y value1 (§2.1, App. A.5). The authors follow pre-training practice in fitting in-distribution held-out performance and report AIME-24 curves only as a downstream check1 (§2.1, §7). Fits start after about 1.5k GPU-hours, roughly one epoch, because the early near-linear phase and transient instabilities do not follow the sigmoid1 (App. A.6, A.7).
1.1 Three claims, and the one this module audits
The abstract states three findings: recipes differ in their asymptotic performance; "loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote"; and stable recipes follow predictable trajectories that extrapolate from small runs1. Module 11 (Section 7.2) audited the aggregation and normalization part and found the spreads inside the paper's own run-to-run margin. Section 3 here checks the rest, in particular the off-policy-algorithm clause, against the paper's forward and leave-one-out numbers, and finds that the paper itself is not consistent about it.
The authors motivate the work with the growth of RL budgets: R1-Zero used 100,000 H800 GPU-hours, which they put at 3.75% of its pre-training compute, and they cite more than a 10× increase in RL compute from o1 to o3 and a similar leap from Grok-3 to Grok-41 (§1). The R1-Zero figure is checkable and Section 7 does so; the o1-to-o3 and Grok figures are not primary data in this course.
2. The sigmoid and what each parameter buys
2.1 The equation
ScaleRL's Equation 1 models the held-out pass rate \(R_C\) after \(C\) GPU-hours as
$$R_C - R_0 \;=\; (A - R_0)\,\underbrace{\frac{1}{1+(C_{mid}/C)^{B}}}_{g(C)}$$with \(0 \le A \le 1\) the asymptotic pass rate, \(B>0\) a scaling exponent that sets the steepness, and \(C_{mid}\) the compute at which half of the total gain \(A-R_0\) has been realized1 (Eq. 1, Fig. 3). I write \(g(C)\) for the factor, the fraction of the asymptotic gain realized at compute \(C\). The text I read does not say how \(R_0\) is obtained. The authors group the parameters by role: \(B\) and \(C_{mid}\) "primarily affect the efficiency of the run" and \(A\) is the ceiling1 (§2.1, App. A.8). Their advice follows from that split: raise \(A\) first and tune efficiency second, because a less efficient design can still end higher (Fig. 13b)1.
2.2 What the curve implies, worked through
Three consequences follow from Equation 1 by algebra (derived), and they are what you use when reading any fit. First, the compute needed to realize a fraction \(p\) of the asymptotic gain is found by setting \(g(C)=p\):
$$C_p \;=\; C_{mid}\Big(\frac{p}{1-p}\Big)^{1/B}$$For the base ScaleRL run (\(C_{mid}=2{,}542\), \(B=1.92\)1, Table 1) the 90% point is \(2{,}542\times 9^{1/1.92}=7{,}983\) GPU-hours. For the longer-context run (\(C_{mid}=11{,}272\), \(B=1.89\)) it is \(11{,}272\times 9^{1/1.89}=36{,}049\) GPU-hours, 4.5 times more. Second, the marginal return of compute, per unit of \(\ln C\), is
$$\frac{dR}{d\ln C} \;=\; (A-R_0)\,B\,g(1-g),$$which peaks at \(g=\tfrac12\) with value \((A-R_0)B/4\). With the assumed \(R_0=0.40\), the base run gains 7.0 pass-rate points from doubling compute at \(C_{mid}\), 2.5 points at 8,000 GPU-hours (\(g=0.90\)), 0.77 points at 16,000 and 0.09 points at 50,000. Third, for \(C\gg C_{mid}\) the curve becomes \(R_C\approx A-D/C^{B}\) with \(D=(A-R_0)C_{mid}^{B}\), the identity the authors give in their Appendix A.41; each doubling of compute then divides the remaining gap to \(A\) by \(2^{B}\), which is 3.78 for the base run.
Figure 2 makes the cost of the higher ceiling concrete. The base run has realized 26.6% of its gain before 1.5k GPU-hours, takes 57% in the next two doublings, and gains under 5% after 12k. The long-context run has realized 2.2% by 1.5k and takes 57% between 6k and 24k. A leave-one-out experiment at 16,000 GPU-hours (Section 3) therefore examines a base-recipe curve that has already realized \(g(16{,}000)=0.972\) of its gain: such experiments compare approaches to the ceiling more than rises toward it.
2.3 Why a sigmoid and not a power law
Pre-training curves are usually fitted with \(R_C=A-D/C^{B}\) beyond a threshold compute. ScaleRL's Appendix A.4 gives three reasons to prefer a sigmoid for RL. Accuracy is bounded. A power law is unbounded at low compute and must be fitted past a threshold, which throws away points when a run has only about 75 evaluations to fit. And empirically it extrapolates badly: on the 100,000 GPU-hour 8B run, a power law fitted on 1.5k to 50k GPU-hours predicts \(A=1.0\) against an observed plateau near 0.65, a fit on 5k to 50k predicts 0.74, and the sigmoid gives 0.645 in both windows; the power law recovers the right asymptote only when fitted on 30k to 60k, the regime one is trying to predict1 (App. A.4). The two forms coincide at high compute, as the identity above shows, so the sigmoid's advantage is the mid-range where most runs live.
A second scaling study uses a power law and no ceiling. Tan et al. trained 63 models from the Qwen2.5 family (0.5B to 72B) with GRPO on more than 50k math problems, define test loss as \(L=1-R/R_{max}\), and fit \(\log L(N,X)=-k(N)\log X+E(N)\) for compute or data \(X\), with a learning-efficiency term \(k(N)\) that saturates at a limit \(K_{max}\) as model size grows2 (Eqs. 1-2, §3.1). Loss falling as a pure power of compute implies the pass rate tends to 1, the behavior ScaleRL's Appendix A.4 reports failing on its data. The papers fit different quantities over different ranges and neither tests the other; the forms agree only while \(1-R\gg 1-A\) (derived reading).
2.4 The fitting procedure, its margin, and code
The paper fits by grid search: \(A\in\{0.450,0.455,\dots,0.800\}\) and 100 linearly spaced values of \(C_{mid}\) in \([100, 40{,}000]\); for each pair, \(B\) is fitted with SciPy's curve_fit, and the pair with the lowest sum of squared residuals wins1 (App. A.5). The authors release a fitting repository; its URL is printed in the paper. The function below is my transcription of that procedure; it takes \(R_0\) as an input because the paper does not say how it sets it.
import numpy as np
from scipy.optimize import curve_fit
def sigmoid(C, B, A, Cmid, R0):
return R0 + (A - R0) / (1.0 + (Cmid / C) ** B)
def fit_scaling_curve(C, R, R0, c_start=1500.0):
"""Grid over (A, Cmid) as in ScaleRL App. A.5; B by least squares.
C: GPU-hours at each evaluation; R: held-out pass rate (mean@16)."""
keep = C >= c_start # fits skip the first ~1.5k GPU-hours
C, R = C[keep], R[keep]
best = None
for A in np.arange(0.450, 0.8001, 0.005):
for Cmid in np.linspace(100, 40000, 100):
f = lambda c, B: sigmoid(c, B, A, Cmid, R0)
try:
(B,), _ = curve_fit(f, C, R, p0=[2.0], bounds=(0.1, 20.0))
except RuntimeError:
continue
sse = float(np.sum((f(C, B) - R) ** 2))
if best is None or sse < best[0]:
best = (sse, float(A), float(B), float(Cmid))
return dict(sse=best[0], A=best[1], B=best[2], Cmid=best[3])
The margin comes from replication. Three independent ScaleRL runs with the same settings gave fitted \(A\) varying by at most \(\pm0.015\), so the authors adopt 0.02 as the error margin on \(A\); their printed fits are (B, A) = (1.92, 0.610), (1.97, 0.610) and (2.20, 0.595)1 (App. A.5, Fig. 8a). They do not give a margin for \(B\), because algorithms with different \(A\) have different \(B\) errors. Their rule is that if two methods have \(A\) within 0.02, the one with the higher \(B\) after refitting with a common \(A\) is at least as efficient1 (App. A.5). One consequence: across three identical runs \(B\) spans 1.92 to 2.20, 15% of its smallest value (derived), so efficiency differences of that size cannot be separated from replication noise.
| Fitting window (GPU-hours) | B | A |
|---|---|---|
| 1.5k to 50k (the paper's default) | 1.70 | 0.645 |
| 5k to 50k | 1.67 | 0.645 |
| 0 to 50k | 1.7 | 0.645 |
| 0 to 100k | 1.56 | 0.655 |
3. Which choices move the asymptote
Two things are tangled in every recipe comparison: where the curve flattens (\(A\)) and how fast it gets there (\(B\), \(C_{mid}\)). Figure 3 collects every fitted \(A\) the paper prints for a before-and-after pair, with the 0.02 replication margin as a scale bar. The abstract's claim about "details" survives in the leave-one-out block and for aggregation and normalization, and fails for loss type, precision and the lag bound.
3.1 Recipes differ in ceiling
The authors re-implemented four published recipes inside their own framework and fitted them next to ScaleRL (Figure 2 and Appendix A.16 of the paper). The ceilings differ by 0.12, six times the replication margin.
| Recipe | Loss | Off-policy scheme | Batch (completions) | B | A |
|---|---|---|---|---|---|
| ScaleRL | CISPO, prompt average | PipelineRL-8 | 768 | 1.97 | 0.610 |
| MiniMax-M13 | CISPO, FP32 LM head, prompt average | PPO-off-policy | 1,280 (80 × 16), zero-variance prompts dropped | 1.77 | 0.610 |
| Magistral | DAPO-style | PipelineRL | not stated in the passage read | 1.44 | 0.535 |
| Qwen2.5 recipe (DAPO)4 | DAPO loss, εmax 0.26, prompt average | PPO-off-policy-8 | 1,280, zero-variance prompts dropped | 1.85 | 0.515 |
| DeepSeek recipe (GRPO)5 | GRPO, ε 0.2, sample average | PPO-off-policy-8 | not stated; base setup is 768 | 1.17 | 0.490 |
Two details qualify the table. DAPO and MiniMax were given a larger batch because the authors' codebase could not refill a batch as DAPO's dynamic sampling does, which if anything advantaged DAPO1 (App. A.16). The GRPO run became unstable after about 6k GPU-hours as truncations rose, and extended runs "align closely with the extrapolated curves for stable recipes like ScaleRL and MiniMax"1 (Fig. 2 caption). Whether the original labs' systems show the same ceilings is untested here.
3.2 Forward tests: large moves, from a weak baseline
The forward ablations start from a GRPO-like baseline without a KL term and with DAPO's asymmetric clip, adopt PipelineRL-8, and fit each variant at 3.5k to 4k GPU-hours1 (§2, §3.2). The legends of the paper's Figures 5a, 5b, 6a and 6b print: DAPO loss \(A=0.520\) (\(B=4.55\)), GSPO 0.590 (1.80), CISPO 0.595 (2.18); BF16 head 0.520 (4.55) against FP32 head 0.610 (1.79); no zero-variance filtering 0.520 (4.55) against filtering 0.540 (2.68); uniform prompt sampling 0.520 (4.55) against No-Positive-Resampling 0.535 (3.85)1. Aggregation and normalization (App. A.9) span only 0.015 and 0.010 in \(A\); Module 11 (Section 7.2) reads them against the margin and Module 12 (Table 2) tabulates the loss-type comparison.
Two features matter for reading these fits. First, \(B\) moves opposite to \(A\): the baseline (\(A=0.520\), \(B=4.55\)) is the control in each pair, every variant with a higher \(A\) has a lower \(B\) (1.79 to 3.85), and the lag-bound variants at or below baseline \(A\) have \(B\) of 4.14 and 4.22. A curve that flattens early at a low ceiling is steep, which is why the paper pins \(A\) before comparing efficiency. Second, the clip constant alone moves the ceiling: sweeping \(\epsilon_{max}\) in the DAPO-style loss gave \(A=0.500, 0.500, 0.510, 0.520, 0.530, 0.480\) for \(\epsilon_{max}=0.20, 0.22, 0.23, 0.25, 0.26, 0.27\), with \(B\) wandering between 1.78 and 7.321 (Fig. 19a). The authors read this as \(\epsilon_{max}\) governing "the asymptotic error itself"; one run per value and a 0.05 range (2.5 times the margin) make it real but coarse, and the spread in \(B\) shows how little a short fit says about efficiency.
3.3 Leave-one-out: ceilings converge, costs do not
Starting from the full recipe and reverting one choice at a time, at 16,000 GPU-hours each, all eight variants have a fitted \(A\) between 0.590 and 0.6101 (Fig. 7). The paper compares efficiency by averaging \(A\) over the runs (0.605), refitting, and comparing the slope \(B\); Module 11 (Figure 7 there) plots those pinned values. The paper never converts them to GPU-hours, so I do: from the printed \(C_{mid}\), \(B\), \(A\) of each variant, the compute to reach a fixed pass rate of 0.57 is \(C_{mid}\,(p/(1-p))^{1/B}\) with \(p=(0.57-R_0)/(A-R_0)\), using the assumed \(R_0=0.40\) and checking \(R_0\in\{0.35,0.45\}\).
| Reversion from the full recipe | Cmid (GPU-h) | B | A | GPU-hours to 0.57 (R0 0.40) | Cost vs full recipe | Range over R0 0.35-0.45 |
|---|---|---|---|---|---|---|
| PPO-off-policy-8 instead of PipelineRL-8 | 3,409 | 1.89 | 0.605 | 7,867 | 1.46× | 1.45 to 1.46 |
| No zero-variance filtering | 2,583 | 1.84 | 0.595 | 7,321 | 1.36× | 1.34 to 1.36 |
| DAPO loss instead of CISPO | 3,093 | 1.77 | 0.610 | 7,005 | 1.30× | 1.28 to 1.31 |
| Uniform prompt sampling (no NPR) | 2,621 | 1.62 | 0.605 | 6,953 | 1.29× | 1.24 to 1.32 |
| Prompt-level advantage normalization | 2,463 | 2.09 | 0.590 | 6,857 | 1.27× | 1.26 to 1.29 |
| Sample-average loss aggregation | 2,621 | 1.93 | 0.600 | 6,439 | 1.19× | 1.19 |
| Length penalty instead of interruptions | 2,660 | 1.68 | 0.610 | 6,294 | 1.17× | 1.14 to 1.19 |
| No FP32 at the LM head | 2,857 | 1.88 | 0.610 | 6,168 | 1.14× | 1.14 to 1.15 |
Read with the replication caveat of Section 2.4, the table supports three statements. The ceilings converge once the stability fixes are in place. The off-policy scheme and the loss type are the costliest reversions, matching the paper's list of "the most important decisions": off-policy algorithm, loss function and model precision1 (§7). Precision is the exception inside this recipe: the FP32 head is the cheapest reversion on a dense 8B model, where the paper says the fix "makes little difference" within ScaleRL but gives "large gains in GRPO/DAPO-style losses"1 (§4), and on the 17B×16 MoE its removal lowers the fitted \(A\) from 0.710 to 0.700 (Fig. 8b); Module 14 covers why the head matters. What the table does not support is reading "efficiency only" as unimportant: aggregation, normalization and curriculum, which the abstract files under efficiency, cost 19% to 29% more GPU-hours to a fixed score, though a gap in \(B\) under about 15% cannot be separated from replication (derived).
3.4 Two cautions on the forward numbers
The paper is not consistent about the off-policy algorithm. The abstract lists it among the details that "primarily modulate compute efficiency"1, while Appendix A.11 says the PipelineRL-versus-PPO distinction "affects the asymptotic performance c of the scaling curve, not just the efficiency exponent b. Very few axes shift the asymptote in this way". The numbers favor the abstract for the choice of schedule (\(A=0.520\) for PPO-off-policy-1, PPO-off-policy-8 and PipelineRL-8, Fig. 4a) and A.11 for the lag bound (\(A=0.490\) at \(k=12\) against 0.520 at \(k=4\) and \(k=8\), Fig. 4b)1.
The forward \(A\) values are also extrapolations from runs that were about to fail. The baseline destabilizes beyond about 3,500 GPU-hours as truncations exceed 10% of generations; variants survive to about 5k (\(\epsilon_{max}=0.26\)), 6k (FP32 head) and 7k (CISPO) GPU-hours1 (App. A.6). A fitted \(A=0.520\) is where an unstable recipe's curve bent before it broke, not a measured ceiling. The leave-one-out numbers, from a recipe stable to 100k, are the cleaner evidence about a single choice; the forward numbers measure "survives longer and flattens higher" jointly.
4. The ScaleRL recipe, component by component
ScaleRL achieves predictable scaling "by integrating existing methods, rather than inventing novel methods": an asynchronous PipelineRL setup with a maximum lag of 8, forced length interruptions, the CISPO loss, prompt-level loss averaging, batch-level advantage normalization, FP32 logits, zero-variance filtering and No-Positive-Resampling1 (§1, §4). Its objective, with \(\mathrm{sg}\) the stop-gradient and \(\rho_{i,t}\) the token-level ratio of trainer to generator probability, is
$$\mathcal J_{\text{ScaleRL}}(\theta)=\mathbb E\Bigg[\frac{1}{\sum_{g=1}^{G}|y_g|}\sum_{i=1}^{G}\sum_{t=1}^{|y_i|}\mathrm{sg}\big(\min(\rho_{i,t},\epsilon)\big)\,\hat A_i^{\mathrm{norm}}\,\log\pi^{\theta}_{train}(y_{i,t})\Bigg],\qquad \hat A_i^{\mathrm{norm}}=\hat A_i/\hat A_{std},$$restricted to prompts with \(0<\mathrm{mean}(r)<1\) and a historical pass rate below 0.91 (§4). Here \(\hat A_i=r_i-\mathrm{mean}(r)\) is the group-centered advantage and \(\hat A_{std}\) the standard deviation of all such advantages in the batch. Module 11 derives the token weights this implies and Module 12 covers CISPO's gradient; Table 4 records what each component is for and what the paper printed for it.
| Component | Setting | Paper's reason | Evidence | See |
|---|---|---|---|---|
| Off-policy schedule | PipelineRL, lag k = 8 | Streaming generation cuts idle time and stays closer to on-policy | Forward: B 4.55 vs 2.68, A equal at 0.520. LOO: 1.46× GPU-hours | Section 5; Module 1 |
| Length control | Forced interruption: append "Okay, time is up. Let me stop thinking and formulate a final answer now. </think>" at a random position in 10k-12k tokens | Truncated generations destabilize training | LOO length penalty: A 0.610, 1.17×. Swapping in a length penalty "does not improve performance" | Module 23 |
| Loss type | CISPO, token level, \(\mathrm{sg}(\min(\rho,\epsilon))\) | Clip the importance weight, keep every token's gradient; robust to \(\epsilon_{max}\in\{4,5,8\}\) | Forward: A 0.595 vs DAPO 0.520; GSPO 0.590. LOO DAPO: 1.30× | Module 12 |
| Loss aggregation | Prompt average | "Achieves the highest asymptotic performance" | Forward A: prompt 0.535, token 0.535, sample 0.520. LOO sample: 1.19× | Module 11 |
| Advantage normalization | Batch level: divide group-centered advantages by the std of all advantages in the batch | "Theoretically sound and marginally better"; all three variants "similar" | Forward A: none 0.525, batch 0.530, prompt 0.520. LOO prompt-level: 1.27× | Module 11 |
| Precision | FP32 at the LM head, generator and trainer | Generator and trainer kernels disagree most at the head | Forward A 0.520 to 0.610. LOO dense: 1.14×. Scout LOO: A 0.710 vs 0.700 | Module 14 |
| Zero-variance filtering | Drop prompts with all-equal rewards from the loss; no refill | Zero advantage means zero gradient; shrinks the effective batch | Forward A 0.520 to 0.540. LOO: 1.36× | Module 11 |
| Curriculum | No-Positive-Resampling: drop prompts whose historical pass rate reaches 0.9 | Once easy, a prompt stays easy and costs compute without signal | Forward A 0.520 to 0.535. LOO uniform sampling: 1.29× | Module 11 |
Two points about the table. "Batch-level" normalization means different things in two codebases. ScaleRL divides the group-centered advantages by the standard deviation of the advantages in the batch; TRL's scale_rewards="batch" pairs the group mean with a batch-level standard deviation, of the rewards in Module 11's reading of the source6. Both apply one scalar per batch, but they are not the same scalar (Module 11, Section 3.2). And the loss-type row hides a stability result that Module 12 (Section 4.5) develops: the paper's GSPO runs "diverged mid-training" on multiple occasions, restarting from a stable checkpoint recovered the 8B runs, and on the 17B×16 MoE "instability persisted despite repeated resetting to a stable checkpoint", with no implementation bug found1 (App. A.17.4). The GSPO ceiling (0.590) is within 0.005 of CISPO's, so the choice of CISPO rests on stability and on robustness to its clip constant, not on the asymptote.
What the recipe leaves out is also a choice. The base algorithm has no KL term "in line with large-scale training reports", rewards are \(\pm1\), and generations per prompt stay at 161 (§2, App. A.3). Module 13 covers the keep-or-remove-KL disagreement. The one monitoring signal the authors insist on is the truncation rate, because instabilities in their runs "were often linked to truncations"1 (App. A.15).
5. PipelineRL against PPO-off-policy: where the efficiency comes from
The one design choice the paper treats as independent of all the others is the asynchronous schedule1 (§3.1). PPO-off-policy-k, the default asynchronous scheme in Qwen3 and ProRL, has the old policy generate for a batch of \(B\) prompts; each gradient update consumes a minibatch of \(\hat B=48\) prompts, so one generation batch feeds \(k=B/\hat B\) updates, and the paper tests \(k\in\{1,8\}\). PipelineRL-k, from Piche et al. and used by Magistral, streams: generators run continuously, whenever the trainers finish an update the new weights are pushed to them at once, and they continue "with the updated weights but a stale KV cache from the old policy"; the trainers wait only if they get \(k\) steps ahead of the generators1 (§3.1).
The ServiceNow write-up of PipelineRL adds the mechanism an inference engineer would ask about. Inference is paused only "for just the time needed to receive the new weights", the in-flight sequences "proceed with stale keys and values in the KV cache", and the authors report that this "does not adversely affect stability"7. Lag there is counted in optimizer steps. The write-up reports no speedup figure and says more analysis is needed to measure it7, so ScaleRL's fits are the quantitative evidence in the source set. Whether ScaleRL stores per-token probabilities from whichever weights were live is not stated in the passages I read.
5.1 What the fits say
| Setup | B | A | Cmid (GPU-h) | GPU-hours to 0.57 |
|---|---|---|---|---|
| PPO-off-policy-1 | 2.15 | 0.520 | not printed | not derived |
| PPO-off-policy-8 | 2.68 | 0.520 | not printed | not derived |
| PipelineRL-4 | 4.14 | 0.520 | not printed | not derived |
| PipelineRL-8 | 4.55 | 0.520 | not printed | not derived |
| PipelineRL-12 | 4.22 | 0.490 | not printed | not derived |
| Full recipe (PipelineRL-8) | 1.92 | 0.610 | 2,542 | 5,401 |
| Full recipe with PPO-off-policy-8 | 1.89 | 0.605 | 3,409 | 7,867 |
On the weak baseline, PipelineRL-8 has a steepness 1.70 times that of PPO-off-policy-8 (4.55 against 2.68) at the same ceiling1 (Fig. 4a). Inside the full recipe the gap is smaller and clearer in compute units: \(C_{mid}\) is 34% higher (3,409 against 2,542) and the compute to a pass rate of 0.57 is 46% higher (derived). The authors attribute the advantage to reduced idle time and to a "tighter feedback loop" that keeps "training closer to the on-policy regime"1 (§3.1, App. A.11). No experiment separates the two. One feature of Table 5 argues against reading it as a pure staleness effect: within PPO-off-policy, the 8-update setting beats the 1-update setting (2.68 against 2.15). With \(k=1\) the generation batch is just 48 prompts, and small generation batches underuse the generators, the effect the PipelineRL authors describe as "producing small amounts of data with many GPUs is inefficient because this means the per-GPU batch size is small"7. So utilization and staleness trade against each other, and the best \(k\) is where they balance (derived reading).
The systems side is measurable independently. Olmo 3 ablated its RL infrastructure on a 2-hour benchmark, one 8×A100 node for training and one for inference, and reports utilization for inference only because inference is the bottleneck8 (Table 23, p. 51).
| Configuration | Total tokens (Mtok) | Tokens/s | MFU (%) | MBU (%) | Tokens/s vs previous row |
|---|---|---|---|---|---|
| Olmo 2 infrastructure | 6.34 | 881 | 0.30 | 12.90 | — |
| + continuous batching | 7.02 | 975 | 0.33 | 14.29 | 1.11× |
| + better threading | 9.77 | 1,358 | 0.46 | 19.89 | 1.39× |
| + inflight updates (Olmo 3) | 21.23 | 2,949 | 1.01 | 43.21 | 2.17× |
In-flight updates are the largest single step, 2.17 times the token rate, and the full change is 3.35 times (2,949 against 881; derived). The same report puts the idle compute of static batching at "up to 54%" for a 32K cap and a 14,628-token mean, which is \((L_{max}-\bar L)/L_{max}\): 54.3% with a 32,000 cap and 55.4% with 32,768 (derived; Module 1, Section 6)8 (§4.4.3). Its footnote 41 gives one measured step from a representative run: about 1,000 seconds, of which 125 were training, so generation was the bottleneck (Module 1, Section 5.1). The MFU column is the reminder from inference practice: decode is memory-bound, so 1.01% MFU at 43.21% MBU is the healthy end of this table, not a failure (Module 1, Section 5.3). The generator-to-trainer split is tuned too: ScaleRL runs 64 generators to 16 trainers (4 to 1)1, and Olmo 3's Table 49 gives 3.5 to 1 for 7B Think, 2.5 to 1 for 32B Think, 7 to 1 for 7B Instruct and 8 to 1 for 7B RL-Zero (derived ratios)8. Kimi K3 attacks the same idle time differently: generation pauses once a fraction \(\lambda\) of trajectories has finished, unfinished ones resume next iteration, and staleness is left to a per-token regularization in the loss9 (§4.1.2; Module 12, Section 5.1).
6. The 100,000 GPU-hour run and the scaling axes
6.1 What the validation extrapolates
The headline test fits the sigmoid on one run and checks it against the same run extended. For the 8B dense model the fit uses data up to 50,000 GPU-hours and the run continues to 100,000; for the 17B×16 Llama-4 Scout MoE the fit uses up to 16,000 and the run continues to 50,000, with the extrapolation checked to 45,000. The runs took 7,400 and 7,100 steps, which the authors put at 3.5 times ProRL's length1 (Fig. 1 caption). The caption says the extrapolated curve "closely follows extended training".
How much does that test extrapolate? The 100,000 GPU-hour run is the batch-2048 configuration (§5), and its fit in Appendix A.7 (B 1.70, A 0.645) matches the batch-2048 row of Table 7 (\(C_{mid}=10{,}909\))1. Evaluating \(g\) there gives 0.930 at 50,000 GPU-hours and 0.977 at 100,000: the check spans the last 4.7 points of the gain, about 1.2 pass-rate points with the assumed \(R_0=0.40\) (0.9 to 1.4 for \(R_0\) from 0.45 to 0.35), a test of the approach to the ceiling (derived). For Scout it runs from \(g=0.899\) at 16,000 to 0.980 at 45,000 (derived). The test rules out a curve that keeps climbing, as the power-law fit of Section 2.3 predicted; it cannot distinguish ceilings 0.01 apart, and the authors' own fit of \(A\) moves from 0.645 to 0.655 when the window changes (Table 1).
Units are the weak point of the headline. "7,400 steps" cannot be turned into a sample count: the batch-size sweep of §5 is headed "prompts" while the base setup's 768 is completions (48 times 16), and the completions per step of the 100,000 GPU-hour run are not given1. Cost per step can be derived: 100,000 GPU-hours over 7,400 steps is 13.5 GPU-hours per step, against about 1.5 in the base setup (about 1,000 steps per 1.5k GPU-hours, App. A.6), a factor of 9. A step is not a unit of work, even inside one paper.
6.2 Which axes raise the ceiling, and what each costs
| Run | What changed | Cmid (GPU-h) | B | A | GPU-hours to 90% of gain |
|---|---|---|---|---|---|
| ScaleRL (base) | 8B dense; 14k tokens; batch 768; 16 generations | 2,542 | 1.92 | 0.610 | 7,983 |
| ScaleRL-32k | generation budget 14k to 32k | 11,272 | 1.89 | 0.645 | 36,049 |
| ScaleRL-bs512 | batch 512 | 2,818 | 1.77 | 0.605 | 9,751 |
| ScaleRL-bs2048 | batch 2048 (the 100k-GPU-hour run) | 10,909 | 1.70 | 0.645 | 39,729 |
| ScaleRL-8gen | 8 generations per prompt, same total batch | 2,542 | 2.44 | 0.585 | 6,255 |
| ScaleRL-24gen | 24 generations per prompt | 3,054 | 2.22 | 0.590 | 8,217 |
| ScaleRL-32gen | 32 generations per prompt | 2,936 | 2.07 | 0.595 | 8,487 |
| ScaleRL-Scout | 17B×16 MoE | 4,242 | 1.65 | 0.710 | 16,066 |
| Math+code, math curve | joint math and code run | 2,896 | 2.05 | 0.595 | 8,458 |
| Math+code, code curve | joint math and code run | 1,675 | 1.09 | 0.615 | 12,574 |
Every axis that raised the ceiling also moved the midpoint right. Context length raised \(A\) by 0.035 and moved \(C_{mid}\) 4.4 times to the right; a 2048 batch raised \(A\) by the same 0.035 and moved it 4.3 times; the MoE raised \(A\) by 0.100 and moved it 1.7 times (derived from Table 7). The authors describe the pattern for batch size as an inverse trend: small batches look better at low compute and are overtaken as compute grows1 (§5, Fig. 10). They call long context a "ceiling-raising knob rather than a mere efficiency trade-off", and it is one only past a break-even. With the assumed \(R_0\), the 14k-token curve and the long-context curve cross at about 28,000 GPU-hours (Figure 1), so a 16,000 GPU-hour budget is better spent on the shorter generation limit and a 50,000 GPU-hour budget on the longer one (derived; the shorter run has flattened at 0.61 by then). Two other reports reached the same direction without fitting curves: Kimi k1.5 scaled the RL context window to 128k and calls context length "a key dimension of the continued scaling of RL with LLMs", and found that a smaller model can match a larger one with longer chains while the larger model is more token-efficient10 (§1, ablations); MiniMax-M1 raised its output limit in steps from 40K to 80K3 (§5).
The remaining axes are smaller. Generations per prompt at fixed total batch (8, 16, 24, 32) gave \(A\) between 0.585 and 0.610, which the authors call "essentially unchanged" and a "second-order choice"1 (§5); the 0.025 spread sits just outside the replication margin and is not monotone in group size. Model scale is the largest effect (\(A\) 0.710 against 0.610): the MoE exceeds the 8B model's performance with one sixth of its RL compute, at truncation rates below 2%1 (§5, App. A.15). The math-plus-code run fits each domain on its own curve, math with \(A=0.595\) and code with \(A=0.615\) but a shallow \(B=1.09\): the code curve has realized 92% of its gain at 16,000 GPU-hours against 97% for math (derived). The authors call the multi-task results preliminary1 (§7).
6.3 Limits of the evidence
Three limits apply to everything above. Each axis is one run, and the paper's error margin covers \(A\) and not \(B\) or \(C_{mid}\). The data are one math set plus one code set, the dense model is a single 8B model (its base is not named in the sections I read) and the MoE one Llama-4 Scout run, so nothing here says the sigmoid fits agentic or multi-turn RL, which the authors list as future work1 (§7). And the fitted ceiling is an extrapolation by construction: \(A\) is searched on a grid that stops at 0.800 and the longest fit window is 50,000 GPU-hours1 (App. A.5). The other scaling study in the source set varies model size at fixed method and finds learning efficiency saturating, with the 32B model ahead of the 72B at equal compute because it takes more steps2 (§3.1); it does not use ScaleRL's recipe, so the two cannot be merged into one law.
7. Reading a report's compute claim
Module 1 (Sections 3 and 4) derives the GPU-hour and sample arithmetic for R1-Zero, MiniMax-M1 and Olmo 3. This section places those budgets against what ScaleRL fitted and adds two normalizations: hardware and steps.
7.1 GPU-hours and percent of pretraining
GPU-hours depend on the GPU. NVIDIA lists a GB200 NVL72 rack at 180 PFLOPS dense FP16/BF16 for 72 GPUs ("dense is one-half sparse"), which is 2.5 PFLOPS per GPU11, and an H100 SXM at 1,979 TFLOPS BF16 "with sparsity", 989.5 TFLOPS dense under the same one-half rule12. One GB200 GPU-hour is then 2.53 times an H100 GPU-hour in peak dense BF16 (derived), so ScaleRL's 100,000 GPU-hour run is about 253,000 H100-hour equivalents. I did not fetch an H800 specification and treat it as H100-class, an assumption. Peak ratios overstate the difference in work done, since rollout decode runs near 1% MFU (Table 6); read them as a bound on comparability.
Percent of pretraining depends on a denominator that is rarely defined. DeepSeek-V3.2 says its RL "allocates a post-training computational budget exceeding 10% of the pre-training cost"13 (§1) without defining the denominator. R1-Zero's 100,000 H800 GPU-hours are 3.75% of the 2,664K H800 GPU-hours of DeepSeek-V3's pre-training (derived)14 (Table 1). If V3.2's denominator is also V3's 2,664K, then 10% is 266,400 H800 GPU-hours, 2.6 times R1-Zero's 101,376 (derived, under that assumption). Olmo 3 shows both ends in one family: Module 1 (Section 4.4) derives about 2.8% of pretraining for the initial 5-day RL run of the 32B Think model and 11.6% after the 21-day continuation8. The 3.75% and the "more than 10%" are two settings of a dial, not a conflict.
The ruler shows where the evidence stops. ScaleRL's fits cover up to 50,000 GPU-hours and its largest validated run is 100,000 GB200 GPU-hours. R1-Zero sits at that scale in nominal GPU-hours; MiniMax-M1 and DeepSeek-V3.2 sit about 2.6 times above it, on different hardware, with no published fit. No source in the set fits a compute curve to a production run.
7.2 Claims stated in steps
The efficiency claims in the literature are mostly in steps, and a step is not a unit of work. Table 8 lists each claim with what its step contains; Module 1 (Table 3 and the common-error box in its Section 3.1) carries the full run-size vocabulary.
| Claim | What a counted step is | Completions per step | Max response (tokens) | Not stated |
|---|---|---|---|---|
| DAPO: 50 AIME-24 points in 50% of R1-Zero-Qwen-32B's steps4 (Fig. 1, §3.2, §4.1) | Gradient update: 512 prompts × 16 = 8,192 completions per rollout step, minibatch 512, so 16 updates per rollout step | 512 (derived) | 20,480 | GPU-hours; dynamic sampling oversamples and discards prompts |
| R1-Zero-Qwen-32B: over 10K steps5 (Supp. B.4.1, F.1) | 32 questions × 16 samples, 16 minibatches per 8,192-output rollout | 512 | 32,768 | GPU-hours for this run (Table 7 covers R1-Zero and R1) |
| CISPO: matches DAPO with 50% of the steps3 (§3.1) | An update inside "16 rounds of off-policy updates per generation batch" | not stated | not stated | Samples per step; GPU-hours; seeds |
| ProRL: over 2k steps, about 16k GPU-hours15 (§1, §3.2) | Batch 256 and minibatch 64, 16 samples per prompt, 4 updates per rollout; read here as prompts, which the text does not say | 1,024 per update; 4,096 per rollout (if prompts) | 8,096; 16k in the last ~200 steps | Whether "steps" are rollouts or updates; derived 8 GPU-hours per step |
| Qwen3-235B-A22B: AIME-24 70.1 to 85.1 in 170 steps16 (§4.2) | Not defined; 3,995 query-verifier pairs in the RL set | not stated | not stated | Batch, rollouts per query, GPU-hours |
| Olmo 3 Think: 750 steps (32B), 1,400 (7B)8 (Table 49) | One minibatch per rollout; 128 × 8 prompts (32B), 64 × 8 (7B) | 1,024 (32B); 512 (7B) | 32,768 | GPU-hours per run (day counts only) |
| ScaleRL: 7,400 steps (8B), 7,100 (Scout)1 (Fig. 1) | One PipelineRL trainer update | 768 in the base setup; not stated for the 100k run | 14,336 generation | Completions per step in the large runs; 13.5 GPU-hours per step (derived) |
| Kimi K3 and K2.5: "scaling RL FLOPs" improves scores917 (K3 Fig. 8; K2.5 Fig. 2) | No steps: the axis is RL FLOPs | — | — | The RL-FLOPs axes carry no tick values, so no FLOP figure can be read |
Convert before comparing. R1-Zero-Qwen-32B's "over 10K steps" at 512 completions each is over 5.12 million completions (derived). DAPO's 50% of that is half the gradient updates at the same 512 completions per update, but the updates are fed by oversampling: DAPO keeps sampling until the batch holds only prompts whose accuracy is neither 0 nor 14 (§3.2), and the extra generation depends on the dead-group fraction, which Module 11 (Section 4.1) derives as \(p^G+(1-p)^G\) and which grows during training. DAPO's only compute statement is that training time was "not significantly affected"4 (§4.2), so "50% of the steps" is not "50% of the compute". Olmo 3's Figure 38 plots its 7B RL-Zero run against DAPO by gradient step and by GPU-hour, with DAPO's GPU-hours "taken from the DAPO reproduction on verl"8; the gap between the curves is about 2.5 times on the step axis and about 5 times on the GPU-hour axis (read off the plot; different base models). ScaleRL's fit sidesteps this by working in GPU-hours, the unit that does not depend on how a lab counts steps.
7.3 Against the alternative use of the same GPU-hours
A compute claim also needs a comparator. Qwen3's report ran both options from the same off-policy-distilled 8B checkpoint (AIME-24 55.0, AIME-25 42.8): further RL reached 67.6 and 55.5 at 17,920 GPU-hours, and on-policy distillation reached 74.4 and 65.5 at 1,800 GPU-hours, 9.96 times fewer (derived); on AIME-24 pass@64, RL stayed at 90.0 and distillation reached 93.316 (Table 21). The passage I read does not say whether the teacher's forward passes are in the 1,800. R1's supplementary comparison is the same direction at larger scale: Qwen2.5-32B trained for over 10K RL steps scored 47.0 on AIME-24 against 72.6 for the R1-distilled 32B model5 (Supp. F.1, Table 16). Module 21 covers on-policy distillation itself; the point here is that the cost of an RL recipe has to be read against a recipe that uses a stronger signal, not only against another RL recipe.
8. What a saturating pass-rate curve does not say
The fitted quantity is a mean pass rate over 16 samples per held-out prompt, an estimate of expected pass@1 on prompts from the training distribution1 (§2.1). Anything that moves probability mass toward answers the model could already reach raises it, so \(A\) is a ceiling on pass@1-type performance and says nothing about pass@k at large \(k\). Yue et al. report the pattern that matters here: RLVR-trained models beat their base models at small \(k\), base models win at large \(k\), and across training pass@256 falls while pass@1 rises (their Fig. 1), with six RLVR algorithms performing similarly and "far from optimal"18. ProRL reports the opposite for a 1.5B model trained over 2,000 steps with KL control, reference resets and diverse tasks, with boundary gains that correlate with how weak the base model was and with training duration15, and a later analysis proposes that both are partly right, as an exploitation stage followed by an exploration stage19. ScaleRL measures no pass@k, so a difference in fitted \(A\) between two recipes could be a difference in sharpening and not in what the model can solve; Module 20 treats that evidence.
The paper also cuts against a popular predictor. Entropy fell in every run it tracked, across batch sizes, tasks, generation lengths and model sizes, and the batch-768 and batch-2048 runs followed nearly identical entropy trajectories per step although the larger-batch run was much stronger downstream. The authors conclude that entropy "may not always offer a predictive insight into the performance, as proposed by some recent works like Cui et al."1 (App. A.12, Fig. 16). Cui et al.'s law \(R=-a\exp(H)+b\) fits performance against entropy along a run and puts the ceiling where entropy reaches zero20. The two statements concern different comparisons, a trajectory within one configuration against the same entropy curve across configurations, and neither paper tests the other (my reading). Module 13 develops the entropy law.
Where experts disagree
Two questions divide the sources. Do the details of an RL recipe move the ceiling or only the speed? And is more RL compute the right use of the next GPU-hours? The first is the Module 11 normalization debate seen through compute curves; the second is the question behind every budget in Figure 5.
A. Details move the speed, not the ceiling
Khatri et al., ScaleRL1 (abstract, App. A.8)
Loss aggregation, normalization, curriculum and the off-policy scheme "primarily modulate compute efficiency without materially shifting the asymptote". Pick choices that raise \(A\) first and tune \(B\) and \(C_{mid}\) second; stable recipes extrapolate, so small runs can rank them.
- A replicated margin (\(\pm0.02\) on \(A\), three identical runs) and eight leave-one-out runs with \(A\) between 0.590 and 0.610.
- Predictions held to 100,000 GPU-hours, and it agrees with Module 11's reading of the aggregation and normalization spreads.
- No margin for \(B\); a 15% spread across identical runs is as large as most reversions' efficiency gaps.
- The leave-one-out starts from a recipe whose stability fixes already absorb the choices that move \(A\), and the same paper's forward tests move \(A\) by up to 0.09.
B. Details decide whether the run survives and where it flattens
DAPO4 (Table 1); MiniMax-M13 (§1, §3.2); ScaleRL's forward tests1 (Figs. 5, 6, App. A.11)
DAPO's four techniques lift AIME-24 from 30 to 50 on one base; MiniMax found that a precision mismatch between training and inference kernels "prevents reward growth" until the LM head runs in FP32; ScaleRL's own forward tests move \(A\) from 0.520 to 0.595 (loss type) and 0.610 (FP32 head), and its Appendix A.11 says the off-policy choice shifts the asymptote.
- Gaps of 0.07 to 0.09 in \(A\) are 3.5 to 4.5 times the replication margin.
- The GRPO recipe became unstable after about 6k GPU-hours and the forward-test baseline beyond about 3.5k, so survival is itself an outcome.
- DAPO's ladder is cumulative, one run per rung and one benchmark (Module 11, Section 7.1).
- The forward \(A\) values extrapolate from runs that were breaking (Section 3.4), and the effects shrink inside a full recipe (Table 3's FP32 row).
C. Spend more RL compute: it still pays
DeepSeek-V3.213 (§1, §4.1); Kimi K39 (§4.1.2); ProRL15; ScaleRL's scaling axes1
V3.2 spends more than 10% of its pre-training cost on RL, reports "consistent performance improvements correlating with extended RL training budget" and hypothesizes more would help. K3 reports that scaling RL FLOPs improves knowledge, reasoning, vision, agent and coding scores, and ProRL's gains continued past 2,000 steps.
- Production budgets keep rising, and ScaleRL identifies three axes (context, batch, model) that lift \(A\).
- The fitted curve gives a way to forecast a budget's return.
- No published fit at production scale, and K3's and K2.5's FLOP axes have no ticks (Section 7.2).
- The base recipe has realized 97% of its gain by 16,000 GPU-hours (Section 2.2); later gains come from changing axes, each of which moves \(C_{mid}\) right 1.7 to 4.4 times (Table 7).
D. For small models, spend it on distillation
DeepSeek-R15 (Supp. F.1); Qwen316 (Table 21)
R1 found that 10K+ steps of RL on Qwen2.5-32B only matched QwQ-32B-Preview while distilling R1 was "significantly better"; Qwen3-8B reached higher scores with on-policy distillation at about a tenth of the GPU-hours of RL.
- 9.96 times fewer GPU-hours for higher AIME scores in the one measured pair (derived), and a higher pass@64.
- Cheap enough to repeat for every model size.
- The teacher is itself an RL product, so compute is moved, not removed; R1 says going beyond "the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning".
- One pair, one model size, and no statement in the passage read of whether teacher compute is counted.
A and B describe different regimes, and both are right where they are measured. Inside a recipe that already carries the stability fixes (an asynchronous scheme with bounded lag, a weight-clipped or sequence-level loss, an FP32 head, zero-variance filtering), the evidence supports A about the ceiling: eight reversions land within 0.02 of each other. B is right about the failure side, where the fitted \(A\) is the point at which an unstable run bent. Neither licenses "efficiency does not matter": the reversions cost 14% to 46% more GPU-hours to reach the same pass rate (derived, with an assumed \(R_0\)), a tax on every ablation and sweep, and Olmo 3 reports running each post-training stage several times to sweep hyperparameters8; it does not change the ceiling. The ranking among the efficiency choices is the weakest part, because \(B\) has no margin. What would settle it: three seeds per leave-one-out variant, the same table on a second model family and a non-math domain, and a fit on a production run's held-out curve.
C and D answer different questions. If a stronger teacher exists and the target is a smaller model, D has the only compute-matched evidence in the source set. If you are building the model that sets the frontier, there is no teacher above you and C is the only option. For that case the fits say to choose by ceiling once the budget exceeds the crossover between candidate recipes (about 28,000 GPU-hours for context length in Figure 1) and by efficiency below it. The compute-matched comparison that would settle C against D, with teacher compute included, has not been published in the sources read.
Source-level conflict. ScaleRL cites RL at 3.75% of pre-training for R1-Zero and V3.2 reports more than 10%113. These are not contradictory: the shares are settings of one dial, differ in denominator and hardware, and Olmo 3's own day counts span 2.8% to 11.6% inside one model family (Section 7.1).
What is changing
As of 2026-10-27, ScaleRL is a single arXiv version (v1, 15 Oct 2025, 28 pages and 20 figures) with no venue listed on its arXiv page, and none of the sources read reproduces it1. Tan et al. is at v4 (17 Apr 2026) and states acceptance at the ACL 2026 main conference2; the two scaling frameworks have not been compared on common data. The authors list the next uses of their framework as multi-turn and agentic RL, dense rewards and compute-heavy generative verifiers1 (§7), settings that Kimi K3 and DeepSeek-V3.2 both train on at scale913, so a sigmoid fit to an agentic run would be the most informative missing result. Two production reports plot RL FLOPs without axis values917 and one states its budget as a share of pre-training13; either form leaves the comparison in Section 7 unresolved until a report prints GPU-hours, steps and completions together. The GSPO instability ScaleRL reports (Section 4) rests on one lab's framework and is the finding most likely to be confirmed or overturned by a second replication.
References
- paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R., Duvvuri, S. S., Zaheer, M., Dhillon, I. S., Brandfonbrener, D., Agarwal, R. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-27. Used for: Eq. 1 and its parameters (§1-§2.1, Fig. 3), fitting procedure and margin (App. A.4-A.8), forward ablations (§3, Figs. 4-6, 14, 19), leave-one-out (§4, Fig. 7, Fig. 15), scaling axes (§5, Figs. 9-11, Table 1), truncation analysis (App. A.15), compared recipes (Fig. 2, App. A.16), GSPO instability (App. A.17.4).
- paperTan, Z., Geng, H., Yu, X., Zhang, M. et al. Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning. arXiv:2509.25300v4 (v1 29 Sep 2025, v4 17 Apr 2026; ACL 2026 Main per arXiv comments). https://arxiv.org/abs/2509.25300. Accessed 2026-10-27. Used for: the power-law form with saturating k(N), 63 Qwen2.5 models, test-loss definition (Eqs. 1-2, 6; §2-§3.1).
- paperMiniMax (Chen, A., Li, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-27. Used for: 3 weeks on 512 H800 and $534,700 (abstract, §1, §3), CISPO versus DAPO "50% of the steps" and 16 off-policy rounds (§3.1), precision mismatch (§3.2), staged 40K to 80K length extension (§5).
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-27. Used for: 50 AIME points in 50% of steps (Fig. 1), ablation ladder (Table 1), dynamic sampling (§3.2), batch and minibatch sizes and 20,480 maximum length (§4.1), training-time statement (§4.2).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (Nature version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-27. Used for: R1-Zero GPU-hours, 64×8 H800 for about 198 hours (Supp. B.4.4, Table 7); R1-Zero-Qwen-32B settings (Supp. B.4.1); distillation versus RL on Qwen2.5-32B (Supp. F.1, Table 16); GRPO recipe the ScaleRL authors re-implemented (via ref 1).
- officialHugging Face. TRL documentation: GRPO Trainer, main branch. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the
scale_rewardsoptions and the description of "batch" as group mean with batch-level standard deviation (first 100,000 characters read). - blogPiche, A., Pardinas, R., Kamalloo, E., Bahdanau, D. (ServiceNow). PipelineRL. Hugging Face blog, 25 Apr 2025. https://huggingface.co/blog/ServiceNow/pipelinerl. Accessed 2026-10-27. Used for: in-flight weight updates, pause only to receive weights, stale KV cache, lag counted in optimizer steps, no speedup reported (sections "Conventional RL vs PipelineRL", the stability discussion, and "What's next").
- paperTeam Olmo (Ettinger, A., Bertsch, A. et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2 (14 Apr 2026; v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-27. Used for: infrastructure ablation (Table 23, p. 51), static-batching waste and long-tail description (§4.4.3), day counts and RL allocation (§2.4, Table 49, p. 109), Fig. 38 (p. 107).
- paperKimi Team (Moonshot AI). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-27. Used for: partial rollouts and staleness tolerance (§4.1.2), scaling RL FLOPs claim and Fig. 8 (p. 12-13).
- paperKimi Team (Moonshot AI). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599v4. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-27. Used for: RL context window scaled to 128k and the context-length observation (§1), model-size versus context ablation (ablation studies).
- officialNVIDIA. NVIDIA GB200 NVL72 product page, specifications table. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-10-27. Used for: 360 PFLOPS sparse FP16/BF16 per 72-GPU rack and the "dense is one-half sparse" note. The page may change.
- officialNVIDIA. NVIDIA H100 Tensor Core GPU product page, specifications table. https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-10-27. Used for: H100 SXM BF16 Tensor Core 1,979 TFLOPS "with sparsity".
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-27. Used for: post-training budget "exceeding 10% of the pre-training cost" (§1, §4.1), the hypothesis that more budget helps (§4.1).
- paperDeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437. https://arxiv.org/abs/2412.19437. Accessed 2026-10-27. Used for: pre-training 2,664K H800 GPU-hours and total 2,788K (Table 1, §1).
- paperLiu, M., Diao, S., Lu, X. et al. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-27. Used for: over 2k steps (§1), 16k GPU-hours on 4 nodes of 8 H100, batch and minibatch sizes and context window (§3.2), boundary-expansion claims (abstract).
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Alibaba Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-27. Used for: 170 RL steps and 3,995 query-verifier pairs (§4.2), RL versus on-policy distillation on Qwen3-8B with GPU-hours (Table 21, p. 21).
- paperKimi Team (Moonshot AI). Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276v2. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-27. Used for: vision-RL curves plotted against RL FLOPs (Fig. 2).
- paperYue, Y., Chen, Z., Lu, R., Zhao, A. et al. (Tsinghua LeapLab / SJTU). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837v5. https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-27. Used for: pass@k crossover between RLVR and base models, pass@256 falling as pass@1 rises (abstract, Fig. 1).
- paperYao, X., Yu, L., Hu, X., Teng, F. et al. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028, 5 Oct 2025. https://arxiv.org/abs/2510.04028. Accessed 2026-10-27. Used for: the exploitation-then-exploration reconciliation (abstract only).
- paperCui, G., Zhang, Y., Chen, J. et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617v1. https://arxiv.org/pdf/2505.22617v1. Accessed 2026-10-27. Used for: the entropy-performance law R = -a exp(H) + b (abstract).