Reinforcement Learning for LLM Post-Training · Module 15

RL Compute Scaling: Sigmoidal Fits, the ScaleRL Recipe, and What Longer Runs Actually Buy

ScaleRL's study of more than 400,000 GPU-hours fits its RL runs with a saturating sigmoid, and the fit separates what raises a recipe's ceiling from what only gets it there sooner. This module derives the fit and its limits, audits each ScaleRL design choice against the paper's own numbers, shows how little of the curve the 100,000 GPU-hour validation actually extrapolates, and gives a procedure for normalizing the compute claims in R1, DAPO, MiniMax-M1, Olmo 3, ProRL and DeepSeek-V3.2. You can fit a saturating curve to a run's early data and say whether a change moved the ceiling or only the speed.

Course 45 min read 20 sources As of 2026-10-27 Prerequisites: Module 1, Module 11, Module 13
Three sigmoid compute-performance curves with asymptote A, steepness B and midpoint C_mid labelled; the curve with the higher asymptote starts slower and crosses the other near 28,000 GPU-hours, and a third curve has the same asymptote as the first but a later midpoint 0.45 0.50 0.55 0.60 0.65 0.40 1k 2k 5k 10k 20k 50k 100k RL compute, GPU-hours (log scale) validation pass rate A = 0.645 A = 0.610 C_mid 2,542 C_mid 11,272 curves cross, about 28k 14k tokens: B 1.92, A 0.610 32k tokens: B 1.89, A 0.645 PPO schedule: B 1.89, A 0.605 half of the gain (C_mid)
Figure 1. Equation 1 of ScaleRL drawn for two of its runs, using the (Cmid, B, A) printed in the paper's Table 1: the base run with a 14k-token generation budget (Cmid 2,542, B 1.92, A 0.610) and the run with a longer budget, listed as 32k (Cmid 11,272, B 1.89, A 0.645)1. The pass rate at zero compute, R0, is not printed; I use R0 = 0.40, near where the 8B curve starts in the paper's Figure 1, so heights are illustrative while the midpoints, shapes and the crossover follow from the printed parameters (derived). The solid curves cross at about 28k GPU-hours (24k to 31.5k for R0 between 0.45 and 0.35, derived): the run with the higher ceiling stays behind until then. The dashed curve is the same full recipe with the PPO-off-policy-8 schedule (Cmid 3,409, B 1.89, A 0.605; Fig. 7): the same ceiling within the 0.02 margin, reached with 34% more compute to the midpoint.

1. What the study measured

ScaleRL (Khatri, Madaan et al.; Meta, UT Austin, UCL, Berkeley, Harvard, Periodic Labs) is the only study in this course's source set that treats RL compute as a quantity to fit rather than a budget to spend. It reports more than 400,000 GPU-hours of experiments on NVIDIA GB200 GPUs, almost all on one 8B dense model trained on a verifiable math task, and uses them to separate recipes by where their curves flatten and how quickly they get there1 (abstract, §1). Everything in Sections 2 to 6 rests on that one paper, so the setup matters.

Model
8B dense (main); Llama-4 Scout 17B×16 MoE (scale check)1
Start point
SFT checkpoint: about 4 epochs, 32B tokens, 32 H100 nodes1
Data and reward
Polaris-53K math; ±1 from Sympy or Math-Verify checkers1
Batch
768 completions = 48 prompts × 161
Sequence budget
16,384 tokens = 12,288 thinking + 2,048 solution + 2,048 prompt1
Optimizer
AdamW, constant 5×10−7, ε = 10−15, 100 warmup steps1
GPUs per run
80 GB200: 64 generators, 16 trainers1
Budget tiers
3.5-4K GPU-hours per design-choice run; 16K per leave-one-out run; 30K-100K for large runs1

What is fitted is not a downstream benchmark. 1,000 prompts are held out from Polaris-53K, and every 100 training steps the model samples 16 generations per held-out prompt; the mean pass rate is the y value1 (§2.1, App. A.5). The authors follow pre-training practice in fitting in-distribution held-out performance and report AIME-24 curves only as a downstream check1 (§2.1, §7). Fits start after about 1.5k GPU-hours, roughly one epoch, because the early near-linear phase and transient instabilities do not follow the sigmoid1 (App. A.6, A.7).

1.1 Three claims, and the one this module audits

The abstract states three findings: recipes differ in their asymptotic performance; "loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote"; and stable recipes follow predictable trajectories that extrapolate from small runs1. Module 11 (Section 7.2) audited the aggregation and normalization part and found the spreads inside the paper's own run-to-run margin. Section 3 here checks the rest, in particular the off-policy-algorithm clause, against the paper's forward and leave-one-out numbers, and finds that the paper itself is not consistent about it.

The authors motivate the work with the growth of RL budgets: R1-Zero used 100,000 H800 GPU-hours, which they put at 3.75% of its pre-training compute, and they cite more than a 10× increase in RL compute from o1 to o3 and a similar leap from Grok-3 to Grok-41 (§1). The R1-Zero figure is checkable and Section 7 does so; the o1-to-o3 and Grok figures are not primary data in this course.

2. The sigmoid and what each parameter buys

2.1 The equation

ScaleRL's Equation 1 models the held-out pass rate \(R_C\) after \(C\) GPU-hours as

$$R_C - R_0 \;=\; (A - R_0)\,\underbrace{\frac{1}{1+(C_{mid}/C)^{B}}}_{g(C)}$$

with \(0 \le A \le 1\) the asymptotic pass rate, \(B>0\) a scaling exponent that sets the steepness, and \(C_{mid}\) the compute at which half of the total gain \(A-R_0\) has been realized1 (Eq. 1, Fig. 3). I write \(g(C)\) for the factor, the fraction of the asymptotic gain realized at compute \(C\). The text I read does not say how \(R_0\) is obtained. The authors group the parameters by role: \(B\) and \(C_{mid}\) "primarily affect the efficiency of the run" and \(A\) is the ceiling1 (§2.1, App. A.8). Their advice follows from that split: raise \(A\) first and tune efficiency second, because a less efficient design can still end higher (Fig. 13b)1.

2.2 What the curve implies, worked through

Three consequences follow from Equation 1 by algebra (derived), and they are what you use when reading any fit. First, the compute needed to realize a fraction \(p\) of the asymptotic gain is found by setting \(g(C)=p\):

$$C_p \;=\; C_{mid}\Big(\frac{p}{1-p}\Big)^{1/B}$$

For the base ScaleRL run (\(C_{mid}=2{,}542\), \(B=1.92\)1, Table 1) the 90% point is \(2{,}542\times 9^{1/1.92}=7{,}983\) GPU-hours. For the longer-context run (\(C_{mid}=11{,}272\), \(B=1.89\)) it is \(11{,}272\times 9^{1/1.89}=36{,}049\) GPU-hours, 4.5 times more. Second, the marginal return of compute, per unit of \(\ln C\), is

$$\frac{dR}{d\ln C} \;=\; (A-R_0)\,B\,g(1-g),$$

which peaks at \(g=\tfrac12\) with value \((A-R_0)B/4\). With the assumed \(R_0=0.40\), the base run gains 7.0 pass-rate points from doubling compute at \(C_{mid}\), 2.5 points at 8,000 GPU-hours (\(g=0.90\)), 0.77 points at 16,000 and 0.09 points at 50,000. Third, for \(C\gg C_{mid}\) the curve becomes \(R_C\approx A-D/C^{B}\) with \(D=(A-R_0)C_{mid}^{B}\), the identity the authors give in their Appendix A.41; each doubling of compute then divides the remaining gap to \(A\) by \(2^{B}\), which is 3.78 for the base run.

Share of the total asymptotic gain bought by each doubling of compute for the 14k-token and 32k-token ScaleRL runs 0 10 20 30 40 50 60 % of total gain (A minus R0) 26.6 2.2 below 1.5k 31.2 5.4 1.5-3k 26.0 15.7 3-6k 11.3 29.7 6-12k 3.5 27.7 12-24k 1.0 13.3 24-48k 0.3 4.4 48-96k compute interval, GPU-hours (each interval is one doubling) 14k-token run 32k-token run
Figure 2. Share of each run's total asymptotic gain realized in each doubling of compute, for the 14k-token and 32k-token runs. Bars are \(g(2C)-g(C)\) from the printed (Cmid, B) of the paper's Table 1; the first group is \(g(1500)\), everything realized before the fits begin. No R0 is needed, so the bars are exact functions of the printed parameters (derived)1.

Figure 2 makes the cost of the higher ceiling concrete. The base run has realized 26.6% of its gain before 1.5k GPU-hours, takes 57% in the next two doublings, and gains under 5% after 12k. The long-context run has realized 2.2% by 1.5k and takes 57% between 6k and 24k. A leave-one-out experiment at 16,000 GPU-hours (Section 3) therefore examines a base-recipe curve that has already realized \(g(16{,}000)=0.972\) of its gain: such experiments compare approaches to the ceiling more than rises toward it.

2.3 Why a sigmoid and not a power law

Pre-training curves are usually fitted with \(R_C=A-D/C^{B}\) beyond a threshold compute. ScaleRL's Appendix A.4 gives three reasons to prefer a sigmoid for RL. Accuracy is bounded. A power law is unbounded at low compute and must be fitted past a threshold, which throws away points when a run has only about 75 evaluations to fit. And empirically it extrapolates badly: on the 100,000 GPU-hour 8B run, a power law fitted on 1.5k to 50k GPU-hours predicts \(A=1.0\) against an observed plateau near 0.65, a fit on 5k to 50k predicts 0.74, and the sigmoid gives 0.645 in both windows; the power law recovers the right asymptote only when fitted on 30k to 60k, the regime one is trying to predict1 (App. A.4). The two forms coincide at high compute, as the identity above shows, so the sigmoid's advantage is the mid-range where most runs live.

A second scaling study uses a power law and no ceiling. Tan et al. trained 63 models from the Qwen2.5 family (0.5B to 72B) with GRPO on more than 50k math problems, define test loss as \(L=1-R/R_{max}\), and fit \(\log L(N,X)=-k(N)\log X+E(N)\) for compute or data \(X\), with a learning-efficiency term \(k(N)\) that saturates at a limit \(K_{max}\) as model size grows2 (Eqs. 1-2, §3.1). Loss falling as a pure power of compute implies the pass rate tends to 1, the behavior ScaleRL's Appendix A.4 reports failing on its data. The papers fit different quantities over different ranges and neither tests the other; the forms agree only while \(1-R\gg 1-A\) (derived reading).

2.4 The fitting procedure, its margin, and code

The paper fits by grid search: \(A\in\{0.450,0.455,\dots,0.800\}\) and 100 linearly spaced values of \(C_{mid}\) in \([100, 40{,}000]\); for each pair, \(B\) is fitted with SciPy's curve_fit, and the pair with the lowest sum of squared residuals wins1 (App. A.5). The authors release a fitting repository; its URL is printed in the paper. The function below is my transcription of that procedure; it takes \(R_0\) as an input because the paper does not say how it sets it.

import numpy as np
from scipy.optimize import curve_fit

def sigmoid(C, B, A, Cmid, R0):
    return R0 + (A - R0) / (1.0 + (Cmid / C) ** B)

def fit_scaling_curve(C, R, R0, c_start=1500.0):
    """Grid over (A, Cmid) as in ScaleRL App. A.5; B by least squares.
    C: GPU-hours at each evaluation; R: held-out pass rate (mean@16)."""
    keep = C >= c_start               # fits skip the first ~1.5k GPU-hours
    C, R = C[keep], R[keep]
    best = None
    for A in np.arange(0.450, 0.8001, 0.005):
        for Cmid in np.linspace(100, 40000, 100):
            f = lambda c, B: sigmoid(c, B, A, Cmid, R0)
            try:
                (B,), _ = curve_fit(f, C, R, p0=[2.0], bounds=(0.1, 20.0))
            except RuntimeError:
                continue
            sse = float(np.sum((f(C, B) - R) ** 2))
            if best is None or sse < best[0]:
                best = (sse, float(A), float(B), float(Cmid))
    return dict(sse=best[0], A=best[1], B=best[2], Cmid=best[3])

The margin comes from replication. Three independent ScaleRL runs with the same settings gave fitted \(A\) varying by at most \(\pm0.015\), so the authors adopt 0.02 as the error margin on \(A\); their printed fits are (B, A) = (1.92, 0.610), (1.97, 0.610) and (2.20, 0.595)1 (App. A.5, Fig. 8a). They do not give a margin for \(B\), because algorithms with different \(A\) have different \(B\) errors. Their rule is that if two methods have \(A\) within 0.02, the one with the higher \(B\) after refitting with a common \(A\) is at least as efficient1 (App. A.5). One consequence: across three identical runs \(B\) spans 1.92 to 2.20, 15% of its smallest value (derived), so efficiency differences of that size cannot be separated from replication noise.

Table 1. Sensitivity of the fit to its window, 100,000 GPU-hour run on the 8B dense model1 (App. A.7). The asymptote moves by 0.010; the exponent by 0.14.
Fitting window (GPU-hours)BA
1.5k to 50k (the paper's default)1.700.645
5k to 50k1.670.645
0 to 50k1.70.645
0 to 100k1.560.655

3. Which choices move the asymptote

Two things are tangled in every recipe comparison: where the curve flattens (\(A\)) and how fast it gets there (\(B\), \(C_{mid}\)). Figure 3 collects every fitted \(A\) the paper prints for a before-and-after pair, with the 0.02 replication margin as a scale bar. The abstract's claim about "details" survives in the leave-one-out block and for aggregation and normalization, and fails for loss type, precision and the lag bound.

Fitted asymptote A before and after each design change in ScaleRL, with the plus or minus 0.02 run-to-run margin shown as a scale bar 0.50 0.55 0.60 0.65 0.70 Loss type: DAPO to CISPO LM-head precision: BF16 to FP32 Zero-variance filtering: off to on Prompt curriculum: uniform to NPR Loss aggregation: sample to prompt Advantage norm: prompt to batch Async: PPO-off-pol-8 to PipelineRL-8 equal PipelineRL max lag k: 8 to 12 Generation length: 14k to 32k Batch: 768 to 2048 Generations per prompt: 8 to 32 Model: 8B dense to 17Bx16 MoE Leave one out, eight reversions Recipe: DeepSeek GRPO to ScaleRL scale bar: ±0.02 run-to-run (three identical ScaleRL runs) before after (blue: within 0.02; orange: beyond) top block: forward tests; then scaling axes; then leave one out; then whole recipe
Figure 3. Fitted asymptote \(A\) before (open circle) and after (filled) each change in ScaleRL, 8B dense model unless the row says otherwise. Blue segments are within the 0.02 run-to-run margin, orange segments exceed it. Forward tests start from a baseline with \(A=0.520\); scaling axes and leave-one-out start from the full recipe (\(A=0.610\)); the leave-one-out row spans the eight reversions of Table 3. One run per point. Values transcribed from the paper's Figs. 2, 4, 5, 6, 7, 14 and Table 11; the whole-recipe row compares the authors' re-implementation of the R1 recipe with ScaleRL (Table 2).

3.1 Recipes differ in ceiling

The authors re-implemented four published recipes inside their own framework and fitted them next to ScaleRL (Figure 2 and Appendix A.16 of the paper). The ceilings differ by 0.12, six times the replication margin.

Table 2. Prevalent recipes as re-run by the ScaleRL authors on the 8B dense model and Polaris-53K, with fitted B and A1 (Fig. 2, App. A.16). These are re-implementations in one codebase, not the original labs' systems.
RecipeLossOff-policy schemeBatch (completions)BA
ScaleRLCISPO, prompt averagePipelineRL-87681.970.610
MiniMax-M13CISPO, FP32 LM head, prompt averagePPO-off-policy1,280 (80 × 16), zero-variance prompts dropped1.770.610
MagistralDAPO-stylePipelineRLnot stated in the passage read1.440.535
Qwen2.5 recipe (DAPO)4DAPO loss, εmax 0.26, prompt averagePPO-off-policy-81,280, zero-variance prompts dropped1.850.515
DeepSeek recipe (GRPO)5GRPO, ε 0.2, sample averagePPO-off-policy-8not stated; base setup is 7681.170.490

Two details qualify the table. DAPO and MiniMax were given a larger batch because the authors' codebase could not refill a batch as DAPO's dynamic sampling does, which if anything advantaged DAPO1 (App. A.16). The GRPO run became unstable after about 6k GPU-hours as truncations rose, and extended runs "align closely with the extrapolated curves for stable recipes like ScaleRL and MiniMax"1 (Fig. 2 caption). Whether the original labs' systems show the same ceilings is untested here.

3.2 Forward tests: large moves, from a weak baseline

The forward ablations start from a GRPO-like baseline without a KL term and with DAPO's asymmetric clip, adopt PipelineRL-8, and fit each variant at 3.5k to 4k GPU-hours1 (§2, §3.2). The legends of the paper's Figures 5a, 5b, 6a and 6b print: DAPO loss \(A=0.520\) (\(B=4.55\)), GSPO 0.590 (1.80), CISPO 0.595 (2.18); BF16 head 0.520 (4.55) against FP32 head 0.610 (1.79); no zero-variance filtering 0.520 (4.55) against filtering 0.540 (2.68); uniform prompt sampling 0.520 (4.55) against No-Positive-Resampling 0.535 (3.85)1. Aggregation and normalization (App. A.9) span only 0.015 and 0.010 in \(A\); Module 11 (Section 7.2) reads them against the margin and Module 12 (Table 2) tabulates the loss-type comparison.

Two features matter for reading these fits. First, \(B\) moves opposite to \(A\): the baseline (\(A=0.520\), \(B=4.55\)) is the control in each pair, every variant with a higher \(A\) has a lower \(B\) (1.79 to 3.85), and the lag-bound variants at or below baseline \(A\) have \(B\) of 4.14 and 4.22. A curve that flattens early at a low ceiling is steep, which is why the paper pins \(A\) before comparing efficiency. Second, the clip constant alone moves the ceiling: sweeping \(\epsilon_{max}\) in the DAPO-style loss gave \(A=0.500, 0.500, 0.510, 0.520, 0.530, 0.480\) for \(\epsilon_{max}=0.20, 0.22, 0.23, 0.25, 0.26, 0.27\), with \(B\) wandering between 1.78 and 7.321 (Fig. 19a). The authors read this as \(\epsilon_{max}\) governing "the asymptotic error itself"; one run per value and a 0.05 range (2.5 times the margin) make it real but coarse, and the spread in \(B\) shows how little a short fit says about efficiency.

3.3 Leave-one-out: ceilings converge, costs do not

Starting from the full recipe and reverting one choice at a time, at 16,000 GPU-hours each, all eight variants have a fitted \(A\) between 0.590 and 0.6101 (Fig. 7). The paper compares efficiency by averaging \(A\) over the runs (0.605), refitting, and comparing the slope \(B\); Module 11 (Figure 7 there) plots those pinned values. The paper never converts them to GPU-hours, so I do: from the printed \(C_{mid}\), \(B\), \(A\) of each variant, the compute to reach a fixed pass rate of 0.57 is \(C_{mid}\,(p/(1-p))^{1/B}\) with \(p=(0.57-R_0)/(A-R_0)\), using the assumed \(R_0=0.40\) and checking \(R_0\in\{0.35,0.45\}\).

Table 3. What each reversion costs in GPU-hours to reach a held-out pass rate of 0.57, 8B dense model, leave-one-out runs1 (Fig. 7 values for Cmid, B, A). The last two columns are derived with an assumed R0; the full recipe needs 5,401 GPU-hours at R0 = 0.40 (6,177 at 0.35; 4,505 at 0.45). No margin is printed for B.
Reversion from the full recipeCmid (GPU-h)BAGPU-hours to 0.57 (R0 0.40)Cost vs full recipeRange over R0 0.35-0.45
PPO-off-policy-8 instead of PipelineRL-83,4091.890.6057,8671.46×1.45 to 1.46
No zero-variance filtering2,5831.840.5957,3211.36×1.34 to 1.36
DAPO loss instead of CISPO3,0931.770.6107,0051.30×1.28 to 1.31
Uniform prompt sampling (no NPR)2,6211.620.6056,9531.29×1.24 to 1.32
Prompt-level advantage normalization2,4632.090.5906,8571.27×1.26 to 1.29
Sample-average loss aggregation2,6211.930.6006,4391.19×1.19
Length penalty instead of interruptions2,6601.680.6106,2941.17×1.14 to 1.19
No FP32 at the LM head2,8571.880.6106,1681.14×1.14 to 1.15

Read with the replication caveat of Section 2.4, the table supports three statements. The ceilings converge once the stability fixes are in place. The off-policy scheme and the loss type are the costliest reversions, matching the paper's list of "the most important decisions": off-policy algorithm, loss function and model precision1 (§7). Precision is the exception inside this recipe: the FP32 head is the cheapest reversion on a dense 8B model, where the paper says the fix "makes little difference" within ScaleRL but gives "large gains in GRPO/DAPO-style losses"1 (§4), and on the 17B×16 MoE its removal lowers the fitted \(A\) from 0.710 to 0.700 (Fig. 8b); Module 14 covers why the head matters. What the table does not support is reading "efficiency only" as unimportant: aggregation, normalization and curriculum, which the abstract files under efficiency, cost 19% to 29% more GPU-hours to a fixed score, though a gap in \(B\) under about 15% cannot be separated from replication (derived).

3.4 Two cautions on the forward numbers

The paper is not consistent about the off-policy algorithm. The abstract lists it among the details that "primarily modulate compute efficiency"1, while Appendix A.11 says the PipelineRL-versus-PPO distinction "affects the asymptotic performance c of the scaling curve, not just the efficiency exponent b. Very few axes shift the asymptote in this way". The numbers favor the abstract for the choice of schedule (\(A=0.520\) for PPO-off-policy-1, PPO-off-policy-8 and PipelineRL-8, Fig. 4a) and A.11 for the lag bound (\(A=0.490\) at \(k=12\) against 0.520 at \(k=4\) and \(k=8\), Fig. 4b)1.

The forward \(A\) values are also extrapolations from runs that were about to fail. The baseline destabilizes beyond about 3,500 GPU-hours as truncations exceed 10% of generations; variants survive to about 5k (\(\epsilon_{max}=0.26\)), 6k (FP32 head) and 7k (CISPO) GPU-hours1 (App. A.6). A fitted \(A=0.520\) is where an unstable recipe's curve bent before it broke, not a measured ceiling. The leave-one-out numbers, from a recipe stable to 100k, are the cleaner evidence about a single choice; the forward numbers measure "survives longer and flattens higher" jointly.

4. The ScaleRL recipe, component by component

ScaleRL achieves predictable scaling "by integrating existing methods, rather than inventing novel methods": an asynchronous PipelineRL setup with a maximum lag of 8, forced length interruptions, the CISPO loss, prompt-level loss averaging, batch-level advantage normalization, FP32 logits, zero-variance filtering and No-Positive-Resampling1 (§1, §4). Its objective, with \(\mathrm{sg}\) the stop-gradient and \(\rho_{i,t}\) the token-level ratio of trainer to generator probability, is

$$\mathcal J_{\text{ScaleRL}}(\theta)=\mathbb E\Bigg[\frac{1}{\sum_{g=1}^{G}|y_g|}\sum_{i=1}^{G}\sum_{t=1}^{|y_i|}\mathrm{sg}\big(\min(\rho_{i,t},\epsilon)\big)\,\hat A_i^{\mathrm{norm}}\,\log\pi^{\theta}_{train}(y_{i,t})\Bigg],\qquad \hat A_i^{\mathrm{norm}}=\hat A_i/\hat A_{std},$$

restricted to prompts with \(0<\mathrm{mean}(r)<1\) and a historical pass rate below 0.91 (§4). Here \(\hat A_i=r_i-\mathrm{mean}(r)\) is the group-centered advantage and \(\hat A_{std}\) the standard deviation of all such advantages in the batch. Module 11 derives the token weights this implies and Module 12 covers CISPO's gradient; Table 4 records what each component is for and what the paper printed for it.

Table 4. ScaleRL components, the paper's stated reason, and the evidence it prints (8B dense, math)1 (§3-§4, App. A.9-A.11, A.17). Forward = from the weak baseline (A 0.520); LOO = reverted from the full recipe (A 0.610); costs from Table 3.
ComponentSettingPaper's reasonEvidenceSee
Off-policy schedulePipelineRL, lag k = 8Streaming generation cuts idle time and stays closer to on-policyForward: B 4.55 vs 2.68, A equal at 0.520. LOO: 1.46× GPU-hoursSection 5; Module 1
Length controlForced interruption: append "Okay, time is up. Let me stop thinking and formulate a final answer now. </think>" at a random position in 10k-12k tokensTruncated generations destabilize trainingLOO length penalty: A 0.610, 1.17×. Swapping in a length penalty "does not improve performance"Module 23
Loss typeCISPO, token level, \(\mathrm{sg}(\min(\rho,\epsilon))\)Clip the importance weight, keep every token's gradient; robust to \(\epsilon_{max}\in\{4,5,8\}\)Forward: A 0.595 vs DAPO 0.520; GSPO 0.590. LOO DAPO: 1.30×Module 12
Loss aggregationPrompt average"Achieves the highest asymptotic performance"Forward A: prompt 0.535, token 0.535, sample 0.520. LOO sample: 1.19×Module 11
Advantage normalizationBatch level: divide group-centered advantages by the std of all advantages in the batch"Theoretically sound and marginally better"; all three variants "similar"Forward A: none 0.525, batch 0.530, prompt 0.520. LOO prompt-level: 1.27×Module 11
PrecisionFP32 at the LM head, generator and trainerGenerator and trainer kernels disagree most at the headForward A 0.520 to 0.610. LOO dense: 1.14×. Scout LOO: A 0.710 vs 0.700Module 14
Zero-variance filteringDrop prompts with all-equal rewards from the loss; no refillZero advantage means zero gradient; shrinks the effective batchForward A 0.520 to 0.540. LOO: 1.36×Module 11
CurriculumNo-Positive-Resampling: drop prompts whose historical pass rate reaches 0.9Once easy, a prompt stays easy and costs compute without signalForward A 0.520 to 0.535. LOO uniform sampling: 1.29×Module 11

Two points about the table. "Batch-level" normalization means different things in two codebases. ScaleRL divides the group-centered advantages by the standard deviation of the advantages in the batch; TRL's scale_rewards="batch" pairs the group mean with a batch-level standard deviation, of the rewards in Module 11's reading of the source6. Both apply one scalar per batch, but they are not the same scalar (Module 11, Section 3.2). And the loss-type row hides a stability result that Module 12 (Section 4.5) develops: the paper's GSPO runs "diverged mid-training" on multiple occasions, restarting from a stable checkpoint recovered the 8B runs, and on the 17B×16 MoE "instability persisted despite repeated resetting to a stable checkpoint", with no implementation bug found1 (App. A.17.4). The GSPO ceiling (0.590) is within 0.005 of CISPO's, so the choice of CISPO rests on stability and on robustness to its clip constant, not on the asymptote.

What the recipe leaves out is also a choice. The base algorithm has no KL term "in line with large-scale training reports", rewards are \(\pm1\), and generations per prompt stay at 161 (§2, App. A.3). Module 13 covers the keep-or-remove-KL disagreement. The one monitoring signal the authors insist on is the truncation rate, because instabilities in their runs "were often linked to truncations"1 (App. A.15).

10-15%truncation at batch 768 destabilized training1
< 5%ScaleRL, 8B dense, for over 90% of training1
about 7%ScaleRL at batch 2048, occasionally1
< 2%ScaleRL on the 17B×16 MoE throughout1

5. PipelineRL against PPO-off-policy: where the efficiency comes from

The one design choice the paper treats as independent of all the others is the asynchronous schedule1 (§3.1). PPO-off-policy-k, the default asynchronous scheme in Qwen3 and ProRL, has the old policy generate for a batch of \(B\) prompts; each gradient update consumes a minibatch of \(\hat B=48\) prompts, so one generation batch feeds \(k=B/\hat B\) updates, and the paper tests \(k\in\{1,8\}\). PipelineRL-k, from Piche et al. and used by Magistral, streams: generators run continuously, whenever the trainers finish an update the new weights are pushed to them at once, and they continue "with the updated weights but a stale KV cache from the old policy"; the trainers wait only if they get \(k\) steps ahead of the generators1 (§3.1).

Schematic of two asynchronous RL schedules: alternating generation and training phases with idle lanes, against streaming generation with in-flight weight updates a. PPO-off-policy-k: alternating phases generatorstrainers batch of k x 48 prompts tail idle next batch tail idle idle 1 2 3 4 idle 1 2 3 4 weights pushed after k updates b. PipelineRL-k: streaming generation, in-flight updates generatorstrainers new weights new weights one sequence spans three weight updates step step step step step gaps between steps: trainers wait only if they are k = 8 steps ahead of the generators generation few long sequences left optimizer step idle lane
Figure 4. Schematic of the two schedules; durations are not to scale. Panel a follows the paper's description of alternating generation and training phases with one weight push per \(k\) updates1 (§3.1, App. A.11); the "few long sequences left" tail follows Olmo 3's account of long-tail generation8 (§4.4.3). Panel b follows the paper and the PipelineRL write-up: weights are updated in the inference servers after each optimizer step without stopping generation, so a sequence in flight continues with a stale KV cache7.

The ServiceNow write-up of PipelineRL adds the mechanism an inference engineer would ask about. Inference is paused only "for just the time needed to receive the new weights", the in-flight sequences "proceed with stale keys and values in the KV cache", and the authors report that this "does not adversely affect stability"7. Lag there is counted in optimizer steps. The write-up reports no speedup figure and says more analysis is needed to measure it7, so ScaleRL's fits are the quantitative evidence in the source set. Whether ScaleRL stores per-token probabilities from whichever weights were live is not stated in the passages I read.

5.1 What the fits say

Table 5. Off-policy schedules fitted on the same baseline, 8B dense model1 (Figs. 4a, 4b, 7). The first five rows are forward tests at 3.5k-4k GPU-hours; the last two are leave-one-out runs inside the full recipe, with the GPU-hours to a pass rate of 0.57 derived as in Table 3.
SetupBACmid (GPU-h)GPU-hours to 0.57
PPO-off-policy-12.150.520not printednot derived
PPO-off-policy-82.680.520not printednot derived
PipelineRL-44.140.520not printednot derived
PipelineRL-84.550.520not printednot derived
PipelineRL-124.220.490not printednot derived
Full recipe (PipelineRL-8)1.920.6102,5425,401
Full recipe with PPO-off-policy-81.890.6053,4097,867

On the weak baseline, PipelineRL-8 has a steepness 1.70 times that of PPO-off-policy-8 (4.55 against 2.68) at the same ceiling1 (Fig. 4a). Inside the full recipe the gap is smaller and clearer in compute units: \(C_{mid}\) is 34% higher (3,409 against 2,542) and the compute to a pass rate of 0.57 is 46% higher (derived). The authors attribute the advantage to reduced idle time and to a "tighter feedback loop" that keeps "training closer to the on-policy regime"1 (§3.1, App. A.11). No experiment separates the two. One feature of Table 5 argues against reading it as a pure staleness effect: within PPO-off-policy, the 8-update setting beats the 1-update setting (2.68 against 2.15). With \(k=1\) the generation batch is just 48 prompts, and small generation batches underuse the generators, the effect the PipelineRL authors describe as "producing small amounts of data with many GPUs is inefficient because this means the per-GPU batch size is small"7. So utilization and staleness trade against each other, and the best \(k\) is where they balance (derived reading).

The systems side is measurable independently. Olmo 3 ablated its RL infrastructure on a 2-hour benchmark, one 8×A100 node for training and one for inference, and reports utilization for inference only because inference is the bottleneck8 (Table 23, p. 51).

Table 6. Olmo 3's RL infrastructure ablation: each row adds one component to the row above8 (Table 23). Ratios derived.
ConfigurationTotal tokens (Mtok)Tokens/sMFU (%)MBU (%)Tokens/s vs previous row
Olmo 2 infrastructure6.348810.3012.90—
+ continuous batching7.029750.3314.291.11×
+ better threading9.771,3580.4619.891.39×
+ inflight updates (Olmo 3)21.232,9491.0143.212.17×

In-flight updates are the largest single step, 2.17 times the token rate, and the full change is 3.35 times (2,949 against 881; derived). The same report puts the idle compute of static batching at "up to 54%" for a 32K cap and a 14,628-token mean, which is \((L_{max}-\bar L)/L_{max}\): 54.3% with a 32,000 cap and 55.4% with 32,768 (derived; Module 1, Section 6)8 (§4.4.3). Its footnote 41 gives one measured step from a representative run: about 1,000 seconds, of which 125 were training, so generation was the bottleneck (Module 1, Section 5.1). The MFU column is the reminder from inference practice: decode is memory-bound, so 1.01% MFU at 43.21% MBU is the healthy end of this table, not a failure (Module 1, Section 5.3). The generator-to-trainer split is tuned too: ScaleRL runs 64 generators to 16 trainers (4 to 1)1, and Olmo 3's Table 49 gives 3.5 to 1 for 7B Think, 2.5 to 1 for 32B Think, 7 to 1 for 7B Instruct and 8 to 1 for 7B RL-Zero (derived ratios)8. Kimi K3 attacks the same idle time differently: generation pauses once a fraction \(\lambda\) of trajectories has finished, unfinished ones resume next iteration, and staleness is left to a per-token regularization in the loss9 (§4.1.2; Module 12, Section 5.1).

6. The 100,000 GPU-hour run and the scaling axes

6.1 What the validation extrapolates

The headline test fits the sigmoid on one run and checks it against the same run extended. For the 8B dense model the fit uses data up to 50,000 GPU-hours and the run continues to 100,000; for the 17B×16 Llama-4 Scout MoE the fit uses up to 16,000 and the run continues to 50,000, with the extrapolation checked to 45,000. The runs took 7,400 and 7,100 steps, which the authors put at 3.5 times ProRL's length1 (Fig. 1 caption). The caption says the extrapolated curve "closely follows extended training".

How much does that test extrapolate? The 100,000 GPU-hour run is the batch-2048 configuration (§5), and its fit in Appendix A.7 (B 1.70, A 0.645) matches the batch-2048 row of Table 7 (\(C_{mid}=10{,}909\))1. Evaluating \(g\) there gives 0.930 at 50,000 GPU-hours and 0.977 at 100,000: the check spans the last 4.7 points of the gain, about 1.2 pass-rate points with the assumed \(R_0=0.40\) (0.9 to 1.4 for \(R_0\) from 0.45 to 0.35), a test of the approach to the ceiling (derived). For Scout it runs from \(g=0.899\) at 16,000 to 0.980 at 45,000 (derived). The test rules out a curve that keeps climbing, as the power-law fit of Section 2.3 predicted; it cannot distinguish ceilings 0.01 apart, and the authors' own fit of \(A\) moves from 0.645 to 0.655 when the window changes (Table 1).

Units are the weak point of the headline. "7,400 steps" cannot be turned into a sample count: the batch-size sweep of §5 is headed "prompts" while the base setup's 768 is completions (48 times 16), and the completions per step of the 100,000 GPU-hour run are not given1. Cost per step can be derived: 100,000 GPU-hours over 7,400 steps is 13.5 GPU-hours per step, against about 1.5 in the base setup (about 1,000 steps per 1.5k GPU-hours, App. A.6), a factor of 9. A step is not a unit of work, even inside one paper.

6.2 Which axes raise the ceiling, and what each costs

Table 7. Fits for the large-scale runs of ScaleRL's Section 51 (Table 1: Cmid, B, A as printed). The last column is derived: \(C_{mid}\,9^{1/B}\), the GPU-hours to realize 90% of each run's own gain. The table lists the long-context run as "32k"; the legend of the paper's Fig. 9a prints "34k" with B 1.86 and A 0.650.
RunWhat changedCmid (GPU-h)BAGPU-hours to 90% of gain
ScaleRL (base)8B dense; 14k tokens; batch 768; 16 generations2,5421.920.6107,983
ScaleRL-32kgeneration budget 14k to 32k11,2721.890.64536,049
ScaleRL-bs512batch 5122,8181.770.6059,751
ScaleRL-bs2048batch 2048 (the 100k-GPU-hour run)10,9091.700.64539,729
ScaleRL-8gen8 generations per prompt, same total batch2,5422.440.5856,255
ScaleRL-24gen24 generations per prompt3,0542.220.5908,217
ScaleRL-32gen32 generations per prompt2,9362.070.5958,487
ScaleRL-Scout17B×16 MoE4,2421.650.71016,066
Math+code, math curvejoint math and code run2,8962.050.5958,458
Math+code, code curvejoint math and code run1,6751.090.61512,574

Every axis that raised the ceiling also moved the midpoint right. Context length raised \(A\) by 0.035 and moved \(C_{mid}\) 4.4 times to the right; a 2048 batch raised \(A\) by the same 0.035 and moved it 4.3 times; the MoE raised \(A\) by 0.100 and moved it 1.7 times (derived from Table 7). The authors describe the pattern for batch size as an inverse trend: small batches look better at low compute and are overtaken as compute grows1 (§5, Fig. 10). They call long context a "ceiling-raising knob rather than a mere efficiency trade-off", and it is one only past a break-even. With the assumed \(R_0\), the 14k-token curve and the long-context curve cross at about 28,000 GPU-hours (Figure 1), so a 16,000 GPU-hour budget is better spent on the shorter generation limit and a 50,000 GPU-hour budget on the longer one (derived; the shorter run has flattened at 0.61 by then). Two other reports reached the same direction without fitting curves: Kimi k1.5 scaled the RL context window to 128k and calls context length "a key dimension of the continued scaling of RL with LLMs", and found that a smaller model can match a larger one with longer chains while the larger model is more token-efficient10 (§1, ablations); MiniMax-M1 raised its output limit in steps from 40K to 80K3 (§5).

The remaining axes are smaller. Generations per prompt at fixed total batch (8, 16, 24, 32) gave \(A\) between 0.585 and 0.610, which the authors call "essentially unchanged" and a "second-order choice"1 (§5); the 0.025 spread sits just outside the replication margin and is not monotone in group size. Model scale is the largest effect (\(A\) 0.710 against 0.610): the MoE exceeds the 8B model's performance with one sixth of its RL compute, at truncation rates below 2%1 (§5, App. A.15). The math-plus-code run fits each domain on its own curve, math with \(A=0.595\) and code with \(A=0.615\) but a shallow \(B=1.09\): the code curve has realized 92% of its gain at 16,000 GPU-hours against 97% for math (derived). The authors call the multi-task results preliminary1 (§7).

6.3 Limits of the evidence

Three limits apply to everything above. Each axis is one run, and the paper's error margin covers \(A\) and not \(B\) or \(C_{mid}\). The data are one math set plus one code set, the dense model is a single 8B model (its base is not named in the sections I read) and the MoE one Llama-4 Scout run, so nothing here says the sigmoid fits agentic or multi-turn RL, which the authors list as future work1 (§7). And the fitted ceiling is an extrapolation by construction: \(A\) is searched on a grid that stops at 0.800 and the longest fit window is 50,000 GPU-hours1 (App. A.5). The other scaling study in the source set varies model size at fixed method and finds learning efficiency saturating, with the 32B model ahead of the 72B at equal compute because it takes more steps2 (§3.1); it does not use ScaleRL's recipe, so the two cannot be merged into one law.

7. Reading a report's compute claim

Module 1 (Sections 3 and 4) derives the GPU-hour and sample arithmetic for R1-Zero, MiniMax-M1 and Olmo 3. This section places those budgets against what ScaleRL fitted and adds two normalizations: hardware and steps.

7.1 GPU-hours and percent of pretraining

GPU-hours depend on the GPU. NVIDIA lists a GB200 NVL72 rack at 180 PFLOPS dense FP16/BF16 for 72 GPUs ("dense is one-half sparse"), which is 2.5 PFLOPS per GPU11, and an H100 SXM at 1,979 TFLOPS BF16 "with sparsity", 989.5 TFLOPS dense under the same one-half rule12. One GB200 GPU-hour is then 2.53 times an H100 GPU-hour in peak dense BF16 (derived), so ScaleRL's 100,000 GPU-hour run is about 253,000 H100-hour equivalents. I did not fetch an H800 specification and treat it as H100-class, an assumption. Peak ratios overstate the difference in work done, since rollout decode runs near 1% MFU (Table 6); read them as a bound on comparability.

Percent of pretraining depends on a denominator that is rarely defined. DeepSeek-V3.2 says its RL "allocates a post-training computational budget exceeding 10% of the pre-training cost"13 (§1) without defining the denominator. R1-Zero's 100,000 H800 GPU-hours are 3.75% of the 2,664K H800 GPU-hours of DeepSeek-V3's pre-training (derived)14 (Table 1). If V3.2's denominator is also V3's 2,664K, then 10% is 266,400 H800 GPU-hours, 2.6 times R1-Zero's 101,376 (derived, under that assumption). Olmo 3 shows both ends in one family: Module 1 (Section 4.4) derives about 2.8% of pretraining for the initial 5-day RL run of the 32B Think model and 11.6% after the 21-day continuation8. The 3.75% and the "more than 10%" are two settings of a dial, not a conflict.

Reported and derived RL compute budgets on a log GPU-hour axis, with the range ScaleRL fitted shaded 3k 10k 30k 100k 300k 1M ScaleRL design-choice runs ScaleRL leave-one-out runs ProRL, 1.5B model Qwen3-8B RL stage ScaleRL 17Bx16 MoE run ScaleRL 8B dense run DeepSeek-R1-Zero Olmo 3.1 Think 32B extension MiniMax-M1 DeepSeek-V3.2 (lower bound) GPU-hours (log scale); shaded: range ScaleRL fitted (up to 50k) GB200 H100 H800 hardware unstated dashed red: largest validated run (100k)
Figure 5. RL compute on a log GPU-hour axis. ScaleRL's tiers and its two large runs are as printed1; ProRL used about 16k GPU-hours on 4 nodes of 8 H10015 (§3.2); the Qwen3-8B RL stage used 17,920 GPU-hours16 (Table 21, hardware not stated). R1-Zero is 64×8 H800 for about 198 hours, 101,376 GPU-hours5 (Supp. B.4.4). Olmo 3.1 Think 32B is the 21-day continuation on 224 H100 GPUs, 112,896 GPU-hours8 (§2.4, Table 49; derived). MiniMax-M1 is 512 H800 for three weeks, 258,048 GPU-hours3 (derived from the report's figures; $534,700 stated). DeepSeek-V3.2 is plotted at the lower bound that "exceeding 10%" implies if the denominator is V3's 2,664K H800 GPU-hours1314 (assumption). Units are not equal across hardware (text above).

The ruler shows where the evidence stops. ScaleRL's fits cover up to 50,000 GPU-hours and its largest validated run is 100,000 GB200 GPU-hours. R1-Zero sits at that scale in nominal GPU-hours; MiniMax-M1 and DeepSeek-V3.2 sit about 2.6 times above it, on different hardware, with no published fit. No source in the set fits a compute curve to a production run.

7.2 Claims stated in steps

The efficiency claims in the literature are mostly in steps, and a step is not a unit of work. Table 8 lists each claim with what its step contains; Module 1 (Table 3 and the common-error box in its Section 3.1) carries the full run-size vocabulary.

Table 8. Compute claims stated in steps, and what each step is. "Completions per step" is the number of sampled responses consumed by one counted step. Derived entries marked.
ClaimWhat a counted step isCompletions per stepMax response (tokens)Not stated
DAPO: 50 AIME-24 points in 50% of R1-Zero-Qwen-32B's steps4 (Fig. 1, §3.2, §4.1)Gradient update: 512 prompts × 16 = 8,192 completions per rollout step, minibatch 512, so 16 updates per rollout step512 (derived)20,480GPU-hours; dynamic sampling oversamples and discards prompts
R1-Zero-Qwen-32B: over 10K steps5 (Supp. B.4.1, F.1)32 questions × 16 samples, 16 minibatches per 8,192-output rollout51232,768GPU-hours for this run (Table 7 covers R1-Zero and R1)
CISPO: matches DAPO with 50% of the steps3 (§3.1)An update inside "16 rounds of off-policy updates per generation batch"not statednot statedSamples per step; GPU-hours; seeds
ProRL: over 2k steps, about 16k GPU-hours15 (§1, §3.2)Batch 256 and minibatch 64, 16 samples per prompt, 4 updates per rollout; read here as prompts, which the text does not say1,024 per update; 4,096 per rollout (if prompts)8,096; 16k in the last ~200 stepsWhether "steps" are rollouts or updates; derived 8 GPU-hours per step
Qwen3-235B-A22B: AIME-24 70.1 to 85.1 in 170 steps16 (§4.2)Not defined; 3,995 query-verifier pairs in the RL setnot statednot statedBatch, rollouts per query, GPU-hours
Olmo 3 Think: 750 steps (32B), 1,400 (7B)8 (Table 49)One minibatch per rollout; 128 × 8 prompts (32B), 64 × 8 (7B)1,024 (32B); 512 (7B)32,768GPU-hours per run (day counts only)
ScaleRL: 7,400 steps (8B), 7,100 (Scout)1 (Fig. 1)One PipelineRL trainer update768 in the base setup; not stated for the 100k run14,336 generationCompletions per step in the large runs; 13.5 GPU-hours per step (derived)
Kimi K3 and K2.5: "scaling RL FLOPs" improves scores917 (K3 Fig. 8; K2.5 Fig. 2)No steps: the axis is RL FLOPs——The RL-FLOPs axes carry no tick values, so no FLOP figure can be read

Convert before comparing. R1-Zero-Qwen-32B's "over 10K steps" at 512 completions each is over 5.12 million completions (derived). DAPO's 50% of that is half the gradient updates at the same 512 completions per update, but the updates are fed by oversampling: DAPO keeps sampling until the batch holds only prompts whose accuracy is neither 0 nor 14 (§3.2), and the extra generation depends on the dead-group fraction, which Module 11 (Section 4.1) derives as \(p^G+(1-p)^G\) and which grows during training. DAPO's only compute statement is that training time was "not significantly affected"4 (§4.2), so "50% of the steps" is not "50% of the compute". Olmo 3's Figure 38 plots its 7B RL-Zero run against DAPO by gradient step and by GPU-hour, with DAPO's GPU-hours "taken from the DAPO reproduction on verl"8; the gap between the curves is about 2.5 times on the step axis and about 5 times on the GPU-hour axis (read off the plot; different base models). ScaleRL's fit sidesteps this by working in GPU-hours, the unit that does not depend on how a lab counts steps.

7.3 Against the alternative use of the same GPU-hours

A compute claim also needs a comparator. Qwen3's report ran both options from the same off-policy-distilled 8B checkpoint (AIME-24 55.0, AIME-25 42.8): further RL reached 67.6 and 55.5 at 17,920 GPU-hours, and on-policy distillation reached 74.4 and 65.5 at 1,800 GPU-hours, 9.96 times fewer (derived); on AIME-24 pass@64, RL stayed at 90.0 and distillation reached 93.316 (Table 21). The passage I read does not say whether the teacher's forward passes are in the 1,800. R1's supplementary comparison is the same direction at larger scale: Qwen2.5-32B trained for over 10K RL steps scored 47.0 on AIME-24 against 72.6 for the R1-distilled 32B model5 (Supp. F.1, Table 16). Module 21 covers on-policy distillation itself; the point here is that the cost of an RL recipe has to be read against a recipe that uses a stronger signal, not only against another RL recipe.

8. What a saturating pass-rate curve does not say

The fitted quantity is a mean pass rate over 16 samples per held-out prompt, an estimate of expected pass@1 on prompts from the training distribution1 (§2.1). Anything that moves probability mass toward answers the model could already reach raises it, so \(A\) is a ceiling on pass@1-type performance and says nothing about pass@k at large \(k\). Yue et al. report the pattern that matters here: RLVR-trained models beat their base models at small \(k\), base models win at large \(k\), and across training pass@256 falls while pass@1 rises (their Fig. 1), with six RLVR algorithms performing similarly and "far from optimal"18. ProRL reports the opposite for a 1.5B model trained over 2,000 steps with KL control, reference resets and diverse tasks, with boundary gains that correlate with how weak the base model was and with training duration15, and a later analysis proposes that both are partly right, as an exploitation stage followed by an exploration stage19. ScaleRL measures no pass@k, so a difference in fitted \(A\) between two recipes could be a difference in sharpening and not in what the model can solve; Module 20 treats that evidence.

The paper also cuts against a popular predictor. Entropy fell in every run it tracked, across batch sizes, tasks, generation lengths and model sizes, and the batch-768 and batch-2048 runs followed nearly identical entropy trajectories per step although the larger-batch run was much stronger downstream. The authors conclude that entropy "may not always offer a predictive insight into the performance, as proposed by some recent works like Cui et al."1 (App. A.12, Fig. 16). Cui et al.'s law \(R=-a\exp(H)+b\) fits performance against entropy along a run and puts the ceiling where entropy reaches zero20. The two statements concern different comparisons, a trajectory within one configuration against the same entropy curve across configurations, and neither paper tests the other (my reading). Module 13 develops the entropy law.

Where experts disagree

Two questions divide the sources. Do the details of an RL recipe move the ceiling or only the speed? And is more RL compute the right use of the next GPU-hours? The first is the Module 11 normalization debate seen through compute curves; the second is the question behind every budget in Figure 5.

A. Details move the speed, not the ceiling

Khatri et al., ScaleRL1 (abstract, App. A.8)

Loss aggregation, normalization, curriculum and the off-policy scheme "primarily modulate compute efficiency without materially shifting the asymptote". Pick choices that raise \(A\) first and tune \(B\) and \(C_{mid}\) second; stable recipes extrapolate, so small runs can rank them.

  • A replicated margin (\(\pm0.02\) on \(A\), three identical runs) and eight leave-one-out runs with \(A\) between 0.590 and 0.610.
  • Predictions held to 100,000 GPU-hours, and it agrees with Module 11's reading of the aggregation and normalization spreads.
  • No margin for \(B\); a 15% spread across identical runs is as large as most reversions' efficiency gaps.
  • The leave-one-out starts from a recipe whose stability fixes already absorb the choices that move \(A\), and the same paper's forward tests move \(A\) by up to 0.09.

B. Details decide whether the run survives and where it flattens

DAPO4 (Table 1); MiniMax-M13 (§1, §3.2); ScaleRL's forward tests1 (Figs. 5, 6, App. A.11)

DAPO's four techniques lift AIME-24 from 30 to 50 on one base; MiniMax found that a precision mismatch between training and inference kernels "prevents reward growth" until the LM head runs in FP32; ScaleRL's own forward tests move \(A\) from 0.520 to 0.595 (loss type) and 0.610 (FP32 head), and its Appendix A.11 says the off-policy choice shifts the asymptote.

  • Gaps of 0.07 to 0.09 in \(A\) are 3.5 to 4.5 times the replication margin.
  • The GRPO recipe became unstable after about 6k GPU-hours and the forward-test baseline beyond about 3.5k, so survival is itself an outcome.
  • DAPO's ladder is cumulative, one run per rung and one benchmark (Module 11, Section 7.1).
  • The forward \(A\) values extrapolate from runs that were breaking (Section 3.4), and the effects shrink inside a full recipe (Table 3's FP32 row).

C. Spend more RL compute: it still pays

DeepSeek-V3.213 (§1, §4.1); Kimi K39 (§4.1.2); ProRL15; ScaleRL's scaling axes1

V3.2 spends more than 10% of its pre-training cost on RL, reports "consistent performance improvements correlating with extended RL training budget" and hypothesizes more would help. K3 reports that scaling RL FLOPs improves knowledge, reasoning, vision, agent and coding scores, and ProRL's gains continued past 2,000 steps.

  • Production budgets keep rising, and ScaleRL identifies three axes (context, batch, model) that lift \(A\).
  • The fitted curve gives a way to forecast a budget's return.
  • No published fit at production scale, and K3's and K2.5's FLOP axes have no ticks (Section 7.2).
  • The base recipe has realized 97% of its gain by 16,000 GPU-hours (Section 2.2); later gains come from changing axes, each of which moves \(C_{mid}\) right 1.7 to 4.4 times (Table 7).

D. For small models, spend it on distillation

DeepSeek-R15 (Supp. F.1); Qwen316 (Table 21)

R1 found that 10K+ steps of RL on Qwen2.5-32B only matched QwQ-32B-Preview while distilling R1 was "significantly better"; Qwen3-8B reached higher scores with on-policy distillation at about a tenth of the GPU-hours of RL.

  • 9.96 times fewer GPU-hours for higher AIME scores in the one measured pair (derived), and a higher pass@64.
  • Cheap enough to repeat for every model size.
  • The teacher is itself an RL product, so compute is moved, not removed; R1 says going beyond "the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning".
  • One pair, one model size, and no statement in the passage read of whether teacher compute is counted.
Assessment

A and B describe different regimes, and both are right where they are measured. Inside a recipe that already carries the stability fixes (an asynchronous scheme with bounded lag, a weight-clipped or sequence-level loss, an FP32 head, zero-variance filtering), the evidence supports A about the ceiling: eight reversions land within 0.02 of each other. B is right about the failure side, where the fitted \(A\) is the point at which an unstable run bent. Neither licenses "efficiency does not matter": the reversions cost 14% to 46% more GPU-hours to reach the same pass rate (derived, with an assumed \(R_0\)), a tax on every ablation and sweep, and Olmo 3 reports running each post-training stage several times to sweep hyperparameters8; it does not change the ceiling. The ranking among the efficiency choices is the weakest part, because \(B\) has no margin. What would settle it: three seeds per leave-one-out variant, the same table on a second model family and a non-math domain, and a fit on a production run's held-out curve.

C and D answer different questions. If a stronger teacher exists and the target is a smaller model, D has the only compute-matched evidence in the source set. If you are building the model that sets the frontier, there is no teacher above you and C is the only option. For that case the fits say to choose by ceiling once the budget exceeds the crossover between candidate recipes (about 28,000 GPU-hours for context length in Figure 1) and by efficiency below it. The compute-matched comparison that would settle C against D, with teacher compute included, has not been published in the sources read.

Source-level conflict. ScaleRL cites RL at 3.75% of pre-training for R1-Zero and V3.2 reports more than 10%113. These are not contradictory: the shares are settings of one dial, differ in denominator and hardware, and Olmo 3's own day counts span 2.8% to 11.6% inside one model family (Section 7.1).

What is changing

As of 2026-10-27, ScaleRL is a single arXiv version (v1, 15 Oct 2025, 28 pages and 20 figures) with no venue listed on its arXiv page, and none of the sources read reproduces it1. Tan et al. is at v4 (17 Apr 2026) and states acceptance at the ACL 2026 main conference2; the two scaling frameworks have not been compared on common data. The authors list the next uses of their framework as multi-turn and agentic RL, dense rewards and compute-heavy generative verifiers1 (§7), settings that Kimi K3 and DeepSeek-V3.2 both train on at scale913, so a sigmoid fit to an agentic run would be the most informative missing result. Two production reports plot RL FLOPs without axis values917 and one states its budget as a share of pre-training13; either form leaves the comparison in Section 7 unresolved until a report prints GPU-hours, steps and completions together. The GSPO instability ScaleRL reports (Section 4) rests on one lab's framework and is the finding most likely to be confirmed or overturned by a second replication.

References

  1. paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R., Duvvuri, S. S., Zaheer, M., Dhillon, I. S., Brandfonbrener, D., Agarwal, R. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-27. Used for: Eq. 1 and its parameters (§1-§2.1, Fig. 3), fitting procedure and margin (App. A.4-A.8), forward ablations (§3, Figs. 4-6, 14, 19), leave-one-out (§4, Fig. 7, Fig. 15), scaling axes (§5, Figs. 9-11, Table 1), truncation analysis (App. A.15), compared recipes (Fig. 2, App. A.16), GSPO instability (App. A.17.4).
  2. paperTan, Z., Geng, H., Yu, X., Zhang, M. et al. Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning. arXiv:2509.25300v4 (v1 29 Sep 2025, v4 17 Apr 2026; ACL 2026 Main per arXiv comments). https://arxiv.org/abs/2509.25300. Accessed 2026-10-27. Used for: the power-law form with saturating k(N), 63 Qwen2.5 models, test-loss definition (Eqs. 1-2, 6; §2-§3.1).
  3. paperMiniMax (Chen, A., Li, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-27. Used for: 3 weeks on 512 H800 and $534,700 (abstract, §1, §3), CISPO versus DAPO "50% of the steps" and 16 off-policy rounds (§3.1), precision mismatch (§3.2), staged 40K to 80K length extension (§5).
  4. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-27. Used for: 50 AIME points in 50% of steps (Fig. 1), ablation ladder (Table 1), dynamic sampling (§3.2), batch and minibatch sizes and 20,480 maximum length (§4.1), training-time statement (§4.2).
  5. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (Nature version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-27. Used for: R1-Zero GPU-hours, 64×8 H800 for about 198 hours (Supp. B.4.4, Table 7); R1-Zero-Qwen-32B settings (Supp. B.4.1); distillation versus RL on Qwen2.5-32B (Supp. F.1, Table 16); GRPO recipe the ScaleRL authors re-implemented (via ref 1).
  6. officialHugging Face. TRL documentation: GRPO Trainer, main branch. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the scale_rewards options and the description of "batch" as group mean with batch-level standard deviation (first 100,000 characters read).
  7. blogPiche, A., Pardinas, R., Kamalloo, E., Bahdanau, D. (ServiceNow). PipelineRL. Hugging Face blog, 25 Apr 2025. https://huggingface.co/blog/ServiceNow/pipelinerl. Accessed 2026-10-27. Used for: in-flight weight updates, pause only to receive weights, stale KV cache, lag counted in optimizer steps, no speedup reported (sections "Conventional RL vs PipelineRL", the stability discussion, and "What's next").
  8. paperTeam Olmo (Ettinger, A., Bertsch, A. et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2 (14 Apr 2026; v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-27. Used for: infrastructure ablation (Table 23, p. 51), static-batching waste and long-tail description (§4.4.3), day counts and RL allocation (§2.4, Table 49, p. 109), Fig. 38 (p. 107).
  9. paperKimi Team (Moonshot AI). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-27. Used for: partial rollouts and staleness tolerance (§4.1.2), scaling RL FLOPs claim and Fig. 8 (p. 12-13).
  10. paperKimi Team (Moonshot AI). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599v4. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-27. Used for: RL context window scaled to 128k and the context-length observation (§1), model-size versus context ablation (ablation studies).
  11. officialNVIDIA. NVIDIA GB200 NVL72 product page, specifications table. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-10-27. Used for: 360 PFLOPS sparse FP16/BF16 per 72-GPU rack and the "dense is one-half sparse" note. The page may change.
  12. officialNVIDIA. NVIDIA H100 Tensor Core GPU product page, specifications table. https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-10-27. Used for: H100 SXM BF16 Tensor Core 1,979 TFLOPS "with sparsity".
  13. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-27. Used for: post-training budget "exceeding 10% of the pre-training cost" (§1, §4.1), the hypothesis that more budget helps (§4.1).
  14. paperDeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437. https://arxiv.org/abs/2412.19437. Accessed 2026-10-27. Used for: pre-training 2,664K H800 GPU-hours and total 2,788K (Table 1, §1).
  15. paperLiu, M., Diao, S., Lu, X. et al. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864v1. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-27. Used for: over 2k steps (§1), 16k GPU-hours on 4 nodes of 8 H100, batch and minibatch sizes and context window (§3.2), boundary-expansion claims (abstract).
  16. paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Alibaba Qwen Team). Qwen3 Technical Report. arXiv:2505.09388v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-27. Used for: 170 RL steps and 3,995 query-verifier pairs (§4.2), RL versus on-policy distillation on Qwen3-8B with GPU-hours (Table 21, p. 21).
  17. paperKimi Team (Moonshot AI). Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276v2. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-27. Used for: vision-RL curves plotted against RL FLOPs (Fig. 2).
  18. paperYue, Y., Chen, Z., Lu, R., Zhao, A. et al. (Tsinghua LeapLab / SJTU). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837v5. https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-27. Used for: pass@k crossover between RLVR and base models, pass@256 falling as pass@1 rises (abstract, Fig. 1).
  19. paperYao, X., Yu, L., Hu, X., Teng, F. et al. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028, 5 Oct 2025. https://arxiv.org/abs/2510.04028. Accessed 2026-10-27. Used for: the exploitation-then-exploration reconciliation (abstract only).
  20. paperCui, G., Zhang, Y., Chen, J. et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617v1. https://arxiv.org/pdf/2505.22617v1. Accessed 2026-10-27. Used for: the entropy-performance law R = -a exp(H) + b (abstract).