Reinforcement Learning for LLM Post-Training · Module 23
Length and Reasoning-Effort Control: Length Rewards, Token Budgets, Toggle, and Effort-Conditioned Experts
Response length is the one quantity that every reasoning-RL run sees growing and every inference bill pays for. This module reads what the reports actually do about it: the within-group length reward in Kimi k1.5, the per-problem budget overrides in K2.5 and K3, DeepSeek-V4's three trained effort modes, Qwen3's mode fusion, and the small-scale reward designs (Arora and Zanette, L1) that expose the arithmetic. After it you can pick a reward-side, loss-side, data-side or expert-side control for a target length-accuracy trade-off, and say what each one does to accuracy, tokens and decode cost.
1. Why a length control exists, and where it can act
1.1 The bill for a long chain
Outcome-reward RL pays for correct answers, not for brevity, and every report that watches the length curve sees it rise. Kimi k1.5 names the effect: "the model's response length significantly increases during RL training"; this "leads to better performance", but "an excessively lengthy reasoning process is costly during training and inference, and overthinking is often not preferred by humans"1 (§2.3.3). K2 adds the allocation argument: longer responses let the model spend test-time compute on hard reasoning, but "the benefits often do not justify its inference cost in non-reasoning domains"3 (§3.2.3). Chen et al. framed the same waste earlier as misallocation, reasoning models spending "excessive computational resources" on "simple problems with minimal benefit" (abstract only; I did not read the body)13. Why the length curve rises, and how to tell growth from runaway repetition, is Module 13 §5; the loss-normalizer cause is Module 11 §2. This module is about what the reports do on purpose.
Two different goals hide behind the phrase "length control". The first is efficiency: remove tokens the answer did not need while keeping the ceiling on hard problems. Arora and Zanette's design targets this, and the same coefficient removes 83% of tokens on GSM8K but 27% on AIME 2024 (7B model, α = 0.2)10 (§5.3.1). The second is selectability: the user, not the training run, chooses how much to think. DeepSeek-V4 trains three modes6, K3 trains low, high and max experts5, and gpt-oss exposes low, medium and high through the system prompt14. The goals interact: Toggle exists because efficiency training under rigid budgets produced "a length-overfitting phenomenon": models that "cannot effectively leverage additional inference-time tokens"4 (§4.4.2).
1.2 Five places a control can act
Table 1 sorts the controls by where they act. Read it along two axes. A penalty teaches the model that long is bad, so the trade-off is fixed at training time by a coefficient; an input (an effort label, a prompt-given target) teaches the model to condition on a request, so the trade-off is chosen per request. And a relative term (k1.5, Arora and Zanette) compares a response with its own group and so adapts to difficulty, while an absolute term (DAPO's 16,384 tokens, L1's target, a K3 budget) compares it with a number someone chose.
| Where it acts | Mechanism | Reports | Reported effect | Caveat |
|---|---|---|---|---|
| Serving time (not trained) | Stop thinking at a threshold and append a closing string | Qwen37, s1 budget forcing12 | Qwen3: smooth gains with budget (Fig. 2)7; s1: "Wait" appended, AIME24 50% to 57%12 | Truncates mid-reasoning; L1 reports 20 to 25 points absolute over s1 at 512 and 1,024 tokens (1.5B)11 |
| Reward: length term | Relative or absolute length added to, or multiplied into, the correctness reward | k1.51, Arora and Zanette10, L111, DAPO soft2, V3.215 | A&Z: tokens -36% for accuracy -2.2% on MATH (7B, α = 0.1)10 | Coefficient to tune; interacts with advantage normalization (Section 2.3) |
| Reward: budget override | Token budget per problem or task; an over-budget response is truncated, zeroed or set to -1 | K23, K2.5 Toggle4, K35 | Toggle: output tokens -25% to -30%, "negligible" effect4 | Rigid budgets overfit length; Toggle's ρ, λ and m are not in the text read |
| Loss normalizer | Divide by a constant or a batch token total instead of 1/|oi| | Dr. GRPO9, DAPO2, TRL16 | Removes one source of growth; gives no target length | Demonstrated at 1.5B to 3B (Module 11 §2.3) |
| Data | Train on shortest correct samples, length-ranked DPO pairs, truncated traces or effort-labelled SFT data | k1.5 long2short1, Nemotron 3 Ultra8, K2-V217 | k1.5-short: 60.8 on AIME at 3,272 tokens on average1 | Needs a long-CoT teacher; no RL signal on length |
| Mode or expert | Effort label as an input; one expert per level, then consolidate | V46, K35, Nemotron 3 Ultra8, gpt-oss14 | Nemotron medium effort: about 2.5x fewer tokens for about 7% lower accuracy8 | gpt-oss's training method is one sentence long14 |
2. Reward-side controls: a length term inside the verifier reward
2.1 Kimi k1.5: the length reward is relative to the group
For each problem, k1.5 samples \(k\) responses, takes the shortest and longest lengths in that group, \(\ell_{\min}\) and \(\ell_{\max}\), and defines the length reward from where a response sits between them. If all lengths are equal the length reward is zero for every response; otherwise1 (§2.3.3, printed equation on p. 6; I checked the form against the rendered page)
$$\lambda_i=\tfrac12-\frac{\ell_i-\ell_{\min}}{\ell_{\max}-\ell_{\min}},\qquad r^{\text{len}}_i=\begin{cases}\lambda_i & y_i\text{ correct}\\ \min(0,\lambda_i) & y_i\text{ wrong}\end{cases}$$The authors' summary is that the term promotes shorter responses and penalizes longer ones among the correct, "while explicitly penalizing long responses with incorrect answers", and that it is "added to the original reward with a weighting parameter"1. The weight is not given in the text I read. They also report that "length penalty may slow down training during the initial phases", so they warm it up: standard policy optimization with no length term first, then a constant length penalty for the rest of training1.
Three properties follow from the formula, none of them stated in the report (derived).
- Position, not length. The term depends only on where a response falls between the group's shortest and longest, so a group spanning 2,100 to 8,100 tokens and a group spanning 21,000 to 81,000 produce identical length rewards for identical positions. It adapts to problem difficulty automatically, but it has no absolute target: on an easy problem where every sample is 400 to 420 tokens, a 20-token difference swings the reward by a full 0.5.
- No bonus for short and wrong. The shorter half of the wrong responses gets exactly 0, so the term cannot reward a model for stopping early with a wrong answer.
- A weight ceiling. The total is \(r+w\,r^{\text{len}}\). The longest correct response scores \(1-0.5w\); the shorter-half wrong responses score 0. The ordering "any correct answer beats a short wrong one" therefore breaks when \(w>2\). The report does not give \(w\), so this is a bound on what \(w\) could be, not a finding.
| Sample | Length (tokens) | Correct? | Relative length | λ | Length reward | Total, w = 1 |
|---|---|---|---|---|---|---|
| 1 | 2,100 | yes | 0.000 | +0.500 | +0.500 | 1.500 |
| 2 | 2,400 | yes | 0.050 | +0.450 | +0.450 | 1.450 |
| 3 | 3,000 | yes | 0.150 | +0.350 | +0.350 | 1.350 |
| 4 | 3,600 | no | 0.250 | +0.250 | 0.000 | 0.000 |
| 5 | 4,400 | yes | 0.383 | +0.117 | +0.117 | 1.117 |
| 6 | 5,200 | no | 0.517 | −0.017 | −0.017 | −0.017 |
| 7 | 6,800 | no | 0.783 | −0.283 | −0.283 | −0.283 |
| 8 | 8,100 | yes | 1.000 | −0.500 | −0.500 | 0.500 |
Read the table the way the optimizer does. The mean of the eight totals is 0.702 (derived). Samples 1 to 3 and 5 sit above it and gain advantage. Sample 8 is correct, yet its total of 0.500 is below the mean, so it receives a negative advantage: at \(w=1\) the longest correct response is pushed down along with the wrong ones. The gap between samples 1 and 8, both correct, is 1.0 of reward, the same size as the correctness reward itself. That is the lever and also the risk: near \(w=1\) the length term competes with correctness on equal terms inside every group where length varies.
DAPO's overlong shaping is a different object: an absolute cap-handling device with a 4,096-token ramp before a 16,384-token expected maximum, derived and plotted in Module 11 §5, meant to stop truncated-but-sound reasoning from being scored as wrong rather than to steer length inside the cap2 (§3.4). The data-side companions of k1.5's length reward are in Section 6.2.
2.2 Arora and Zanette: a bounded factor multiplied into correctness
Arora and Zanette keep the correctness indicator and multiply a bounded, per-prompt-normalized length factor into it10 (§4, Eqs. 4 and 5):
$$R(x,y)=\mathbb{1}\{y=y^*\}\,\bigl(1-\alpha\,\sigma(z)\bigr),\qquad z=\frac{\mathrm{LEN}(y)-\mathrm{MEAN}(x)}{\mathrm{STD}(x)}$$\(\mathrm{MEAN}(x)\) and \(\mathrm{STD}(x)\) are the mean and standard deviation of the lengths of the correct responses to that prompt, estimated from the rollouts during training; \(\sigma\) is the sigmoid; \(\alpha\in[0,1)\) is the one knob. The authors give the design reasons: per-prompt normalization means long chains on hard problems are not penalized relative to short chains on easy ones, the sigmoid keeps the reward bounded in \([1-\alpha,1]\) "even for abnormally long or short generations", and "correct responses, even if long, are always preferred to incorrect ones"10 (§4). Compared with k1.5 the two are alike in being relative to the group (a z-score has the same spread whether the group spans 20 tokens or 20,000), and unlike in two ways: this form is multiplicative, so a wrong response is always 0 and never gains or loses anything for its length, and it is bounded, so the ordering problem of Section 2.1 cannot occur.
They train DeepSeek-R1-Distill-Qwen-1.5B and 7B with \(\alpha\in\{0,0.05,0.1,0.2,0.4\}\). On the 7B model, \(\alpha=0.1\) cuts MATH chains by 36% (about 4,000 to about 2,600 tokens) for a 2.2% accuracy loss, and \(\alpha=0.2\) cuts AIME 2024 tokens by 27% (about 13,000 to about 9,000) "while incurring only a 4% accuracy drop"; the same \(\alpha=0.2\) saves 83% on GSM8K10 (§5.3.1). They read the gap between 27% and 83% as easy problems having a larger relative spread, so reasoning models are "more wasteful on easier problems". At equal token usage the online-RL models beat SFT and DPO baselines (their Fig. 1), and the single coefficient gives a family of models, one per \(\alpha\)10. What \(\alpha\) does not do is hit a target length: the authors list this as a limitation and point to L1 for that case10 (§6).
2.3 Standard-deviation division can erase the coefficient
Arora and Zanette run an ablation worth deriving, because it applies to any small additive or multiplicative length term inside a GRPO-style group. Take a prompt where all \(k\) responses are correct, so every reward is \(1-\alpha f_i\) with \(f_i\in[0,1]\) spread evenly, which is the paper's assumption (uniform on \([1-\alpha,1]\)). The mean is \(1-\alpha/2\) and the standard deviation is \(\alpha/\sqrt{12}\). The advantage of the shortest correct response, with \(r=1\), is
$$A_{\text{short}}=\frac{1-(1-\alpha/2)}{\alpha/\sqrt{12}}=\frac{\alpha/2}{\alpha/\sqrt{12}}=\frac{\sqrt{12}}{2}=\sqrt{3}\approx 1.73,$$and the longest gets \(-\sqrt{3}\): "their difference is independent of α"10 (§5.4.1). Without the division, with a mean baseline only, the endpoints are \(\pm\alpha/2\): 0.025 at \(\alpha=0.05\), 0.1 at \(\alpha=0.2\), 0.2 at \(\alpha=0.4\) (derived). For a finite group of \(k=8\) evenly spaced rewards the endpoint is 1.43 with the sample (\(n-1\)) standard deviation and 1.53 with the population one; \(\sqrt3\) is the continuum limit (derived). For scale, a group with four correct and four wrong answers gives a correct response an advantage of about +1 under the same division (mean 0.5, standard deviation about 0.5; derived). So in a group where everything is correct, the length term hands out advantages larger than the correctness signal in a half-right group, whatever \(\alpha\) is. The authors observe that the resulting length decrease is "generally too substantial for the model to absorb, leading to a sharp drop in accuracy", and they do not normalize advantages10 (§5.4.1, Fig. 4).
One limit on the derivation, from me and not from the paper: it assumes the length term is the only source of reward variance in the group. In groups that mix correct and wrong answers the correctness variance enters the standard deviation and the length term's share of the advantage shrinks. I have not found a report that measures where the crossover lies.
2.4 The reward forms side by side
| Form | Length quantity | Combination | Range of the length effect | Long correct below wrong? |
|---|---|---|---|---|
| k1.51 | Position between group min and max | Added, weight not given | -0.5 to +0.5 before weight | Yes if the weight exceeds 2 (derived) |
| DAPO soft overlong2 | Absolute length beyond 16,384 tokens, 4,096-token ramp | Added | 0 to -1 | No: a truncated correct answer ties a wrong answer inside the cap at 0 (derived) |
| Arora and Zanette10 | z-score among correct responses | Multiplied into correctness | Correct responses in [1-α, 1] | No (α < 1) |
| L1-Exact11 | Distance to a target in the prompt | Added, α = 0.0003 | Unbounded below | Yes beyond 1/α = 3,333 tokens off target (derived) |
| L1-Max11 | Excess over a target in the prompt | Multiplied, δ = 0.5 | [0, 1] | No: ties at 0 (derived) |
| K2 budget3 | Per-sample budget by task type | Truncate and penalize | Not stated | Not stated |
| K2.5 Toggle4 | Budget from a percentile of correct lengths | Reward zeroed over budget | {0, r} | No: ties at 0 (derived) |
| K35 | Budget τ·b0 per problem | Reward overridden with -1 | -1 over budget | Yes if the task reward is non-negative; its range is not stated in the text read (derived) |
The frontier reports (K2, K2.5, K3, V4) use hard or gated budgets and say little about the numerical form, while the small-scale papers (Arora and Zanette, L1) publish the full reward and its ablations. Section 7 returns to that asymmetry.
3. Budget overrides: K2, Toggle and K3
3.1 K2: a per-sample budget by task type
K2 adopts k1.5's policy-optimization algorithm as its foundation3 (§3.2.3) and adds a budget rule: "we enforce a per-sample maximum token budget throughout RL training, where the budget is determined based on the type of task. Responses that exceed this token budget are truncated and assigned a penalty". The report says this "significantly enhances the model's token efficiency" and gives no budgets, penalty size or numbers in the text I read3. The structural point is that the budget belongs to the task type and is set by the lab. The two later designs replace it with a budget derived from the model itself: a percentile of its own correct lengths (Toggle) or a multiple of the cold-start model's per-problem budget (K3).
3.2 K2.5 Toggle: alternate the constraint on and off
Toggle starts from the failure the K2 recipe produced. Models "trained under rigid budget constraints often fail to generalize to higher compute scales" and default to "truncated reasoning patterns"4 (§4.4.2). The fix is to stop applying the constraint all the time. For learning iteration \(t\), with \(K\) rollouts per problem, the reward is4
$$\tilde r(x,y_i)=\begin{cases} r(x,y_i)\cdot\mathbb{1}\!\left[\bar r(x)<\lambda\ \ \text{or}\ \ |y_i|\le \mathrm{budget}(x)\right] & \lfloor t/m\rfloor \bmod 2=0\ \ (\text{Phase 0})\\[2pt] r(x,y_i) & \lfloor t/m\rfloor \bmod 2=1\ \ (\text{Phase 1})\end{cases}\qquad \mathrm{budget}(x)=\mathrm{Percentile}\bigl(\{|y_j|: r(x,y_j)=1\},\,\rho\bigr)$$where \(\bar r(x)=\frac1K\sum_i r(x,y_i)\) is the problem's mean accuracy and the percentile is over the lengths of the correct responses. The printed indicator carries the index \(y_i\) inside a sum; I read it per response. Phase 0 is budget-limited, Phase 1 is the standard objective, and they swap every \(m\) iterations. Four properties follow.
- A quality gate. The budget binds only on problems whose mean accuracy is already at least \(\lambda\); the authors say this prevents "a premature sacrifice of quality for efficiency"4. Below the threshold the problem is trained on correctness alone.
- A fixed budget. It is "estimated once at the beginning of training and remains fixed thereafter"4, so the target does not ratchet downward as the policy shortens.
- One-sided. Over-budget responses get \(\tilde r=0\). For a binary reward that is the same as being wrong, and wrong responses are untouched, so unlike k1.5 nothing pushes long wrong answers further down (derived).
- Half the iterations. Phase 1 restores the unconstrained objective, which "encourag[es] the model to leverage computation for better inference-time scaling"4. The authors call the whole scheme "a stochastic alternating optimization for a bi-objective problem".
The values of \(\rho\), \(\lambda\) and \(m\) are not given in the text I read. A toy shows what \(\rho\) controls. Suppose a problem's five correct responses have lengths 3.2k, 4.1k, 4.8k, 6.0k and 9.5k tokens (illustrative). With linear-interpolation percentiles, \(\rho=50\) gives a budget of 4.8k and \(\rho=75\) gives 6.0k (derived). At \(\rho=50\), a budget estimated from the starting policy puts two of the five correct responses over budget in the first Phase 0, and at \(\rho=75\) one of five. A median budget therefore starts by zeroing roughly half of what the model currently gets right on that problem, on every problem where the quality gate is open.
What it bought. The report states that Toggle "decreases output tokens by 25∼30% with a negligible impact on performance", that redundant patterns such as repeated verification and mechanical calculation decrease substantially, and that it generalizes across domains: trained only on mathematics and programming, it still cut tokens on GPQA and MMLU-Pro "with only marginal degradation"4 (§4.4.2, Fig. 5; evaluated on K2 Thinking). Figure 5 plots the per-benchmark values as printed on the radar charts; I read them from the rendered page, because the text gives only the average.
Absolute deltas rank benchmarks by length, not by how much Toggle helped, since baselines differ by an order of magnitude. The report presents one run per benchmark and no variance in the text I read, so a 0.3-point overall gain and a 1.0-point GPQA loss are within what the figure cannot distinguish.
3.3 Kimi K3: a per-problem budget, annealed into effort levels
K3 turns the same device into a family of models. Each problem \(x\) carries a token budget \(b_0(x)\) "estimated from the cold-start model", and any trajectory whose total exceeds a scaled threshold has its task reward overridden5 (§4.1.2, "Reasoning Effort RL"):
$$\tilde r(x,y)=\begin{cases}-1 & T(y)>\tau\,b_0(x)\\ r(x,y) & \text{otherwise}\end{cases}$$For general tasks \(T(y)\) counts thinking tokens; for agentic tasks it counts "cumulative output tokens, including both reasoning traces and tool-call arguments"5. Training "follows a stage-wise curriculum over the budget multiplier \(\tau\)": a max-budget variant first, "with a relatively large \(\tau\), while still capping the maximum budget to suppress excessive overthinking", then \(\tau\) annealed to smaller values to obtain the high- and low-effort experts, with "the adjustment of \(\tau\) ... configured per domain under human-in-the-loop guidance"5. Three domains (general tasks, general agents, coding agents) crossed with three efforts (low, high, max) give nine experts5 (§4.1.2). The values of \(\tau\), the estimator for \(b_0\) and the task-reward range are not stated in the text I read.
Four comparisons with Toggle, all derived from the two printed forms. First, K3 does not alternate in time; it separates in space. The max-budget expert plays the role of Toggle's Phase 1 and is trained as its own model, and the lower-effort experts are annealed from it. Second, the override value differs: Toggle zeroes, K3 sets \(-1\), which is strictly worse than a wrong answer if the task reward is non-negative (its range is not stated). Third, both budgets are set before RL begins, K3's from the cold-start model (how \(b_0\) is estimated is not stated) and Toggle's from a percentile of correct lengths, so neither ratchets down as the policy shortens. Fourth, K3 counts tool-call arguments as output tokens for agentic tasks, so a long-horizon agent is charged for what it writes into tool calls but not for tool results it reads.
The effort level reaches the final model through distillation. K3's per-token reward is \(\mathrm{clip}\bigl(\mathrm{sg}[\log \pi^{(d,e)}_{\text{teacher}}(y_t\mid x,y_{<t})-\log \pi_\theta(y_t\mid e,x,y_{<t})],-R_{\max},R_{\max}\bigr)\): the student is conditioned on \(e\), the teacher is selected by domain \(d\) and effort \(e\), and trajectories from all nine experts feed both SFT and the distillation stage5 (§4.1.2, §4.1.3, Eq. 15). The distillation mechanics belong to Module 22. The report's only effort-specific result in the text I read is a cost claim: on Kimi Code Bench 2.0 "at high effort it already matches Claude Opus 4.8's maximum-effort score at roughly one third of the cost", with costs measured internally and other models run through Claude Code5 (§6.4). K3 applies the same budget form to its judge as a verbosity guard; that guard and its failure modes are in Module 16 §6.4 and Module 19 §4.1.
4. Effort as an input: prompt-given targets, trained modes, and what the reports disclose
4.1 L1: the target length is in the prompt
L1 makes length an explicit input. Each training prompt gets the suffix "Think for \(n_{\text{gold}}\) tokens." with \(n_{\text{gold}}\) drawn uniformly from 100 to 4,000, and GRPO optimizes11 (§3, Eqs. 1 and 2)
$$r_{\text{Exact}}=\mathbb{1}[y=y^*]-\alpha\,|n_{\text{gold}}-n_y|,\qquad r_{\text{Max}}=\mathbb{1}[y=y^*]\cdot\mathrm{clip}\bigl(\alpha\,(n_{\text{gold}}-n_y)+\delta,\,0,\,1\bigr)$$with \(\alpha=0.0003\) in Eq. 1 and \(\delta=0.5\) in Eq. 2 (the text I read gives no separate \(\alpha\) for Eq. 2). The base is DeepScaleR-1.5B-Preview, trained 700 steps for L1-Exact and 120 more for L1-Max, with a 4K-token training context and 8K at evaluation11 (§4). The arithmetic of Eq. 1 is simple. Each token of deviation costs 0.0003, so a response 1,000 tokens off target loses 0.3; a correct 2,548-token answer to a 2,048-token target scores 0.85; and a correct answer more than \(1/\alpha=3{,}333\) tokens off scores below a wrong answer of exactly the right length (derived). Since training targets reach 4,000, the penalty can reach about 1.17 (derived), so Eq. 1 has the ordering problem of Table 3 in principle. The multiplicative Eq. 2 does not: a correct answer far over budget ties a wrong one at 0.
The results are about control, not just savings. Against s1's budget forcing, which truncates generation and inserts "Final Answer", L1 gains "over 100-150% relative and 20-25% absolute" at 512 and 1,024 tokens, because "S1 often truncates mid-reasoning"; its mean length error on math is "close to 3%"; and L1-Max matches the unconstrained DeepScaleR-4K by adapting to difficulty, often with up to 2x fewer tokens than L1-Exact11 (§5, Figs. 2, 4, 11). One ablation carries a design lesson. Replacing the multiplicative gate with an additive length penalty gave "a 0% budget violation rate" because "the model collapses to generating extremely short CoTs"11 (App. A.7): an additive term can be satisfied by abandoning correctness, a gated one cannot. The scale caveat is large: a 1.5B model trained at 4K tokens, against V4's 128K and 384K evaluation windows below.
4.2 DeepSeek-V3.2, Speciale and V4's three modes
DeepSeek-V3.2's reasoning and agent rewards combine a rule-based outcome reward, a length penalty and a language-consistency reward, with no functional form given15 (§3). Its high-compute variant V3.2-Speciale "was trained exclusively on reasoning data with a reduced length penalty during RL", plus DeepSeekMath-V2's proof data and reward15 (§3). The report's Table 3 lists accuracy and output tokens for both. Across its eight rows Speciale spends 1.39x to 2.29x the tokens of V3.2 (derived): AIME 2025 23k against 16k tokens for 96.0 against 93.1; GPQA-Diamond 16k against 7k for 85.7 against 82.4; HLE 35k against 21k for 30.6 against 25.115 (Table 3). It is not a clean penalty ablation, since the training data differ too. The authors say plainly that Speciale's token efficiency "remains significantly inferior" to Gemini-3.0-Pro's, that the official V3.2 "imposed stricter token constraints during the training" to balance performance and cost, and that its numbers are "constrained by a length constraint reward model"15 (§4.1, §4.2).
V4 makes the two V3.2 operating points a product. "DeepSeek-V4-Pro and DeepSeek-V4-Flash both support three specific reasoning effort modes. For each mode, we apply distinct length penalties and context windows during RL training", and the specialists are trained "under divergent RL configurations"6 (§5.1.1). The modes are Non-think (response format "</think> summary"), Think High, and Think Max, which adds a system-prompt instruction beginning "Reasoning Effort: Absolute maximum with no shortcuts permitted."6 (Tables 2 and 3). Evaluation context windows are 8K, 128K and 384K tokens for the three modes, 16x and then 3x steps (derived)6 (§5.3.1). The report says Max "employs longer contexts and reduced length penalties in RL" and "outperforms the High mode on the most challenging tasks"6 (§5.3.2). No penalty values are given. Table 4 and Figure 7 show what the modes buy.
| Benchmark | Flash Non-think | Flash High | Flash Max | Pro Non-think | Pro High | Pro Max |
|---|---|---|---|---|---|---|
| HLE | 8.1 | 29.4 | 34.8 | 7.7 | 34.5 | 37.7 |
| HLE total tokens, about | - | 35k | 86k | - | 35k | 53k |
| Terminal Bench 2.0 | 49.1 | 56.6 | 56.9 | 59.1 | 63.3 | 67.9 |
| Terminal Bench total tokens, about | 37k | 47k | 57k | 28k | 36k | 50k |
| GPQA Diamond | 71.2 | 87.4 | 88.1 | 72.9 | 89.1 | 90.1 |
| Apex | 1.0 | 19.1 | 33.0 | 0.4 | 27.4 | 38.3 |
| SWE Verified (resolved) | 73.7 | 78.6 | 79.0 | 73.6 | 79.4 | 80.6 |
Combining Table 4's accuracy and token rows gives marginal returns (derived, with the token readings' uncertainty). On HLE, V4-Flash's Max mode spends about 2.4x High's tokens (86k against 35k) for 5.4 points, and V4-Pro's spends about 1.5x (53k against 35k) for 3.2 points. On Terminal Bench 2.0, Flash Max costs about 1.2x High's tokens for 0.3 points, while Pro Max costs about 1.4x for 4.6. The report's own reading of the same figure is that on HLE "DeepSeek-V4-Pro demonstrates higher token efficiency than DeepSeek-V3.2"6 (§5.3.2, Fig. 10). My cross-check on the readings: Speciale's HLE position in Figure 10 lands at about 35k, matching its 35k in the V3.2 report's Table 315.
4.3 Nemotron 3 Ultra: a medium-effort mode from 2.5% of the RL prompts
Nemotron 3 Ultra is "trained for three reasoning modes: reasoning-off, regular and medium-effort", and the last two can be combined with inference-time budget control8 (§3.5). Medium effort begins in SFT with two data components: samples generated by GPT-OSS-120B in its medium-effort mode on math, STEM and instruction-following prompts, and samples whose "reasoning traces are truncated to random reasoning budgets while the responses remain the same", with the closing </think> token masked from the loss in truncated samples8 (§3.1.1). RLVR then optimizes it: "Approximately 2.5% of the RLVR training prompts are in the medium-effort mode", covering math, STEM and coding, "and length-based adjustments are applied on the RL rewards for them"; the effect "generalize[s] to a variety of tasks beyond" those, and the mode is calibrated "by adjusting hyperparameters"8 (§3.5). The adjustment's form and size are not given. The headline is that medium effort "uses on average around 2.5X less tokens than the regular mode at the cost of approximately 7% drop in accuracy" on the Artificial Analysis Intelligence Index V4, averaged over its 10 tasks with token use measured relative to Qwen 3.5 397B8 (§3.5, Fig. 11).
Reading the plotted points (about ±0.03 on the token axis), regular Ultra sits near 1.06 and medium near 0.37, a ratio of about 2.9 rather than 2.5, and the index falls from about 49.8 to about 42.7, about 7 index points rather than 7% relative (derived from the figure). The report does not say how it averages, so the gap may be a mean of per-task ratios against a ratio of means. The design lesson is that a small, effort-labelled slice of RL prompts with a length adjustment on its rewards moved a whole mode, which is consistent with Toggle's domain generalization.
4.4 How the level reaches the model
| Model | Levels | How selected | How trained, as disclosed |
|---|---|---|---|
| gpt-oss14 | low, medium, high | System-prompt keyword such as "Reasoning: low" | "We train the models to support three reasoning levels" (§2.5.2); nothing more. Accuracy rises smoothly with average CoT length (Fig. 3) |
| K2-V217 | low, medium, high | Special tokens <think_faster>, <think_fast>, <think> in the chat template | SFT only, 17.4B tokens per epoch for 3 epochs, data regenerated with GPT-OSS-120B at the three efforts (§6.1, §6.2) |
| Qwen37 | thinking, non-thinking, plus a budget | /think and /no_think flags; budget by truncating and inserting a stop string | Continual SFT on both modes, then general RL; the budget "is not explicitly trained" (§4.3) |
| Kimi K35 | low, high, max (four reserved: low, medium, high, max) | A "thinking-effort" option message stating the level in natural language | Per-effort RL experts with annealed budgets, then MOPD (§4.1.2, §4.1.3; App. F) |
| DeepSeek-V46 | Non-think, High, Max | Response format by mode; a system-prompt instruction for Max | Distinct length penalties and context windows per mode in RL (§5.1.1) |
| Nemotron 3 Ultra8 | off, regular, medium | Mode, optionally with inference-time budget control | SFT from GPT-OSS-120B medium traces and truncated traces; 2.5% of RLVR prompts with length-adjusted rewards (§3.1.1, §3.5) |
K3 describes its option message as acting "as a generation-constraint instruction" that, "instead of modifying the generation prefix or exposing a token budget", states the level in natural language; because the pretrained model already follows such instructions, new options can be added "with little or no additional training"5 (App. F). Lambert's survey groups these under "toggleable reasoning with system prompt (or length control)" and says that for gpt-oss and K2-V2 "training methods for this type of behavior are not as well documented"18 (§7.3.3). K2-V2's report claims to be "the first open recipe" for three effort levels and shows the SFT-only route works at 70B17 (§6), which is a data-side result, not an RL one.
5. Qwen3: two modes in one model, and a budget that was never trained
Qwen3 is the report on the opposite design from K3 and V4: one set of weights, two modes, and a thinking budget that no objective ever optimized. The flagship's third stage, Thinking Mode Fusion, continues SFT from the reasoning-RL model to add non-thinking behavior7 (§4.3).
- Reasoning-RL model. Stage 2 is thinking-only; Qwen3-32B scores 83.8 on AIME'24 and 68.4 on LiveCodeBench v5 here7 (Table 22).
- Fusion SFT. The thinking data is rejection-sampled from the Stage 2 model itself so its performance is not eroded; the non-thinking data is curated across coding, math, instruction following, multilingual, creative writing, QA and role-play. A /think or /no_think flag in the user or system message selects the mode, a non-thinking sample keeps an empty thinking block, and in multi-turn chats the model follows the last flag7 (§4.3, Table 9).
- General RL. Mode-switching accuracy on the in-house ThinkFollow benchmark moves from 88.7 to 98.9 (+10.2), while thinking-mode AIME'24 slips to 81.4 and LiveCodeBench v5 to 65.77 (§4.7, Table 22).
- Budget at serving time. When thinking reaches a user-set threshold, the runtime halts it and inserts "Considering the limited time by the user, I have to give the solution based on the thinking directly now." followed by the closing tag; the model then answers from its partial reasoning7 (§4.3).
The costs and the gap are in Table 22, which tracks Qwen3-32B through the stages7 (§4.7). Thinking-mode AIME'24 goes 83.8, 81.9, 81.4 across Stages 2, 3 and 4, and LiveCodeBench v5 goes 68.4, 67.2, 65.7. The non-thinking mode of the same weights scores 31.0 and 31.3 at Stage 4, so the mode switch is worth about 50 points on AIME'24 and 34 on LiveCodeBench (derived). The authors conjecture the thinking-mode drop is "due to the model being trained on a broader range of general tasks" and choose to "accept this performance trade-off to enhance the model's overall versatility"7 (§4.7). They report "scalable and smooth performance improvements correlated to the allocated thinking budget" for Qwen3-235B-A22B on four benchmarks and expect gains beyond 32K tokens7 (Fig. 2).
The stop-string mechanism is the same family as s1's budget forcing, which "forcefully terminat[es] the model's thinking process" or lengthens it by appending "Wait"12. What fusion adds is that a truncated prefix is a state the model has seen. What no RL objective in the Qwen3 pipeline measures is the quality of the answer at a given budget, because the budget is imposed after training. That is the specific thing K3's annealed experts, V4's per-mode penalties and L1's prompt-given targets train directly, and L1's comparison against truncation at 512 and 1,024 tokens is evidence, at 1.5B scale only, that training at the target wins at small budgets11.
6. Loss-side and data-side controls
6.1 The loss normalizer: removing a push is not setting a target
Module 11 §2 derives why GRPO's per-response \(1/|o_i|\) makes shorter correct responses receive larger per-token updates and longer wrong ones smaller penalties, and Module 11 §2.3 gives Dr. GRPO's evidence, at 1.5B to 3B and 3,000-token caps. Dr. GRPO removes the length and standard-deviation normalizers and reports 43.3% on AIME 2024 with a 7B base9 (abstract, §3.2); DAPO averages over all tokens in the batch2 (§3.3); TRL defaults to loss_type="dapo" and offers "dr_grpo", which divides by a constant that "is recommended to be the maximum completion length"16.
This module's sources add one independent data point. Arora and Zanette's \(\alpha=0\) control, with no length penalty at all, still shortened responses on MATH and AIME. They traced it to the OpenRLHF implementation of RLOO, which normalizes the loss by response length, and "removing the length bias from the implementation leads to no change in the response length of the model after training"10 (§5.3.1). So a normalizer alone can move length, here toward shorter, with no reward term. What it cannot do is set a target: the fix removes a push and leaves the model to settle wherever the task and the cap put it. The reward-side and mode-side controls exist because a target is wanted. I found no report in the set that ablates a loss-side fix against a reward-side term at matched scale, so the two are not ranked by evidence.
6.2 Data: shorten the traces, then train on them
k1.5's long2short section is the data-side counterpart of its length reward. It lists four routes for moving a long-CoT model's strength into a shorter-output model1 (§2.4):
- Model merging averages the weights of a long-CoT and a short-CoT model, "without training".
- Shortest rejection sampling samples each question \(n=8\) times and keeps the shortest correct response for SFT.
- DPO takes the shortest correct solution as the positive and treats longer responses as negatives, "including both wrong longer responses and correct longer responses (1.5 times longer than the chosen positive sample)".
- Long2short RL picks the best token-efficiency trade-off from standard RL as the base, then applies the Section 2.1 length penalty with a "significantly" reduced maximum rollout length.
The report finds long2short RL "demonstrates the highest token efficiency" of the four, with k1.5-short reaching 60.8 on AIME 2024 (mean of 8 runs) at 3,272 tokens on average1 (§3.4, Fig. 7). Its abstract states short-CoT results of 60.8 on AIME, 94.6 on MATH-500 and 47.3 on LiveCodeBench, above GPT-4o and Claude Sonnet 3.5 "by a large margin (up to +550%)"1. The DPO route turns a known failure into the signal: Module 7 §4.3 covers how DPO drifts toward verbosity, and here the preference pairs are length-ranked on purpose, so the same sensitivity becomes the mechanism (derived).
The other data-side examples are SFT-only. Nemotron 3 Ultra's truncated-trace samples give the model the experience of answering from a partial chain, with the closing tag masked from the loss, which is the training-time analogue of Qwen3's serving-time stop string8 (§3.1.1). K2-V2 gets three effort levels from SFT alone, on data regenerated at three efforts by GPT-OSS-120B17 (§6.1). Both depend on a teacher that already behaves the way the student should, and neither puts a reward on length. They are the cheapest route when an effort-controlled teacher exists.
7. Reading the accuracy-versus-tokens evidence
7.1 One benchmark, several recipes
Almost every report states its result as a token reduction against its own baseline, and none shares a baseline with another. One benchmark gives a common axis. HLE appears in the V3.2 report's Table 3 with accuracy and output tokens for five models, and in V4's Table 7 and Figure 10 for four V4 configurations. Figure 9 puts them on one plot. Accuracies are as printed; token counts are exact (to the nearest 1k) for the V3.2-report models and read off Figure 10's axis for V4.
Two things stand out. Inside a family the last effort level is the expensive one: V4-Flash Max spends about 2.4x High's tokens for 5.4 points, while V4-Pro High reaches 34.5 at about the token count of Speciale's 30.6, which is what the V4 report means by higher token efficiency. And across labs the token axis spreads by about 3.5x at a fixed score, a gap that no length term in this module is designed to close.
7.2 What each claim is measured against
| Report | Claimed change | Measured against | Accuracy change | Limits |
|---|---|---|---|---|
| K2.5 Toggle4 | Output tokens -25% to -30% | K2 Thinking before Toggle; average over benchmarks | "negligible"; -2.0 to +2.2 per benchmark (Fig. 5) | One run; baselines not printed |
| Arora and Zanette10 | -36% MATH (α = 0.1); -27% AIME and -83% GSM8K (α = 0.2) | R1-Distill-Qwen-7B | -2.2% MATH; "only a 4% accuracy drop" AIME | 1.5B and 7B; approximate token counts |
| Nemotron 3 Ultra8 | About 2.5x fewer tokens | Regular mode of the same model | About 7% (about 7 index points by the plot) | 10-task index; verbosity relative to another model |
| V3.2 to Speciale15 | 1.39x to 2.29x more tokens | V3.2 Thinking | +2.9 to +6.7 points on seven benchmarks | Data also differ (reasoning-only, proof data) |
| V4 Flash, High to Max6 | About 2.4x more tokens on HLE | V4-Flash High | +5.4 points | Tokens read off a plot |
| k1.5-short1 | 3,272 tokens on average | Other short-CoT models (Fig. 7) | 60.8 on AIME 2024 | A model comparison, not a before and after |
| L111 | Length error about 3% | s1 budget forcing at 512 and 1,024 tokens | +20 to +25 points absolute | 1.5B, 4K training context |
The token measure itself changes row to row: output tokens (V3.2, Toggle), "total tokens" including tool traffic (V4's Terminal Bench panel), thinking tokens only for K3's general tasks and cumulative output including tool-call arguments for its agents5, and a verbosity ratio against a third model (Nemotron). Read the right-hand column first.
7.3 Choosing a control
| If you need | Start with | Why | Watch for |
|---|---|---|---|
| Stop unexplained length growth | Fix the loss normalizer, then monitor (Module 13 §5) | One source of growth removed, no knob910 | No target length results |
| Fewer tokens on easy prompts, long chains on hard ones | A relative, gated reward term, no standard-deviation division, warmed up after standard RL | 83% against 27% saving at one α10; warm-up1 | Section 2.3 identity; coefficient tuning |
| A hard cap without length overfitting | A budget with a quality gate, alternated with an unconstrained phase | Toggle's design response to rigid budgets4 | ρ, λ, m are undisclosed |
| User-selectable effort at inference | Effort label as input, per-effort rewards or experts, or a prompt-given target | V4, K3, Nemotron658; L111 | Penalty values and \(\tau\) are undisclosed |
| Effort levels without RL infrastructure | SFT on teacher traces generated at each effort | K2-V217 | Needs an effort-controlled teacher |
| A budget on a model already trained | Serving-time truncation, or long2short | Qwen37; k1.51 | Quality at a budget is never measured by an objective |
8. Serving-side implications (my inference, not the reports')
None of the reports analyzes serving cost beyond a token count; this section is my reading, built on the standard memory-bound decode model. A decode step at context length \(c\) reads the weights, amortized over a batch of \(B\) requests, plus the request's own KV cache. With \(W\) the weight bytes read per step and \(k\) the KV bytes per token, a request that generates \(L\) tokens after a prompt of \(P\) tokens moves
$$\sum_{t=1}^{L}\Bigl(\tfrac{W}{B}+k\,(P+t)\Bigr)=L\Bigl(\tfrac{W}{B}+kP\Bigr)+\tfrac{k}{2}L^2=a\,L\Bigl(1+\tfrac{\kappa}{2}L\Bigr),\qquad a=\tfrac{W}{B}+kP,\ \ \kappa=\tfrac{k}{a}.$$When \(\kappa L\ll1\), cost is linear in tokens and a token reduction is a one-for-one throughput gain: Toggle's 25% to 30% fewer output tokens4 is \(1/0.75\) to \(1/0.70\), 1.33x to 1.43x more requests per decode-second, and Nemotron's medium effort at about 2.5x fewer tokens8 is about 2.5x. When KV traffic matters, the quadratic term makes the saving larger than the token ratio; the calculator lets you set \(\kappa\). Prefill, speculative decoding and compute-bound regimes are outside this model.
Relative decode traffic of a request before and after an effort cut
Formula: \( C(L)=L\,(1+\kappa L/2) \) with \( \kappa \) in units of 1e-6 per token, the ratio of per-token KV traffic to the per-token fixed traffic \( a \). The defaults (10,000 to 4,000 tokens, \( \kappa=5\times10^{-5} \)) are illustrative: the 2.5x token ratio is Nemotron's8; \( \kappa \) is not from any source and depends on your model, batch size and prompt length.
Four properties of effort modes matter to an engine, each grounded in a report but extended by me.
- Reservation scales with the mode. V4's evaluation windows are 8K, 128K and 384K tokens for Non-think, High and Max6 (§5.3.1), so a Max request can reserve up to 48x the KV of a Non-think one (derived). Admission control can key on the mode instead of the worst case.
- Length variance differs by level. K2-V2 observes that its low and medium efforts "have fairly consistent generation length, while high has a large spread"17 (§6.1). A tail of long generations is what stalls batches, and why K3 uses partial rollouts in training5 (§4.1.2).
- The effort setting sits in the prefix. K3 places "the reasoning-effort setting" before all input messages because global options "rarely change, so modifying them invalidates the KV cache anyway"5 (App. F). A per-request effort router that changes this message forfeits prefix-cache reuse across modes; V4's Max instruction is also a system-prompt prefix6 (Table 3).
- One model replaces two deployments. Qwen3 cites "reducing the cost and complexity of deploying separate models for thinking and non-thinking tasks"7 (§4.3), and K2-V2 argues a single model with a chat-template setting avoids serving separate instruct and thinking variants17 (§6).
Where experts disagree
The reports agree that unconstrained RL lengthens chains and that something should be done about it. They disagree about what kind of object length is: a cost to penalize, a budget to enforce, a bias to remove from the loss, or an input the user sets. Frontier recipes mix these, so the positions below are the pure forms.
1. Penalize length inside the reward, continuously and relative to the group
Moonshot's k1.51; Arora and Zanette10; DeepSeek-V3.2 and its reduced-penalty Speciale15
A smooth term adapts to difficulty automatically and one coefficient yields a family of models.
2. Enforce a budget, then alternate or anneal it
A hard override has no relative weight to tune. Alternation keeps the ability to scale up, and annealing the multiplier produces levels.
- Output tokens down 25% to 30% with "negligible" accuracy impact, and the cut transfers to benchmarks outside the training mix4
- A quality gate prevents trading accuracy away on problems the model cannot yet solve4
- Rigid budgets caused the length-overfitting Toggle was built to fix4
- \(\rho\), \(\lambda\), \(m\), \(\tau\) and the override's size are not disclosed in the text read
- Evidence is one run per benchmark, no variance
3. Do not train length: fix the loss, cut at serving time
Dr. GRPO9; Qwen3's untrained budget7; s1's budget forcing12
Part of the growth is a normalizer artefact, and a model fused across modes can be stopped early without an objective for it.
- No coefficient and no capability tax from a penalty
- Qwen3 reports smooth gains with budget7; A&Z's \(\alpha=0\) control shows a normalizer alone moved length10
- No target length results from a loss fix
- Quality at a budget is never optimized; L1 beats s1 truncation by 20 to 25 points at 512 and 1,024 tokens (1.5B)11
4. Make effort an input and train each level
DeepSeek-V46; K35; Nemotron 3 Ultra8; L111; gpt-oss14; K2-V217
Users need different trade-offs, so train several operating points and let one model read the label.
These are layers, not rivals. V3.2 used position 1 and V4 moved to position 4; K3 uses position 2 to manufacture the levels position 4 exposes. For one operating point at lower cost, a relative gated reward term (1) or a gated budget with alternation (2) is the evidence-backed choice, after the Section 2.3 check on advantage normalization. For several operating points, position 4 is where the frontier reports converged, with 2 producing the lower levels. Position 3 is the right first step because it removes a bias for free, and it is not sufficient alone. The evidence gap is the finding: frontier reports give no ablation numbers for their length terms, and the small-scale papers that do (1.5B to 7B, 4K to 16K contexts) are not comparable to them. Settling it takes one base and one dataset with a relative penalty, a gated override and an effort expert, each swept over at least three budgets with seeds, scored on one token metric and on out-of-domain benchmarks.
What is changing
As of October 2026, effort is becoming a first-class API object. K3's chat template reserves four levels and supports a subset5; V4 ships three6; gpt-oss, K2-V2 and Nemotron converge on three14178. Effort conditioning is moving into distillation: K3 consolidates nine domain-by-effort experts by MOPD with the level as an input5, the subject of Module 22. Open items: no shared accuracy-per-token metric; whether SFT-only effort (K2-V2) matches RL-trained effort (Nemotron) has no head-to-head; and whether Toggle's alternation helps outside the one model it was reported on. A bibliographic caution: the V4 report's arXiv id is 2606.19348 but the stamp on its PDF reads 26 Apr 20266, so cite it by title and id and do not date it from the id.
References
- paperKimi Team (Du, A., Gao, B., Xing, B., et al.). Kimi k1.5: Scaling Reinforcement Learning with LLMs. Moonshot AI, arXiv v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: length reward and warm-up (§2.3.3; printed equation checked on p. 6), policy-optimization gradient (§2.3.2, Eq. 3), long2short methods (§2.4), long2short results (§3.4, Fig. 7), abstract short-CoT numbers.
- paperYu, Q., Zhang, Z., Zhu, R., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. ByteDance Seed and Tsinghua AIR, arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: token-level loss (§3.3), soft overlong punishment (§3.4, Eq. 13), training settings (§4.1).
- paperKimi Team (Bai, Y., Bao, Y., Charles, Y., et al.). Kimi K2: Open Agentic Intelligence. Moonshot AI, arXiv v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: algorithm foundation and budget control (§3.2.3).
- paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.). Kimi K2.5: Visual Agentic Intelligence. Moonshot AI, arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: Toggle definition, budget percentile and results (§4.4.2, Eq. 2, Fig. 5 on p. 10; per-benchmark values read from the rendered figure).
- paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.). Kimi K3: Open Frontier Intelligence. Moonshot AI, arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: reasoning-effort RL (§4.1.2), effort-conditioned MOPD reward (§4.1.3, Eq. 15), cost claim (§6.4), effort option message and KV-cache placement (Appendix F).
- paperDeepSeek-AI (Xu, A., Lin, B., Xue, B., et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348v1 (PDF stamp 26 Apr 2026). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: reasoning-effort modes (§5.1.1, Tables 2 and 3), evaluation context windows (§5.3.1), Table 7 (p. 39), Figure 10 (p. 41; token positions read off the axis).
- paperYang, A., Li, A., Yang, B., et al. (Qwen Team). Qwen3 Technical Report. Alibaba, arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: Thinking Mode Fusion and thinking budget (§4.3, Table 9), Table 22 stage results and Figure 2 (§4.7).
- paperNVIDIA (Blakeman, A., Thomas, A., et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: effort-control SFT data (§3.1.1), reasoning modes and medium-effort RLVR share (§3.5), Figure 11 (p. 31; points read off the plot).
- paperLiu, Z., Chen, C., Li, W., et al. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025, arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: Dr. GRPO's loss-side length fix and its 43.3% AIME 2024 result (abstract, §3.2).
- paperArora, D., Zanette, A. Training Language Models to Reason Efficiently. NeurIPS 2025, arXiv v4, 3 Nov 2025. https://arxiv.org/abs/2502.04463. Accessed 2026-10-27. Used for: reward form (§4, Eqs. 4 and 5), results (§5.3.1), advantage-normalization ablation (§5.4.1), limitations (§6). Added by the writer; full text read.
- paperAggarwal, P., Welleck, S. L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. COLM 2025, arXiv v2, 3 Oct 2025. https://arxiv.org/abs/2503.04697. Accessed 2026-10-27. Used for: LCPO rewards (§3, Eqs. 1 and 2), setup (§4), results (§5, Figs. 2, 4, 11), additive-reward ablation (App. A.7). Added by the writer; full text read.
- paperMuennighoff, N., Yang, Z., Shi, W., et al. s1: Simple test-time scaling. arXiv 2501.19393, 2025. https://arxiv.org/abs/2501.19393. Accessed 2026-10-27. Used for: budget forcing (abstract only). Added by the writer.
- paperChen, X., Xu, J., Liang, T., et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. arXiv 2412.21187, Dec 2024. https://arxiv.org/abs/2412.21187. Accessed 2026-10-27. Used for: the overthinking framing (abstract only; the body was not read). Added by the writer.
- officialOpenAI. gpt-oss-120b & gpt-oss-20b Model Card. arXiv v1, 5 Aug 2025. https://arxiv.org/abs/2508.10925. Accessed 2026-10-26. Used for: variable-effort levels set by system prompt (§2.5.2) and accuracy against CoT length (Fig. 3). Added by the writer.
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: length penalty in the reward and the Speciale variant (§3), advantage without std division (§3.1), Table 3 accuracy and output tokens (§4.2), token-efficiency statements (§4.2, §5).
- officialHugging Face. TRL documentation: GRPO Trainer. main branch. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-26. Used for:
loss_typeoptions ("dapo" default, "dr_grpo") and thescale_rewardsdefault ("group"); text extracted by a fetch tool, not machine-verified. - paperK2 Team, Institute of Foundation Models, MBZUAI. K2-V2: A 360-Open, Reasoning-Enhanced LLM. IFM Technical Report, arXiv 2512.06201, 2 Dec 2025. https://arxiv.org/abs/2512.06201. Accessed 2026-10-27. Used for: SFT-only three reasoning efforts, special tokens and generation-length spread (§6.1), TxT360-3efforts data (§6.2). Added by the writer; the relevant sections were read.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: the survey of toggleable-reasoning practice (§7.3.3); a secondary source, not used for any number.