Reinforcement Learning for LLM Post-Training · Module 23

Length and Reasoning-Effort Control: Length Rewards, Token Budgets, Toggle, and Effort-Conditioned Experts

Response length is the one quantity that every reasoning-RL run sees growing and every inference bill pays for. This module reads what the reports actually do about it: the within-group length reward in Kimi k1.5, the per-problem budget overrides in K2.5 and K3, DeepSeek-V4's three trained effort modes, Qwen3's mode fusion, and the small-scale reward designs (Arora and Zanette, L1) that expose the arithmetic. After it you can pick a reward-side, loss-side, data-side or expert-side control for a target length-accuracy trade-off, and say what each one does to accuracy, tokens and decode cost.

Course 50 min read 18 sources As of 2026-10-27 Prerequisites: Modules 11, 13, 22
The RL loop in blue with the six places where reports attach a length or effort control in orange, and a mode-level strip outside the loop The loop Promptand training data RolloutsG samples, decode Rewardverifier or judge Advantagerelative to group Loss, updatetoken weighting Controls Data and modeeffort label in promptshort-trace SFT dataK3, V4, Nemotron, k1.5 Cut when servingstop-thinking stringnot trainedQwen3 budget, s1 Reward termslength term: k1.5,A&Z, L1, DAPO softoverride: K2, Toggle,K3 Std divisionrescales a smalllength termSection 2.3 Loss normalizer1/|o| or constantor token meanDr. GRPO, DAPO Outside the loop Train one expert per effort level, then consolidate into one model K3: 3 domains x 3 efforts = 9 experts; V4: 3 modes; consolidation by distillation is Module 22
Figure 1. The RL loop (blue) and where the reports attach a length or effort control (orange). The reward term (Section 2) and the budget override (Section 3) act on the same box but fail differently; the standard-deviation division (Section 2.3) can silently erase a small length term; the loss normalizer is Module 11's subject; serving-time truncation (dashed) is not trained. Reports: Kimi k1.51, DAPO2, Kimi K23, K2.54, K35, DeepSeek-V46, Qwen37, Nemotron 3 Ultra8, Dr. GRPO9, Arora and Zanette10, L111, s112.

1. Why a length control exists, and where it can act

1.1 The bill for a long chain

Outcome-reward RL pays for correct answers, not for brevity, and every report that watches the length curve sees it rise. Kimi k1.5 names the effect: "the model's response length significantly increases during RL training"; this "leads to better performance", but "an excessively lengthy reasoning process is costly during training and inference, and overthinking is often not preferred by humans"1 (§2.3.3). K2 adds the allocation argument: longer responses let the model spend test-time compute on hard reasoning, but "the benefits often do not justify its inference cost in non-reasoning domains"3 (§3.2.3). Chen et al. framed the same waste earlier as misallocation, reasoning models spending "excessive computational resources" on "simple problems with minimal benefit" (abstract only; I did not read the body)13. Why the length curve rises, and how to tell growth from runaway repetition, is Module 13 §5; the loss-normalizer cause is Module 11 §2. This module is about what the reports do on purpose.

Two different goals hide behind the phrase "length control". The first is efficiency: remove tokens the answer did not need while keeping the ceiling on hard problems. Arora and Zanette's design targets this, and the same coefficient removes 83% of tokens on GSM8K but 27% on AIME 2024 (7B model, α = 0.2)10 (§5.3.1). The second is selectability: the user, not the training run, chooses how much to think. DeepSeek-V4 trains three modes6, K3 trains low, high and max experts5, and gpt-oss exposes low, medium and high through the system prompt14. The goals interact: Toggle exists because efficiency training under rigid budgets produced "a length-overfitting phenomenon": models that "cannot effectively leverage additional inference-time tokens"4 (§4.4.2).

1.2 Five places a control can act

Table 1 sorts the controls by where they act. Read it along two axes. A penalty teaches the model that long is bad, so the trade-off is fixed at training time by a coefficient; an input (an effort label, a prompt-given target) teaches the model to condition on a request, so the trade-off is chosen per request. And a relative term (k1.5, Arora and Zanette) compares a response with its own group and so adapts to difficulty, while an absolute term (DAPO's 16,384 tokens, L1's target, a K3 budget) compares it with a number someone chose.

Table 1. Length and effort controls by where they act. Effect sizes are the authors' own and are not comparable across rows (different models, benchmarks and token measures; Section 7).
Where it actsMechanismReportsReported effectCaveat
Serving time (not trained)Stop thinking at a threshold and append a closing stringQwen37, s1 budget forcing12Qwen3: smooth gains with budget (Fig. 2)7; s1: "Wait" appended, AIME24 50% to 57%12Truncates mid-reasoning; L1 reports 20 to 25 points absolute over s1 at 512 and 1,024 tokens (1.5B)11
Reward: length termRelative or absolute length added to, or multiplied into, the correctness rewardk1.51, Arora and Zanette10, L111, DAPO soft2, V3.215A&Z: tokens -36% for accuracy -2.2% on MATH (7B, α = 0.1)10Coefficient to tune; interacts with advantage normalization (Section 2.3)
Reward: budget overrideToken budget per problem or task; an over-budget response is truncated, zeroed or set to -1K23, K2.5 Toggle4, K35Toggle: output tokens -25% to -30%, "negligible" effect4Rigid budgets overfit length; Toggle's ρ, λ and m are not in the text read
Loss normalizerDivide by a constant or a batch token total instead of 1/|oi|Dr. GRPO9, DAPO2, TRL16Removes one source of growth; gives no target lengthDemonstrated at 1.5B to 3B (Module 11 §2.3)
DataTrain on shortest correct samples, length-ranked DPO pairs, truncated traces or effort-labelled SFT datak1.5 long2short1, Nemotron 3 Ultra8, K2-V217k1.5-short: 60.8 on AIME at 3,272 tokens on average1Needs a long-CoT teacher; no RL signal on length
Mode or expertEffort label as an input; one expert per level, then consolidateV46, K35, Nemotron 3 Ultra8, gpt-oss14Nemotron medium effort: about 2.5x fewer tokens for about 7% lower accuracy8gpt-oss's training method is one sentence long14

2. Reward-side controls: a length term inside the verifier reward

2.1 Kimi k1.5: the length reward is relative to the group

For each problem, k1.5 samples \(k\) responses, takes the shortest and longest lengths in that group, \(\ell_{\min}\) and \(\ell_{\max}\), and defines the length reward from where a response sits between them. If all lengths are equal the length reward is zero for every response; otherwise1 (§2.3.3, printed equation on p. 6; I checked the form against the rendered page)

$$\lambda_i=\tfrac12-\frac{\ell_i-\ell_{\min}}{\ell_{\max}-\ell_{\min}},\qquad r^{\text{len}}_i=\begin{cases}\lambda_i & y_i\text{ correct}\\ \min(0,\lambda_i) & y_i\text{ wrong}\end{cases}$$

The authors' summary is that the term promotes shorter responses and penalizes longer ones among the correct, "while explicitly penalizing long responses with incorrect answers", and that it is "added to the original reward with a weighting parameter"1. The weight is not given in the text I read. They also report that "length penalty may slow down training during the initial phases", so they warm it up: standard policy optimization with no length term first, then a constant length penalty for the rest of training1.

Kimi k1.5 length reward as a function of relative length within the group, for correct and wrong responses, with the eight toy responses of Table 2 marked +0.5 +0.25 0 −0.25 −0.5 00.250.50.751 shortest correct: +0.5 longest correct: −0.5 wrong and short: 0 wrong and long: penalized correct response wrong response relative length (len − min) / (max − min), within the k samples of one problem length reward, before weight
Figure 2. The k1.5 length reward plotted from the printed equation1 (§2.3.3). Correct responses get +0.5 for the shortest in the group down to -0.5 for the longest; wrong responses get nothing for being short and a penalty only in the longer half (shaded). Dots are the eight toy responses of Table 2.

Three properties follow from the formula, none of them stated in the report (derived).

  • Position, not length. The term depends only on where a response falls between the group's shortest and longest, so a group spanning 2,100 to 8,100 tokens and a group spanning 21,000 to 81,000 produce identical length rewards for identical positions. It adapts to problem difficulty automatically, but it has no absolute target: on an easy problem where every sample is 400 to 420 tokens, a 20-token difference swings the reward by a full 0.5.
  • No bonus for short and wrong. The shorter half of the wrong responses gets exactly 0, so the term cannot reward a model for stopping early with a wrong answer.
  • A weight ceiling. The total is \(r+w\,r^{\text{len}}\). The longest correct response scores \(1-0.5w\); the shorter-half wrong responses score 0. The ordering "any correct answer beats a short wrong one" therefore breaks when \(w>2\). The report does not give \(w\), so this is a bound on what \(w\) could be, not a finding.
Table 2. A toy group of eight samples (illustrative lengths) pushed through the k1.5 formula. The total column uses \(w=1\), an assumption, since the report gives no weight.
SampleLength (tokens)Correct?Relative lengthλLength rewardTotal, w = 1
12,100yes0.000+0.500+0.5001.500
22,400yes0.050+0.450+0.4501.450
33,000yes0.150+0.350+0.3501.350
43,600no0.250+0.2500.0000.000
54,400yes0.383+0.117+0.1171.117
65,200no0.517−0.017−0.017−0.017
76,800no0.783−0.283−0.283−0.283
88,100yes1.000−0.500−0.5000.500

Read the table the way the optimizer does. The mean of the eight totals is 0.702 (derived). Samples 1 to 3 and 5 sit above it and gain advantage. Sample 8 is correct, yet its total of 0.500 is below the mean, so it receives a negative advantage: at \(w=1\) the longest correct response is pushed down along with the wrong ones. The gap between samples 1 and 8, both correct, is 1.0 of reward, the same size as the correctness reward itself. That is the lever and also the risk: near \(w=1\) the length term competes with correctness on equal terms inside every group where length varies.

DAPO's overlong shaping is a different object: an absolute cap-handling device with a 4,096-token ramp before a 16,384-token expected maximum, derived and plotted in Module 11 §5, meant to stop truncated-but-sound reasoning from being scored as wrong rather than to steer length inside the cap2 (§3.4). The data-side companions of k1.5's length reward are in Section 6.2.

2.2 Arora and Zanette: a bounded factor multiplied into correctness

Arora and Zanette keep the correctness indicator and multiply a bounded, per-prompt-normalized length factor into it10 (§4, Eqs. 4 and 5):

$$R(x,y)=\mathbb{1}\{y=y^*\}\,\bigl(1-\alpha\,\sigma(z)\bigr),\qquad z=\frac{\mathrm{LEN}(y)-\mathrm{MEAN}(x)}{\mathrm{STD}(x)}$$

\(\mathrm{MEAN}(x)\) and \(\mathrm{STD}(x)\) are the mean and standard deviation of the lengths of the correct responses to that prompt, estimated from the rollouts during training; \(\sigma\) is the sigmoid; \(\alpha\in[0,1)\) is the one knob. The authors give the design reasons: per-prompt normalization means long chains on hard problems are not penalized relative to short chains on easy ones, the sigmoid keeps the reward bounded in \([1-\alpha,1]\) "even for abnormally long or short generations", and "correct responses, even if long, are always preferred to incorrect ones"10 (§4). Compared with k1.5 the two are alike in being relative to the group (a z-score has the same spread whether the group spans 20 tokens or 20,000), and unlike in two ways: this form is multiplicative, so a wrong response is always 0 and never gains or loses anything for its length, and it is bounded, so the ordering problem of Section 2.1 cannot occur.

They train DeepSeek-R1-Distill-Qwen-1.5B and 7B with \(\alpha\in\{0,0.05,0.1,0.2,0.4\}\). On the 7B model, \(\alpha=0.1\) cuts MATH chains by 36% (about 4,000 to about 2,600 tokens) for a 2.2% accuracy loss, and \(\alpha=0.2\) cuts AIME 2024 tokens by 27% (about 13,000 to about 9,000) "while incurring only a 4% accuracy drop"; the same \(\alpha=0.2\) saves 83% on GSM8K10 (§5.3.1). They read the gap between 27% and 83% as easy problems having a larger relative spread, so reasoning models are "more wasteful on easier problems". At equal token usage the online-RL models beat SFT and DPO baselines (their Fig. 1), and the single coefficient gives a family of models, one per \(\alpha\)10. What \(\alpha\) does not do is hit a target length: the authors list this as a limitation and point to L1 for that case10 (§6).

2.3 Standard-deviation division can erase the coefficient

Arora and Zanette run an ablation worth deriving, because it applies to any small additive or multiplicative length term inside a GRPO-style group. Take a prompt where all \(k\) responses are correct, so every reward is \(1-\alpha f_i\) with \(f_i\in[0,1]\) spread evenly, which is the paper's assumption (uniform on \([1-\alpha,1]\)). The mean is \(1-\alpha/2\) and the standard deviation is \(\alpha/\sqrt{12}\). The advantage of the shortest correct response, with \(r=1\), is

$$A_{\text{short}}=\frac{1-(1-\alpha/2)}{\alpha/\sqrt{12}}=\frac{\alpha/2}{\alpha/\sqrt{12}}=\frac{\sqrt{12}}{2}=\sqrt{3}\approx 1.73,$$

and the longest gets \(-\sqrt{3}\): "their difference is independent of α"10 (§5.4.1). Without the division, with a mean baseline only, the endpoints are \(\pm\alpha/2\): 0.025 at \(\alpha=0.05\), 0.1 at \(\alpha=0.2\), 0.2 at \(\alpha=0.4\) (derived). For a finite group of \(k=8\) evenly spaced rewards the endpoint is 1.43 with the sample (\(n-1\)) standard deviation and 1.53 with the population one; \(\sqrt3\) is the continuum limit (derived). For scale, a group with four correct and four wrong answers gives a correct response an advantage of about +1 under the same division (mean 0.5, standard deviation about 0.5; derived). So in a group where everything is correct, the length term hands out advantages larger than the correctness signal in a half-right group, whatever \(\alpha\) is. The authors observe that the resulting length decrease is "generally too substantial for the model to absorb, leading to a sharp drop in accuracy", and they do not normalize advantages10 (§5.4.1, Fig. 4).

Advantage of the shortest correct response in an all-correct group, with and without division by the group standard deviation, for three values of the length coefficient alpha 0.5 1.0 1.5 0 0.0251.43 0.11.43 0.21.43 α = 0.05α = 0.2α = 0.4 mean baseline only divided by std, k = 8 1.0: correct answer at p = 0.5 advantage of shortest correct
Figure 3. Derived from Arora and Zanette's argument10 (§5.4.1): in a group where every response is correct and rewards are \(1-\alpha f_i\) with \(f_i\) evenly spread over \([0,1]\), the shortest response's advantage is \(\alpha/2\) with a mean baseline and about 1.43 (sample std, \(k=8\); \(\sqrt3\) in the limit) once divided by the group standard deviation, whatever \(\alpha\) is. The dashed line is the advantage of a correct answer in a half-right group.

One limit on the derivation, from me and not from the paper: it assumes the length term is the only source of reward variance in the group. In groups that mix correct and wrong answers the correctness variance enters the standard deviation and the length term's share of the advantage shrinks. I have not found a report that measures where the crossover lies.

2.4 The reward forms side by side

Table 3. Length-related reward forms in the reports read. "Long correct below wrong?" asks whether a correct response that is long enough can score below a wrong one; entries marked "derived" are my reading of the printed form.
FormLength quantityCombinationRange of the length effectLong correct below wrong?
k1.51Position between group min and maxAdded, weight not given-0.5 to +0.5 before weightYes if the weight exceeds 2 (derived)
DAPO soft overlong2Absolute length beyond 16,384 tokens, 4,096-token rampAdded0 to -1No: a truncated correct answer ties a wrong answer inside the cap at 0 (derived)
Arora and Zanette10z-score among correct responsesMultiplied into correctnessCorrect responses in [1-α, 1]No (α < 1)
L1-Exact11Distance to a target in the promptAdded, α = 0.0003Unbounded belowYes beyond 1/α = 3,333 tokens off target (derived)
L1-Max11Excess over a target in the promptMultiplied, δ = 0.5[0, 1]No: ties at 0 (derived)
K2 budget3Per-sample budget by task typeTruncate and penalizeNot statedNot stated
K2.5 Toggle4Budget from a percentile of correct lengthsReward zeroed over budget{0, r}No: ties at 0 (derived)
K35Budget τ·b0 per problemReward overridden with -1-1 over budgetYes if the task reward is non-negative; its range is not stated in the text read (derived)

The frontier reports (K2, K2.5, K3, V4) use hard or gated budgets and say little about the numerical form, while the small-scale papers (Arora and Zanette, L1) publish the full reward and its ablations. Section 7 returns to that asymmetry.

3. Budget overrides: K2, Toggle and K3

3.1 K2: a per-sample budget by task type

K2 adopts k1.5's policy-optimization algorithm as its foundation3 (§3.2.3) and adds a budget rule: "we enforce a per-sample maximum token budget throughout RL training, where the budget is determined based on the type of task. Responses that exceed this token budget are truncated and assigned a penalty". The report says this "significantly enhances the model's token efficiency" and gives no budgets, penalty size or numbers in the text I read3. The structural point is that the budget belongs to the task type and is set by the lab. The two later designs replace it with a budget derived from the model itself: a percentile of its own correct lengths (Toggle) or a multiple of the cold-start model's per-problem budget (K3).

3.2 K2.5 Toggle: alternate the constraint on and off

Toggle starts from the failure the K2 recipe produced. Models "trained under rigid budget constraints often fail to generalize to higher compute scales" and default to "truncated reasoning patterns"4 (§4.4.2). The fix is to stop applying the constraint all the time. For learning iteration \(t\), with \(K\) rollouts per problem, the reward is4

$$\tilde r(x,y_i)=\begin{cases} r(x,y_i)\cdot\mathbb{1}\!\left[\bar r(x)<\lambda\ \ \text{or}\ \ |y_i|\le \mathrm{budget}(x)\right] & \lfloor t/m\rfloor \bmod 2=0\ \ (\text{Phase 0})\\[2pt] r(x,y_i) & \lfloor t/m\rfloor \bmod 2=1\ \ (\text{Phase 1})\end{cases}\qquad \mathrm{budget}(x)=\mathrm{Percentile}\bigl(\{|y_j|: r(x,y_j)=1\},\,\rho\bigr)$$

where \(\bar r(x)=\frac1K\sum_i r(x,y_i)\) is the problem's mean accuracy and the percentile is over the lengths of the correct responses. The printed indicator carries the index \(y_i\) inside a sum; I read it per response. Phase 0 is budget-limited, Phase 1 is the standard objective, and they swap every \(m\) iterations. Four properties follow.

  • A quality gate. The budget binds only on problems whose mean accuracy is already at least \(\lambda\); the authors say this prevents "a premature sacrifice of quality for efficiency"4. Below the threshold the problem is trained on correctness alone.
  • A fixed budget. It is "estimated once at the beginning of training and remains fixed thereafter"4, so the target does not ratchet downward as the policy shortens.
  • One-sided. Over-budget responses get \(\tilde r=0\). For a binary reward that is the same as being wrong, and wrong responses are untouched, so unlike k1.5 nothing pushes long wrong answers further down (derived).
  • Half the iterations. Phase 1 restores the unconstrained objective, which "encourag[es] the model to leverage computation for better inference-time scaling"4. The authors call the whole scheme "a stochastic alternating optimization for a bi-objective problem".

The values of \(\rho\), \(\lambda\) and \(m\) are not given in the text I read. A toy shows what \(\rho\) controls. Suppose a problem's five correct responses have lengths 3.2k, 4.1k, 4.8k, 6.0k and 9.5k tokens (illustrative). With linear-interpolation percentiles, \(\rho=50\) gives a budget of 4.8k and \(\rho=75\) gives 6.0k (derived). At \(\rho=50\), a budget estimated from the starting policy puts two of the five correct responses over budget in the first Phase 0, and at \(\rho=75\) one of five. A median budget therefore starts by zeroing roughly half of what the model currently gets right on that problem, on every problem where the quality gate is open.

Toggle schedule: Phase 0 and Phase 1 alternate every m iterations; below, a toy problem whose correct responses are checked against a percentile budget in Phase 0 Schedule over learning iterations t Phase 0budget-limited Phase 1standard Phase 0budget-limited Phase 1standard Phase 0budget-limited Phase 1standard 0m2m3m4m5m6m phase = floor(t / m) mod 2; half of all iterations are budget-limited One problem, Phase 0 (toy lengths) 2k4k6k8k10k budget(x) = 50th percentile = 4.8k keep r (within budget) reward zeroed (over budget) tokens in a correct response; gate: the constraint applies only if the problem's mean accuracy is at least lambda budget is estimated once at the start of training and then held fixed
Figure 4. Toggle's alternation and its budget4 (§4.4.2, Eq. 2 and the reward definition). The schedule and the percentile rule are from the report; the five lengths and the choice \(\rho=50\) are a toy of mine, since the report's \(\rho\), \(\lambda\) and \(m\) are not stated in the text read.

What it bought. The report states that Toggle "decreases output tokens by 25∼30% with a negligible impact on performance", that redundant patterns such as repeated verification and mechanical calculation decrease substantially, and that it generalizes across domains: trained only on mathematics and programming, it still cut tokens on GPQA and MMLU-Pro "with only marginal degradation"4 (§4.4.2, Fig. 5; evaluated on K2 Thinking). Figure 5 plots the per-benchmark values as printed on the radar charts; I read them from the rendered page, because the text gives only the average.

Per-benchmark change in output tokens and in accuracy for Kimi K2 Thinking after Toggle, as labelled in the K2.5 report Output tokens, after minus beforeAccuracy, percentage points −9,000−4,5000 −20+2 AIME 2025−6,179 HMMT Feb 2025−7,967 HMMT Nov 2025−8,127 GPQA-Diamond−4,912 MMLU-Pro−817 LiveCodeBench v6−745 Overall−4,791 +1.1 +0.6 +0.8 −1.0 −2.0 +2.2 +0.3
Figure 5. Toggle on Kimi K2 Thinking: change in output tokens (left, cyan) and accuracy (right; green improved, red degraded) per benchmark, as labelled in Figure 5 of the K2.5 report4 (§4.4.2). All seven token counts fall; five accuracies rise and two fall, the two on GPQA-Diamond and MMLU-Pro, the benchmarks outside the math-and-code training mix. Baseline token counts are not printed (the radar axes are bucketed, e.g. 20K-35K for AIME 2025), so the 25% to 30% average cannot be reproduced from these deltas. Values read from the rendered figure.

Absolute deltas rank benchmarks by length, not by how much Toggle helped, since baselines differ by an order of magnitude. The report presents one run per benchmark and no variance in the text I read, so a 0.3-point overall gain and a 1.0-point GPQA loss are within what the figure cannot distinguish.

3.3 Kimi K3: a per-problem budget, annealed into effort levels

K3 turns the same device into a family of models. Each problem \(x\) carries a token budget \(b_0(x)\) "estimated from the cold-start model", and any trajectory whose total exceeds a scaled threshold has its task reward overridden5 (§4.1.2, "Reasoning Effort RL"):

$$\tilde r(x,y)=\begin{cases}-1 & T(y)>\tau\,b_0(x)\\ r(x,y) & \text{otherwise}\end{cases}$$

For general tasks \(T(y)\) counts thinking tokens; for agentic tasks it counts "cumulative output tokens, including both reasoning traces and tool-call arguments"5. Training "follows a stage-wise curriculum over the budget multiplier \(\tau\)": a max-budget variant first, "with a relatively large \(\tau\), while still capping the maximum budget to suppress excessive overthinking", then \(\tau\) annealed to smaller values to obtain the high- and low-effort experts, with "the adjustment of \(\tau\) ... configured per domain under human-in-the-loop guidance"5. Three domains (general tasks, general agents, coding agents) crossed with three efforts (low, high, max) give nine experts5 (§4.1.2). The values of \(\tau\), the estimator for \(b_0\) and the task-reward range are not stated in the text I read.

K3 effort RL: the budget multiplier tau is annealed through three stages to produce max, high and low effort experts, which then feed one student that sees the effort level in its prompt Per-problem budget, tau annealed (schematic) 1. max-budget expert tau_max * b0 2. high-effort expert tau_high * b0 3. low-effort expert tau_low * b0 within budget: task reward r −1−1−1 anneal tokens T(y), in multiples of b0(x); tau values are not given in the report Consolidation generalagentscoding lowhighmax 9 experts One studenteffort e in its promptteacher (d, e) per token MOPD, Module 22
Figure 6. K3's effort recipe5 (§4.1.2, §4.1.3). Left: the same per-problem budget \(b_0(x)\) with a multiplier annealed from the max-budget variant to the high- and low-effort experts; the lane lengths are schematic because no \(\tau\) value is reported. Right: nine domain-by-effort experts consolidated by multi-teacher on-policy distillation, in which the student sees the effort level \(e\) and the teacher is the expert for (domain, effort).

Four comparisons with Toggle, all derived from the two printed forms. First, K3 does not alternate in time; it separates in space. The max-budget expert plays the role of Toggle's Phase 1 and is trained as its own model, and the lower-effort experts are annealed from it. Second, the override value differs: Toggle zeroes, K3 sets \(-1\), which is strictly worse than a wrong answer if the task reward is non-negative (its range is not stated). Third, both budgets are set before RL begins, K3's from the cold-start model (how \(b_0\) is estimated is not stated) and Toggle's from a percentile of correct lengths, so neither ratchets down as the policy shortens. Fourth, K3 counts tool-call arguments as output tokens for agentic tasks, so a long-horizon agent is charged for what it writes into tool calls but not for tool results it reads.

The effort level reaches the final model through distillation. K3's per-token reward is \(\mathrm{clip}\bigl(\mathrm{sg}[\log \pi^{(d,e)}_{\text{teacher}}(y_t\mid x,y_{<t})-\log \pi_\theta(y_t\mid e,x,y_{<t})],-R_{\max},R_{\max}\bigr)\): the student is conditioned on \(e\), the teacher is selected by domain \(d\) and effort \(e\), and trajectories from all nine experts feed both SFT and the distillation stage5 (§4.1.2, §4.1.3, Eq. 15). The distillation mechanics belong to Module 22. The report's only effort-specific result in the text I read is a cost claim: on Kimi Code Bench 2.0 "at high effort it already matches Claude Opus 4.8's maximum-effort score at roughly one third of the cost", with costs measured internally and other models run through Claude Code5 (§6.4). K3 applies the same budget form to its judge as a verbosity guard; that guard and its failure modes are in Module 16 §6.4 and Module 19 §4.1.

4. Effort as an input: prompt-given targets, trained modes, and what the reports disclose

4.1 L1: the target length is in the prompt

L1 makes length an explicit input. Each training prompt gets the suffix "Think for \(n_{\text{gold}}\) tokens." with \(n_{\text{gold}}\) drawn uniformly from 100 to 4,000, and GRPO optimizes11 (§3, Eqs. 1 and 2)

$$r_{\text{Exact}}=\mathbb{1}[y=y^*]-\alpha\,|n_{\text{gold}}-n_y|,\qquad r_{\text{Max}}=\mathbb{1}[y=y^*]\cdot\mathrm{clip}\bigl(\alpha\,(n_{\text{gold}}-n_y)+\delta,\,0,\,1\bigr)$$

with \(\alpha=0.0003\) in Eq. 1 and \(\delta=0.5\) in Eq. 2 (the text I read gives no separate \(\alpha\) for Eq. 2). The base is DeepScaleR-1.5B-Preview, trained 700 steps for L1-Exact and 120 more for L1-Max, with a 4K-token training context and 8K at evaluation11 (§4). The arithmetic of Eq. 1 is simple. Each token of deviation costs 0.0003, so a response 1,000 tokens off target loses 0.3; a correct 2,548-token answer to a 2,048-token target scores 0.85; and a correct answer more than \(1/\alpha=3{,}333\) tokens off scores below a wrong answer of exactly the right length (derived). Since training targets reach 4,000, the penalty can reach about 1.17 (derived), so Eq. 1 has the ordering problem of Table 3 in principle. The multiplicative Eq. 2 does not: a correct answer far over budget ties a wrong one at 0.

The results are about control, not just savings. Against s1's budget forcing, which truncates generation and inserts "Final Answer", L1 gains "over 100-150% relative and 20-25% absolute" at 512 and 1,024 tokens, because "S1 often truncates mid-reasoning"; its mean length error on math is "close to 3%"; and L1-Max matches the unconstrained DeepScaleR-4K by adapting to difficulty, often with up to 2x fewer tokens than L1-Exact11 (§5, Figs. 2, 4, 11). One ablation carries a design lesson. Replacing the multiplicative gate with an additive length penalty gave "a 0% budget violation rate" because "the model collapses to generating extremely short CoTs"11 (App. A.7): an additive term can be satisfied by abandoning correctness, a gated one cannot. The scale caveat is large: a 1.5B model trained at 4K tokens, against V4's 128K and 384K evaluation windows below.

4.2 DeepSeek-V3.2, Speciale and V4's three modes

DeepSeek-V3.2's reasoning and agent rewards combine a rule-based outcome reward, a length penalty and a language-consistency reward, with no functional form given15 (§3). Its high-compute variant V3.2-Speciale "was trained exclusively on reasoning data with a reduced length penalty during RL", plus DeepSeekMath-V2's proof data and reward15 (§3). The report's Table 3 lists accuracy and output tokens for both. Across its eight rows Speciale spends 1.39x to 2.29x the tokens of V3.2 (derived): AIME 2025 23k against 16k tokens for 96.0 against 93.1; GPQA-Diamond 16k against 7k for 85.7 against 82.4; HLE 35k against 21k for 30.6 against 25.115 (Table 3). It is not a clean penalty ablation, since the training data differ too. The authors say plainly that Speciale's token efficiency "remains significantly inferior" to Gemini-3.0-Pro's, that the official V3.2 "imposed stricter token constraints during the training" to balance performance and cost, and that its numbers are "constrained by a length constraint reward model"15 (§4.1, §4.2).

V4 makes the two V3.2 operating points a product. "DeepSeek-V4-Pro and DeepSeek-V4-Flash both support three specific reasoning effort modes. For each mode, we apply distinct length penalties and context windows during RL training", and the specialists are trained "under divergent RL configurations"6 (§5.1.1). The modes are Non-think (response format "</think> summary"), Think High, and Think Max, which adds a system-prompt instruction beginning "Reasoning Effort: Absolute maximum with no shortcuts permitted."6 (Tables 2 and 3). Evaluation context windows are 8K, 128K and 384K tokens for the three modes, 16x and then 3x steps (derived)6 (§5.3.1). The report says Max "employs longer contexts and reduced length penalties in RL" and "outperforms the High mode on the most challenging tasks"6 (§5.3.2). No penalty values are given. Table 4 and Figure 7 show what the modes buy.

Table 4. DeepSeek-V4 by reasoning mode6. Accuracy rows are Table 7 of the report (Pass@1, % unless noted; p. 39). Token rows are my readings of the axis of Figure 10 (about ±2k), not printed values; "-" means not read or not run.
BenchmarkFlash Non-thinkFlash HighFlash MaxPro Non-thinkPro HighPro Max
HLE8.129.434.87.734.537.7
HLE total tokens, about-35k86k-35k53k
Terminal Bench 2.049.156.656.959.163.367.9
Terminal Bench total tokens, about37k47k57k28k36k50k
GPQA Diamond71.287.488.172.989.190.1
Apex1.019.133.00.427.438.3
SWE Verified (resolved)73.778.679.073.679.480.6
DeepSeek-V4 Flash and Pro accuracy on HLE and Terminal Bench 2.0 in Non-think, High and Max modes HLE (Pass@1, %)Terminal Bench 2.0 (accuracy, %; axis starts at 45) 20040 57.54570 Non-thinkHighMax Non-thinkHighMax 8.17.7 29.434.5 34.837.7 49.159.1 56.663.3 56.967.9 V4-Flash V4-Pro
Figure 7. Accuracy by effort mode, plotted from Table 7 of the V4 report6 (p. 39). On HLE the gain from High to Max is 5.4 points for Flash and 3.2 for Pro; on Terminal Bench 2.0 it is 0.3 for Flash and 4.6 for Pro (derived). The Non-think to High step dominates on HLE (21.3 and 26.8 points). Terminal Bench's axis starts at 45, so compare slopes, not heights above the baseline.

Combining Table 4's accuracy and token rows gives marginal returns (derived, with the token readings' uncertainty). On HLE, V4-Flash's Max mode spends about 2.4x High's tokens (86k against 35k) for 5.4 points, and V4-Pro's spends about 1.5x (53k against 35k) for 3.2 points. On Terminal Bench 2.0, Flash Max costs about 1.2x High's tokens for 0.3 points, while Pro Max costs about 1.4x for 4.6. The report's own reading of the same figure is that on HLE "DeepSeek-V4-Pro demonstrates higher token efficiency than DeepSeek-V3.2"6 (§5.3.2, Fig. 10). My cross-check on the readings: Speciale's HLE position in Figure 10 lands at about 35k, matching its 35k in the V3.2 report's Table 315.

4.3 Nemotron 3 Ultra: a medium-effort mode from 2.5% of the RL prompts

Nemotron 3 Ultra is "trained for three reasoning modes: reasoning-off, regular and medium-effort", and the last two can be combined with inference-time budget control8 (§3.5). Medium effort begins in SFT with two data components: samples generated by GPT-OSS-120B in its medium-effort mode on math, STEM and instruction-following prompts, and samples whose "reasoning traces are truncated to random reasoning budgets while the responses remain the same", with the closing </think> token masked from the loss in truncated samples8 (§3.1.1). RLVR then optimizes it: "Approximately 2.5% of the RLVR training prompts are in the medium-effort mode", covering math, STEM and coding, "and length-based adjustments are applied on the RL rewards for them"; the effect "generalize[s] to a variety of tasks beyond" those, and the mode is calibrated "by adjusting hyperparameters"8 (§3.5). The adjustment's form and size are not given. The headline is that medium effort "uses on average around 2.5X less tokens than the regular mode at the cost of approximately 7% drop in accuracy" on the Artificial Analysis Intelligence Index V4, averaged over its 10 tasks with token use measured relative to Qwen 3.5 397B8 (§3.5, Fig. 11).

Reading the plotted points (about ±0.03 on the token axis), regular Ultra sits near 1.06 and medium near 0.37, a ratio of about 2.9 rather than 2.5, and the index falls from about 49.8 to about 42.7, about 7 index points rather than 7% relative (derived from the figure). The report does not say how it averages, so the gap may be a mean of per-task ratios against a ratio of means. The design lesson is that a small, effort-labelled slice of RL prompts with a length adjustment on its rewards moved a whole mode, which is consistent with Toggle's domain generalization.

4.4 How the level reaches the model

Table 5. Interfaces for selecting effort and what each report says about training it.
ModelLevelsHow selectedHow trained, as disclosed
gpt-oss14low, medium, highSystem-prompt keyword such as "Reasoning: low""We train the models to support three reasoning levels" (§2.5.2); nothing more. Accuracy rises smoothly with average CoT length (Fig. 3)
K2-V217low, medium, highSpecial tokens <think_faster>, <think_fast>, <think> in the chat templateSFT only, 17.4B tokens per epoch for 3 epochs, data regenerated with GPT-OSS-120B at the three efforts (§6.1, §6.2)
Qwen37thinking, non-thinking, plus a budget/think and /no_think flags; budget by truncating and inserting a stop stringContinual SFT on both modes, then general RL; the budget "is not explicitly trained" (§4.3)
Kimi K35low, high, max (four reserved: low, medium, high, max)A "thinking-effort" option message stating the level in natural languagePer-effort RL experts with annealed budgets, then MOPD (§4.1.2, §4.1.3; App. F)
DeepSeek-V46Non-think, High, MaxResponse format by mode; a system-prompt instruction for MaxDistinct length penalties and context windows per mode in RL (§5.1.1)
Nemotron 3 Ultra8off, regular, mediumMode, optionally with inference-time budget controlSFT from GPT-OSS-120B medium traces and truncated traces; 2.5% of RLVR prompts with length-adjusted rewards (§3.1.1, §3.5)

K3 describes its option message as acting "as a generation-constraint instruction" that, "instead of modifying the generation prefix or exposing a token budget", states the level in natural language; because the pretrained model already follows such instructions, new options can be added "with little or no additional training"5 (App. F). Lambert's survey groups these under "toggleable reasoning with system prompt (or length control)" and says that for gpt-oss and K2-V2 "training methods for this type of behavior are not as well documented"18 (§7.3.3). K2-V2's report claims to be "the first open recipe" for three effort levels and shows the SFT-only route works at 70B17 (§6), which is a data-side result, not an RL one.

5. Qwen3: two modes in one model, and a budget that was never trained

Qwen3 is the report on the opposite design from K3 and V4: one set of weights, two modes, and a thinking budget that no objective ever optimized. The flagship's third stage, Thinking Mode Fusion, continues SFT from the reasoning-RL model to add non-thinking behavior7 (§4.3).

Stage 2thinking modelAIME'24 83.8 Stage 3: fusion SFT/think, /no_think flagsempty think block Stage 4: general RLThinkFollow 88.7to 98.9 Servinguser budgethalts thinking One request in thinking mode thinking tokens, up to the budget inserted stop stringthen </think> final answer user-defined threshold reached: halt the budget ability is "not explicitly trained"; it emerges from fusion
  1. Reasoning-RL model. Stage 2 is thinking-only; Qwen3-32B scores 83.8 on AIME'24 and 68.4 on LiveCodeBench v5 here7 (Table 22).
  2. Fusion SFT. The thinking data is rejection-sampled from the Stage 2 model itself so its performance is not eroded; the non-thinking data is curated across coding, math, instruction following, multilingual, creative writing, QA and role-play. A /think or /no_think flag in the user or system message selects the mode, a non-thinking sample keeps an empty thinking block, and in multi-turn chats the model follows the last flag7 (§4.3, Table 9).
  3. General RL. Mode-switching accuracy on the in-house ThinkFollow benchmark moves from 88.7 to 98.9 (+10.2), while thinking-mode AIME'24 slips to 81.4 and LiveCodeBench v5 to 65.77 (§4.7, Table 22).
  4. Budget at serving time. When thinking reaches a user-set threshold, the runtime halts it and inserts "Considering the limited time by the user, I have to give the solution based on the thinking directly now." followed by the closing tag; the model then answers from its partial reasoning7 (§4.3).

The costs and the gap are in Table 22, which tracks Qwen3-32B through the stages7 (§4.7). Thinking-mode AIME'24 goes 83.8, 81.9, 81.4 across Stages 2, 3 and 4, and LiveCodeBench v5 goes 68.4, 67.2, 65.7. The non-thinking mode of the same weights scores 31.0 and 31.3 at Stage 4, so the mode switch is worth about 50 points on AIME'24 and 34 on LiveCodeBench (derived). The authors conjecture the thinking-mode drop is "due to the model being trained on a broader range of general tasks" and choose to "accept this performance trade-off to enhance the model's overall versatility"7 (§4.7). They report "scalable and smooth performance improvements correlated to the allocated thinking budget" for Qwen3-235B-A22B on four benchmarks and expect gains beyond 32K tokens7 (Fig. 2).

The stop-string mechanism is the same family as s1's budget forcing, which "forcefully terminat[es] the model's thinking process" or lengthens it by appending "Wait"12. What fusion adds is that a truncated prefix is a state the model has seen. What no RL objective in the Qwen3 pipeline measures is the quality of the answer at a given budget, because the budget is imposed after training. That is the specific thing K3's annealed experts, V4's per-mode penalties and L1's prompt-given targets train directly, and L1's comparison against truncation at 512 and 1,024 tokens is evidence, at 1.5B scale only, that training at the target wins at small budgets11.

6. Loss-side and data-side controls

6.1 The loss normalizer: removing a push is not setting a target

Module 11 §2 derives why GRPO's per-response \(1/|o_i|\) makes shorter correct responses receive larger per-token updates and longer wrong ones smaller penalties, and Module 11 §2.3 gives Dr. GRPO's evidence, at 1.5B to 3B and 3,000-token caps. Dr. GRPO removes the length and standard-deviation normalizers and reports 43.3% on AIME 2024 with a 7B base9 (abstract, §3.2); DAPO averages over all tokens in the batch2 (§3.3); TRL defaults to loss_type="dapo" and offers "dr_grpo", which divides by a constant that "is recommended to be the maximum completion length"16.

This module's sources add one independent data point. Arora and Zanette's \(\alpha=0\) control, with no length penalty at all, still shortened responses on MATH and AIME. They traced it to the OpenRLHF implementation of RLOO, which normalizes the loss by response length, and "removing the length bias from the implementation leads to no change in the response length of the model after training"10 (§5.3.1). So a normalizer alone can move length, here toward shorter, with no reward term. What it cannot do is set a target: the fix removes a push and leaves the model to settle wherever the task and the cap put it. The reward-side and mode-side controls exist because a target is wanted. I found no report in the set that ablates a loss-side fix against a reward-side term at matched scale, so the two are not ranked by evidence.

6.2 Data: shorten the traces, then train on them

k1.5's long2short section is the data-side counterpart of its length reward. It lists four routes for moving a long-CoT model's strength into a shorter-output model1 (§2.4):

  • Model merging averages the weights of a long-CoT and a short-CoT model, "without training".
  • Shortest rejection sampling samples each question \(n=8\) times and keeps the shortest correct response for SFT.
  • DPO takes the shortest correct solution as the positive and treats longer responses as negatives, "including both wrong longer responses and correct longer responses (1.5 times longer than the chosen positive sample)".
  • Long2short RL picks the best token-efficiency trade-off from standard RL as the base, then applies the Section 2.1 length penalty with a "significantly" reduced maximum rollout length.

The report finds long2short RL "demonstrates the highest token efficiency" of the four, with k1.5-short reaching 60.8 on AIME 2024 (mean of 8 runs) at 3,272 tokens on average1 (§3.4, Fig. 7). Its abstract states short-CoT results of 60.8 on AIME, 94.6 on MATH-500 and 47.3 on LiveCodeBench, above GPT-4o and Claude Sonnet 3.5 "by a large margin (up to +550%)"1. The DPO route turns a known failure into the signal: Module 7 §4.3 covers how DPO drifts toward verbosity, and here the preference pairs are length-ranked on purpose, so the same sensitivity becomes the mechanism (derived).

The other data-side examples are SFT-only. Nemotron 3 Ultra's truncated-trace samples give the model the experience of answering from a partial chain, with the closing tag masked from the loss, which is the training-time analogue of Qwen3's serving-time stop string8 (§3.1.1). K2-V2 gets three effort levels from SFT alone, on data regenerated at three efforts by GPT-OSS-120B17 (§6.1). Both depend on a teacher that already behaves the way the student should, and neither puts a reward on length. They are the cheapest route when an effort-controlled teacher exists.

7. Reading the accuracy-versus-tokens evidence

7.1 One benchmark, several recipes

Almost every report states its result as a token reduction against its own baseline, and none shares a baseline with another. One benchmark gives a common axis. HLE appears in the V3.2 report's Table 3 with accuracy and output tokens for five models, and in V4's Table 7 and Figure 10 for four V4 configurations. Figure 9 puts them on one plot. Accuracies are as printed; token counts are exact (to the nearest 1k) for the V3.2-report models and read off Figure 10's axis for V4.

HLE accuracy against output tokens for GPT-5 High, Gemini 3.0 Pro, Kimi K2 Thinking, DeepSeek-V3.2, V3.2-Speciale and four DeepSeek-V4 configurations 2025303540 020k40k60k80k same score, about 3.5x the tokens GPT-5 High 26.3Gemini-3.0 Pro 37.7K2 Thinking 23.9 V3.2 25.1Speciale 30.6 V4-Flash High 29.4V4-Flash Max 34.8V4-Pro High 34.5V4-Pro Max 37.7 other labs (V3.2 Table 3) V3.2 and Speciale V4 (tokens read off plot) output tokens per problem (V4 positions read off Figure 10, about +-2k) HLE Pass@1, %
Figure 9. HLE accuracy against output tokens. Accuracy: V3.2 report Table 3 for GPT-5 High, Gemini-3.0 Pro, Kimi-K2 Thinking, V3.2 and Speciale15; V4 report Table 7 for the four V4 points6. Tokens: Table 3 (thousands) for the first five; Figure 10 axis for V4 (blue; Flash dashed). V4-Pro Max matches Gemini-3.0 Pro's 37.7 at about 53k tokens against 15k (derived), which the V3.2 authors flag as an open problem ("token efficiency remains a challenge")15.

Two things stand out. Inside a family the last effort level is the expensive one: V4-Flash Max spends about 2.4x High's tokens for 5.4 points, while V4-Pro High reaches 34.5 at about the token count of Speciale's 30.6, which is what the V4 report means by higher token efficiency. And across labs the token axis spreads by about 3.5x at a fixed score, a gap that no length term in this module is designed to close.

7.2 What each claim is measured against

Table 6. Headline trade-offs, with the baseline each is measured against. The numbers are the authors' own and are not comparable across rows.
ReportClaimed changeMeasured againstAccuracy changeLimits
K2.5 Toggle4Output tokens -25% to -30%K2 Thinking before Toggle; average over benchmarks"negligible"; -2.0 to +2.2 per benchmark (Fig. 5)One run; baselines not printed
Arora and Zanette10-36% MATH (α = 0.1); -27% AIME and -83% GSM8K (α = 0.2)R1-Distill-Qwen-7B-2.2% MATH; "only a 4% accuracy drop" AIME1.5B and 7B; approximate token counts
Nemotron 3 Ultra8About 2.5x fewer tokensRegular mode of the same modelAbout 7% (about 7 index points by the plot)10-task index; verbosity relative to another model
V3.2 to Speciale151.39x to 2.29x more tokensV3.2 Thinking+2.9 to +6.7 points on seven benchmarksData also differ (reasoning-only, proof data)
V4 Flash, High to Max6About 2.4x more tokens on HLEV4-Flash High+5.4 pointsTokens read off a plot
k1.5-short13,272 tokens on averageOther short-CoT models (Fig. 7)60.8 on AIME 2024A model comparison, not a before and after
L111Length error about 3%s1 budget forcing at 512 and 1,024 tokens+20 to +25 points absolute1.5B, 4K training context

The token measure itself changes row to row: output tokens (V3.2, Toggle), "total tokens" including tool traffic (V4's Terminal Bench panel), thinking tokens only for K3's general tasks and cumulative output including tool-call arguments for its agents5, and a verbosity ratio against a third model (Nemotron). Read the right-hand column first.

7.3 Choosing a control

Table 7. A selection guide built from the evidence above; the "why" column cites where the evidence is.
If you needStart withWhyWatch for
Stop unexplained length growthFix the loss normalizer, then monitor (Module 13 §5)One source of growth removed, no knob910No target length results
Fewer tokens on easy prompts, long chains on hard onesA relative, gated reward term, no standard-deviation division, warmed up after standard RL83% against 27% saving at one α10; warm-up1Section 2.3 identity; coefficient tuning
A hard cap without length overfittingA budget with a quality gate, alternated with an unconstrained phaseToggle's design response to rigid budgets4ρ, λ, m are undisclosed
User-selectable effort at inferenceEffort label as input, per-effort rewards or experts, or a prompt-given targetV4, K3, Nemotron658; L111Penalty values and \(\tau\) are undisclosed
Effort levels without RL infrastructureSFT on teacher traces generated at each effortK2-V217Needs an effort-controlled teacher
A budget on a model already trainedServing-time truncation, or long2shortQwen37; k1.51Quality at a budget is never measured by an objective

8. Serving-side implications (my inference, not the reports')

None of the reports analyzes serving cost beyond a token count; this section is my reading, built on the standard memory-bound decode model. A decode step at context length \(c\) reads the weights, amortized over a batch of \(B\) requests, plus the request's own KV cache. With \(W\) the weight bytes read per step and \(k\) the KV bytes per token, a request that generates \(L\) tokens after a prompt of \(P\) tokens moves

$$\sum_{t=1}^{L}\Bigl(\tfrac{W}{B}+k\,(P+t)\Bigr)=L\Bigl(\tfrac{W}{B}+kP\Bigr)+\tfrac{k}{2}L^2=a\,L\Bigl(1+\tfrac{\kappa}{2}L\Bigr),\qquad a=\tfrac{W}{B}+kP,\ \ \kappa=\tfrac{k}{a}.$$

When \(\kappa L\ll1\), cost is linear in tokens and a token reduction is a one-for-one throughput gain: Toggle's 25% to 30% fewer output tokens4 is \(1/0.75\) to \(1/0.70\), 1.33x to 1.43x more requests per decode-second, and Nemotron's medium effort at about 2.5x fewer tokens8 is about 2.5x. When KV traffic matters, the quadratic term makes the saving larger than the token ratio; the calculator lets you set \(\kappa\). Prefill, speculative decoding and compute-bound regimes are outside this model.

Relative decode traffic of a request before and after an effort cut

Token ratio (before / after)
Traffic after / before
Requests per decode-second gained

Formula: \( C(L)=L\,(1+\kappa L/2) \) with \( \kappa \) in units of 1e-6 per token, the ratio of per-token KV traffic to the per-token fixed traffic \( a \). The defaults (10,000 to 4,000 tokens, \( \kappa=5\times10^{-5} \)) are illustrative: the 2.5x token ratio is Nemotron's8; \( \kappa \) is not from any source and depends on your model, batch size and prompt length.

Four properties of effort modes matter to an engine, each grounded in a report but extended by me.

  • Reservation scales with the mode. V4's evaluation windows are 8K, 128K and 384K tokens for Non-think, High and Max6 (§5.3.1), so a Max request can reserve up to 48x the KV of a Non-think one (derived). Admission control can key on the mode instead of the worst case.
  • Length variance differs by level. K2-V2 observes that its low and medium efforts "have fairly consistent generation length, while high has a large spread"17 (§6.1). A tail of long generations is what stalls batches, and why K3 uses partial rollouts in training5 (§4.1.2).
  • The effort setting sits in the prefix. K3 places "the reasoning-effort setting" before all input messages because global options "rarely change, so modifying them invalidates the KV cache anyway"5 (App. F). A per-request effort router that changes this message forfeits prefix-cache reuse across modes; V4's Max instruction is also a system-prompt prefix6 (Table 3).
  • One model replaces two deployments. Qwen3 cites "reducing the cost and complexity of deploying separate models for thinking and non-thinking tasks"7 (§4.3), and K2-V2 argues a single model with a chat-template setting avoids serving separate instruct and thinking variants17 (§6).

Where experts disagree

The reports agree that unconstrained RL lengthens chains and that something should be done about it. They disagree about what kind of object length is: a cost to penalize, a budget to enforce, a bias to remove from the loss, or an input the user sets. Frontier recipes mix these, so the positions below are the pure forms.

1. Penalize length inside the reward, continuously and relative to the group

Moonshot's k1.51; Arora and Zanette10; DeepSeek-V3.2 and its reduced-penalty Speciale15

A smooth term adapts to difficulty automatically and one coefficient yields a family of models.

  • Difficulty-adaptive: 83% fewer tokens on GSM8K against 27% on AIME at one \(\alpha\)10
  • Full reward forms and ablations are public at small scale1011
  • The multiplicative form cannot rank a long correct answer below a wrong one
  • Coefficient-sensitive; the standard-deviation identity of Section 2.3 can erase it
  • k1.5 needed a warm-up because early application "may slow down training"1
  • V3.2's authors say their score is "constrained by a length constraint reward model"15

2. Enforce a budget, then alternate or anneal it

Kimi K23, K2.5's Toggle4, K35

A hard override has no relative weight to tune. Alternation keeps the ability to scale up, and annealing the multiplier produces levels.

  • Output tokens down 25% to 30% with "negligible" accuracy impact, and the cut transfers to benchmarks outside the training mix4
  • A quality gate prevents trading accuracy away on problems the model cannot yet solve4
  • Rigid budgets caused the length-overfitting Toggle was built to fix4
  • \(\rho\), \(\lambda\), \(m\), \(\tau\) and the override's size are not disclosed in the text read
  • Evidence is one run per benchmark, no variance

3. Do not train length: fix the loss, cut at serving time

Dr. GRPO9; Qwen3's untrained budget7; s1's budget forcing12

Part of the growth is a normalizer artefact, and a model fused across modes can be stopped early without an objective for it.

  • No coefficient and no capability tax from a penalty
  • Qwen3 reports smooth gains with budget7; A&Z's \(\alpha=0\) control shows a normalizer alone moved length10
  • No target length results from a loss fix
  • Quality at a budget is never optimized; L1 beats s1 truncation by 20 to 25 points at 512 and 1,024 tokens (1.5B)11

4. Make effort an input and train each level

DeepSeek-V46; K35; Nemotron 3 Ultra8; L111; gpt-oss14; K2-V217

Users need different trade-offs, so train several operating points and let one model read the label.

  • Large, selectable gaps: V4-Pro HLE 7.7, 34.5 and 37.7 across its three modes6
  • One model replaces separate deployments17
  • 2.5% of RL prompts sufficed for Nemotron's medium mode8
  • Training details are thin: gpt-oss gives one sentence14; V4 and K3 give no penalty or \(\tau\) values
  • Needs per-level experts plus distillation (Module 22) or a large SFT corpus
  • Returns diminish at the top: Flash on Terminal Bench gains 0.3 points from High to Max6
Assessment

These are layers, not rivals. V3.2 used position 1 and V4 moved to position 4; K3 uses position 2 to manufacture the levels position 4 exposes. For one operating point at lower cost, a relative gated reward term (1) or a gated budget with alternation (2) is the evidence-backed choice, after the Section 2.3 check on advantage normalization. For several operating points, position 4 is where the frontier reports converged, with 2 producing the lower levels. Position 3 is the right first step because it removes a bias for free, and it is not sufficient alone. The evidence gap is the finding: frontier reports give no ablation numbers for their length terms, and the small-scale papers that do (1.5B to 7B, 4K to 16K contexts) are not comparable to them. Settling it takes one base and one dataset with a relative penalty, a gated override and an effort expert, each swept over at least three budgets with seeds, scored on one token metric and on out-of-domain benchmarks.

What is changing

As of October 2026, effort is becoming a first-class API object. K3's chat template reserves four levels and supports a subset5; V4 ships three6; gpt-oss, K2-V2 and Nemotron converge on three14178. Effort conditioning is moving into distillation: K3 consolidates nine domain-by-effort experts by MOPD with the level as an input5, the subject of Module 22. Open items: no shared accuracy-per-token metric; whether SFT-only effort (K2-V2) matches RL-trained effort (Nemotron) has no head-to-head; and whether Toggle's alternation helps outside the one model it was reported on. A bibliographic caution: the V4 report's arXiv id is 2606.19348 but the stamp on its PDF reads 26 Apr 20266, so cite it by title and id and do not date it from the id.

References

  1. paperKimi Team (Du, A., Gao, B., Xing, B., et al.). Kimi k1.5: Scaling Reinforcement Learning with LLMs. Moonshot AI, arXiv v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: length reward and warm-up (§2.3.3; printed equation checked on p. 6), policy-optimization gradient (§2.3.2, Eq. 3), long2short methods (§2.4), long2short results (§3.4, Fig. 7), abstract short-CoT numbers.
  2. paperYu, Q., Zhang, Z., Zhu, R., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. ByteDance Seed and Tsinghua AIR, arXiv v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: token-level loss (§3.3), soft overlong punishment (§3.4, Eq. 13), training settings (§4.1).
  3. paperKimi Team (Bai, Y., Bao, Y., Charles, Y., et al.). Kimi K2: Open Agentic Intelligence. Moonshot AI, arXiv v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: algorithm foundation and budget control (§3.2.3).
  4. paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.). Kimi K2.5: Visual Agentic Intelligence. Moonshot AI, arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: Toggle definition, budget percentile and results (§4.4.2, Eq. 2, Fig. 5 on p. 10; per-benchmark values read from the rendered figure).
  5. paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.). Kimi K3: Open Frontier Intelligence. Moonshot AI, arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: reasoning-effort RL (§4.1.2), effort-conditioned MOPD reward (§4.1.3, Eq. 15), cost claim (§6.4), effort option message and KV-cache placement (Appendix F).
  6. paperDeepSeek-AI (Xu, A., Lin, B., Xue, B., et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348v1 (PDF stamp 26 Apr 2026). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: reasoning-effort modes (§5.1.1, Tables 2 and 3), evaluation context windows (§5.3.1), Table 7 (p. 39), Figure 10 (p. 41; token positions read off the axis).
  7. paperYang, A., Li, A., Yang, B., et al. (Qwen Team). Qwen3 Technical Report. Alibaba, arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: Thinking Mode Fusion and thinking budget (§4.3, Table 9), Table 22 stage results and Figure 2 (§4.7).
  8. paperNVIDIA (Blakeman, A., Thomas, A., et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: effort-control SFT data (§3.1.1), reasoning modes and medium-effort RLVR share (§3.5), Figure 11 (p. 31; points read off the plot).
  9. paperLiu, Z., Chen, C., Li, W., et al. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025, arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: Dr. GRPO's loss-side length fix and its 43.3% AIME 2024 result (abstract, §3.2).
  10. paperArora, D., Zanette, A. Training Language Models to Reason Efficiently. NeurIPS 2025, arXiv v4, 3 Nov 2025. https://arxiv.org/abs/2502.04463. Accessed 2026-10-27. Used for: reward form (§4, Eqs. 4 and 5), results (§5.3.1), advantage-normalization ablation (§5.4.1), limitations (§6). Added by the writer; full text read.
  11. paperAggarwal, P., Welleck, S. L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. COLM 2025, arXiv v2, 3 Oct 2025. https://arxiv.org/abs/2503.04697. Accessed 2026-10-27. Used for: LCPO rewards (§3, Eqs. 1 and 2), setup (§4), results (§5, Figs. 2, 4, 11), additive-reward ablation (App. A.7). Added by the writer; full text read.
  12. paperMuennighoff, N., Yang, Z., Shi, W., et al. s1: Simple test-time scaling. arXiv 2501.19393, 2025. https://arxiv.org/abs/2501.19393. Accessed 2026-10-27. Used for: budget forcing (abstract only). Added by the writer.
  13. paperChen, X., Xu, J., Liang, T., et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. arXiv 2412.21187, Dec 2024. https://arxiv.org/abs/2412.21187. Accessed 2026-10-27. Used for: the overthinking framing (abstract only; the body was not read). Added by the writer.
  14. officialOpenAI. gpt-oss-120b & gpt-oss-20b Model Card. arXiv v1, 5 Aug 2025. https://arxiv.org/abs/2508.10925. Accessed 2026-10-26. Used for: variable-effort levels set by system prompt (§2.5.2) and accuracy against CoT length (Fig. 3). Added by the writer.
  15. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: length penalty in the reward and the Speciale variant (§3), advantage without std division (§3.1), Table 3 accuracy and output tokens (§4.2), token-efficiency statements (§4.2, §5).
  16. officialHugging Face. TRL documentation: GRPO Trainer. main branch. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-26. Used for: loss_type options ("dapo" default, "dr_grpo") and the scale_rewards default ("group"); text extracted by a fetch tool, not machine-verified.
  17. paperK2 Team, Institute of Foundation Models, MBZUAI. K2-V2: A 360-Open, Reasoning-Enhanced LLM. IFM Technical Report, arXiv 2512.06201, 2 Dec 2025. https://arxiv.org/abs/2512.06201. Accessed 2026-10-27. Used for: SFT-only three reasoning efforts, special tokens and generation-length spread (§6.1), TxT360-3efforts data (§6.2). Added by the writer; the relevant sections were read.
  18. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: the survey of toggleable-reasoning practice (§7.3.3); a secondary source, not used for any number.