Reinforcement Learning for LLM Post-Training · Module 28

Case Study: Kimi k1.5 to K3: Mirror-Descent Policy Optimization, Self-Critic Rewards, Token-Budget RL, and Nine-Expert MOPD

Moonshot AI published four technical reports in eighteen months, and each one states a problem with the previous recipe and changes one part of it. This module reads them as a lineage: an objective that is not GRPO, a reward surface that widens from tests to rubrics to a judge that writes its own rubric, three generations of length control, a gradient mask for off-policy drift, and a final stage that distills nine specialists into one student. Afterwards you can say which Kimi design choice answered which stated failure, which choices have no ablation behind them, and which numbers the reports do not give.

Course 45 min read 20 sources As of 2026-10-27 Prerequisites: Modules 9, 12, 16, 18, 22, 23
Four Kimi reports as columns and five design dimensions as rows, with changed cells highlighted k1.5 · 22 Jan 2025 K2 · 28 Jul 2025 K2.5 · 2 Feb 2026 K3 · 27 Jul 2026 Objective Reward Length Rollout Final stage Mirror descentmean baseline Same lossMuon optimizer Log-ratio maskignores adv. sign Follows K2.5+ per-token reg. Verifier + CoT RMtests, answer match Gym + self-criticpairwise rubrics GRM rubrics+ vision rewards Agentic judgewrites own rubric Length rewardin-group, warm-up Budget per tasktruncate, penalize Togglebudget / free phases τ·b0 budget-1 over, 3 efforts Partial rolloutstoken cap per pass Partial + Gym APIagentic long tail Async coroutinesup to 100K tasks Pause at λNKspans policy versions long2shortmerge, RS, DPO, RL Joint RLafter SFT + tool data Text + vision RL+ PARL orchestrator 9 experts → MOPD3 domains × 3 efforts first report unchanged from previous changed or added
Figure 1. The lineage in one grid. Column dates are arXiv v1 dates; every column was revised later (k1.5 v4 3 Jun 2025, K2 v2 3 Feb 2026, K2.5 v2 and K3 v2 7 Aug 2026)1234. K2 keeps the k1.5 loss and adds budget control, a PTX loss and temperature decay to it; K2.5 changes the loss; K3 states that its policy optimization follows K2.5 and adds a staleness-tolerant rollout scheme. Cell labels are the writer's compression of the sections cited in Sections 2 to 5.

1. Four reports, one lineage

All four documents are authored as "Kimi Team" (Moonshot AI) and each says what it inherits. K2 states "we adopt the policy optimization algorithm introduced in K1.5 as the foundation for K2"2 (§3.2.3). K2.5 says its loss "departs from the policy optimization algorithm used in K1.5"3 (§4.4.2). K3 says its policy optimization "follows the algorithm in Kimi K2.5"4 (§4.1.2). That chain of self-citations is what makes a four-report case study coherent: you can follow one lineage of objective, reward and rollout design without switching labs.

1.1 Dates, names, and what each report is

Kimi k1.5 appeared on 22 January 2025, the same day as DeepSeek-R1's first version15. The two reports reached opposite answers on the optimizer: R1 used GRPO, k1.5 used a mirror-descent objective with a mean-reward baseline, and Module 9 derives both. Module 27 covers R1 itself.

Table 1. The four reports at a glance. "RL compute" is what the text of each report states about the cost of its RL runs; "not stated" means no GPU-hour, dollar or FLOP figure appears in the sections read1234.
Reportv1 / latest readModelPost-training stages namedRL objectiveRL compute
Kimi k1.522 Jan 2025 / v4, 3 Jun 2025multimodal; parameter count not statedvanilla SFT, long-CoT SFT, RL, long2shortonline policy mirror descentnot stated
Kimi K228 Jul 2025 / v2, 3 Feb 2026MoE, 1.04T total, 32B activeSFT with agentic data synthesis, joint RLthe k1.5 lossnot stated
Kimi K2.52 Feb 2026 / v2, 7 Aug 2026K2 backbone plus MoonViT-3Dzero-vision SFT, joint text-vision RL, PARLk1.5-derived loss with a token-level masknot stated
Kimi K327 Jul 2026 / v2, 7 Aug 2026MoE, 2.8T total, 104B active, 1M contextSFT, 9 RL experts, MOPDK2.5's algorithm plus staleness handling1M-context experiment "within a few hundred GPUs"

1.2 What the reports can and cannot support

Two limits apply to everything below. First, none of the four reports states an RL GPU-hour or dollar figure, so the cost arithmetic of Module 1 has no Kimi counterpart. K3 comes closest: co-located RL keeps "each 1M-context Kimi K3 RL experiment within a few hundred GPUs"4 (§5.3.1), a footprint with no duration. Second, none of the reports, in the sections read, runs its objective against GRPO under matched conditions. k1.5 has the only direct algorithm ablation (against ReST, Section 2.4); the others describe choices and show curves1234. Where this module says a choice "answered" a problem, the problem is one the next report states itself, and the link is the report's wording, not a measured effect.

2. k1.5 (January 2025): value-free RL with a long context

2.1 The stance: flatten the search tree into the context

k1.5's abstract names what it leaves out: "Monte Carlo tree search, value functions, and process reward models"1. The argument for leaving them out is in §2.3.1. A planning algorithm grows a tree of partial solutions guided by a critic; the report rewrites that as a mapping on a flat sequence of reasoning steps, so that "all information stored in the search tree ... is flattened into the full context", the number of thought tokens plays the role of the planner's compute budget, and a long enough context lets the model "run an implicit search over the reasoning space directly via auto-regressive predictions"1. The introduction compresses this to one sentence: "An increased context length has an effect of increasing the number of search steps." Two ingredients follow: scale the RL context window to 128k tokens, and use an objective that does not need a critic.

The objective is online policy mirror descent. At iteration \(i\) the current policy \(\pi_{\theta_i}\) is the reference, and the report maximizes expected reward minus \(\tau\,\mathrm{KL}(\pi_\theta\Vert\pi_{\theta_i})\), whose closed-form optimum is \(\pi^*\propto\pi_{\theta_i}e^{r/\tau}\). That yields a squared-error surrogate in which \(\tau\log Z\) is replaced in practice by the empirical mean reward \(\bar r\) of \(k\) samples1 (§2.3.2). Module 9 derives it and sets it beside GRPO and RLOO7; the point for this lineage is what survives into later reports.

Derivation: the k1.5 gradient and K2's printed loss are the same object

Write \(\rho=\pi_\theta/\pi_{\theta_i}\) for a sampled response. The k1.5 surrogate is \(L=\mathbb{E}\big[(r-\tau\log Z-\tau\log\rho)^2\big]\). Using \(\nabla\log\rho=\nabla\log\pi_\theta\) and \(\nabla(\log\rho)^2=2\log\rho\,\nabla\log\pi_\theta\),

$$-\frac{1}{2\tau}\nabla_\theta L=(r-\tau\log Z)\,\nabla\log\pi_\theta-\frac{\tau}{2}\nabla(\log\rho)^2 .$$

With \(\tau\log Z\) replaced by \(\bar r\), the right-hand side is the report's Eq. 3 (mean-reward baseline plus an \(\ell_2\) regularizer on the log-ratio)1. K2 then prints the loss directly as \(L_{RL}=\mathbb{E}\big[\tfrac1K\sum_i\big(r(x,y_i)-\bar r(x)-\tau\log\tfrac{\pi_\theta(y_i|x)}{\pi_{old}(y_i|x)}\big)^2\big]\)2 (§3.2.3, p. 13). Derived by the writer from the two printed forms; each report states its form, and the identity is the writer's.

The report's case against a value function is a hypothesis, not an experiment: "we also hypothesize that the conventional use of value functions for credit assignment ... may not be suitable"1. An oracle value function would give a negative advantage to an exploratory reasoning step that contains an error and later recovers, yet that recovery is the behavior long-CoT training wants. Module 9 treats the debate; the report offers no run with a critic for comparison.

2.2 Data engineering: the part Module 9 does not cover

k1.5 spends as much text on the RL set as on the loss. A prompt set needs diverse coverage, balanced difficulty and "accurate evaluability"1 (§2.1). Difficulty is measured: an SFT model answers each prompt ten times at a relatively high temperature and the pass rate is the proxy. Two filters protect the reward. Multiple-choice, true/false and proof-based questions are dropped because guessable answers give false positives, a right answer by a wrong route. For general question answering, a model guesses the answer with no chain of thought; a prompt answered correctly within \(N\) attempts is "easy-to-hack" and removed, with \(N=8\) removing "the majority" of such prompts1. The fraction of the pool removed is not given.

Coding rewards need tests, and web problems often lack them. The recipe (§2.3.5) uses the CYaRon generator library: the base model writes a test generator from the problem statement, the generator makes 50 test cases, and 10 ground-truth submissions are run on each. A case is valid if at least 7 of 10 submissions produce the same result; a problem enters the training set only if at least 9 of 10 submissions pass all the selected cases1. The report gives one yield measurement, drawn in Figure 2.

Funnel from 1,000 sampled contest problems to 323 kept coding problems 1,000 sampled online contest problems 100% ≈ 614 need no special judge 61.4% of 1,000 463 generators, ≥ 40 valid tests 46.3% of 1,000 323 kept for training 32.3% of 1,000; 52.6% of 614 Per problem 1. Base model writes a CYaRon generator; it emits 50 test cases 2. 10 ground-truth submissions run on every case; a case is valid at ≥ 7 of 10 agreeing 3. The problem is kept only if ≥ 9 of 10 submissions pass all selected cases
Figure 2. k1.5's test-case yield on a sample of 1,000 online contest problems: about 614 need no special judge, 463 generators produced at least 40 valid cases, and 323 problems entered the training set1 (§2.3.5). The percentages are derived. The 52.6% shares 323 against 614 and assumes the 463 generators were built for the 614 problems, which the report implies but does not state.

Math has the opposite problem: many written forms for one answer. The report trained two answer-checking models on roughly 800k labelled examples each. A "classic" value-head reward model takes question, reference answer and response and emits one scalar; a chain-of-thought reward model writes a step-by-step judgment first and ends in a JSON verdict. On "manual spot checks" the classic model scored about 84.4 and the chain-of-thought model 98.5, and RL used the second1 (§2.3.5). The spot-check size is not given, so 98.5 is a vendor figure. A learned model sits in the reward path here, on the day of a release whose expanded version says it will "abstain from applying neural reward models" for reasoning tasks because of reward hacking5 (§2.2); the difference is that k1.5's model decides whether an answer matches a reference, not which of two answers is better. Vision RL data has three sources: real-world (science questions with figures, location guessing, charts), synthetic scenes, and text rendered into images so a query gets a consistent answer as text or as a screenshot1.

2.3 Sampling, length, and rollouts

Two sampling rules ride on top of the loss (§2.3.4). Curriculum sampling moves from easy to hard problems, because a weak initial model gets few correct samples on hard ones. Prioritized sampling draws problem \(i\) with probability proportional to \(1-s_i\), one minus its tracked success rate1. For \(s=(0.9,\,0.5,\,0.1)\) the weights are \((0.1,\,0.5,\,0.9)\) and the draw probabilities \((6.7\%,\,33.3\%,\,60.0\%)\) (derived). A problem at \(s=1\) is never drawn, and \(s=0\) gets the largest weight although a group of all-zero rewards carries no advantage signal under a mean baseline (Module 9). The only sampling ablation is the curriculum (a warm-up on the full set, then hard problems only, switching at iteration 24, against uniform sampling), reported as "significantly" better with the gap shown only as a plot1 (Figure 9).

Length control and rollouts each have a module of their own. The length reward is computed per problem across the \(k\) samples, favors the shorter correct responses, penalizes long wrong ones, and is warmed up: standard optimization first, then a constant penalty, because it "may slow down training during the initial phases" (Module 23)1 (§2.3.3). Partial rollouts cap output tokens per pass and continue the unfinished tail from a replay buffer, so "only the current iteration (iter n) requires on-policy computation" (Modules 1 and 18)1 (§2.6.2). The Megatron-plus-vLLM system is covered in System Design Modules 36 to 38.

2.4 Results, ablations, and what the report leaves out

Table 2. k1.5 long-CoT against the models in the report's Table 2 (p. 12), with the protocol notes from Appendix C. Pass@1 and exact match (EM) as the report labels them; "-" means not reported1.
Benchmark (metric)QwQ-32B Previewo1-miniQVQ-72B Previewo1k1.5Protocol note (App. C)
MATH-500 (EM)90.690.0-94.896.2500 problems (Lightman et al.)
AIME 2024 (Pass@1)50.063.6-74.477.5the 2024 AIME questions
Codeforces (percentile)6288-9494majority vote over model-written tests; percentile mapped from Elo using an OpenAI talk
LiveCodeBench (Pass@1)40.653.1-67.262.5v5, questions from 2412 to 2502
MathVista-Test (Pass@1)--71.471.074.9vision-math benchmark
MMMU-Val (Pass@1)--70.377.370.0college-level multimodal questions
MathVision-Full (Pass@1)--35.9-38.63,040 problems with visual context
Dumbbell plot of OpenAI o1 against Kimi k1.5 on six benchmarks, axis from 50 to 100 OpenAI o1 Kimi k1.5 5060708090100 score as labelled in the report (EM, Pass@1, or percentile); axis starts at 50 MATH-500 (EM) 94.896.2 AIME 2024 (Pass@1) 74.477.5 Codeforces (percentile) 94 both LiveCodeBench v5 (Pass@1) 62.567.2 MathVista-Test (Pass@1) 71.074.9 MMMU-Val (Pass@1) 70.077.3
Figure 3. "Matching o1" is an average statement. k1.5 is ahead on MATH-500, AIME 2024 and MathVista, ties on the Codeforces percentile, and is behind on LiveCodeBench and MMMU. Data: the report's Table 2 on p. 121; the axis is truncated at 50.

Table 2 and Figure 3 qualify the abstract's "matching OpenAI's o1"1. The Codeforces entry is the least comparable: Appendix C.2 says the model's submissions were voted over tests "generated by the same model", and the percentile comes from an OpenAI talk's Elo mapping, so it is not a pass@1 on a fixed judge. The short-CoT headline, "up to +550%", reproduces from Table 3: AIME 2024 60.8 against GPT-4o's 9.3 is \((60.8-9.3)/9.3=5.54\), a gain of 554% (derived), and against Claude 3.5 Sonnet's 16.0 it is 280%. The report also says its Table 3 IFEval entry "derived from an intermediate model"1 (App. C.1).

The long2short results are the first of three length-control designs (Section 6). Four methods turn the long-CoT model into a token-efficient short-CoT one: weight averaging with a short-CoT model; shortest rejection sampling (8 samples per problem, SFT on the shortest correct one); DPO with the shortest correct response as positive and, as negatives, wrong long responses plus correct ones 1.5 times longer; and a second RL phase with the length penalty and a much shorter maximum rollout length. In the report's Figure 7 the last is the most token-efficient: 60.8 on AIME 2024 (Pass@1 averaged over 8 runs) at 3,272 tokens on average1 (§2.4, §3.4).

Two ablations beyond the curriculum carry the scaling argument. A smaller model trained with longer CoTs "can achieve comparable performance" to a larger one, but the larger model is more token-efficient and "has a higher upper bound", so a test-time-compute budget favors the smaller model and a quality target the larger (§3.5, Figure 8). ReST, which refits only the best sampled responses with no negative gradient, has worse sample complexity than the k1.5 objective (Figure 10), the evidence that penalizing wrong long chains does work. Not given anywhere in the body or appendices: the parameter count, \(k\), \(\tau\), the RL learning rate and batch size, the number of RL steps, and any GPU-hour figure1.

3. K2 (July 2025): the same loss, a wider reward surface

3.1 What K2 kept, and what changed about the job

K2 is a mixture-of-experts model with 1.04T total and 32B activated parameters, pre-trained on 15.5T tokens with the MuonClip optimizer2 (abstract, §1). Its RL objective is k1.5's, printed on p. 13 as \(L_{RL}(\theta)=\mathbb{E}_x\big[\tfrac1K\sum_{i=1}^{K}\big(r(x,y_i)-\bar r(x)-\tau\log\tfrac{\pi_\theta(y_i|x)}{\pi_{old}(y_i|x)}\big)^2\big]\) and minimized with Muon2 (§3.2.3); the box in Section 2.1 shows it is k1.5's surrogate with the mean reward replacing \(\tau\log Z\). What changed is the job. k1.5 was a long-CoT reasoning model; K2 is evaluated against every baseline in non-thinking mode, with an 8,192-token output cap on every benchmark except one SWE-bench mode2 (§4.1.1). The length pressure reverses: k1.5 fights "overthinking" in a model meant to think, while K2's budget control exists because longer responses "do not justify" their cost "in non-reasoning domains".

K2's RL runs on a colocated architecture inherited from k1.5, where training and inference engines share workers and a checkpoint engine pushes the full parameter set to the inference replicas, which the report says completes "with less than 30 seconds" for the 1T model2 (§3.3.2). That is a systems result; System Design Module 37 covers weight-sync engineering. The report gives no \(\tau\), \(K\), learning rate, step count or RL compute.

3.2 The verifiable-reward gym

K2 describes "a Gym-like extensible framework" with "a large number of tasks with verifiable rewards", plus a self-critic reward for subjective tasks2 (§3.2). The prompt-set rule is the one k1.5 used for difficulty: "neither too easy nor too hard", judged by the SFT model's pass@k, keeping only moderate problems. Table 3 lists the task families and what scores each.

Table 3. K2's RL task families and their scorers, from §3.2.12.
FamilyPrompts or settingWhat produces the reward
Math, STEM, logicexpert annotation, internal QA extraction, open datasets; logic puzzles (24-game, Sudoku, cryptarithms, Morse decoding) and table-reasoning tasksverifiable answers (the checker is not detailed)
Complex instruction followingexpert-written conditional prompts, agentic augmentation, a model fine-tuned to probe failure modescode interpreter for checkable constraints, LLM-as-judge for nuanced ones, plus a "hack-check" layer for false claims of compliance
Faithfulnessmulti-turn tool use, self-generated reasoning, open-environment interactiona sentence-level faithfulness judge trained for the purpose, used as a reward model
Competition codingopen-source and synthetic problems with judgesjudge verdicts, with human-written unit tests retrieved from pre-training data
Software engineeringGitHub pull requests and issuesexecutable unit tests in Kubernetes sandboxes, over 10,000 concurrent instances
Safetyhuman seed prompts evolved by an attack modela judge model applies a task-specific rubric to give a binary success or failure label

The agentic rollout section (§3.3.4) adds the operational facts: heavy environments run as dedicated services, many concurrent rollouts amortize slow environment calls, long-tail trajectories are paused and resumed with k1.5's partial rollouts, and a Gym-style unified interface integrates new environments2. Whether tool-observation tokens are masked out of the loss is not stated in the K2 text read; that is an absence in the report, not evidence that K2 does not mask (Module 18 covers the mechanics).

3.3 Self-critic reward: what the report gives and withholds

For subjective prompts, K2's actor generates responses and the K2 critic "ranks all results by performing pairwise evaluations" against a mix of core rubrics, prescriptive rubrics meant "to eliminate reward hacking", and human-written ones. The critic is refined during RL on rollouts from verifiable-reward prompts, so that "the critic continuously recalibrates its evaluation standards in lockstep with the policy's evolution"2 (§3.2.2). Module 16 reads the loop in detail. Three things are absent from the K2 text: the critic's loss, the rule that converts a ranking into a scalar reward for the policy, and any experiment against a separate or frozen judge. The closed loop is therefore a vendor claim: a design with no ablation behind it.

K2's appendix is more candid than its body. The prescriptive rubrics forbid opening praise and sentences that explain why a response is good, and they prohibit self-assessments and hedging such as "I might be wrong", so Appendix F.3 concedes that the framework "may favor responses that appear confident and assertive", and that "the model may occasionally overstate certainty in areas where ambiguity, nuance, or epistemic modesty would be more appropriate". It adds that future iterations "may incorporate more fine-grained handling of calibrated uncertainty"2. A reward that penalizes hedging is a documented calibration cost inside a production recipe; Module 24 develops why.

3.4 Three additions to the loss

Budget control sets a per-sample maximum token budget "determined based on the type of task"; over-budget responses are truncated and penalized. An auxiliary PTX loss on hand-selected high-quality samples limits forgetting during joint RL. A temperature decay runs from exploration to exploitation: high temperature early avoids "premature convergence", while a high temperature later "introduces excessive randomness"2 (§3.2.3). No budget values, PTX weight, decay schedule or ablation are given (Modules 13 and 23). K2's limitations name its own length problem: on hard reasoning tasks or unclear tool definitions the model "may generate excessive tokens, sometimes leading to truncated outputs or incomplete tool calls"2 (§5). The next report picks up that thread.

4. K2.5 (February 2026): a masked loss, Toggle, and a trained orchestrator

4.1 The loss: one equation, two descriptions

K2.5 is the first report in the line to change the objective, and it says why: the new loss introduces "a token-level clipping mechanism designed to mitigate the off-policy divergence amplified by discrepancies between training and inference frameworks"3 (§4.4.2). Its description of the mechanism is a gradient mask: "policy gradients are computed normally for tokens with log-ratios within the interval [α, β], while gradients for tokens falling outside this range are zeroed out", and, unlike PPO's clip8, it "relies strictly on the log-ratio ... regardless of the sign of the advantages". The report adds that the mechanism is "essential for maintaining training stability" in long-horizon tool-use domains, with no ablation shown. Module 12 places this beside GSPO and CISPO; this section is about the report's own text.

The printed equation (Eq. 1, p. 8) does not read the same way as that prose. With \(\rho_{j,i}=\pi_\theta(y_j^i\mid x,y_j^{0:i})/\pi_{old}(\cdot)\) the per-token ratio, \(\ell=\log\rho\), and \(A_j=r(x,y_j)-\bar r(x)\) the sequence advantage, Eq. 1 is

$$L_{RL}(\theta)=\mathbb{E}_{x}\Big[\frac1N\sum_{j=1}^{K}\sum_{i=1}^{|y_j|}\Big(\mathrm{Clip}\big(\rho_{j,i},\,\alpha,\,\beta\big)\,A_j-\tau\,\ell_{j,i}\Big)^2\Big],\qquad N=\sum_{j}|y_j|,$$

where \(\alpha,\beta,\tau>0\) are hyperparameters3. Three differences from K2's loss and from the prose follow. First, the unit changes. K2's ratio is that of a whole response and its average is over \(K\) responses; here the ratio is per token and the average is over all \(N\) tokens in the batch, a token-level loss in the sense of Module 11. Second, the clip acts on the ratio \(\rho\), not on \(\ell\); a band \([\alpha,\beta]\) on the ratio and the same symbols on the log-ratio are different bands (a PPO-style ratio band \([0.8,1.2]\)8 is \([-0.223,\,0.182]\) in log-ratio, derived). Third, a clip is constant outside its band, so the gradient that flows through the clipped ratio vanishes there, but the \(-\tau\ell\) path does not, and the clipped constant still enters the squared error as its target; "gradients for tokens falling outside this range are zeroed out" is a stronger statement than the equation makes (derived below). Figure 4 draws the three readings.

Which gradient paths are active across the per-token log-ratio, for K2's loss, K2.5's prose and K2.5's printed Eq. 1 αβ K2 loss (p. 13) path via ratio in A-term absent: A is a constant path via −τ log-ratio active at every token K2.5 prose (§4.4.2) path via ratio in A-term zero active inside [α, β] zero path via −τ log-ratio zero active inside [α, β] zero K2.5 printed Eq. 1 (p. 8) path via ratio in A-term zero active inside [α, β] zero path via −τ log-ratio still active outside the band per-token log-ratio ℓ = log(π_θ / π_old)
Figure 4. Schematic of which gradient paths are active, per token, across the log-ratio. K2's loss has one path, through −τℓ, with the advantage as a constant. K2.5's prose zeroes the token's whole gradient outside [α, β]. K2.5's printed Eq. 1, read literally, has two paths: the one through Clip(ρ) is constant, hence zero, outside the band, but the −τℓ path remains and still regresses ℓ toward the clipped value divided by τ (derived). α, β and τ are not reported, so the band position is illustrative23.
Derivation: the gradient of the printed Eq. 1

For one token let \(c=\mathrm{Clip}(\rho,\alpha,\beta)\) and use \(\nabla\rho=\rho\nabla\ell\). Then \(\nabla (cA-\tau\ell)^2=2\,(cA-\tau\ell)\,(A\,c'(\rho)\,\rho-\tau)\,\nabla\ell\). Outside the band \(c'=0\) and \(c\in\{\alpha,\beta\}\) is a constant, so the gradient is \(-2\tau(cA-\tau\ell)\nabla\ell\): the squared error still pulls \(\ell\) toward \(cA/\tau\), and the sign of \(A\) still matters through the constant \(c\). Inside the band \(c'=1\) and it is \(2(\rho A-\tau\ell)(\rho A-\tau)\nabla\ell\), which at \(\rho=1\) is \(2A(A-\tau)\nabla\ell\). K2's loss gives \(-2\tau(A-\tau\ell)\nabla\ell\), which at \(\ell=0\) is \(-2\tau A\nabla\ell\). So if gradients flow through the clipped ratio, the descent direction for a positive-advantage token reverses when \(A>\tau\), which K2's form does not do at \(\ell=0\). The report does not say whether the ratio inside Clip is stop-gradiented, and it does not give \(\tau\); for 0/1 rewards \(|A|\le 1\), so the reversal needs \(\tau<1\). Derived by the writer from the printed forms in K2 (p. 13) and K2.5 (p. 8) and not checked against an implementation.

Which reading the code implements is unknowable from the report; Module 12 follows the prose. The printed form deserves a flag because the readings differ in whether out-of-band tokens keep any gradient at all, and the report's own rationale for the mask is a drift argument. Appendix D adds the system half: the framework "strictly follows a Token-in-Token-out paradigm" and records "log probabilities for all inference engine outputs to perform train-inference mismatch correction"3 (p. 24). The mask needs the sampler's own per-token probabilities as \(\pi_{old}\), so it belongs with the mismatch controls of Module 14 as much as with the clipping family.

4.2 Rewards, zero-vision SFT, and transfer from vision RL to text

K2.5's reward section widens K2's in three directions. Verifiable tasks keep a rule-based outcome reward, now with a budget-control reward alongside it. General tasks use Generative Reward Models that "function not as binary adjudicators, but as fine-grained evaluators", applied "on top of verified reward signals" across chat, coding, search and artifact-generating agents, with "multiple alternative GRM rubrics" to limit hacking and overfitting to one preference signal (Module 16)3 (§4.4.2). Vision tasks get task-specific, graded rewards: an F1 reward with soft matching for grounding (from IoU) and point localization (from Gaussian-weighted distance), segmentation IoU after rasterizing the predicted polygon, normalized edit distance for OCR, and absolute difference for counting; synthesized visual puzzles are scored by an LLM verifier, Kimi K2 itself3. Module 17 covers when graded rewards help.

The cold-start problem is that a pretrained vision-language model does not naturally call tools on images. The conventional fix is hand-annotated chain-of-thought data, which the report says is limited to "simple diagrams and primitive tool manipulations (crop, rotate, flip)". Zero-vision SFT instead uses only text SFT data and proxies every image manipulation through programmatic operations in an IPython tool, and the report finds that "adding human-designed visual trajectories at this stage hurts generalization"; its preliminary text-vision SFT gave "much worse performance on visual, agentic tasks, possibly because of the lack of high-quality vision data"3 (§1, §2.2). Outcome-based RL on grounding and counting, chart and document understanding, and vision-critical STEM problems then supplies the vision behavior.

Table 4. Text benchmarks before and after K2.5's outcome-based vision RL (percent), from the report's Table 2, p. 43.
BenchmarkBefore vision RLAfter vision RLChange (points)
MMLU-Pro84.786.4+1.7
GPQA-Diamond84.386.4+2.1
LongBench v256.758.9+2.2

Text gains from vision RL are the report's evidence for training one policy on both modalities, and it organizes RL domains "not by input modality but by abilities" (knowledge, reasoning, coding, agentic). Read the explanation with care. The report says its analysis "suggests that visual RL enhances calibration in areas requiring structured information extraction"3, but it reports no calibration metric (no expected calibration error, no Brier score) in the text read, and the three rows are single before-and-after pairs without seeds or intervals. The word is used loosely there; Module 24 defines the measurable sense.

4.3 Toggle and PARL in the lineage

Toggle starts from a failure of the previous generation that K2.5 states itself. Problem-dependent token budgets (k1.5, K2) shorten reasoning, but "models trained under rigid budget constraints often fail to generalize to higher compute scales", ending up "defaulting to truncated reasoning patterns" when given more tokens3 (§4.4.2). The fix alternates, every \(m\) iterations, between a budget-limited phase and an unconstrained phase, with the budget set once at the start as the \(\rho\)-th percentile of correct-sample lengths and enforced only where the model's mean accuracy on the problem is high enough. On K2 Thinking it cuts output tokens by 25 to 30% with "a negligible impact on performance", and training on math and code alone still reduced tokens on GPQA and MMLU-Pro. \(\rho\), the accuracy threshold \(\lambda\) and \(m\) are not disclosed, and the per-benchmark changes are visible only in Figure 5 on p. 10 (Module 23).

PARL trains an orchestrator that creates frozen subagents; the subagents' trajectories "are excluded from the optimization objective" and their outputs are treated as environment observations, which sidesteps credit assignment across agents3 (§1, §3; Modules 18 and 19 cover the mechanism and the reward guards). Its reported effect on the wide-search benchmark is an item-level F1 rise from 72.7% to 79.0% and BrowseComp from 60.6% to 78.4% (+17.8 points, derived), with 3 to 4.5 times shorter wall-clock to reach a target F1. The report's introduction prints the first figure as 72.8%; §5.1, Table 4 and Table 6 print 72.7%, which this module uses3 (pp. 2, 11, 12, 14).

5. K3 (July 2026): nine experts, then one student

5.1 The pipeline: experts per domain and effort, then consolidation

K3 is a 2.8T-parameter mixture-of-experts model with 104B activated parameters and a 1M-token context4 (abstract). Its post-training "follows a three-stage paradigm": SFT for a cold-start policy, RL to develop "specialized domain experts at varying reasoning effort", and Multi-Teacher On-Policy Distillation (MOPD) to consolidate them (§4.1). It does not train a model per task: "Rather than training specialized RL models for individual tasks, we scale RL across three broad domains", with one expert per domain "at every reasoning effort level". The domains are general tasks (reasoning, vision, faithfulness, search, knowledge work), general agents (long-horizon assistant tasks, deep research, writing) and coding agents (software engineering, coding experience, kernel tasks, web development). Three domains crossed with low, high and max effort give "a total of nine expert models"4 (§4.1.2). Lambert's survey records the same count beside Nemotron 3 Ultra and DeepSeek-V4, both at more than ten experts, and notes that labs "are far from unanimous" on MOPD9 (§3.2.4).

K3 post-training: SFT, three domains each with max, high and low effort experts, then MOPD into one model SFTcold start Generalreasoning, vision General agentsresearch, writing Coding agentsSWE, kernels, web max effort high effort low effort max effort high effort low effort max effort high effort low effort per domain: max first (large τ), then τ annealed for high and low MOPD teacher: expert (d, e) student sees effort e clip(log ratio, ±R_max) on student rollouts one K3 model Reward forms the report names (their mapping to the nine experts is not stated) verifiable answers · agentic judge for non-verifiable tasks · kernel correctness then speed web checks plus an internal reward model · hidden verifiers in autonomous execution tasks
Figure 5. K3's three stages. Nine experts come from three domains crossed with three efforts; the report also says trajectories from the experts at all effort levels are collected for supervised fine-tuning and for MOPD. The MOPD reward is the clipped log-ratio of the effort-matched teacher to the student on the student's own tokens (Module 22). Data: K3 §4.1 to §4.24; the reward forms are listed in Sections 5.3 and 5.4.

Two properties of this arrangement matter for an RL engineer, and both are the writer's reading, since the report does not discuss them. Each expert is a separate policy, so reward and rollout settings can differ by domain without interference; and the final model sees one on-policy distillation loss rather than a mixture of heterogeneous rewards. The cost is nine RL runs plus a distillation stage, with no GPU-hours stated (Table 1). The scaling evidence is the report's Figure 8: eight panels (coding experience, tool use, web development, agentic search, professional workflows, office deliverables, chart understanding, visual puzzles) of score and average assistant steps against RL FLOPs, captioned "By scaling RL FLOPs, tool-call steps scale up consistently"4 (p. 13). The panels carry no printed axis values, so the size of the gains cannot be read off, and the curves do not isolate the experts, the effort control or MOPD.

5.2 Partial rollouts: a trajectory that spans policy versions

K3 extends k1.5's partial rollouts from a token cap per pass to a completion fraction. In each iteration it samples \(K\) completions for each of \(N\) prompts, keeping \(N\times K\) trajectories active, and "the generation phase pauses as soon as a fraction \(\lambda\in(0,1)\) of trajectories completes (i.e., \(\lambda NK\))". Paused rollouts "are enqueued and prioritized for resumption at the start of the next iteration"; once all \(K\) responses of a prompt are complete they are dispatched to policy optimization, "which follows the algorithm in Kimi K2.5"4 (§4.1.2). The consequence is stated plainly: "an individual long-horizon trajectory naturally spans multiple iterations, introducing data staleness that threatens training stability", and the answer is in the loss: the algorithm "inherently tolerates such an extreme off-policy regime through a per-token regularization" that constrains updates "within a localized neighborhood". Figure 6 steps through a toy case. Module 18 compares this with the other staleness policies (cap it, avoid it, never create it).

iteration n iteration n+1 A · sample 1 A · sample 2 B · sample 1 B · sample 2 generation progress (tokens) π_n π_n π_n π_n pause: λNK = 3 of 4 done donedonedone unfinished, enqueued group A trains π_n → π_n+1 B · 1 waits for its sibling π_n+1 resumed first, sandbox restored group B trains stale prefix one trajectory, two policy versions
  1. Sample under π_n. Toy values: N = 2 prompts, K = 2 samples each, all four sampled by the current policy. The report gives no N, K or λ.
  2. Pause at a completion fraction. Generation stops when λNK trajectories are done; with λ = 0.75 that is 3 of 4. The fourth is unfinished and is enqueued for the next iteration.
  3. Dispatch complete groups. A prompt's group goes to the optimizer once all K of its responses are complete, so A trains and π_n becomes π_n+1. B's finished sample waits for its sibling (the writer's reading of the dispatch rule).
  4. Resume under the newer policy. In iteration n+1 the paused trajectory continues first, now sampled by π_n+1, with its sandbox state restored.
  5. Train on a mixed-version trajectory. When it finishes, B trains. The prefix came from π_n and the suffix from π_n+1, so the update must tolerate a stale prefix; the report assigns that to its per-token regularization.

Figure 6 is a stepper; use the controls or the arrow keys. Toy values are the writer's; the sequence is K3 §4.1.24.

Not in the report: which behavior probabilities the per-token terms use for tokens from different policy versions, any bound on how many versions a trajectory may span, and any comparison with synchronous rollouts or a staleness histogram. K2.5's framework records the engine's log-probabilities for every output token3 (App. D), so a per-token old-policy probability is available in principle; whether K3 uses it that way is not stated (the writer's inference). The algorithm also moves cost into the serving stack. K3 notes that partial rollout "exacerbates" prefix-cache misses "at the beginning of each iteration, due to many unfinished long prefill requests from the previous iteration arriving at the same time", where at 1M tokens a miss is "extremely expensive", and answers with a write-back external KV-cache pool in CPU DRAM and resumable microVM sandboxes4 (§5.3). For an inference engineer that is the lesson of the staleness choice: pausing and resuming turns rollout scheduling into cache management (System Design Module 37).

5.3 Effort RL, the agentic judge, and the verbosity guard

Effort control is the third generation of the length mechanism (Section 6). Each problem \(x\) gets an initial token budget \(b_0(x)\) "estimated from the cold-start model", and a trajectory whose budgeted tokens \(T(y)\) exceed \(\tau\cdot b_0(x)\) has its task reward overridden with \(-1\). For general tasks \(T(y)\) counts thinking tokens; for agentic tasks, cumulative output tokens, "including both reasoning traces and tool-call arguments". Training is "a stage-wise curriculum over the budget multiplier \(\tau\)": a max-effort variant first, with a relatively large \(\tau\), then \(\tau\) annealed to smaller values for the high and low experts, "configured per domain under human-in-the-loop guidance"4 (§4.1.2). At inference the level is requested by a global thinking-effort message that "states the requested level in natural language" rather than exposing a token budget; the schema reserves low, medium, high and max, and K3 supports a subset4 (Appendix F, p. 47). Not given: \(\tau\), the \(b_0\) estimator, the task-reward range (which decides whether \(-1\) is worse than every other outcome) and any ablation of the curriculum. Module 23 compares the form with Toggle.

The only cost-versus-effort evidence is a vendor comparison on an internal benchmark. On Kimi Code Bench 2.0, K3 "is 4.0 points behind Claude Fable 5 at 38% of its cost, and at high effort it already matches Claude Opus 4.8's maximum-effort score at roughly one third of the cost"4 (§6.4). That is a claim about the effort ladder as a product feature, not a measurement of what the \(\tau\) curriculum buys in training.

For non-verifiable general tasks K3 uses an agentic generative reward model "retaining the tournament-style group reward with binary comparisons as in Kimi K2.5", plus a mandatory four-step judge protocol (read the output, generate a rubric, score each candidate against it, record the scores in a scorepad) and a verbosity guard under which a candidate longer than \(\sigma\cdot\ell_0\) "automatically loses the binary comparison"4 (§4.1.2). Two cautions apply. The K2.5 text read for this module never uses the word "tournament" or "pairwise", so the sentence points at a K2.5 design that is not described there; the nearest documented ancestor is K2's pairwise ranking2. And the report does not give the aggregation rule, the judge's tools, whether rubrics are reused across candidates, or the values of \(\sigma\) and \(\ell_0\). Module 16 reads the judge design and Module 19 the guard.

5.4 Consolidation and environments

The final stage conditions the student on the effort level and routes each sampled query to the matching teacher among the nine. The per-token reward is \(\mathrm{clip}\big(\mathrm{sg}[\log\pi^{(d,e)}_{teacher}-\log\pi_\theta],\,-R_{max},\,R_{max}\big)\) evaluated on the student's own tokens, which the report says "seamlessly integrates into our RL framework", including partial rollouts for long-horizon tasks; top-k distillation objectives showed "no clear advantage in either convergence speed or final performance"4 (§4.1.3, Eq. 15). Module 22 derives the objective and compares the alternatives. What K3 does not publish is \(R_{max}\), the number of MOPD steps, or a comparison of MOPD against mixed RL on the same experts, so the case for the consolidation rests on the separate MOPD paper, which reports on Qwen3-30B-A3B that MOPD beats mixed RL, cascade RL, off-policy fine-tuning and parameter merging and inherits "nearly all of each teacher's capability"10 (abstract).

The environments are covered in Module 17: a white-box harness assembled from composable modules (tools, prompts, context policy, skills, memory, subagents) so the policy does not overfit one harness, knowledge-graph-guided task synthesis, verifiable professional agentic problems, kernel tasks with a correctness gate and a hack detector, mock applications for multi-day assistant work, autonomous execution tasks with hidden verifiers, and web development4 (§4.2). The scale statement is the sandbox count: "a total of 51,219,741 sandboxes across 1,505,678 images were created" over training and evaluation, about 34 per image (derived)4 (§5.3.2).

5.5 What K3 offers as evidence

The headline claim is relative: K3 "trails the strongest proprietary systems overall", Claude Fable 5 and GPT-5.6 Sol, "and is consistently ahead of the other open and proprietary models evaluated in our suite"4 (§1). The third-party entry the report lists is Artificial Analysis's Intelligence Index v4.1, where K3 scores 57.1, ranked fourth of 580 models behind Claude Fable 5 (59.9) and GPT-5.6 Sol (58.9), as of 23 July 2026 (§6.3). These are the report's statements of third-party results, not numbers this module re-checked at the source. What is missing is any run that varies one post-training choice: no comparison of nine experts with one mixed-RL policy, no with-and-without effort curriculum, no staleness measurement. The report is at v2 (7 Aug 2026) and may change.

6. Reading the lineage: stated problems and what followed

The useful way to read four reports from one group is to line each stated problem up with the next change. Table 5 does that, using only problems the reports state in their own words; where a report states none, the row says so.

Table 5. Problems each report states, and the change in the next report or stage. "Evidence" is what the report shows for the change, not whether the change was the cause123410.
Stated problemWhereChange that followsEvidence shown
Response length "significantly increases" during RL; overthinking is costlyk1.5 §2.3.3in-group length reward; long2short (merge, shortest RS, DPO, RL)k1.5-short: AIME 60.8 at 3,272 tokens (Fig. 7)
Length penalty "may slow down training" earlyk1.5 §2.3.3warm-up: standard RL first, then a constant penaltynone reported
Longer responses are not worth their cost "in non-reasoning domains"K2 §3.2.3per-sample budget by task type; truncate and penalize"significantly enhances" token efficiency; no numbers
Rigid budgets overfit length and do not scale to more computeK2.5 §4.4.2Toggle: alternate budgeted and free phases25 to 30% fewer tokens on K2 Thinking (Fig. 5)
Training and inference frameworks amplify off-policy divergenceK2.5 §4.4.2token-level log-ratio gradient mask; engine log-probs recorded"essential" for stability; no ablation
Partial rollouts introduce "data staleness that threatens training stability"K3 §4.1.2per-token regularization in the K2.5 lossRL curves (Fig. 8); no staleness data
Judge drifts "toward increasingly verbose outputs"K3 §4.1.2verbosity guard: over σ·ℓ0 loses the comparisonnone reported
Rubrics penalize hedging and may overstate certaintyK2 App. F.3future work "may incorporate" calibrated uncertainty; none in K2.5 or K3 text readopen
Pretrained VLMs do not naturally call tools on imagesK2.5 §2.2zero-vision SFT with IPython-proxied image operationsRL curves; Table 4 transfer to text
Joint RL across domains (K2, K2.5); no reason stated for the switchK3 §4.1nine experts, then MOPD; the MOPD paper names Mix-RL's "see-saw effect"MOPD paper's Qwen3-30B-A3B study; none in K3

6.1 Four observations

Length control went through three forms, each repairing the last. k1.5's reward compares a response with its own group (relative, added to the task reward); K2's is a per-task cap with truncation and a penalty (absolute); Toggle alternates phases and sets the budget from the model's own correct lengths; K3 multiplies a per-problem budget by an annealed \(\tau\) and sets the reward to \(-1\) outside it, then trains one expert per level (Module 23 compares the reward arithmetic). K2's cap and Toggle each repair a limit that their own reports state; K3 states no limitation of Toggle as its motive for the effort ladder. None reports an ablation against the previous form.

The loss is one of two live lineages. The Kimi line keeps a squared-error surrogate with a mean-reward baseline and no standard-deviation division, then adds a per-token mask. DeepSeek's GRPO line converged on the same baseline: V3.2 defines the advantage as the group's mean reward subtracted from the output's, without dividing by the standard deviation11 (§3.1), which is what k1.5's gradient already used. The two lines now differ mainly in how they bound drift (Kimi's masks and regularizers, DeepSeek's sequence masking and routing replay; Modules 12 and 30).

Each widening of the reward surface added a judge and then a guard. K2 adds LLM judges for instructions and a hack-check layer, K2.5 adds GRMs with multiple rubrics, K3 adds an agentic judge and a verbosity limit, and its kernel and autonomous-execution tasks add detectors and hidden verifiers. The guards are reactive ("continuously extend it with new safeguards as new hacking strategies are observed", for kernels), and none reports a hack rate with and without the guard (Module 19).

The structural change in K3 has the thinnest stated rationale. K2 and K2.5 run joint RL after SFT; K3 trains nine experts and distills. The K3 text read names no failure of joint RL as the reason. The reasons given for MOPD come from the separate Xiaomi paper (mixed RL's "see-saw effect" across domains)10, and Lambert records that labs are not unanimous9.

Table 6. Feature by version. "n.s." means not stated in the sections read1234.
Featurek1.5K2K2.5K3
Policy objectivesquared error, mean baseline, response-level ratiosametoken-level ratio, log-ratio mask, 1/N average"follows" K2.5, plus per-token regularization
RL optimizern.s.MuonMuonClipn.s. in RL section
Verifiable rewardgenerated tests (CYaRon); CoT answer-matching RMgym: math, logic, instructions, code, SWE, safetyrule outcome reward; graded vision rewardssearch, professional tasks, kernels, web checks, hidden verifiers
Non-verifiable rewardn.s.self-critic pairwise rubrics, closed loopGRMs, several rubricsagentic judge, tournament-style group reward
Length controlin-group length reward, warm-upbudget by task typeToggleτ·b0 budget, 3 effort levels
Rolloutpartial rollouts by token cappartial rollouts, Gym-style interfacecoroutines, up to 100,000 concurrent taskspause at λNK; resumable sandboxes
Drift controlreference reset to the current iterate each iterationsame: ratio to π_old in the losslog-ratio mask; engine log-probsper-token regularization
Last stagelong2shortjoint RLjoint text-vision RL, PARL9 experts, MOPD
Table 7. Quantities the reports use and do not give, as far as the sections read show1234.
QuantityWhere it appearsValue given?
τ, regularization strength in the lossk1.5, K2, K2.5no
samples per prompt (k or K), steps, batch, learning rateall fourno
α, β of the log-ratio maskK2.5no
temperature-decay schedule; PTX weight; budget valuesK2no
Toggle ρ, λ, mK2.5no
λ, N, K of partial rollout; staleness boundK3no
τ curriculum, b0 estimator, reward rangeK3no
σ, ℓ0, tournament aggregation, judge toolsK3no
R_max, MOPD stepsK3no
RL GPU-hours or costall fourno; K3 only "a few hundred GPUs" per 1M-context experiment

Where experts disagree

The Kimi line takes a side on four questions that the rest of the field answers differently. For each, the Kimi position is set against the main alternative, with what the evidence is and what it is not.

Critic or no critic

No value model: a sampled-mean baseline is enough

Kimi k1.51 (§2.3.2); DeepSeekMath12 (§4.1.1); Ahmadian et al.7 (§1)

k1.5 drops the value network and argues that oracle credit assignment would penalize recoverable exploratory steps. DeepSeekMath's reasons are cost and fit: the value model is another policy-sized network and rewards arrive only at the last token. Ahmadian et al. report REINFORCE-style training ahead of PPO by 3.2% to 20.3% win-rate.

  • No second policy-sized model to train, store or sync.
  • The K2, K2.5 and K3 reports run at 1T-class or larger scale on this footing.
  • k1.5's argument is a stated hypothesis; no run with a critic is shown.
  • A mean baseline gives zero signal on groups with equal rewards.

A well-initialized value model still earns its cost

Tulu 313 (§6.2.1, Limitations); Xu et al.14

Tulu 3's RLVR runs initialized the value model from a general reward model and found this gave the best GSM8K and average scores; the report lists value-free GRPO as untested future work. Xu et al. find carefully tuned PPO ahead of DPO on every testbed, which supports tuned PPO but does not ablate the critic.

  • Per-token credit assignment in principle; a measured value-init ablation.
  • Evidence is short-horizon math and instruction tasks at 8B to 405B, not long-CoT agentic RL.
Assessment

For sparse, outcome-only rewards on long reasoning and agentic trajectories, value-free is the working consensus (Modules 9 and 11), and the Kimi reports are consistent with it. They are not evidence for it, because no Kimi report trains a critic. The Tulu 3 result covers a regime with short responses and a good reward-model initialization. A matched-compute long-CoT comparison with a properly initialized value model is what would settle it, and none has been published in the sources read.

Importance ratio and clipping

Mask or regularize per token on the log-ratio

Kimi K2.53 (§4.4.2); K34 (§4.1.2)

Bound drift directly: zero a token's gradient outside a log-ratio band whatever the advantage's sign (K2.5), and rely on a per-token regularization to tolerate stale partial rollouts (K3).

  • Simple, and the same rule covers training-inference mismatch and staleness.
  • α, β and τ are unreported, the printed equation differs from the prose (Section 4.1), and no ablation is shown.

Change the unit or the clipped quantity

GSPO15 (§1, §5.3); MiniMax-M1's CISPO16 (§3.1); DAPO's Clip-Higher17 (§3.1)

GSPO diagnoses GRPO's token-level weights as "a misapplication of importance sampling weights" and moves the ratio and clip to the sequence, which removes the need for MoE routing replay. CISPO clips the importance weight instead of the update so rare "fork" tokens keep contributing and, in a controlled Qwen2.5-32B study, matched DAPO with half the steps. DAPO keeps PPO's clip and widens the upper bound.

  • Each has a controlled comparison against a baseline.
  • The comparisons are against other clip variants, not against a sign-blind mask.
Assessment

No source compares a log-ratio mask with GSPO or CISPO on the same code, data and model; Module 12 reaches the same conclusion. The families trade differently: a sign-blind mask discards gradient that PPO's min would keep8, CISPO keeps every token with a bounded weight, and GSPO changes the unit and so needs no replay on MoE. Choose by failure mode: a mask or regularizer where staleness is extreme (partial rollouts), a weight clip where rare tokens matter, a sequence ratio on MoE. The K2.5 ambiguity means the Kimi position cannot yet be reproduced from the report.

How to combine many skills

One mixed RL stage, or staged RL with cross-stage distillation

DeepSeek-V3.211 (§3); GLM-518 (§1, §3)

V3.2 merges reasoning, agent and alignment training into one RL stage to avoid "the catastrophic forgetting issues commonly associated with multi-stage training paradigms". GLM-5 runs three RL stages in sequence with on-policy cross-stage distillation to limit forgetting. Kimi K2 and K2.5 also run joint RL.

  • Avoids multi-stage forgetting by construction; V3.2's post-training budget exceeded 10% of pre-training cost11.
  • The MOPD paper attributes cross-domain interference ("see-saw") to mixed RL10.

Experts first, then on-policy distillation

Kimi K34 (§4.1); DeepSeek-V419 (§5.1); MOPD paper10

Train each domain to its peak, then distill on the student's own rollouts with a dense signal. V4 replaces V3.2's mixed RL this way, with more than ten specialists, arguing that logit-level alignment avoids the degradation of merging and mixed RL. The MOPD paper adds that same-origin teachers are "critical for stable optimization".

  • Teachers can be developed in parallel and revised separately.
  • Only the MOPD paper has a controlled comparison, on one model family; Lambert records that labs are "far from unanimous"9.
Assessment

Experts plus MOPD is the 2026 direction (K3, V4, and Xiaomi's MiMo-V2-Flash, the first to document it9), and the mechanism is sound where domain rewards interfere. The evidence is thinner than the adoption: K3 publishes no comparison with its own K2.5-style joint RL, and V4's pipeline mirrors V3.2's except for this stage19, yet the two models also differ in architecture and data, so that comparison is not controlled. Joint RL remains reasonable when the domains share rewards and verifiers, or when nine expert runs are unaffordable. A K3-scale ablation, nine experts against one mixed run at equal RL compute, would settle it; Module 22 sets out the conditions.

Controlling length and effort

Shape the reward inside one policy, or ladder the effort levels

k1.51, K22, K2.5's Toggle3; K34 and DeepSeek-V419

The Kimi reports shorten responses through the reward: an in-group length term, a per-task cap, alternating budget phases. K3 and V4 go further and train separate effort levels, V4 as three modes with distinct length penalties and context windows, K3 as low, high and max experts distilled into one conditioned student.

  • Toggle: 25 to 30% fewer tokens at a "negligible" cost; effort ladders give users a control.
  • No report ablates one form against another; K3's cost claim is on an internal benchmark.

Fix the length bias in the loss instead

Dr. GRPO20 (§3.1); DAPO17 (§3.3)

Dr. GRPO traces part of the length growth to dividing each response's loss by its length, which lets wrong answers grow longer at lower penalty, and removes the normalizer. DAPO moves to a token-level loss and shapes overlong outputs with a soft penalty.

  • Removes a bias instead of adding a term; no new hyperparameter for the budget.
  • Does not give a user-selectable effort, and the bias analysis is for GRPO's normalizer, not Kimi's squared loss.
Assessment

These address different things. Loss fixes remove unintended growth; budgets and effort experts decide how much thinking to buy. K2.5 states why plain budgets fail (they do not scale to more compute), which is the best-argued step in the lineage, and Toggle's token reduction is a measured effect on K2 Thinking. The effort ladder is a product decision more than a training finding until a report compares it with a single budget-conditioned policy. See Module 23 for the arithmetic.

What is changing

Three things in this lineage are moving. The K3 and K2.5 reports are at v2 (7 Aug 2026) and K2.5's printed loss could be corrected; if it is, Table 6's objective row changes. The Kimi line and the GRPO line are converging on a mean-reward advantage with no standard-deviation division but still differ on how to bound drift, and whichever reports an ablation first will move Module 12's comparison. And K3's nine-expert design is one of several 2026 MOPD recipes that Lambert records; the missing piece is a published comparison against joint RL at equal compute9. Module 29 covers Qwen3's route to the same goal for small models, distillation instead of RL, and Module 30 the DeepSeek line.

References

  1. paperKimi Team (Du, A., Gao, B., Xing, B., et al.), Moonshot AI. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv 2501.12599 v4, 3 Jun 2025 (v1 22 Jan 2025). https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: §1, §2.1 to §2.4 (stance, prompt curation, objective, length penalty, sampling, test cases, reward models, long2short); §2.6.2; Tables 2 and 3 (p. 12, page image); Figures 7 to 10; Appendix C (p. 25).
  2. paperKimi Team (Bai, Y., Bao, Y., Charles, Y., et al.), Moonshot AI. Kimi K2: Open Agentic Intelligence. arXiv 2507.20534 v2, 3 Feb 2026 (v1 28 Jul 2025). https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: scale (§1); SFT data pipeline (§3.1.1); gym (§3.2.1); self-critic (§3.2.2); loss and add-ons (§3.2.3, p. 13 page image); infrastructure (§3.3); evaluation (§4.1.1); limitations (§5); Appendix F.
  3. paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.), Moonshot AI. Kimi K2.5: Visual Agentic Intelligence. arXiv 2602.02276 v2, 7 Aug 2026 (v1 2 Feb 2026). https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: zero-vision SFT and transfer (§1, §2.2, §2.3, Table 2 on p. 4); PARL (§3); Eq. 1, rewards, Toggle (§4.4.2, p. 8 page image); swarm results (pp. 2, 11, 12, 14); Appendix D (p. 24).
  4. paperKimi Team (Bai, T., Bai, Y., Bao, Y., et al.), Moonshot AI. Kimi K3: Open Frontier Intelligence. arXiv 2607.24653 v2, 7 Aug 2026 (v1 27 Jul 2026). https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: scale (§1); post-training, experts, partial rollouts, effort RL, judge, MOPD (§4.1, Figure 8 on p. 13 page image, Eq. 15); environments (§4.2); infrastructure and sandboxes (§5.3); results (§6.1, §6.3, §6.4); thinking-effort message (Appendix F, p. 47).
  5. paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948 v2 (expanded Nature version, 4 Jan 2026); v1 22 Jan 2025. https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the v1 date shared with k1.5 and the statement that neural reward models are not applied to reasoning tasks (§2.2).
  6. paperK2 Team (Liu, Z., Tang, L., Jin, L., et al.). K2-V2: A 360-Open, Reasoning-Enhanced LLM. arXiv 2512.06201 v1, 5 Dec 2025. https://arxiv.org/abs/2512.06201. Accessed 2026-10-27. Used for: confirming that "K2-V2" is a different model from a different group than Moonshot's Kimi K2 (abstract page only).
  7. paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv 2402.14740 v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Used for: RLOO and the REINFORCE-over-PPO margin of 3.2% to 20.3% win-rate (§1, §2.3).
  8. paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv 1707.06347 v2, 28 Aug 2017. https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Used for: the clipped surrogate and the suggested ε = 0.2 (§3, Eq. 7).
  9. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv 2504.12501 v12, 11 Sep 2026; web edition last built 14 Oct 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: expert counts for K3, Nemotron 3 Ultra and DeepSeek-V4, MiMo-V2-Flash as the first documented MOPD, and the statement that labs are not unanimous on MOPD (§3.2.4).
  10. paperMa, W., Wei, J., Zhao, L., Zhang, H., et al. (Xiaomi). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv 2606.30406 v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-26. Used for: the comparison against Mix-RL, Cascade RL, off-policy fine-tuning and parameter merging on Qwen3-30B-A3B, the "see-saw effect" and the same-origin-teacher finding (abstract, §1).
  11. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv 2512.02556 v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the advantage defined as reward minus the group mean (§3.1); mixed RL in one stage (§3); off-policy sequence masking and Keep Routing (§3.1); post-training budget above 10% of pre-training cost (§1).
  12. paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv 2402.03300 v3, 27 Apr 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the reasons to drop PPO's value model, a policy-sized network and last-token rewards (§4.1.1).
  13. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv 2411.15124 v5, 14 Apr 2025 (v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: value-model initialization from a general reward model in RLVR PPO (§6.2.1) and value-free GRPO listed as untested (Limitations).
  14. paperXu, S., Fu, W., Gao, J., Ye, W., et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv 2404.10719 v3, 10 Oct 2024. https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Used for: the claim that tuned PPO beats DPO on every testbed (abstract); abstract only.
  15. paperZheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., et al. Group Sequence Policy Optimization. arXiv 2507.18071 v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: the diagnosis of token-level importance weights, the sequence-level ratio and the MoE routing-replay result (§1, §5.3).
  16. paperMiniMax (Chen, A., Li, A., et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv 2506.13585 v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: CISPO, clipping the importance weight rather than the update, and the Qwen2.5-32B controlled comparison (§3.1).
  17. paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv 2503.14476 v2, 20 May 2025. https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: Clip-Higher (§3.1), token-level loss and overlong reward shaping (§3.3, §3.4).
  18. paperGLM-5 Team (Zeng, A., Lv, X., et al.). GLM-5: from Vibe Coding to Agentic Engineering. arXiv 2602.15763 v2, 24 Feb 2026 (v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: three sequential RL stages with on-policy cross-stage distillation (§1, §3).
  19. paperDeepSeek-AI (Xu, A., Lin, B., Xue, B., et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348 v1 (the arXiv API's publication date is inconsistent with the id month). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: the pipeline that mirrors V3.2 with mixed RL replaced by OPD, specialists with GRPO, more than ten teachers, and three effort modes with distinct length penalties (§5.1).
  20. paperLiu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., Lin, M. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv 2503.20783 v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: the response-length bias from 1/|o_i| normalization and Dr. GRPO's constant normalizer (§3.1, §3.2).