Reinforcement Learning for LLM Post-Training · Module 32

Case Study: GLM-5 and Nemotron 3 Ultra: Asynchronous Agentic RL, a Deterministic Indexer Top-k, Unified RLVR over Environments the Report Calls Open, and the Limits of MOPD

Two 2026 reports describe agent-era post-training from opposite ends. GLM-5 runs reasoning, agentic and general RL in sequence, closes with on-policy distillation from its own earlier checkpoints, and found that a non-deterministic top-k kernel in its sparse-attention indexer broke RL within a few steps. Nemotron 3 Ultra runs one unified RLVR stage over fifteen environment groups, merges more than ten specialists by on-policy distillation, and states where that merge stops working. Afterwards you can say what each lab found that the other did not, which of their knobs are disclosed and which are not, and what to audit in an RL rollout path.

Course 50 min read 10 sources As of 2026-10-27 Prerequisites: Module 14, Module 17, Module 18, Module 22
GLM-5 and Nemotron 3 Ultra post-training as two lanes; orange marks the step where separately acquired skills are merged GLM-5: three RL stages in sequence, then distillation SFT interleaved thinking Reasoning RL GRPO + IcePop Agentic RL asynchronous, 10K+ envs General RL rules + ORM + GRM Cross-stage OPD own earlier checkpoints group 32, batch 32 weights pushed every K group 1, batch 1,024 Nemotron 3 Ultra: one RL stage, then ten-plus teachers merged SFT general, two stages Unified RLVR batch 8,192, 16 rollouts MOPD warmup light SFT, once MOPD 1 1,024 prompts x 1 MOPD 2 teachers refreshed More than ten specialist teachers own SFT and RL pipelines, trained in parallel round 2 branches from MOPD 1 integration step
Figure 1. The two recipes as lanes. GLM-5 sequences three RL stages and finishes with on-policy cross-stage distillation from the final checkpoints of earlier stages1 (§1, Figure 5, §3.5). Nemotron 3 Ultra trains a student with SFT and unified RLVR, trains more than ten teachers separately, and merges them in two MOPD iterations after a light warmup SFT2 (§3, Figures 9 and 10). Whether Ultra's 8,192 counts prompts or rollouts is not stated (Section 5.2); GLM-5's batch of 32 has no stated unit either.

1. Two recipes, one frame

1.1 The reports and what they disclose

GLM-5 is a 744B-parameter mixture-of-experts model with 40B active, trained on 28.5T tokens and built on DeepSeek Sparse Attention (DSA)1 (§1, §2.1). Its report is dated 17 Feb 2026, with v2 on 24 Feb 2026. Nemotron 3 Ultra is a 550B total, 55B active hybrid Mamba-attention mixture-of-experts model pre-trained on 20T tokens and posted 12 Jun 20262 (abstract). Ultra says its RL training "largely follows the asynchronous GRPO algorithm" with the stability optimizations of the Nemotron 3 Super report2 (§3.2), so the algorithm's details are in Super3 (§3.2.4), posted 14 Apr 2026. As of 2026-10-27 arXiv lists v2 as the latest GLM-5 version and v1 for both NVIDIA reports4.

Neither report states a total RL budget in GPU-hours or FLOPs in the text searched, and neither tabulates its RL recipe against alternatives. GLM-5's one ablation section compares attention variants (§2.1.2); Ultra's ablations cover MOPD design (the warmup in Table 4, rollouts per prompt in §3.3.1, logit matching in §3.3.5) and the MTP draft length (Figure 12). That sets the evidential ceiling for everything below: two labs describing what they did and what they say worked.

Table 1. The two recipes by the same rows. "Not stated" means the sections named do not give it. Sources: GLM-51, Nemotron 3 Ultra2, Nemotron 3 Super3.
RowGLM-5Nemotron 3 Ultra
Base model744B total, 40B active; 28.5T tokens; DSA indexer with k = 2,0481 (§1, §2.1, §3.2)550B total, 55B active; 20T tokens; hybrid Mamba-attention2 (abstract)
Post-training orderMulti-task SFT; Reasoning RL; Agentic RL; General RL; on-policy cross-stage distillation1 (§1, Figure 5)SFT; unified RLVR; more than ten teachers trained separately; MOPD warmup; two MOPD iterations; MTP boosting2 (Figures 9, 10)
RL algorithmReasoning: GRPO with IcePop, no KL term, on-policy (§3.2). Agentic: group-wise objective with direct double-sided importance sampling (§4.1.2)1Asynchronous GRPO with the stability optimizations of Super23; PPO-style clip around a proximal policy in MOPD2 (§3.3.1)
Group and batchReasoning: group 32, batch 32, unit of the batch not stated. Distillation: group 1, batch 1,024. Agentic: traces per problem and batch not stated1RLVR: global batch 8,192, 16 rollouts per sample, unit of 8,192 not stated. MOPD: 1,024 prompts, one rollout each2 (§3.2, §3.3.1)
EnvironmentsMore than 10,000 SWE environments, terminal tasks, multi-hop search, slide generation1 (§4.2)Fifteen groups from terminal use to structured outputs, with several harnesses per task family2 (§3.2, Appendix A.2)
Skill integrationSequential stages, then distillation from the model's own earlier checkpoints1Unified RLVR first, then specialists as teachers2

1.2 Section map and flags

2. GLM-5: three RL stages and the distillation that closes them

2.1 The SFT starting point

The RL stages start from a multi-task SFT model with a 202,752-token context and three thinking behaviors: interleaved thinking (reason before every response and tool call), preserved thinking (keep thinking blocks across turns in coding-agent use) and turn-level control of whether to think1 (§3.1). One SFT detail anticipates the RL rules of Section 4: in coding and agent trajectories, "erroneous segments" are kept but masked out of the loss, which lets the model "learn error correction behaviors without reinforcing incorrect actions"1 (§3.1).

2.2 Reasoning RL: GRPO, IcePop, no KL

Reasoning RL is GRPO with IcePop to handle the gap between the training policy \(\pi^{train}\) and the inference policy \(\pi^{infer}\), with the KL term removed "to accelerate RL improvement"1 (§3.2). Per token:

$$\mathcal L(\theta)=-\,\mathbb E\Big[\tfrac1G\sum_{i=1}^{G}\tfrac1{|y_i|}\sum_{t=1}^{|y_i|}\mathrm{pop}\big(\rho_{i,t},\tfrac1\beta,\beta\big)\cdot\min\big(r_{i,t}\hat A_{i,t},\ \mathrm{clip}(r_{i,t},1-\epsilon_{low},1+\epsilon_{high})\hat A_{i,t}\big)\Big]$$

where \(\rho_{i,t}=\pi^{train}_{\theta_{old}}/\pi^{infer}_{\theta_{old}}\) is the train-to-inference ratio, \(r_{i,t}=\pi^{train}_{\theta}/\pi^{train}_{\theta_{old}}\) is the usual PPO ratio, \(\mathrm{pop}(\rho,1/\beta,\beta)\) equals \(\rho\) inside the band and \(0\) outside it, and \(\hat A_{i,t}\) is the group-normalized advantage with the standard-deviation division kept1 (Eq. 1). The stated values are \(\beta=2\), \(\epsilon_{low}=0.2\), \(\epsilon_{high}=0.28\), group size 32 and batch size 32, "entirely on-policy". Module 12 (Section 5.2) places this mask among the other log-ratio gates; Module 14 covers where \(\rho\) comes from.

Two readings of the numbers. With \(\beta=2\), a token receives no gradient when its train-to-inference probability ratio leaves \([0.5,\,2]\), that is when \(|\ln\rho|>\ln 2=0.693\) (derived). And if each rollout batch feeds a single update, \(r_{i,t}=1\) exactly and the clip at 0.2 and 0.28 is idle, so the pop mask is the only guard against drift (my reading; the report does not give updates per batch or the unit of "batch size 32").

The data side is prose only. Four domains are mixed, roughly balanced: mathematics, science, code and tool-integrated reasoning. Problems are those GLM-4.7 solves "only rarely or fails consistently" and stronger teachers can solve; domain-specific judges or evaluation systems return binary outcome rewards1 (§3.2). The report claims "stable and significant gains in each domain" without printing them, and prints no run with the KL term kept (the KL question is Modules 6 and 13).

2.3 General RL: three reward types and a human anchor

The third stage targets foundational correctness, emotional intelligence and task-specific quality with a hybrid of rule-based rewards, outcome reward models (ORMs) and generative reward models (GRMs). The report says ORMs give "low-variance signals and high training efficiency" but are "more susceptible to reward hacking", while GRMs are "more robust to such exploitation" but "exhibit higher variance"1 (§3.4). It also adds expert human-written responses as "stylistic and qualitative anchors", on the stated observation that purely model-generated optimization converges toward "recognizably 'model-like' patterns". No reward weights, data sizes or hacking measurements appear in the section (judges are Module 16; the hacking taxonomy is Module 19).

2.4 Cross-stage distillation: the loss and what the report leaves out

GLM-5 ends with "on-policy cross-stage distillation as the final stage", because "sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities". The teachers are "the final checkpoints from the preceding training stages"; prompts are "sampled from the corresponding teachers' RL training sets and mixed in appropriate proportions"1 (§3.5). The loss reuses Eq. 1 with the advantage replaced by a stop-gradient log-ratio (Eq. 2):

$$\hat A_{i,t}=\mathrm{sg}\Big[\log\frac{\pi^{infer}_{teacher}(y_{i,t}\mid x,y_{i,<t})}{\pi^{train}_{\theta}(y_{i,t}\mid x,y_{i,<t})}\Big]$$

Group size is set to 1 and batch size to 1,024 because "it is no longer necessary to maintain a large group of samples per prompt to estimate advantages; the advantage is computed directly from the gap with the teacher models instead". Teacher logits come from the inference engine; the authors plan to move the inference backend into the training engine and use the MQA mode of MLA for inference, so that \(\pi^{infer}\rightarrow\pi^{train}\)1 (§3.5). The labels in Eq. 2 show why: the numerator is scored in the inference engine and the denominator in the training engine, so any engine mismatch enters the advantage as a bias indistinguishable from teacher-student gap (my reading; Module 14 gives the size of such gaps). Module 22 (Section 6.3) treats the objective itself.

Cross-stage distillation in GLM-5: stage checkpoints act as teachers, the student samples its own rollouts, and the advantage is the teacher-student log-ratio Teachers: final checkpoint of each stage SFT Reasoning RL Agentic RL General RL Student rollout y sampled, G = 1 Teacher scoring logits from inference engine Advantage sg[log T - log S] GRPO loss batch 1,024 prefixes update student Dashed: no logits arrow in the report's Figure 5 T = teacher, S = student (training engine)
Figure 2. Wiring of GLM-5's on-policy cross-stage distillation. The log-ratio advantage replaces the GRPO advantage in Eq. 1, with group size 1 and batch size 1,0241 (§3.5, Eq. 2). The arrows from SFT, Reasoning RL and General RL follow the report's Figure 5; the absence of an arrow from Agentic RL is a reading of that figure, not a statement in the text. The teacher is scored in the inference engine and the student in the training engine; the report plans to unify them. The mix of prompts across teachers is not given.

What the report leaves out is the measurement. Its stated motive is cumulative degradation, yet the sections read print no before-and-after scores for this stage, no teacher count, and no statement of whether Agentic RL's skills reach the final model by distillation or only through General RL starting from the agentic checkpoint, which the "sequential" description implies (my reading). Ultra's analogous stage prints scores for every stage on eleven benchmarks (Section 6.3). Any comparison with GLM-5's distillation rests on the report's reasoning, not on a table.

3. A top-k kernel as an RL hyperparameter

3.1 A discrete branch inside attention

DSA adds a lightning indexer to attention. For a query token \(t\) it scores every preceding token \(s\) as \(I_{t,s}=\sum_{j=1}^{H^I} w^I_{t,j}\,\mathrm{ReLU}(\mathbf q^I_{t,j}\cdot\mathbf k^I_s)\), chosen for throughput and implementable in FP8; a selection step keeps the key-value entries with the top-\(k\) scores and attention runs over that set only5 (§2.1, Eqs. 1-2). GLM-5 uses \(k=2{,}048\), with 32 indexer heads of dimension 1281 (§3.2, Table 10). Selection is a discrete function of continuous scores: the attended set changes when two scores swap order. It is the same kind of branch as MoE routing and top-p truncation. If the rollout engine and the trainer disagree on the set, they compute different distributions for the same sampled tokens, and the mismatch of Module 14 appears with no optimizer update at all.

DeepSeek-V3.2's answer to the two discrete branches it met was replay: Keep Routing reuses the sampling-time expert paths in training, and Keep Sampling Mask reuses the top-p and top-k truncation masks5 (§3.1; Module 30 covers both). GLM-5 reports that for the indexer, replay is not the answer.

3.2 What the report states

Table 2. GLM-5's DSA finding, split into what §3.2 states and what it leaves out1. Quoted phrases are the report's.
StatedNot stated in the sections read
The retrieved top-k results "are critical for RL stability", by analogy with MoE routing replayAny measurement of how often the selected set differs between engines
Storing the indexer's indices at every token position is "clearly impractical": \(k=2048\) is "much larger than the k typically used in MoE", with "enormous storage costs" and "significant communication overhead" between enginesByte counts for that storage or traffic (Section 3.4 derives them)
"[O]ther non-deterministic top-k operators (e.g., CUDA or TileLang implementations) caused drastic performance degradation during RL after only a few steps, accompanied by a sharp drop in entropy"Step counts, reward or entropy curves, or which operators were tried beyond those two names
Naive torch.topk is "slightly slower but deterministic", gives "more consistent outputs" and "substantial RL gains"; it is the default top-k operator "in our training engine" in all RL stagesThe size of the slowdown; the size of the gains; which top-k the rollout engine runs after the fix (the comparison names the CUDA operator in SGLang's DSA indexer)
The indexer parameters are frozen by default during RL "to accelerate training and prevent unstable learning in the indexer"Any run with the indexer trained

Two precision points. The fix is stated for the training engine; whether the SGLang rollout path was also changed is not said, so "deterministic top-k fixed the mismatch" should be read as the report's framing, with the operator on each side unspecified. And the report gives no mechanism for the failure. Module 14 (Figure 6) draws the qualitative curves the text describes and labels them a schematic; nothing below adds data to them.

3.3 Two candidate mechanisms (my inference)

Two mechanisms would explain a non-deterministic top-k hurting RL, and the report does not choose between them. First, a call-to-call difference inside one engine: when scores tie at the cut, a parallel selection kernel may return either tied token on different runs with identical inputs. The trainer's two forward passes at the same weights, the one that fixes \(\pi_{old}\) and the one that computes \(\pi_\theta\), can then disagree, so the ratio \(r\) that should be exactly 1 on the first update is not, and the clip and IcePop masks act at random. Second, a cross-engine difference: each engine may break ties, or round scores, differently, which is the \(\rho\) of Section 2.2 acting through the selected set. Ties are plausible after a ReLU with low-precision scores. For scale, DeepSeek-V4 (a different model and indexer) quantized its index scores from FP32 to BF16 and kept a 99.7% recall of the selected KV entries6 (§5.2.1), so a small perturbation changes a fraction of a percent of entries. Figure 3 draws the tie case. A fixed batch scored twice under each operator, with the spread of \(\ln\rho\) and \(\ln r\) logged, would separate the mechanisms; the report shows none.

A tie at the top-k boundary lets two calls with identical inputs select different tokens, giving different attended sets Indexer scores, one query, sorted score 9 10 tie rank by score; k = 9 in this sketch (2,048 in GLM-5) Two calls, same input Call 1 9 Call 2 10 Different attended set different log-probabilities Illustration only: the report says "non-deterministic", not that ties are the mechanism.
Figure 3. Illustration of the tie hypothesis in Section 3.3 (my sketch, not data from the report). Scores at the cut are equal, so a selection kernel that does not fix its tie-break can return token 9 on one call and token 10 on another. GLM-5 reports the outcome of using non-deterministic CUDA and TileLang top-k operators, "drastic performance degradation during RL after only a few steps" with a sharp entropy drop1 (§3.2), and gives no mechanism.

3.4 Why replay was judged impractical: the arithmetic

The report calls replay impractical without giving sizes. They follow from printed quantities. Per token and per layer, MoE routing replay stores 8 expert ids (8 routed experts of 256, Table 10), while indexer replay stores 2,048 indices:

$$\frac{k}{k_{\mathrm{MoE}}}=\frac{2048}{8}=256\times$$

For a sequence of \(L=10^5\) tokens, with \(N_{idx}\) layers that each carry an indexer and \(b\) bytes per index,

$$S_{\mathrm{idx}}=L\,N_{idx}\,k\,b=10^{5}\cdot 78\cdot 2048\cdot 4\ \mathrm{B}\approx 63.9\ \mathrm{GB},\qquad S_{\mathrm{MoE}}=10^{5}\cdot 75\cdot 8\cdot 1\ \mathrm{B}=60\ \mathrm{MB}.$$

Packing each index into 18 bits (derived: \(2^{18}=262{,}144\) exceeds the 202,752-token context) gives 35.9 GB. The ratios to routing replay are about 1,065 times for int32 and 599 times packed. The assumptions: \(N_{idx}=78\), the 3 dense and 75 MoE layers of Table 10 (the text of §2.1 says 80 layers, while Table 10 lists those 78 plus one MTP layer), with an indexer in each. The second assumption is supported by the GLM-5 repository, whose README says GLM-5.2 introduces IndexShare, which "reuses the same indexer across every four sparse attention layers"7; there would be nothing to share if GLM-5 had one indexer in total. For scale, a 576-dimension latent KV entry (the MLA cache size §2.1 discusses) in BF16 is 1,152 B per token per layer, so the latent cache of the same sequence is about 9.0 GB (derived; the dtype is assumed), and the replay payload would be 4 to 7 times larger than the cache it describes. Sharing one selection across four layers would cut it about fourfold, to roughly 16 GB, still about 266 times the routing payload (my extrapolation from the README line).

Derived storage to replay routing ids versus indexer indices for one 100,000-token sequence, on a log axis Replay payload per 100,000-token sequence 0.01 0.1 1 10 100 MoE routing ids 0.06 GB Indexer, 18-bit packed 35.9 GB Indexer, int32 63.9 GB GB per sequence (log axis)
Figure 4. Derived, not reported: bytes needed to replay the discrete selections of one 100,000-token sequence. Routing ids: 100,000 tokens × 75 MoE layers × 8 routed experts × 1 byte (256 experts fit in a byte). Indexer: 100,000 × 78 layers × 2,048 indices at 4 bytes (int32) or 18 bits (packed; 218 = 262,144 exceeds the 202,752-token context). Inputs: k = 2,048 (§3.2) and layer and expert counts (Table 10) of the GLM-5 report1; one indexer per layer is an assumption (Section 3.4). The ratios are about 1,065× (int32) and 599× (packed).

4. GLM-5 agentic RL: asynchrony as a set of hygiene rules

4.1 The objective and what it ignores

For each problem \(x\), GLM-5 samples \(K\) agent traces from the previous policy and optimizes a group-wise objective whose advantage is the trace reward minus the group mean reward, \(r(x,y_i)-\bar r(x)\); "only model-generated tokens are used for optimization, and the environment feedback is ignored in loss computation"1 (§4.1). Unlike the reasoning loss of Section 2.2, the printed agentic advantage has no standard-deviation division (Module 11 covers the choice), and the printed objective carries only the advantage term; the policy term arrives in Eq. 3 below. The report uses \(K\) both for traces per problem and for the weight-push interval below; neither value is given. Module 18 covers the masking of environment tokens and the multi-turn credit problem; this section is about what the report builds around the objective so that a learner can be fed by long, uneven rollouts.

4.2 The loop, one stage at a time

Synchronous RL "suffers from severe GPU idle time during long-horizon agent rollouts"1 (§3.3), so the report decouples the engines onto different GPUs. The inference engine generates continuously; once the number of trajectories reaches a threshold, the batch goes to the trainer. Figure 5 steps through it.

GLM-5's asynchronous agentic RL loop: task services, orchestrator, token-in-token-out gateway, inference engines, filtered buffer, trainer, and weight push Task services SWE, terminal, search Orchestrator multi-task, server-based TITO gateway token IDs in and out Inference engines continuous generation Buffer and filters Trainer per-task ratio, speed over 1k concurrent rollouts records token IDs rollout ID to one DP rank batch at threshold drop: version gap above tau drop: sandbox failures mask outside the band ratio uses rollout log-probs weights every K updates optimizer reset
Figure 5. GLM-5's asynchronous agentic RL loop, stepped through. Components and rules from §4.1.1 and §4.1.2 of the report1; the values of \(K\), \(\tau\), \(\epsilon_\ell\) and \(\epsilon_h\) are not stated. The drawn layout is mine.
  1. Register and balance. Each task is a microservice with its own rollout and reward logic, registered with the orchestrator, which sets the per-task rollout ratio and generation speed and standardizes every trajectory into one message-list format. The report cites more than 1,000 concurrent rollouts.
  2. Record the tokens. The TITO gateway intercepts every generation request and stores token IDs and metadata, so the learner never re-tokenizes text. Requests from one rollout are hashed to one data-parallel rank so the shared prefix hits the KV cache.
  3. Filter the batch. Once enough trajectories exist the batch moves on. A sample is dropped if its oldest weight version is more than \(\tau\) behind the current one, or if its sandbox failed; a group with more than half valid samples is padded by repeating them, otherwise dropped.
  4. Update with a double-sided mask. The ratio is the current policy over the logged rollout probability; tokens outside \([1-\epsilon_\ell,\,1+\epsilon_h]\) get no gradient; no old-policy checkpoints are tracked.
  5. Push the weights. Every \(K\) gradient updates the weights return to the inference engines and the optimizer is reset, because the new weights pose a different optimization problem.

Module 14 (Section 2.5) explains why the token-in-token-out gateway matters, and Module 18 (Section 4) why a trajectory spanning several weight versions needs per-token behavior probabilities. The point of this section is the combination, written as a ledger of the knobs and which of them the report gives.

Table 3. GLM-5's agentic-RL mechanisms, the failure each targets, and whether the value is printed1 (§4.1.1, §4.1.2, §3.6.2).
MechanismFailure it targetsValue
Weight push every \(K\) updates, optimizer resetPolicy lag; the new weights pose a different optimization problem\(K\) not stated
TITO gatewayRe-tokenization changes boundaries, whitespace, truncation or special tokens and misaligns actions and advantagesNo measurement
Direct double-sided importance samplingTracking \(\pi_{old}\) across many weight versions per trajectory is "computationally prohibitive"\(\epsilon_\ell\), \(\epsilon_h\) not stated
Stale-sample drop: discard if \(w'-w_0>\tau\)Overlong trajectories "become highly off-policy"\(\tau\) not stated
Environment-failure drop; pad the group if more than half is valid, else drop itSandbox crashes unrelated to the model add reward noiseThreshold stated: half the group
DP-aware routing by consistent hashingRepeated prefill across data-parallel ranks in multi-turn rolloutsNo speedup printed
Orchestrator ratios and speedsHeterogeneous tools and rollout times across tasksOver 1k concurrent rollouts
Tail-latency stack in slime: multi-node serving (EP64, DP64 over 8 nodes), FP8 rollouts, MTP, prefill-decode disaggregation, heartbeat fault toleranceOne straggler stalls the stepConfigurations named, no latency numbers

4.3 What Eq. 3 optimizes

The report writes the token-level objective as \(L(\theta)=\mathbb E_t[f(r_t(\theta),\epsilon_\ell,\epsilon_h)\,\hat A_t\log\pi_\theta(a_t\mid s_t)]\) with \(r_t=\exp(\log\pi_\theta-\log\pi_{rollout})\) and \(f(x)=x\) for \(1-\epsilon_\ell<x<1+\epsilon_h\), else 01 (§4.1.2, Eqs. 3-5). It prints no stop-gradient on \(f\). Read as a constant weight, as Module 12 does, the gradient is

$$\nabla_\theta L=\mathbb E_t\Big[\mathbb 1\{1-\epsilon_\ell<r_t<1+\epsilon_h\}\;r_t\,\hat A_t\,\nabla_\theta\log\pi_\theta\Big]=\mathbb E_t\Big[\mathbb 1\{\cdot\}\,\hat A_t\,\frac{\nabla_\theta\pi_\theta(a_t\mid s_t)}{\pi_{rollout}(a_t\mid s_t)}\Big],$$

the off-policy policy gradient with the logged rollout probability as behavior policy, restricted to in-band tokens. Because \(\pi_{rollout}\) is whatever the engine held when it sampled each token, one stored number per token serves however many weight versions a trajectory crossed. The report says reusing rollout log-probabilities "accepts a controlled degree of off-policy bias" to avoid historical tracking and that this improves stability; it prints no comparison against tracking \(\pi_{old}\).

4.4 Environments, in four lines

Module 17 covers construction and validity checks; the sizes the report prints are these. SWE: more than 10,000 verifiable environments across thousands of repositories and nine languages, with fail-to-pass and pass-to-pass tests extracted by LLM-written log parsers (§4.2.1). Terminal: thousands of Harbor-format tasks with Docker construction accuracy "exceeding 90%" (§4.2.2). Search: a web knowledge graph over more than two million pages, and a filter that removes any question a tool-free reasoning model answers in at least one of eight attempts (§4.2.3). Slides: a three-level HTML reward whose renderer was patched after reward hacking, with 16:9 compliance rising from 40% to 92% (§4.2.5; Module 19, Section 4.3)1. The report also describes a keep-recent-\(k\) context rule for search agents (\(k=5\), BrowseComp from 55.3% to 62.0%) in its inference section (§4.2.4); whether training rollouts used it is not stated.

5. Nemotron 3 Ultra: one unified RLVR stage

5.1 From Super to Ultra: two RL stages become teacher recipes

Ultra's post-training is "substantially redesigned from the pipeline used for Nemotron 3 Super"2 (§3). Super ran three RL stages after SFT, multi-environment RLVR, a separate end-to-end SWE-RL stage and RLHF, then healed its MTP heads3 (§3.2). It split SWE-RL out because "SWE rollouts are substantially slower to generate and typically require longer context lengths, creating a throughput bottleneck when co-trained with shorter-horizon environments". Ultra keeps the unified first stage. The other two stages survive as teacher recipes: its SWE teacher ends in end-to-end SWE-RL, and its chat teacher is trained by RLHF against a generative reward model2 (§3.3.2). The mapping is my reading; the report does not draw it. The rollout-heterogeneity problem behind Super's separate stage returns as Ultra's stated limit in Section 6.4.

Table 4. The RL stage of Nemotron 3 Super and Nemotron 3 Ultra by the same rows. Sources: Super3, Ultra2.
RowNemotron 3 Super (14 Apr 2026)Nemotron 3 Ultra (12 Jun 2026)
RL stages after SFTMulti-environment RLVR; end-to-end SWE-RL; RLHF; MTP healing (§3.2)Unified RLVR, then MOPD over teachers whose recipes include SWE-RL and RLHF (§3.2, §3.3)
Environments21 environments, 37 datasets (§3.2.1)Fifteen groups; environment count not stated (§3.2)
Per step256 prompts, 16 responses per prompt, batch 4,096, one gradient update per rollout (§3.2.4)Global batch 8,192, 16 rollouts per sample, unit of 8,192 not stated (§3.2)
Maximum generation49K tokens, later 64K (§3.2.4)48K tokens, later 64K (§3.2); 48 × 1,024 = 49,152, so probably the same value (my reading)
Effort mode in RLLow-effort on 2% of RL prompts, later 1%; reward depends on correctness and token count (§3.2.1)Medium-effort on about 2.5% of RLVR prompts; length-based reward adjustment, form not disclosed (§3.5)
SWE harnessesOpenHands loop with OpenCode and Codex agent classes that imitate Claude Code and Codex CLI tool formats (§3.2.2)At least two of six harness families per task vertical (Appendix A.2)

5.2 The algorithm, and a batch whose unit is not stated

Super's algorithm section is the specification Ultra points to. Training and inference run on separate GPUs; workers fill a rollout buffer; a batch trains once enough trajectories exist, "a single gradient update per rollout"; updated weights are pushed "as soon as a new model version is available", so one trajectory can hold tokens from different versions and the KV cache is not recomputed after an update; workers stay "at most one step behind"; and "we mask the importance sampling ratio computed from the training and inference logprobs"3 (§3.2.4, §3.2.5). Ultra's MOPD loss names the same masking "IcePop strategy"2 (§3.3.1), which is what GLM-5's reasoning stage uses (Section 2.2). Two labs and three reports therefore converge on masking tokens whose train-to-inference ratio leaves a band (Module 14 covers the design space). Neither NVIDIA report prints the band in the sections read.

5.3 Fifteen environment groups and several harnesses

The unified stage spans "terminal usage, office and productivity workflows, software engineering, search, general tool-calling, math, code, STEM, safety, chat, instruction following, long-context QA, inductive and transductive reasoning, structured outputs, and general model usability": fifteen groups by my count2 (§3.2). For harness-based environments the training data use "a diverse collection of harness implementations and interaction formats", to reduce "overfitting to any particular harness design". The report profiles rewards before training and takes its mixture and curriculum from "the Gaussian-based approach introduced in NVIDIA (2025b)", an earlier report I did not read; Super describes dropping prompts the SFT model answers correctly "consistently" and sorting the rest by difficulty3 (§3.2.1). Appendix A.2 sorts the task distributions into five verticals (terminal use and software engineering, bug fixing in existing repositories, office productivity, general knowledge, search) and trains each under at least two of Stirrup, OpenHands, OpenCode, Terminus, Droid and internal harnesses2; which harnesses serve which vertical is not stated. Module 17 covers the environment library.

The same appendix prints the only per-harness numbers, as the cell labels of Figure 17. Table 5 transcribes them from a 400 dpi crop; the printed Avg column reproduces from the seven cells in all six rows (derived).

Table 5. Accuracy (%) by agent harness, from Figure 17 of the Nemotron 3 Ultra report2 (Appendix A.2). Top block: SWE-bench Verified. Bottom block: Terminal-Bench 2.1, whose first column is Terminus 2.0.0 instead of Mini SWE Agent 2.3.0. GLM-5.1 and Kimi K2.6 are the report's comparison models. Highlighted cells are the best in their column; Ultra's lowest cells are marked as worst.
ModelMini SWE / TerminusOpenCodePiClaudeHermesOpenHandsCodexAvg
SWE-bench Verified
GLM-5.171.374.874.474.474.573.773.673.8
Kimi K2.670.771.173.774.272.573.365.871.6
Nemotron 3 Ultra65.067.370.460.369.970.321.160.6
Terminal-Bench 2.1
GLM-5.159.361.861.451.971.355.551.258.9
Kimi K2.667.251.758.057.561.463.848.358.3
Nemotron 3 Ultra55.146.752.147.252.852.635.548.9
Per-harness scores of GLM-5.1, Kimi K2.6 and Nemotron 3 Ultra on SWE-bench Verified and Terminal-Bench 2.1, one dot per harness SWE-bench Verified 20 30 40 50 60 70 80 GLM-5.1 Mini SWE 71.3 OpenCode 74.8 Kimi K2.6 Codex 65.8 Claude 74.2 Ultra Codex 21.1 Pi 70.4 score (%) Terminal-Bench 2.1 20 30 40 50 60 70 80 GLM-5.1 Codex 51.2 Hermes 71.3 Kimi K2.6 Codex 48.3 Terminus 67.2 Ultra Codex 35.5 Terminus 55.1 score (%)
Figure 6. Each dot is one agent harness; the labelled dots are the lowest and highest per model. Values were read from the cell labels of Figure 17 in the Nemotron 3 Ultra report2 (Appendix A.2, page 65) at 400 dpi; the printed Avg column reproduces from the seven cells in all six rows. "Ultra" is Nemotron 3 Ultra; GLM-5.1 and Kimi K2.6 are the comparison models in that figure, not the GLM-5 of the GLM-5 report. On SWE-bench Verified Ultra's range is 49.3 points (10.1 without the Codex harness), against 3.5 for GLM-5.1; on Terminal-Bench 2.1 all three models span 19 to 20 points.

Three observations, all derived from the table. First, Ultra is below both comparison models in every one of the 14 cells: by 3.4 to 52.5 points against GLM-5.1 on SWE-bench Verified and 2.9 to 18.5 on Terminal-Bench 2.1. The introduction claims "on-par accuracy across a wide range of agentic and reasoning benchmarks" at 5.9 times GLM-5.1's throughput2 (§1); that is a claim about a suite and a throughput point, and Figure 17 is the same report's finer view. Second, harness spread is large: 19 to 20 points for every model on Terminal-Bench 2.1, and for Ultra on SWE-bench Verified a range of 49.3 points of which the Codex harness (21.1) contributes 39.2. The report gives no reason. Third, membership in the training list does not sort the scores. Pi, which no training list in the sections read names, is Ultra's best SWE-bench Verified cell (70.4), above OpenHands (70.3) and OpenCode (67.3), both named in Appendix A.2; Claude and Codex, also unnamed, are its two lowest. No single-harness training run is reported, so the figure cannot say whether diversity helped, and Module 17's held-out-harness experiment still needs that control.

GLM-5 reports a smaller harness check. On Terminal-Bench 2.0 it scores 56.2 under Terminus-2 and 56.2 under Claude Code (60.7 and 61.1 on the verified version), while GLM-4.7 scored 41.0 and 32.8, an 8.2-point gap; the text says GLM-5 "shows consistent performance across both frameworks"1 (§6.1.2, Table 7). Whether GLM-5's agentic RL used more than one harness is not stated; the orchestrator standardizes trajectories into "a unified message-list representation", which is a data format, not harness variety (§4.1.1).

6. Nemotron 3 Ultra: from unified RLVR to MOPD, and where it stops

6.1 Why a second stage, and who the teachers are

Ultra's reason for MOPD is dilution: "as the number of environments continues to grow, each domain contributes only a relatively small number of samples to any given training batch, diluting the per-domain learning signal and making it increasingly difficult to balance training across domains". The remedy is "more than ten specialized teacher models, each optimized through its own domain-specific training pipeline"; the student, "obtained from RLVR", "generates rollouts across all domains and receives dense reward signals from the corresponding teacher models"2 (§3.3). Figure 10 of the report draws ten first-round teachers, then five new or refreshed second-round teachers beside five reused ones (my count), with the RLVR student as self-teacher where no specialist covers a domain. Module 22 (Sections 4 and 6) covers the objective, the warmup SFT and the same-origin condition. Table 6 lists how each teacher is trained.

Table 6. Ultra's specialist teachers as described in §3.3.22. "Rollout shape" is my classification from the description; "not stated" means the paragraph gives no rollout detail.
TeacherTraining recipe as describedRollout shape
Software engineeringSFT on agentic data; PivotRL on single-step environments; then end-to-end SWE-RL: hidden-test binary reward in GRPO, 192K generation, at most 200 agent turns, unfinished trajectories masked, two gold-patch leak channels closed (Modules 18, 19)End-to-end, multi-turn
Office and workplaceAfterQuery tasks sharing structure with GDPval; strong-model rollouts; light SFT on the student, then PivotRL inside the MOPD stageTurn-local (pivots)
SearchSFT on trajectories with context management (discard-all resets, summary compression); no RL stage describedSFT only
Terminal useExpert trajectories for tasks with timeouts up to one hour; PivotRL, re-profiling when accuracy saturatesTurn-local (pivots)
Conversational tool useSuper's data and recipe with PivotRL, expanded to dependent multi-step actionsTurn-local (pivots)
Model usabilityStructured outputs (five schema types), document extraction, citation and free-form formatting; environments in NeMo GymNot stated
Agentic safetyIndirect prompt-injection tasks, red-teamed against a defender model; a deterministic verifier checks whether the target tool was calledNot stated
ChatUltra-based principle-following generative reward model; several RLHF iterations with targeted data after eachNot stated
Instruction following and factualityDomain-focused RLVR on the RL checkpoint with instruction-following, abstention and RLHF environments; abstention reward "dynamically calibrated" for accuracy against hallucinationNot stated
General reasoning (STEM)SFT on 40B tokens, 58.75% science, generated with DeepSeek-V4-Pro; then RL on non-STEM prompts, 128 prompts per batch, global batch 2,048Single-turn reasoning
Competitive codingRL on the general reasoning teacher; prompts it solves 8 of 8 are removed, leaving 3.5K; +2.4 on LiveCodeBench v6Single-turn reasoning

Read as a whole (derived), one of the eleven recipes ends in end-to-end multi-turn RL, three use PivotRL, and one is SFT-only. The teachers' own training mostly avoids long end-to-end agentic rollouts too, which is the context for the limit in Section 6.4. Module 19 covers the SWE teacher's leak channels and Module 24 the calibration questions raised by the abstention reward.

6.2 MOPD in four reports

Table 7. How four 2026 recipes place distillation relative to RL. GLM-51 (§3.5), Nemotron 3 Ultra2 (§3.3), DeepSeek-V46 (§5.1), Kimi K38 (§4.1). "Not stated" means the sections named do not say. Lambert's survey lists Ultra at more than ten experts, K3 at nine and V4 at more than ten, and groups GLM-5 with MOPD recipes9 (§3.2.4).
ReportTeachersPer-token signalBatchTiming relative to RLAgentic rollouts
GLM-5Final checkpoints of earlier stages (SFT, Reasoning RL and General RL named); count not statedSampled-token log-ratio as the GRPO advantageGroup 1, batch 1,024As the final stage, after all three RL stagesNot stated
Nemotron 3 UltraMore than ten, trained in parallel; second round re-branched from MOPD 1Sampled-token log-ratio against a proximal policy, PPO clip, IcePop mask1,024 prompts, one rollout each; up to 192K tokensAfter unified RLVR and a light warmup SFT; two iterationsSingle-turn, PivotRL-style, for most agentic tasks
DeepSeek-V4More than ten domain teachers, weighted \(w_i\) by importanceFull-vocabulary reverse KL, because the token-level estimate has "high variance" and "training instability"Not statedReplaces the mixed RL stage entirelyNot stated
Kimi K3Nine: three domains at three effort levelsPer-token clipped log-ratio; top-k variants showed "no clear advantage"Not statedAfter SFT and expert RLLong-horizon kept: dense reward "naturally enabl[es]" partial rollouts

The columns that differ most are the signal and the agentic rollouts. Three of four use the cheap sampled-token estimate; only V4 pays for the full vocabulary, while Ultra reports that logit matching "consistently underperformed the sampled-token objective on some agentic benchmarks such as Terminal Bench"2 (§3.3.5; Module 22, Section 5). GLM-5's teachers are same-origin by construction; Ultra's first-round teachers came from different SFT pipelines and needed the warmup. On agentic rollouts Ultra avoids end-to-end rollouts, K3 keeps long-horizon ones, and GLM-5 says nothing.

6.3 What each stage added

Ultra's Table 5 prints SFT, RLVR, MOPD 1, MOPD 2 and teacher scores on eleven benchmarks, and a recovery column defined as \((\text{MOPD2}-\text{RLVR})/(\text{Teacher}-\text{RLVR})\)2 (§3.3.4). I recomputed all eleven recovery values from the printed scores; all eleven match to the printed precision (derived), so the column can be used as printed. Module 22 (Figure 4) plots recovery. Figure 7 plots what it does not: the points each stage contributed.

Points added on each of eleven benchmarks by the unified RLVR stage and by the two MOPD iterations in Nemotron 3 Ultra RLVR minus SFT MOPD 2 minus RLVR 0 10 20 30 40 Terminal Bench 2.0 +10.0 +9.5 GDPVal +5.7 +17.8 SWE-Bench Verified +2.3 +5.9 TauBench Telecom +27.0 +10.2 BrowseComp +16.7 +13.4 LiveCodeBench v6 +1.9 +1.6 IMOAnswerBench -0.6 +4.1 HLE (no tools) +5.9 +1.1 OmniScience non-halluc. +41.5 +32.4 IFBench +16.1 +3.3 Multi-Challenge +7.0 +3.5 benchmark points added by the stage
Figure 7. Points added by each stage, derived from the scores printed in Table 5 of the Nemotron 3 Ultra report2 (§3.3.4): blue is RLVR minus SFT, orange is MOPD 2 minus RLVR. MOPD adds more than the first RL stage on three benchmarks (GDPVal, SWE-Bench Verified, IMOAnswerBench) and less on eight. Not a controlled comparison: the MOPD gain includes whatever the teachers' own SFT and RL pipelines added. OmniScience non-hallucination is a non-hallucination score; the report prints no accuracy counterpart.

Three readings, all derived. The first RL stage contributes more than MOPD 2 on eight of eleven rows; MOPD contributes more on GDPVal (+17.8 against +5.7), SWE-Bench Verified (+5.9 against +2.3) and IMOAnswerBench (+4.1 against -0.6). The three largest MOPD gains, OmniScience non-hallucination (+32.4), GDPVal (+17.8) and BrowseComp (+13.4), are the rows where the RLVR student sat furthest below its teacher (40.7, 20.6, 20.0 points). And MOPD 2 beats its own teacher on two rows, Terminal Bench 2.0 (54.0 against 50.0) and Multi-Challenge (63.8 against 63.3), and trails it on the other nine; the report suggests the Terminal Bench case, on data-science tasks, is "potential knowledge transfer" from the office teacher.

6.4 The reported limit: rollouts that differ in length

Ultra lists three unresolved MOPD problems. Two are about teachers (logit matching, and giving teacher and student a shared SFT stage first). The third is about rollouts: "When mixing end-to-end agentic environments with reasoning environments in MOPD, we observed substantial training inefficiency because rollout times can differ dramatically. Balancing efficiency and accuracy requires sophisticated training infrastructure together with careful asynchronous algorithm design. In practice, we use single-turn rollouts, similar to PivotRL (Yi et al., 2026), for most of the agentic tasks. This approach performs relatively well, but whether end-to-end rollouts can yield further gains, and how to make it robust to potential distribution mismatch (Wang et al., 2026a), remains an open area for exploration"2 (§3.3.5). The sentence states a cost and a choice, not a comparison: the sections read contain no run of end-to-end agentic MOPD against the single-turn version.

The cost is a straggler cost, and the report's own infrastructure numbers show how it enters. During RL and MOPD the pipeline overlaps rollout generation with the policy update, so "the step time is bounded by whichever stage is slower", typically generation, "dominated by a small fraction of straggler generations"2 (§3.6.1). With \(T_{step}=\max(T_{gen},T_{train})\) and \(a=T_{gen}/T_{train}\ge1\), a generator made 1.46 times faster (the measured MTP gain at \(k=5\), concentrated on the slowest trajectories) shortens the step to \(\max(T_{gen}/1.46,\,T_{train})\):

$$\text{speedup}_{\text{overlapped}}=\min(1.46,\;a),\qquad \text{speedup}_{\text{sequential}}=\frac{a+1}{a/1.46+1}.$$

At \(a=1.2\) the overlapped pipeline gains 1.2 times, not 1.46; at \(a=3\) it gains the full 1.46 (derived; Figure 8). Two consequences. In a generation-bound pipeline every added second of straggler time lands in the step time one for one, which is how a few 200-turn, 192K-token SWE rollouts in a batch of single-turn reasoning rollouts can idle a trainer (my reading of the report's statement). And rollout-side speedups (MTP, FP8, prefill-decode splitting, as GLM-5 also uses1 (§3.6.2)) are worth their full value only while \(a\) stays above the speedup. The remedy Ultra chose changes the work, not the engine: shorter, turn-local rollouts (my reading).

Step-time speedup from a 1.46 times faster rollout generator, as a function of generation time over training time, for an overlapped and a sequential pipeline 1 2 3 4 5 1.0 1.2 1.4 1.6 generator speedup 1.46 knee at 1.46 overlapped: max(gen, train) sequential: gen + train a = generation time / training time step-time speedup
Figure 8. Derived from one reported number. Nemotron 3 Ultra measured 1.46× faster mean rollout generation with MTP speculative decoding at k = 5 and runs a one-step off-policy asynchronous setup in which the step time is set by the slower stage2 (§3.6.1, Figure 12). With \(a=T_{gen}/T_{train}\), the overlapped speedup is \(\min(1.46,\,a)\) for \(a\ge1\); the sequential speedup is \((a+1)/(a/1.46+1)\). The report says generation is typically the slower stage but gives no value of \(a\).

Kimi K3 keeps long-horizon rollouts inside distillation by pausing generation once a fraction of trajectories has finished and resuming the rest next iteration8 (§4.1.2, §4.1.3), trading the straggler cost for staleness (Module 18, Section 4); GLM-5 states no agentic distillation design. The agentic rows of Ultra's Table 5 recover 67.0% to 172.7% of the teacher gap, with single-turn rollouts for most agentic tasks, and the report reads MOPD as most effective "when the teacher's advantage can be expressed as token-level preferences over trajectories that the student is already able to sample, such as tool-use decisions, environment interactions, abstention behavior, and multi-step execution patterns"2 (§3.3.4). That supports "good enough" on these benchmarks, not "no gain left".

7. What each lab found that the other did not

The reports share more than they differ in: asynchronous RL, ratio masking, distillation after RL, no RL budget total, no ablation of the KL removal. Table 8 lists what is one-sided.

Table 8. One-sided findings. "Evidence" says what the claim rests on. Sources: GLM-51, Nemotron 3 Ultra2, Nemotron 3 Super3.
TopicGLM-5Nemotron 3 UltraEvidence
A discrete selection in the rollout pathNon-deterministic top-k in the DSA indexer degraded RL within a few steps; torch.topk in the training engine fixed itNo comparable finding in the sections read; no indexer in the architectureDescription, no curves
Teacher-student supportSame-origin teachers by construction; no warmup mentionedTeachers from different SFT pipelines "cannot be effectively combined" by a plain merge; a light warmup SFT lifts GDPVal from 35.3 to 46.7 and BrowseComp from 33.0 to 44.4, HLE only from 26.3 to 26.7 (Table 4)Ablation, one run per cell
Measuring the integration stageNo scores for cross-stage distillation in the sections readScores for SFT, RLVR, MOPD 1, MOPD 2 and teacher on eleven benchmarks (Table 5)Ultra measured; GLM-5 asserts
Harness evidenceOne pair: Terminal-Bench 2.0 under two harnesses, 56.2 and 56.2Seven harnesses on two benchmarks; at least two harnesses per vertical in trainingMeasured; no single-harness control
KL to referenceRemoved from the policy loss "to accelerate RL improvement"No KL term mentioned in Super's algorithm text or Ultra's RLVR description; KL to a teacher is the MOPD objectiveNeither prints an ablation

7.1 What is released

Ultra's report says it is "open-sourcing the training recipes, data, and RL environments" with the checkpoints, including a 550B-A55B GenRM2 (§1). The Nemotron repository README, fetched 2026-10-27, lists the Ultra program as "Pretrain → SFT → RLVR → MOPD → MTP Boosting (this recipe covers Pretrain → SFT)" and says the recipes "train exclusively on the open-sourced subset of training data. Results will differ from the tech report benchmarks, which used additional proprietary data"10. The RLVR and MOPD stages this module is about are named there but not in that recipe; the report's "RL environments" may live in the NeMo Gym and NeMo RL repositories, which I did not inspect. Super's recipe entry lists "3× RLVR + 2× SWE-RL + RLHF across 21 reward environments"10. GLM-5's repository README links slime as "the reinforcement learning framework used by the GLM team" and lists weights; it states no release of RL data or environments7.

Where experts disagree

Neither report is an experiment. Each states a design and its authors' judgment that it worked, so the disagreements are between designs, and each block ends with what would settle it.

A. How should separately trained skills end up in one model?

A1. Sequential stages, healed by distilling from earlier checkpoints

GLM-51 (§3.5)

Run reasoning, agentic and general RL in order, then distill from the final checkpoints of earlier stages to undo "cumulative degradation".

  • Needs only checkpoints the run already produced; teachers are same-origin, so no warmup.
  • Stages stay serial; no scores printed for the healing step; teacher count and prompt mix not given.

A2. Unified RL first, specialists as teachers second

Nemotron 3 Ultra2 (§3.2, §3.3)

Train one student on all environments, then close per-domain gaps with more than ten teachers trained in parallel.

  • Measured: 67.0% to 172.7% recovery on the agentic rows of Table 5.
  • Teachers train in parallel and can be refreshed.
  • Pays for both stages; HLE recovers only 16.9%, which the report attributes to capabilities the student rarely samples.

A3. One mixed stage, or distillation replacing it

DeepSeek-V3.2 mixed GRPO5 (§3); DeepSeek-V4 replaced it with OPD6 (§5.1); Module 30

V3.2 merged reasoning, agent and alignment RL into one stage to avoid forgetting; V4 dropped that stage for distillation.

  • One trainer, one recipe.
  • Ultra's dilution argument applies; V4 gives no ablation for its reversal.
Assessment

The reports do not measure the same thing on the same base, so neither "A1 is enough" nor "A2 is better" is established. Under few domains with a natural order, A1 costs least. Under many environments of different rollout shape, Ultra's dilution argument is sound and its Table 5 is the only measurement of a merge stage in these two reports. What would settle it is one SFT checkpoint taken through both routes at matched GPU-hours over the same domains, scored per domain (Module 22 reaches the same verdict).

B. What closes the training-inference gap?

B1. Mask or correct in the loss

GLM-5 reasoning RL1; Nemotron 3 Super3; Ultra's MOPD2

Drop or reweight tokens whose train-to-inference ratio is out of band.

  • Engine-agnostic; one band to tune.
  • Discards tokens; the NVIDIA bands are not printed.

B2. Make the two engines agree

GLM-5's deterministic top-k1 (§3.2); Kimi K3's shared quantization scheme8 (§4.1.4)

Remove the cause: same operator, same numerics on both sides.

  • Fixes what a mask cannot: GLM-5 reports collapse with non-deterministic top-k.
  • Costs speed ("slightly slower"); per-operator engineering; no measured comparison.

B3. Replay the discrete state

DeepSeek-V3.2's Keep Routing and Keep Sampling Mask5 (§3.1)

Reuse the sampling-time choice in training.

  • Exact for small state such as 8 expert ids per layer.
  • For the indexer the derived payload is hundreds of times larger (Section 3.4); GLM-5 rejects it.
Assessment

They are layers, not rivals. GLM-5 runs B1 and B2 together; B3 fits only where the state is small. If IcePop was active in GLM-5's failed top-k runs (the report does not say), the finding shows masking alone cannot absorb this failure. A run that logs \(\ln\rho\) and \(\ln r\) under both operators would settle which mechanism of Section 3.3 operates.

C. Which rollouts should agentic training consume?

C1. Turn-local, single-turn rollouts

Nemotron 3 Ultra2 (§3.3.5); Super3 (§3.2.4)

Use PivotRL-style turns for most agentic tasks.

  • No stragglers; the agentic rows still recover 67.0% to 172.7%.
  • Untested against end-to-end; distribution mismatch is listed as open.

C2. End-to-end, balanced by an orchestrator

GLM-51 (§4.1)

Train on whole trajectories, with per-task rollout ratio and speed control and staleness hygiene.

  • Trains the distribution that is deployed.
  • Needs the Table 3 rules; no throughput figures printed.

C3. Partial rollouts and a staleness-tolerant loss

Kimi K38 (§4.1.2, §4.1.3)

Pause at a fraction of finished trajectories and resume the rest later.

  • Keeps the long tail in the batch.
  • Trajectories span policy versions; tolerance is asserted.
Assessment

Unsettled: no report compares the three. Figure 8 gives the way to choose: if generation is the slower stage, stragglers lengthen every step, which favors C1 or C3; if training dominates, they are hidden and C2's fidelity is cheaper. Measuring \(T_{gen}/T_{train}\) on your own rollout mix comes before choosing. The experiment that would settle the quality question is Ultra's own open item: end-to-end against single-turn distillation on the same teachers.

D. Should the policy loss keep a KL term to a reference?

D1. Drop it

GLM-5 reasoning RL1 (§3.2); no KL term appears in Super's algorithm text or Ultra's RLVR description3

GLM-5 removes it "to accelerate RL improvement".

  • Faster movement; no reference-model forward pass.
  • No printed ablation; no restraint where rewards are soft.

D2. Keep a small, corrected term, tuned by domain

DeepSeek-V3.25 (§3.1)

V3.2 corrects the estimator with an importance ratio and says that for mathematics "a relatively weak KL penalty or even omitting it entirely can yield improved performance".

  • A leash where a judge can be exploited.
  • One more knob per domain; no values printed.
Assessment

The reports split by reward type, and V3.2's own sentence agrees. GLM-5 states the removal for reasoning RL only; whether its General RL stage, whose rewards partly come from reward models the report itself calls hackable (§3.4), keeps a KL term is not stated. The run that would settle it is General RL with and without the term, scored for reward hacking; neither report has it (Modules 6 and 13).

What is changing

Both reports have successors. The GLM-5 repository's README now lists GLM-5.1, 5.2 and 5.3, and says GLM-5.2 shares one indexer across every four sparse-attention layers7; if that carries to the selections, the replay problem of Section 3.4 shrinks about fourfold without vanishing. I did not read those reports. Ultra's own figures already compare against GLM-5.1 and Kimi K2.62. The README of the Nemotron repository lists a 3.5 Super recipe with two-phase MOPD10, which I also did not read. Every number here is from a v1 or v2 report that may be revised; MOPD is a family of recipes, not a specification (Module 22).

References

  1. paperGLM-5 Team (Zeng, A., Lv, X. et al.; Zhipu AI and Tsinghua University). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2, 24 Feb 2026 (v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: §3.1 SFT; §3.2 reasoning-RL loss (Eq. 1), hyperparameters and the DSA top-k finding; §3.4 General RL; §3.5 cross-stage distillation (Eq. 2); §3.6 slime; §4.1 asynchronous agentic RL, TITO, double-sided importance sampling (Eqs. 3-5); §4.2 environments; §6.1.2 and Table 7 (Terminal-Bench 2.0 under two harnesses); Table 10; Figure 5 (page 4, read as an image).
  2. paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: §1 release and throughput claims; §3.2 unified RLVR; §3.3 MOPD (§3.3.1 algorithm, §3.3.2 teachers, §3.3.3 warmup, §3.3.4 results, §3.3.5 limits), Tables 3-5, Figure 10; §3.5 and Figure 11; §3.6; Appendix A.2 and Figure 17 (page 65, cell values read at 400 dpi).
  3. paperNVIDIA. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2604.12374v1, 14 Apr 2026. https://arxiv.org/abs/2604.12374 (PDF fetched from https://arxiv.org/pdf/2604.12374). Accessed 2026-10-27. Used for: §3.2.1-§3.2.5 RL stages, algorithm, batch figures, harness diversity and asynchronous infrastructure. Not in the source map.
  4. officialarXiv. arXiv API records for 2602.15763, 2606.15007 and 2604.12374. https://export.arxiv.org/api/query?id_list=2602.15763, https://export.arxiv.org/api/query?id_list=2606.15007, https://export.arxiv.org/api/query?id_list=2604.12374. Accessed 2026-10-27. Used for: latest-version check (GLM-5 v2; Ultra v1; Super v1) and dates. Not in the source map.
  5. paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: §2.1 lightning indexer (Eqs. 1-2); §3 mixed RL stage; §3.1 Keep Routing and Keep Sampling Mask.
  6. paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1 (version date not asserted here; the arXiv identifier month and the API's publication date disagree). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: §5.1.2 on-policy distillation (Eq. 29), more than ten teachers and the full-vocabulary choice; §5.2.1 index-score quantization and 99.7% recall.
  7. repozai-org. GLM-5 repository README (GLM-5.3, GLM-5.2, GLM-5.1, GLM-5). https://github.com/zai-org/GLM-5 (raw README: https://raw.githubusercontent.com/zai-org/GLM-5/main/README.md). Accessed 2026-10-27. Used for: the GLM-5.2 IndexShare statement, the slime link, the model list and what is and is not listed as released. Not in the source map.
  8. paperKimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2, 7 Aug 2026 (v1 27 Jul 2026). https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: §4.1.2 expert RL and partial rollouts; §4.1.3 MOPD; §4.1.4 shared quantization scheme.
  9. textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12, 11 Sep 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: §3.2.4 expert counts across MOPD recipes and the grouping of GLM-5 with them.
  10. repoNVIDIA-NeMo. Nemotron repository README (training recipes). https://github.com/NVIDIA-NeMo/Nemotron (raw README: https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/README.md). Accessed 2026-10-27. Used for: the Nemotron 3 Ultra recipe entry (program stages and the stages the recipe covers), the open-data-only note and the Nemotron 3 Super recipe entry. Not in the source map.