Reinforcement Learning for LLM Post-Training · Module 18

Agentic and Multi-Turn RL: Trajectories as Episodes, Credit Assignment, Observation Tokens, and Staleness

An episode is no longer one response: it is several model turns interleaved with tool output, and that changes four things in the objective. This module covers which tokens carry loss, how one trajectory-level reward is split across turns (GiGPO, PARL's critical steps), what partial rollouts do to the policy version behind each token, and why Nemotron 3 Ultra's distillation stage uses single-turn rollouts for most agentic tasks. Afterwards you can specify the loss mask, the advantage and the staleness rule for a multi-turn recipe, and say which reported design fits which task structure.

Course 40 min read 17 sources As of 2026-10-27 Prerequisites: Module 12, Module 14, Module 17
A single-turn episode (one response, one reward) next to a multi-turn episode (model turns a1 to a3 interleaved with environment outputs o1 and o2, one reward), with four numbered places where multi-turn training differs Single-turn: one response, one reward prompt sampled response: every token has a loss r one scalarper sequence Multi-turn: T model turns interleaved with environment output prompt a1 o1 a2 o2 a3 R mask m 010101 T = 3 turns,one scalar R 1 2 3 4 1 Loss mask: only the model's owntokens may carry gradient (Section 1) 2 Credit: one reward must be splitacross T decisions (Sections 2, 3) 3 Tails: duration is turns plus tool waits;stale tokens follow (Section 4) 4 Unit: train on the whole trajectoryor on one turn at a time (Sections 5, 6)
Figure 1. The four places where a multi-turn episode changes the training recipe. Blue segments are tokens the model sampled (a), grey segments are environment output the model only reads (o). GLM-5 states that only model-generated tokens are used for optimization1; Kimi K3 pauses generation once a fraction of trajectories has finished, so tails cross policy versions2; PivotRL and Nemotron 3 Ultra train on single turns instead of whole trajectories3.

1. The trajectory is a token stream

1.1 What the trainer receives, and the mask

A single-turn RLVR sample is a prompt and one sampled response, and the sequence the trainer scores is the sequence the engine sampled. A multi-turn sample is a message list that grows during rollout: the model emits a turn (reasoning plus a tool call), the harness executes the call and appends the result, and the model continues until it answers, hits a turn or token limit, or the environment ends the episode. NeMo Gym describes the environment in these terms: a dataset of tasks, an agent harness that decides how the model interacts with the world, a verifier, and per-task state4 (an unversioned README snapshot, release v0.7.0). Gymnasium's step returns separate terminated (task completed or failed) and truncated (time limit) flags5. For an LLM the difference matters: a truncated trajectory has no outcome yet, and Section 5.3 shows Nemotron masking exactly those.

Write the rollout as tokens \(y_1,\dots,y_L\) and let \(I(y_t)=1\) for tokens the policy sampled and \(0\) for tokens the environment returned. The policy-gradient estimator differentiates the probability of the actions the policy chose (Module 2). Observation tokens are drawn from the environment's dynamics, not from \(\pi_\theta\), so neither \(\nabla\log\pi_\theta\) nor the ratio \(\rho=\pi_\theta/\pi_{old}\) means anything for them. Search-R1 puts this into the GRPO objective, and applies the same mask inside the KL term6 (§3.1, Eq. 3):

$$ \mathcal J(\theta)=\mathbb E\Big[\frac1G\sum_{i=1}^{G}\frac{1}{\sum_t I(y_{i,t})}\sum_{t:\,I(y_{i,t})=1}\min\big(\rho_{i,t}\hat A_{i,t},\ \mathrm{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_{i,t}\big)\Big]-\beta D_{KL}. $$

Two things differ from the single-turn loss of Module 11. The normalizer counts sampled tokens only, so a long tool output does not dilute the gradient of the model's own tokens. And in outcome-reward GRPO the advantage \(\hat A_{i,t}=\hat A_i\) is the same for every unmasked token of trajectory \(i\), however many turns it contains (Section 2 is about that).

The ids in this sequence must be the ids the engine sampled. The gateway that guarantees it (GLM-5's token-in-token-out) is Module 14, Section 2.5, and is not repeated here. Kimi K2.5 states the same design choice: its framework "strictly follows a Token-in-Token-out paradigm" and records the log-probabilities of every engine output for mismatch correction7 (Appendix D). What multi-turn adds is that observation segments travel through the same id stream and need a mask, not a probability.

1.2 What each report says about masking

The source map did not record whether the reports mask environment tokens, so I searched the full text of each. The result is thin: only GLM-5 states the rule for tool output in the passages searched, and Search-R1 is the only paper that measures it.

Table 1. Statements about loss on non-model tokens, by source. "Not stated" means a keyword search of the full report text found no such statement; it does not mean the lab does not mask.
SourceWhat it saysWhere
Search-R1Loss masked on retrieved tokens in PPO, GRPO and the KL term; normalizer counts sampled tokens6§3.1, Table 4
RAGEN (StarPO)Response masking enabled vs disabled on Sokoban with a 0.5B model; enabling it gave higher success (figure only)8Appendix D, Fig. 12
GLM-5"only model-generated tokens are used for optimization, and the environment feedback is ignored in loss computation"1§4.1
Kimi K2.5Frozen subagents' trajectories "excluded from the optimization objective"; the orchestrator's own tool results: not stated7Abstract, §3
Kimi K3A pure-JSON fallback block that occurs only in input tokens has its loss masked; tool results: not stated2Chat-template appendix
Nemotron 3 UltraLoss masked on unfinished trajectories; negative advantage on tokens of malformed reasoning or tool calls; tool results: not stated9§3.3.2 (SWE teacher)
Kimi k1.5Under partial rollouts "certain segments can be excluded from loss computation"10§2.6.2
Kimi K2, DeepSeek-V3.2, DeepSeek-V4Not stated111213-
Grouped bar chart of exact-match scores on seven question-answering sets and their mean, with and without loss masking of retrieved tokens, Qwen2.5-7B-Base trained with PPO 00.250.50 exact match .480.388 .638.567 .457.391 .433.325 .382.321 .196.108 .432.304 NQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogle with loss mask (mean .431) without (mean .343)
Figure 2. Masking retrieved tokens out of the loss, Qwen2.5-7B-Base trained with PPO, exact match on seven QA sets. The mean is 0.431 with the mask and 0.343 without, +0.088 absolute (derived; +26% relative). On Qwen2.5-3B-Base the mean is 0.303 against 0.262, but MuSiQue (0.049 vs 0.053) and Bamboogle (0.088 vs 0.104) go the other way. Data: Search-R1 Tables 4 and 66.

RAGEN's version of the same ablation ran at 0.5B on Sokoban and reports the direction only8. Both are small-model results on short-horizon tasks with PPO, so they show that the mask is not cosmetic, not how large the effect is at frontier scale.

1.3 History is part of the policy

The policy at turn \(t\) conditions on whatever the harness shows it, and the harness can rewrite history. DeepSeek-V3.2 found that R1's rule of discarding reasoning when the next round arrives "forces the model to redundantly re-reason" through the problem at every tool call. Its rule drops reasoning only when a new user message arrives, keeps it while only tool outputs are appended, and always keeps the tool-call history12 (§3.2.1). DeepSeek-V4 keeps the complete reasoning history across all rounds in tool-calling scenarios, including across user messages13. Kimi K3 "supports only preserved thinking": the think channel stays in history even when empty2. Other rules shrink history instead: DeepSeek-V3.2's search agent applies Summary, Discard-75% or Discard-all once usage passes 80% of the context window12 (§4.4), Nemotron 3 Ultra's search teacher is fine-tuned on trajectories with discard-all and summary behavior9, and K3 lists context-management strategies among the composable modules of its training harness2 (§4.2.1).

This decides how a trajectory becomes training samples (derived; none of the reports describes its packing). If every turn appends to the previous context, the trajectory is one sequence: the engine's log-probability for each token was computed given exactly the prefix that the trainer sees, and one forward pass reproduces all conditionals (Figure 3). If a rule deletes or rewrites earlier tokens, the final sequence is not an extension of the context at turn 2, so the trainer must rebuild each segment's own context and one trajectory becomes several sequences that share one reward. verl-agent, the code base behind GiGPO, takes the second route by design: a "step-wise multi-turn" paradigm that avoids concatenating full histories, with a configurable memory module and a history length of 2 for ALFWorld and WebShop14 (Appendix A, E.2).

Token stream of a three-turn tool-using trajectory with segment widths proportional to illustrative token counts, the loss mask, and the context window the policy saw at each turn One trajectory as the trainer sees it task prompt2,000 tok a1600 o1 tool result1,500 a2600 o2 tool result1,500 a3600 R loss mask m 010101 advantage on masked-in tokens A A A same scalar A context the policy conditioned on for a1 for a2: prompt, a1, o1 for a3: prompt, a1, o1, a2, o2 append-only: one sequence rewritten history: one per turn
Figure 3. A three-turn trajectory with illustrative token counts (chosen for drawing, not measured; the same values feed Table 2). Blue segments carry loss (m=1), grey segments are environment output. Every blue token receives the same trajectory-level advantage in outcome-reward GRPO. If the harness appends only, the context for each turn is a prefix of the final sequence; if it drops or summarizes earlier tokens, that prefix property fails (Section 1.3).

The cost of the choice is plain arithmetic. Let \(p\) be the initial prompt, \(g\) the sampled tokens per turn, \(o\) the observation tokens per turn, \(T\) the turns and \(h\) a history window. One concatenated sequence has \(p+Tg+(T-1)o\) tokens. Per-turn samples with full history sum to \(T(p+g)+(g+o)T(T-1)/2\), quadratic in \(T\). A window of \(h\) previous turns costs \(T(p+g)+(g+o)\sum_{t=1}^{T}\min(t-1,h)\). For scale, GiGPO notes that an ALFWorld episode can run to 50 steps and over 20k tokens14.

Table 2. Tokens the trainer processes for one trajectory under three layouts. Parameters are illustrative (\(p=2{,}000\), \(g=600\), \(o=1{,}500\), \(T=30\), \(h=2\)); all columns are derived from the formulas above.
LayoutSamplesTokens (k)vs concatenatedLoss-token share (%)
Concatenated, append-only163.51.0x28.3
Per-turn, full history30991.515.6x1.8
Per-turn, window \(h=2\)30197.73.1x9.1

Trainer tokens for one trajectory under each layout

Concatenated
Per-turn, full history
Per-turn, window h
Full / concatenated
Loss tokens in concatenated

Formulas as above; token counts, not FLOPs. Attention cost inside a sequence is superlinear in its length, so the concatenated layout is cheaper per token than the token count alone suggests only when the sequence still fits the trainer's context and activation budget.

Rollout-side prefill has the same shape. GLM-5 routes every request of one rollout to the same data-parallel rank by consistent hashing so the shared prefix hits the KV cache; with that, "prefill cost remains proportional to incremental tokens rather than total context length"1 (§4.1.2). The infrastructure is System Design Modules 27 and 37; the algorithmic point is only that the trainer pays the quadratic bill unless history is append-only or windowed.

2. Splitting credit across turns

2.1 What a trajectory-level baseline does

Group-relative advantage compares trajectory \(i\)'s reward with its group and hands the result to every sampled token (Modules 9 and 11). Every agentic recipe in this module's source set does exactly that. GLM-5's objective is \(r(x,y_i)-\bar r(x)\) over agent traces1 (§4.1). Kimi K2.5's loss uses the same mean-reward baseline inside a token-level objective7 (Eq. 1). DeepSeek-V3.2 uses \(A=R-\operatorname{mean}(R)\) in GRPO12 (§3.1), and Nemotron 3 Ultra's end-to-end SWE stage feeds a binary hidden-test reward to GRPO9. A full-text search of K2, K2.5, K3, V3.2, V4, GLM-5, Nemotron 3 Ultra, MiMo-V2-Flash and Olmo 3 finds no mention of GiGPO, anchor states or step-level advantages.

The consequence is that a wrong turn inside a successful trajectory is reinforced, and a correct turn inside a failed one is punished. On average the signal points the right way, because good turns co-occur with success more often, but the estimator's variance grows with the number of turns that share one scalar (Module 2). What the frontier reports show in response is more samples per task and difficulty filtering of tasks (DeepSeek-V3.2 keeps tasks with non-zero pass@100, Section 6.2), not finer credit.

2.2 GiGPO: step-level groups from repeated states

GiGPO keeps the critic-free group structure and adds a second grouping. Sample \(N\) trajectories of one task from identical initial conditions. The episode-level advantage is GRPO's, with \(F_{norm}\) either the standard deviation or 1 (a rescaled leave-one-out estimator). Collect the set of distinct states \(\tilde s\) that occur anywhere in the group, including repeat visits inside one trajectory, and for each build a step group of the actions taken there and the discounted return that followed14 (§4.1-4.3, Eqs. 3-8):

$$ A^E(\tau_i)=\frac{R(\tau_i)-\operatorname{mean}_j R(\tau_j)}{F_{norm}},\quad A^S(a^{(i)}_t)=\frac{R^{(i)}_t-\operatorname{mean}_{G^S(\tilde s)}R}{F_{norm}},\quad R^{(i)}_t=\sum_{k=t}^{T}\gamma^{k-t}r^{(i)}_k,\quad A=A^E+\omega A^S. $$

The clipped objective with a KL term is then applied per step with \(A\) (Eq. 9). Grouping is a hash-map lookup on the state, so it needs no extra rollouts and no value model; the paper uses \(\omega=1\) without tuning, \(\gamma=0.95\) and \(N=8\) on ALFWorld and WebShop, with a reward of 10 for success, 0 for failure and -0.1 for an invalid action (Appendix E.1). The comparison group for an action is "what else happened from this same state", which is the closest a critic-free method gets to a baseline \(V(s)\).

Six trajectories visiting the same anchor state, the step group built from them with discounted returns and step-level advantages, and the resulting credit for the two clicks in trajectory 3 under GRPO and under GiGPO Trajectories reaching the anchor state steps t = 1, 2, ... left to right τ1item1R=1 τ2item1R=1 τ3item2item1R=1 τ4nextR=0 τ5nextR=0 τ6item2R=0 τ7, τ8 fail without reaching this state: 3 of 8 succeed anchor state: search-results page Step group at the anchor (7 entries) actionR_tA^SA^EA item1 (τ1, τ2, τ3@4)0.810+0.369+0.625+0.994 item2 (τ3@2)0.656+0.215+0.625+0.840 item2 (τ6)0.000-0.441-0.375-0.816 next (τ4, τ5)0.000-0.441-0.375-0.816 γ = 0.9, F_norm = 1, ω = 1; group mean of R_t = 0.441 Credit given to the two clicks inside τ3 GRPO: item2 click (wrong)+0.625 GRPO: item1 click (right)+0.625 GiGPO: item2 click (wrong)+0.840 GiGPO: item1 click (right)+0.994 bar length is proportional to advantage; the right click now outranks the wrong one inside the same trajectory
Figure 4. A constructed example of GiGPO's two-level advantage; the mechanism is from the paper's Eqs. 3-8, the numbers are mine. The toy uses \(\gamma=0.9\) and reward 1 at the end of a successful trajectory (the paper's experiments use \(\gamma=0.95\) and reward 10), so a click at step \(t\) in a trajectory that succeeds at step \(T\) has return \(0.9^{T-t}\): 0.81 for the first click in \(\tau_1\) (\(T=4\), \(t=2\)) and 0.656 for \(\tau_3\)'s wrong click (\(T=6\)). The paper's own Figure 3 gives only the ordering \(A^S(\text{1st item})>A^S(\text{2nd item})>A^S(\text{next page})\)14.

2.3 Evidence and limits

The evidence is two text-environment benchmarks at 1.5B and 7B, three seeds, and a search-QA set. With \(F_{norm}=1\) GiGPO beats GRPO by 13.3 and 10.6 points (ALFWorld success, WebShop success) at 1.5B and by 12.6 and 9.1 points at 7B, which are the abstract's "> 12%" and "> 9%" in absolute points14 (§5.2, Table 1).

Table 3. GiGPO against its baselines (Qwen2.5-Instruct, mean of 3 seeds, % success unless marked). "w/ std" uses \(F_{norm}=\) std, "w/o std" uses 1. Source: Table 1 of the GiGPO paper14.
ModelMethodALFWorld successWebShop scoreWebShop success
1.5BPPO (with critic)54.473.851.5
1.5BRLOO69.773.952.1
1.5BGRPO72.875.856.8
1.5BGiGPO w/ std86.783.165.0
1.5BGiGPO w/o std86.183.567.4
7BPPO (with critic)80.481.468.7
7BRLOO75.580.365.7
7BGRPO77.679.366.1
7BGiGPO w/ std90.884.472.8
7BGiGPO w/o std90.286.275.2

Three details bound the claim. First, the overhead: anchor grouping takes 0.01 s and the step-advantage arithmetic 0.53 s per iteration, against 362.83 s for the whole iteration (1.5B, ALFWorld). That is \(0.54/362.83=0.15\%\); the paper prints "< 0.002%", which matches a fraction (0.0015) and not a percentage, so I read the unit as a slip. Either way grouping is free next to rollout. For scale, the 1.5B runs used 2 H100 GPUs for 150 iterations (Appendix E.1), which at 362.83 s per iteration is about 15 hours, or 30 GPU-hours (derived). Second, recurrence is empirical: singleton step groups stay under 35% of groups through training, so over 65% of states recur in ALFWorld, and the authors read the early large groups (10 or more members, over 20% of groups at iteration 10) as looping by an immature policy14 (§5.5). Third, when no state repeats \(A^S=0\) and the method reduces to GRPO. The search-QA runs used similarity-based grouping instead: two states match when their longest-matching-subsequence similarity exceeds 0.9. For coding and terminal agents, where each command changes file and process state, exact repeats are plausibly rare after the first few steps, so the approach may degrade towards GRPO there.

2.4 Critics, process rewards and shaping

DeepSeek-R1's supplement gives three reasons it dropped a process reward model: a fine-grained step is hard to define, whether a step is correct is hard to determine, and a model-based PRM "inevitably leads to reward hacking"15 (§G.2). In an agent loop the first reason disappears, because the harness delimits turns, but the second remains. GiGPO sidesteps both by using the environment's own returns, grouped by state (my reading).

The critic is not dead at small scale. At 7B, PPO with a critic beats GRPO and RLOO on ALFWorld (80.4 against 77.6 and 75.5, Table 3). Search-R1 finds GRPO converges faster but "may exhibit instability after trained for a number of steps", PPO is more stable but slower6 (Fig. 2a). RAGEN sees "reward variability cliffs and gradient spikes" in multi-turn training, calls the collapse mode the Echo Trap, notes that declining reward standard deviation often precedes it, and stabilizes by training only on the top \(p\%\) of prompts by reward standard deviation (default 25%): under PPO, keeping 75% extends FrozenLake stability from 100 to 140 steps and keeping 50% avoids collapse8 (§4.2). This is Module 11's dynamic sampling, applied because a multi-turn group has more ways to go uniform.

Shaping lives outside the advantage. Nemotron's SWE teacher says the final reward "does not always help with trajectory-level behaviors", so it masks unfinished trajectories and gives malformed reasoning or tool calls a negative advantage on the offending tokens9 (§3.3.2). Kimi K3 overrides the reward with \(-1\) when a trajectory's budgeted tokens, counting reasoning and tool-call arguments for agentic tasks, exceed \(\tau\) times a per-problem budget2 (§4.1.2). Its personal-assistant environments attach an evaluation criterion to each event, which makes the environment's signal denser than a final pass/fail; the report does not say how event scores enter the advantage (§4.2.5).

3. Orchestrators and frozen subagents: PARL

3.1 What is trained, and the reward

Kimi K2.5's Agent Swarm splits one task across subagents that an orchestrator creates and schedules. Parallel-Agent RL (PARL) trains the orchestrator only. The subagents are "instantiated from fixed intermediate policy checkpoints", their execution trajectories are "excluded from the optimization objective", and their outputs are treated "as environmental observations rather than differentiable decision points"7 (§3). In token-stream terms from Section 1, a subagent's return is an observation segment with mask 0. The stated reason is credit ambiguity: "a correct final answer does not guarantee flawless subagent execution, just as a failure does not imply universal subagent error".

The reward adds two auxiliary terms to the task outcome7 (§3):

$$ r_{\text{PARL}}(x,y)=\lambda_1\,r_{\text{parallel}}+\lambda_2\,r_{\text{finish}}+r_{\text{perf}}(x,y). $$

\(r_{\text{perf}}\) is the task-level outcome. \(r_{\text{parallel}}\), an instantiation reward, fights "serial collapse", the local optimum in which the orchestrator never spawns anything. \(r_{\text{finish}}\), the sub-agent finish rate, fights "spurious parallelism", where the orchestrator inflates a parallelism metric by spawning many subagents without a real decomposition. Both \(\lambda\) are annealed to zero during training, so the final policy is optimized for \(r_{\text{perf}}\) alone. The passages I read give no values for \(\lambda_1,\lambda_2\) or the annealing schedule. The pair is a small instance of the pattern Module 19 treats: an auxiliary reward to get past a bad optimum, removed before it becomes the thing the policy exploits.

3.2 Critical steps: a resource constraint that rewards parallelism

Counting total steps would charge the orchestrator for every parallel step. K2.5 instead measures cost by analogy to a critical path. With stages \(t=1,\dots,T\), where \(S^{(t)}_{main}\) is the main agent's steps in stage \(t\) (typically 1) and \(S^{(t)}_{sub,i}\) the steps of the \(i\)-th subagent in a parallel group7 (§3):

$$ \text{CriticalSteps}=\sum_{t=1}^{T}\Big(S^{(t)}_{main}+\max_i S^{(t)}_{sub,i}\Big). $$

Training and evaluation are constrained by this quantity, so creating subtasks that do not shorten the longest parallel branch gains nothing under it. Figure 5 counts a toy episode (numbers mine). Stage 1 spawns four subagents that take 5, 3, 8 and 2 steps; stage 2 spawns six that take 4, 4, 9, 2, 3 and 5; stage 3 is the orchestrator's final answer. Critical steps are \((1+8)+(1+9)+1=20\), total steps are \(3+18+27=48\), so the swarm is 2.4 times shorter than one agent doing the same 48 steps in sequence. A perfectly balanced split of the same subagent work (\(\lceil 18/4\rceil=5\), \(\lceil 27/6\rceil=5\)) would give \(6+6+1=13\) critical steps, 3.7 times shorter. The gap between 20 and 13 is the value of decomposing evenly, which is what the metric pays for.

Critical-steps count for a toy parallel-agent episode: three stages, with the longest subagent in each stage setting the stage length, next to a diagram of which tokens the orchestrator is trained on Toy episode, 1 step = 22 px stage 1 5 3 8 2 1 + 8 = 9 stage 2 4 4 9 2 3 5 1 + 9 = 10 stage 3 1 1 + 0 = 1 critical steps = 9 + 10 + 1 = 20 total steps = 3 + 18 + 27 = 48 (2.4x longer) balanced split of the same work: 6 + 6 + 1 = 13 (3.7x) orchestrator step frozen subagent longest in stage Orchestrator (trainable) own tokens: m = 1 returns: m = 0 subagent outputs are observations gradient flows through m = 1 tokens only Subagents (frozen checkpoints) sub 1 sub 2 sub 3 trajectories excluded from the loss task text result
Figure 5. Left: the critical-steps count for a toy episode (the numbers are constructed to exercise the formula; each band is one stage, bars within a band start together, stages run in sequence). The longest subagent sets each stage's length. Right: which tokens the orchestrator is optimized on. Formula and the freezing/exclusion rule: Kimi K2.5 §37.

3.3 Why freeze, what it costs, and what is reported

Freezing makes the orchestrator's problem an ordinary single-policy RL problem with a stochastic environment (the subagents), which is what Sections 1 and 2 already handle. The costs are visible in the recipe. The orchestrator is first trained against small subagents and then larger ones, the cluster's inference-instance ratio between subagents and orchestrator is adjusted dynamically, and subagents never improve from the swarm's reward during that stage7 (§3). The report contains no co-training baseline in the passages I read, so "frozen is more stable" is the authors' stated reason and not a measured comparison.

Reported outcomes are for inference, on wide-search tasks: latency "up to 4.5x" lower with item-level F1 up from 72.8% to 79.0% against single-agent baselines (the introduction prints 72.8%, the evaluation text 72.7%), and a swarm that stays at 0.6-1.6 times a baseline latency as the target F1 rises from 30% to 70%, where the single agent grows from about 1.8 to over 7.0 times it (evaluation section, Fig. 8). The report also argues the swarm is "proactive context management": subagents keep bounded local contexts and "only task-relevant outputs" return, which shards context instead of truncating it and, per the authors, beat Discard-all on BrowseComp with fewer critical steps7 (Fig. 7). For training, that has a plain consequence: the orchestrator's trained sequence stays short because subagent traces never enter it. K2.5's rollout manager runs every task as an asynchronous coroutine, lets a task recursively trigger sub-task rollouts, and handles up to 100,000 concurrent agent tasks (Appendix D). Module 28 reads the K1.5-to-K3 line stage by stage.

4. Partial rollouts and staleness

4.1 Three ways to stop waiting for the slowest trajectory

Agentic training lengthens the tail as it succeeds: Kimi K3 reports that average tool-call steps rise consistently as RL FLOPs scale2 (Fig. 8). Module 1 (Section 6) has the long-tail arithmetic and System Design Module 37 the infrastructure taxonomy; this section covers what the algorithm must absorb.

Kimi k1.5 gives each iteration a fixed output-token budget. A trajectory that exceeds it has its unfinished portion saved to a replay buffer and continued in the next iteration. Only the current iteration (iter n) "requires on-policy computation", while segments from iterations \(n-m\) to \(n-1\) "can be efficiently reused from the buffer", and "certain segments can be excluded from loss computation"10 (§2.6.2). Kimi K2 applies the same technique to multi-turn agentic rollouts: long-tail unfinished tasks "paused, and resumed in the next RL iteration"11 (§3.3.4, Agentic Rollout).

Kimi K3 replaces the token budget with a completion fraction. Each iteration keeps \(N\times K\) trajectories in flight, and "the generation phase pauses as soon as a fraction \(\lambda\in(0,1)\) of trajectories completes (i.e., \(\lambda NK\))". Paused rollouts are enqueued and resumed first in the next iteration; once all \(K\) responses of a prompt are done they go straight to policy optimization2 (§4.1.2). A trajectory "naturally spans multiple iterations", and the report's answer to the resulting staleness is that the objective "inherently tolerates such an extreme off-policy regime through a per-token regularization"2. Pausing an agent mid-episode needs the environment to survive the pause, which is why K3 pairs it with a microVM sandbox whose incremental checkpoint and resume take as little as 133 ms and 49 ms (§5.3.2; sandbox design is System Design Module 27). MiMo-V2-Flash also partitions overlong trajectories across steps, "while limiting staleness and the proportion of partial samples in each batch", with "staleness-aware truncated importance sampling"16 (§4.6.2).

Timeline of four iterations with five trajectories; trajectories C, D and E cross policy-version boundaries, D spans three versions, and its older tokens are enclosed in a dashed region where the importance ratio departs from one iteration niteration n+1iteration n+2 engine at v_nengine at v_n+1engine at v_n+2 pause at λNK done; update Adone Bdone Cdone Ddone Edone stale tokens of D (dashed):ratio departs from 1 K=all done: to the optimizer sampled by v_n sampled by v_n+1 sampled by v_n+2 If D is consumed by the update after iteration n+2, token lag is 2, 1 and 0 for the three segments. Schematic of the scheme described in Kimi K3 §4.1.2; lengths and counts are drawn, not measured.
Figure 6. Partial rollouts with a completion fraction: generation pauses when \(\lambda NK\) trajectories have finished, unfinished ones resume next iteration under newer weights, and a trajectory can end up with tokens from several policy versions. Schematic of the scheme in Kimi K32; the dashed region is where K3 relies on its per-token regularization.

4.2 What pausing at a fraction does

Model trajectory durations as i.i.d. exponential with mean \(\mu\), an assumption chosen for a closed form and not taken from any report. Waiting for all \(n=NK\) costs the expected maximum \(\mu H_n\approx\mu(\ln n+0.577)\). Pausing at fraction \(\lambda\) costs the \(\lambda\)-quantile \(t_\lambda=-\mu\ln(1-\lambda)\). For \(n=8{,}192\) and \(\lambda=0.9\): \(H_n=9.59\), \(t_\lambda=2.30\,\mu\), a 4.2-fold shorter iteration; to first order the longest trajectory then spans about \(9.59/2.30\approx4.2\) iterations, so it touches five policy versions. The same model prices the selection effect: the first \(\lambda\) fraction to finish has mean length \(\mu\,[1-(1+q)(1-\lambda)]/\lambda\) with \(q=-\ln(1-\lambda)\), which is \(0.744\,\mu\) at \(\lambda=0.9\), 26% shorter than the population.

Pause-at-fraction under exponential durations (illustrative model)

Wait-for-all iteration time (units of mean duration)
Pause-at-λ iteration time (same units)
Iteration speed-up
Iterations spanned by the longest trajectory (approx.)
Mean length of first-λ finishers vs population

Formulas: \(E[\max]\approx\ln n+0.577\), \(t_\lambda=-\ln(1-\lambda)\), kept-set mean \(=[1-(1+q)(1-\lambda)]/\lambda\). Agent trajectories with variable turn counts and tool latency are plausibly heavier-tailed than the exponential; if so the real speed-up and span are larger. Use the shape, not the values.

The selection effect is why resuming matters. DeepSeek-V4 states the failure for preempted generation: "it is mathematically incorrect to regenerate unfinished requests from scratch, as this introduces length bias", because "shorter responses are more likely to survive interruption"; its fix is a token-granular write-ahead log plus saved KV cache, so decoding continues instead of restarting13 (§5.2.3). The same logic covers pausing: if the unfinished 10% were dropped, the optimizer would see a batch biased to short trajectories. K3's scheme avoids that by resuming, and pays in staleness. The staleness is also not random. It concentrates on the longest trajectories, and GLM-5 notes that "overly long trajectories can become highly off-policy, which may destabilize training"1 (§4.1.2).

4.3 What the loss sees across versions

Give each token a behavior version \(b_t\), the weights that sampled it. The ratio the learner needs is \(\pi_\theta(y_t\mid s_t)/\pi_{b_t}(y_t\mid s_t)\). If the engine returns \(\log\pi_{b_t}\) with every token, as K2.5's framework does7 (Appendix D), that is one stored number per token however many versions the trajectory crossed. This is why GLM-5 can drop \(\pi_{old}\) and use the rollout log-probabilities as "a direct behavior proxy" when "rollout engines may undergo multiple updates during a single trajectory generation"1 (§4.1.2), and why token-level objectives carry over unchanged to partial rollouts. GLM-5 names the result Direct Double-sided Importance Sampling: the ratio is \(\pi_\theta/\pi_{rollout}\) and tokens outside \([1-\epsilon_\ell,1+\epsilon_h]\) are masked from the gradient, controlling off-policy bias "without tracking historical policy checkpoints". The objectives themselves (K2.5's log-ratio band, K3's per-token regularization, GLM-5's double-sided mask) are Module 12, Sections 5.1 and 5.2; the mismatch between engines that produces a second, version-independent ratio is Module 14.

Two derived cautions. A sequence-level ratio, as in GSPO, can still be assembled from per-token recorded log-probabilities, but its clip range (3e-4 and 4e-4 in GSPO's experiments17, §4) was set for one-version sequences, and none of the reports in this set tests it on multi-version ones. And importance weights correct the probability of each action given its context, not the distribution of contexts: a stale action's context holds observations produced by older actions the current policy may not take. PPO-style methods ignore that term, and so does every report here.

4.4 Bounding the lag

GLM-5 treats staleness as something to cap. It logs for each response the sequence of weight versions \(w_0<\dots<w_k\) that generated it and discards a sample when \(w'-w_0>\tau\), with \(w'\) the current version; the report does not state \(\tau\). The training engine pushes weights to the rollout engine every \(K\) gradient updates, and GLM-5 also resets the optimizer after each such push1 (§4.1.1-4.1.2). Nemotron 3 Ultra runs "a one-step off-policy asynchronous RL setup" in which rollout generation overlaps the previous update, so lag is at most one step by construction; for MOPD it separates the behavior policy from a proximal policy used as the trust-region center9 (§3.3.1, §3.6.1). K3 states a tolerance and no bound, and publishes no staleness histogram or comparison with synchronous rollouts in the passages I read. Section 7 lines the designs up.

5. Turn-local RL: PivotRL and Nemotron 3 Ultra

5.1 Train on the turns that carry signal

The cheapest way to avoid long, stale, uneven trajectories is not to roll them out. PivotRL, from NVIDIA, starts from existing SFT trajectories and decomposes each at assistant boundaries into state-action pairs \((s_t,a^*_t)\), where \(s_t\) is the full history before the \(t\)-th assistant turn. Offline, it samples \(K\) local rollouts from the reference policy at each candidate state, scores them with a verifier, and keeps only turns with nonzero reward variance and a low reward mean (\(\hat\sigma^2(s)>0\), \(\hat\mu(s)<\lambda_{\text{diff}}\)); these are the pivots. Online, it samples \(G\) actions at a pivot, runs only the short rollout needed to score each, and applies a GRPO-style update with a verifier reward \(r_{\text{func}}(s,a)=\mathbb{1}[a\in\mathcal M(s)]\), where \(\mathcal M(s)\) is the set of locally acceptable actions3 (§3.1, Eqs. 4-8). The verifier may be a normalized string or schema check, a task-specific equivalence rule, or a lightweight LLM judge.

Both design choices answer a measured problem. First, a group whose rewards are all equal has zero advantage (Module 11, Section 4); in turn space the paper measures that 71% of randomly sampled turns on \(\tau^2\)-Bench and SWE-Bench give no learning signal. Second, rewarding only an exact match with the demonstrated action is too strict in a generative action space, where many tool calls are valid.

5.2 What the numbers show, and what they do not

Left: tau-squared-Bench accuracy for the base model, same-data SFT, and three PivotRL variants. Right: cumulative rollout turns each method needed to reach 32.67 percent on SWE-Bench Verified, PivotRL about 133 thousand against about 542 thousand for end-to-end RL τ²-Bench accuracy (%) 02550 44.35 58.44 57.34 59.68 63.81 Base SFTsame data strictreward no pivotfilter fullPivotRL Turns to reach 32.67% on SWE-Bench PivotRL ~133K (130 x 1,024) End-to-end RL ~542K 72 steps x 512 trajectories x ~14.7 turns (derived) 4.1x in turns; ~5.5x in wall-clock (reported) Qwen3-30B-A3B-Thinking, OpenHands, same nodes
Figure 7. Left: ablation on τ²-Bench (best checkpoint over training): both removing the pivot filter and replacing the functional reward with exact match lose accuracy, and strict-reward local RL ends below same-data SFT. Right: rollout turns to a matched 32.67% SWE-Bench accuracy. Data: PivotRL Tables 4 and 6, Section 4.2 and Appendix A.23; the 14.7 turns per trajectory is derived from the reported 542K turns.

The turn count in Figure 7 checks out. PivotRL trains with a batch of 1,024 (64 prompts times 16 generations) at one turn per sample and reaches 32.67% at step 130, so \(130\times1{,}024=133{,}120\) rollout turns. End-to-end RL uses 512 trajectories per step (16 prompts times 32 generations), reaches the same accuracy at about step 72 and about 542K cumulative turns, which implies \(542{,}000/(72\times512)\approx14.7\) turns per trajectory, inside the reported 12-25. The ratio is 4.1 in turns and the report gives about 5.5 in wall-clock, so an end-to-end turn costs roughly 1.35 times a single-turn one in time (derived; long contexts and tool latency are the likely cause, which the report does not break out)3 (§4.2, App. A.2).

Four limits sit around the headline. The count is training samples; the \(K\) offline profiling rollouts per candidate turn are not part of that total in the text I read. The 32.67% is below same-data SFT on the same benchmark (37.40%, Table 1), so on SWE-Bench PivotRL's case is cost against end-to-end RL and retention outside the domain, not in-domain accuracy. The SWE-Bench local verifier "matches tool-call names only", so what is learned turn by turn there is which kind of operation to run next, with final success judged only by the full harness. And the out-of-domain advantage (+0.21 against SFT's -9.83 average change; the introduction prints -9.48) comes from one base model, Qwen3-30B-A3B-Thinking-25073 (Tables 1-2, App. A.2).

5.3 How Nemotron 3 Ultra mixes both

Nemotron 3 Ultra does not choose. Its unified RLVR stage spans terminal, office, SWE, search, tool-calling and non-agentic environments, with a deliberately diverse set of harness implementations "reducing overfitting to any particular harness design", asynchronous GRPO, "a global batch size of 8192, with each sample generating 16 rollouts", and a 48K maximum generation length later raised to 64K9 (§3.2). The report does not say whether 8,192 counts prompts (then 131,072 rollouts per step) or rollouts (then 512 prompts), so both readings are shown rather than resolved. Its SWE teacher runs three stages: SFT on agentic data, PivotRL on single-step agentic environments, then an end-to-end SWE-RL stage where the model works a repository over multiple turns and a binary hidden-test reward feeds GRPO, with 192K generation length and at most 200 agent turns. That stage masks unfinished trajectories (maximum turns or timeouts), penalizes malformed reasoning and tool calls, and closes two channels by which the agent could read the gold patch: future commits are physically deleted from the in-container repository, and a runtime filter blocks remote git operations and downloads from GitHub domains (§3.3.2; Module 19 covers the exploit class).

In the MOPD stage (Module 22), the student's rollouts get dense per-token rewards from a domain teacher, with 1,024 prompts per batch and one rollout per prompt, because "using multiple rollouts did not yield additional benefits" in their ablations. The report is candid about agentic tasks there: end-to-end agentic rollouts mixed with reasoning rollouts caused "substantial training inefficiency because rollout times can differ dramatically", so "we use single-turn rollouts, similar to PivotRL (Yi et al., 2026), for most of the agentic tasks". Whether end-to-end rollouts would gain more, and how to make them robust to distribution mismatch, is listed as open9 (§3.3.1, §3.3.5).

6. GLM-5 and DeepSeek-V3.2: two end-to-end designs

6.1 GLM-5: asynchronous traces, sample hygiene

GLM-5's agentic stage is a plain group-wise objective over agent traces (advantage \(r-\bar r\), model-generated tokens only), wrapped in infrastructure that keeps the learner fed. The training and inference engines sit on different GPUs; the inference engine generates continuously, a batch goes to the learner once a trajectory-count threshold is reached, and updated weights return every \(K\) gradient updates1 (§4.1.1). Because tasks differ in tools and rollout logic, each task runs as an independent microservice registered with a central Multi-Task Rollout Orchestrator that sets the per-task rollout ratio and generation speed. All tasks' trajectories are standardized into one message-list format, and the orchestrator supports "over 1k concurrent rollouts" and dynamic changes to task sampling ratios. The scheduling is infrastructure (System Design Modules 37 and 38); the algorithm-side content is what happens to the samples.

GLM-5 excludes samples whose failure came from the sandbox ("environment crashes"), on the grounds that they "reflect environment instability rather than the model's capability". Exclusion breaks GRPO's fixed group size, so the rule is: if the valid samples number more than half the group, pad by repeating valid ones; otherwise drop the group1 (§4.1.2). With a group of 16, that pads groups with 9 to 15 valid samples and drops groups with 8 or fewer. The report does not say how the repeats are chosen. Two consequences follow (derived). Padding by repetition gives some samples double weight in the group mean. And if sandboxes fail more often on long or hard tasks, the survivors skew easy, the same survival effect as the length bias of Section 4.2. Neither is measured in the report.

6.2 DeepSeek-V3.2: synthesized tasks inside one mixed stage

DeepSeek-V3.2's agentic RL is not a separate algorithm. It is GRPO in the same single stage that merges reasoning, agent and alignment training, with rule-based outcome reward, a length penalty and a language-consistency reward for reasoning and agent tasks12 (§3). What is agent-specific is the task supply and the context rule of Section 1.3. Table 4 counts the task pool; the four rows sum to 85,267, the "85,000 complex prompts" of the introduction. Only the general-agent row (4,417 tasks, 1,827 environments) is synthesized end to end; the other three populations run in real environments.

Table 4. DeepSeek-V3.2 agent RL task pool (Table 1 of the report)12. The total is derived.
Agent typeTasksEnvironmentPrompts
Code agent24,667realextracted
Search agent50,275realsynthesized
General agent4,417synthesizedsynthesized
Code interpreter5,908realextracted
Total85,267

The general-agent pool is built by an agent: for a task category it generates a sandbox database, task-specific tools, a task with a Python solution and a Python verifier that must agree, and then raises the difficulty step by step. After an RL pass with DeepSeek-V3.2 the pipeline keeps only instances with non-zero pass@100, which leaves 1,827 environments and 4,417 tasks (the table's general-agent row). On a 50-instance sample the difficulty is real: pass@1 is 12% for DeepSeek-V3.2-Exp against 34% (Sonnet-4.5), 51% (Gemini-3.0 Pro) and 62% (GPT-5-Thinking)12 (§3.2.3, §4.3, Table 5). The ablation that carries the argument is qualitative: RL on synthetic general-agent data alone, from the SFT checkpoint in non-thinking mode, improves Tau2Bench, MCP-Mark and MCP-Universe, while RL on code and search environments alone does not (Figure 5, no numbers in the text). Environment design is Module 17's subject; the point for this module is that V3.2 improves tool-use generalization by changing the data, and the multi-turn mechanics are the single-turn recipe unchanged.

7. The designs side by side

Table 5. Rollout unit, credit, staleness handling and treatment of unfinished or failed samples, as stated in each report. "Not stated" means not found in the passages read. Cross-references are to the loss and ratio details in Module 12.
ReportRollout unitAdvantage and creditStalenessUnfinished or failed
Kimi k1.510One long response, split across iterations by a token budgetMean-reward baselineEarlier segments reused from a replay buffer; current segment on-policySegments may be excluded from loss; repeats stopped and penalized
Kimi K2.57Trajectory; swarm trains the orchestrator only\(r-\bar r\) token-level; PARL auxiliary rewards annealed to 0Gradient mask outside a log-ratio bandSubagent trajectories excluded
Kimi K32Trajectory to 1M-token context; pause at \(\lambda NK\)K2.5 algorithm; -1 override above \(\tau b_0\) tokensPer-token regularization; no bound statedResumed next iteration in a checkpointed sandbox
GLM-51Message-list trace, 1k+ concurrent\(r-\bar r\); model tokens onlyRollout log-probs as behavior proxy; two-sided mask; drop if \(w'-w_0>\tau\)Sandbox crashes excluded; group padded or dropped
Nemotron 3 Ultra9Single-turn for most MOPD agentic tasks; end-to-end SWE to 200 turnsBinary reward in GRPO; dense teacher reward in MOPD; negative advantage on malformed tokensOne-step off-policy; behavior and proximal policies in MOPDLoss masked on unfinished trajectories
DeepSeek-V3.212Trajectory; reasoning kept across tool results\(A=R-\operatorname{mean}(R)\); outcome, length and language rewardsSequence masking and routing replay (Module 12); nothing agent-specificTasks filtered to non-zero pass@100; unfinished: not stated
MiMo-V2-Flash16Trajectory; overlong ones partitioned across stepsNot statedStaleness-aware truncated IS; caps on staleness and on partial-sample shareNot stated

Three patterns hold across the rows. Advantage is trajectory-level everywhere; the only step-level credit in this set is GiGPO's, outside the frontier reports. Staleness has four answers: tolerate (K3), cap (GLM-5, MiMo), avoid by construction (Nemotron's one-step lag), or never create it (V3.2 has no partial rollouts in the passages read). Unfinished trajectories are resumed (K3), masked (Nemotron) or dropped (GLM-5, under its staleness rule). Masking and dropping remove trajectories by length or timeout, which reintroduces the survivor bias of Section 4.2 (derived); only resumption keeps the long tail in the batch, at the price of staleness.

Where experts disagree

Three questions are open for multi-turn RL: what unit receives credit, whether to train on whole trajectories or on single turns, and how much policy staleness to accept. The reports answer them differently and almost never ablate one answer against another, so the positions below are designs, not measured rivals.

Trajectory-level outcome reward is enough

GLM-5, Kimi K2.5 and K3, DeepSeek-V3.2, Nemotron 3 Ultra's end-to-end SWE stage17129; RAGEN's StarPO is trajectory-level too8

Scale and filtering average out turn-level noise. A step-level signal needs a state definition or a learned model, and the simple objective already runs at 1,000 to 100,000 concurrent rollouts.

  • No extra machinery; the Module 12 and 14 fixes apply unchanged
  • Works with any harness, including ones that rewrite history
  • No frontier report ablates it against step-level credit
  • RAGEN reports Echo-Trap collapse at small scale; whether frontier runs hit it is not reported

Add step-level credit from the environment, with no learned PRM

GiGPO (NTU, NeurIPS 2025)14; R1's supplement gives the case against learned PRMs15

Compare actions taken from the same state across trajectories and add that to the episode advantage. It is critic-free and nearly free to compute.

  • +12.6 to +13.3 points ALFWorld success over GRPO at 1.5B and 7B (3 seeds)
  • Falls back to GRPO when no state repeats
  • Needs recurring states; evidence stops at 7B on ALFWorld, WebShop and search QA
  • None of the nine frontier-class reports searched mentions it

Stop rolling out whole trajectories; train on pivots

PivotRL (NVIDIA, UC Berkeley)3; Nemotron 3 Ultra's MOPD uses single-turn rollouts for most agentic tasks9

Most turns carry no gradient under group normalization, and local rollouts remove tails and staleness entirely.

  • About 4.1 times fewer rollout turns and 5.5 times less wall-clock to a matched SWE-Bench accuracy
  • Keeps out-of-domain skills (+0.21 against SFT's -9.83)
  • 32.67% is below same-data SFT (37.40%) on SWE-Bench Verified, and the SWE verifier matches tool names only
  • Nemotron lists end-to-end rollouts as an open question

Keep end-to-end rollouts and manage staleness: tolerate or cap

Tolerate: Kimi K32. Cap: GLM-5 and MiMo-V2-Flash116. Resume instead of restarting: DeepSeek-V413

Tails are the cost of long-horizon training, so keep them and decide where the off-policy error goes: into the objective (K3's per-token regularization) or into a rule that drops samples older than \(\tau\) versions (GLM-5).

  • Trains on the behavior that only long episodes show
  • Resuming avoids the length bias of restarting
  • No staleness histogram or synchronous baseline is published for K3
  • \(\tau\) and the clip values are not stated in the passages read
Assessment

Trajectory-level outcome reward is the default for good reason: every large run in this set uses it and none, in the passages read, reports a failure attributed to coarse credit. That is absence of evidence, not a result. GiGPO is the strongest case where states recur and horizons are tens of steps; for coding and terminal agents I would not expect exact state matches to fire, so the experiment that would settle it is GiGPO with similarity-based state keys against GRPO on a SWE-style task at 30B or more, with the group-size histogram reported. Turn-local RL is a cost and forgetting tool, not an accuracy tool: use it to bootstrap agentic skill from SFT data and where rollout cost dominates, and keep an end-to-end stage for recovery and long-horizon behavior, which is what Nemotron's SWE teacher does. On staleness, cap by default, because a cap is cheap to implement and a failure is visible in the data; K3's tolerance claim would be credible with a lag sweep, which no report has. Three of the four positions are compatible, which suggests the disagreement is about emphasis and budget more than correctness.

What is changing

Context length is the driver. Kimi K3 trains agents at 1M-token context with co-located partial rollouts, an external KV pool and resumable sandboxes2, which turns the quadratic cost of Section 1.3 and the staleness of Section 4 into first-order design constraints. Training-time harness diversity (K3's composable modules, Nemotron's mixed harnesses) pushes history rules into the training distribution, so Section 1.3's segment bookkeeping will matter more. PivotRL's adoption in Nemotron 3 Super and Ultra3 suggests turn-local RL becomes a standard first stage. Open items: step-level credit at frontier scale, co-training orchestrator and subagents, and how multi-teacher distillation (Module 22) handles long-horizon rollouts. The reports used here are dated 2025-2026 and the K3, K2.5 and Nemotron 3 Ultra figures are likely to be revised in later versions.

References

  1. paperGLM-5 Team (Aohan Zeng, Xin Lv, et al.), Zhipu AI and Tsinghua University. GLM-5: from Vibe Coding to Agentic Engineering. arXiv 2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: agentic RL objective and the "only model-generated tokens" statement (§4.1), asynchronous design and Multi-Task Rollout Orchestrator (§4.1.1), TITO, rollout log-probability importance sampling, version-lag drop rule, sample hygiene and DP-aware routing (§4.1.2).
  2. paperKimi Team (Tongtong Bai, Yifan Bai, Yiping Bao, et al.), Moonshot AI. Kimi K3: Open Frontier Intelligence. arXiv 2607.24653v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: partial rollouts with completion fraction and per-token regularization (§4.1.2), budget override, white-box harness (§4.2.1), personal-assistant environments (§4.2.5), resumable sandbox (§5.3.2), preserved thinking and masked JSON block (chat-template appendix), steps scaling with RL FLOPs (Fig. 8).
  3. paperYi, J., Mosk-Aoyama, D., Huang, B., Gala, R., Wang, C., Devare, S. D., Bhardwaj, K., Gupta, A., Kuchaiev, O., Jiao, J., Zhang, J., Srinivasan, V. PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost. NVIDIA and UC Berkeley, arXiv 2603.21383v1, 22 Mar 2026. https://arxiv.org/abs/2603.21383. Accessed 2026-10-27. Used for: pivot selection and functional reward (§3.1), 71% zero-signal turns (§2), Tables 1-2 and 4-6, rollout-turn comparison with end-to-end RL (§4.2, App. A.2). Not in the source map; full text read.
  4. repoNVIDIA. NeMo Gym README. main branch snapshot showing release v0.7.0. https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/main/README.md. Accessed 2026-10-26. Used for: the definition of an environment as dataset, agent harness, verifier and per-task state. Unversioned snapshot read through a page extraction.
  5. officialFarama Foundation. Gymnasium documentation: Basic Usage. Live documentation, no version stamp. https://gymnasium.farama.org/introduction/basic_usage/. Accessed 2026-10-26. Used for: the terminated and truncated flags returned by step. Quote is from a page extraction.
  6. paperJin, B., Zeng, H., Yue, Z., Yoon, J., Arık, S. Ö., Wang, D., Zamani, H., Han, J. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025; arXiv 2503.09516v5, 5 Aug 2025. https://arxiv.org/abs/2503.09516. Accessed 2026-10-27. Used for: retrieved-token loss masking and normalizer (§3.1, Eqs. 2-3), masking ablation (Tables 4 and 6, Fig. 3), PPO against GRPO stability (Fig. 2a). Not in the source map; full text read.
  7. paperKimi Team (Tongtong Bai, Yifan Bai, Yiping Bao, et al.), Moonshot AI. Kimi K2.5: Visual Agentic Intelligence. arXiv 2602.02276v2, 7 Aug 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: PARL architecture, reward and critical steps (§3), objective (§4.4.2, Eq. 1), Agent Swarm results (Figs. 7-8), rollout manager and Token-in-Token-out (Appendix D).
  8. paperWang, Z., Wang, K., Wang, Q., Zhang, P., et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv 2504.20073v2, 26 May 2025. https://arxiv.org/abs/2504.20073. Accessed 2026-10-27. Used for: trajectory-level StarPO, Echo Trap and uncertainty filtering in StarPO-S (§4.2), response-masking ablation (Appendix D, Fig. 12). Not in the source map; full text read.
  9. paperNVIDIA (Aaron Blakeman, Aaron Thomas, et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv 2606.15007v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: unified RLVR stage (§3.2), MOPD algorithm and asynchronous setup (§3.3.1), SWE teacher and search teacher (§3.3.2), limitations (§3.3.5), one-step off-policy training (§3.6.1).
  10. paperKimi Team (Angang Du, Bofei Gao, Bowei Xing, et al.), Moonshot AI. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv 2501.12599v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: partial rollouts, replay buffer and segment exclusion (§2.6.2), mean-reward baseline.
  11. paperKimi Team (Yifan Bai, Yiping Bao, Y. Charles, et al.), Moonshot AI. Kimi K2: Open Agentic Intelligence. arXiv 2507.20534v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: partial rollout in agentic rollout (§3.3.4); full-text search for masking statements.
  12. paperDeepSeek-AI (Aixin Liu, Aoxue Mei, Bangcai Lin, et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv 2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: mixed RL and rewards (§3), advantage (§3.1), thinking context management (§3.2.1), agent task pool (§3.2.3, Table 1), synthetic-task ablation and Table 5 (§4.3), test-time context management (§4.4).
  13. paperDeepSeek-AI (Anyi Xu, Bangcai Lin, Bing Xue, et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: interleaved thinking rule, preemptible rollout service and the length-bias argument against regenerating from scratch (§5.2.3).
  14. paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025; arXiv 2505.10978v3, 28 Oct 2025. https://arxiv.org/abs/2505.10978. Accessed 2026-10-27. Used for: anchor-state grouping and Eqs. 3-9 (§4), Tables 1-2, ablation, step-group dynamics and compute budget (§5.4-5.6), limitations, verl-agent step-wise paradigm (Appendix A). The source map had the abstract only; full text read.
  15. paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948v2 (expanded version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the three limitations of process reward models (Supplement G.2).
  16. paperLLM-Core Xiaomi. MiMo-V2-Flash Technical Report. arXiv 2601.02780v2, 8 Jan 2026. https://arxiv.org/pdf/2601.02780v2. Accessed 2026-10-26. Used for: partial rollout with staleness-aware truncated importance sampling and caps on staleness and partial-sample share (§4.6.2). Abstract-only pointer in the source map; the cached full text was read.
  17. paperZheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., et al. (Qwen Team, Alibaba). Group Sequence Policy Optimization. arXiv 2507.18071v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: the sequence-level clipping ranges 3e-4 and 4e-4 (§4).