Reinforcement Learning for LLM Post-Training · Module 18
Agentic and Multi-Turn RL: Trajectories as Episodes, Credit Assignment, Observation Tokens, and Staleness
An episode is no longer one response: it is several model turns interleaved with tool output, and that changes four things in the objective. This module covers which tokens carry loss, how one trajectory-level reward is split across turns (GiGPO, PARL's critical steps), what partial rollouts do to the policy version behind each token, and why Nemotron 3 Ultra's distillation stage uses single-turn rollouts for most agentic tasks. Afterwards you can specify the loss mask, the advantage and the staleness rule for a multi-turn recipe, and say which reported design fits which task structure.
a), grey segments are environment output the model only reads (o). GLM-5 states that only model-generated tokens are used for optimization1; Kimi K3 pauses generation once a fraction of trajectories has finished, so tails cross policy versions2; PivotRL and Nemotron 3 Ultra train on single turns instead of whole trajectories3.1. The trajectory is a token stream
1.1 What the trainer receives, and the mask
A single-turn RLVR sample is a prompt and one sampled response, and the sequence the trainer scores is the sequence the engine sampled. A multi-turn sample is a message list that grows during rollout: the model emits a turn (reasoning plus a tool call), the harness executes the call and appends the result, and the model continues until it answers, hits a turn or token limit, or the environment ends the episode. NeMo Gym describes the environment in these terms: a dataset of tasks, an agent harness that decides how the model interacts with the world, a verifier, and per-task state4 (an unversioned README snapshot, release v0.7.0). Gymnasium's step returns separate terminated (task completed or failed) and truncated (time limit) flags5. For an LLM the difference matters: a truncated trajectory has no outcome yet, and Section 5.3 shows Nemotron masking exactly those.
Write the rollout as tokens \(y_1,\dots,y_L\) and let \(I(y_t)=1\) for tokens the policy sampled and \(0\) for tokens the environment returned. The policy-gradient estimator differentiates the probability of the actions the policy chose (Module 2). Observation tokens are drawn from the environment's dynamics, not from \(\pi_\theta\), so neither \(\nabla\log\pi_\theta\) nor the ratio \(\rho=\pi_\theta/\pi_{old}\) means anything for them. Search-R1 puts this into the GRPO objective, and applies the same mask inside the KL term6 (§3.1, Eq. 3):
$$ \mathcal J(\theta)=\mathbb E\Big[\frac1G\sum_{i=1}^{G}\frac{1}{\sum_t I(y_{i,t})}\sum_{t:\,I(y_{i,t})=1}\min\big(\rho_{i,t}\hat A_{i,t},\ \mathrm{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_{i,t}\big)\Big]-\beta D_{KL}. $$Two things differ from the single-turn loss of Module 11. The normalizer counts sampled tokens only, so a long tool output does not dilute the gradient of the model's own tokens. And in outcome-reward GRPO the advantage \(\hat A_{i,t}=\hat A_i\) is the same for every unmasked token of trajectory \(i\), however many turns it contains (Section 2 is about that).
The ids in this sequence must be the ids the engine sampled. The gateway that guarantees it (GLM-5's token-in-token-out) is Module 14, Section 2.5, and is not repeated here. Kimi K2.5 states the same design choice: its framework "strictly follows a Token-in-Token-out paradigm" and records the log-probabilities of every engine output for mismatch correction7 (Appendix D). What multi-turn adds is that observation segments travel through the same id stream and need a mask, not a probability.
1.2 What each report says about masking
The source map did not record whether the reports mask environment tokens, so I searched the full text of each. The result is thin: only GLM-5 states the rule for tool output in the passages searched, and Search-R1 is the only paper that measures it.
| Source | What it says | Where |
|---|---|---|
| Search-R1 | Loss masked on retrieved tokens in PPO, GRPO and the KL term; normalizer counts sampled tokens6 | §3.1, Table 4 |
| RAGEN (StarPO) | Response masking enabled vs disabled on Sokoban with a 0.5B model; enabling it gave higher success (figure only)8 | Appendix D, Fig. 12 |
| GLM-5 | "only model-generated tokens are used for optimization, and the environment feedback is ignored in loss computation"1 | §4.1 |
| Kimi K2.5 | Frozen subagents' trajectories "excluded from the optimization objective"; the orchestrator's own tool results: not stated7 | Abstract, §3 |
| Kimi K3 | A pure-JSON fallback block that occurs only in input tokens has its loss masked; tool results: not stated2 | Chat-template appendix |
| Nemotron 3 Ultra | Loss masked on unfinished trajectories; negative advantage on tokens of malformed reasoning or tool calls; tool results: not stated9 | §3.3.2 (SWE teacher) |
| Kimi k1.5 | Under partial rollouts "certain segments can be excluded from loss computation"10 | §2.6.2 |
| Kimi K2, DeepSeek-V3.2, DeepSeek-V4 | Not stated111213 | - |
RAGEN's version of the same ablation ran at 0.5B on Sokoban and reports the direction only8. Both are small-model results on short-horizon tasks with PPO, so they show that the mask is not cosmetic, not how large the effect is at frontier scale.
1.3 History is part of the policy
The policy at turn \(t\) conditions on whatever the harness shows it, and the harness can rewrite history. DeepSeek-V3.2 found that R1's rule of discarding reasoning when the next round arrives "forces the model to redundantly re-reason" through the problem at every tool call. Its rule drops reasoning only when a new user message arrives, keeps it while only tool outputs are appended, and always keeps the tool-call history12 (§3.2.1). DeepSeek-V4 keeps the complete reasoning history across all rounds in tool-calling scenarios, including across user messages13. Kimi K3 "supports only preserved thinking": the think channel stays in history even when empty2. Other rules shrink history instead: DeepSeek-V3.2's search agent applies Summary, Discard-75% or Discard-all once usage passes 80% of the context window12 (§4.4), Nemotron 3 Ultra's search teacher is fine-tuned on trajectories with discard-all and summary behavior9, and K3 lists context-management strategies among the composable modules of its training harness2 (§4.2.1).
This decides how a trajectory becomes training samples (derived; none of the reports describes its packing). If every turn appends to the previous context, the trajectory is one sequence: the engine's log-probability for each token was computed given exactly the prefix that the trainer sees, and one forward pass reproduces all conditionals (Figure 3). If a rule deletes or rewrites earlier tokens, the final sequence is not an extension of the context at turn 2, so the trainer must rebuild each segment's own context and one trajectory becomes several sequences that share one reward. verl-agent, the code base behind GiGPO, takes the second route by design: a "step-wise multi-turn" paradigm that avoids concatenating full histories, with a configurable memory module and a history length of 2 for ALFWorld and WebShop14 (Appendix A, E.2).
m=1), grey segments are environment output. Every blue token receives the same trajectory-level advantage in outcome-reward GRPO. If the harness appends only, the context for each turn is a prefix of the final sequence; if it drops or summarizes earlier tokens, that prefix property fails (Section 1.3).The cost of the choice is plain arithmetic. Let \(p\) be the initial prompt, \(g\) the sampled tokens per turn, \(o\) the observation tokens per turn, \(T\) the turns and \(h\) a history window. One concatenated sequence has \(p+Tg+(T-1)o\) tokens. Per-turn samples with full history sum to \(T(p+g)+(g+o)T(T-1)/2\), quadratic in \(T\). A window of \(h\) previous turns costs \(T(p+g)+(g+o)\sum_{t=1}^{T}\min(t-1,h)\). For scale, GiGPO notes that an ALFWorld episode can run to 50 steps and over 20k tokens14.
| Layout | Samples | Tokens (k) | vs concatenated | Loss-token share (%) |
|---|---|---|---|---|
| Concatenated, append-only | 1 | 63.5 | 1.0x | 28.3 |
| Per-turn, full history | 30 | 991.5 | 15.6x | 1.8 |
| Per-turn, window \(h=2\) | 30 | 197.7 | 3.1x | 9.1 |
Trainer tokens for one trajectory under each layout
Formulas as above; token counts, not FLOPs. Attention cost inside a sequence is superlinear in its length, so the concatenated layout is cheaper per token than the token count alone suggests only when the sequence still fits the trainer's context and activation budget.
Rollout-side prefill has the same shape. GLM-5 routes every request of one rollout to the same data-parallel rank by consistent hashing so the shared prefix hits the KV cache; with that, "prefill cost remains proportional to incremental tokens rather than total context length"1 (§4.1.2). The infrastructure is System Design Modules 27 and 37; the algorithmic point is only that the trainer pays the quadratic bill unless history is append-only or windowed.
2. Splitting credit across turns
2.1 What a trajectory-level baseline does
Group-relative advantage compares trajectory \(i\)'s reward with its group and hands the result to every sampled token (Modules 9 and 11). Every agentic recipe in this module's source set does exactly that. GLM-5's objective is \(r(x,y_i)-\bar r(x)\) over agent traces1 (§4.1). Kimi K2.5's loss uses the same mean-reward baseline inside a token-level objective7 (Eq. 1). DeepSeek-V3.2 uses \(A=R-\operatorname{mean}(R)\) in GRPO12 (§3.1), and Nemotron 3 Ultra's end-to-end SWE stage feeds a binary hidden-test reward to GRPO9. A full-text search of K2, K2.5, K3, V3.2, V4, GLM-5, Nemotron 3 Ultra, MiMo-V2-Flash and Olmo 3 finds no mention of GiGPO, anchor states or step-level advantages.
The consequence is that a wrong turn inside a successful trajectory is reinforced, and a correct turn inside a failed one is punished. On average the signal points the right way, because good turns co-occur with success more often, but the estimator's variance grows with the number of turns that share one scalar (Module 2). What the frontier reports show in response is more samples per task and difficulty filtering of tasks (DeepSeek-V3.2 keeps tasks with non-zero pass@100, Section 6.2), not finer credit.
2.2 GiGPO: step-level groups from repeated states
GiGPO keeps the critic-free group structure and adds a second grouping. Sample \(N\) trajectories of one task from identical initial conditions. The episode-level advantage is GRPO's, with \(F_{norm}\) either the standard deviation or 1 (a rescaled leave-one-out estimator). Collect the set of distinct states \(\tilde s\) that occur anywhere in the group, including repeat visits inside one trajectory, and for each build a step group of the actions taken there and the discounted return that followed14 (§4.1-4.3, Eqs. 3-8):
$$ A^E(\tau_i)=\frac{R(\tau_i)-\operatorname{mean}_j R(\tau_j)}{F_{norm}},\quad A^S(a^{(i)}_t)=\frac{R^{(i)}_t-\operatorname{mean}_{G^S(\tilde s)}R}{F_{norm}},\quad R^{(i)}_t=\sum_{k=t}^{T}\gamma^{k-t}r^{(i)}_k,\quad A=A^E+\omega A^S. $$The clipped objective with a KL term is then applied per step with \(A\) (Eq. 9). Grouping is a hash-map lookup on the state, so it needs no extra rollouts and no value model; the paper uses \(\omega=1\) without tuning, \(\gamma=0.95\) and \(N=8\) on ALFWorld and WebShop, with a reward of 10 for success, 0 for failure and -0.1 for an invalid action (Appendix E.1). The comparison group for an action is "what else happened from this same state", which is the closest a critic-free method gets to a baseline \(V(s)\).
2.3 Evidence and limits
The evidence is two text-environment benchmarks at 1.5B and 7B, three seeds, and a search-QA set. With \(F_{norm}=1\) GiGPO beats GRPO by 13.3 and 10.6 points (ALFWorld success, WebShop success) at 1.5B and by 12.6 and 9.1 points at 7B, which are the abstract's "> 12%" and "> 9%" in absolute points14 (§5.2, Table 1).
| Model | Method | ALFWorld success | WebShop score | WebShop success |
|---|---|---|---|---|
| 1.5B | PPO (with critic) | 54.4 | 73.8 | 51.5 |
| 1.5B | RLOO | 69.7 | 73.9 | 52.1 |
| 1.5B | GRPO | 72.8 | 75.8 | 56.8 |
| 1.5B | GiGPO w/ std | 86.7 | 83.1 | 65.0 |
| 1.5B | GiGPO w/o std | 86.1 | 83.5 | 67.4 |
| 7B | PPO (with critic) | 80.4 | 81.4 | 68.7 |
| 7B | RLOO | 75.5 | 80.3 | 65.7 |
| 7B | GRPO | 77.6 | 79.3 | 66.1 |
| 7B | GiGPO w/ std | 90.8 | 84.4 | 72.8 |
| 7B | GiGPO w/o std | 90.2 | 86.2 | 75.2 |
Three details bound the claim. First, the overhead: anchor grouping takes 0.01 s and the step-advantage arithmetic 0.53 s per iteration, against 362.83 s for the whole iteration (1.5B, ALFWorld). That is \(0.54/362.83=0.15\%\); the paper prints "< 0.002%", which matches a fraction (0.0015) and not a percentage, so I read the unit as a slip. Either way grouping is free next to rollout. For scale, the 1.5B runs used 2 H100 GPUs for 150 iterations (Appendix E.1), which at 362.83 s per iteration is about 15 hours, or 30 GPU-hours (derived). Second, recurrence is empirical: singleton step groups stay under 35% of groups through training, so over 65% of states recur in ALFWorld, and the authors read the early large groups (10 or more members, over 20% of groups at iteration 10) as looping by an immature policy14 (§5.5). Third, when no state repeats \(A^S=0\) and the method reduces to GRPO. The search-QA runs used similarity-based grouping instead: two states match when their longest-matching-subsequence similarity exceeds 0.9. For coding and terminal agents, where each command changes file and process state, exact repeats are plausibly rare after the first few steps, so the approach may degrade towards GRPO there.
2.4 Critics, process rewards and shaping
DeepSeek-R1's supplement gives three reasons it dropped a process reward model: a fine-grained step is hard to define, whether a step is correct is hard to determine, and a model-based PRM "inevitably leads to reward hacking"15 (§G.2). In an agent loop the first reason disappears, because the harness delimits turns, but the second remains. GiGPO sidesteps both by using the environment's own returns, grouped by state (my reading).
The critic is not dead at small scale. At 7B, PPO with a critic beats GRPO and RLOO on ALFWorld (80.4 against 77.6 and 75.5, Table 3). Search-R1 finds GRPO converges faster but "may exhibit instability after trained for a number of steps", PPO is more stable but slower6 (Fig. 2a). RAGEN sees "reward variability cliffs and gradient spikes" in multi-turn training, calls the collapse mode the Echo Trap, notes that declining reward standard deviation often precedes it, and stabilizes by training only on the top \(p\%\) of prompts by reward standard deviation (default 25%): under PPO, keeping 75% extends FrozenLake stability from 100 to 140 steps and keeping 50% avoids collapse8 (§4.2). This is Module 11's dynamic sampling, applied because a multi-turn group has more ways to go uniform.
Shaping lives outside the advantage. Nemotron's SWE teacher says the final reward "does not always help with trajectory-level behaviors", so it masks unfinished trajectories and gives malformed reasoning or tool calls a negative advantage on the offending tokens9 (§3.3.2). Kimi K3 overrides the reward with \(-1\) when a trajectory's budgeted tokens, counting reasoning and tool-call arguments for agentic tasks, exceed \(\tau\) times a per-problem budget2 (§4.1.2). Its personal-assistant environments attach an evaluation criterion to each event, which makes the environment's signal denser than a final pass/fail; the report does not say how event scores enter the advantage (§4.2.5).
3. Orchestrators and frozen subagents: PARL
3.1 What is trained, and the reward
Kimi K2.5's Agent Swarm splits one task across subagents that an orchestrator creates and schedules. Parallel-Agent RL (PARL) trains the orchestrator only. The subagents are "instantiated from fixed intermediate policy checkpoints", their execution trajectories are "excluded from the optimization objective", and their outputs are treated "as environmental observations rather than differentiable decision points"7 (§3). In token-stream terms from Section 1, a subagent's return is an observation segment with mask 0. The stated reason is credit ambiguity: "a correct final answer does not guarantee flawless subagent execution, just as a failure does not imply universal subagent error".
The reward adds two auxiliary terms to the task outcome7 (§3):
$$ r_{\text{PARL}}(x,y)=\lambda_1\,r_{\text{parallel}}+\lambda_2\,r_{\text{finish}}+r_{\text{perf}}(x,y). $$\(r_{\text{perf}}\) is the task-level outcome. \(r_{\text{parallel}}\), an instantiation reward, fights "serial collapse", the local optimum in which the orchestrator never spawns anything. \(r_{\text{finish}}\), the sub-agent finish rate, fights "spurious parallelism", where the orchestrator inflates a parallelism metric by spawning many subagents without a real decomposition. Both \(\lambda\) are annealed to zero during training, so the final policy is optimized for \(r_{\text{perf}}\) alone. The passages I read give no values for \(\lambda_1,\lambda_2\) or the annealing schedule. The pair is a small instance of the pattern Module 19 treats: an auxiliary reward to get past a bad optimum, removed before it becomes the thing the policy exploits.
3.2 Critical steps: a resource constraint that rewards parallelism
Counting total steps would charge the orchestrator for every parallel step. K2.5 instead measures cost by analogy to a critical path. With stages \(t=1,\dots,T\), where \(S^{(t)}_{main}\) is the main agent's steps in stage \(t\) (typically 1) and \(S^{(t)}_{sub,i}\) the steps of the \(i\)-th subagent in a parallel group7 (§3):
$$ \text{CriticalSteps}=\sum_{t=1}^{T}\Big(S^{(t)}_{main}+\max_i S^{(t)}_{sub,i}\Big). $$Training and evaluation are constrained by this quantity, so creating subtasks that do not shorten the longest parallel branch gains nothing under it. Figure 5 counts a toy episode (numbers mine). Stage 1 spawns four subagents that take 5, 3, 8 and 2 steps; stage 2 spawns six that take 4, 4, 9, 2, 3 and 5; stage 3 is the orchestrator's final answer. Critical steps are \((1+8)+(1+9)+1=20\), total steps are \(3+18+27=48\), so the swarm is 2.4 times shorter than one agent doing the same 48 steps in sequence. A perfectly balanced split of the same subagent work (\(\lceil 18/4\rceil=5\), \(\lceil 27/6\rceil=5\)) would give \(6+6+1=13\) critical steps, 3.7 times shorter. The gap between 20 and 13 is the value of decomposing evenly, which is what the metric pays for.
3.3 Why freeze, what it costs, and what is reported
Freezing makes the orchestrator's problem an ordinary single-policy RL problem with a stochastic environment (the subagents), which is what Sections 1 and 2 already handle. The costs are visible in the recipe. The orchestrator is first trained against small subagents and then larger ones, the cluster's inference-instance ratio between subagents and orchestrator is adjusted dynamically, and subagents never improve from the swarm's reward during that stage7 (§3). The report contains no co-training baseline in the passages I read, so "frozen is more stable" is the authors' stated reason and not a measured comparison.
Reported outcomes are for inference, on wide-search tasks: latency "up to 4.5x" lower with item-level F1 up from 72.8% to 79.0% against single-agent baselines (the introduction prints 72.8%, the evaluation text 72.7%), and a swarm that stays at 0.6-1.6 times a baseline latency as the target F1 rises from 30% to 70%, where the single agent grows from about 1.8 to over 7.0 times it (evaluation section, Fig. 8). The report also argues the swarm is "proactive context management": subagents keep bounded local contexts and "only task-relevant outputs" return, which shards context instead of truncating it and, per the authors, beat Discard-all on BrowseComp with fewer critical steps7 (Fig. 7). For training, that has a plain consequence: the orchestrator's trained sequence stays short because subagent traces never enter it. K2.5's rollout manager runs every task as an asynchronous coroutine, lets a task recursively trigger sub-task rollouts, and handles up to 100,000 concurrent agent tasks (Appendix D). Module 28 reads the K1.5-to-K3 line stage by stage.
4. Partial rollouts and staleness
4.1 Three ways to stop waiting for the slowest trajectory
Agentic training lengthens the tail as it succeeds: Kimi K3 reports that average tool-call steps rise consistently as RL FLOPs scale2 (Fig. 8). Module 1 (Section 6) has the long-tail arithmetic and System Design Module 37 the infrastructure taxonomy; this section covers what the algorithm must absorb.
Kimi k1.5 gives each iteration a fixed output-token budget. A trajectory that exceeds it has its unfinished portion saved to a replay buffer and continued in the next iteration. Only the current iteration (iter n) "requires on-policy computation", while segments from iterations \(n-m\) to \(n-1\) "can be efficiently reused from the buffer", and "certain segments can be excluded from loss computation"10 (§2.6.2). Kimi K2 applies the same technique to multi-turn agentic rollouts: long-tail unfinished tasks "paused, and resumed in the next RL iteration"11 (§3.3.4, Agentic Rollout).
Kimi K3 replaces the token budget with a completion fraction. Each iteration keeps \(N\times K\) trajectories in flight, and "the generation phase pauses as soon as a fraction \(\lambda\in(0,1)\) of trajectories completes (i.e., \(\lambda NK\))". Paused rollouts are enqueued and resumed first in the next iteration; once all \(K\) responses of a prompt are done they go straight to policy optimization2 (§4.1.2). A trajectory "naturally spans multiple iterations", and the report's answer to the resulting staleness is that the objective "inherently tolerates such an extreme off-policy regime through a per-token regularization"2. Pausing an agent mid-episode needs the environment to survive the pause, which is why K3 pairs it with a microVM sandbox whose incremental checkpoint and resume take as little as 133 ms and 49 ms (§5.3.2; sandbox design is System Design Module 27). MiMo-V2-Flash also partitions overlong trajectories across steps, "while limiting staleness and the proportion of partial samples in each batch", with "staleness-aware truncated importance sampling"16 (§4.6.2).
4.2 What pausing at a fraction does
Model trajectory durations as i.i.d. exponential with mean \(\mu\), an assumption chosen for a closed form and not taken from any report. Waiting for all \(n=NK\) costs the expected maximum \(\mu H_n\approx\mu(\ln n+0.577)\). Pausing at fraction \(\lambda\) costs the \(\lambda\)-quantile \(t_\lambda=-\mu\ln(1-\lambda)\). For \(n=8{,}192\) and \(\lambda=0.9\): \(H_n=9.59\), \(t_\lambda=2.30\,\mu\), a 4.2-fold shorter iteration; to first order the longest trajectory then spans about \(9.59/2.30\approx4.2\) iterations, so it touches five policy versions. The same model prices the selection effect: the first \(\lambda\) fraction to finish has mean length \(\mu\,[1-(1+q)(1-\lambda)]/\lambda\) with \(q=-\ln(1-\lambda)\), which is \(0.744\,\mu\) at \(\lambda=0.9\), 26% shorter than the population.
Pause-at-fraction under exponential durations (illustrative model)
Formulas: \(E[\max]\approx\ln n+0.577\), \(t_\lambda=-\ln(1-\lambda)\), kept-set mean \(=[1-(1+q)(1-\lambda)]/\lambda\). Agent trajectories with variable turn counts and tool latency are plausibly heavier-tailed than the exponential; if so the real speed-up and span are larger. Use the shape, not the values.
The selection effect is why resuming matters. DeepSeek-V4 states the failure for preempted generation: "it is mathematically incorrect to regenerate unfinished requests from scratch, as this introduces length bias", because "shorter responses are more likely to survive interruption"; its fix is a token-granular write-ahead log plus saved KV cache, so decoding continues instead of restarting13 (§5.2.3). The same logic covers pausing: if the unfinished 10% were dropped, the optimizer would see a batch biased to short trajectories. K3's scheme avoids that by resuming, and pays in staleness. The staleness is also not random. It concentrates on the longest trajectories, and GLM-5 notes that "overly long trajectories can become highly off-policy, which may destabilize training"1 (§4.1.2).
4.3 What the loss sees across versions
Give each token a behavior version \(b_t\), the weights that sampled it. The ratio the learner needs is \(\pi_\theta(y_t\mid s_t)/\pi_{b_t}(y_t\mid s_t)\). If the engine returns \(\log\pi_{b_t}\) with every token, as K2.5's framework does7 (Appendix D), that is one stored number per token however many versions the trajectory crossed. This is why GLM-5 can drop \(\pi_{old}\) and use the rollout log-probabilities as "a direct behavior proxy" when "rollout engines may undergo multiple updates during a single trajectory generation"1 (§4.1.2), and why token-level objectives carry over unchanged to partial rollouts. GLM-5 names the result Direct Double-sided Importance Sampling: the ratio is \(\pi_\theta/\pi_{rollout}\) and tokens outside \([1-\epsilon_\ell,1+\epsilon_h]\) are masked from the gradient, controlling off-policy bias "without tracking historical policy checkpoints". The objectives themselves (K2.5's log-ratio band, K3's per-token regularization, GLM-5's double-sided mask) are Module 12, Sections 5.1 and 5.2; the mismatch between engines that produces a second, version-independent ratio is Module 14.
Two derived cautions. A sequence-level ratio, as in GSPO, can still be assembled from per-token recorded log-probabilities, but its clip range (3e-4 and 4e-4 in GSPO's experiments17, §4) was set for one-version sequences, and none of the reports in this set tests it on multi-version ones. And importance weights correct the probability of each action given its context, not the distribution of contexts: a stale action's context holds observations produced by older actions the current policy may not take. PPO-style methods ignore that term, and so does every report here.
4.4 Bounding the lag
GLM-5 treats staleness as something to cap. It logs for each response the sequence of weight versions \(w_0<\dots<w_k\) that generated it and discards a sample when \(w'-w_0>\tau\), with \(w'\) the current version; the report does not state \(\tau\). The training engine pushes weights to the rollout engine every \(K\) gradient updates, and GLM-5 also resets the optimizer after each such push1 (§4.1.1-4.1.2). Nemotron 3 Ultra runs "a one-step off-policy asynchronous RL setup" in which rollout generation overlaps the previous update, so lag is at most one step by construction; for MOPD it separates the behavior policy from a proximal policy used as the trust-region center9 (§3.3.1, §3.6.1). K3 states a tolerance and no bound, and publishes no staleness histogram or comparison with synchronous rollouts in the passages I read. Section 7 lines the designs up.
5. Turn-local RL: PivotRL and Nemotron 3 Ultra
5.1 Train on the turns that carry signal
The cheapest way to avoid long, stale, uneven trajectories is not to roll them out. PivotRL, from NVIDIA, starts from existing SFT trajectories and decomposes each at assistant boundaries into state-action pairs \((s_t,a^*_t)\), where \(s_t\) is the full history before the \(t\)-th assistant turn. Offline, it samples \(K\) local rollouts from the reference policy at each candidate state, scores them with a verifier, and keeps only turns with nonzero reward variance and a low reward mean (\(\hat\sigma^2(s)>0\), \(\hat\mu(s)<\lambda_{\text{diff}}\)); these are the pivots. Online, it samples \(G\) actions at a pivot, runs only the short rollout needed to score each, and applies a GRPO-style update with a verifier reward \(r_{\text{func}}(s,a)=\mathbb{1}[a\in\mathcal M(s)]\), where \(\mathcal M(s)\) is the set of locally acceptable actions3 (§3.1, Eqs. 4-8). The verifier may be a normalized string or schema check, a task-specific equivalence rule, or a lightweight LLM judge.
Both design choices answer a measured problem. First, a group whose rewards are all equal has zero advantage (Module 11, Section 4); in turn space the paper measures that 71% of randomly sampled turns on \(\tau^2\)-Bench and SWE-Bench give no learning signal. Second, rewarding only an exact match with the demonstrated action is too strict in a generative action space, where many tool calls are valid.
5.2 What the numbers show, and what they do not
The turn count in Figure 7 checks out. PivotRL trains with a batch of 1,024 (64 prompts times 16 generations) at one turn per sample and reaches 32.67% at step 130, so \(130\times1{,}024=133{,}120\) rollout turns. End-to-end RL uses 512 trajectories per step (16 prompts times 32 generations), reaches the same accuracy at about step 72 and about 542K cumulative turns, which implies \(542{,}000/(72\times512)\approx14.7\) turns per trajectory, inside the reported 12-25. The ratio is 4.1 in turns and the report gives about 5.5 in wall-clock, so an end-to-end turn costs roughly 1.35 times a single-turn one in time (derived; long contexts and tool latency are the likely cause, which the report does not break out)3 (§4.2, App. A.2).
Four limits sit around the headline. The count is training samples; the \(K\) offline profiling rollouts per candidate turn are not part of that total in the text I read. The 32.67% is below same-data SFT on the same benchmark (37.40%, Table 1), so on SWE-Bench PivotRL's case is cost against end-to-end RL and retention outside the domain, not in-domain accuracy. The SWE-Bench local verifier "matches tool-call names only", so what is learned turn by turn there is which kind of operation to run next, with final success judged only by the full harness. And the out-of-domain advantage (+0.21 against SFT's -9.83 average change; the introduction prints -9.48) comes from one base model, Qwen3-30B-A3B-Thinking-25073 (Tables 1-2, App. A.2).
5.3 How Nemotron 3 Ultra mixes both
Nemotron 3 Ultra does not choose. Its unified RLVR stage spans terminal, office, SWE, search, tool-calling and non-agentic environments, with a deliberately diverse set of harness implementations "reducing overfitting to any particular harness design", asynchronous GRPO, "a global batch size of 8192, with each sample generating 16 rollouts", and a 48K maximum generation length later raised to 64K9 (§3.2). The report does not say whether 8,192 counts prompts (then 131,072 rollouts per step) or rollouts (then 512 prompts), so both readings are shown rather than resolved. Its SWE teacher runs three stages: SFT on agentic data, PivotRL on single-step agentic environments, then an end-to-end SWE-RL stage where the model works a repository over multiple turns and a binary hidden-test reward feeds GRPO, with 192K generation length and at most 200 agent turns. That stage masks unfinished trajectories (maximum turns or timeouts), penalizes malformed reasoning and tool calls, and closes two channels by which the agent could read the gold patch: future commits are physically deleted from the in-container repository, and a runtime filter blocks remote git operations and downloads from GitHub domains (§3.3.2; Module 19 covers the exploit class).
In the MOPD stage (Module 22), the student's rollouts get dense per-token rewards from a domain teacher, with 1,024 prompts per batch and one rollout per prompt, because "using multiple rollouts did not yield additional benefits" in their ablations. The report is candid about agentic tasks there: end-to-end agentic rollouts mixed with reasoning rollouts caused "substantial training inefficiency because rollout times can differ dramatically", so "we use single-turn rollouts, similar to PivotRL (Yi et al., 2026), for most of the agentic tasks". Whether end-to-end rollouts would gain more, and how to make them robust to distribution mismatch, is listed as open9 (§3.3.1, §3.3.5).
6. GLM-5 and DeepSeek-V3.2: two end-to-end designs
6.1 GLM-5: asynchronous traces, sample hygiene
GLM-5's agentic stage is a plain group-wise objective over agent traces (advantage \(r-\bar r\), model-generated tokens only), wrapped in infrastructure that keeps the learner fed. The training and inference engines sit on different GPUs; the inference engine generates continuously, a batch goes to the learner once a trajectory-count threshold is reached, and updated weights return every \(K\) gradient updates1 (§4.1.1). Because tasks differ in tools and rollout logic, each task runs as an independent microservice registered with a central Multi-Task Rollout Orchestrator that sets the per-task rollout ratio and generation speed. All tasks' trajectories are standardized into one message-list format, and the orchestrator supports "over 1k concurrent rollouts" and dynamic changes to task sampling ratios. The scheduling is infrastructure (System Design Modules 37 and 38); the algorithm-side content is what happens to the samples.
GLM-5 excludes samples whose failure came from the sandbox ("environment crashes"), on the grounds that they "reflect environment instability rather than the model's capability". Exclusion breaks GRPO's fixed group size, so the rule is: if the valid samples number more than half the group, pad by repeating valid ones; otherwise drop the group1 (§4.1.2). With a group of 16, that pads groups with 9 to 15 valid samples and drops groups with 8 or fewer. The report does not say how the repeats are chosen. Two consequences follow (derived). Padding by repetition gives some samples double weight in the group mean. And if sandboxes fail more often on long or hard tasks, the survivors skew easy, the same survival effect as the length bias of Section 4.2. Neither is measured in the report.
6.2 DeepSeek-V3.2: synthesized tasks inside one mixed stage
DeepSeek-V3.2's agentic RL is not a separate algorithm. It is GRPO in the same single stage that merges reasoning, agent and alignment training, with rule-based outcome reward, a length penalty and a language-consistency reward for reasoning and agent tasks12 (§3). What is agent-specific is the task supply and the context rule of Section 1.3. Table 4 counts the task pool; the four rows sum to 85,267, the "85,000 complex prompts" of the introduction. Only the general-agent row (4,417 tasks, 1,827 environments) is synthesized end to end; the other three populations run in real environments.
| Agent type | Tasks | Environment | Prompts |
|---|---|---|---|
| Code agent | 24,667 | real | extracted |
| Search agent | 50,275 | real | synthesized |
| General agent | 4,417 | synthesized | synthesized |
| Code interpreter | 5,908 | real | extracted |
| Total | 85,267 |
The general-agent pool is built by an agent: for a task category it generates a sandbox database, task-specific tools, a task with a Python solution and a Python verifier that must agree, and then raises the difficulty step by step. After an RL pass with DeepSeek-V3.2 the pipeline keeps only instances with non-zero pass@100, which leaves 1,827 environments and 4,417 tasks (the table's general-agent row). On a 50-instance sample the difficulty is real: pass@1 is 12% for DeepSeek-V3.2-Exp against 34% (Sonnet-4.5), 51% (Gemini-3.0 Pro) and 62% (GPT-5-Thinking)12 (§3.2.3, §4.3, Table 5). The ablation that carries the argument is qualitative: RL on synthetic general-agent data alone, from the SFT checkpoint in non-thinking mode, improves Tau2Bench, MCP-Mark and MCP-Universe, while RL on code and search environments alone does not (Figure 5, no numbers in the text). Environment design is Module 17's subject; the point for this module is that V3.2 improves tool-use generalization by changing the data, and the multi-turn mechanics are the single-turn recipe unchanged.
7. The designs side by side
| Report | Rollout unit | Advantage and credit | Staleness | Unfinished or failed |
|---|---|---|---|---|
| Kimi k1.510 | One long response, split across iterations by a token budget | Mean-reward baseline | Earlier segments reused from a replay buffer; current segment on-policy | Segments may be excluded from loss; repeats stopped and penalized |
| Kimi K2.57 | Trajectory; swarm trains the orchestrator only | \(r-\bar r\) token-level; PARL auxiliary rewards annealed to 0 | Gradient mask outside a log-ratio band | Subagent trajectories excluded |
| Kimi K32 | Trajectory to 1M-token context; pause at \(\lambda NK\) | K2.5 algorithm; -1 override above \(\tau b_0\) tokens | Per-token regularization; no bound stated | Resumed next iteration in a checkpointed sandbox |
| GLM-51 | Message-list trace, 1k+ concurrent | \(r-\bar r\); model tokens only | Rollout log-probs as behavior proxy; two-sided mask; drop if \(w'-w_0>\tau\) | Sandbox crashes excluded; group padded or dropped |
| Nemotron 3 Ultra9 | Single-turn for most MOPD agentic tasks; end-to-end SWE to 200 turns | Binary reward in GRPO; dense teacher reward in MOPD; negative advantage on malformed tokens | One-step off-policy; behavior and proximal policies in MOPD | Loss masked on unfinished trajectories |
| DeepSeek-V3.212 | Trajectory; reasoning kept across tool results | \(A=R-\operatorname{mean}(R)\); outcome, length and language rewards | Sequence masking and routing replay (Module 12); nothing agent-specific | Tasks filtered to non-zero pass@100; unfinished: not stated |
| MiMo-V2-Flash16 | Trajectory; overlong ones partitioned across steps | Not stated | Staleness-aware truncated IS; caps on staleness and on partial-sample share | Not stated |
Three patterns hold across the rows. Advantage is trajectory-level everywhere; the only step-level credit in this set is GiGPO's, outside the frontier reports. Staleness has four answers: tolerate (K3), cap (GLM-5, MiMo), avoid by construction (Nemotron's one-step lag), or never create it (V3.2 has no partial rollouts in the passages read). Unfinished trajectories are resumed (K3), masked (Nemotron) or dropped (GLM-5, under its staleness rule). Masking and dropping remove trajectories by length or timeout, which reintroduces the survivor bias of Section 4.2 (derived); only resumption keeps the long tail in the batch, at the price of staleness.
Where experts disagree
Three questions are open for multi-turn RL: what unit receives credit, whether to train on whole trajectories or on single turns, and how much policy staleness to accept. The reports answer them differently and almost never ablate one answer against another, so the positions below are designs, not measured rivals.
Trajectory-level outcome reward is enough
GLM-5, Kimi K2.5 and K3, DeepSeek-V3.2, Nemotron 3 Ultra's end-to-end SWE stage17129; RAGEN's StarPO is trajectory-level too8
Scale and filtering average out turn-level noise. A step-level signal needs a state definition or a learned model, and the simple objective already runs at 1,000 to 100,000 concurrent rollouts.
- No extra machinery; the Module 12 and 14 fixes apply unchanged
- Works with any harness, including ones that rewrite history
- No frontier report ablates it against step-level credit
- RAGEN reports Echo-Trap collapse at small scale; whether frontier runs hit it is not reported
Add step-level credit from the environment, with no learned PRM
GiGPO (NTU, NeurIPS 2025)14; R1's supplement gives the case against learned PRMs15
Compare actions taken from the same state across trajectories and add that to the episode advantage. It is critic-free and nearly free to compute.
- +12.6 to +13.3 points ALFWorld success over GRPO at 1.5B and 7B (3 seeds)
- Falls back to GRPO when no state repeats
- Needs recurring states; evidence stops at 7B on ALFWorld, WebShop and search QA
- None of the nine frontier-class reports searched mentions it
Stop rolling out whole trajectories; train on pivots
PivotRL (NVIDIA, UC Berkeley)3; Nemotron 3 Ultra's MOPD uses single-turn rollouts for most agentic tasks9
Most turns carry no gradient under group normalization, and local rollouts remove tails and staleness entirely.
- About 4.1 times fewer rollout turns and 5.5 times less wall-clock to a matched SWE-Bench accuracy
- Keeps out-of-domain skills (+0.21 against SFT's -9.83)
- 32.67% is below same-data SFT (37.40%) on SWE-Bench Verified, and the SWE verifier matches tool names only
- Nemotron lists end-to-end rollouts as an open question
Keep end-to-end rollouts and manage staleness: tolerate or cap
Tolerate: Kimi K32. Cap: GLM-5 and MiMo-V2-Flash116. Resume instead of restarting: DeepSeek-V413
Tails are the cost of long-horizon training, so keep them and decide where the off-policy error goes: into the objective (K3's per-token regularization) or into a rule that drops samples older than \(\tau\) versions (GLM-5).
- Trains on the behavior that only long episodes show
- Resuming avoids the length bias of restarting
- No staleness histogram or synchronous baseline is published for K3
- \(\tau\) and the clip values are not stated in the passages read
Trajectory-level outcome reward is the default for good reason: every large run in this set uses it and none, in the passages read, reports a failure attributed to coarse credit. That is absence of evidence, not a result. GiGPO is the strongest case where states recur and horizons are tens of steps; for coding and terminal agents I would not expect exact state matches to fire, so the experiment that would settle it is GiGPO with similarity-based state keys against GRPO on a SWE-style task at 30B or more, with the group-size histogram reported. Turn-local RL is a cost and forgetting tool, not an accuracy tool: use it to bootstrap agentic skill from SFT data and where rollout cost dominates, and keep an end-to-end stage for recovery and long-horizon behavior, which is what Nemotron's SWE teacher does. On staleness, cap by default, because a cap is cheap to implement and a failure is visible in the data; K3's tolerance claim would be credible with a lag sweep, which no report has. Three of the four positions are compatible, which suggests the disagreement is about emphasis and budget more than correctness.
What is changing
Context length is the driver. Kimi K3 trains agents at 1M-token context with co-located partial rollouts, an external KV pool and resumable sandboxes2, which turns the quadratic cost of Section 1.3 and the staleness of Section 4 into first-order design constraints. Training-time harness diversity (K3's composable modules, Nemotron's mixed harnesses) pushes history rules into the training distribution, so Section 1.3's segment bookkeeping will matter more. PivotRL's adoption in Nemotron 3 Super and Ultra3 suggests turn-local RL becomes a standard first stage. Open items: step-level credit at frontier scale, co-training orchestrator and subagents, and how multi-teacher distillation (Module 22) handles long-horizon rollouts. The reports used here are dated 2025-2026 and the K3, K2.5 and Nemotron 3 Ultra figures are likely to be revised in later versions.
References
- paperGLM-5 Team (Aohan Zeng, Xin Lv, et al.), Zhipu AI and Tsinghua University. GLM-5: from Vibe Coding to Agentic Engineering. arXiv 2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: agentic RL objective and the "only model-generated tokens" statement (§4.1), asynchronous design and Multi-Task Rollout Orchestrator (§4.1.1), TITO, rollout log-probability importance sampling, version-lag drop rule, sample hygiene and DP-aware routing (§4.1.2).
- paperKimi Team (Tongtong Bai, Yifan Bai, Yiping Bao, et al.), Moonshot AI. Kimi K3: Open Frontier Intelligence. arXiv 2607.24653v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Used for: partial rollouts with completion fraction and per-token regularization (§4.1.2), budget override, white-box harness (§4.2.1), personal-assistant environments (§4.2.5), resumable sandbox (§5.3.2), preserved thinking and masked JSON block (chat-template appendix), steps scaling with RL FLOPs (Fig. 8).
- paperYi, J., Mosk-Aoyama, D., Huang, B., Gala, R., Wang, C., Devare, S. D., Bhardwaj, K., Gupta, A., Kuchaiev, O., Jiao, J., Zhang, J., Srinivasan, V. PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost. NVIDIA and UC Berkeley, arXiv 2603.21383v1, 22 Mar 2026. https://arxiv.org/abs/2603.21383. Accessed 2026-10-27. Used for: pivot selection and functional reward (§3.1), 71% zero-signal turns (§2), Tables 1-2 and 4-6, rollout-turn comparison with end-to-end RL (§4.2, App. A.2). Not in the source map; full text read.
- repoNVIDIA. NeMo Gym README. main branch snapshot showing release v0.7.0. https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/main/README.md. Accessed 2026-10-26. Used for: the definition of an environment as dataset, agent harness, verifier and per-task state. Unversioned snapshot read through a page extraction.
- officialFarama Foundation. Gymnasium documentation: Basic Usage. Live documentation, no version stamp. https://gymnasium.farama.org/introduction/basic_usage/. Accessed 2026-10-26. Used for: the
terminatedandtruncatedflags returned bystep. Quote is from a page extraction. - paperJin, B., Zeng, H., Yue, Z., Yoon, J., Arık, S. Ö., Wang, D., Zamani, H., Han, J. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025; arXiv 2503.09516v5, 5 Aug 2025. https://arxiv.org/abs/2503.09516. Accessed 2026-10-27. Used for: retrieved-token loss masking and normalizer (§3.1, Eqs. 2-3), masking ablation (Tables 4 and 6, Fig. 3), PPO against GRPO stability (Fig. 2a). Not in the source map; full text read.
- paperKimi Team (Tongtong Bai, Yifan Bai, Yiping Bao, et al.), Moonshot AI. Kimi K2.5: Visual Agentic Intelligence. arXiv 2602.02276v2, 7 Aug 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Used for: PARL architecture, reward and critical steps (§3), objective (§4.4.2, Eq. 1), Agent Swarm results (Figs. 7-8), rollout manager and Token-in-Token-out (Appendix D).
- paperWang, Z., Wang, K., Wang, Q., Zhang, P., et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv 2504.20073v2, 26 May 2025. https://arxiv.org/abs/2504.20073. Accessed 2026-10-27. Used for: trajectory-level StarPO, Echo Trap and uncertainty filtering in StarPO-S (§4.2), response-masking ablation (Appendix D, Fig. 12). Not in the source map; full text read.
- paperNVIDIA (Aaron Blakeman, Aaron Thomas, et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv 2606.15007v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: unified RLVR stage (§3.2), MOPD algorithm and asynchronous setup (§3.3.1), SWE teacher and search teacher (§3.3.2), limitations (§3.3.5), one-step off-policy training (§3.6.1).
- paperKimi Team (Angang Du, Bofei Gao, Bowei Xing, et al.), Moonshot AI. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv 2501.12599v4, 3 Jun 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: partial rollouts, replay buffer and segment exclusion (§2.6.2), mean-reward baseline.
- paperKimi Team (Yifan Bai, Yiping Bao, Y. Charles, et al.), Moonshot AI. Kimi K2: Open Agentic Intelligence. arXiv 2507.20534v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: partial rollout in agentic rollout (§3.3.4); full-text search for masking statements.
- paperDeepSeek-AI (Aixin Liu, Aoxue Mei, Bangcai Lin, et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv 2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: mixed RL and rewards (§3), advantage (§3.1), thinking context management (§3.2.1), agent task pool (§3.2.3, Table 1), synthetic-task ablation and Table 5 (§4.3), test-time context management (§4.4).
- paperDeepSeek-AI (Anyi Xu, Bangcai Lin, Bing Xue, et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: interleaved thinking rule, preemptible rollout service and the length-bias argument against regenerating from scratch (§5.2.3).
- paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025; arXiv 2505.10978v3, 28 Oct 2025. https://arxiv.org/abs/2505.10978. Accessed 2026-10-27. Used for: anchor-state grouping and Eqs. 3-9 (§4), Tables 1-2, ablation, step-group dynamics and compute budget (§5.4-5.6), limitations, verl-agent step-wise paradigm (Appendix A). The source map had the abstract only; full text read.
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948v2 (expanded version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: the three limitations of process reward models (Supplement G.2).
- paperLLM-Core Xiaomi. MiMo-V2-Flash Technical Report. arXiv 2601.02780v2, 8 Jan 2026. https://arxiv.org/pdf/2601.02780v2. Accessed 2026-10-26. Used for: partial rollout with staleness-aware truncated importance sampling and caps on staleness and partial-sample share (§4.6.2). Abstract-only pointer in the source map; the cached full text was read.
- paperZheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., et al. (Qwen Team, Alibaba). Group Sequence Policy Optimization. arXiv 2507.18071v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: the sequence-level clipping ranges 3e-4 and 4e-4 (§4).