System Design · Module 37
RL Training Infrastructure: Environments as Services, Rollout Generation, and Trainer/Rollout Decoupling
Module 36 showed that RLHF wall-clock time is dominated by sample generation, not gradient updates. This module treats that observation as an architecture problem: it packages the state/action/reward loop as a reusable "environment" service, shows that generating a rollout is the same continuous-batching, paged-KV-cache inference workload you already know from serving, and derives the dominant answer to the resulting question -- an inference pool and a training pool running continuously in parallel, trading policy weights across a channel, with a deliberate policy for how stale a rollout is allowed to get before the trainer stops trusting it.
1. Environments as services
Every RL post-training run needs the same three task-specific ingredients -- a dataset of prompts or starting states, a way for the policy to act on the world (tool calls, a shell, a browser, a math checker), and a function that turns the result of an action into a scalar reward -- wired into the same shared machinery: a rollout engine that runs the policy, a trainer that consumes the resulting trajectories, and the staleness and scheduling logic in between. The practical problem this creates is combinatorial: N tasks times M pieces of shared infrastructure should not require N×M integrations, each hand-rolled by whichever team owns the task. Prime Intellect's Environments Hub states the fix directly: an environment is a package that "defines the world, rules and feedback loop of state, action and reward"1, built specifically so that "effort" -- the dataset, the tool harness, the reward function -- stays task-specific while everything downstream of it (the eval runner, the RL trainer) is reused unmodified across every environment that implements the same interface1. The word "downstream" is doing real work in that sentence: the same environment package is explicitly framed as reusable for "evaluation or RL training"1, not RL training alone, because a benchmark run and a training rollout batch are, mechanically, the same call into the same step() function -- an eval harness simply never feeds the resulting trajectories to a trainer. One environment implementation therefore serves both purposes with no extra engineering, which is the same "write once, reuse everywhere" argument that motivates most of the interface boundaries elsewhere in this course, applied here to a much smaller and more specialized contract than a general service API.
This is the same interface-boundary discipline Module 10 establishes for ordinary microservices, applied to a much narrower contract: instead of an arbitrary RPC surface, an environment exposes exactly one loop -- reset() -> state, step(action) -> (next_state, reward, done) -- and everything an RL framework needs to drive training (batching prompts, collecting trajectories, computing advantages, feeding a replay buffer) is written once against that loop rather than once per task. Prime Intellect's own trainer, prime-rl, is built to consume environments natively through this interface1, and the practical proof that the boundary holds is adoption speed rather than a benchmark number: in a private beta the week before the Hub's public launch, "over 30 researchers and companies" had already contributed environments built against the same interface1 -- thirty independent task implementations reusing one piece of shared RL infrastructure is exactly the payoff a well-drawn service boundary is supposed to produce.
1.1 Sandboxes as verifiers
The reward function is where this boundary gets interesting for anything beyond a multiple-choice-style task with a string-match answer. A coding task's reward function has to actually run the model's proposed patch against a test suite; a tool-use task has to actually execute the tool call and observe what comes back. Both require running untrusted, model-generated code, which is precisely the problem Module 27 designs a solution for at the harness layer: an isolated execution environment (microVM, gVisor, or a comparable mechanism) that runs code without letting it read another tenant's data, reach the network unexpectedly, or escape to the host. Prime Intellect's framing makes the reuse explicit rather than incidental: "sandboxes... plug directly into Verifier Environments for secure code execution"1 -- the sandbox is not a training-specific invention, it is the same isolation layer Module 27 already specifies for a chat agent's code-execution tool, now called from inside an environment's step() function instead of from a user-facing harness turn.
Nothing about the isolation-mechanism trade-offs changes when the caller is a reward function instead of a chat session: a verifier that runs one candidate solution per rollout, at whatever concurrency the rollout batch size demands, is still choosing between a microVM's stronger isolation and slower cold start, a user-space kernel's syscall interception, or a container's shared-kernel speed, for exactly the reasons Module 27 works through in depth. What is different is the traffic shape. A chat harness spins up a sandbox per user session and expects it to live for minutes to hours; a verifier spun up inside an RL training loop is created and destroyed once per rollout, and a training run generating hundreds of rollouts per batch (Section 2.3 works the concrete arithmetic) needs hundreds of verifier sandboxes cycling in parallel, on the same schedule as the inference pool that is generating the actions those sandboxes will grade. Warm-pool reuse and fast snapshot/restore -- Module 27's answer to "don't destroy a sandbox when a turn ends" -- matter even more here, because a slow-to-provision verifier sits directly on the training loop's critical path in a way a slow-to-provision chat sandbox merely degrades one user's latency. Module 27's own fleet-density argument -- that most sandboxes are idle most of the time, so a host can be safely oversubscribed well past its literal memory and CPU capacity -- applies to a fleet of verifier sandboxes exactly as it does to a fleet of chat-session sandboxes; the only change is that a verifier fleet's occupancy is driven by the rollout pool's batch size and group size (Section 2.2) instead of by how many human users happen to be active at once.
reset()/step() contract1. The sandbox box is Module 27's isolation-mechanism menu called from inside a reward function rather than from a chat harness.2. Rollout generation is an inference-serving workload
Strip away the RL vocabulary -- environment, rollout, trajectory -- and the rollout/inference pool in Figure 1 is doing exactly what a multi-tenant LLM serving deployment does: taking a batch of prompts (here, environment states) and running autoregressive decode until each sequence terminates, at whatever concurrency the accelerator's memory allows. Every mechanism the multi-node LLM inference course builds up for that workload transfers here unmodified, because it is not analogous machinery -- it is the same machinery, called from a training loop instead of a public API.
2.1 Continuous batching transfers unmodified
A naive batch-generation loop waits for every sequence in a batch to finish before starting the next batch, which wastes the accelerator on every sequence that terminates early while longer sequences in the same batch keep running. Orca's iteration-level scheduling fixes this by scheduling at the granularity of one decode iteration rather than one whole request: the engine runs a single forward pass over whatever sequences are currently active, and a sequence that finishes is evicted and replaced by a new one from the queue on the very next iteration, without waiting for its batch-mates3. Because a Transformer's attention operation cannot naively batch sequences of different lengths the way its feed-forward layers can, Orca pairs iteration-level scheduling with selective batching -- batching only the operations (linear layers, layer norm) that tolerate mixed-length inputs, and running attention per-sequence -- and reports a 36.9× throughput improvement over NVIDIA FasterTransformer at matched latency on a 175B-parameter model as a result3. This pairing is what every serving engine you already work with (vLLM, SGLang, TensorRT-LLM) calls continuous batching, and a rollout pool generating environment actions has precisely the same shape of workload that motivated it: many sequences of unpredictable, data-dependent length (a coding agent's turn can be five tokens or five thousand), started and finished continuously rather than in synchronized rounds. A rollout engine that instead waits for an entire batch of rollouts to complete before admitting new prompts is reintroducing the exact inefficiency Orca's scheduling change was built to remove -- there is no RL-specific reason to make that mistake, and no RL-specific technique that improves on the fix.
2.2 Paged KV-cache allocation transfers unmodified
The second piece is memory, not scheduling. PagedAttention breaks a sequence's key-value cache into fixed-size blocks addressed through a per-sequence page table, the same idea as OS virtual memory applied to attention state, which lets the allocator hand out cache capacity in small increments instead of reserving a worst-case-length contiguous buffer per sequence4. The measured effect against FasterTransformer and Orca's own (non-paged) implementation is a 2-4× throughput improvement at equivalent latency, "near-zero waste" in KV-cache memory, and -- the feature that matters most for a rollout pool specifically -- "flexible sharing of KV cache within and across requests"4. Module 36 documents exactly this sharing mechanism already carrying its weight inside a training pipeline: Llama 3's rejection-sampling step draws K=10-30 completions from one identical prompt, and PagedAttention lets the entire prompt-length KV cache be written once and referenced by all K continuations, contributing most of a measured >2× rejection-sampling speedup4.
RL rollout generation reproduces the same access pattern at a larger scale: GRPO-family algorithms (Module 38 names several -- GRPO, DAPO, GSPO, RLOO) compute an advantage by sampling a group of completions per prompt and comparing them against each other, which means every rollout batch is, by construction, a set of prompts each sampled many times over -- exactly the shared-prefix structure PagedAttention's cross-request page sharing was built to exploit. A rollout pool that did not page its KV cache would pay for the same prompt prefix once per sample instead of once per prompt, multiplying its memory footprint by the group size for no benefit. Nothing about this argument is specific to training: it is the identical prefix-sharing case Module 28 covers for prompt caching in ordinary serving, reappearing because the workload -- many correlated completions of a shared prefix -- is the same workload, not a coincidence.
To see the magnitude, decompose Section 2.3's 512-sequence rollout batch by an illustrative group size: a group of \(G = 16\) samples per prompt (an ordinary GRPO configuration) turns 512 sequences into 32 distinct prompts, each generated 16 times. Without page sharing, the rollout pool reserves 512 independent copies of each prompt's prefix KV cache; with it, the pool holds 32 -- a 16× reduction in prefix-cache footprint before a single output token is generated, freed up for more concurrent sequences or a larger group size at the same memory budget. This is an illustrative decomposition of the already-cited batch size, not a separately measured figure, but it is exactly the same arithmetic Module 36 works through for Llama 3's K=10-30 rejection-sampling groups, applied to an RL group instead of a rejection-sampling group.
2.3 Why generation dominates wall-clock time
Module 36 already establishes the headline number from the framework side: OpenRLHF states plainly that "RLHF training spends 80% of the time on sample generation"5, against roughly 20% for the batched forward-and-backward training step. Section 2.1-2.2 explain the mechanism behind that split -- generation is memory-bandwidth-bound autoregressive decode, one token per sequence per forward pass, while a training step is a fully batched, compute-dense pass over sequences whose length is already known -- but the number is worth grounding in absolute time, not just a percentage, because a percentage does not tell you whether an iteration takes ninety seconds or four hours.
The Hugging Face survey of open RL libraries works this arithmetic directly for a realistic rollout batch: for a 32B-parameter model generating 32K output tokens per rollout across 512 sequences, at roughly 1.2K tokens/second of decode throughput on a single H100, total generation time for that one batch is approximately 3.7 hours2. That is not a pathological configuration -- 32K-token rollouts are ordinary for a reasoning-heavy RLVR task where the policy has to think through a proof or a multi-step agent trajectory before the environment can score it, and 512 sequences is a modest batch size for a group-sampling algorithm running even a small group size across a few dozen prompts.
The composite in Figure 3 is deliberately conservative about what it claims: the 3.7-hour figure is a real, sourced number for one concrete configuration2; the 0.9-hour training bar is that same figure rescaled by a ratio measured on a different framework and workload5, shown to make the relative-scale argument visible rather than to assert a precise measured duration. Both readings support the same conclusion: whether the true ratio is 80/20 or something else entirely for a given model and rollout length, generation is not a rounding error next to the training step, and any architecture that makes the training GPUs sit idle for the full length of the orange bar before starting the blue bar is leaving most of a training run's wall-clock time on the table. That is the entire motivation for Section 3's decoupled architecture.
2.4 What does not transfer: the verifier round-trip
The "unmodified" claim in this section's title has one real exception, and it is worth stating precisely rather than glossing over. An ordinary serving reply is handed back to the client and the sequence's resources are freed the moment the last token streams out -- the engine's job ends there. A rollout's sequence is not done at that point: Section 1.1's environment still has to execute the action (run the test suite, call the tool, check the proof) and return a reward before that trajectory is usable, and until the reward comes back, the trainer does not know whether this rollout belongs in the admitted batch at all. If the verifier -- Module 27's sandbox mechanisms, called from inside the environment's step() function -- is slow relative to generation, it becomes a second bottleneck sitting after the one Figure 3 already measures, not a free postscript to it.
This is also where Module 36's PagedAttention admission-control nuance resurfaces in a new form. Module 36 shows Llama 3 declining PagedAttention's dynamic, memory-pressure-driven scheduling for rejection sampling specifically because an offline batch job can bound its output length in advance and avoid the risk of a mid-generation swap. An RL rollout pool faces a related but distinct version of the same question: whether to free a finished sequence's KV pages immediately (assuming the verifier will succeed and the trajectory is likely admissible) or hold them until the reward actually comes back (in case a fast retry or a partial-credit re-check needs the same prefix again). Neither this module's sources nor Module 36's document which choice production frameworks make here -- it is a genuine gap in what the survey (S63) and the framework READMEs disclose, worth flagging rather than assuming away.
3. The dominant decoupled architecture
The Hugging Face survey looked across sixteen open-source RL libraries -- AReaL, ART, Atropos, MILES, NeMo-RL, OAT, open-instruct, PipelineRL, PRIME-RL, ROLL, SkyRL, slime, TorchForge, Tunix, verl, and verifiers-rl2 -- built by different organizations, on different codebases, targeting different algorithms, and found that nearly all of them converge on the same top-level shape: "the inference pool runs vLLM or SGLang continuously; the training pool runs the optimizer continuously. The two pools communicate via a weight synchronisation protocol"2. That is Figure 1's rollout pool and training pool, restated as the field's actual, converged answer rather than this module's own framing.
3.1 Two pools, one channel
The word "continuously" is the load-bearing part of that sentence. A lock-step design -- generate a full batch, stop the inference engine, run the training step, stop the trainer, resume generation with new weights -- is a valid way to implement the same dataflow, and Section 4 of Module 36 shows it inside OpenRLHF's tightly-coupled placement: the framework's dataflow keeps generation and training on a shared schedule, paying the 80/20 imbalance from Section 2.3 as idle time on whichever pool of GPUs is not currently active. The disaggregated answer instead keeps the inference pool decoding continuously -- accepting new prompts on every finished sequence, per Section 2.1's iteration-level scheduling -- and keeps the training pool consuming whatever admitted batch (Section 4) is ready, on its own schedule, rather than waiting for the entire fleet of rollout GPUs to finish one synchronized round. The two pools are physically separate: a rollout GPU is running an inference engine's kernels (paged attention, continuous-batching scheduler) and a training GPU is running the optimizer's kernels (fused Adam, gradient all-reduce), and nothing requires them to be the same physical device or even the same node count. Because Figure 3's imbalance puts the throughput bottleneck squarely on generation, the two pools can be sized independently of each other -- as many GPUs as the rollout workload's decode throughput demands on one side, only as many as the much shorter batched training step needs on the other -- rather than both being forced to match whatever a single shared allocation happens to provide.
This is the continuous, disaggregated end of a placement spectrum Module 36 already opens from the framework side: whether the actor, critic, reference, and reward roles run on dedicated GPU pools that scale independently, or share a colocated hybrid engine that time-multiplexes one pool of GPUs between generation and training. Module 36's own verdict is that dedicated pools win when the models differ enough in size for independent scaling to matter and a team values debuggability, while colocated engines win when GPU budget is the harder constraint and the idle-pool waste of a dedicated layout is too expensive to leave on the table. Figure 4 draws the dedicated-pool, always-on version of that trade-off; a colocated implementation keeps the same environment interface, the same staleness gate, and the same weight-sync channel in Table 1, but time-slices a smaller set of physical GPUs between the two roles instead of running each on its own hardware -- the architecture this section describes and the placement decision Module 36 covers are two different axes of the same design, not competing answers to one question.
3.2 What crosses the weight-sync channel
Decoupling the pools creates exactly one new piece of required machinery: a channel that gets the training pool's updated weights into the rollout pool's GPUs without stalling either side for longer than necessary. The survey documents a spread of implementations differing by more than two orders of magnitude in latency, from a naive collective broadcast to an engineering trick that avoids a synchronization point altogether.
| Strategy | Latency | Mechanism |
|---|---|---|
| Standard NCCL broadcast | 100-500 ms | Trainer broadcasts the full parameter set as one collective; rollout engine pauses until the broadcast completes. |
| Bucketed broadcast | ≈20 ms | Parameters are chunked into buckets and streamed, overlapping transfer with the rollout engine reloading earlier buckets. |
| Per-forward-pass swap (PipelineRL) | 1-10 ms | Weights swap between forward passes; an in-flight sequence is never interrupted2. |
The standard-broadcast number is the naive implementation: treat weight sync as one more NCCL collective, the same primitive used for gradient all-reduce during training, and pay whatever latency a full-parameter broadcast to every rollout replica costs. A broadcast's latency is dominated by the volume of parameters moving across whatever interconnect links the training and inference pools -- for a model in the tens of billions of parameters, that is enough data that even a well-implemented collective measures in the hundreds of milliseconds, and every rollout replica has to receive the full update before it is safe to resume decoding with the new weights. That cost is not free, but it is also not the dominant cost next to Figure 3's hours-long generation phase -- which is exactly why it is a defensible default and why several frameworks stop there. Bucketing shaves an order of magnitude by chunking the parameter set and streaming it, so a rollout replica can start reloading the first bucket while later buckets are still in flight and can keep serving in-progress sequences on old weights for the buckets not yet received, overlapping the transfer with continued (partial) use of the old weights rather than blocking the whole engine on one large collective. PipelineRL's outlier design removes the synchronization point altogether by treating a weight update as a per-sequence event rather than a pool-wide barrier: because a sequence's forward pass is already broken into discrete steps, the engine can swap in new weights in the gap between one step and the next, so "sequences are never interrupted"2 and there is no moment where the whole rollout pool sits idle waiting for a broadcast to finish. Module 38 examines PipelineRL and the other fifteen surveyed frameworks' concrete implementations of this channel in depth; the point to take from Table 1 here is that weight-sync latency is a real, measured design axis, not an implementation detail below the level this module's architecture operates at, and it interacts directly with the staleness question Section 4 covers -- a channel that takes 500 ms to complete lets more generation happen on stale weights than one that takes 5 ms, for the same update frequency.
3.3 The formal version: HybridFlow's dataflow graph
HybridFlow (the paper behind the open-source verl framework, examined concretely in Module 38) gives the two-pool picture a more precise formal treatment. It models the entire post-training computation as a directed acyclic graph in which "each node represents computation of a neural network... and each edge denotes data dependencies between the NNs"6 -- the actor (policy), critic, reference, and reward models are nodes; the weight-sync channel and every other cross-model data transfer are edges. Two things about that formalism matter for the architecture question this section is answering, independent of verl's own implementation choices, which Module 38 covers.
First, an edge in this graph is frequently "a many-to-many multicast"6, not a simple point-to-point copy, because the actor's training-time parallelism layout (say, tensor-parallel across 8 GPUs with a particular sharding) is rarely the same layout the rollout engine wants for generation (which may prefer a different tensor-parallel degree, or none at all, to maximize decode throughput). A weight-sync channel is not just moving bytes from one place to another; it is resharding a parameter tensor from one parallelism layout to a different one, which is why HybridFlow's own contribution -- the 3D-HybridEngine -- is specifically an actor-resharding mechanism built "to enable efficient execution of training and generation of the actor model, introducing zero memory redundancy and significantly reduced communication overhead during model parameter resharding between the training and generation stages"6. Table 1's latency numbers are, underneath, measuring how well a given implementation solves this resharding problem for its own weight layout choices.
Second, HybridFlow's paper frames the architectural choice underneath its own design as a spectrum rather than a binary, and its own position on that spectrum is a deliberate hybrid: "a single-controller paradigm is inefficient due to the substantial overhead of dispatching operators to distributed accelerators... [while] without central control, [a multi-controller paradigm] is inflexible to implement various RLHF dataflow[s]"6, so HybridFlow uses "the single-controller paradigm on the inter-node level" -- one coordinator deciding how data reshards between the actor, critic, reference, and reward nodes -- while "integrating the multi-controller paradigm within intra-node computation" for the actual GPU-local kernels6. Measured against prior-generation baselines, this combination yields a "1.53×~20.57× throughput improvement... running various RLHF algorithms"6. This module does not re-derive verl's specific implementation of that hybrid coordination, its 3D-HybridEngine's resharding algorithm in detail, or how its numbers compare against the other five frameworks Module 38 surveys -- the point here is narrower: the dominant architecture's weight-sync channel is doing real, non-trivial distributed-systems work (resharding across parallelism layouts, not just copying bytes), and how tightly that resharding couples the two pools in time is precisely what separates the synchronous and asynchronous positions in this module's disagreement section.
4. Staleness management: three families
Decoupling the two pools (Section 3) buys back the idle time Figure 3 makes visible, but it creates the problem every asynchronous producer/consumer system creates: the consumer (the trainer) can now fall behind the producer (the rollout pool), or run ahead of it, and the two are no longer guaranteed to be looking at the same policy version at the same moment. Concretely: a rollout generated with policy version \(v\) may not reach the trainer until the trainer has already advanced to version \(v+2\) or \(v+3\), because the rollout took 3.7 hours to generate (Figure 3) while training steps complete in a fraction of that time once fed. A policy-gradient algorithm's math assumes the actions in its training batch were sampled from the current policy; a batch sampled from a stale policy is, strictly, off-policy data being fed to an on-policy algorithm, and how much of that mismatch a given framework tolerates -- and what it does about the mismatch it does tolerate -- is the single largest design fork in the RL-infrastructure survey. The survey's own synthesis groups every framework's answer into exactly three, largely orthogonal, categories2.
4.1 Hard version rejection
The simplest policy: attach a version number to every rollout at generation time, and when the trainer is ready to consume a batch, drop any sample "whose version falls behind the current policy by more than a threshold"2. This is the staleness-management equivalent of Module 24's quota-checked-at-admission pattern -- reject at the boundary rather than accept and deal with the consequences later -- and it has the same virtue: the trainer never sees data it was not designed to handle, so the underlying RL algorithm's on-policy assumptions stay closer to true. The cost is wasted work: every rejected rollout consumed real generation time (Figure 3's 3.7-hour bar, at whatever fraction of a batch gets discarded) for nothing, and the threshold itself is a tuned hyperparameter with no universal correct value -- too tight, and a slow-but-otherwise-fine rollout gets discarded along with a genuinely-too-stale one; too loose, and the mechanism stops doing anything.
4.2 Bounded-depth queue
Instead of a hard age cutoff, this strategy bounds the quantity of staleness in flight: "the queue or buffer between generation and training has a bounded capacity"2, so at most a fixed number of not-yet-consumed rollouts can accumulate before the oldest is evicted to make room for a new one. This is a backpressure mechanism in the sense Module 10 already covers -- a bounded queue with an eviction policy, rather than an unbounded buffer that would let staleness grow without limit if the trainer ever fell behind for an extended period. The advantage over hard rejection is that it self-regulates: if generation and training happen to be running at similar rates, the queue rarely fills and almost nothing is discarded; if generation pulls far ahead, the bound caps how bad the worst-case staleness can get without a human-tuned age threshold deciding sample-by-sample. The disadvantage is the same one every bounded-queue design has: what gets evicted is whatever is oldest at the moment the queue is full, which is a capacity-driven decision, not a quality- or staleness-severity-driven one -- a queue at exactly its capacity limit will evict a borderline-stale sample just as readily as a very stale one.
The capacity bound converts directly into a staleness bound through the same relationship Little's Law gives any steady-state queue: for a queue holding at most \(N\) rollouts, drained by the training pool at a steady-state rate of \(r\) rollouts per second, the time any rollout can wait before being consumed (or evicted) is bounded by \(N / r\) seconds. A shallow queue (small \(N\)) caps staleness tightly but has almost no slack to absorb a temporary slowdown in the training pool before it starts evicting rollouts that were still perfectly usable; a deep queue tolerates that slowdown gracefully but, by the same formula, raises the ceiling on how many policy versions behind a surviving sample is allowed to be. Queue depth is therefore not a free knob -- it is a direct trade between how strictly staleness is bounded and how much transient rate mismatch between the two pools the system can absorb without discarding otherwise-good rollouts, which is exactly the same shallow-buffer-versus-deep-buffer tension Module 8 covers for a message broker's retention window, applied here to policy-version age instead of wall-clock time.
4.3 Importance-sampling correction
The third family accepts stale samples rather than discarding or bounding them, and instead corrects for the staleness inside the loss: "stale samples that reach the trainer are reweighted by the importance sampling ratio, typically clipped (Truncated IS)"2. The importance-sampling ratio compares the probability the current policy would have assigned to the sampled action against the probability the (older) policy that actually generated it assigned, and scales the sample's contribution to the gradient by that ratio -- a sample the current policy finds much less likely than the old policy did gets down-weighted; one the current policy finds more likely gets up-weighted. Clipping the ratio (the same idea PPO already applies to its own policy-ratio term, one algorithm layer up from this infrastructure-layer correction) prevents a single, very-off-policy sample from producing an outsized gradient contribution when the ratio itself becomes extreme. The advantage is that no generation work is ever discarded -- every rollout, however stale, contributes something to the gradient. The disadvantage is statistical rather than architectural: importance weights have variance that grows with how stale the sample is, so a framework leaning on this correction alone, with no depth bound at all, is trading discarded compute for a noisier, potentially higher-variance gradient estimate as staleness grows unchecked.
Formally, for an action \(a\) sampled from behavior policy \(\pi_{\theta_{\text{old}}}\) at generation time, the correction reweights that sample's contribution to the current policy \(\pi_\theta\)'s gradient by the ratio
$$ \rho(a \mid s) = \frac{\pi_\theta(a \mid s)}{\pi_{\theta_{\text{old}}}(a \mid s)}, \qquad \hat{\rho} = \min\!\big(\rho(a \mid s),\, c\big) $$where \(c\) is the clip constant. Two things about this identity explain both why the correction works and why it stops working as staleness grows. First, in expectation over actions actually drawn from \(\pi_{\theta_{\text{old}}}\), weighting each sample's contribution by \(\rho\) recovers an unbiased estimate of the gradient under \(\pi_\theta\) -- the whole point of the correction, and the reason it is a statistically principled fix rather than a heuristic. Second, unbiasedness says nothing about variance: if the trainer has advanced several versions past the policy that generated a sample, \(\pi_{\theta_{\text{old}}}\) and \(\pi_\theta\) can disagree sharply on the same action's probability, driving \(\rho\) toward zero (the action is now judged near-impossible) or toward very large values (judged now much more likely), and clipping to \(c\) trims only the second failure mode -- a stale sample the current policy judges near-impossible still contributes almost nothing useful to the gradient no matter how it is clipped, which is precisely the case a depth bound or a hard age cutoff would have removed before it ever reached this formula. That is the mechanical reason the three categories compose rather than compete: importance-sampling correction fixes the bias a stale sample introduces, but only a bound on how stale a sample is allowed to get (categories 1 and 2) keeps that correction's variance from growing without limit.
These three categories are not mutually exclusive engineering choices so much as three independent knobs a given framework can set anywhere from off to maximal, and production systems frequently combine them -- a depth-bounded queue that also applies importance-sampling correction to whatever survives inside the bound is a coherent design, not a contradiction. AReaL's own staleness control illustrates the point: a single parameter (the survey and AReaL's own documentation call staleness tolerance a first-class, named setting rather than an emergent property of queue depth) governs how far behind the current policy a rollout is allowed to be before it is treated as too stale to use, and setting that tolerance to its strictest value collapses the framework back to a synchronous algorithm7 -- one parameter smoothly interpolating between category 1's hard cutoff and full asynchrony, rather than three categories bolted together. Module 38 covers AReaL's specific parameter and five other frameworks' concrete instantiations of Figure 5's three categories in depth; this module's claim is only that the three categories are the complete, orthogonal design space the field has converged on describing, whatever combination a given framework picks within it.
| Strategy | What is bounded | What it costs | Instance |
|---|---|---|---|
| Hard version rejection | Sample age (policy-version delta) | Discarded generation compute for every rejected rollout | Age-threshold check at batch admission |
| Bounded-depth queue | Queue capacity between the two pools | Eviction is capacity-driven, not severity-driven | Fixed-size buffer with oldest-evicted-first |
| Importance-sampling correction | Nothing discarded; gradient contribution reweighted | Estimator variance grows with staleness | AReaL's tunable staleness-tolerance parameter7, collapsing to synchronous at its strictest setting |
Where experts disagree
Section 3 established that nearly every surveyed framework agrees on the top-level shape -- an inference pool and a training pool, connected by a weight-sync channel2. What the field has not converged on is how tightly the two sides of that channel should be coupled in time: whether the trainer and rollout pool should run as genuinely independent, fully asynchronous services with staleness bounded after the fact, or whether they should stay coordinated through an explicit resharding step that keeps both sides working from a dataflow graph a single controller can reason about.
Fully asynchronous, microservice-decoupled coupling wins on throughput
AReaL's own reported results7; PipelineRL's per-forward-pass weight swap, per the 16-library survey2
AReaL 2.0's refactor into "independent training, inference, agent, and weight-update services"7 treats generation and training as separate long-running processes with no shared execution barrier at all -- staleness is bounded by a single tunable parameter after the fact, not enforced by keeping the two sides in lockstep. AReaL reports a "2.77× speedup... while delivering comparable or superior training performance" against synchronous systems7, and PipelineRL's design goes further, eliminating the weight-sync barrier itself by swapping weights between forward passes rather than pausing the whole pool2 -- the strongest asynchronous position on the spectrum, with staleness near zero without ever blocking generation. The argument is that Figure 3's multi-hour generation phase is too expensive to spend any of it waiting on a barrier, and that bounded, corrected staleness (Section 4) is a solved-enough problem that the throughput this design buys back is worth it.
- Removes idle GPU-time on both pools, not just the imbalance Figure 3 shows -- neither side ever blocks on the other's schedule.
- A single staleness-tolerance knob is simple to reason about and to tune against a specific algorithm's off-policy sensitivity.
- Measured, reported speedups (AReaL's 2.77×) are large enough to matter at frontier training-run cost scales.
- Two independently-scheduled services (inference, training) are two independent failure domains and two logs to correlate when a run misbehaves, instead of one dataflow graph.
- Staleness tolerance is itself a hyperparameter with no universal correct value, and getting it wrong trades the throughput win for a training instability that is harder to diagnose than a slow but correct synchronous run.
Explicit resharding through a single coordinator wins on correctness and debuggability
HybridFlow/verl's hybrid single-/multi-controller design6; verl's own repository8; NeMo-RL's Ray-based resource management9
HybridFlow's own framing of the alternative it rejected is instructive here: a pure multi-controller design (each node deciding its own dataflow independently, the shape a fully decoupled microservice architecture tends toward) is "inflexible to implement various RLHF dataflow[s]" because "changing one node's dataflow requires changing all dependent nodes"6. verl's answer keeps a single controller coordinating how data reshards between the actor, critic, reference, and reward computations6, which means the entire training/generation transition for one iteration is one function call the coordinator can trace, retry, and reason about as a unit -- and it reports substantial measured throughput gains of its own, "1.53×~20.57×" over prior baselines6, without giving up the single-coordinator structure. NeMo-RL takes the same coordinated-resource-management position on a different substrate, using Ray for "scalable and flexible deployment across different hardware configurations"9 rather than a hand-rolled set of independent microservices.
- One coordinator, one dataflow graph -- a failure or a debugging session has a single place to look, not two async services whose relative timing has to be reconstructed after the fact.
- Explicit resharding (the 3D-HybridEngine) can target zero memory redundancy6 for a specific known transition, which a general-purpose async weight-sync channel does not get for free.
- Reported throughput multiples (1.53×-20.57×) show tight coupling is not automatically slower than decoupling -- the gain depends on how well the resharding step itself is engineered.
- A resharding step, however efficient, is still a synchronization point the survey's own dominant-architecture description names as characteristic of the tighter-coupling side2 -- some amount of Figure 3's idle-time cost is not eliminated, only reduced.
- Coordinating logic that spans every node's dataflow is more code to get right than two independently-deployable services, even if each individual piece is simpler.
Both positions are live in production, not a settled question -- the survey's own conflict note is explicit that the field shows "three genuinely different staleness-tolerance philosophies... rather than one converged answer"2, and the same split shows up one layer up, in how tightly the two pools are coupled in the first place. The deciding variable is not throughput in isolation -- both sides report large, real speedups against their own prior baselines -- it is how expensive a wrong or hard-to-diagnose training run is for a given team's iteration budget. A frontier lab running one enormous training job for weeks, where every hour of idle GPU time is a large absolute cost and the team has the engineering depth to operate two independent failure domains, should weight AReaL's and PipelineRL's fully asynchronous position more heavily: the 2.77× speedup compounds across a long run in a way that dwarfs the added operational complexity. A smaller team iterating quickly on algorithm or reward-function changes, where a training run that silently degrades because staleness tolerance was mistuned costs more in debugging time than the throughput loss from a coordinated resharding step, should weight HybridFlow/verl's position more heavily -- a single dataflow graph is a much easier thing to add a breakpoint to than two services exchanging messages on independent clocks. What would settle it is not available yet: none of the sources here report a controlled comparison of the same model, same algorithm, and same hardware budget across a fully-async and a tightly-coupled implementation -- every reported speedup is against that framework's own prior generation or a different baseline system, not a head-to-head with the opposing philosophy.
What is changing
Module 36 already flags that the generation share of RLHF wall-clock time is not fixed at the roughly 80/20 split Section 2.3 uses for illustration -- OpenRLHF's own later paper revises the estimate upward to "over 90%" of total RLHF (or RLVR) runtime specifically for long-chain-of-thought training10, where a policy generates thousands of reasoning tokens before an environment can score the result. Every argument in Section 3 for decoupling the two pools, and every cost Section 4's staleness taxonomy imposes for keeping them loosely synchronized, gets stronger as that ratio moves further from parity: a rollout pool that dominates wall-clock time even more heavily has even less tolerance for a training-side barrier stalling it, and a verifier fleet (Section 1.1) scored against ever-longer chains of reasoning has correspondingly less slack to be the slow half of Section 2.4's round-trip.
The weight-sync channel is also still diversifying rather than settling on one implementation. Table 1's three-tier latency spread -- a 25-50× range from a naive NCCL broadcast down to bucketing, and a further order of magnitude down to PipelineRL's per-forward-pass swap2 -- spans techniques published within roughly the same period, which suggests the channel itself, not just the staleness policy layered on top of it, is an active engineering target rather than a solved primitive. The 16-library survey's own orchestration-substrate finding -- eight of sixteen frameworks built on Ray, four on native Python async/threading, and outliers on HTTP microservices and an actor-model runtime2 -- points the same direction: Module 38 covers that fork directly, but its existence at all, this late into the field's development, is itself evidence that the dominant architecture's implementation details are still being worked out even where its top-level shape (Section 3) is not in dispute.
References
- blogPrime Intellect. Environments Hub: A Community Hub To Scale RL To Open AGI. 2026. https://www.primeintellect.ai/blog/environments. Accessed 2026-09-17. Used for: the environment interface as state/action/reward loop, task-specific-vs-shared-infrastructure framing, sandboxes plugging into Verifier Environments, the 30-contributor adoption figure.
- blogHugging Face. Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries. 2026. https://huggingface.co/blog/async-rl-training-landscape. Accessed 2026-09-17. Used for: the dominant inference-pool/training-pool architecture description, the three-category staleness taxonomy, weight-sync latency figures (Table 1), the 512-sequence/32K-token/3.7-hour generation arithmetic, the 16-framework list and orchestration-substrate split.
- paperYu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., Chun, B.-G. Orca: A Distributed Serving System for Transformer-Based Generative Models. 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022. https://www.usenix.org/conference/osdi22/presentation/yu. Accessed 2026-09-18. Used for: iteration-level scheduling and selective batching as the origin of continuous batching, the 36.9x throughput figure over FasterTransformer.
- paperKwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023, arXiv:2309.06180. https://arxiv.org/abs/2309.06180. Accessed 2026-09-18. Used for: paged KV-cache allocation, near-zero memory waste and 2-4x throughput figures, cross-request KV-cache page sharing.
- repoOpenRLHF contributors. OpenRLHF README. GitHub, 2026. https://github.com/OpenRLHF/OpenRLHF. Accessed 2026-09-17. Used for: the 80%-of-wall-clock-time-on-generation figure underlying Figure 3's training-step bar.
- paperSheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., Wu, C. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256, EuroSys '25, 2024. https://arxiv.org/pdf/2409.19256. Accessed 2026-09-17. Used for: the RLHF dataflow-as-DAG formalism, many-to-many resharding edges, the single-controller/multi-controller hybrid design, the 3D-HybridEngine's zero-memory-redundancy resharding, the 1.53x-20.57x throughput figure.
- repoinclusionAI (AReaL contributors). AReaL README. GitHub, main branch AReaL 2.0, 2026. https://github.com/inclusionAI/AReaL. Accessed 2026-09-17. Used for: the microservice-decoupled training/inference/agent/weight-update architecture, the tunable staleness-tolerance parameter collapsing to a synchronous algorithm at its strictest setting, the 2.77x speedup figure.
- repoverl contributors (ByteDance Seed / HKU). volcengine/verl README. GitHub, main branch, 2026. https://github.com/volcengine/verl. Accessed 2026-09-17. Used for: the 3D-HybridEngine's resharding description as verl's own framing, in the disagreement section's coordinated-resharding position.
- repoNVIDIA NeMo team. NVIDIA-NeMo/RL README. GitHub, main branch, 2026. https://github.com/NVIDIA-NeMo/RL. Accessed 2026-09-17. Used for: Ray-based resource management as a second instance of the coordinated-coupling position in the disagreement section.
- paperHu, J., Wu, X., Shen, W., Liu, J. K., Zhu, Z., Wang, W., Jiang, S., Wang, H., Chen, H., Chen, B., Fang, W., Xianyu, Cao, Y., Xu, H., Liu, Y. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv:2405.11143v6, 2025. https://arxiv.org/abs/2405.11143. Accessed 2026-09-18. Used for: the >90%-of-runtime generation-time figure for long-chain-of-thought RLVR training.