System Design · Module 38
Comparing the Open-Source RL Frameworks: verl, OpenRLHF, SkyRL, AReaL, slime, and NeMo-RL
Six frameworks answer the same trainer/rollout-decoupling question differently, from verl's hybrid single-/multi-controller resharding to AReaL's fully asynchronous microservices, read against a 16-framework survey that found three orthogonal staleness philosophies and no consensus on orchestration substrate. After this module you can place any of these six (or a seventh you encounter later) on the synchronous-to-asynchronous spectrum, name which of Module 37's staleness categories it implements, and know which of its own numbers to check before trusting a reported speedup.
1. Six frameworks, one open question
Module 37 established the shape of the problem: RL post-training runs an inference workload (rollout generation) and a training workload (optimizer steps) that want opposite things from the same GPUs, and every framework that decouples them has to decide how stale a rollout is allowed to be before the sample it produced can no longer be trusted for a gradient update. Module 37 named three answers — hard version-rejection, a bounded-depth queue between generation and training, and importance-sampling correction applied to whatever staleness gets through — as the taxonomy the field actually uses. This module applies that taxonomy to six frameworks a practitioner is likely to pick up in 2026: HybridFlow/verl, OpenRLHF, SkyRL, AReaL, slime, and NeMo-RL.
These six are not arbitrary. A Hugging Face survey published in 2026 catalogued sixteen open-source RL libraries actively maintained at the time — AReaL, ART, Atropos, MILES, NeMo-RL, OAT, open-instruct, PipelineRL, PRIME-RL, ROLL, SkyRL, SLIME, TorchForge, Tunix, verl, and verifiers-rl8 — and found no convergence on either staleness philosophy or orchestration substrate. The six named in this module's scope cover four of the survey's sixteen directly (verl, SkyRL, AReaL, SLIME), add NeMo-RL as a fifth entry from the same survey, and add OpenRLHF as a sixth framework the survey does not include at all — OpenRLHF predates most of the async-RL tooling discussion and is included here because it is still the most widely cited Ray-plus-vLLM RLHF framework in practitioner use, not because the survey vouches for it.
The pattern across all six: five build on Ray for cluster orchestration, and the sixth (AReaL) deliberately does not. All six ship at least a synchronous mode; four of the six ship a documented, named staleness-control mechanism you can point to a specific parameter for. Table 1 is the orientation; the rest of this module is the mechanism behind each cell.
| Framework | Orchestration | Default coupling | Staleness category | Reported speedup |
|---|---|---|---|---|
| verl / HybridFlow | Ray (single-controller inter-node + multi-controller intra-node) | Colocated, synchronous (hybrid_engine) | 2 (optional: staleness_threshold, 0 = sync) | 1.53x-20.57x vs. prior baselines1; +23-40% one-step-off9 |
| OpenRLHF | Ray | Colocated, synchronous (Hybrid Engine Scheduling) | 1 (none documented) | not a standardized throughput number3 |
| SkyRL | Ray (RayPPOTrainer) | Configurable: sync, one-step-off, or fully async | 2 (queue depth 1, or in-flight updates) | task-accuracy result only, no systems multiplier4 |
| AReaL | Native async (asyncio/threading), microservices | Fully decoupled by design | 2 + 3 (max_head_offpolicyness + staleness-enhanced PPO) | 2.77x vs. synchronous systems5 |
| slime | Ray | Colocated by default, sync weight-push per step | 2 (fully-async example, no published bound) | not published; production framework, not a benchmark suite6 |
| NeMo-RL | Ray | Configurable: sync GRPO, or async GRPO | 2 + 3 (max_trajectory_age_steps + mandatory IS correction) | not a single published multiplier710 |
Two things stand out immediately. First, only AReaL reports a headline, apples-to-apples "x times faster than synchronous" number with a fixed configuration behind it (2.77x, the boba2 configuration)5; the others either report a range that depends heavily on model size and hardware (verl's 1.53x-20.57x spans multiple RLHF algorithms and baselines1) or report no systems-level multiplier at all, only a downstream task result. Do not treat these numbers as comparable across rows — they were measured on different models, different hardware, and different baselines by each project's own team. Second, "staleness category" is not a fixed property of a framework the way orchestration substrate is: verl and NeMo-RL both ship a synchronous default and an opt-in asynchronous recipe with its own staleness knob, so the category column names what is available, not what every user actually runs.
What all six are actually solving underneath the staleness question is the same resharding problem HybridFlow's paper names directly: the RLHF dataflow is a DAG "where each node represents computation of a neural network... and each edge denotes data dependencies between the NNs," and "the edge represents data resharding, which is often a many-to-many multicast" between models laid out differently across GPUs1 — the same DAG Module 36 introduced for the SFT/reward-modeling/DPO pipeline, now with a training-time actor and a generation-time actor as two different physical layouts of the same weights. The six frameworks differ mainly in where they choose to pay that resharding cost. verl pays it in place, inside a single process boundary, via the 3D-HybridEngine's zero-redundancy reshard (Section 3). OpenRLHF, SkyRL, slime, and NeMo-RL pay it by moving data through Ray's distributed object store between separate actors, relying on Ray's reference-passing to avoid an unnecessary copy but still crossing a process (and often a network) boundary11. AReaL pays it explicitly over its own RPC layer, since it has no Ray object store to lean on — the update_weights request Section 6 describes is that payment made visible as a named message type rather than an implicit object-store transfer. None of the six frameworks makes the resharding cost disappear; each one is a different answer to where in the stack that cost is easiest to control.
2. Orchestration substrate: Ray, native-async, and the rest
Before comparing how each framework manages staleness, it is worth separating that question from a different one: what actually schedules the processes, holds the distributed object references, and restarts a worker that dies. The Hugging Face survey's clearest finding is that the RL-framework ecosystem has not converged on an answer. Of the sixteen frameworks it catalogued, eight run on Ray (verl, SkyRL, NeMo-RL, SLIME, MILES, ROLL, OAT, open-instruct), four use native Python asyncio/threading with no cluster-scheduling framework underneath (verifiers-rl, PipelineRL, ART, AReaL), one uses HTTP microservices (Atropos), and one uses the Monarch actor model (TorchForge)8. The remaining two — PRIME-RL and Tunix — are not assigned to any of those four categories in the survey.
2.1 What Ray gives an RL framework for free
The appeal of building on Ray rather than raw multiprocessing is threefold, per Ray's own documentation: tasks and actors as a small set of primitives for distributed Python, actors as stateful workers that execute their method calls serially in the order received (preserving per-actor consistency without an explicit lock), and a distributed object store that avoids unnecessary copies by passing references between components11. For an RLHF pipeline, this maps directly onto the DAG Module 36 described: an Actor model, a Reward model, a Reference model, and a Critic model are each naturally a Ray actor holding its own GPU-resident weights, and the object store is what lets a rollout batch move from the generation actor to the training actor without an extra serialization round trip. OpenRLHF's own framing of its design is explicit about this: "OpenRLHF leverages Ray for efficient distributed scheduling. It separates the Actor, Reward, Reference, and Critic models across different GPUs, enabling scalable training for models up to 70B+ parameters"3. verl, SkyRL, slime, and NeMo-RL all make the same substrate choice for the same reason — none of them re-implements cluster membership, failure detection, or object placement, all of which Module 6 and Module 34's primitives already solve at the Ray layer.
The serial-execution guarantee matters specifically for the weight-versioning problem Sections 6 and 8 return to. An actor "executes method calls serially in the order it receives them, preserving consistency"11, which means a rollout actor cannot receive an update_weights call and a generate call out of order and apply them in the wrong sequence — Ray's scheduler, not the RL framework's own code, is what guarantees a weight update issued before a generation request is fully applied before that generation request executes. AReaL's native-async design, covered next, has to build this same ordering guarantee itself rather than getting it from an underlying actor framework, which is part of the implementation cost its own README concedes when describing AReaL-lite as a smaller, 80%-less-code alternative5.
2.2 The outlier: AReaL's native-async design
AReaL is the one framework among the six that does not build on Ray. Its README describes AReaL 2.0 as refactored "into a microservice architecture with independent training, inference, agent, and weight-update services"5 — independent OS processes coordinating over their own RPC boundaries rather than Ray actors sharing an object store. Section 6 covers why: a fully asynchronous design where a rollout worker's in-flight generation must be interruptible mid-sequence (to load new weights) benefits from precise control over process lifecycle and networking that a general-purpose actor framework does not have to optimize for by default. The trade-off is that AReaL forgoes Ray's free cluster-membership and object-store machinery and has to build its own equivalent — a cost the Hugging Face survey's four-way split suggests at least four of sixteen teams judged worth paying8.
3. verl: single-controller/multi-controller hybrid and the 3D-HybridEngine
verl is the open-source implementation of HybridFlow, a EuroSys '25 paper from ByteDance and the University of Hong Kong1. The paper's starting observation is that existing RLHF frameworks forced a choice between two bad options: a single-controller paradigm, where one process drives every distributed operation and pays the dispatch overhead of pushing operators out to billion-parameter distributed models, or a multi-controller paradigm, where each node runs its own copy of the driver logic and is efficient but inflexible — changing one node's dataflow means changing every dependent node's code by hand1. HybridFlow's answer is to use both, at different granularities: "utilizing the single-controller paradigm on the inter-node level enables flexible expression of various data dependencies and easy coordination of inter-node data resharding with minimal overhead, while integrating the multi-controller paradigm within intra-node computation enhances computation efficiency substantially"1. Concretely, one Python-level controller expresses the RLHF DAG (actor generates, reward scores, critic estimates, actor updates) as ordinary function calls across node boundaries, while inside each node the actual tensor-parallel or pipeline-parallel execution runs with the multi-controller pattern any distributed training framework already uses.
The mechanism that makes this efficient rather than merely flexible is the 3D-HybridEngine. The actor model needs two very different physical layouts: a training layout (say, FSDP- or Megatron-sharded, optimized for backward passes and gradient synchronization) and a generation layout (optimized for the inference engine's tensor-parallel or expert-parallel serving shape, matching what Aman's inference-serving background already covers). Naively, switching between them means writing the training-layout weights to disk or a staging buffer, then re-loading them into a second, separately-allocated copy of the model in the generation layout — doubling peak memory for the duration of the switch. The 3D-HybridEngine instead reshards the actor's parameters in place: HybridFlow's own description is that it enables "efficient execution of training and generation of the actor model, introducing zero memory redundancy and significantly reduced communication overhead during model parameter resharding between the training and generation stages"1, and the verl repository restates the same property as a shipped feature: "Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases"2.
The "3D" in 3D-HybridEngine is not marketing shorthand; the paper defines it as three literal parallel dimensions the actor's training-time layout is constructed from: "pp·tt·dd denote 3D parallel groups constructed for actor training, corresponding to the set of GPUs to host pp pipeline stages, tt tensor shards, and dd model replicas"1. The resharding step Figure 3 draws as a single arrow is really a remapping across all three of those dimensions simultaneously — the training layout's pipeline-stage and tensor-shard boundaries do not need to coincide with the generation engine's tensor- or expert-parallel boundaries, and the 3D-HybridEngine's job is to compute the minimal data movement between the two arbitrary shardings rather than assuming they line up.
The measured payoff HybridFlow reports depends heavily on model size and baseline, and the paper is specific enough to show the shape of that dependence rather than only a headline range. All experiments ran on 16 machines of 8 NVIDIA A100-80GB GPUs each (128 GPUs total), with 600GB/s NVLink within a machine and 200Gbps between machines1 — a cluster description Aman can compare directly against the interconnect topologies Module 33 covers. Table 2 breaks the reported PPO speedup down by model size.
| Model size | Speedup range | Notable single comparison |
|---|---|---|
| 7B | 1.68x - 8.63x | vs. DeepSpeed-Chat: 3.67x average, up to 7.84x |
| 13B | 2.70x - 18.96x | vs. OpenRLHF: 3.25x average, up to 5.93x |
| 34B | 2.41x - 20.57x | vs. NeMo-Aligner: 12.52x average, up to 20.57x |
| 70B | 5.17x - 17.98x | largest model tested; smallest low-end speedup of the four |
The pattern worth noticing is that the widest speedups are not at the largest model size: the 34B row's 20.57x ceiling (against NeMo-Aligner specifically) is wider than the 70B row's. This is consistent with the mechanism rather than surprising — NeMo-Aligner's multi-controller design pays a fixed per-node coordination cost that a smaller model's proportionally larger wasted-idle-time fraction exposes more clearly, while at 70B every baseline is closer to its own hardware ceiling regardless of coordination overhead, so HybridFlow's relative advantage narrows even though its absolute throughput is highest at that size. This is also the concrete answer to why Table 1's single "1.53x-20.57x" range spans an order of magnitude: it is three different baselines crossed with four different model sizes, not four repeated measurements of the same comparison1. The repository's own changelog separately reports a ~1.4x speedup over verl's own prior release2, which is the more useful number if you are asking "should I upgrade" rather than "should I use verl at all," and verl's algorithm list has grown well past the paper's original PPO focus to include GRPO, DAPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME, and DrGPO2, all sharing the same 3D-HybridEngine resharding mechanism underneath.
verl's default execution mode is synchronous and colocated: the same GPUs run both the training and generation layouts of the actor, alternating via the hybrid engine's reshard, which by construction produces zero staleness — every rollout batch is generated under the exact parameters the immediately following optimizer step will update. verl also ships two opt-in asynchronous recipes that expose Module 37's bounded-depth-queue category directly as a configuration surface. The staleness_threshold parameter controls how far the rollout worker is allowed to run ahead of the trainer: at staleness_threshold=0, "Rollouter will generate a fixed number of samples between two parameter updates" — recovering the synchronous case exactly — while a positive value lets the rollout worker generate up to (1+staleness_threshold) times the per-sync sample count before the trainer must catch up, with the documentation recommending "it is recommended to set this value to less than 1" to keep the policy gap small. A simpler, fixed-depth-one variant is the "one-step off-policy" recipe, which deploys the trainer and rollout worker on disaggregated resources and always trains on samples generated exactly one step behind the current policy. Measured on FSDP2 it delivered roughly a 23% wall-clock improvement (19h18m to 15h34m over a fixed number of training steps) and on Megatron roughly 40% (18h21m to 13h06m) versus the colocated synchronous baseline on the same hardware allocation9. The fully-async recipe (unbounded staleness threshold, disaggregated resources) reports a 2.35x-2.67x speedup on a 128-GPU configuration, attributed specifically to overlapping rollout and training time rather than serializing them: "in the colocate case, using more resources for rollout cannot solve the idleness caused by long-tail samples... the overlap in their time consumption reduces the end-to-end time consumption"12.
The documentation's formula for the fully-async cap is explicit enough to plug numbers into: the rollout worker may generate at most \((1+\text{staleness\_threshold}) \times (\text{trigger\_parameter\_sync\_step} \times \text{require\_batches} \times \text{ppo\_mini\_batch\_size})\) samples between two parameter updates12. Take a configuration where the trainer synchronizes weights every trigger_parameter_sync_step=1 step, needs require_batches=4 such batches before running an optimizer update, and uses a ppo_mini_batch_size of 256:
At staleness_threshold=0 this 1024-sample figure is also the hard cap — the rollout worker cannot get ahead at all, which is exactly the synchronous case restated as a formula. At the documentation's own recommended ceiling of staleness_threshold=0.5, the cap becomes
— the rollout worker is permitted to run 512 samples, or half a sync's worth, ahead of the trainer before it must pause and wait for a weight update. That is the concrete, numeric meaning behind the documentation's qualitative advice to "set this value to less than 1": at staleness_threshold=1 the rollout worker could get a full extra sync-cycle ahead (2048 samples outstanding), which is the point at which the policy that generated the oldest outstanding sample and the policy currently training have diverged by an entire update, not half of one.
4. OpenRLHF: Ray-native separation and the 80% generation problem
OpenRLHF is the framework Module 36 already introduced as the concrete instance of the Actor/Reward/Reference/Critic DAG: "OpenRLHF leverages Ray for efficient distributed scheduling. It separates the Actor, Reward, Reference, and Critic models across different GPUs, enabling scalable training for models up to 70B+ parameters"3. Its generation step runs on vLLM with automatic tensor and pipeline parallelism, motivated by a number the project states directly: "RLHF training spends 80% of the time on sample generation. Powered by vLLM with Auto Tensor Parallelism (AutoTP) and Pipeline Parallelism (PP), OpenRLHF delivers high-throughput, memory-efficient generation"3 — the same 80% figure that motivates Module 37's entire framing of RL as an inference workload wearing a training hat. Training itself is built on DeepSpeed ZeRO-3 with deepcompile, AutoTP, and RingAttention, explicitly to avoid depending on a heavier training framework while still working directly against HuggingFace model checkpoints3. OpenRLHF supports PPO, REINFORCE++, REINFORCE++-baseline, GRPO, RLOO, Dr.GRPO, and FlashREINFORCE, and states that these algorithms are "decoupled from agent execution modes, meaning they work with both single-turn and multi-turn agent executors"3.
The coupling OpenRLHF ships by default is what it calls Hybrid Engine Scheduling: "All models and vLLM engines can share GPU resources — minimizing idle time and maximizing GPU utilization"3. This is a colocated, synchronous design — the same resource pool alternates between generation and training rather than running them concurrently on separate pools. Unlike verl, NeMo-RL, or SkyRL, OpenRLHF's own documentation does not describe a staleness parameter, a bounded queue, or an importance-sampling correction path for asynchronous rollouts; its own project is also absent from the sixteen frameworks the Hugging Face survey catalogued8. That absence is worth stating plainly rather than filling in: OpenRLHF sits at category 1 in Module 37's taxonomy — no staleness tolerated, by construction, because generation and training never run far enough apart to produce any — not because a design decision explicitly rejected the other categories. The multi-turn agent executor support is worth flagging for the same reason Module 37's sandbox-as-verifier framing (built on Module 27) matters here: a multi-turn rollout that calls a tool, waits on a sandboxed execution result, and resumes generation stretches the wall-clock time a single rollout occupies well past the single-forward-pass case, which pushes the 80%-generation-time figure higher still and makes OpenRLHF's synchronous, no-staleness-tolerance default a more expensive default to keep as agentic task length grows — the same pressure that pushes SkyRL, AReaL, and slime toward the async recipes Sections 5 through 7 describe.
5. SkyRL: modularity as the design axis, and three execution modes
SkyRL's own description of itself leads with modularity rather than a specific algorithmic or systems claim: "skyrl-train" is "a modular, performant training framework for RL," and the project frames this as the organizing principle across its whole stack rather than a single trained-model result4. The modularity claim extends past the trainer itself: the project's own framing is that "skyrl combines our previous work" into one repository, pairing skyrl-train with skyrl-tx, a separate inference backend component the project maintains alongside the trainer rather than depending exclusively on an external serving engine for the InferenceEngine role described below4. The one concrete outcome its README does report is SkyRL-SQL-7B, trained on 653 samples, reported to outperform both GPT-4o and o4-mini on its target text-to-SQL task4 — a task-accuracy result, not a systems throughput number, and not directly comparable to any other row in Table 1.
SkyRL's own documentation names five architectural components that map cleanly onto the RLHF DAG13: a Generator that "generates complete trajectories and computes their rewards," encompassing both inference and environment execution plus custom logic such as context management or tree search; a Trainer that performs the optimization step and updates model parameters from generated trajectories and rewards; an InferenceEngine that executes inference on the policy model, with the documentation noting that "multiple InferenceEngines are deployed to process prompts in parallel"; an Environment that presents a task and executes the policy's actions; and a Controller that manages "physical placement, initialization, and control flow" across the other four. The Trainer class SkyRL exposes, RayPPOTrainer, together with its PPORayActorGroup abstraction for a group of training workers implemented as Ray actors, confirms Ray as SkyRL's orchestration substrate13 — consistent with its place in the Hugging Face survey's Ray-based column8. SkyRL naming Environment as a first-class component in its own right, separate from both the Generator and the Trainer, is the same environments-as-services idea Module 37 introduces at the framework-agnostic level: the task-specific logic (what state the agent sees, what actions are legal, how a reward is computed) is meant to be swappable without touching the Generator, Trainer, or Controller code at all, so a new benchmark or a new tool-use task is, in principle, a new Environment implementation and nothing else.
SkyRL exposes staleness control as a choice of execution mode rather than a single tunable number. The default is synchronous. A one-step off-policy pipelining mode "separates the generation and training phases into two parallel coroutines" connected by an asyncio.Queue(maxsize=1) — a bounded-depth queue with a hard depth of exactly one, Module 37's category 2 at its simplest. This mode requires disaggregated resources: full colocation must be disabled (colocate_all=false) so generation and training run as genuinely separate worker pools rather than time-sharing the same GPUs, and it is supported with both the FSDP and Megatron training backends14. Beyond one-step-off, SkyRL's own README links to a dedicated write-up titled "Fully Async RL with In-Flight Weight Updates in SkyRL," naming a third mode that lets new parameters reach the inference engine mid-generation rather than only between batches — the same in-flight-update idea Section 6 covers for AReaL and Section 9 covers as a weight-sync-latency question in its own right4.
6. AReaL: fully asynchronous by design
AReaL is the framework in this module built around asynchrony from the ground up rather than as an opt-in recipe layered on a synchronous default. Its README states the design goal directly: "a fully asynchronous RL training paradigm... AReaL 2.0... refactors AReaL into a microservice architecture with independent training, inference, agent, and weight-update services"5. The academic paper behind the system (AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning, arXiv:2505.24298) names the finer-grained components underneath that framing: an Interruptible Rollout Worker that "handles two types of requests: (1) The generate request generates responses given prompts. (2) The update_weights request interrupts all ongoing generations and loads parameters"; a Reward Service that "evaluates the accuracy of the responses generated by the model"; Trainer Workers that "continuously sample from the replay buffer, accumulating data until reaching the configured training batch size" before performing PPO updates; and a Rollout Controller that "serves as a critical bridge between the rollout workers, reward service, and the model workers"15. The Reward Service is AReaL's concrete instance of the verifier role Module 37 introduces as a sandbox plugged into an environment: for a reasoning or code task, that service is precisely where a Module-27-style sandboxed execution happens to check whether a generated answer or program is actually correct, and its output is what the Rollout Controller forwards into the replay buffer alongside the trajectory itself. Running the verifier as its own independent service, rather than as a step inside the rollout worker, is also what lets AReaL 2.0's refactor name it separately in the four-service split its README describes5.
max_head_offpolicyness515.The staleness bound itself has a precise definition in the paper: writing \(N_r\) for the number of trajectories generated so far, \(B\) for the training batch size, and \(i\) for the trainer's current policy version, the system enforces
$$ \left\lfloor \frac{N_r - 1}{B} \right\rfloor \le i + \eta $$which says the rollout workers may run at most \(\eta\) full batches ahead of the trainer before generation must pause and wait15 — a bounded-depth queue in Module 37's sense, parameterized directly rather than left implicit. The paper's ablation varies \(\eta\) over 0, 1, 2, 4, 8, 16, and \(\infty\), and reports that moderate staleness preserves final task performance while improving throughput15. The README exposes the same idea under the name max_head_offpolicyness, and states plainly that "algorithms support asynchronous versions by setting max_head_offpolicyness=0" to recover a synchronous algorithm5 — the same zero-recovers-sync pattern verl's staleness_threshold and NeMo-RL's async GRPO configuration both share. AReaL layers a second mechanism on top of the bound: rather than treating every trajectory that passes the η check as equally trustworthy, it "adopts a staleness-enhanced PPO variant to better handle outdated training samples"15 — an importance-sampling-style correction, Module 37's category 3, applied to whatever the bounded queue lets through. AReaL is therefore the cleanest example among the six of a framework combining categories 2 and 3 rather than choosing one.
The reported payoff is a 2.77x training speedup compared to synchronous systems using the same number of GPUs, "while delivering comparable or superior training performance," in the project's boba2 configuration5 — consistent with the paper's separately reported "up to 2.57x" figure for an earlier configuration5, both numbers describing the same underlying decoupling rather than two different results. The project also ships AReaL-lite, a reduced implementation that "maintains 90% of AReaL's performance" while using "80% fewer lines of code"5 — a concession that the fully asynchronous microservice design carries real implementation cost, not only a throughput benefit.
Module 37's own worked arithmetic gives a way to see what a 2.77x speedup is actually worth in wall-clock terms. That module computes generation time for a 32B model producing 32K tokens per rollout across 512 sequences at roughly 1.2K tokens/second on a single H100: \(512 \times 32{,}000 / 1{,}200 \approx 13{,}653\) seconds, or about 3.7 hours, versus a training step that takes a much smaller fraction of that time8. In a fully synchronous design, one full iteration (generation, then training) costs approximately that 3.7 hours plus the training step; a framework achieving AReaL's reported 2.77x on the same workload does not shrink the generation time itself — a single H100 still produces tokens at the same 1.2K tokens/second — it instead overlaps that 3.7-hour generation window with the previous iteration's training step, so the marginal wall-clock cost per iteration after the pipeline fills approaches \(3.7 / 2.77 \approx 1.34\) hours rather than the full 3.7. The 2.77x is therefore a statement about pipeline overlap recovering otherwise-idle GPU-hours, not about generation itself becoming faster — the same distinction Section 3's derivation makes for verl's fully-async recipe.
7. slime: SGLang-native and the framework actually in production
slime is the framework in this set for which "in production" is not a marketing claim but a documented fact: its README states it is "the RL framework behind GLM-5.3, GLM-5.2, GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, and GLM-4.5"6 — six successive releases of Zhipu AI's flagship model family, plus documented support for Qwen, DeepSeek-V3, and Llama 36. That production pedigree, rather than a novel systems mechanism, is slime's main claim on a practitioner's attention: it is evidence a Ray-orchestrated, SGLang-native RL stack survives the transition from a research repository to a frontier-lab release pipeline repeatedly.
Architecturally, slime splits into exactly three components, mirroring the training/rollout decoupling this module has seen five times already: a training module that "reads data from the Data Buffer, and synchronizes parameters to the rollout module after training," implemented on top of Megatron; a rollout module that "generates new data (including rewards/verifier outputs) and stores it in the Data Buffer," implemented on top of SGLang plus a router; and the Data Buffer itself, "a bridge module that manages prompt initialization, custom data, and rollout generation methods"6. The detail that distinguishes slime from the other five frameworks is how tightly it couples to SGLang specifically: rather than wrapping SGLang behind a generic inference-engine interface the way verl or OpenRLHF treat vLLM as one of several interchangeable backends, slime "optimizes deeply for SGLang so RL workloads can use SGLang-specific serving, routing, caching, disaggregation, and weight-sync behavior directly," and exposes SGLang's own command-line arguments through the framework with a --sglang- prefix rather than re-implementing or hiding them6. For a reader already fluent in SGLang's serving internals, this is the framework where that fluency transfers directly rather than through an abstraction layer: "caching" here is SGLang's own radix-attention prefix cache, and "disaggregation" is SGLang's own prefill/decode-disaggregated serving mode, not a rollout-specific reimplementation of either idea6. A framework that wraps an inference engine behind a generic interface has to either re-expose every such feature one flag at a time or leave it unreachable; slime's --sglang- passthrough means a serving-side optimization that lands in SGLang upstream is available to slime's rollout module the same release cycle, with no separate integration work on slime's side.
slime's default coupling is synchronous and colocated, with the training module pushing updated weights to the rollout module once per training step6. Like verl, it also ships a documented fully-async rollout path — an examples/fully_async configuration the repository describes as "useful for long-tail agentic generation where some samples take much longer than others"6, which is Module 37's category 2 in its least formalized shape: the mechanism exists and is shipped, but slime's own documentation does not publish a single staleness-bound parameter or a measured speedup number the way verl's staleness_threshold or AReaL's η do. That gap is worth stating rather than papering over: slime is the clearest illustration in this module that "ships an async mode" and "publishes a staleness-management policy with numbers behind it" are two different claims, and only some frameworks make both.
8. NeMo-RL: two training backends, one staleness knob
NeMo-RL is NVIDIA's post-training library, and its most consequential design decision has nothing to do with staleness at all: it ships two interchangeable training backends rather than committing to one. DTensor is "PyTorch's next-generation distributed training with improved memory efficiency (PyTorch-native TP, SP, PP, CP, and FSDP2)," positioned for "hackable research prototypes," while Megatron is "NVIDIA's high-performance training framework for scaling to large models with 6D parallelisms," positioned for maximum scale7. Resource management across both runs through Ray, "enabling scalable and flexible deployment across different hardware configurations... both small-scale experiments and massive multi-GPU, multi-node deployments"7. Supported algorithms include GRPO, GSPO, and DAPO, plus SFT (with LoRA), DPO, and on-policy distillation, with the framework additionally emphasizing "process isolation between RL Actors" as a reliability property rather than only a performance one7. The dual-backend decision is a direct concession that "hackable" and "maximum scale" are different, sometimes conflicting requirements at different points in a project's life: DTensor's PyTorch-native TP/SP/PP/CP/FSDP2 stack lets a researcher iterate on an algorithm change without touching a Megatron-specific code path, while a team that has already committed a model to Megatron-style 6D parallelism for pre-training (the same parallelism dimensions Aman's distributed-training background and Module 33's cluster-topology treatment both assume) can keep that parallelism strategy through the RL stage rather than re-sharding into a different framework's convention purely to run post-training.
NeMo-RL's staleness mechanism sits inside its asynchronous GRPO path. The design decouples generation from optimization using an Async Trajectory Collector — a background Ray actor that "continuously generates trajectories using current policy weights" — running concurrently with the main training loop, which samples trajectories from a ReplayBuffer and runs policy training steps in parallel with that background generation10. The bound on how old a trajectory may be before it is discarded is a single named parameter, max_trajectory_age_steps, and its semantics are stated precisely: "trajectory generated with weights v10 can be used for training steps v10 or v11. At training step v12, trajectories from v10 are too old and discarded"10 — a bounded-depth queue, Module 37's category 2, expressed directly in units of training steps rather than a fraction or a ratio. What makes NeMo-RL the cleanest textbook illustration of categories 2 and 3 working together is that the documentation treats the importance-sampling correction as mandatory rather than optional once any staleness is tolerated at all: "importance sampling correction... ensures that we have unbiased gradient estimates and stable convergence" for exactly the trajectories the age-based filter allows through10. Where AReaL exposes its correction as "a staleness-enhanced PPO variant" bundled with the system, NeMo-RL states the causal link explicitly: tolerate staleness at all (category 2), and category 3 is not an enhancement but a requirement for the resulting gradient estimate to remain unbiased.
9. Weight synchronization: the other place staleness gets decided
Every bounded-depth queue and every importance-sampling correction in this module assumes something about how long it takes for a fresh set of parameters to actually reach the rollout workers, once the trainer has produced them. That transfer — the weight-synchronization step — is itself a latency number, and it varies by roughly two orders of magnitude across the ecosystem the Hugging Face survey measured: a standard NCCL broadcast of the full parameter set takes on the order of 100-500ms; libraries that implement bucketing strategies (splitting the broadcast into pipelined chunks rather than one blocking call) bring this down to roughly 20ms; and one framework, PipelineRL, is described as categorically different rather than merely faster, because "sequences are never interrupted. Weights swap between forward passes (~1-10ms gap)"8 — the update happens inside the natural boundary between one token-generation forward pass and the next, rather than requiring generation to pause and resume around a broadcast.
Placing the six named frameworks on this range: verl's default hybrid-engine path resolves weight sync as an in-place reshard rather than a network broadcast at all, so the comparison does not directly apply to its colocated mode; once verl moves to a disaggregated recipe (one-step-off or fully async), it falls back to an explicit parameter-transfer step between the training and rollout resource pools, in the same 100ms-to-low-tens-of-ms band the survey describes for bucketed transfers9. AReaL's controller-triggered update_weights request explicitly interrupts all ongoing generations before loading new parameters15 — a design that accepts a pause-the-world cost at each sync in exchange for a bound the controller can enforce precisely via η, rather than pursuing PipelineRL's zero-pause approach. slime's tight coupling to SGLang lets it use "SGLang-specific... weight-sync behavior directly"6 rather than a generic broadcast primitive, though its README does not publish a specific latency number the way the survey does for the ecosystem overall. The general lesson is that a framework's headline staleness-bound parameter (η, staleness_threshold, max_trajectory_age_steps) is only half the story — the other half is how expensive one unit of that bound is to pay in wall-clock time, and that number depends on the weight-sync mechanism, not the staleness policy.
It is also worth remembering that a weight-sync broadcast does not travel over a dedicated, uncontended link. Module 33 describes the three-layer Clos network a large training cluster actually runs on, with a 1:7 oversubscription ratio at the pod-to-cluster aggregation layer specifically because cross-pod bandwidth is the scarcer resource. A weight-sync broadcast issued from a training pool to a rollout pool that happens to sit in a different pod competes for exactly that oversubscribed link, alongside checkpoint traffic and any other cross-pod communication the cluster is running at the same moment — which is one reason the survey's reported ranges are ranges rather than single numbers, and one reason frameworks that colocate training and rollout on the same rack (avoiding the cross-pod hop entirely) can report tighter, more predictable sync latency than the ecosystem-wide spread in Figure 5 suggests.
Where experts disagree
Module 37 raised synchronous-versus-asynchronous trainer/rollout coupling as an open question in the abstract; the six frameworks in this module turn it into a question with concrete numbers on both sides, and the numbers do not agree on an answer. A second, structurally unrelated disagreement resurfaces here from Module 34: five of these six frameworks depend on Ray for orchestration, and Ray itself has to be deployed on top of either an HPC batch scheduler or a cloud-native orchestrator, which reopens the question Module 34 left open rather than closing it.
Synchronous, colocated coupling is the right default
verl's and OpenRLHF's own default configurations13; implicitly, any team that has not adopted an async recipe
The 3D-HybridEngine's zero-memory-redundancy resharding1 and OpenRLHF's Hybrid Engine Scheduling3 both make the case that colocated synchronous execution is not merely simpler but can be made nearly free: if switching between training and generation layouts costs no extra memory and only "significantly reduced" (not eliminated, but small) communication overhead, then the complexity of a bounded queue, a staleness parameter, and an importance-sampling correction buys throughput the colocated design has already mostly captured through engineering. Every rollout is generated under the exact policy that will be updated from it, so there is no bias-correction machinery to build, test, or get wrong.
- No importance-sampling correction needed; the gradient estimate is unbiased by construction.
- Simpler operational model: one resource pool, one failure domain, no separate staleness-tuning problem.
- verl's own repository reports the resharding overhead is small enough that the synchronous default remains the recommended starting point.
- Cannot overlap the generation and training phases in time, so GPUs assigned to the idle half of the alternation sit unused — the exact inefficiency Module 37's 3.7-hour generation-versus-training arithmetic quantifies.
- Long-tail rollouts (a handful of sequences that generate far longer than the median) stall the entire batch under a synchronous design, since training cannot start until every sequence in the batch finishes.
Fully asynchronous decoupling is necessary at scale
AReaL's design and measured results515; verl's and SkyRL's own opt-in async recipes91213
AReaL's 2.77x speedup over synchronous systems at comparable or better final task performance5 is the strongest single data point in this module for the opposite view: decoupling generation and training into separate resource pools, and tolerating a bounded, corrected amount of staleness, recovers GPU-time that a synchronous design structurally cannot use. verl's own fully-async recipe reporting 2.35x-2.67x on a 128-GPU configuration12 — measured by the same project that ships the synchronous default — is notable precisely because it is not a competitor's number; the team that built the zero-memory-redundancy synchronous path also built and measured a faster asynchronous one on top of it.
- Overlapping rollout generation and training time directly addresses the long-tail-stall failure mode a synchronous design cannot avoid.
- Every framework that reports a controlled speedup number for its own async mode (AReaL 2.77x5, verl one-step-off +23-40%9, verl fully-async 2.35x-2.67x12) reports it as a genuine improvement, not a wash.
- Requires an importance-sampling correction (or an equivalent staleness-enhanced algorithm variant) to keep the gradient estimate unbiased — extra machinery that has to be tuned and validated, not merely enabled.
- Disaggregated resource pools mean GPUs dedicated to generation cannot be reclaimed for training during a training-bound phase, trading one form of idleness for another under an unfavorable generation/training ratio.
- AReaL-lite's own framing — 90% of AReaL's performance at 80% fewer lines of code5 — is itself an admission that the full microservice design's implementation cost is real.
The two positions are not actually symmetric once model size and rollout-length variance are held fixed. For small-to-mid models with short, low-variance rollouts (the common case for many SFT-adjacent RLHF runs), the synchronous colocated path's simplicity dominates: the generation/training time ratio is close enough to 1 that overlap buys little, and the 3D-HybridEngine's near-zero resharding cost means the "waste" a synchronous design incurs is genuinely small. As rollout length grows and its variance grows with it — the long-reasoning, multi-turn agentic regime AReaL, slime's fully-async example, and SkyRL's in-flight-update mode all target — the long-tail-stall problem becomes the dominant cost, and every measured async number in this module (AReaL 2.77x, verl 2.35x-2.67x) reflects exactly that regime. The evidence that would settle this for a specific workload is not a general benchmark; it is the ratio of generation time to training time the workload itself produces, which is precisely the number Module 37's arithmetic teaches you to compute before choosing an architecture, not after.
An HPC batch scheduler underneath Ray is the pragmatic choice
SchedMD's own architectural framing of Slurm16; the training clusters Module 34 described as Slurm-first
Slurm's centralized slurmctld/slurmd heartbeat model, its pending-work queue for contention arbitration, and its long-standing role as the default scheduler on GPU-dense HPC clusters16 mean that a research or training team already running Slurm does not need to introduce a second, unfamiliar scheduling layer just to run a Ray-orchestrated RL framework — Ray's own documentation provides an explicit deployment path for exactly this: starting a Ray cluster inside an sbatch job using multiple srun commands to launch the head node and worker nodes17. verl's own multi-node documentation confirms this is not merely theoretical: the project has "verified the GSM8K example on a Slurm cluster under a multi-node setting"9.
- No second cluster manager to operate, secure, or reconcile with existing HPC allocation and accounting policy.
- Slurm's admission-control-by-quota and job-queue model (Module 34) already solve fair-share scheduling across users, a problem Ray does not natively address.
- Slurm's fundamental assumption — one job submitted per node — is a poor fit for Ray's head-worker architecture, which needs one coordinating process that all other nodes connect to17; Ray's own documentation calls this out as a direct design conflict requiring careful
--tasks-per-nodeallocation to work around. - Multi-tenant port allocation (node manager, object manager, and worker ports) must be manually managed to avoid collisions between simultaneous Ray-on-Slurm jobs17 — an operational cost Kubernetes' per-pod network namespace avoids by construction.
A cloud-native orchestrator (Kubernetes/KubeRay) is the better long-term substrate
The declarative-reconciliation model Module 34 described as Borg's and Kubernetes' lineage; Ray's own ray symmetric-run feature as an admission the head-worker model needs first-class multi-node tooling17
Ray's introduction of ray symmetric-run in version 2.49 — a command that "will start a Ray cluster on all nodes with given CPU and GPU resources and run your entrypoint script only on the head node"17 — exists specifically because the Slurm-plus-srun deployment path is awkward enough that Ray's own maintainers built tooling to smooth over it. A cloud-native orchestrator with native per-pod networking, a declarative reconciliation loop, and (via KubeRay) first-class support for the Ray head-worker topology avoids the port-collision and one-job-per-node friction points Slurm's design was never built to anticipate, at the cost of needing a Kubernetes-fluent operations team.
- Per-pod network isolation removes the manual port-range management Ray-on-Slurm requires17.
- Declarative reconciliation (Module 34) matches Ray's own dynamic actor/task scheduling model more naturally than a static Slurm allocation does.
- Most existing large GPU training clusters, including the ones Module 34 and Module 33 describe, were provisioned Slurm-first; migrating the scheduler layer under a production training pipeline is a nontrivial project in its own right.
- Kubernetes' GPU-topology and RDMA-networking support for large training jobs has historically lagged Slurm's, a gap Module 34 already documents as unresolved.
This is the same unresolved fork Module 34 described, and nothing in this module's six frameworks resolves it — if anything, the fact that Ray itself now ships a Slurm-specific compatibility command (ray symmetric-run) rather than requiring users to migrate off Slurm is evidence the field expects both substrates to coexist rather than one replacing the other. The practical decision variable is not which scheduler is "better" in the abstract but which one a team's training cluster was provisioned under before the RL framework decision was made: retrofitting Ray onto an existing Slurm allocation is a one-time integration cost documented well enough (by both Ray and verl) to be low-risk, while migrating an established Slurm-based training pipeline to Kubernetes purely to simplify Ray deployment is very unlikely to be justified by that reason alone.
What is changing
The number sixteen in this module's survey is already a lower bound on churn, not a stable census. The same source that catalogued sixteen actively maintained RL libraries in 20268 was itself compiled because the field had outgrown any single team's ability to track it by memory — a year earlier, a comparable list would have been shorter, and a year from now it will likely be longer or restructured around a different set of leaders. Two of this module's six (AReaL and slime) reached their current asynchronous or SGLang-native designs through public major-version refactors (AReaL 2.0's microservice rewrite5; slime's SGLang-argument passthrough) rather than incremental releases, which suggests the architectural question is still being actively re-litigated inside individual projects, not only across them. The most likely near-term convergence is not on a single winning architecture but on vocabulary: five of the six frameworks in this module now expose a named staleness-bound parameter a user can set to zero to recover synchronous training, which was not true of the RLHF tooling landscape even two years earlier. Whether that convergence extends to orchestration substrate — Ray versus native async versus the Monarch actor model Figure 2 shows TorchForge betting on — is the question Module 39 returns to without expecting a different answer than the one this module found.
References
- paperSheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., Wu, C. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256 / EuroSys '25, 2024. https://arxiv.org/pdf/2409.19256. Accessed 2026-09-17. Used for: single-controller/multi-controller hybrid design (§1), 3D-HybridEngine resharding, 1.53x-20.57x throughput result, RLHF-as-DAG framing (§2.1).
- repoverl contributors (ByteDance Seed / HKU). volcengine/verl README. GitHub, main branch, 2026. https://github.com/volcengine/verl. Accessed 2026-09-17. Used for: 3D-HybridEngine feature description, supported algorithm list, vLLM/SGLang/FSDP/Megatron integration, ~1.4x version-over-version speedup.
- repoOpenRLHF contributors. OpenRLHF README. GitHub, main branch, 2026. https://github.com/OpenRLHF/OpenRLHF. Accessed 2026-09-17. Used for: Ray-based Actor/Reward/Reference/Critic separation, 80% generation-time figure, Hybrid Engine Scheduling, supported algorithms.
- repoNovaSky-AI. SkyRL README. GitHub, main branch, 2026. https://github.com/NovaSky-AI/SkyRL. Accessed 2026-09-17. Used for: modularity framing, SkyRL-SQL-7B result, link to the in-flight-weight-update write-up.
- repoinclusionAI. AReaL README. GitHub, main branch (AReaL 2.0), 2026. https://github.com/inclusionAI/AReaL. Accessed 2026-09-17. Used for: microservice refactor description, max_head_offpolicyness parameter, 2.77x/2.57x speedup figures, AReaL-lite 90%/80% figures, supported algorithm list.
- repoTHUDM. slime README. GitHub, main branch, 2026. https://github.com/THUDM/slime. Accessed 2026-09-17 (architecture and production-usage facts); re-accessed 2026-09-18 (fully_async example, SGLang argument passthrough). Used for: training/rollout/data-buffer split, --sglang- argument passthrough, GLM model family production usage, fully-async rollout example.
- repoNVIDIA NeMo team. NVIDIA-NeMo/RL README. GitHub, main branch, 2026. https://github.com/NVIDIA-NeMo/RL. Accessed 2026-09-17. Used for: DTensor vs. Megatron backend framing, Ray-based resource management, supported algorithms, process isolation between RL Actors.
- blogHugging Face. Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries. 2026. https://huggingface.co/blog/async-rl-training-landscape. Accessed 2026-09-17. Used for: the 16-framework list, the three-category staleness taxonomy applied at ecosystem scale, the 8/4/1/1/2 orchestration-substrate split (Fig. 2), weight-sync latency ranges (Fig. 5), the OpenRLHF-not-included observation.
- officialverl documentation. Recipe: One Step Off Policy Async Trainer. verl.readthedocs.io, 2026. https://verl.readthedocs.io/en/latest/advance/one_step_off.html. Accessed 2026-09-18. Used for: one-step-off staleness=1 design, disaggregated-resource requirement, +23%/+40% measured speedups, Slurm multi-node verification note.
- officialNVIDIA. Async GRPO. NeMo-RL documentation, docs.nvidia.com, 2026. https://docs.nvidia.com/nemo/rl/latest/guides/async-grpo.html. Accessed 2026-09-18. Used for: Async Trajectory Collector / ReplayBuffer design, max_trajectory_age_steps semantics, mandatory importance-sampling correction.
- officialAnyscale (Ray project). Ray Core Walkthrough. docs.ray.io, 2026. https://docs.ray.io/en/latest/ray-core/walkthrough.html. Accessed 2026-09-17. Used for: tasks/actors/object-store primitives, per-actor serial execution guarantee.
- officialverl documentation. Recipe: Fully Async Policy Trainer. verl.readthedocs.io, 2026. https://verl.readthedocs.io/en/latest/advance/fully_async.html. Accessed 2026-09-18. Used for: staleness_threshold semantics (0 = sync), 2.35x-2.67x fully-async speedup on 128 GPUs, colocation-vs-isolation trade-off rationale.
- officialSkyRL documentation. System Overview. docs.skyrl.ai, 2026. https://docs.skyrl.ai/docs/getting-started/overview. Accessed 2026-09-18. Used for: Generator/Trainer/InferenceEngine/Environment/Controller components, RayPPOTrainer/PPORayActorGroup.
- officialSkyRL documentation. Async Training (one-step-off). docs.skyrl.ai, 2026. https://docs.skyrl.ai/docs/tutorials/one_step_off_async. Accessed 2026-09-18. Used for: asyncio.Queue(maxsize=1) one-step-off pipelining, colocate_all=false requirement, FSDP/Megatron backend support.
- paperFu, W., Gao, J., Shen, X., Zhu, C., Mei, Z., He, C., Xu, S., Wei, G., Mei, J., Wang, J., Yang, T., Yuan, B., Wu, Y. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. arXiv:2505.24298, 2025 (v2, 2026). https://arxiv.org/abs/2505.24298. Accessed 2026-09-18. Used for: Interruptible Rollout Worker / Reward Service / Trainer Workers / Rollout Controller component breakdown (§4.1), the staleness-bound inequality with η (Eq. 3), the η-ablation (Table 2/Fig. 5), staleness-enhanced PPO variant.
- officialSchedMD. Slurm Workload Manager Overview. slurm.schedmd.com, 2026. https://slurm.schedmd.com/overview.html. Accessed 2026-09-17. Used for: slurmctld/slurmd centralized heartbeat architecture, pending-work queue, optional accounting/REST daemons.
- officialAnyscale (Ray project). Deploying on Slurm. docs.ray.io, 2026. https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html. Accessed 2026-09-18. Used for: head-worker vs. one-job-per-node design conflict, srun/sbatch deployment pattern, ray symmetric-run (v2.49+), port-allocation caveat.