Multi-Node LLM Inference: From One Node to N · Module 12
SGLang: Router, PD Disaggregation, and Wide-EP Deployment Patterns
SGLang scales across nodes with three pieces that plug into the generic patterns from earlier modules: a standalone Rust router (sglang-router, now rebuilt as the SGLang Model Gateway) with a cache-aware load-balancing algorithm, two selectable transfer backends behind the bootstrap-handshake PD mechanism, and a small set of CLI flags that turn into a concrete DeepSeek-class multi-node deployment. After this module you can read sglang-router's actual routing algorithm rather than its feature list, choose between its two PD transfer backends, and reproduce the exact flags behind SGLang's own 96-H100 wide-EP deployment.
1. SGLang's disaggregation motivation, and why it differs from the interference argument
SGLang's own documentation frames prefill/decode disaggregation as solving two distinct problems on a unified engine, not one: Prefill Interruption, where incoming prefill batches delay token generation for requests already decoding, and DP Attention Imbalance, where data-parallel workers end up processing mismatched mixes of prefill and decode work1. Module 4 introduced this framing alongside the bootstrap handshake mechanism it motivates and noted, in passing, that it is a different diagnosis from DistServe's and Mooncake's interference argument (Module 3) even though both conclude that separate pools help. This module stays on the SGLang side of that split and goes one level further into why the two framings diverge, because the difference matters for how you read SGLang's own deployment guidance.
DistServe's and Mooncake's case for disaggregation is a queueing argument: a compute-bound prefill job occupies the GPU long enough to stall the memory-bound decode steps of requests already in flight, so tail latency (TBT/ITL) degrades in proportion to how much prefill work is mixed into a batch (Module 3). SGLang's "Prefill Interruption" is the same underlying symptom, described the same way. "DP Attention Imbalance" is a second, separate failure mode that only exists because of how SGLang implements data-parallel attention across replicas: when --enable-dp-attention is active, each DP rank runs attention against its own private request slice and KV cache while MoE layers dispatch across the wider expert-parallel group (Module 5, §5.3). On a colocated engine, nothing guarantees that the prefill and decode requests land evenly across DP ranks — one rank can be doing three prefills while its neighbor idles on a single decode step, and because attention-DP ranks do not share KV cache or request queues, that imbalance cannot self-correct the way a single shared-queue engine's load naturally would. Disaggregating prefill and decode into separate pools removes DP Attention Imbalance by construction: each pool's DP ranks only ever see one kind of work, so the imbalance has nowhere to come from.
The practical upshot is that SGLang's own case for disaggregation is partly attention-DP-specific — a consequence of the parallelism layout introduced for wide-EP scaling (Module 5) — rather than a general claim that holds for every tensor-parallel-only deployment. A TP-only SGLang deployment without --enable-dp-attention still suffers Prefill Interruption exactly as DistServe describes it, but DP Attention Imbalance specifically requires the DP-attention layout to exist in the first place. This is worth keeping straight when reading SGLang's disaggregation documentation against Module 3's more general goodput argument: SGLang is not disagreeing with DistServe, it is adding a second, layout-specific reason on top.
The rest of this module covers what sits underneath that motivation on the SGLang side: the router that decides which prefill and which decode worker a request goes to, the two transfer backends that move the KV cache between them, the exact flags that turn TP16/DP8 attention-DP into a running multi-node deployment, and which parts of this stack SGLang's own issue tracker describes as settled versus still being hardened.
2. sglang-router / SGLang Model Gateway: architecture
Module 2's router-comparison table could only describe sglang-router at the level its PyPI package page documents: "a high-performance Rust-based load balancer for SGLang with multiple routing algorithms and prefill-decode disaggregation support," currently version 0.3.2, released 2026-01-152, with the caveat that the package page does not itemize the internal scoring formula. The project's own repository documentation does itemize it, under a name change: the same Rust binary and Python launcher (python3 -m sglang_router.launch_router) are now documented as the SGLang Model Gateway, describing itself as "high-performance model routing control and data plane for large-scale LLM deployments"3. The CLI flags and Python module name are unchanged from the sglang-router package Module 2 cites — this is the same project with its documentation and internal architecture rewritten, not a replacement.
The architecture splits into a control plane and a data plane3:
- Control plane. A
WorkerManagervalidates workers, discovers their capabilities via/server_infoand/get_model_info, and keeps a shared worker registry in sync. AJobQueueserializes background add/remove operations so registration never blocks a client request. A background health checker and load monitor keep per-worker circuit breakers and the routing policies informed of current load. Optional Kubernetes service discovery reconciles the registry against pod state directly, with separate selectors for prefill and decode pods in PD mode. - Data plane. HTTP routers for regular and PD traffic implement
/generateand the OpenAI-compatible endpoints (/v1/chat/completions,/v1/completions,/v1/embeddings,/v1/responses,/v1/rerank,/v1/classify); a separate gRPC router streams tokenized requests directly to SGLang's gRPC workers with tokenizer, reasoning-parser, and tool-call-parser logic implemented in Rust rather than shelling out to Python. A resilience layer — token-bucket rate limiting, request queuing, a retry executor, and per-worker circuit breakers — sits in front of both.
Two details matter for anyone deploying this behind an SGLang fleet rather than reading about it. First, the gateway also has an OpenAI-router mode that proxies requests to remote OpenAI-compatible vendors (OpenAI, xAI, Gemini) and can front a multi-model fleet under one process via --enable-igw (inference-gateway mode), applying a distinct load-balancing policy per model3 — this is out of scope for a single-model multi-node SGLang deployment but explains why the project renamed itself from a "router" to a "gateway." Second, the observability surface is large: over 40 Prometheus metrics grouped by layer (smg_http_*, smg_router_*, smg_worker_*, smg_worker_cb_* for circuit breakers, smg_discovery_*), plus OpenTelemetry tracing with W3C trace-context propagation into upstream worker requests3. None of this is itemized on the PyPI page Module 2 could cite in detail; it lives in the project's own README.
The resilience layer's own defaults are concrete enough to reason about a fleet-wide failure budget from: retries default to a maximum of 5 attempts with exponential backoff, triggered on HTTP 408, 429, 500, 502, 503, and 504 responses from a worker; per-worker circuit breakers open after a configurable failure threshold and half-open after a configurable timeout window; and the upstream HTTP client pools connections with a 50-second idle timeout, a 10-second connect timeout, and up to 500 idle connections per worker host3. None of these numbers were configurable at all in the router design the architecture-improvement proposal in §8 was written against — they are part of what "resilience layer" means for the rewritten gateway specifically, not a carryover from the earlier router.
3. Five load-balancing policies and the cache-aware algorithm
The router ships five selectable policies, set with --policy (or --prefill-policy/--decode-policy independently in PD mode, §4)3:
| Policy | Mechanism | Config knobs | Where it appears in this module's examples |
|---|---|---|---|
random | Uniform random worker selection | none | debug / MiniLB only |
round_robin | Sequential rotation via atomic counter | none | — |
power_of_two | Samples two random workers, routes to the one the Load Monitor reports as lighter | integrates with background load monitor | decode-side policy in the project's own PD quick-start (§4) |
cache_aware | Maintains an approximate prefix tree of routed prompts to predict per-worker cache-hit ratio, rebalancing on load thresholds | --cache-threshold (0.3), --balance-abs-threshold (64), --balance-rel-threshold (1.5), --eviction-interval (120s), --max-tree-size (67,108,864 nodes) | used in every quick-start example in the project's own README, including the default single-stage and PD examples |
bucket | Divides workers into load buckets with dynamic boundaries | not itemized beyond "dynamic boundaries" in the current README | — |
cache_aware is the only policy of the five that reasons about KV-cache locality rather than raw load, and it is the one every quick-start command in the project's documentation reaches for first3. Its mechanism was introduced in SGLang v0.4 (December 4, 2024) as a "cache-aware load balancer": the router maintains an approximate radix tree that mirrors the real radix tree each SGLang worker keeps for its own prefix cache, and predicts which worker holds the longest matching prefix for an incoming request without querying the workers directly4. The design goal stated in that announcement is explicitly communication-free: "the tree is lazily updated with almost no overhead," and no worker-to-router synchronization of cache state is required — the router infers the shadow tree's state purely from the requests it has already routed and their observed responses4. This is the same family of idea as Module 2's approximate block-hashing scheme (llm-d's prefix scorer), but at the granularity of a full radix tree over raw prompt text rather than fixed-size token blocks, and built into the router process itself rather than layered on top via a separate event stream.
The v0.4 announcement also published the first measured numbers for this policy against a round-robin baseline: prefill throughput rose from 82,665 to 158,596 tokens/second (a 1.9× increase), and the cache hit rate rose from 20% to 75% (a 3.8× increase), with the gain reported as increasing with worker count4. Those are two-year-old numbers relative to this module's as-of date and measured on whatever hardware and workload the v0.4 benchmark used — treat them as evidence the mechanism works, not as a number that transfers to an arbitrary deployment.
Why the policy needs two separate imbalance thresholds, not one
The cache-aware policy's rebalancing behavior is governed by two published default constants: --balance-abs-threshold = 64 and --balance-rel-threshold = 1.53. The documentation does not publish the exact boolean combination of the two, but the reason a single threshold cannot do this job on its own is straightforward arithmetic. Consider two workers with loads \(L_{min}\) and \(L_{max}\) in requests currently in flight:
- At low absolute load (\(L_{min}=10\), \(L_{max}=20\)): the relative ratio \(L_{max}/L_{min} = 2.0\) already exceeds 1.5, even though the absolute gap of 10 requests is nowhere near 64. A relative-only rule would trigger rebalancing here.
- At high absolute load (\(L_{min}=200\), \(L_{max}=260\)): the relative ratio is only \(260/200 = 1.3\), under the 1.5 threshold, even though the absolute gap of 60 requests is close to the 64-request cutoff and represents a much larger real queueing difference. A relative-only rule would not trigger here, and an absolute-only rule set to catch this case would have already over-triggered on much smaller fleets.
An absolute threshold alone over-triggers at small fleet sizes (a 5-request gap on 2 lightly loaded workers looks huge in relative terms but is operationally meaningless); a relative threshold alone under-triggers at high load (a 30% gap on heavily loaded workers is a large absolute number of misrouted requests). Publishing both and presumably requiring some combination of the two is what lets the same default configuration behave sensibly whether the fleet is idling at single-digit concurrency or saturated at hundreds of requests per worker — the same problem Module 2's llm-d load scorer solves with its own load-aware scoring layered on top of prefix-cache scoring, just with two explicit numeric knobs instead of a weighted-scorer combination.
4. PD-aware routing: per-role policies and worker registration
In PD mode the router accepts separate --prefill and --decode worker lists (a prefill entry can carry an optional bootstrap port, e.g. --prefill http://prefill1:30001 9001), and can apply an independent policy to each side via --prefill-policy and --decode-policy3. The project's own quick-start for PD mode pairs cache_aware on the prefill side with power_of_two on the decode side3:
./target/release/sgl-model-gateway \
--pd-disaggregation \
--prefill http://prefill1:30001 9001 \
--prefill http://prefill2:30002 \
--decode http://decode1:30011 \
--decode http://decode2:30012 \
--policy cache_aware \
--prefill-policy cache_aware \
--decode-policy power_of_two
That pairing is not arbitrary. Prefix locality is a prefill-side concern — which worker already holds a matching prefix in its KV cache determines how much of the prompt that worker can skip recomputing, which is exactly what cache_aware predicts. By the time a request reaches the decode side, the bootstrap handshake (Module 4, §3) has already fixed which decode worker receives the transferred KV cache; there is no longer a prefix-match decision to make, only a load-balancing one, which is what power_of_two is for. The router "merges prefill metadata with decode outputs and streams results back to the client," and for gRPC workers a single router process automatically selects the PD pipeline once the worker registry contains prefill and decode workers rather than requiring a separate deployment mode3.
Workers, including PD-role workers, can also be registered and removed at runtime rather than only via CLI flags at launch, through a small control-plane API3:
| Method | Path | What it does |
|---|---|---|
| POST | /workers | Queue worker registration (prefill/decode/regular); returns 202 Accepted while the job queue processes it |
| GET | /workers | List workers with health, load, policy metadata, queued-job status |
| PUT | /workers/{worker_id} | Queue a worker update by UUID |
| DELETE | /workers/{worker_id} | Queue worker removal by UUID |
| POST | /flush_cache | Trigger a cache flush across HTTP workers — used before redeploying PD workers |
Dynamic worker registration is a real capability change worth flagging against the project's own history. A router architecture-improvement proposal filed against the pre-rewrite design (June 25, 2025) listed "no dynamic worker management in PD mode" — requiring a full restart to add or remove PD workers — as one of seven concrete gaps, alongside workers being represented internally as bare strings with no health or load tracking5. The POST /workers / DELETE /workers/{worker_id} API documented for the current SGLang Model Gateway directly answers that gap. Section 8 returns to what in that 2025 gap list is and is not yet fully closed.
5. Two transfer backends: Mooncake vs. NIXL
The router picks which worker pair talks to which; the actual KV cache bytes move over one of two selectable transfer backends, configured on the SGLang engine processes themselves rather than in the router1. Both backends implement the same bootstrap-handshake contract described in Module 4 — they differ only in the transport underneath the handshake.
Mooncake is installed via uv pip install mooncake-transfer-engine and selected as the default backend. Its distinguishing configuration surface is --disaggregation-ib-device, which accepts three forms: a single shared device list (mlx5_0 or mlx5_0,mlx5_1), a per-GPU JSON mapping ({"0": "mlx5_0,mlx5_1", "1": "mlx5_2,mlx5_3"}), or a path to a JSON file containing that same per-GPU mapping1. The per-GPU form matters on any node where InfiniBand HCAs are not uniformly reachable from every GPU — common on 8-GPU nodes where NIC placement follows PCIe topology rather than a flat any-GPU-to-any-NIC design — because it lets each GPU's transfer traffic go out its topologically nearest HCA instead of contending on a shared default device. Mooncake itself, its Conductor scheduler, and its own transfer-engine bandwidth numbers are Module 7's subject; here it is one of two backend choices a flag selects.
NIXL is installed via pip install nixl and selected with --disaggregation-transfer-backend nixl. Its own backend choice is a second-level selection controlled by the environment variable SGLANG_DISAGGREGATION_NIXL_BACKEND, defaulting to UCX, with LIBFABRIC or any other installed NIXL plugin available as alternatives1. NIXL's own architecture — Memory Sections, Transfer Backend Interface, Metadata Handler — and its role as the shared substrate under Dynamo's and TensorRT-LLM's default KV transfer paths are Module 8's subject; SGLang treats it as an alternate backend behind the same bootstrap protocol, not its default.
| Property | Mooncake | NIXL |
|---|---|---|
| Install | uv pip install mooncake-transfer-engine | pip install nixl |
| Select | default backend | --disaggregation-transfer-backend nixl |
| Device mapping | per-GPU IB device JSON via --disaggregation-ib-device | backend-internal (UCX/LIBFABRIC plugin selection) |
| Sub-backend selector | — | SGLANG_DISAGGREGATION_NIXL_BACKEND (UCX default, LIBFABRIC, or other plugin) |
| Full project treatment | Module 7 | Module 8 |
Neither backend is documented as the categorically faster choice — the selection is about what is already installed and topologically available rather than a raw-throughput comparison. Mooncake's per-GPU IB device mapping is the more direct control surface when the deployment target's InfiniBand HCA-to-GPU topology is already known and asymmetric; NIXL's environment-variable backend selection is the path already shared with Dynamo (Module 9) and TensorRT-LLM (Module 13), so a fleet that runs more than one of these engines side by side has one fewer transport library to operate if it standardizes on NIXL everywhere it is available.
6. Multi-node DeepSeek-class deployment recipe
SGLang's documentation gives a concrete multi-node command set for a DeepSeek-class model, splitting prefill and decode onto separate master-node groups1. The core flags on the prefill side:
python3 -m sglang.launch_server \
--disaggregation-mode prefill \
--host ${local_ip} --port 30000 \
--dist-init-addr ${prefill_master_ip}:5000 \
--nnodes 2 --node-rank 0 \
--tp-size 16 --dp-size 8 --enable-dp-attention \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8
and on the decode side, the same topology shape with the disaggregation mode and a request-admission cap flipped:
python3 -m sglang.launch_server \
--disaggregation-mode decode \
--host ${local_ip} --port 30001 \
--dist-init-addr ${decode_master_ip}:5000 \
--nnodes 2 --node-rank 0 \
--tp-size 16 --dp-size 8 --enable-dp-attention \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8 --max-running-requests 128
Each pool repeats the command with --node-rank 1 (and each additional node it needs) pointed at the same --dist-init-addr, so the master IP is the one piece of address information every rank in a given pool must agree on1. Nothing in these two commands is new mechanism: --tp-size/--dp-size/--enable-dp-attention/--moe-a2a-backend deepep are exactly the attention-DP-plus-expert-parallel flags Module 5 covers mechanically (§5.3), and --disaggregation-mode plus the master-address/rank flags are the multi-node bootstrap plumbing every SGLang worker uses whether or not PD is involved. What this recipe adds is only the fact that prefill and decode are two independently addressed, independently sized pools — a prefill pool can run on different node count, node rank, and even different --mem-fraction-static than a decode pool, because they are separate process groups that only communicate through the transfer backend (§5) and the bootstrap handshake (Module 4), never through a shared distributed-init group.
--dist-init-addr groups with independent node counts and ranks; the flags shown are exactly SGLang's own documented multi-node DeepSeek recipe1.This is the class of recipe SGLang's own team used to reproduce a DeepSeek-scale deployment independently: 12 nodes of 8 H100 GPUs each (96 GPUs total) on Atlas Cloud, with prefill tested at EP32 across 4 nodes and decode tested at EP72 across 9 nodes6 — the wide-EP mechanics behind those EP degrees are Module 5's subject and the full reproduction, including its per-node throughput and cost numbers against DeepSeek's own published H800 deployment, is Module 6's subject. What is worth restating here, because it is the direct payoff of this section's flags, is that the reproduction's simulated-optimal-load-balancing prefill throughput landed within 5.6% of DeepSeek's own official profile, decode landed within 6.6%, and the resulting cost came to $0.20 per million output tokens — about one-fifth of the official DeepSeek Chat API price at the time6. Those numbers are evidence that the flags in this section are not a toy example; they are the same shape of command that produced a validated, published, near-DeepSeek-parity deployment.
7. Heterogeneous TP with the GPU staging buffer
Every example so far uses matching parallelism on both sides of the PD split (TP16/DP8 on both prefill and decode). That symmetry is not required, and SGLang has a specific mechanism for when it is deliberately broken: prefill and decode often want different tensor-parallel degrees, because prefill is compute-bound and benefits from a wider TP group to parallelize the initial forward pass, while decode is memory-bandwidth-bound and can run efficiently — sometimes more efficiently — at a narrower TP degree with more replicas. Running heterogeneous TP across the PD boundary means the KV cache produced by a 16-way-sharded prefill worker has to be reshaped into whatever partition scheme a differently-sized decode worker expects, instead of a direct block-for-block copy.
SGLang's answer is a GPU-resident staging buffer, enabled with SGLANG_DISAGG_STAGING_BUFFER=1 and sized with SGLANG_DISAGG_STAGING_POOL_SIZE_MB (default 4096)1. Rather than transferring KV cache directly per-token-slice across mismatched TP partitions, the staging buffer holds an intermediate GPU-resident copy that both sides can read from and write to at their own partitioning — decoupling the transfer schedule from the reshape. SGLang's own documentation reports this yields a 2–5× throughput improvement over the default per-token slice approach at high concurrency, and gets within roughly 5% of a homogeneous-TP baseline's throughput despite the extra reshape step1.
The mechanism is explicitly restricted: it "is designed for non-MLA models (e.g. GQA, MHA)," and models using Multi-head Latent Attention — DeepSeek-V2/V3 among them — should not enable the flag1. This is a direct consequence of what MLA does to the KV cache layout: MLA compresses the key/value representation into a single low-rank latent vector shared across all attention heads rather than storing per-head K/V tensors, so there is no per-head partition for a staging buffer built around reshaping a head-sharded tensor to reconcile against. The heterogeneous-TP staging buffer and DeepSeek-class wide-EP deployment (§6) are consequently closer to mutually exclusive than composable: the deployment recipe in §6 is exactly the case — an MLA model at DeepSeek scale — where this section's mechanism does not apply, and heterogeneous TP would need a different reshape strategy or matching TP degrees on both sides instead.
For non-MLA models — a dense Llama-class or GQA/MHA MoE model without DeepSeek's specific attention design — heterogeneous TP with the staging buffer is the mechanism that lets a deployer size prefill and decode pools independently by compute/memory profile rather than being forced into matching TP degrees purely because the KV transfer path can't otherwise reconcile them.
8. Failure detection, and what is production-ready versus still hardening
Independent of transport backend, the engine layer detects a dead or stuck prefill/decode peer with four environment-variable-tunable defaults1:
| Variable | Default | What it protects against |
|---|---|---|
| SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL | 5.0 s | How often a decode worker pings a prefill peer it has an in-flight transfer with |
| SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE | 2 | Consecutive missed heartbeats before the peer is declared dead |
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT | 300 s | Ceiling on the bootstrap handshake itself (Module 4, §3) completing before the request is failed |
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT | 300 s | Ceiling on a decode worker waiting for a KV transfer to complete once bootstrap has finished |
Two consecutive missed 5-second heartbeats (10 seconds) is the fast-path failure signal for a peer that has already established a transfer; the two 300-second timeouts are the slow-path ceilings for a handshake or transfer that never completes cleanly rather than one that stalls after starting successfully. All four are per-request-transfer state, not fleet-wide — a single hung prefill worker triggers failure detection only for the requests actually bound to it.
Aman asked specifically what in this stack is production-ready versus experimental, and SGLang's own issue tracker gives a dated, first-party answer rather than requiring guesswork from documentation tone. A roadmap issue opened 2026-03-30 to track Q2 2026 PD work opens by stating that "Prefill-Decode Disaggregation has been an important and widely adopted feature for LLM inference production," and that the maintainers consider themselves to have achieved broad compatibility "across different model architectures, all parallel strategies, and almost all advanced features"7. That is a first-party claim, not an independent audit, but it is dated close to this module's as-of date and it is candid about what remains unfinished: the same issue lists open work for new model architectures (hybrid linear-attention models), deeper integration with cache-management features (HiSparse, HiCache, RadixCache for agentic workloads), routing-strategy enhancements for heterogeneous instance pools, and an "Encoder-Prefill-Decode" extension for vision-language models still under active development7. Read together, the core prefill/decode split and the flags in §§4–7 are the settled part of the stack; new model-architecture coverage and adjacent feature integration are the part still moving.
The router specifically has its own, separate history here, distinct from the PD mechanism's maturity. A router architecture-improvement proposal filed June 25, 2025 — before the Model Gateway rewrite described in §2 — listed concrete production-readiness gaps: workers represented internally as bare strings with no health or load tracking, routing logic duplicated between the regular and PD code paths, a router-enum design that made adding new policies risky, metrics scattered across files with inconsistent naming, service discovery with no retry logic (so a single transient failure could permanently drop a healthy worker), and no dynamic PD worker management without a full restart5. Every one of those gaps maps onto a feature this module has already described as present in the current SGLang Model Gateway: a typed WorkerManager and shared worker registry (§2), unified metrics under a consistent smg_* prefix (§2), circuit breakers and a documented retry executor with exponential backoff (§2), and the POST /workers/DELETE /workers/{worker_id} dynamic registration API (§4). The rewrite appears to have closed most of the specific 2025 gap list, but there is no independently published changelog confirming each item, item by item, against that original proposal — the correspondence here is this module's own reading of the before/after documentation, not a project-published closure report.
9. Wide-EP deployment patterns and the SGL-JAX preview
Everything in §6's deployment recipe that touches expert parallelism — --enable-dp-attention and --moe-a2a-backend deepep — is SGLang's implementation of the attention-DP-plus-expert-parallel layout Module 5 covers mechanically: TP=1-equivalent attention replicated per DP rank against a private KV cache, with MoE layers dispatched across the wider expert-parallel group via DeepEP's kernels1. This module has not re-derived that mechanism; the flags in §6 are the concrete SGLang-side entry point into it, and the EP-degree choices, placement policy (EPLB), and measured DeepEP bandwidth numbers behind them are Module 5's subject. The 96-H100 reproduction cited in §6 is the validated end-to-end proof that this flag combination, at DeepSeek scale, reproduces DeepSeek's own published per-node throughput within single-digit percentage points6 — the strongest evidence available that the router, PD, and wide-EP pieces covered separately across this module compose correctly in a real multi-node deployment.
One extension sits outside the GPU-centric stack this course otherwise assumes: SGLang runs on Google TPUs today through a bridge called SGL-JAX, supporting model families including Gemma, Qwen, DeepSeek, GLM, Mimo, Kimi, Ling, MiniMax, and Grok, plus the diffusion models Wan and Flux8. A second, PyTorch-native TPU backend, SGL-torchtpu, is planned with eager execution, MPMD support, and the same parallelism menu (data, tensor, expert, context, and pipeline parallelism) plus Radix Cache, HiCache, quantization, and speculative decoding running on TPU Pallas kernels8. This module does not cover TPU-side ICI topology, Pathways orchestration, or JetStream — that is Module 18's full treatment — but the existence of SGL-JAX matters here as a scope boundary: the router, PD-disaggregation, and wide-EP mechanisms described in §§2–7 are documented and versioned against SGLang's GPU backend, and how completely each one carries over to the JAX/TPU backend is not yet itemized in SGL-JAX's own announcement in the way §§4–7's GPU-side flags are.
Where experts disagree
This module's mechanisms do not disagree with each other or with earlier modules on facts. Where the open-source stack genuinely diverges is on a design question this module's evidence bears on directly: should routing and PD-awareness live inside a single lightweight process tightly coupled to one inference engine, the way SGLang builds it, or inside a separate orchestration layer that sits above the inference engine and can span several of them, the way NVIDIA Dynamo and llm-d build it (Modules 9 and 10)? Both are shipping, maintained answers to the same architectural question, not a settled-versus-experimental split.
Keep routing and PD-awareness in one process
SGLang's sglang-router / Model Gateway design32
A single Rust binary owns worker discovery, health checking, load-balancing policy, PD-role assignment, retries, circuit breakers, rate limiting, and observability, all as one deployable artifact with no dependency beyond the SGLang workers it fronts. The router is not aware of Kubernetes unless --service-discovery is explicitly turned on, and works identically on bare-metal, VM, or containerized workers.
- Fewest moving parts: one process to deploy, version, and debug in front of a fleet.
- Fast iteration — the June-2025-to-current rewrite (§8) shows the project can close design gaps by rebuilding the router itself, without coordinating a change across a separate control-plane project.
- No extra network hop between routing decision and the request reaching a worker.
- Engine-specific: this router only understands SGLang workers, not a mixed vLLM/SGLang/TensorRT-LLM fleet.
- Until the Model Gateway rewrite, it carried real production gaps (untyped workers, no dynamic PD worker management, scattered metrics) that a smaller, less-resourced router project might not fix as quickly (§8).
Pull routing and PD-awareness into a separate orchestration layer
NVIDIA Dynamo's Router and Planner9 (Module 9); llm-d's Endpoint Picker built on the Kubernetes Gateway API Inference Extension (Module 10)
Dynamo describes itself explicitly as "the orchestration layer above inference engines," not a replacement for them, and supports vLLM, SGLang, and TensorRT-LLM as interchangeable backends behind the same Router and SLA-driven Planner9. llm-d's Endpoint Picker is a reference implementation of a Kubernetes-standard protocol (Module 10) that any inference engine's pods can sit behind. Both put routing intelligence one layer up, outside any single engine's process.
- Engine-agnostic: one control plane can front a heterogeneous fleet, or let an operator swap engines without re-architecting routing.
- Kubernetes-native by design (llm-d) or with first-class Kubernetes gang-scheduling (Dynamo's Grove) — fits an already-Kubernetes-standardized platform team's operating model directly.
- More scoring signals available in principle: llm-d's layered filter-score-select pipeline (Module 2) composes several independent scorers rather than one router's fixed policy set.
- More moving parts and at least one more network hop between the orchestration layer's decision and the engine that serves the request.
- Kubernetes-shaped complexity persists even for a single-cluster, single-engine deployment that does not need multi-engine flexibility.
For a fleet that only ever runs SGLang, the embedded-router approach is the better default: it has fewer components to operate, and the specific production gaps a mid-2025 architecture review identified in the pre-rewrite router (§8) appear to have been closed by the same team that owns the inference engine, rather than requiring cross-project coordination. For a fleet that mixes engines, already standardizes on Kubernetes' Gateway API, or wants routing decisions that survive a future engine migration, the orchestration-layer approach is the more defensible choice, at the cost of the extra hop and component count. Neither position is backed by a published head-to-head benchmark comparing sglang-router/Model Gateway against Dynamo's Router or llm-d's EPP under an identical PD topology and identical hardware as of this module's 2026-09-17 as-of date — that comparison, if either project publishes it, would be the evidence that actually settles which architecture wins for a given fleet composition rather than which one is easier to reason about in the abstract.
References
- officialSGLang contributors. PD Disaggregation (SGLang Documentation). docs.sglang.io (live), 2026. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: two-problem disaggregation motivation, transfer-backend selection flags (Mooncake/NIXL), heterogeneous-TP staging-buffer flags and numbers, failure-detection defaults, multi-node DeepSeek deployment recipe flags.
- repoSGLang contributors. sglang-router (PyPI package page). SGLang / RadixArk, 0.3.2, 2026-01-15. https://pypi.org/project/sglang-router/. Accessed 2026-09-17. Used for: package description and version number (matching Module 2's citation).
- repoSGLang / RadixArk contributors. SGLang Model Gateway (README). sgl-project/sglang repository, main branch. https://github.com/sgl-project/sglang/blob/main/sgl-model-gateway/README.md. Accessed 2026-09-17. Used for: control-plane/data-plane architecture, the five load-balancing policies and their config flags, PD worker-registration API, reliability primitives (retries, circuit breakers, rate limiting), observability metrics.
- blogLMSYS Org. SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs. lmsys.org, 2024-12-04. https://www.lmsys.org/blog/2024-12-04-sglang-v0-4/. Accessed 2026-09-17. Used for: the cache-aware load balancer's approximate-radix-tree mechanism and its measured throughput/hit-rate numbers.
- reposgl-project/sglang GitHub. SGLang Router Architecture Improvement Proposal. Issue #7532, opened 2025-06-25. https://github.com/sgl-project/sglang/issues/7532. Accessed 2026-09-17. Used for: pre-rewrite router production-readiness gaps (untyped workers, no dynamic PD worker management, scattered observability).
- blogLMSYS Org. Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs. lmsys.org, 2025-05-05. https://www.lmsys.org/blog/2025-05-05-large-scale-ep/. Accessed 2026-09-17. Used for: the 96-H100 reproduction's hardware layout, EP degrees, per-node throughput, cost, and accuracy against DeepSeek's own published profile, cross-referenced from Module 6.
- reposgl-project/sglang GitHub. [Roadmap] Prefill-Decode Disaggregation Roadmap (2026 Q2). Issue #21703, opened 2026-03-30. https://github.com/sgl-project/sglang/issues/21703. Accessed 2026-09-17. Used for: first-party production-adoption claim and the open Q2 2026 gap list (new model architectures, EPD, cache-management integration).
- blogLMSYS Org / RadixArk. RadixArk Joins Forces with Google to Bring Full SGLang Features to TPUs. lmsys.org, 2026-07-30. https://www.lmsys.org/blog/2026-07-30-sglang-google-tpu/. Accessed 2026-09-17. Used for: SGL-JAX's supported model families and the planned SGL-torchtpu backend, previewed ahead of Module 18.
- repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo (README). GitHub, v1.4.2 (container tags), 2026. https://github.com/ai-dynamo/dynamo. Accessed 2026-09-17. Used for: Dynamo's self-description as an orchestration layer above inference engines, in the disagree section.