Multi-Node LLM Inference: From One Node to N · Module 4
KV Transfer and Scheduling Across the Prefill/Decode Boundary
Module 3 made the case for splitting prefill and decode into separate pools. This module is the runtime "how": the generic connector abstraction every disaggregation backend implements, why vLLM and TensorRT-LLM make a decode worker wait for its own transfer to finish while SGLang lets it start admitting the request during that transfer, how a prefill worker gets paired with a decode worker, what happens when a worker joins, drains, or dies mid-transfer, and exactly which of these behaviors are documented defaults versus still-open gaps. After reading, you can trace a request's path from a prefill worker to its first decoded token, step by step, in vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo.
1. The connector abstraction: producer, consumer, and three generic primitives
Every disaggregation backend solves the same problem: a prefill worker holds KV cache for a request it will never decode, and a decode worker needs that KV cache before it can emit a token. Strip away engine-specific naming and the shape is always producer/consumer — the prefill worker is a kv producer, the decode worker is a kv consumer — connected by something that moves tensors between two GPUs that may be on different nodes. vLLM's documentation names this shape most explicitly and is the cleanest reference vocabulary for the pattern1.
vLLM decomposes the connector into three abstractions1:
- Connector — the top-level interface. It lets a kv consumer retrieve the KV caches of a batch of requests from a kv producer. Everything below is plumbing the Connector composes.
- LookupBuffer — a small key-value-store-like layer with two operations that read like SQL:
insertis non-blocking and stores a KV cache tensor keyed by request identity;drop_selectis blocking and retrieves a matching tensor while removing it from the buffer, so a tensor is delivered to exactly one consumer and then freed. - Pipe — a single-direction FIFO for raw tensor transmission, exposing
send_tensorandrecv_tensor. The Pipe doesn't know about requests or KV semantics; it moves bytes.
A second split that matters operationally: vLLM implements this as two connector types living in two different processes1. A scheduler connector runs inside the scheduler process and decides when a KV transfer should happen — it schedules the operation against the request's lifecycle. A worker connector runs inside each GPU worker process and actually executes the transfer once scheduled. This separation is why the mechanism composes cleanly with vLLM's existing scheduler loop: the scheduler doesn't touch tensors, and the worker doesn't make admission decisions.
vLLM ships eight named connectors that implement this trio against different transports and policies, all configured through the same --kv-transfer-config flag1. NIXL and Mooncake are treated purely as connector backends here — how they move bytes is Module 7's and Module 8's subject; this module cares only about the abstraction each one plugs into.
| Connector | Backs onto | Notes |
|---|---|---|
| ExampleConnector | in-repo reference implementation | minimal, for reading the interface |
| LMCacheConnectorV1 | NIXL, underneath LMCache's policy layer | LMCache handles chunking/reuse, NIXL moves the bytes (Module 8) |
| NixlConnector | NIXL directly, backend list configurable | fully async send/recv; supports UCX and GDS backends |
| MooncakeConnector | Mooncake's transfer engine | dedicated connector for Mooncake-based clusters (Module 7) |
| MoRIIOConnector | ROCm I/O path | ROCm-exclusive; the AMD-side disaggregation connector |
| MultiConnector | an ordered list of other connectors | orchestrates several connectors together via kwargs list |
| OffloadingConnector | CPU memory | offloads KV to host RAM, configurable block size/allocation |
| FlexKVConnectorV1 | a distributed KV store | multi-level cache management across tiers |
2. Request-ID-keyed transfer: TensorRT-LLM's cache-transceiver
TensorRT-LLM uses different vocabulary — context servers for prefill, generation servers for decode — but the same producer/consumer shape sits underneath. The component responsible for moving the KV cache between them is KvCacheTransceiverV2. Its own documentation describes it as responsible for "efficient transmission and reception of the cache, promptly releasing cache space, and performing cache layout conversions during the exchange process"2. That last clause matters: a context server and a generation server can run different parallelism strategies (different TP/PP degrees, different attention layouts), so the transceiver has to re-tile the KV blocks in flight, not just copy bytes.
2.1 The disagg_request_id binding mechanism
Because context and generation are separate processes with no shared memory, TensorRT-LLM needs a single identifier that both sides agree names the same logical request. That identifier is DisaggregatedParams.disagg_request_id. The documentation is direct about the failure mode this exists to prevent: "the ctx↔gen KV-cache transfer is keyed by it, so a collision (two in-flight requests with the same ID) corrupts the transfer"2. Get the ID scheme wrong and you don't get an error — you get one request's tokens attending over another request's KV cache.
The ID is a 64-bit snowflake-style value, laid out as2:
[ 0 (1 bit) | timestamp_ms (39 bits) | node_id (8 bits) | process_id (6 bits) | counter (10 bits) ]
node_id defaults to a hash of the machine's MAC address but is overridable in the disaggregated config; process_id (0–63) is set automatically per worker by the launcher via TRTLLM_DISAGG_WORKER_PROCESS_ID, specifically so that "co-located workers never emit the same ID in the same millisecond"2. Global IDs occupy the range \([2^{40}, 2^{63})\) and worker-local IDs occupy \([0, 2^{40})\), which keeps the two ID namespaces from colliding with each other by construction.
disagg_request_id snowflake ID2. See the worked derivation below for what this headroom means in practice.Worked derivation: ID-space headroom per node
The 39-bit millisecond timestamp field gives a wraparound period of
$$ \frac{2^{39}\ \text{ms}}{1000 \times 3600 \times 24 \times 365.25} \approx 17.42\ \text{years} $$which is why the field is a millisecond counter rather than a full Unix epoch in seconds — it buys multi-decade uniqueness within 39 bits instead of needing 64. Per node, the process and counter fields bound how many distinct requests can be ID'd in a single millisecond: 6 bits of process_id allow \(2^6 = 64\) concurrent worker processes per node, and 10 bits of counter allow \(2^{10} = 1024\) IDs per process per millisecond, for a per-node ceiling of
— several orders of magnitude above any request-admission rate a single node will see, which is the point of a snowflake scheme: pick field widths so collisions are structurally impossible at realistic load rather than statistically unlikely.
2.2 Backend selection and GB200 caveats
The cache-transceiver's default backend is NIXL, and NIXL's own internal transport is selected by the environment variable TRTLLM_NIXL_KVCACHE_BACKEND, with valid values UCX (the default) or LIBFABRIC; an unsupported value logs a warning and falls back to UCX2. The transceiver has no default backend configured at all — it must be set explicitly in both the context and generation workers' config files, and a missing setting produces the error "Disaggregated serving is not enabled, please check the configuration"2. On GB200-class NVLink domains, a documented failure mode is workers hanging or running slow when different NVLink domains are involved, with the fix being to set UCX_CUDA_IPC_ENABLE_MNNVL: n and adjust UCX_NET_DEVICES2 — a reminder that the transport-selection knobs interact with the interconnect topology from Module 1, not just with the disaggregation logic.
One overlap optimization is documented directly: "TensorRT LLM overlaps the KV cache transmission with computation for multiple independent requests"2. Note the scope of that sentence — it overlaps one request's transfer with another request's compute, at the batch level. It says nothing about a single request's own decode step starting before its own transfer finishes. That distinction is the subject of section 4.
3. SGLang's bootstrap handshake
SGLang frames disaggregation as solving two problems on a unified engine: Prefill Interruption, where incoming prefill batches delay token generation for requests already decoding, and DP Attention Imbalance, where data-parallel workers end up processing mismatched mixes of prefill and decode work3. That framing is worth noticing on its own terms — it is a different diagnosis from DistServe's and Mooncake's interference argument covered in Module 3, even though both conclude that separate pools help.
The mechanism SGLang uses to hand a request from a prefill worker to a decode worker is a bootstrap handshake, run per request during initialization3:
- Decode initiates. The decode worker allocates destination KV cache slots for the incoming request and sends those indices to the assigned prefill worker.
- Prefill collects. The prefill worker receives and holds the destination indices during the bootstrap phase.
- Prefill prepares. Once its forward pass produces the KV blocks, the prefill worker slices out exactly the blocks the decode side asked for.
- Backend transfers. The KV slices move over the selected transfer backend (Mooncake or NIXL).
- Decode scatters. The decode worker receives the transferred data and writes it into its own KV cache pages at the pre-allocated slots.
Each bootstrap session is identified by a room_id together with the decode worker's host and port — SGLang's equivalent of TensorRT-LLM's disagg_request_id, serving the same collision-avoidance role but carried as connection metadata rather than a single packed integer. A background cleanup task runs on an interval that defaults to 120 seconds, releasing bootstrap-side memory left behind by decode nodes that disconnected without completing the handshake3.
3.1 Heterogeneous TP with a GPU staging buffer
Prefill and decode often want different tensor-parallel degrees — prefill is compute-bound and benefits from more TP to spread FLOPs, decode is memory-bandwidth-bound and can run leaner. But different TP sizes mean the KV cache is sharded differently on each side, and a naive per-token slice-by-slice transfer pays a latency tax reassembling those shards. SGLang's fix is a GPU staging buffer: it gathers KV head slices into a contiguous buffer on the prefill side, performs one bulk RDMA transfer, then scatters into the correct KV cache pages on the decode side3. Measured gain is 2–5× throughput improvement over the default per-token slice approach at high concurrency3.
Two scope limits apply. First, staging only activates when prefill and decode use different TP sizes with the Mooncake transfer backend specifically — when both sides match, staging is automatically bypassed even if the flag is enabled3. Second, the staging buffer targets non-MLA attention (GQA, MHA); MLA models such as DeepSeek-V2/V3 should not enable it3, because MLA's compressed latent KV representation doesn't decompose into the same per-head slices the staging buffer assumes.
| Engine | Mechanism | Carried as |
|---|---|---|
| TensorRT-LLM | disagg_request_id | packed 64-bit snowflake int, one field per request |
| SGLang | room_id + decode host/port | bootstrap connection metadata (bootstrap_info) |
| vLLM | block IDs + connection info | kv_transfer_params, passed through engine output |
| Dynamo (backend-agnostic) | whichever of the three above the chosen backend uses | opaque per-backend metadata (opaque_state for TensorRT-LLM) |
4. Synchronous vs. overlapped transfer
This is the design decision that most changes what a request's time-to-first-token looks like, and it's easy to state imprecisely. Two distinct properties are in play, and they don't always move together.
4.1 Two different kinds of "non-blocking"
The first property is engine-level non-blocking: does the GPU running the prefill worker sit idle while one request's KV cache streams out, or does it keep serving its batch? NVIDIA Dynamo's documentation states this as a general property of NIXL-based transfer: "the KV transfer is non-blocking, allowing GPU forward passes to continue serving other requests during the transfer"4. TensorRT-LLM's cache-transceiver does the same thing at the batch level — it "overlaps the KV cache transmission with computation for multiple independent requests"2. Both statements are about the source GPU's utilization across different requests, not about any single request's own critical path.
The second property is request-level synchronicity: for one specific request, does its decode phase wait until its own KV transfer has fully landed, or can the decode worker start doing useful work on that request before the transfer completes? This is where the three engines diverge, and Dynamo's documentation states the divergence directly by comparing what each backend does4:
| Backend | Transfer metadata | Execution model |
|---|---|---|
| SGLang | bootstrap_info (host, port, room_id) | asynchronous — prefill runs as a background task; decode phase begins immediately while transfer proceeds in parallel |
| vLLM | kv_transfer_params (block IDs, connection info) | synchronous — decode waits for prefill to complete before proceeding |
| TensorRT-LLM | opaque_state (serialized internal metadata) | synchronous — decode waits for prefill to complete before proceeding |
SGLang's overlap is only possible because of the bootstrap handshake from section 3: the decode worker has already reserved destination KV slots and registered a room_id before the transfer starts, so its scheduler can admit the request into decode-side bookkeeping without waiting for bytes to arrive — the actual generation of the first token still needs the KV cache to be present, but the scheduling and slot-reservation overhead that would otherwise sit serially in front of it is pulled forward. vLLM's synchronous default reflects the blocking semantics of LookupBuffer.drop_select from section 1: the consumer side calls a blocking retrieval and only proceeds once it returns. TensorRT-LLM's cache-transceiver is synchronous per request for the same reason — the generation server needs a complete, layout-converted KV cache before it can run attention correctly — while still keeping the GPU busy on other requests in the interim via the batch-level overlap from 4.1.
5. Scheduling across two pools: pairing prefill and decode workers
Disaggregation turns one scheduling decision into two: which prefill worker gets this request, and — once prefill is done — which decode worker receives its KV cache. Module 2 covered the first decision in depth (prefix-cache-aware scoring, load/SLO signals, the filter-score-select pipeline); it applies to prefill-pool placement unchanged. What's new here is the second decision, made after prefill completes, and each engine wires it differently.
5.1 Dynamo's PrefillRouter
Dynamo names this component the PrefillRouter. It selects a decode worker using "KV-aware routing (based on cache overlap scores and load) or simple load balancing"4 — the same two-signal family Module 2 described for prefill-pool routing, reapplied to the decode side. Once a decode worker is chosen, the router "injects the prefill result into the decode request, then routes to the decode worker"4: the prefill output (KV transfer metadata, generated tokens so far) is attached to the request object before it's handed off, rather than the decode worker having to separately discover where its input came from.
5.2 SGLang's router
SGLang's standalone Rust load balancer, sglang-router, exposes disaggregation-aware routing through a dedicated launch mode: sglang_router.launch_router --pd-disaggregation --prefill http://IP:PORT --decode http://IP:PORT, with pool membership declared explicitly per URL and a configurable health-check-interval-secs for liveness checking3. The router receives the client request, dispatches the prefill portion to a prefill instance, and routes the decode continuation to a decode instance3. The documentation is explicit about a related statelessness constraint: prefill and decode run in separate serving processes whose "process-local Responses stores cannot provide coherent shared state"3 — the router itself does not maintain conversation history across the split, so a client relying on server-side session state has to send explicit context rather than assume continuity.
5.3 TensorRT-LLM's disaggregated orchestrator
TensorRT-LLM's orchestration layer receives client requests and dispatches them to the appropriate context and generation servers over the OpenAI-compatible REST API, marking a dispatched request as "context-only" (skip the generation phase) when it goes to a context server and "generation-only" (skip the context phase) when it goes to a generation server2. The handoff metadata is called ctx_params: it moves from the context instance to the generation instance and is "used by the generation instances to establish communication with the context instance and retrieve the KV cache blocks"2 — so, unlike Dynamo's inject-then-route pattern, the generation server pulls the KV cache itself once it has the context instance's address, rather than the orchestrator pushing the payload. At the fleet level, coordinator/worker processes bind the same port so that "the kernel load-balances incoming connections across them by 4-tuple hash"2 — a detail about how multiple coordinator or worker replicas share inbound connections, not about which specific context server pairs with which generation server for a given request.
| Engine | Who decides | Selection signal | Handoff direction |
|---|---|---|---|
| Dynamo | PrefillRouter | KV-cache overlap score + load, or plain load balancing | router injects result, pushes to decode worker |
| SGLang | sglang-router | declared pool membership + health checks | router dispatches prefill, then routes continuation to decode |
| TensorRT-LLM | disaggregated orchestrator | REST dispatch to registered ctx/gen servers | orchestrator hands ctx_params; generation server pulls KV from context server |
5.4 Admission signals feeding the pairing decision
Pairing is only half the problem; the router also needs to know a decode worker has room before committing a request to it. The three engines expose that signal at different granularities. SGLang controls decode-side admission with a static, per-worker ceiling: the documented multi-node DeepSeek deployment sets --max-running-requests 128 on the decode cluster3, a hard cap the router's pool/health logic operates within rather than a continuously updated score. Dynamo ties admission to the same RuntimeConfig that a worker publishes on joining the pool (section 6): it includes KV capacity, and the PrefillRouter's load-based selection criterion draws on load alongside cache-overlap scores4 — the documentation connects the two without spelling out the exact scoring formula that combines a capacity figure with a load figure into one selection decision. For vLLM and TensorRT-LLM, once a pairing is made, admission of the transferred request into the decode worker's running batch falls back to each engine's own single-node continuous-batching admission logic — the same mechanism Aman already knows from single-node serving, just fed by a connector instead of a locally-computed KV cache.
6. xPyD runtime reconfiguration
A production disaggregated deployment runs some number \(x\) of prefill workers and \(y\) of decode workers — shorthand for this is "xPyD." Fixed pool sizes are a planning assumption, not an operational reality: load shifts, hardware fails, and capacity needs to move between the two pools without a full redeploy. Dynamo documents this capability as runtime-reconfigurable xPyD, meaning workers can be added and removed while the system is serving traffic4.
The two paths are asymmetric4:
- Add: a new worker registers with the discovery service and publishes its
RuntimeConfig, which includes its KV capacity. The router "automatically discovers new workers via the discovery service and incorporates them into routing decisions"4 — no explicit router restart or reconfiguration step is described; discovery is push-based from the worker side and pull-based from the router side. - Remove: a worker being taken out of the pool first drains its active requests, then deregisters from discovery4. Draining means in-flight requests are allowed to finish on that worker rather than being killed; only after they complete does the worker stop advertising itself and disappear from the router's view.
TensorRT-LLM's own documentation draws a sharp line under the equivalent capability on its side: dynamic node joining and leaving "is being built on top of" the existing cache-transceiver mechanism2 — explicitly in progress, not generally available. This is not a disagreement about how the mechanism should work; it's a maturity gap between two projects documenting the same category of capability at different points in its development. Dynamo's docs describe xPyD as something that already functions; TensorRT-LLM's docs describe the equivalent as future work layered on infrastructure (the cache-transceiver and disagg_request_id scheme from section 2) that already exists for the static case. Treat "runtime reconfigurable pool sizing" as a capability whose availability you need to check per engine and per release, not as a property of disaggregation in general.
7. Failure detection: heartbeats, bootstrap timeouts, and what's actually specified
A disaggregated request has a partner dependency that a colocated engine never has: the decode side is waiting on a specific remote prefill worker, and that worker can die, hang, or lose network reachability mid-transfer. How each engine detects and responds to that is documented to very different depths.
7.1 SGLang: the most fully specified defaults
SGLang publishes concrete numeric defaults for every timeout in the bootstrap path3:
| Setting | Default | Protects against |
|---|---|---|
SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL | 5.0 s | decode-side polling of prefill worker liveness |
SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE | 2 consecutive | false positives from a single dropped heartbeat |
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT | 300 s | prefill-side: handshake that never completes |
SGLANG_DISAGGREGATION_WAITING_TIMEOUT | 300 s | decode-side: transfer that never arrives |
With the defaults, a decode worker treats a prefill server as offline after two missed heartbeats at 5-second spacing — roughly 10 seconds of undetected failure in the worst case before the health signal flips. Both the bootstrap and waiting timeouts default to 300 seconds and can be relaxed; the documentation gives export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 as an example for slower networks or larger clusters3. On the cleanup side, the 120-second bootstrap-entry cleanup interval from section 3 is explicitly about releasing prefill-side memory left behind when "a running decode node loses connection"3 — note this documents the decode-dies-first direction; the sources for this module do not spell out the symmetric case of a request-level response when the prefill side dies mid-transfer for a request already past the bootstrap step, beyond the heartbeat eventually marking that prefill server offline for future routing decisions.
7.2 TensorRT-LLM: startup-time coordinator health, not mid-transfer partner death
TensorRT-LLM documents a different failure surface: a fleet worker's relationship to its own coordinator, not one worker's relationship to its transfer partner. "A fleet worker fails fast if its coordinator is unreachable: on startup it probes the coordinator's /cluster_info with bounded retry (up to --server_start_timeout seconds) and exits with an error"2. This guards against a worker starting up into a cluster whose coordinator is down or unreachable — a bring-up-time check, not a running-request check. Separately, the documentation notes an NVLink-domain hang failure mode on GB200-class hardware (section 2.2) and an HTTP connection-pool issue resolved by raising server_keep_alive_timeout when clients hold large idle connection pools2. None of this is the "context server dies while a generation server is mid-pull of its KV cache" case; the sources for this module do not document that scenario for TensorRT-LLM.
7.3 Dynamo: documented at the worker-lifecycle level only
Dynamo's disaggregated-serving documentation covers worker registration, draining, and deregistration (section 6) but does not specify heartbeat intervals, discovery-service polling periods, or a request-level response to a partner dying mid-transfer4. This is worth stating plainly rather than papering over: as of this writing, failure handling for the KV-transfer boundary specifically is the least-documented part of the three engines' disaggregation implementations, and Dynamo's own docs are the thinnest of the three on this exact point.
| Engine | Detects | Default | Not documented (per these sources) |
|---|---|---|---|
| SGLang | prefill liveness via decode-side heartbeat; unfinished bootstrap; unfinished transfer | 5.0 s interval, 2 failures, 300 s bootstrap/waiting timeout | exact per-request outcome (retry vs. fail) once prefill is marked offline mid-request |
| TensorRT-LLM | worker-to-coordinator reachability at startup; NVLink-domain hangs; idle HTTP pools | --server_start_timeout bounded retry | context-server death mid-transfer for an in-flight request |
| Dynamo | worker presence via discovery-service registration/deregistration | not specified | heartbeat interval; timeout values; mid-transfer partner-death response |
The one resilience property that does carry across all three, stated most explicitly by Dynamo, is engine-level non-blocking transfer from section 4.1: whatever happens to one request's transfer, it does not stall the source GPU's other in-flight work4. That protects throughput for everyone else on the same worker even when a specific transfer is struggling; it does not, by itself, tell you what happens to the struggling request.
Where experts disagree
The three engines don't disagree about whether disaggregation should exist — Module 3 already made that case. They disagree, in their implementation choices if not in explicit prose, about whether the transfer window should be on or off a request's critical path. That's a real design trade-off with measurable consequences, not a documentation gap, so it earns the same treatment as a sourced controversy.
Overlap-first: let decode admission proceed during transfer
SGLang's bootstrap-handshake design3, described comparatively in NVIDIA Dynamo's backend documentation4
Reserve the decode-side KV slots and register the request before the transfer even starts, so the decode worker's scheduling and admission machinery runs in parallel with the byte movement instead of after it. This is what makes the heterogeneous-TP GPU staging buffer's 2–5× throughput gain reachable at all (section 3.1) — it requires prefill and decode to already be decoupled at the point the transfer begins.
- Shrinks the portion of TTFT that is serially exposed to full transfer latency.
- Makes staging-buffer-style optimizations reachable, since they need early decoupling between the two sides.
- Documented, tunable failure detection: explicit heartbeat interval, failure-count threshold, and two separate 300 s timeouts (Table 5).
- More moving parts to operate correctly: bootstrap room IDs, a background cleanup task, heartbeat tuning.
- The staging-buffer path that benefits most from this design is explicitly incompatible with MLA models.
- The sources reviewed do not specify the exact per-request outcome when a partner dies after bootstrap but before transfer completion (section 7.1).
Synchronous-first: gate decode admission on transfer completion
vLLM's default connector behavior via LookupBuffer.drop_select1; TensorRT-LLM's cache-transceiver2; both confirmed as the synchronous case in Dynamo's own backend comparison4
Don't let the decode worker touch a request until its KV cache has fully, correctly landed. TensorRT-LLM still keeps the source GPU busy by overlapping one request's transfer with a different request's compute (section 2.2), and vLLM's blocking drop_select call means a tensor is handed to exactly one consumer with no possibility of a partial or stale read.
- Simpler correctness argument: no window where the decode worker could attend over an incomplete KV cache.
- TensorRT-LLM's batch-level overlap recovers some of the GPU-utilization cost without touching per-request ordering.
- Fewer distinct failure states to reason about, since there is no "admitted but not yet transferred" state to define behavior for.
- The full transfer duration sits serially in front of the first decode step for every request, with no documented mitigation at the per-request level.
- vLLM's own documentation concedes the whole disaggregated-prefill path "does not improve throughput" and remains labeled experimental1.
- TensorRT-LLM's equivalent of xPyD — dynamic node joining/leaving — is explicitly still being built, not GA (section 6), suggesting the synchronous implementation is also the less operationally mature one on that specific axis.
No source reviewed for this module publishes a controlled, head-to-head TTFT_P90 comparison between a synchronous connector (vLLM's NixlConnector or TensorRT-LLM's cache-transceiver) and SGLang's overlapped bootstrap path on identical hardware, model, and ISL:OSL distribution — that is the experiment that would actually settle which design wins by how much, and it does not appear to exist publicly as of 2026-09-17. Absent that number, the trade is structural rather than empirical: overlap buys latency at the cost of a wider failure-mode surface to operate (more timeouts, more state, more things that can be misconfigured), while synchronous transfer buys operational and correctness simplicity at the cost of paying the full transfer time on every request's critical path. For a deployment where Module 3's TTFT_P90 SLO is the binding constraint, the overlap design is worth its added complexity. For a first disaggregated deployment, or one where the transfer subsystem itself is still labeled experimental (as vLLM's is), starting with the synchronous default and only adopting the overlapped path once the operational failure modes are well understood is the more defensible sequencing — which is also, in effect, what each project's own maturity signals already suggest.
What is changing
All four primary sources for this module are live, versioned documentation rather than papers, which means the specific numbers in this module — timeout defaults, flag names, whether a capability is GA — are the most likely of anything in this course to drift within a year. Three trends to watch specifically: first, TensorRT-LLM's explicit "in progress" label on dynamic node joining/leaving (section 6) is a stated near-term roadmap item, built on infrastructure (the cache-transceiver and disagg_request_id scheme) that already exists for the static case — the gap with Dynamo's already-documented xPyD is likely to narrow, not persist. Second, failure handling for the mid-transfer partner-death case is the thinnest-documented mechanism across all three engines (section 7) — as disaggregation moves from early production deployments toward being the default topology for large models, expect this to get filled in with explicit retry/failover semantics rather than left to heartbeat-driven routing decisions alone. Third, the connector list itself is not fixed: vLLM's eight named connectors (Table 1) and SGLang's two transfer backends (Mooncake, NIXL) are the current state of two actively developed projects, and Modules 7 and 8 treat the transfer-engine layer underneath them in more depth, including the version churn that layer is going through.
References
- officialvLLM contributors. Disaggregated Prefilling. vLLM Documentation, docs.vllm.ai/en/latest/features/disagg_prefill/, package v0.29.0 (2026-09-09). https://docs.vllm.ai/en/latest/features/disagg_prefill/. Accessed 2026-09-17. Used for: Connector/LookupBuffer/Pipe abstraction, scheduler-vs-worker connector split, the eight named connectors and their backends, the "does not improve throughput" and experimental-status caveats.
- officialNVIDIA / TensorRT-LLM contributors. Disaggregated Serving (disagg-serving.md). TensorRT-LLM documentation, main branch. https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disagg-serving.md. Accessed 2026-09-17. Used for: KvCacheTransceiverV2 mechanism, disagg_request_id bit layout and binding rationale, NIXL/UCX/LIBFABRIC backend selection, GB200 NVLink-domain caveat, coordinator fail-fast behavior, dynamic node joining/leaving in-progress status.
- officialSGLang contributors. PD Disaggregation. SGLang Documentation, docs.sglang.io/advanced_features/pd_disaggregation.html. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: bootstrap handshake steps, room_id/host/port correlation, cleanup interval, heartbeat and timeout defaults, Mooncake/NIXL backend selection, heterogeneous-TP staging buffer mechanism and scope limits, sglang-router launch flags and stateless-design note, multi-node DeepSeek deployment flags.
- officialNVIDIA. Disaggregated Serving (NVIDIA Dynamo Documentation). docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. Accessed 2026-09-17. Used for: the generic three-stage pipeline framing, NIXL non-blocking VRAM-to-VRAM transfer, the backend-specific transfer-metadata/execution-model comparison table, PrefillRouter selection criteria, xPyD worker add/remove mechanism.