Multi-Node LLM Inference: From One Node to N · Module 4

KV Transfer and Scheduling Across the Prefill/Decode Boundary

Module 3 made the case for splitting prefill and decode into separate pools. This module is the runtime "how": the generic connector abstraction every disaggregation backend implements, why vLLM and TensorRT-LLM make a decode worker wait for its own transfer to finish while SGLang lets it start admitting the request during that transfer, how a prefill worker gets paired with a decode worker, what happens when a worker joins, drains, or dies mid-transfer, and exactly which of these behaviors are documented defaults versus still-open gaps. After reading, you can trace a request's path from a prefill worker to its first decoded token, step by step, in vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo.

Course 30 min read 4 sources As of 2026-09-17 Prerequisite: Module 3
Generic request lifecycle across the prefill/decode boundary: router, prefill worker, connector transfer, decode worker incoming request Router / EPP Prefill worker kv producer computes full prompt, emits KV blocks connector Transfer LookupBuffer / Pipe / bootstrap / cache-transceiver Decode worker kv consumer admits request, emits tokens first token vLLM: Connector retrieves from kv producer via LookupBuffer.drop_select over a Pipe, decode waits for completion. SGLang: bootstrap handshake reserves decode-side slots first; decode admission overlaps the transfer. TensorRT-LLM: cache-transceiver keyed by disagg_request_id.
Figure 1. The three-stage generic pipeline — prefill compute, KV transfer, decode compute — that NVIDIA Dynamo's own documentation uses to describe disaggregated serving4, with the three engine-specific transfer mechanisms this module covers overlaid at the bottom132.

1. The connector abstraction: producer, consumer, and three generic primitives

Every disaggregation backend solves the same problem: a prefill worker holds KV cache for a request it will never decode, and a decode worker needs that KV cache before it can emit a token. Strip away engine-specific naming and the shape is always producer/consumer — the prefill worker is a kv producer, the decode worker is a kv consumer — connected by something that moves tensors between two GPUs that may be on different nodes. vLLM's documentation names this shape most explicitly and is the cleanest reference vocabulary for the pattern1.

vLLM decomposes the connector into three abstractions1:

  • Connector — the top-level interface. It lets a kv consumer retrieve the KV caches of a batch of requests from a kv producer. Everything below is plumbing the Connector composes.
  • LookupBuffer — a small key-value-store-like layer with two operations that read like SQL: insert is non-blocking and stores a KV cache tensor keyed by request identity; drop_select is blocking and retrieves a matching tensor while removing it from the buffer, so a tensor is delivered to exactly one consumer and then freed.
  • Pipe — a single-direction FIFO for raw tensor transmission, exposing send_tensor and recv_tensor. The Pipe doesn't know about requests or KV semantics; it moves bytes.

A second split that matters operationally: vLLM implements this as two connector types living in two different processes1. A scheduler connector runs inside the scheduler process and decides when a KV transfer should happen — it schedules the operation against the request's lifecycle. A worker connector runs inside each GPU worker process and actually executes the transfer once scheduled. This separation is why the mechanism composes cleanly with vLLM's existing scheduler loop: the scheduler doesn't touch tensors, and the worker doesn't make admission decisions.

vLLM's Connector, LookupBuffer, and Pipe abstractions between a kv producer and a kv consumer kv producer prefill worker insert() non-blocking Connector LookupBuffer Pipe: send_tensor / recv_tensor kv consumer decode worker drop_select() blocking scheduler connector (scheduler process) decides when to transfer worker connector (GPU worker process) executes the transfer Concrete connectors implement this trio against a real transport — NIXL, Mooncake, LMCache, or CPU offload.
Figure 2. vLLM's Connector / LookupBuffer / Pipe decomposition, and the scheduler-process / worker-process split each concrete connector implements against1.

vLLM ships eight named connectors that implement this trio against different transports and policies, all configured through the same --kv-transfer-config flag1. NIXL and Mooncake are treated purely as connector backends here — how they move bytes is Module 7's and Module 8's subject; this module cares only about the abstraction each one plugs into.

Table 1. Named KV connectors in vLLM and what each backs onto1.
ConnectorBacks ontoNotes
ExampleConnectorin-repo reference implementationminimal, for reading the interface
LMCacheConnectorV1NIXL, underneath LMCache's policy layerLMCache handles chunking/reuse, NIXL moves the bytes (Module 8)
NixlConnectorNIXL directly, backend list configurablefully async send/recv; supports UCX and GDS backends
MooncakeConnectorMooncake's transfer enginededicated connector for Mooncake-based clusters (Module 7)
MoRIIOConnectorROCm I/O pathROCm-exclusive; the AMD-side disaggregation connector
MultiConnectoran ordered list of other connectorsorchestrates several connectors together via kwargs list
OffloadingConnectorCPU memoryoffloads KV to host RAM, configurable block size/allocation
FlexKVConnectorV1a distributed KV storemulti-level cache management across tiers

2. Request-ID-keyed transfer: TensorRT-LLM's cache-transceiver

TensorRT-LLM uses different vocabulary — context servers for prefill, generation servers for decode — but the same producer/consumer shape sits underneath. The component responsible for moving the KV cache between them is KvCacheTransceiverV2. Its own documentation describes it as responsible for "efficient transmission and reception of the cache, promptly releasing cache space, and performing cache layout conversions during the exchange process"2. That last clause matters: a context server and a generation server can run different parallelism strategies (different TP/PP degrees, different attention layouts), so the transceiver has to re-tile the KV blocks in flight, not just copy bytes.

2.1 The disagg_request_id binding mechanism

Because context and generation are separate processes with no shared memory, TensorRT-LLM needs a single identifier that both sides agree names the same logical request. That identifier is DisaggregatedParams.disagg_request_id. The documentation is direct about the failure mode this exists to prevent: "the ctx↔gen KV-cache transfer is keyed by it, so a collision (two in-flight requests with the same ID) corrupts the transfer"2. Get the ID scheme wrong and you don't get an error — you get one request's tokens attending over another request's KV cache.

The ID is a 64-bit snowflake-style value, laid out as2:

[ 0 (1 bit) | timestamp_ms (39 bits) | node_id (8 bits) | process_id (6 bits) | counter (10 bits) ]

node_id defaults to a hash of the machine's MAC address but is overridable in the disaggregated config; process_id (0–63) is set automatically per worker by the launcher via TRTLLM_DISAGG_WORKER_PROCESS_ID, specifically so that "co-located workers never emit the same ID in the same millisecond"2. Global IDs occupy the range \([2^{40}, 2^{63})\) and worker-local IDs occupy \([0, 2^{40})\), which keeps the two ID namespaces from colliding with each other by construction.

Bit-field layout of TensorRT-LLM's 64-bit disagg_request_id 1 reserved timestamp_ms 39 bits ≈ 17.4 years of range node_id 8 bits 256 nodes proc_id 6 bits 64 procs/node counter 10 bits 1024/ms/proc 64-bit int64, MSB to LSB left to right
Figure 3. Bit-field layout of the disagg_request_id snowflake ID2. See the worked derivation below for what this headroom means in practice.
Worked derivation: ID-space headroom per node

The 39-bit millisecond timestamp field gives a wraparound period of

$$ \frac{2^{39}\ \text{ms}}{1000 \times 3600 \times 24 \times 365.25} \approx 17.42\ \text{years} $$

which is why the field is a millisecond counter rather than a full Unix epoch in seconds — it buys multi-decade uniqueness within 39 bits instead of needing 64. Per node, the process and counter fields bound how many distinct requests can be ID'd in a single millisecond: 6 bits of process_id allow \(2^6 = 64\) concurrent worker processes per node, and 10 bits of counter allow \(2^{10} = 1024\) IDs per process per millisecond, for a per-node ceiling of

$$ 64 \times 1024 \times 1000\ \text{ms/s} = 65{,}536{,}000\ \text{IDs/s per node} $$

— several orders of magnitude above any request-admission rate a single node will see, which is the point of a snowflake scheme: pick field widths so collisions are structurally impossible at realistic load rather than statistically unlikely.

2.2 Backend selection and GB200 caveats

The cache-transceiver's default backend is NIXL, and NIXL's own internal transport is selected by the environment variable TRTLLM_NIXL_KVCACHE_BACKEND, with valid values UCX (the default) or LIBFABRIC; an unsupported value logs a warning and falls back to UCX2. The transceiver has no default backend configured at all — it must be set explicitly in both the context and generation workers' config files, and a missing setting produces the error "Disaggregated serving is not enabled, please check the configuration"2. On GB200-class NVLink domains, a documented failure mode is workers hanging or running slow when different NVLink domains are involved, with the fix being to set UCX_CUDA_IPC_ENABLE_MNNVL: n and adjust UCX_NET_DEVICES2 — a reminder that the transport-selection knobs interact with the interconnect topology from Module 1, not just with the disaggregation logic.

One overlap optimization is documented directly: "TensorRT LLM overlaps the KV cache transmission with computation for multiple independent requests"2. Note the scope of that sentence — it overlaps one request's transfer with another request's compute, at the batch level. It says nothing about a single request's own decode step starting before its own transfer finishes. That distinction is the subject of section 4.

3. SGLang's bootstrap handshake

SGLang frames disaggregation as solving two problems on a unified engine: Prefill Interruption, where incoming prefill batches delay token generation for requests already decoding, and DP Attention Imbalance, where data-parallel workers end up processing mismatched mixes of prefill and decode work3. That framing is worth noticing on its own terms — it is a different diagnosis from DistServe's and Mooncake's interference argument covered in Module 3, even though both conclude that separate pools help.

The mechanism SGLang uses to hand a request from a prefill worker to a decode worker is a bootstrap handshake, run per request during initialization3:

Decode worker sends dest. KV indices Prefill worker collects indices during bootstrap Prefill worker prepares KV slices for those indices Mooncake / NIXL transfer backend bulk KV transfer decode scatters into KV pages
  1. Decode initiates. The decode worker allocates destination KV cache slots for the incoming request and sends those indices to the assigned prefill worker.
  2. Prefill collects. The prefill worker receives and holds the destination indices during the bootstrap phase.
  3. Prefill prepares. Once its forward pass produces the KV blocks, the prefill worker slices out exactly the blocks the decode side asked for.
  4. Backend transfers. The KV slices move over the selected transfer backend (Mooncake or NIXL).
  5. Decode scatters. The decode worker receives the transferred data and writes it into its own KV cache pages at the pre-allocated slots.

Each bootstrap session is identified by a room_id together with the decode worker's host and port — SGLang's equivalent of TensorRT-LLM's disagg_request_id, serving the same collision-avoidance role but carried as connection metadata rather than a single packed integer. A background cleanup task runs on an interval that defaults to 120 seconds, releasing bootstrap-side memory left behind by decode nodes that disconnected without completing the handshake3.

3.1 Heterogeneous TP with a GPU staging buffer

Prefill and decode often want different tensor-parallel degrees — prefill is compute-bound and benefits from more TP to spread FLOPs, decode is memory-bandwidth-bound and can run leaner. But different TP sizes mean the KV cache is sharded differently on each side, and a naive per-token slice-by-slice transfer pays a latency tax reassembling those shards. SGLang's fix is a GPU staging buffer: it gathers KV head slices into a contiguous buffer on the prefill side, performs one bulk RDMA transfer, then scatters into the correct KV cache pages on the decode side3. Measured gain is 2–5× throughput improvement over the default per-token slice approach at high concurrency3.

Two scope limits apply. First, staging only activates when prefill and decode use different TP sizes with the Mooncake transfer backend specifically — when both sides match, staging is automatically bypassed even if the flag is enabled3. Second, the staging buffer targets non-MLA attention (GQA, MHA); MLA models such as DeepSeek-V2/V3 should not enable it3, because MLA's compressed latent KV representation doesn't decompose into the same per-head slices the staging buffer assumes.

Table 2. Each engine's request-correlation mechanism for binding a prefill phase to its matching decode phase1234.
EngineMechanismCarried as
TensorRT-LLMdisagg_request_idpacked 64-bit snowflake int, one field per request
SGLangroom_id + decode host/portbootstrap connection metadata (bootstrap_info)
vLLMblock IDs + connection infokv_transfer_params, passed through engine output
Dynamo (backend-agnostic)whichever of the three above the chosen backend usesopaque per-backend metadata (opaque_state for TensorRT-LLM)

4. Synchronous vs. overlapped transfer

This is the design decision that most changes what a request's time-to-first-token looks like, and it's easy to state imprecisely. Two distinct properties are in play, and they don't always move together.

4.1 Two different kinds of "non-blocking"

The first property is engine-level non-blocking: does the GPU running the prefill worker sit idle while one request's KV cache streams out, or does it keep serving its batch? NVIDIA Dynamo's documentation states this as a general property of NIXL-based transfer: "the KV transfer is non-blocking, allowing GPU forward passes to continue serving other requests during the transfer"4. TensorRT-LLM's cache-transceiver does the same thing at the batch level — it "overlaps the KV cache transmission with computation for multiple independent requests"2. Both statements are about the source GPU's utilization across different requests, not about any single request's own critical path.

The second property is request-level synchronicity: for one specific request, does its decode phase wait until its own KV transfer has fully landed, or can the decode worker start doing useful work on that request before the transfer completes? This is where the three engines diverge, and Dynamo's documentation states the divergence directly by comparing what each backend does4:

Table 3. Backend-specific transfer metadata and request-level execution model, per NVIDIA Dynamo's own comparison4.
BackendTransfer metadataExecution model
SGLangbootstrap_info (host, port, room_id)asynchronous — prefill runs as a background task; decode phase begins immediately while transfer proceeds in parallel
vLLMkv_transfer_params (block IDs, connection info)synchronous — decode waits for prefill to complete before proceeding
TensorRT-LLMopaque_state (serialized internal metadata)synchronous — decode waits for prefill to complete before proceeding

SGLang's overlap is only possible because of the bootstrap handshake from section 3: the decode worker has already reserved destination KV slots and registered a room_id before the transfer starts, so its scheduler can admit the request into decode-side bookkeeping without waiting for bytes to arrive — the actual generation of the first token still needs the KV cache to be present, but the scheduling and slot-reservation overhead that would otherwise sit serially in front of it is pulled forward. vLLM's synchronous default reflects the blocking semantics of LookupBuffer.drop_select from section 1: the consumer side calls a blocking retrieval and only proceeds once it returns. TensorRT-LLM's cache-transceiver is synchronous per request for the same reason — the generation server needs a complete, layout-converted KV cache before it can run attention correctly — while still keeping the GPU busy on other requests in the interim via the batch-level overlap from 4.1.

Timeline comparing SGLang's overlapped decode admission against vLLM/TensorRT-LLM's synchronous wait-for-completion transfer SGLang (asynchronous) bootstrap handshake KV transfer (background) decode admitted, scheduling begins first token vLLM / TensorRT-LLM (synchronous) handoff (ID/params) KV transfer (drop_select / transceiver, blocking) decode admitted first token decode admission starts ~100px earlier on the async path, relative to transfer duration
Figure 4. SGLang's decode admission overlaps the KV transfer window; vLLM and TensorRT-LLM gate decode admission on transfer completion4. Bar widths are illustrative of the ordering documented in Table 3, not measured durations — no controlled cross-engine timing benchmark is available in the sources for this module.

5. Scheduling across two pools: pairing prefill and decode workers

Disaggregation turns one scheduling decision into two: which prefill worker gets this request, and — once prefill is done — which decode worker receives its KV cache. Module 2 covered the first decision in depth (prefix-cache-aware scoring, load/SLO signals, the filter-score-select pipeline); it applies to prefill-pool placement unchanged. What's new here is the second decision, made after prefill completes, and each engine wires it differently.

5.1 Dynamo's PrefillRouter

Dynamo names this component the PrefillRouter. It selects a decode worker using "KV-aware routing (based on cache overlap scores and load) or simple load balancing"4 — the same two-signal family Module 2 described for prefill-pool routing, reapplied to the decode side. Once a decode worker is chosen, the router "injects the prefill result into the decode request, then routes to the decode worker"4: the prefill output (KV transfer metadata, generated tokens so far) is attached to the request object before it's handed off, rather than the decode worker having to separately discover where its input came from.

5.2 SGLang's router

SGLang's standalone Rust load balancer, sglang-router, exposes disaggregation-aware routing through a dedicated launch mode: sglang_router.launch_router --pd-disaggregation --prefill http://IP:PORT --decode http://IP:PORT, with pool membership declared explicitly per URL and a configurable health-check-interval-secs for liveness checking3. The router receives the client request, dispatches the prefill portion to a prefill instance, and routes the decode continuation to a decode instance3. The documentation is explicit about a related statelessness constraint: prefill and decode run in separate serving processes whose "process-local Responses stores cannot provide coherent shared state"3 — the router itself does not maintain conversation history across the split, so a client relying on server-side session state has to send explicit context rather than assume continuity.

5.3 TensorRT-LLM's disaggregated orchestrator

TensorRT-LLM's orchestration layer receives client requests and dispatches them to the appropriate context and generation servers over the OpenAI-compatible REST API, marking a dispatched request as "context-only" (skip the generation phase) when it goes to a context server and "generation-only" (skip the context phase) when it goes to a generation server2. The handoff metadata is called ctx_params: it moves from the context instance to the generation instance and is "used by the generation instances to establish communication with the context instance and retrieve the KV cache blocks"2 — so, unlike Dynamo's inject-then-route pattern, the generation server pulls the KV cache itself once it has the context instance's address, rather than the orchestrator pushing the payload. At the fleet level, coordinator/worker processes bind the same port so that "the kernel load-balances incoming connections across them by 4-tuple hash"2 — a detail about how multiple coordinator or worker replicas share inbound connections, not about which specific context server pairs with which generation server for a given request.

Table 4. How each engine decides which decode/generation worker receives a completed prefill234.
EngineWho decidesSelection signalHandoff direction
DynamoPrefillRouterKV-cache overlap score + load, or plain load balancingrouter injects result, pushes to decode worker
SGLangsglang-routerdeclared pool membership + health checksrouter dispatches prefill, then routes continuation to decode
TensorRT-LLMdisaggregated orchestratorREST dispatch to registered ctx/gen serversorchestrator hands ctx_params; generation server pulls KV from context server

5.4 Admission signals feeding the pairing decision

Pairing is only half the problem; the router also needs to know a decode worker has room before committing a request to it. The three engines expose that signal at different granularities. SGLang controls decode-side admission with a static, per-worker ceiling: the documented multi-node DeepSeek deployment sets --max-running-requests 128 on the decode cluster3, a hard cap the router's pool/health logic operates within rather than a continuously updated score. Dynamo ties admission to the same RuntimeConfig that a worker publishes on joining the pool (section 6): it includes KV capacity, and the PrefillRouter's load-based selection criterion draws on load alongside cache-overlap scores4 — the documentation connects the two without spelling out the exact scoring formula that combines a capacity figure with a load figure into one selection decision. For vLLM and TensorRT-LLM, once a pairing is made, admission of the transferred request into the decode worker's running batch falls back to each engine's own single-node continuous-batching admission logic — the same mechanism Aman already knows from single-node serving, just fed by a connector instead of a locally-computed KV cache.

6. xPyD runtime reconfiguration

A production disaggregated deployment runs some number \(x\) of prefill workers and \(y\) of decode workers — shorthand for this is "xPyD." Fixed pool sizes are a planning assumption, not an operational reality: load shifts, hardware fails, and capacity needs to move between the two pools without a full redeploy. Dynamo documents this capability as runtime-reconfigurable xPyD, meaning workers can be added and removed while the system is serving traffic4.

The two paths are asymmetric4:

  • Add: a new worker registers with the discovery service and publishes its RuntimeConfig, which includes its KV capacity. The router "automatically discovers new workers via the discovery service and incorporates them into routing decisions"4 — no explicit router restart or reconfiguration step is described; discovery is push-based from the worker side and pull-based from the router side.
  • Remove: a worker being taken out of the pool first drains its active requests, then deregisters from discovery4. Draining means in-flight requests are allowed to finish on that worker rather than being killed; only after they complete does the worker stop advertising itself and disappear from the router's view.
State diagram for adding and removing a worker at runtime in Dynamo's xPyD model Add path worker starts register + publish RuntimeConfig (incl. KV capacity) router discovers, includes in routing Remove path removal requested drain active requests to completion deregister from discovery, worker gone
Figure 5. Worker add versus worker remove in Dynamo's runtime-reconfigurable xPyD model4. The two paths are not mirror images — add is push-then-discover, remove is drain-then-deregister.

TensorRT-LLM's own documentation draws a sharp line under the equivalent capability on its side: dynamic node joining and leaving "is being built on top of" the existing cache-transceiver mechanism2 — explicitly in progress, not generally available. This is not a disagreement about how the mechanism should work; it's a maturity gap between two projects documenting the same category of capability at different points in its development. Dynamo's docs describe xPyD as something that already functions; TensorRT-LLM's docs describe the equivalent as future work layered on infrastructure (the cache-transceiver and disagg_request_id scheme from section 2) that already exists for the static case. Treat "runtime reconfigurable pool sizing" as a capability whose availability you need to check per engine and per release, not as a property of disaggregation in general.

7. Failure detection: heartbeats, bootstrap timeouts, and what's actually specified

A disaggregated request has a partner dependency that a colocated engine never has: the decode side is waiting on a specific remote prefill worker, and that worker can die, hang, or lose network reachability mid-transfer. How each engine detects and responds to that is documented to very different depths.

7.1 SGLang: the most fully specified defaults

SGLang publishes concrete numeric defaults for every timeout in the bootstrap path3:

Table 5. SGLang's disaggregation failure-detection defaults3.
SettingDefaultProtects against
SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL5.0 sdecode-side polling of prefill worker liveness
SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE2 consecutivefalse positives from a single dropped heartbeat
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT300 sprefill-side: handshake that never completes
SGLANG_DISAGGREGATION_WAITING_TIMEOUT300 sdecode-side: transfer that never arrives

With the defaults, a decode worker treats a prefill server as offline after two missed heartbeats at 5-second spacing — roughly 10 seconds of undetected failure in the worst case before the health signal flips. Both the bootstrap and waiting timeouts default to 300 seconds and can be relaxed; the documentation gives export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 as an example for slower networks or larger clusters3. On the cleanup side, the 120-second bootstrap-entry cleanup interval from section 3 is explicitly about releasing prefill-side memory left behind when "a running decode node loses connection"3 — note this documents the decode-dies-first direction; the sources for this module do not spell out the symmetric case of a request-level response when the prefill side dies mid-transfer for a request already past the bootstrap step, beyond the heartbeat eventually marking that prefill server offline for future routing decisions.

7.2 TensorRT-LLM: startup-time coordinator health, not mid-transfer partner death

TensorRT-LLM documents a different failure surface: a fleet worker's relationship to its own coordinator, not one worker's relationship to its transfer partner. "A fleet worker fails fast if its coordinator is unreachable: on startup it probes the coordinator's /cluster_info with bounded retry (up to --server_start_timeout seconds) and exits with an error"2. This guards against a worker starting up into a cluster whose coordinator is down or unreachable — a bring-up-time check, not a running-request check. Separately, the documentation notes an NVLink-domain hang failure mode on GB200-class hardware (section 2.2) and an HTTP connection-pool issue resolved by raising server_keep_alive_timeout when clients hold large idle connection pools2. None of this is the "context server dies while a generation server is mid-pull of its KV cache" case; the sources for this module do not document that scenario for TensorRT-LLM.

7.3 Dynamo: documented at the worker-lifecycle level only

Dynamo's disaggregated-serving documentation covers worker registration, draining, and deregistration (section 6) but does not specify heartbeat intervals, discovery-service polling periods, or a request-level response to a partner dying mid-transfer4. This is worth stating plainly rather than papering over: as of this writing, failure handling for the KV-transfer boundary specifically is the least-documented part of the three engines' disaggregation implementations, and Dynamo's own docs are the thinnest of the three on this exact point.

Table 6. What each engine's own documentation specifies about disaggregation failure handling, and what it does not234.
EngineDetectsDefaultNot documented (per these sources)
SGLangprefill liveness via decode-side heartbeat; unfinished bootstrap; unfinished transfer5.0 s interval, 2 failures, 300 s bootstrap/waiting timeoutexact per-request outcome (retry vs. fail) once prefill is marked offline mid-request
TensorRT-LLMworker-to-coordinator reachability at startup; NVLink-domain hangs; idle HTTP pools--server_start_timeout bounded retrycontext-server death mid-transfer for an in-flight request
Dynamoworker presence via discovery-service registration/deregistrationnot specifiedheartbeat interval; timeout values; mid-transfer partner-death response

The one resilience property that does carry across all three, stated most explicitly by Dynamo, is engine-level non-blocking transfer from section 4.1: whatever happens to one request's transfer, it does not stall the source GPU's other in-flight work4. That protects throughput for everyone else on the same worker even when a specific transfer is struggling; it does not, by itself, tell you what happens to the struggling request.

Where experts disagree

The three engines don't disagree about whether disaggregation should exist — Module 3 already made that case. They disagree, in their implementation choices if not in explicit prose, about whether the transfer window should be on or off a request's critical path. That's a real design trade-off with measurable consequences, not a documentation gap, so it earns the same treatment as a sourced controversy.

Overlap-first: let decode admission proceed during transfer

SGLang's bootstrap-handshake design3, described comparatively in NVIDIA Dynamo's backend documentation4

Reserve the decode-side KV slots and register the request before the transfer even starts, so the decode worker's scheduling and admission machinery runs in parallel with the byte movement instead of after it. This is what makes the heterogeneous-TP GPU staging buffer's 2–5× throughput gain reachable at all (section 3.1) — it requires prefill and decode to already be decoupled at the point the transfer begins.

  • Shrinks the portion of TTFT that is serially exposed to full transfer latency.
  • Makes staging-buffer-style optimizations reachable, since they need early decoupling between the two sides.
  • Documented, tunable failure detection: explicit heartbeat interval, failure-count threshold, and two separate 300 s timeouts (Table 5).
  • More moving parts to operate correctly: bootstrap room IDs, a background cleanup task, heartbeat tuning.
  • The staging-buffer path that benefits most from this design is explicitly incompatible with MLA models.
  • The sources reviewed do not specify the exact per-request outcome when a partner dies after bootstrap but before transfer completion (section 7.1).

Synchronous-first: gate decode admission on transfer completion

vLLM's default connector behavior via LookupBuffer.drop_select1; TensorRT-LLM's cache-transceiver2; both confirmed as the synchronous case in Dynamo's own backend comparison4

Don't let the decode worker touch a request until its KV cache has fully, correctly landed. TensorRT-LLM still keeps the source GPU busy by overlapping one request's transfer with a different request's compute (section 2.2), and vLLM's blocking drop_select call means a tensor is handed to exactly one consumer with no possibility of a partial or stale read.

  • Simpler correctness argument: no window where the decode worker could attend over an incomplete KV cache.
  • TensorRT-LLM's batch-level overlap recovers some of the GPU-utilization cost without touching per-request ordering.
  • Fewer distinct failure states to reason about, since there is no "admitted but not yet transferred" state to define behavior for.
  • The full transfer duration sits serially in front of the first decode step for every request, with no documented mitigation at the per-request level.
  • vLLM's own documentation concedes the whole disaggregated-prefill path "does not improve throughput" and remains labeled experimental1.
  • TensorRT-LLM's equivalent of xPyD — dynamic node joining/leaving — is explicitly still being built, not GA (section 6), suggesting the synchronous implementation is also the less operationally mature one on that specific axis.
Assessment

No source reviewed for this module publishes a controlled, head-to-head TTFT_P90 comparison between a synchronous connector (vLLM's NixlConnector or TensorRT-LLM's cache-transceiver) and SGLang's overlapped bootstrap path on identical hardware, model, and ISL:OSL distribution — that is the experiment that would actually settle which design wins by how much, and it does not appear to exist publicly as of 2026-09-17. Absent that number, the trade is structural rather than empirical: overlap buys latency at the cost of a wider failure-mode surface to operate (more timeouts, more state, more things that can be misconfigured), while synchronous transfer buys operational and correctness simplicity at the cost of paying the full transfer time on every request's critical path. For a deployment where Module 3's TTFT_P90 SLO is the binding constraint, the overlap design is worth its added complexity. For a first disaggregated deployment, or one where the transfer subsystem itself is still labeled experimental (as vLLM's is), starting with the synchronous default and only adopting the overlapped path once the operational failure modes are well understood is the more defensible sequencing — which is also, in effect, what each project's own maturity signals already suggest.

What is changing

All four primary sources for this module are live, versioned documentation rather than papers, which means the specific numbers in this module — timeout defaults, flag names, whether a capability is GA — are the most likely of anything in this course to drift within a year. Three trends to watch specifically: first, TensorRT-LLM's explicit "in progress" label on dynamic node joining/leaving (section 6) is a stated near-term roadmap item, built on infrastructure (the cache-transceiver and disagg_request_id scheme) that already exists for the static case — the gap with Dynamo's already-documented xPyD is likely to narrow, not persist. Second, failure handling for the mid-transfer partner-death case is the thinnest-documented mechanism across all three engines (section 7) — as disaggregation moves from early production deployments toward being the default topology for large models, expect this to get filled in with explicit retry/failover semantics rather than left to heartbeat-driven routing decisions alone. Third, the connector list itself is not fixed: vLLM's eight named connectors (Table 1) and SGLang's two transfer backends (Mooncake, NIXL) are the current state of two actively developed projects, and Modules 7 and 8 treat the transfer-engine layer underneath them in more depth, including the version churn that layer is going through.

References

  1. officialvLLM contributors. Disaggregated Prefilling. vLLM Documentation, docs.vllm.ai/en/latest/features/disagg_prefill/, package v0.29.0 (2026-09-09). https://docs.vllm.ai/en/latest/features/disagg_prefill/. Accessed 2026-09-17. Used for: Connector/LookupBuffer/Pipe abstraction, scheduler-vs-worker connector split, the eight named connectors and their backends, the "does not improve throughput" and experimental-status caveats.
  2. officialNVIDIA / TensorRT-LLM contributors. Disaggregated Serving (disagg-serving.md). TensorRT-LLM documentation, main branch. https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disagg-serving.md. Accessed 2026-09-17. Used for: KvCacheTransceiverV2 mechanism, disagg_request_id bit layout and binding rationale, NIXL/UCX/LIBFABRIC backend selection, GB200 NVLink-domain caveat, coordinator fail-fast behavior, dynamic node joining/leaving in-progress status.
  3. officialSGLang contributors. PD Disaggregation. SGLang Documentation, docs.sglang.io/advanced_features/pd_disaggregation.html. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: bootstrap handshake steps, room_id/host/port correlation, cleanup interval, heartbeat and timeout defaults, Mooncake/NIXL backend selection, heterogeneous-TP staging buffer mechanism and scope limits, sglang-router launch flags and stateless-design note, multi-node DeepSeek deployment flags.
  4. officialNVIDIA. Disaggregated Serving (NVIDIA Dynamo Documentation). docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. Accessed 2026-09-17. Used for: the generic three-stage pipeline framing, NIXL non-blocking VRAM-to-VRAM transfer, the backend-specific transfer-metadata/execution-model comparison table, PrefillRouter selection criteria, xPyD worker add/remove mechanism.