Multi-Node LLM Inference: From One Node to N · Module 11

vLLM Multi-Node: Distributed Serving, Disaggregated Prefill, and the Production Stack

vLLM's own account of scaling past one node: the KVTransferConfig surface that every disaggregation connector plugs into, the CLI flags that turn a data-parallel-size number into a running multi-node cluster, and the reference router-plus-observability stack the project ships for production. After this module you can read a vLLM multi-node deployment's flags and configs and know exactly what topology they produce.

Course 30 min read 7 sources As of 2026-09-17 Prerequisite: Module 4, Module 5
vLLM multi-node deployment: router, prefill pool, decode pool, connector, observability client HTTP request router production-stack (§4) prefill pool node 1 DP+EP node 2 DP+EP kv_role: kv_producer NixlConnector / Mooncake / LMCache decode pool node 1 DP+EP node 2 DP+EP kv_role: kv_consumer same connector family KV transfer — §1, §3 Prometheus scrapes both pools — §5 kv_cache_usage_perc, TTFT, ITL histograms Each pool's internal DP+EP layout follows Module 5's EP_SIZE = TP_SIZE × DP_SIZE arithmetic; this module covers what wraps around it.
Figure 1. The shape this module fills in: a router in front of two independently-scaled pools, each internally a multi-node DP+EP vLLM deployment, joined by a KV connector, both observed through one Prometheus surface123.

1. vLLM's disaggregated prefill: status and the KVTransferConfig surface

Module 4 already built the abstraction vLLM disaggregation runs on — a Connector that lets a kv-consumer worker pull KV caches from a kv-producer, a LookupBuffer with insert/drop_select, and a Pipe that moves tensors — and its table of eight named connectors (ExampleConnector, LMCacheConnectorV1, NixlConnector, MooncakeConnector, MoRIIOConnector, MultiConnector, OffloadingConnector, FlexKVConnectorV1) and what transport each backs onto. That does not repeat here. What matters for a multi-node vLLM deployment is the surface you actually configure: a single JSON blob, --kv-transfer-config, passed to every vllm serve invocation1.

1.1 The KVTransferConfig JSON schema

vLLM's own documentation gives three worked configurations, one per connector family. All three share the same top-level fields — kv_connector (which class implements the connector), kv_role (whether this instance produces KV, consumes it, or both), and kv_connector_extra_config (a connector-specific nested object) — with an optional kv_buffer_device when the connector needs to know which memory tier to stage through1:

--kv-transfer-config '{
  "kv_connector": "NixlConnector",
  "kv_role": "kv_both",
  "kv_buffer_device": "cuda",
  "kv_connector_extra_config": {"backends": ["UCX", "GDS"]}
}'
--kv-transfer-config '{
  "kv_connector": "MultiConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {"connectors": ["..."]}
}'
--kv-transfer-config '{
  "kv_connector": "OffloadingConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {"block_size": 64, "cpu_bytes_to_use": 1000000000}
}'

The kv_connector_extra_config object is where each connector's own tuning surface lives, and the three examples above show how differently that surface is shaped. NIXL's backends list is an ordered preference — ["UCX", "GDS"] tells the connector to prefer UCX for host/device transfer and fall back to GPUDirect Storage, the same two actively-supported NIXL backends Module 8 covers at the transport-library level1. OffloadingConnector's block_size and cpu_bytes_to_use fields are a capacity budget, not a transport choice: 64 tokens per block and a fixed byte ceiling for how much host RAM the offload path is allowed to claim, the same knob Module 8 covers on LMCache's side of the offload story but set here at the raw-connector level rather than through LMCache's policy layer. MultiConnector's connectors field is the odd one out structurally — it does not configure a transport at all, it nests other connectors' own configuration objects inside it, which is what lets one engine run, for example, an OffloadingConnector for local CPU spillover and a NixlConnector for cross-pool transfer at the same time, each keyed by its own role.

kv_role is the field that turns a generic connector into a disaggregation half: kv_producer on a prefill instance means "compute KV, then hand it off and drop it"; kv_consumer on a decode instance means "pull KV before generating"; kv_both is what a single colocated engine or an offloading-only setup uses, since it is neither strictly producing for, nor strictly consuming from, a remote peer1. Setting kv_role is the whole difference, at the configuration level, between "this engine offloads its own KV cache to CPU RAM for later reuse" (OffloadingConnector, kv_both) and "this engine is one half of a disaggregated pair" (NixlConnector, kv_producer paired with a second instance running kv_consumer). Section 3 uses exactly this distinction to compose prefill and decode pools.

1.2 The connector roster: nine, not eight

vLLM's disaggregated-prefill documentation states, as of the version checked for this module, that it "now supports 9 types of connectors"1. Module 4's table, built from the same documentation, enumerated eight; the ninth is PyNcclConnector, named in vLLM's own connector listing alongside the eight already covered1. The documentation does not give PyNcclConnector the same one-line gloss it gives the others, so treat its presence as confirmed and its mechanism as unverified beyond the name — a small, telling data point on its own about how fast this part of the stack is still moving (more in "What is changing").

Table 1 restates the roster with a practical lens: not "what does each back onto" (Module 4's framing) but "which one do you reach for, and why."

Table 1. vLLM's KV-connector roster and the deployment situation each fits1.
ConnectorUse it when…kv_role typically set to
NixlConnectorstanding up disaggregated prefill/decode with the lowest-friction, project-maintained transportkv_producer / kv_consumer
MooncakeConnectorthe cluster already runs Mooncake's transfer engine and store (Module 7)kv_producer / kv_consumer
LMCacheConnectorV1you want KV reuse/offload policy (chunking, tiered backends) on top of NIXL, not just raw transfer (Module 8)kv_both, or kv_producer/kv_consumer when composed with disaggregation
OffloadingConnectora single colocated engine needs CPU-RAM spillover, no second pool involvedkv_both
MultiConnectoryou need more than one of the above active at once (e.g. NIXL for cross-pool transfer plus CPU offload for local spillover)kv_both, delegating internally
MoRIIOConnectorthe cluster is ROCm, not CUDA — the AMD-side disaggregation pathkv_producer / kv_consumer
FlexKVConnectorV1KV needs to live behind a distributed, multi-level cache store rather than a point-to-point transferkv_producer / kv_consumer
PyNcclConnectornamed in the current roster; mechanism not documented in the source checked—
ExampleConnectorreading or extending the interface itself, not production traffic—

Two entries in Table 1 are worth a second look because they sit a level above the others: MultiConnector and FlexKVConnectorV1 are not point-to-point transports in the way NixlConnector or MooncakeConnector are, they are composition and indirection layers built on top of the same interface. FlexKVConnectorV1 backs onto a distributed, multi-level KV store rather than a direct producer-consumer link1 — conceptually closer to LMCache's tiered-backend model (Module 8) than to a raw transfer connector, though it is vLLM's own connector implementation rather than a wrapper around LMCache's. Which of LMCacheConnectorV1 and FlexKVConnectorV1 a deployment reaches for is mostly a question of which reuse/store ecosystem it has already standardized on; both solve the same underlying problem of KV living somewhere other than a single engine's HBM.

2. DP+EP across nodes: from the formula to a running cluster

Module 5 already derived the arithmetic (EP_SIZE = TP_SIZE × DP_SIZE), worked the TP=2/DP=4/8-GPU single-node layout, and walked vLLM's own two-node example — --data-parallel-size 16, --data-parallel-size-local 8, --data-parallel-start-rank 8 on the second node — down to the rank numbers2. That arithmetic does not need repeating. What Module 5 left aside is what those same two vllm serve invocations need beyond the DP/EP ranks to actually find each other, agree on a network endpoint, and decide which process talks to clients: the process- and network-level flags. That is this section's job.

2.1 The flags that turn a rank layout into a cluster

vLLM's expert-parallel deployment guide's own two-node example carries four flags that Module 5's arithmetic discussion did not need to unpack2:

Table 2. Process- and network-level flags in vLLM's two-node DP+EP example, beyond the rank arithmetic Module 5 already covers2.
FlagSet onWhat it does
--data-parallel-addressboth nodes, same valueIP of the primary node; every node dials this address to join the DP group
--data-parallel-rpc-portboth nodes, same valueRPC port for inter-node coordination; any port reachable from every node
--api-server-countprimary node onlynumber of API-server processes fronting the primary node for client-request load handling
--headlesssecondary nodes onlyno API server on this process — it runs worker ranks only and accepts no client traffic

Put together with Module 5's rank arithmetic, the full picture for the two-node, DP=16, TP=1, EP=16 example is: node 1 opens an RPC listener on --data-parallel-rpc-port, runs 8 API-server processes (--api-server-count=8) to absorb client concurrency, and hosts local DP ranks 0–7; node 2 dials node 1's address and port, contributes local ranks 0–7 which become global ranks 8–15 via --data-parallel-start-rank 8, and runs --headless so it never accepts a client connection directly2. Only one node in the pair is ever a client-facing endpoint; every other node is a worker that the primary coordinates over the RPC port. This is the same shape Section 4's production-stack router sits in front of — the router never needs to know a decode pool spans two nodes and one of them is headless; it only sees the primary node's API-server endpoint.

Two-node DP+EP process and network topology: primary node with API servers and RPC listener, headless secondary node node 1 — primary 8× API server RPC listener :13345 local DP ranks 0–7 --data-parallel-start-rank 0 (default) EP group spans both nodes (§2.1) node 2 — headless no API server (--headless) local DP ranks 0–7 --data-parallel-start-rank 8 → global ranks 8–15 dials --data-parallel-address 192.168.1.100 Only node 1 is ever client-facing; node 2 is a pure worker discovered over the RPC port.
Figure 2. The process and network layer wrapped around Module 5's rank arithmetic for the same two-node, DP=16 example: node 2 dials node 1's --data-parallel-address/--data-parallel-rpc-port, and only node 1 runs an API server2.

The documentation does not give a closed-form rule for more than two nodes, but the two-node example generalizes directly: for node i of N uniform nodes, each hosting data-parallel-size-local ranks, the start rank is the cumulative local size of every earlier node — "equal to the cumulative local DP size of previous nodes," in the documentation's own words2:

$$ \text{start-rank}(i) = \sum_{j=0}^{i-1} \text{data-parallel-size-local}(j) $$

For the uniform case every node runs, this collapses to start-rank(i) = i × data-parallel-size-local, and the global size is data-parallel-size = N × data-parallel-size-local. Only node 0 carries --api-server-count and omits --headless; every other node carries --headless and its own start-rank.

N-node DP+EP sizing (uniform nodes, from vLLM's flags2)

Total DP ranks (--data-parallel-size)
EP_SIZE (Module 5: TP × DP)
Total GPUs
Last node's start-rank

GPU count assumes one GPU per (TP-rank × DP-rank) slot, the layout vLLM's expert-parallel guide uses throughout2. At the default TP=1, EP_SIZE equals total DP ranks, matching the DeepSeek-V3 two-node example (N=2, dpl=8 → DP=16, EP=16).

Push the same arithmetic past two nodes and the pattern that matters is which node carries which start-rank, not just the totals. Take four nodes, each with --data-parallel-size-local 4 and --tensor-parallel-size 1: the global --data-parallel-size is 4 × 4 = 16, matching the calculator's default DP total but now spread over twice as many, smaller nodes. Node 0 (the only non-headless one) still takes the default start-rank of 0; nodes 1, 2, and 3 take start-ranks 4, 8, and 12 respectively — each one data-parallel-size-local higher than the previous node's, exactly as the cumulative-sum formula requires. EP_SIZE is unchanged at 16 (TP=1 × DP=16), so four smaller nodes and two larger ones produce an identical EP group size and an identical MoE shard width; the only thing that differs is how many separate RPC-connected processes the primary node's rank-0 process has to track, and how much per-node blast radius a single node failure has — a quarter of the DP group instead of half.

2.2 What actually changes crossing from one node to two

vLLM's single-node H200 example for the same DeepSeek-V3 model is four flags: --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel, nothing else2. No address, no RPC port, no headless flag, no start-rank — because a single process owns every rank and there is no second process to coordinate with. The two-node version does not change the parallelism strategy at all (still TP=1, still attention-DP + full-width EP); it only adds the four coordination flags from Table 2 and splits --data-parallel-size 8 into --data-parallel-size 16 spread across two --data-parallel-size-local 8 processes. The mental model worth keeping: going multi-node in vLLM's DP+EP path is not a different parallelism scheme, it is the same scheme plus a discovery/coordination layer bolted on at the process level.

3. Composing disaggregation with DP+EP: two independently-scaled clusters

Sections 1 and 2 are separable knobs — which connector moves KV, and how many nodes and ranks a pool spans — and vLLM lets you turn both at once. A DeepSeek-class deployment with disaggregated prefill and decode is, mechanically, two separate multi-node DP+EP clusters (each sized by Section 2's arithmetic, independently) joined by one kv_producer/kv_consumer connector pair from Section 1. Nothing in vLLM's own configuration surface treats "disaggregated deployment" as a single object with one set of flags; a prefill pool and a decode pool are two entirely independent invocations of everything covered so far, and the only place they are coupled is the connector metadata each one's --kv-transfer-config points at. That has a direct operational consequence: the prefill pool can be resized, restarted, or moved to different hardware without touching the decode pool's DP+EP flags at all, and vice versa, as long as the connector's endpoint metadata stays reachable — the same independent-scaling property Module 3's goodput argument for disaggregation assumes at the architectural level, here visible at the level of two vllm serve command lines that share nothing but a connector role.

3.1 kv_producer meets kv_consumer

The prefill pool's engines start with kv_role: kv_producer in their --kv-transfer-config; the decode pool's engines start with kv_role: kv_consumer1. Both point at the same connector class — typically NixlConnector for a fresh disaggregation setup, or MooncakeConnector if the cluster already runs Mooncake's transfer engine (Module 7) — so the wire format and metadata handshake match on both sides. Nothing forces the two pools to share a parallelism configuration: the prefill pool can run a smaller EP degree tuned for its compute-bound, larger-batch workload while the decode pool runs a wider EP degree tuned for its memory-bound, latency-sensitive workload, exactly the hierarchical-vs-global EPLB policy split Module 5 covers for placement, applied here at the level of two entirely separate vLLM clusters rather than one.

This is also where the KVTransferConfig fields interact with the DP+EP flags rather than sitting beside them: each pool independently sets its own --data-parallel-size, --data-parallel-size-local, and --data-parallel-start-rank family (Section 2), and independently sets its own --kv-transfer-config (Section 1) with the role appropriate to its position in the pipeline. A concrete four-node layout — two nodes of prefill, two nodes of decode — runs four separate vllm serve processes, each with two independent flag families that never need to agree with each other except on the connector's own metadata protocol:

vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 --enable-expert-parallel \
    --data-parallel-size 8 --data-parallel-size-local 4 \
    --data-parallel-address 10.0.0.10 --data-parallel-rpc-port 13345 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 --enable-expert-parallel \
    --data-parallel-size 16 --data-parallel-size-local 8 \
    --data-parallel-address 10.0.1.10 --data-parallel-rpc-port 13345 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

The two DP totals here — 8 for prefill, 16 for decode, each on its own independent two-node group with its own --data-parallel-address — are a synthesis of Sections 1 and 2's flags, built to illustrate the composition rather than quoted from a single documented example; the individual flags and their semantics are each sourced above. The asymmetry (EP=8 for the compute-bound, larger-batch prefill pool; EP=16 for the memory-bound, latency-sensitive decode pool) mirrors the hierarchical-vs-global EPLB placement split Module 5 covers: a narrower EP group with more group-local traffic for prefill, a wider one that spreads load further for decode.

Disaggregated prefill and decode as two independently-sized DP+EP clusters joined by one connector pair prefill pool — 2 nodes node 1 DP-local 4 node 2 DP-local 4 DP=8, EP=8 (TP=1) kv_role: kv_producer smaller EP — hierarchical policy decode pool — 2 nodes node 1 DP-local 8 node 2 DP-local 8 DP=16, EP=16 (TP=1) kv_role: kv_consumer wider EP — global policy NixlConnector Each pool's DP+EP sizing follows §2 independently; only kv_role and the shared connector class link them.
Figure 3. Four physical nodes split into two independently-sized DP+EP clusters — a narrower EP group for prefill, a wider one for decode — joined by a single NixlConnector kv_producer/kv_consumer pair12.

3.2 What is production-ready and what is not

vLLM's documentation is direct about where this stands. The disaggregated-prefill feature as a whole carries an explicit banner: "this feature is experimental and subject to change"1, and the docs state plainly that disaggregation "does not improve throughput"1 — consistent with Module 3's framing that the payoff is goodput under an SLO, not raw tokens/second. DP+EP for MoE models sits on firmer ground: it is the documented, default path for serving large MoE models like DeepSeek-V3 across one or many nodes, with a stable CLI surface (--tensor-parallel-size, --data-parallel-size, --enable-expert-parallel) that the expert-parallel deployment guide treats as production guidance, not an experimental preview2.

Table 3. Production-readiness signals vLLM's own documentation gives for each piece of the multi-node stack, as of the version checked12.
ComponentStatus per vLLM's own docs
Continuous batching, chunked prefill (single engine)default production path
DP+EP for MoE, single or multi-nodedocumented deployment guide, stable CLI surface
EPLB rebalancingdocumented, configurable, default off (num_redundant_experts=0)
Disaggregated prefill (any connector)explicitly labeled experimental, subject to change
PyNcclConnectornamed in the roster; not documented in the text this module drew from

The practical reading: a team standing up a first multi-node vLLM deployment gets a stable, documented path to DP+EP for a wide MoE model with no experimental-status caveat attached. The same team adding disaggregated prefill on top is opting into a feature vLLM's own maintainers have not yet called stable — not a reason to avoid it, since Module 3's goodput numbers and Module 6's DeepSeek production numbers both come from disaggregated deployments, but a reason to expect connector-level API churn (Section 1.2's PyNcclConnector appearing without documentation is one symptom of that churn) and to budget engineering time for it accordingly.

4. The vLLM production-stack reference deployment

Everything above configures individual vllm serve processes. vllm-project/production-stack, released by the vLLM project itself on 2026-01-223, is the reference answer to the next question: how do those processes become a deployment a client actually talks to. Its own framing is "scale from a single vLLM instance to a distributed vLLM deployment without changing any application code"3 — the stack sits entirely outside the model-serving path.

4.1 Three components

The stack has three parts3:

  • Serving engine. The vLLM engines running the actual model(s) — any of the topologies from Sections 1–3, one pool or several.
  • Request router. Sits in front of the serving engines and "directs requests to appropriate backends based on routing keys or session IDs to maximize KV cache reuse"3.
  • Observability stack. Prometheus scrapes the metrics surface (Section 5), Grafana renders it3.
vLLM production-stack architecture: router, serving-engine pods, Prometheus/Grafana client OpenAI API router round-robin session-ID prefix-aware (WIP) serving engine pod 1 serving engine pod 2 serving engine pod N Prometheus + Grafana §5 metrics Deployed as a Helm chart on Kubernetes; each pod's own topology follows §1–3.
Figure 4. The vLLM production-stack's three components: a router with three selectable routing modes in front of N serving-engine pods, all scraped by one Prometheus/Grafana pair3.

4.2 Router modes

The router documents three routing behaviors: round-robin, session-ID-based routing, and prefix-aware routing, the last explicitly marked work-in-progress3. Round-robin and session-ID routing are the two production-ready modes; prefix-aware routing — the KV-cache-overlap-aware scoring Module 2 covers in depth for llm-d's scheduler — is on the production-stack's own roadmap as a future capability, alongside a documented but not-yet-implemented "KV-cache-aware routing algorithm" and improved fault tolerance3. In addition to these routing behaviors, the router supports automatic Kubernetes service discovery, so serving-engine pods can be added or removed from a Deployment without a router restart or manual endpoint list3. Autoscaling driven by vLLM-specific metrics (Section 5's surface) is also roadmap, not shipped3.

Read against Module 2's router taxonomy, the production-stack router sits at the simpler end: it is the "approximate, cheap, no synchronous global state" style of routing taken to its simplest case — round-robin is not even prefix-aware — rather than the KVEvents-driven precise indexing llm-d's EPP implements. That is a deliberate scope difference, not an oversight: production-stack is vLLM's own minimal reference deployment, meant to demonstrate the serving-engine/router/observability shape work out of the box, not to compete with llm-d or Dynamo's routing sophistication.

Session-ID routing is the mode that matters most in practice among the two shipped today, because it is the cheapest way to get most of prefix-cache reuse's benefit without building a KV-index at all: pinning every request that carries the same session identifier to the same backend pod means a multi-turn conversation's growing prompt keeps landing on the one engine that already has its prefix cached, at the cost of no longer load-balancing within a session. Round-robin gives up that property entirely in exchange for perfectly even load distribution across the pool — the right default for single-turn, cache-indifferent traffic, and the wrong one for a chat workload with long multi-turn sessions.

4.3 Kubernetes deployment shape

Production-stack ships as a Helm chart, configured through a values.yaml that declares the serving-engine replica set(s), router, and observability components together3; its own tutorial set includes a worked example for enabling LMCache-backed KV offload (Module 8) directly from the chart3. What the chart does not itself provide is the multi-host sharding primitive a single logical replica needs when Section 2's DP+EP layout, or a plain tensor-parallel shard, spans more than one physical node: that is LeaderWorkerSet (LWS), the Kubernetes SIG API "for deploying a group of pods as a unit of replication," co-designed alongside llm-d specifically to address "multi-host inference workloads where the LLM will be sharded and run across multiple devices on multiple nodes"4. vLLM's own multi-node deployment guides assume an LWS-shaped substrate underneath a production-stack-style router; the full mechanism — leader/worker pod roles, rolling upgrades of a sharded unit, drain-before-remove — is Module 14's subject. Concretely, a two-node DP+EP pool from Section 2 (one primary, one headless) is a natural fit for one LWS "replica": the leader pod runs the primary node's flags and carries the API server, the worker pod runs the headless secondary's flags, and LWS is what keeps the pair scheduled, restarted, and drained together as a unit rather than as two independent Kubernetes pods that happen to reference each other's IP addresses.

4.4 Where production-stack sits relative to Dynamo and llm-d

Production-stack, Dynamo (Module 9), and llm-d (Module 10) all put a router in front of a fleet of vLLM engines, and it is worth being precise about why a deployment would pick one over another rather than treating them as interchangeable. Production-stack's own documented feature set stops at round-robin, session-ID routing, and Kubernetes service discovery, with prefix-aware and KV-cache-aware routing still on its roadmap and autoscaling not yet implemented3. Dynamo's Router does KV-aware routing today and ships an SLA-driven Planner that actively resizes prefill and decode pools against TTFT/ITL targets. llm-d's scheduler implements the same KV-cache-aware scoring behind the standardized Gateway API Inference Extension contract, with both approximate block-hash and precise KVEvents-based indexing modes. None of that makes production-stack the wrong choice: it is vLLM's own minimal, dependency-light reference implementation of "router in front of engines, metrics out the back," useful precisely because it does not require adopting Dynamo's or llm-d's broader component set to get a working multi-engine deployment off the ground. A team that later needs KV-cache-aware routing or SLA-driven autoscaling is not extending production-stack to get it; it is swapping the router layer for Dynamo's or llm-d's, while the serving-engine flags from Sections 1–3 stay the same either way.

5. vLLM's Prometheus metric surface

Everything upstream — the production-stack's roadmapped autoscaling, Module 9's Dynamo Planner, Module 2's load-aware scorers — ultimately reads the same substrate: the Prometheus metrics vLLM's v1 engine emits per instance5. vLLM does not group these formally in its documentation, but they split cleanly into three practical buckets: gauges, counters, and histograms.

Table 4. vLLM v1 Prometheus metrics by type, with the ones an SLO dashboard or autoscaler actually reads called out5.
TypeMetricWhat it reads
Gaugevllm:num_requests_runningcurrently-executing requests — direct load signal
Gaugevllm:kv_cache_usage_percfraction of used KV cache blocks, 0–1 — the headroom signal every scheduler and autoscaler cares about
Gaugevllm:lora_requests_infoper-adapter LoRA request counts
Gaugevllm:cache_config_infostatic cache configuration, for dashboards not alerts
Countervllm:prompt_tokens_total / vllm:generation_tokens_totalcumulative tokens processed — the raw input to tokens/s/$ arithmetic (Module 15)
Countervllm:prefix_cache_queries / vllm:prefix_cache_hitsprefix-cache hit rate, comparable to Mooncake's own hit-ratio curve (Module 7)
Countervllm:request_success_totalfinished requests, broken down by completion reason
Histogramvllm:time_to_first_token_secondsTTFT distribution — the metric Section 4.2's SLO framing and Module 9's Planner default (500 ms) target directly
Histogramvllm:inter_token_latency_secondstime between streamed tokens — ITL, the Planner's other default target (50 ms)
Histogramvllm:request_time_per_output_token_secondsper-request TPOT, the Module 15 formula applied at the metric level
Histogramvllm:e2e_request_latency_secondsfull request duration, prefill through last token
Histogramvllm:request_prefill_time_seconds / vllm:request_decode_time_secondsphase split — directly useful for sizing a disaggregated prefill:decode ratio (Module 3)
Histogramvllm:kv_block_lifetime_seconds, kv_block_idle_before_evict_seconds, kv_block_reuse_gap_secondsKV block-level churn — how long blocks live, sit idle, and get reused before eviction

Two things are worth reading past the raw list. First, kv_cache_usage_perc and num_requests_running are the two gauges that actually gate operational decisions — a router or autoscaler with access only to these two can implement most of the load-aware scoring Module 2 describes without needing the KVEvents stream llm-d's precise indexing subscribes to. Second, vLLM's own design documentation explicitly flags several v0-era metrics as deprecated or dead weight in v1: vllm:num_requests_swapped and vllm:cpu_cache_usage_perc belonged to a CPU-swapping mode v1 removed; vllm:time_in_queue_requests duplicates queue-time information now carried elsewhere and is marked for deprecation; vllm:tokens_total was never actually implemented despite being named; vllm:avg_prompt_throughput_toks_per_s is deprecated with no direct v1 replacement5. A dashboard built against an older vLLM metrics reference will alert on gauges that no longer move.

Building a minimal SLO dashboard for a two-pool disaggregated deployment from this surface takes four series, all sourced from Table 4: vllm:time_to_first_token_seconds scraped from the prefill pool's endpoints, since TTFT is set almost entirely by prefill queueing and compute time; vllm:inter_token_latency_seconds scraped from the decode pool, since ITL is a decode-loop property; vllm:kv_cache_usage_perc on both pools, as the earliest warning signal before either one starts queueing; and vllm:request_prefill_time_seconds / vllm:request_decode_time_seconds together, to see the phase split directly rather than infer it from the end-to-end histogram. Everything else in Table 4 — the LoRA gauge, the static cache-config gauge, the block-lifetime histograms — is diagnostic detail for debugging a specific incident, not something worth an alert threshold on its own.

Where experts disagree

Module 3 already covers the goodput-theoretic case for disaggregation in general. The narrower question this module's sources put in direct tension is whether vLLM's own implementation choices endorse that case as a default, or treat it as a specialized add-on most deployments should skip.

Disaggregation is close to necessary once interference is counted

DistServe (OSDI 2024)6; Mooncake (FAST 2025)7

DistServe measures colocated per-GPU goodput on a single A100 at roughly 1.6 requests/second against 5.6 rps prefill-only and 10 rps decode-only when the phases are separated6 — a >3x goodput gap attributed directly to prefill-decode interference on one engine. Mooncake's authors explicitly revisited whether chunked prefill made this unnecessary and decided against it, keeping their disaggregated architecture for cross-node long-context parallelism and VRAM savings even after chunked prefill became standard7. Under this view, a colocated engine is a compromise a serious deployment eventually moves past.

  • Grounded in controlled, measured goodput numbers on real hardware, not vendor claims
  • Mooncake's reconsideration is evidence the conclusion survived a later technology change (chunked prefill), not just an artifact of when the paper was written
  • Both results come from systems built specifically to prove disaggregation works well; neither measures the operational cost of running two pools instead of one
  • DistServe's numbers are from A100-era hardware and a 13B model — Module 3 covers how the crossover shifts on newer hardware

vLLM's own default path is still one colocated engine

vLLM's disaggregated-prefill documentation1

vLLM ships continuous batching and chunked prefill as the default, unqualified production path. Disaggregated prefill is offered as an opt-in feature explicitly labeled experimental, and the documentation itself says the feature "does not improve throughput"1 — language a project does not attach to something it wants as the default recommendation. Nine named connectors exist, one of them (PyNcclConnector) not even documented in prose, which reads as a surface still being actively reshaped rather than settled.

  • Reflects what the maintainers of the most widely deployed open-source engine are actually willing to call stable, today
  • Correctly separates "does this help goodput under an SLO" from "is this safe to run in production without deep familiarity with its failure modes"
  • Documentation status is a maturity signal, not a technical argument against the mechanism — it says nothing about whether disaggregation pays off, only whether vLLM's implementation of it is battle-tested
  • Module 6's DeepSeek numbers and Module 3's TCO crossover both come from disaggregated production systems, so "experimental" clearly has not stopped large-scale real deployment elsewhere
Assessment

These are not actually competing claims about the same question. DistServe and Mooncake answer "does the mechanism pay off" and vLLM's docs answer "is this specific implementation safe to run without deep operational familiarity" — Module 3's interactivity-dependent TCO crossover already establishes that the first answer is conditional, not universal, so the two positions are compatible: disaggregation pays off above a workload-shape-dependent bar, and vLLM's own connector layer is young enough that adopting it below that bar, or without the SLO pressure to justify the added complexity, is not yet obviously worth it. The evidence that would sharpen this further is a change-log-level maturity signal from vLLM itself — the disaggregated-prefill feature losing its experimental label, or the connector roster stabilizing for several releases in a row — neither of which had happened as of the version checked for this module (package 0.29.0, 2026-09-091).

What is changing

The connector roster grew from the eight Module 4 catalogued to nine in the same documentation pass, with the ninth (PyNcclConnector) named but not yet described in prose1 — treat the exact count and the exact set of names as likely to move within the year. The production-stack router's prefix-aware mode is explicitly work-in-progress, and KV-cache-aware routing and autoscaling are both roadmap items rather than shipped behavior3; a deployment that needs those today has to reach for llm-d (Module 10) or Dynamo (Module 9) instead of production-stack. EPLB's configuration surface has already grown past the two parameters Module 5 documents2, which is itself a signal that the MoE deployment path, despite being the more "stable" of the two features compared in this module's disagreement section, is still under active development. vLLM's own package versioning — 0.29.0 as of 2026-09-091 — moves on a roughly monthly cadence, so any specific flag name or default value here should be re-checked against the installed version before being relied on operationally.

References

  1. officialvLLM contributors. Disaggregated Prefilling. vLLM Documentation, docs.vllm.ai/en/latest/features/disagg_prefill/, package v0.29.0 (2026-09-09). https://docs.vllm.ai/en/latest/features/disagg_prefill/. Accessed 2026-09-17. Used for: KVTransferConfig JSON schema and field-level examples (NixlConnector, MultiConnector, OffloadingConnector), kv_role semantics (kv_producer/kv_consumer/kv_both), the 9-connector roster including PyNcclConnector, the experimental-status banner and "does not improve throughput" statement.
  2. officialvLLM Project. Expert Parallel Deployment. vLLM Documentation, docs.vllm.ai/en/latest/serving/expert_parallel_deployment/. https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/. Accessed 2026-09-17. Used for: two-node DP+EP flag semantics (--data-parallel-address, --data-parallel-rpc-port, --api-server-count, --headless), single-node vs. two-node example contrast, start-rank generalization to N nodes, EPLB extra configuration fields (num_redundant_experts, use_async, log_balancedness).
  3. repovLLM Project contributors. vllm-project/production-stack (README). Released 2026-01-22. https://github.com/vllm-project/production-stack. Accessed 2026-09-17. Used for: the three-component architecture (serving engine, router, observability), router modes (round-robin, session-ID, prefix-aware WIP, Kubernetes service discovery), Helm chart / values.yaml deployment shape, LMCache-offload tutorial, roadmap items (KV-cache-aware routing, autoscaling).
  4. repoKubernetes SIG contributors. kubernetes-sigs/lws (LeaderWorkerSet README). Main branch. https://github.com/kubernetes-sigs/lws/blob/main/README.md. Accessed 2026-09-17. Used for: LWS as the multi-host sharded-replica substrate under vLLM's multi-node deployment guides, co-design with llm-d.
  5. officialvLLM contributors. Metrics. vLLM Design Documentation, docs.vllm.ai/en/latest/design/metrics/ (v1 engine). https://docs.vllm.ai/en/latest/design/metrics/. Accessed 2026-09-17. Used for: full gauge/counter/histogram metric names and descriptions, deprecated v0 metrics list.
  6. paperZhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., Zhang, H. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. USENIX OSDI 2024. arXiv:2401.09670v3 (2024-06-06). https://arxiv.org/pdf/2401.09670. Accessed 2026-09-17. Used for: per-GPU goodput numbers (1.6 rps colocated vs. 5.6/10 rps prefill/decode split on A100).
  7. paperQin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. USENIX FAST 2025 (Best Paper). arXiv:2407.00079v4 (2025-09-03). https://arxiv.org/pdf/2407.00079. Accessed 2026-09-17. Used for: the authors' explicit reconsideration of disaggregation's necessity after chunked prefill, and their reasons for keeping it (§5).