Multi-Node LLM Inference: From One Node to N · Module 9

NVIDIA Dynamo: Router, Planner, KV Block Manager, and Disaggregation Orchestration

Dynamo is not a fourth inference engine sitting next to vLLM, SGLang, and TensorRT-LLM — it is a control plane that sits above all three, making the placement, autoscaling, and KV-lifecycle decisions those engines don't make for themselves. This module works through Dynamo's five named components (Router, Planner, KVBM, ModelExpress, Grove) mechanism by mechanism, contrasts each against the generic routing, KV-transfer, and tiering patterns Modules 2–8 already established, and states plainly — from NVIDIA's own release notes, adoption list, and documented version churn — how production-grade the stack is as of v1.4.2. After reading, you can place any Dynamo deployment flag or metric in its correct architectural layer.

Course 35 min read 16 sources As of 2026-09-17 Prerequisite: Module 2 Prerequisite: Module 4
Dynamo's five named components sitting above three backend inference engines, with NIXL as the shared transfer substrate underneath Dynamo — orchestration layer Router KV-aware placement Planner SLA autoscaler KVBM GPU→CPU→SSD→remote ModelExpress GPU→GPU weights Grove K8s gang scheduling Backend inference engines vLLM · SGLang · TensorRT-LLM (Dynamo runs and coordinates these, it does not replace them) NIXL point-to-point transfer substrate (Module 8) — KVBM, ModelExpress, and disaggregated KV transfer all move bytes through it NVLink · InfiniBand/RoCE · GPUDirect RDMA · NVMe-oF (Module 1)
Figure 1. Dynamo's five named components run as a control plane above the three engines it supports as backends, coordinating them through NIXL rather than reimplementing KV-cache management, batching, or attention kernels inside itself. Component list and backend list per the project README1.

1. Dynamo's position: orchestration layer, not an engine

Every mechanism in Modules 2–8 — prefix-cache-aware routing, prefill/decode connectors, wide-EP dispatch, tiered KV offload, NIXL transfer — is something a serving engine (vLLM, SGLang, TensorRT-LLM) or a companion project (Mooncake, LMCache) implements at the level of one replica's forward pass or one pod's KV cache. Dynamo's own framing draws a hard line above that level: its README states plainly that it is "the orchestration layer above inference engines"1. It does not run attention kernels, does not own a paged-KV allocator, and does not implement continuous batching — those stay inside whichever of vLLM, SGLang, or TensorRT-LLM is running as the backend1. What Dynamo adds is everything above that: which replica a request goes to, how many replicas of each pool should exist right now, where a cold KV block should live when GPU memory runs out, how a new replica gets its weights without hitting shared storage, and how a multi-node PodClique gets scheduled onto Kubernetes hardware as one atomic unit. Each of these is a decision Modules 1–8 already framed generically; this module is about Dynamo's specific, named implementation of each one.

1.1 The five named components at a glance

Figure 1 lays out the shape: five components sit above the three supported backends, and four of the five (Router, KVBM, ModelExpress, and the disaggregation path in §6) move bytes or metadata through NIXL, the point-to-point transfer library Module 8 already covered in full. Grove is the exception — it is Kubernetes-side scheduling, not a runtime data-plane component.

Router
KV-aware request routing: worker load + KV cache overlap1
Planner
SLA-driven autoscaler: profiles workloads, right-sizes pools1
KVBM
Offloads KV cache GPU → CPU → SSD → remote storage1
ModelExpress
Streams model weights GPU-to-GPU via NIXL/NVLink1
Grove
Kubernetes operator for topology-aware gang scheduling1

Backends: vLLM, SGLang, TensorRT-LLM1. A Dynamo deployment is, mechanically, one or more of these engines running as workers, fronted by a Dynamo frontend process, with Router, Planner, and KVBM as separate control-plane services talking to the workers over the same distributed runtime.

2. The Router: KV-aware placement as a cost function, not a scorer chain

Module 2 built two mechanisms for guessing where a prefix is cached — an approximate block-hash-plus-LRU-index path and a precise KVEvents-over-ZeroMQ path — then combined that signal with load through a weighted scorer set (prefix-cache-scorer, inflight-load-producer, and a weighted sum decided by max-score-picker). Dynamo's Router solves the identical problem — which worker minimizes recomputed prefill and queueing delay for this request — but expresses it as a single cost function evaluated per candidate worker, rather than a pluggable filter/score/select pipeline.

2.1 The cost function: overlap score against projected load

The Router's documented cost function is a logit combining two terms: logit = kv_overlap_score_weight × potential_prefill_blocks + potential_active_blocks4. potential_prefill_blocks is the prefill work a worker would still have to do after subtracting whatever blocks of the incoming prompt it already has cached — a large cache overlap lowers this term directly, which is the same "avoid redundant compute" logic Module 2's prefix-cache-scorer expresses as a matched/total ratio, just folded into a cost rather than a normalized score4. potential_active_blocks folds in the worker's current decode load: active prefill blocks, active decode blocks, and optionally active-request count, so the same term that penalizes a cold worker's missing cache also penalizes a busy worker's existing queue4. The router picks the worker with the lowest resulting logit — framed explicitly in NVIDIA's own docs as "a placement decision, not an unconditional affinity rule"4, the same caution Module 2 made about prefix affinity never being the sole signal (§6.2 of that module).

The mechanism supplying potential_prefill_blocks is a two-source split that maps directly onto Module 2's approximate-versus-precise duality, just resolved differently: Cached Blocks are tracked globally by a component called the KvIndexer, which maintains a prefix tree (radix tree) built from KV creation/release events that every worker publishes4 — structurally the same idea as Module 2's precise KVEvents-over-ZMQ path, an event-sourced index rather than an inferred one. Active Decoding Blocks, by contrast, are tracked locally by the router itself across a request's lifecycle rather than reported by the worker4 — the router's own bookkeeping of what it has already sent where, the same shape as Module 2's approximate LRU index, but scoped to active decode state rather than prefix-cache state. Dynamo does not offer a router-level choice between an approximate and a precise mode the way llm-d does (Module 2, §2–3); it always uses the event-sourced KvIndexer for cache overlap and always uses local bookkeeping for active load, because the two signals answer different questions rather than being two implementations of the same signal.

Dynamo Router's cost function combining a cache-overlap term from the global KvIndexer with an active-load term tracked locally, producing one logit per candidate worker KvIndexer (global) radix tree from worker-reported KV creation/release events Router-local bookkeeping active prefill/decode blocks tracked per request in flight potential_prefill_blocks work minus cache hit potential_active_blocks current load logit = w × prefill + active lowest logit wins
Figure 2. The Router evaluates one cost per candidate worker, combining a global, event-sourced cache-overlap term with a locally-tracked load term — structurally the same two-signal idea as Module 2's prefix-cache-scorer plus inflight-load-producer, expressed as a single weighted cost instead of a pluggable weighted-sum scorer chain. Formula per NVIDIA's KV-aware routing documentation4.

2.2 Parameter churn: a tuning surface still in motion

The exact name of the weight term in that cost function is not stable across Dynamo's own documentation. Older configuration docs (v1.2.1) already mark --kv-overlap-score-weight and its sibling --router-kv-overlap-score-weight as deprecated, emitting a warning but still accepted, in favor of a replacement parameter, --router-kv-overlap-score-credit (range 0.0–1.0, default 1), which credits device-local prefix overlap against prefill load, combined with a separate --router-prefill-load-scale multiplier (default 1) applied to the adjusted prompt-side load5. The resulting formula in that same documentation is written as prefill_load_scale × adjusted_prefill_blocks + decode_blocks, where the adjusted-prefill term itself now incorporates separate credit weights for device-cache hits, host-cache hits (default 0.75), disk-cache hits (default 0.25), and shared-cache hits5 — a finer-grained decomposition of "how much does a cache hit at this tier actually save" than the single kv_overlap_score_weight term implies. Read against Module 8's finding that NIXL shipped three functionally significant releases in about one month, this is the same signal from a different layer of the stack: the Router's own tuning surface has already been renamed and subdivided once, and a deployment pinning to a specific flag name in one Dynamo version should expect to re-verify it against the live docs at upgrade time rather than assume the name is stable.

3. KVBM: the KV Block Manager

Module 7 established the generic offload-tier hierarchy — HBM → host DRAM → local SSD → remote — through Mooncake's KVCache-centric architecture, and Module 8 covered NIXL's Memory Sections as the abstraction that lets a single transfer call address any of those tiers uniformly. KVBM is Dynamo's own named component implementing that hierarchy specifically, with its own tier names, its own block lifecycle state machine, and NIXL as its data-movement layer underneath.

3.1 Three logical layers

KVBM's architecture documentation describes three layers stacked vertically6. The top layer is the LLM inference runtime itself — TensorRT-LLM, vLLM, or SGLang — integrated through a dedicated connector module per engine; these connectors act as "translation layers, mapping runtime-specific operations and events into the KVBM's block-oriented memory interface," which decouples KVBM's own memory management from any one engine's internal block representation6. The middle layer is KVBM's own core: it "encapsulates the core logic of the KV block manager and serves as the runtime substrate for managing block memory," handling table lookups, memory allocation, block layout management, lifecycle and state transitions, and reuse/eviction policy6. The bottom layer is NIXL, which "provides unified support for enabling all data and storage transactions" — P2P GPU transfers, RDMA and NVLink remote memory sharing, dynamic block registration, and metadata exchange6. That bottom layer is exactly Module 8's NIXL, unmodified: KVBM does not implement its own transport, it registers memory regions with NIXL and lets NIXL's Memory Sections and Transfer Backend Interface move the bytes.

3.2 Four memory tiers: G1 through G4

KVBM names its tiers G1 through G4 rather than using Module 7's HBM/DRAM/SSD/remote labels directly, though the physical mapping is the same idea with one more layer of hierarchy inside the host tier7:

Table 1. KVBM's four memory tiers compared against Mooncake's tiering (Module 7) and NIXL's Memory Sections (Module 8). Data: KVBM design documentation7, cross-referenced against Mooncake's HBM→DRAM→SSD→remote hierarchy and NIXL's memory/storage unification6.
KVBM tierPhysical locationRole
G1 — Device PoolGPU HBM"GPU-resident KV block pool. Allocates mutable GPU blocks, registers completed blocks (immutable)"7 — the primary inference tier
G2 — Host PoolCPU pinned memorypage-locked host memory, chosen specifically for efficient CUDA transfer to/from G17
G3 — Disk Poollocal NVMe SSD"Local SSD NVMe-backed KV block pool" supporting zero-copy I/O and optional GPUDirect Storage7 — the same GDS path NIXL exposes as a named backend (Module 8)
G4 — Remote Storageremote or cloud-backed store"KVBM treats G4 as an opaque blob store"7 — KVBM does not know or care what is underneath, only that NIXL can address it

The G1→G4 naming is a direct match to Module 7's four-tier hierarchy in substance (HBM, host DRAM, local SSD, remote), with one addition worth flagging: G4 is explicitly documented as opaque to KVBM itself7, which is the same design choice NIXL's Memory Sections make one layer down — unify the addressing, but let the actual storage backend (object store, filesystem, block device) stay a plugin NIXL's Transfer Backend Interface resolves at transfer time, not something KVBM's own core logic needs to understand.

3.3 Block lifecycle: a five-state machine with RAII cleanup

Where Mooncake's block model (Module 7) is defined by a content hash over 512-token blocks and Module 2's approximate/precise routing model is defined by whether the router's view of a block is inferred or event-sourced, KVBM defines its own blocks by an explicit state machine, independent of both7:

KVBM's five-state block lifecycle: Reset, Partial, Complete, Registered, and Remove Reset InactivePool init_sequence() Partial filling with tokens commit() Complete finalized, not visible register() Registered hashed, lookup-visible drop Remove Remove event returns block to Reset / InactivePool
  1. Reset. The block's initial state inside the InactivePool — unused, ready for allocation7.
  2. Partial. init_sequence() moves the block into active use while it is still being filled with tokens7.
  3. Complete. commit() finalizes a full block, but it is not yet visible for lookup by other requests7.
  4. Registered. register() hashes the block by sequence_hash and moves it into a deduplication cache; only now can the block "be used for lookup" by a different request sharing that prefix7.
  5. Remove. Automatically triggered when a handle drops (RAII), which deregisters the block from NIXL and returns it to Reset in the InactivePool7.

The RAII pattern in the final step is a specific engineering choice worth naming: KVBM's own documentation states this design "avoids stale memory access, dangling pointers on GPU or host, manual deregistration bugs"7 — the block's NIXL registration and its logical lifetime are tied to the same handle, so a crashed or evicted request cannot leave a block registered with NIXL but logically abandoned, the class of bug that a manual deregister-on-eviction path would otherwise have to get right on every code path. Lookup itself works through the same sequence_hash mechanism as Registered blocks: a "deduplication cache" keyed by that hash lets a later request's prefix match resolve to an already-Registered block across any of the four tiers, without KVBM having to scan pools7 — conceptually the same content-addressed lookup Mooncake's 512-token block hash performs (Module 7), implemented against KVBM's own block granularity rather than Mooncake's.

4. The Planner: SLA-driven autoscaling

The Router and KVBM answer per-request questions — which worker, which tier. The Planner answers a fleet-level question none of Modules 1–8 addressed directly: given a current traffic pattern and a latency target, how many prefill and decode replicas should exist right now. It is a controller that "adjusts prefill and decode engine replica counts at runtime to meet latency SLAs," driven either by live Prometheus metrics or by a load-predictor's forecast, combined with an engine performance model8.

4.1 Four scaling modes and their composition order

The Planner supports four optimization_target modes, each a different answer to "scale on what signal":

Table 2. Dynamo Planner's four scaling modes. Data: Planner Guide8.
ModeSignalNotes
throughputstatic thresholds on queue depth and KV-cache utilizationthe default; needs no SLA target or profiling data8
latencymore aggressive versions of the same thresholds"scales up earlier and tolerates less queuing" than throughput mode8
loaduser-defined prefill queue-token and decode KV-utilization thresholdsreactive, threshold-driven scaling layered on top of the other modes8
slaa performance model targeting specific TTFT/ITL valuesthe only mode that predicts, rather than reacts to, replica needs (§4.2)8

These are not four mutually exclusive settings; they compose. When optimization_target is throughput, latency, or load, load-based scaling activates automatically alongside it, and throughput-based scaling disables8. When both throughput-based and load-based scaling are active in the same evaluation tick, throughput-based scaling sets the lower-bound replica count first, and load-based scaling can then adjust above that floor, subject to a global GPU budget clamp8 — the manifest's framing of "load-based adjusts above the throughput-derived floor" is this exact mechanism. SLA mode is the odd one out: it requires explicitly enabling at least one of enable_throughput_scaling or enable_load_scaling alongside it, rather than composing automatically8, because SLA mode's own output is a target replica count derived from a performance model, and the deployer has to decide whether that target is additionally floored or bounded by the reactive modes.

4.2 SLA mode: the performance model underneath the prediction

SLA mode is the Planner's most involved path, and the one the manifest for this module singles out. It is built on "the Rust engine performance shim with native AIC estimates when available, plus online FPM tuning or FPM regression fallback, to target specific TTFT/ITL values"8. Unpacked, that shim initializes in one of two ways: if a native AIC (analytical/interpolated cost) performance model and the engine's own limits are available for the specific engine/model/hardware combination, it uses those directly; if no native AIC config is supported for that combination, it falls back automatically to an "observed-FPM regression" — a regression fitted from field performance measurements (FPM) rather than an analytical model8. Either way, the shim needs bootstrap data before it can estimate anything: that data comes from a self-benchmark fetched via the engine's own get_perf_metrics endpoint, or from rapid-mode AIC interpolation, or from profiler results read from a configured profile_results_dir (the npz/JSON profiler output the manifest references)8. Once running, online FPM tuning feeds live observations back into the model at a configured load_adjustment_interval_seconds, so the performance model itself keeps updating against reality rather than staying fixed at its bootstrap estimate8.

SLA-mode's performance model pipeline: AIC estimate or regression fallback, bootstrapped from profiler or self-benchmark data, tuned online, producing a target replica count Bootstrap data self-benchmark / profiler npz-json Native AIC estimate if supported for engine/HW unsupported config → fallback FPM regression fallback observed-performance fit Replica-count decision target TTFT 500ms / ITL 50ms online FPM tuning, every load_adjustment_interval_seconds
  1. Bootstrap. The shim gets initial performance data from a self-benchmark call to the engine's get_perf_metrics endpoint, or from profiler output in a configured directory8.
  2. Native AIC estimate. If a native analytical/interpolated cost model exists for this engine/model/hardware combination, the shim uses it directly8.
  3. Regression fallback. If no native AIC config is supported, the shim falls back to a regression fitted from observed field performance measurements (FPM)8.
  4. Replica-count decision. Either path yields an estimate of engine performance under a given replica count, which the Planner uses to target the configured TTFT/ITL values (500 ms / 50 ms by default, §4.3)8.
  5. Online tuning. Live observations feed back into the model at a configured interval, so the estimate keeps correcting against measured reality rather than staying fixed8.

4.3 Default SLA targets and traffic prediction

The Planner's default SLA targets are a TTFT of 500.0 milliseconds and an ITL of 50.0 milliseconds8 — both configurable, but these are the values a fresh deployment starts from. Feeding the SLA-mode decision is a separate traffic-prediction step: the load_predictor field accepts constant, arima, kalman, or prophet as the forecasting method8, with ARIMA and Prophet specifically used to predict request count, input sequence length (ISL), and output sequence length (OSL), while other runtime signals such as KV-cache hit rate instead use the latest observed value rather than a forecast8. NVIDIA's own Dynamo 0.4 announcement frames the same mechanism from the release-notes side: the Planner "predicts future traffic patterns using advanced time-series models such as ARIMA or Prophet," then calculates the minimum prefill/decode worker counts needed to meet SLA targets under that predicted demand9. The two sources agree on the mechanism; the Planner Guide is the more complete one, since it also documents the kalman and constant predictor options the announcement does not mention89.

5. ModelExpress and Grove: weight streaming and gang scheduling

The Router, KVBM, and Planner all operate on request- and cache-level state. ModelExpress and Grove sit further out, at cluster-lifecycle scale: getting a new replica's weights into GPU memory fast, and getting a multi-node group of pods scheduled onto hardware as one atomic unit in the first place. Neither is unique to serving — both matter equally for a training or RL rollout cluster — but both are load-bearing for how quickly a Dynamo deployment can scale out.

5.1 ModelExpress: weight streaming over NIXL instead of shared storage

The problem ModelExpress solves recurs across three scenarios: a cold start pulling weights from remote storage into GPU memory, an autoscaling event that needs to populate a brand-new replica, and reinforcement-learning post-training continuously moving updated weights between trainer and rollout workers10. Its mechanism is the same NIXL point-to-point transfer Module 8 covered, applied to model weights instead of KV cache: when a serving peer already holds compatible weights in GPU memory, ModelExpress "transfers them directly from GPU to GPU over P2P RDMA via NVIDIA Inference Xfer Library (NIXL), bypassing redundant access to object storage, local disk, and host memory"10. When no such peer exists yet, it falls back to the fastest supported bootstrap path — streaming from object storage via a Model Streamer component, or reading local files directly into GPU memory through GPUDirect Storage10 — the same GDS backend NIXL exposes and KVBM's G3 tier uses (§3.2).

A second mechanism, the Model Cache Service, addresses a different failure mode: many replicas scaling up simultaneously would otherwise each independently hit external storage for the same weights. Instead, "an atomic claim in Metadata Store selects a downloader, while the remaining replicas track its progress and reuse the cached copy. The cluster pays the external download cost once"10 — a single-flight pattern applied to weight loading, conceptually the same "avoid redundant work across replicas" logic the Router applies to KV-cache prefill.

Two separate documented examples give ModelExpress's speedup a concrete range rather than one number to anchor on. NVIDIA's own README headline is 7x faster model startup for DeepSeek-V3 on H2001. A separate, more granular example from NVIDIA's ModelExpress blog post gives an end-to-end figure for a larger model: MX transfers DeepSeek-V4-Pro weights and JIT kernel-cache artifacts from a serving replica into a fresh replica on 8×B200 in under 10 seconds, cutting total startup time from 8 minutes to 1 minute 44 seconds10 — a roughly \(8 / 1.73 \approx 4.6\times\) reduction in that specific case, on different hardware and a different model than the headline 7x figure. Both are real, sourced numbers; neither should be read as the number for an arbitrary model/hardware pair, for the same reason §8.2 treats every headline benchmark claim as workload-specific rather than a general constant.

5.2 Grove: Kubernetes gang scheduling for multi-node inference

Grove solves a Kubernetes gap none of the generic scheduling primitives address directly: a single disaggregated Dynamo deployment is not one pod, or even one uniform group of identical pods — it is several distinct roles (prefill workers, decode workers, a frontend, a router) that must be scheduled and scaled together, sometimes spanning multiple physical nodes for one logical model instance. Grove is "a Kubernetes API that provides a single declarative interface for orchestrating any AI inference workload," from a single pod up to complex multi-node, disaggregated systems at data-center scale11.

Its object model has two levels. A PodClique represents "a group of pods representing a specific role (e.g., leader, worker, frontend)" — all prefill workers, say, or all decode workers — with its own container spec, GPU resource claims, replica count, and scaling logic11. A PodCliqueSet is the top-level object composing multiple PodCliques into one group that is managed and colocated together, with autoscaling and topology-aware replica distribution11. Grove creates a PodGang resource for each PodCliqueSet, which a separate scheduler (KAI Scheduler) reads and enforces as an all-or-nothing constraint116: if only a subset of the required pods across all cliques can be scheduled, none of them start, because a partially-scheduled disaggregated instance — prefill workers up, decode workers pending — is not a usable model instance and would otherwise leave GPUs allocated and idle. On top of that gang-scheduling guarantee, Grove also does topology-aware placement, described as "network optimized placement, e.g. within NVLink domains"11 — the same NVL72-domain reasoning Module 1 introduced for wide expert parallelism, here applied to keeping a PodClique's pods close enough together that intra-clique traffic stays on fast interconnect rather than crossing a rack boundary.

The division of labor between Dynamo and Grove is explicit in NVIDIA's own description: "Dynamo manages the inference control plane at runtime: routing incoming requests, tracking KV cache locations, and dispatching prefill/decode work to the right workers. Grove manages the Kubernetes workload lifecycle: starting pods in the right order, placing them on the right hardware, scaling them as a coordinated unit"11. That split maps cleanly onto this module's two halves so far: Router/Planner/KVBM (§2–4) are the runtime control plane; Grove is the Kubernetes-side scheduling substrate underneath it. Module 14 covers Kubernetes deployment topology in general; Grove is Dynamo's own gang-scheduling answer to that general problem.

6. Disaggregation orchestration built on NIXL

Module 4 established the generic connector pattern — producer/consumer roles, a LookupBuffer, a Pipe — and the backend-specific choice between synchronous and overlapped transfer. Dynamo's own disaggregated-serving implementation is a concrete instance of that pattern, built directly on NIXL rather than a custom transport, and it is worth stating precisely rather than re-deriving, since Module 4 already covers the general shape in full.

NVIDIA's own architecture documentation states the mechanism directly: "Dynamo leverages NIXL to transfer KV cache directly from the VRAM of the prefill engine to the VRAM of the decode engine. The KV transfer is non-blocking, allowing GPU forward passes to continue serving other requests during the transfer"12. That non-blocking property is the same one Module 4 flagged generically (“non-blocking KV transfer”) — here it is Dynamo's specific implementation of it, VRAM-to-VRAM via NIXL's Memory Sections rather than staging through host memory.

Backend-specific synchronization behavior, already covered generically in Module 4, appears in Dynamo's own docs in the same terms: SGLang uses bootstrap coordination that lets the decode phase begin immediately while KV transfer proceeds in parallel, while vLLM and TensorRT-LLM run prefill synchronously, with decode waiting for prefill to complete before proceeding12. This is not a Dynamo-specific mechanism — it is each backend engine's own transfer behavior, which Dynamo's orchestration layer has to accommodate rather than override, since Dynamo does not reimplement any engine's internal KV-transfer logic. NVIDIA's own performance-characteristics documentation is explicit that it deliberately declines to encode the prefill/decode split point as "a universal token threshold"12 — the right split is workload- and engine-dependent, not a constant Dynamo can bake in.

What Dynamo's orchestration layer does add on top of the generic connector pattern is runtime-reconfigurable xPyD: x prefill workers and y decode workers, with both counts changeable at runtime rather than fixed at deployment time12. Workers register with a discovery service and publish a RuntimeConfig that includes their KV capacity when added, and drain their active requests before deregistering when removed12 — the same drain-before-remove pattern Module 4 introduced generically, here as Dynamo's own worker-lifecycle mechanism, and the mechanism the Planner's SLA-mode replica-count decisions (§4.2) actually act on: a Planner decision to add a decode replica has nowhere to take effect without this registration path underneath it.

7. Observability: the Prometheus metrics surface

Every scaling and routing decision in §2 and §4 has to be driven by something measured. Dynamo exposes its own metrics in Prometheus exposition format at a /metrics endpoint, with every Dynamo-generated metric using a dynamo_* name prefix and labels (dynamo_namespace, dynamo_component, dynamo_endpoint) identifying which component and endpoint produced it13. NVIDIA's own Dynamo 0.4 announcement summarizes the categories at a high level — average requests per second and request duration; average TTFT and ITL; average input and output sequence length; GPU utilization and power usage9 — and the live metrics catalog gives the specific named metrics underneath those categories, plus several categories the announcement does not mention (queue depth, cache-hit accounting, error/migration counters):

Table 3. A representative selection of Dynamo's dynamo_* Prometheus metrics, grouped by category. Data: Dynamo metrics catalog13, categories cross-referenced against the Dynamo 0.4 announcement9.
CategoryMetricTypeWhat it measures
Request rate/durationdynamo_frontend_requests_totalcountertotal LLM requests13
Request rate/durationdynamo_frontend_request_duration_secondshistogramend-to-end request duration13
TTFTdynamo_frontend_time_to_first_token_secondshistogramtime to first token13
ITLdynamo_frontend_inter_token_latency_secondshistograminter-token latency13
ISL/OSLdynamo_frontend_input_sequence_tokenshistograminput sequence length in tokens13
ISL/OSLdynamo_frontend_output_sequence_tokenshistogramoutput sequence length in tokens13
Queue / KV cachedynamo_frontend_router_queue_pending_requestsgaugerequests pending in the router scheduler queue13
Queue / KV cachedynamo_component_router_kv_hit_ratehistogrampredicted KV-cache hit rate at routing time — the Router's own overlap-score input (§2.1), exported as a metric13
Queue / KV cachedynamo_component_kv_cache_events_appliedcounterKV-cache events applied to the router's radix-tree index — the same KvIndexer event stream §2.1 describes13
Errors / migrationdynamo_frontend_model_rejection_totalcounterrequests rejected because all workers were busy13
Errors / migrationdynamo_frontend_model_migration_totalcounterrequest migrations due to worker unavailability13
Routing overheaddynamo_router_overhead_total_mshistogramtotal routing overhead per request, broken down into block-hashing, indexer-lookup, and scheduling sub-metrics13

Two rows are worth reading against earlier sections directly. dynamo_component_router_kv_hit_rate and dynamo_component_kv_cache_events_applied are not generic serving metrics — they are the Router's own KvIndexer and cost-function machinery from §2.1, exposed as Prometheus series so an operator can watch the overlap-score input directly rather than infer it from downstream latency. dynamo_router_overhead_total_ms and its sub-metrics are an admission the routing decision itself has a cost — block-hashing time, indexer-lookup time, scheduling time are each separately tracked13, the same category of overhead Module 2 flagged for the tokenization round-trip in precise KV-indexing, here quantified rather than described qualitatively. GPU utilization and power usage are named in the Dynamo 0.4 announcement as part of the observability surface9, but do not appear under the dynamo_* prefix in the metrics catalog reviewed for this module; they are exposed through separate GPU-telemetry integration (DCGM-style exporters) rather than as Dynamo-native metrics, and this module cannot source their exact metric names beyond what the announcement states in prose9.

8. Version, release cadence, and headline benchmarks

8.1 Release cadence as evidence of pace

Dynamo's current tagged release is v1.4.2, dated 2026-08-2914. Reading the three releases immediately preceding it shows a cadence of roughly one release per week, each with functional changes rather than only cosmetic ones:

  1. Base of the v1.4.x series

    Base version for the v1.4.x series, with multiple dev builds extending support to specialized models14.

  2. Classify/pooling endpoints; Router fix

    Adds OpenAI-compatible classify and pooling endpoints to the Dynamo Frontend with vLLM worker support; fixes Router overload recovery, a logprob_token_ids passthrough rejection bug, and vLLM-Omni NIXL object-transfer failures14.

  3. NIXL loader-path patch

    A patch release on top of v1.4.1 that fixes NIXL loader-path resolution in the Frontend and SGLang Runtime container images, where Rust bindings were running against non-functional stubs; also removes an unused profiling plugin and tightens dependency pins314.

  4. Model-specific dev containers

    Multiple v1.5.0-dev tags ship in early September 2026 for specific new models: v1.5.0-gemma-4-31b-dev.1 (Sept. 3) with early TensorRT-LLM 1.3.0rc25 support314, v1.5.0-inkling-dev.1 (Sept. 3) with vLLM 0.28.0 for Thinking Machines' Inkling on GB300, and v1.5.0-deepseek-v4-pro-0813-dev.1 (Sept. 4) adding DeepSeek-V4-Pro-0813 support on GB200/H200 with DSpark speculative decoding and KV-aware routing14.

A patch release fixing a NIXL loader-path bug in the exact week following the previous patch is the same signal Module 8 documented for NIXL's own release history: this is a stack whose transport layer is still being hardened in production, not one where the last bug in that class was found years ago.

8.2 Headline benchmark claims, read with their scope

Dynamo's own README and blog posts carry several large multiplier claims. Read individually, each is tied to a specific model, GPU generation, and mechanism — none is a general "Dynamo makes inference N× faster" constant, and the manifest for this module is explicit that these should be read as vendor-reported.

Table 4. Dynamo's headline performance claims, with the specific workload and hardware each is tied to. Data: project README1 and the Dynamo 0.4 developer blog9.
ClaimWorkload / hardware scopeMechanism credited
7× higher throughput per GPUDeepSeek R1, GB200 NVL721disaggregated serving + wide-EP, not attributed to one component alone
2× faster TTFTQwen3-Coder 480B1KV-aware routing (§2)
750× higher throughputDeepSeek-R1, GB300 NVL721full-stack, hardware-generation-specific; not decomposed by component
7× faster model startupDeepSeek-V3, H2001ModelExpress weight streaming (§5.1)
2.5× higher throughput/GPUDeepSeek-R1 671B, GB200 NVL72, TensorRT-LLM backend9full-stack, dated Aug. 2025 (Dynamo 0.4), superseded in scope by the newer 7× claim above
Up to 4× faster interactivitygpt-oss-120b, B200, very long input sequences, TensorRT-LLM backend9full-stack; explicitly scoped to long-ISL workloads, not a general figure

Three things are worth noting about this table as a whole. First, the 7×-per-GPU and 750× figures are both tied to GB200/GB300 NVL72 and DeepSeek R1 specifically1 — the same hardware and model class Module 6 grounded in DeepSeek's own published deployment, which is the correct reference point for sanity-checking these numbers, not a generic Llama-class deployment. Second, none of the entries in this table decomposes its multiplier into contributions from Router, Planner, and disaggregation separately — "7× higher throughput per GPU" is a full-stack number, and this module cannot attribute what fraction of it comes from KV-aware routing versus disaggregated pools versus wide-EP placement, because NVIDIA's own sources do not publish that decomposition. Third, the 2.5× and 4× figures from the August 2025 Dynamo 0.4 announcement9 predate the current v1.4.2 release by roughly a year and are smaller than the newer 7× claim on similar hardware1 — consistent with the stack having genuinely improved over that year, but also a reminder that any single headline number in this table has a shelf life measured in months, not years.

9. Is Dynamo production-grade today?

Dynamo 1.0 reached general availability on March 16, 2026, at GTC2. NVIDIA's own production-readiness announcement states the year before that date was spent working "collaboratively with the open source ecosystem to harden Dynamo for production-grade performance and large-scale workloads"15, and frames the GA release around four additions: unified configuration, Kubernetes production readiness, multimodal serving, and an agents surface15. The adoption list accompanying the GA announcement is broad and spans several distinct categories rather than one or two logo-slide names: major cloud providers running it in managed Kubernetes (AWS, Microsoft Azure, Google Cloud, Oracle Cloud Infrastructure, Alibaba Cloud), GPU cloud partners (CoreWeave, Crusoe, DigitalOcean, Gcore, GMI Cloud, Lightning AI, Nebius, Nscale, Together AI, Vultr), AI-native companies (Cursor, Hebbia, Perplexity), inference-endpoint providers (Baseten, Deep Infra, Fireworks), and enterprises (AstraZeneca, BlackRock, ByteDance, Coupang, Instacart, Meituan, PayPal, Pinterest, Shopee, SoftBank Corp.)2. That breadth — hyperscalers, GPU clouds, and inference-specific vendors all named as running it — is a stronger production-readiness signal than a GA date alone, because it shows adoption at more than one layer of the serving stack, not just one hyperscaler badge-checking a launch.

Set against that, two sources of counter-evidence in this module keep the assessment from being unqualified. The release cadence in §8.1 shows Dynamo shipping a NIXL-loader-path bug fix in v1.4.2 — a patch release — the same week as v1.4.1's own bug fixes, roughly six months after the GA date314; a project six months past its 1.0 milestone is still finding and fixing loader-level bugs in its container images, which is normal for young infrastructure but is not the profile of a settled, slow-moving system. And Module 4 already noted that dynamic node joining and leaving — letting a prefill or decode pool grow or shrink without a full redeploy, beyond the RuntimeConfig registration path §6 describes — is explicitly in progress, not GA, in the disaggregation stack generally; nothing in this module's Dynamo-specific sources states that capability has since reached GA status specifically inside Dynamo's own xPyD implementation.

Put plainly: Dynamo is production-grade in the sense that matters most — it is GA, it has a real, diverse adoption list including major cloud providers running it in their own managed offerings, and its core mechanisms (Router, Planner, KVBM, disaggregation) are documented, versioned, and exercised by that adoption215. It is not production-grade in the sense of being a finished, slow-churn system: patch releases fixing functional bugs roughly weekly14, a router parameter already renamed and subdivided once (§2.2), and specific capabilities (dynamic node joining) still marked in progress rather than complete, are all live as of this module's access date. Both statements are true simultaneously, and treating either one alone as the full picture would misread the evidence.

Where experts disagree

The disagreement in this module is not about whether Dynamo's mechanisms work — the adoption list in §9 settles that they do, for a broad range of deployers. It is about whether an integrated, vendor-built full-stack orchestration platform is the right architectural bet for this layer of the serving stack at all, versus a modular set of Kubernetes-standard building blocks that any vendor's components can plug into. NVIDIA itself sits on both sides of this question, which is part of what makes it a genuine, unsettled disagreement rather than a marketing claim versus a critique.

An integrated, purpose-built platform is the right bet

Implicit in Dynamo's own design and its adoption evidence12

Dynamo ships Router, Planner, KVBM, ModelExpress, and Grove as one versioned project, tested and released together, with each component already aware of the others — the Router's cost function already accounts for KVBM's tiered cache-hit weights (§2.2), the Planner's replica decisions act through the same worker-registration path the disaggregation layer defines (§6). A deployer adopting Dynamo gets a coherent system where the seams between routing, autoscaling, and KV tiering have already been designed to fit, rather than having to integrate five separately maintained open-source projects and resolve version-compatibility and interface mismatches themselves. The breadth of the GA adoption list — four major clouds and a dozen-plus GPU clouds and inference vendors2 — is direct evidence this integration is worth paying for.

  • One versioned system; components are designed to compose, not integrated after the fact.
  • Broad, diverse production adoption across cloud providers, GPU clouds, and inference vendors as of GA2.
  • NVIDIA-backed engineering investment funding rapid iteration (weekly-cadence patches, §8.1).
  • Coupling to one project's release cadence and versioning; the router-parameter rename in §2.2 shows internal APIs are not yet stable.
  • A single vendor's roadmap decides which backends and features get priority.

Kubernetes-standard modular building blocks are the more durable path

The Gateway API Inference Extension's own design goal16, and NVIDIA's own stated intent to contribute Dynamo pieces into llm-d rather than only build its standalone stack17

The Gateway API Inference Extension (Module 2, §5.2) standardizes the routing/scheduling contract at the Kubernetes API level — InferencePool, the Endpoint Picker protocol — so that any conformant implementation can plug in, and a deployer is not locked into one vendor's own runtime for that layer16. llm-d (Module 10) is one such implementation, built by a multi-vendor consortium rather than one company. The strongest evidence this position deserves real weight is not third-party skepticism, it is NVIDIA's own stated behavior: in a developer blog announcing collaboration with the llm-d community, NVIDIA committed to "bring the benefits of NVIDIA Dynamo Planner to the llm-d Variant Autoscaler component" and to "bring the benefits of the NVIDIA Dynamo KV Cache Manager to the llm-d KV Cache subsystem"17 — contributing the ideas behind two of Dynamo's own named components into a separate, Kubernetes-standard project, rather than treating llm-d purely as a competitor to out-execute.

  • No lock-in to one vendor's runtime for the routing/scheduling API surface; any conformant EPP implementation works.
  • Multi-vendor governance (Module 10: Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA) spreads maintenance burden and reduces single-vendor roadmap risk.
  • Modularity has an integration cost the deployer pays instead of the vendor — components from different projects have to be version-matched and interface-tested independently.
  • The 2025 collaboration announcement17 is over a year old relative to this module's access date; this module found no equally recent, equally specific follow-up confirming how much of that integration actually shipped inside llm-d by v1.4.2's timeframe.
Assessment

These are not mutually exclusive bets, and NVIDIA's own behavior — building Dynamo as an integrated platform while simultaneously contributing pieces of it into a Kubernetes-standard competitor — is the clearest evidence neither side has fully won the argument yet. The integrated-platform case is stronger for a deployer already committed to a single serving stack across their fleet, who values a router cost function that already knows about KVBM's tiers over one they would have to hand-tune themselves; the adoption list in §9 shows this describes a large fraction of current production deployments. The modular-standard case is stronger for a deployer running a multi-vendor or multi-engine fleet, or one for whom avoiding dependency on a single company's release cadence outweighs the integration cost of assembling equivalent capability from separately governed projects. What would settle this more concretely is exactly the follow-up this module could not find: a dated status report on how much of Dynamo's Planner and KV Cache Manager logic has actually landed inside llm-d's own components, and a head-to-head benchmark of the two paths under identical traffic — neither exists in the sources available as of 2026-09-17.

References

  1. repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo (README). v1.4.2 container tags; v1.5.0-dev builds referenced, 2026. https://github.com/ai-dynamo/dynamo. Accessed 2026-09-17. Used for: "orchestration layer above inference engines" framing, five named components (Router, Planner, KVBM, ModelExpress, Grove), backend list (vLLM, SGLang, TensorRT-LLM), headline benchmark claims (7x/GPU, 2x TTFT, 750x, 7x model startup), current version.
  2. officialNVIDIA. NVIDIA Enters Production With Dynamo, the Broadly Adopted Inference Operating System for AI Factories. NVIDIA Newsroom, 2026-03-16. https://nvidianews.nvidia.com/news/dynamo-1-0. Accessed 2026-09-17. Used for: Dynamo 1.0 GA date, full production-adopter list across cloud providers, GPU clouds, AI-native companies, inference vendors, and enterprises.
  3. repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo Releases. v1.4.2 (2026-08-29), v1.5.0-dev builds in progress. https://github.com/ai-dynamo/dynamo/releases. Accessed 2026-09-17. Used for: v1.4.2 release date and NIXL loader-path fix, v1.5.0-dev build dates.
  4. officialNVIDIA. KV-Aware Routing (NVIDIA Dynamo Documentation). docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/kv-aware-routing, 2026. https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/kv-aware-routing. Accessed 2026-09-17. Used for: the Router's cost function (logit = kv_overlap_score_weight × potential_prefill_blocks + potential_active_blocks), KvIndexer's global radix-tree/prefix-tree index built from worker-reported KV events, locally-tracked Active Decoding Blocks, "placement decision, not an unconditional affinity rule" framing.
  5. officialNVIDIA. Router: Configuration and Tuning (NVIDIA Dynamo Documentation). v1.2.1. https://docs.nvidia.com/dynamo/v1.2.1/components/router/configuration-and-tuning. Accessed 2026-09-17. Used for: deprecated kv_overlap_score_weight parameter, replacement router-kv-overlap-score-credit and router-prefill-load-scale parameters and defaults, host/disk cache-hit credit weights (0.75/0.25).
  6. officialNVIDIA. KVBM Architecture (NVIDIA Dynamo Documentation). v-0-8-1. https://docs.nvidia.com/dynamo/v-0-8-1/components/kvbm/architecture. Accessed 2026-09-17. Used for: KVBM's three logical layers (engine connector translation layer, KVBM core, NIXL layer), NIXL's role providing P2P GPU transfer/RDMA/NVLink/dynamic block registration, PodGang/KAI Scheduler gang-scheduling cross-reference.
  7. officialNVIDIA. KVBM Design (NVIDIA Dynamo Design Docs). v1.3.0. https://docs.nvidia.com/dynamo/v1.3.0/design-docs/component-design/kvbm-design. Accessed 2026-09-17. Used for: G1-G4 memory-tier definitions (Device Pool, Host Pool, Disk Pool, Remote Storage), five-state block lifecycle (Reset/Partial/Complete/Registered/Remove), RAII-based cleanup, sequence_hash deduplication lookup.
  8. officialNVIDIA. Planner Guide (NVIDIA Dynamo Documentation). v1.2.1. https://docs.nvidia.com/dynamo/v1.2.1/components/planner/planner-guide. Accessed 2026-09-17. Used for: four scaling modes and composition order, SLA-mode performance model (Rust engine shim, AIC estimates, FPM tuning/regression fallback, bootstrap sources), default SLA targets (TTFT 500ms, ITL 50ms), load_predictor options (constant/arima/kalman/prophet).
  9. blogNVIDIA. NVIDIA Dynamo 0.4 Delivers 4x Faster Performance, SLO-Based Autoscaling, and Real-Time Observability. NVIDIA Developer Blog, published 2025-08-13. https://developer.nvidia.com/blog/dynamo-0-4-delivers-4x-faster-performance-slo-based-autoscaling-and-real-time-observability. Accessed 2026-09-17. Used for: ARIMA/Prophet traffic-prediction framing, Prometheus metric category list, 2.5x DeepSeek-R1/GB200 and 4x gpt-oss-120b/B200 benchmark figures.
  10. blogNVIDIA. ModelExpress: Distributing Model Artifacts at the Speed of Light. NVIDIA Developer Blog. https://developer.nvidia.com/blog/modelexpress-distributing-model-artifacts-at-the-speed-of-light. Accessed 2026-09-17. Used for: ModelExpress's GPU-to-GPU P2P/NIXL transfer mechanism, bootstrap fallback path (Model Streamer, GDS), Model Cache Service single-download pattern, DeepSeek-V4-Pro 8xB200 startup example (8 min to 1 min 44 sec).
  11. repoNVIDIA / ai-dynamo contributors. ai-dynamo/grove (README). https://github.com/ai-dynamo/grove. Accessed 2026-09-17. Used for: PodClique/PodCliqueSet CRD definitions, PodGang gang-scheduling mechanism, NVLink-domain topology-aware placement, Dynamo/Grove division of labor.
  12. officialNVIDIA. Disaggregated Serving (NVIDIA Dynamo Documentation). docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. Accessed 2026-09-17. Used for: non-blocking VRAM-to-VRAM NIXL transfer, backend-specific sync/overlap behavior (SGLang bootstrap vs. vLLM/TensorRT-LLM synchronous), runtime-reconfigurable xPyD and RuntimeConfig registration, rejection of a universal token-threshold split point.
  13. officialNVIDIA. Metrics Catalog (NVIDIA Dynamo Documentation, reference). docs.nvidia.com/dynamo/reference/observability/metrics-catalog. https://docs.nvidia.com/dynamo/reference/observability/metrics-catalog. Accessed 2026-09-17. Used for: named dynamo_* Prometheus metrics grouped by category (request rate/duration, TTFT, ITL, ISL/OSL, queue/KV cache, errors/migration, routing overhead).
  14. repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo Releases (detailed changelog). v1.4.0, v1.4.1 (2026-08-22), v1.4.2 (2026-08-29), v1.5.0-dev builds (Sept. 2026). https://github.com/ai-dynamo/dynamo/releases. Accessed 2026-09-17. Used for: per-release changelog detail (classify/pooling endpoints, Router overload-recovery fix, NIXL loader-path fix, v1.5.0-dev model-specific containers).
  15. blogNVIDIA. How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale. NVIDIA Developer Blog. https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/. Accessed 2026-09-17. Used for: "hardened Dynamo for production-grade performance" quote, the four GA additions (unified configuration, Kubernetes production readiness, multimodal serving, agents surface).
  16. repoKubernetes SIG contributors. kubernetes-sigs/gateway-api-inference-extension (README). GA since v1.0.0, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension. Accessed 2026-09-17. Used for: the Gateway API Inference Extension's design goal as a vendor-neutral, Kubernetes-standard routing/scheduling API any implementation can plug into.
  17. blogNVIDIA. NVIDIA Dynamo Accelerates llm-d Community Initiatives for Advancing Large-Scale Distributed Inference. NVIDIA Developer Blog, published 2025-05-21. https://developer.nvidia.com/blog/nvidia-dynamo-accelerates-llm-d-community-initiatives-for-advancing-large-scale-distributed-inference/. Accessed 2026-09-17. Used for: NVIDIA's stated commitment to contribute Dynamo Planner and KV Cache Manager capabilities into llm-d's Variant Autoscaler and KV Cache subsystem.