Multi-Node LLM Inference: From One Node to N · Module 8
LMCache and NIXL: The Reusable KV Layer and the Transfer-Library Primitive
LMCache turns a serving engine's KV cache from private, per-replica scratch space into a pluggable, storage-backed resource: chunk it, evict it under GPU memory pressure, and reload it on a prefix hit, independent of which engine or which storage tier sits underneath. NIXL solves a narrower problem one layer down — a uniform point-to-point transfer call that works identically whether the destination is another GPU's HBM, a peer's CPU DRAM, or a file on NVMe — and three independently built engines (Dynamo, TensorRT-LLM, SGLang) now share it as a default or alternate KV-transfer backend. This module covers LMCache's offload mechanism and backend list, NIXL's three abstractions and version history, and contrasts both against Mooncake's Transfer Engine from Module 7, so you can say precisely which layer you are touching when you set a connector flag.
LMCacheConnectorV1, Dynamo's LMCache integration); other paths (vLLM's NixlConnector, SGLang's and TensorRT-LLM's own backend selection, detailed in §2.3) reach NIXL or Mooncake's Transfer Engine directly. LMCache itself can dispatch to NIXL, Mooncake, or its own simpler backends (§1.2). Table 2 gives the full per-project mapping. Data: LMCache's backend list1 and NIXL's abstractions/backend list2.1. LMCache: turning KV cache into reusable state
Every project covered so far in this course keeps its KV cache as private, in-process GPU memory: once a request finishes and its blocks are evicted, that computation is gone unless another request happens to land on a replica that still holds a matching prefix in HBM (Module 2's prefix-cache routing) or unless Mooncake's cluster-wide pool still has it (Module 7). LMCache adds a layer underneath the engine that gives KV cache a lifecycle independent of any single request or GPU: it moves cold chunks out of GPU memory before eviction destroys them, and reloads them on a later prefix hit instead of letting the engine recompute1. The project's own framing is explicit about this reversal: LMCache is "a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge"1.
1.1 Mechanism: memory pressure, chunking, transparent retrieval
The trigger is GPU memory pressure. As new requests need KV-cache space, LMCache moves data out of the engine's own paged GPU allocator into a configured backend, then transparently returns it when a later request's prompt shares a prefix with an offloaded chunk — the quickstart doc states plainly that "LMCache will automatically handle storing and managing KV cache in CPU memory" once GPU memory is insufficient3.
Movement happens at LMCache's own chunk granularity, not at whatever block size the underlying engine's paged KV allocator uses internally: the default chunk size is 256 tokens3. This is a separate, independently-configured boundary from Mooncake's 512-token content-hash block covered in Module 75 — LMCache re-chunks at its own size regardless of what physical block granularity the engine or a different cluster-wide cache happens to use.
The economic logic underneath the whole mechanism is the one Module 3 already established for prefill generally: regenerating a previously-computed prefix's KV state costs a full compute-bound forward pass over every token in that prefix, while reading the same state back from a slower tier costs only the bytes moved. LMCache applies that trade one layer down from Module 3's prefill/decode pool split — inside a single engine instance, below the level any pool-level scheduling decision operates at — exchanging capacity on a CPU, disk, or network tier for avoided recompute, chunk by chunk, whenever a prefix happens to still be resident somewhere off the GPU. The trade only pays when a chunk is actually reused; a chunk offloaded and never touched again is pure overhead with no offsetting benefit, which is why the backend and capacity choice in Table 1 is a real tuning decision for a given workload's prefix-reuse rate, not a free upgrade to apply uniformly.
- Memory pressure. As new requests need KV-cache space, the engine's own paged GPU allocator runs low, and LMCache steps in3.
- Chunk boundary. LMCache carves the cold KV data into its own 256-token chunks, a boundary independent of whatever block size the engine's own paged allocator uses internally3.
- Backend selection. Each chunk is written to whichever of LMCache's eight configured backends is active for this deployment (Table 1)1.
- Reuse on a later hit. A later request whose prompt shares that prefix triggers a reload from the backend instead of a recompute on the GPU3.
1.2 Pluggable backends
LMCache does not implement one fixed storage tier; it is a plugin surface. Between the quickstart doc's backend list — CPU memory, local file system, Mooncake Storage, InfiniStore, Redis, ValKey3 — and the project README's broader list — CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS1 — the union is eight named backends spanning three very different reachability classes: in-process host memory, local persistent storage, and network-attached storage or transport libraries.
| Backend | Reachability | Notes |
|---|---|---|
| CPU RAM | in-process, no network hop | the default in LMCache's own quickstart example3 |
| Local disk (SSD) | local filesystem | survives process restart, still local to the host1 |
| Redis / Valkey | network, in-memory KV store | shared across processes/hosts, not just one engine instance3 |
| Mooncake (Storage) | network, Mooncake's own store | composes with Module 7's KVCache-centric architecture35 |
| InfiniStore | network, dedicated KV storage | listed alongside Mooncake as a remote storage option3 |
| S3-compatible object storage | network, object store | largest-capacity tier in the list, listed only in the README1 |
| NIXL | transport library, not a store itself | LMCache dispatches through NIXL's own backend interface (§2) rather than opening its own wire protocol1 |
| GDS | GPUDirect Storage transport | the same GPU-to-storage path NIXL itself exposes as a backend12 |
Reading the eight rows as three concentric reachability tiers, rather than eight independent choices, clarifies the design space more than the raw list does. CPU RAM and local disk are host-local: no network round trip, but capacity bounded by whatever one host provides, and invisible to any other replica1. Redis/Valkey, Mooncake, InfiniStore, and S3 are network-attached: reachable from any replica in the cluster, at the cost of a network hop per chunk, and — for Mooncake specifically — already integrated with the cluster-wide content-addressed pool Module 7 describes3. NIXL and GDS are neither a store nor a tier; they are the transport two of the other backends (Mooncake, and implicitly any GDS-reachable filesystem) can themselves be reached through, which is why they appear in the same list without being a competing capacity option.
Which tier is worth configuring depends on what the deployment is actually optimizing for, not on which backend is individually fastest. A single replica serving traffic with no cross-instance sharing gets most of the achievable benefit from CPU RAM alone: no network hop, and it captures the common case of a user revisiting a similar prompt within that one process's lifetime. A fleet of replicas behind a shared router (Module 2) benefits more from a network-attached tier such as Redis/Valkey or Mooncake, because a chunk offloaded by one replica becomes reachable by whichever other replica the router happens to send the next matching request to, rather than staying trapped behind whichever process first evicted it. A deployment where the KV cache must survive a process restart, a rolling upgrade, or a crash needs a tier that outlives the engine process entirely, which rules out CPU RAM — freed the moment the process exits — and points at local disk, S3, or a dedicated store instead.
The last two rows matter for §3: two of LMCache's eight "backends" are not storage systems at all, they are transport libraries — meaning LMCache's own plugin list already includes NIXL as one option among several, not a competitor to it.
1.3 vLLM integration and a worked example
vLLM enables LMCache through the same --kv-transfer-config flag Module 4 introduced for connectors generally, setting kv_connector to LMCacheConnectorV1 and kv_role to kv_both3. Module 4 introduced the general shape underneath every connector: a kv producer and a kv consumer, with a Connector retrieving KV caches between them, a LookupBuffer exposing insert and drop_select calls, and a Pipe as the single-direction FIFO that actually carries tensors4. LMCache's kv_role=kv_both setting collapses that producer/consumer split onto one instance: within a single vLLM process, LMCache is simultaneously the producer that inserts evicted chunks into its backend and the consumer that later retrieves them, rather than the producer and consumer being two separate instances the way they are for prefill/decode disaggregation in Module 4. vLLM's own reference production deployment treats LMCache as a first-class, documented capability rather than a third-party add-on: the production-stack repository's tutorial set includes "How to Enable KV Cache Offloading with LMCache" as a named tutorial6.
The quickstart doc's own worked configuration gives a concrete sizing example: a CPU memory limit of 5.0 GB, a stated workload ratio of about 1.5 GB per 10,000 tokens, and an observed roughly 7.43x speedup on second-run inference with CPU offloading enabled3. Combined with the 256-token default chunk size, that ratio turns into a concrete cache budget:
$$ \text{tokens cacheable} = \frac{5.0\ \text{GB}}{1.5\ \text{GB} \,/\, 10{,}000\ \text{tok}} = \frac{5.0}{1.5}\times 10{,}000 \approx 33{,}300\ \text{tokens} $$At 256 tokens per chunk, that is roughly \(33{,}300 / 256 \approx 130\) resident chunks before this particular 5.0 GB CPU tier itself fills up and starts evicting. This ratio is an example tied to the specific model and precision used in LMCache's own quickstart, not a universal constant — the arithmetic pattern (memory budget divided by bytes-per-token, divided again by chunk size) is what transfers to a different model, not the 1.5 GB figure itself.
LMCache CPU-tier cache budget
Formula: \( \text{tokens} = (\text{mem} / \text{ratio}) \times 10{,}000 \), \( \text{chunks} = \lfloor \text{tokens} / \text{chunk} \rfloor \). Defaults reproduce LMCache's own quickstart example (5.0 GB budget, 1.5 GB per 10,000 tokens, 256-token chunk)3. The ratio itself is specific to the model and precision in that example, not a general constant — move the slider to see how the token budget scales once you substitute a different model's KV-cache-bytes-per-token figure.
Whether the roughly 7.43x second-run speedup generalizes depends entirely on how much of that second run was cache-hit rather than cache-miss traffic and on the specific model and hardware in the quickstart's own test3; treat it as evidence the mechanism works, not as a deployment-independent multiplier.
1.4 Integration timeline
LMCache's adoption into the two largest orchestration projects in this course happened within about two months of each other in the second half of 2025: the README records that LMCache "extends its turbo-boost to multimodal models in vLLM V1" as of a July 2025 update, and that "NVIDIA Dynamo integrates LMCache, accelerating LLM inference" as of September 20251. Module 9 covers exactly where LMCache sits inside Dynamo's own KV Block Manager (KVBM); here the point is narrower — by late 2025, LMCache had gone from a vLLM-specific offload connector to an offload/reuse layer plugged into the two most widely deployed orchestration stacks in this ecosystem, without changing its own mechanism or backend list. That LMCache could be integrated into two architecturally different orchestration stacks within roughly two months of each other is itself indirect evidence for the design choice in §1.2: because LMCache's interface to a host engine is a connector plus a pluggable backend store, not a fork of that engine's own KV-cache internals, adding a new consumer is a matter of implementing one connector against LMCache's existing API rather than modifying LMCache itself.
2. NIXL: the point-to-point transfer primitive
Module 1 named NIXL's three abstractions in passing, while building the physical bandwidth/latency ladder underneath every transport this course refers to2. This section gives NIXL's architecture in full: what each abstraction does, which backends actually work today versus which are still in development, why three independently built serving engines route their KV transfer through it, and what its release history says about how settled the library is.
2.1 Three abstractions
NIXL (NVIDIA Inference Xfer Library) states its own design goal directly: it is "targeted for accelerating point to point communications in AI inference frameworks such as NVIDIA Dynamo, while providing an abstraction over various types of memory (e.g., CPU and GPU) and storage (e.g., file, block and object store) through a modular plug-in architecture"2. That goal decomposes into three named abstractions2:
- Memory Sections — unifies memory and storage types (GPU HBM, CPU DRAM, files, block devices, object stores) behind one buffer-list primitive, so a transfer call does not need a different code path depending on what kind of address it is reading from or writing to.
- Transfer Backend Interface — abstracts the different wire-level transport backends (UCX, GDS, and whatever else is registered) and selects the one that can actually move bytes between the memory/storage types named in a given transfer.
- Metadata Handler — manages the data needed to establish communication between two NIXL agents: discovery, capability exchange, and cached addressing information, so repeated transfers between the same pair of agents do not re-negotiate from scratch.
NIXL's name states its scope precisely: an inference transfer library, not a general communication library. Module 1 already drew the distinction its design leans on — UCX serves both point-to-point transfers and the ring/tree collective patterns NCCL and RCCL use for all-reduce, and those two traffic shapes have historically been optimized by different code paths even when they share the same transport layer underneath. NIXL targets only the point-to-point side of that split2: a KV-cache handoff from one prefill worker's Memory Section to one decode worker's, a single chunk written to one storage backend, a weight push from one rank to another (Module 9's ModelExpress). It has no notion of a reduction operation, a broadcast tree, or a collective's rank topology — every transfer in NIXL's model is addressed agent-to-agent, which is exactly the shape a KV-cache handoff needs and an all-reduce does not.
2.2 Backends: two actively supported, more in development
Of the backends the Transfer Backend Interface can select, only two are labelled actively supported as of this writing: UCX and NVIDIA Magnum IO GPUDirect Storage (GDS); other filesystem, block, and object-storage backends are explicitly listed as in development2. Through those two backends, NIXL reaches the same physical transport protocols Module 1 built the bandwidth ladder from — RoCE, InfiniBand, GPUDirect RDMA, NVMe-oF, TCP, and NVLink2 — as a uniform call regardless of which one answers on the wire. This is a narrower backend list than Mooncake's own Transfer Engine already ships (§3), which matters directly for how mature each option is to depend on today.
GDS is NVIDIA's GPUDirect Storage product, named explicitly in NIXL's own docs as "NVIDIA Magnum IO GPUDirect Storage"2 — the same GPUDirect product family whose GPU-to-GPU member, GPUDirect RDMA, Module 1 covered as the mechanism that removes the two host-side copies from a cross-node transfer. GDS extends that branding to GPU-to-storage transfers instead of GPU-to-GPU ones: a NIXL transfer that names a file or block device as one endpoint can reach it through GDS as a registered backend rather than being folded into a generic "disk" option the way LMCache's own local-disk backend is (Table 1). That it merits its own named backend, distinct from the storage types Memory Sections otherwise unify generically, is itself a signal that the file/block/object backends still listed as "in development"2 are not simply thinner wrappers around GDS — each storage type is its own integration effort.
2.3 NIXL as shared substrate across three engines
NIXL's stated design goal — one library other inference frameworks depend on rather than each reimplementing point-to-point transfer — is not just aspirational; three independently built serving engines route KV transfer through it as of 2026-09-17:
- Dynamo uses it as the default KV-transfer path: "Dynamo leverages NIXL to transfer KV cache directly from the VRAM of the prefill engine to the VRAM of the decode engine. The KV transfer is non-blocking, allowing GPU forward passes to continue serving other requests during the transfer"7.
- TensorRT-LLM's cache-transceiver (
KvCacheTransceiverV2) defaults to NIXL as its backend; NIXL's own internal transport is then selected by the environment variableTRTLLM_NIXL_KVCACHE_BACKEND, valid valuesUCX(the default) orLIBFABRIC, with an unsupported value logging a warning and falling back to UCX8 — a detail Module 4 already covered from the connector side; here it is the same fact from NIXL's side of the interface. - SGLang ships NIXL as one of exactly two transfer backends, alongside Mooncake: installed via
pip install nixl, using UCX by default with LIBFABRIC available through theSGLANG_DISAGGREGATION_NIXL_BACKENDenvironment variable9. Unlike Dynamo and TensorRT-LLM, SGLang does not default to NIXL — it exposes both NIXL and Mooncake as configuration choices, discussed further in §3.
Three different projects, three different levels of commitment to NIXL — default-only (Dynamo), default-with-fallback (TensorRT-LLM), and equal-choice (SGLang) — but all three depend on the same three abstractions from §2.1 rather than each writing their own. Because these are three separately maintained codebases pointing at the same upstream library, a fix like the v1.4.1 use-after-free closure discussed next (§2.4) benefits all three at once as each project bumps its vendored NIXL version — the alternative, three independent reimplementations of point-to-point transfer, would have needed the same class of bug found and fixed three separate times, on three separate schedules.
2.4 Version churn as evidence of a young library
NIXL's release history over roughly one month (through v1.4.1, dated 2026-09-01) is dense with both bug fixes and internal format changes10:
- Cross-node EFA hang fixed
A cross-node AWS EFA transfer hang introduced in v1.3.1 was fixed through a per-rail flush strategy10.
- Stride descriptors + tracing
Introduced stride (compressed) descriptors, cutting descriptor-list memory from roughly 44 MB to roughly 7 KB, and added a
nixl::tracetracing framework with an NVTX backend for Nsight Systems integration10. - Use-after-free closed
Closed a use-after-free window in the transfer-request path and restored the
num_expertsparameter for CUDA-graph reuse across rank changes10.
The stride-descriptor change alone is worth pausing on: shrinking a per-transfer descriptor structure from roughly 44 MB to roughly 7 KB is a reduction of approximately \(44{,}000{,}000 / 7{,}000 \approx 6{,}300\times\) — the kind of change that only makes sense if the previous representation was carrying far more redundant metadata than the transfer itself needed, which is the sort of inefficiency a library still finds and fixes in its first couple of years, not one still present after a decade of production hardening. The num_experts fix in the same release is worth flagging for Module 5's audience specifically: it means a CUDA graph capturing a wide-EP dispatch/combine pattern can be reused safely across a rank-count change without recapturing10 — exactly the situation EPLB's periodic redundant-expert rebalancing creates. Separately, that same v1.4.0 release added the nixl::trace tracing framework with an NVTX backend for Nsight Systems integration10; no equivalent tracing integration for Mooncake's Transfer Engine appears in any source reviewed for this module, which is itself a data point on relative operational tooling maturity, not just on backend count.
3. Three transport designs, one job
By this point three different movers have appeared across this module and Module 7: NIXL (§2), Mooncake's Transfer Engine, and LMCache's own eight-backend plugin list (§1.2), which itself can dispatch through either of the other two. All three solve the same underlying problem — move KV-cache bytes from one place to another fast — at three different levels of generality, and the field has not converged on one.
Mooncake's Transfer Engine, covered in full in Module 7, already ships a broader raw transport list than NIXL exposes today: the repository names TCP, RDMA, AWS EFA, NVMe-oF, NVLink, HIP, Barex, CXL, and Ascend-family transports as supported11. It ships bundled as one component of the Mooncake project, alongside Mooncake Store (the distributed KV-cache storage engine) and Mooncake EP & Process Group (fault-tolerant execution for large-scale MoE)11, and it is adopted directly as a named backend by vLLM, SGLang, and, since June 2025, LMDeploy11.
NIXL takes the opposite starting point: a generic abstraction over memory and storage first, with only two backends — UCX and GDS — actively supported today and the rest explicitly labelled in development2. What it trades for that narrower backend list is independence from any one storage system's data model: three separately built engines (Dynamo, TensorRT-LLM, SGLang) depend on it as a default or alternate KV-transfer backend (§2.3) without any of them needing to adopt Mooncake's KVCache/Store abstractions to do so.
LMCache's own backends are not really a third transport design at all. For a single-host deployment, LMCache moves bytes through its own CPU RAM, local-disk, Redis/Valkey, or S3 backends without depending on either NIXL or Mooncake1. For a higher-performance or cross-node deployment, it dispatches the same offload/reuse decision through NIXL or through Mooncake Storage as two of its eight backend options (§1.2)1. LMCache is a policy layer that composes with either of the other two rather than competing with them for the same job.
3.1 Two concrete choices
Put concretely, using two deployments this course already sets up: a team standing up wide expert-parallel decoding (Module 5) across GB200 NVL72 racks on TensorRT-LLM inherits NIXL as the cache-transceiver's default with zero extra configuration8 — swapping in Mooncake's Transfer Engine instead would mean replacing a working default to gain a transport list none of this module's sources show that deployment needing beyond what NIXL's UCX backend already reaches. A team that has already built cross-node prefix caching on Mooncake's KVCache-centric Store (Module 7) and is now adding SGLang to the same cluster gets an actual choice, not a default to accept or override: SGLang's own docs list both Mooncake and NIXL as configuration options9, and picking Mooncake keeps the KV-transfer path on the same data model already running in production, at the cost of not gaining NIXL's broader multi-consumer surface for anything the cluster might add beyond that one engine.
| Dimension | NIXL | Mooncake Transfer Engine | LMCache's own backends |
|---|---|---|---|
| Scope | generic memory + storage abstraction2 | bundled with Mooncake's KVCache-centric Store11 | reuse/offload policy layer, not a transport itself1 |
| Actively-supported backends | UCX, GDS; others in development2 | TCP, RDMA, EFA, NVMe-oF, NVLink, HIP, Barex, CXL, Ascend11 | CPU RAM, disk, Redis/Valkey, S3 — plus NIXL, GDS, Mooncake as pass-through options1 |
| Coupling | independent library, no owning storage system2 | coupled to Mooncake's own KVCache data model11 | engine-agnostic by design; not coupled to one store1 |
| Adopted as KV-transfer backend by | Dynamo (default)7, TensorRT-LLM (default)8, SGLang (alternate)9 | vLLM, SGLang, LMDeploy11 | vLLM (LMCacheConnectorV1), Dynamo since Sep 20251 |
| Recent maturity signal | v1.3.2 → v1.4.1 in about one month, active bug fixes10 | USENIX FAST 2025 Best Paper; production since Kimi's launch (Module 7) | production tutorial in vLLM's own reference stack6 |
Reading the "Adopted as KV-transfer backend by" row across both columns is the closest thing to a convergence signal this module has: NIXL is the default for every project NVIDIA either controls or ships an official backend for (Dynamo, TensorRT-LLM), while Mooncake's Transfer Engine is additionally available wherever a team already runs Mooncake's cross-node KVCache store and wants a mover that already speaks its data model (SGLang, LMDeploy, and vLLM's own MooncakeConnector sitting alongside its NixlConnector per Module 4). vLLM and SGLang appear on both sides of the table for the same reason: both shipped connector support for NIXL and for Mooncake, and let the deployer pick, rather than the project itself picking one.
Where experts disagree
Table 2 already lays out the factual differences; the open question is which design point is actually correct for a KV-transfer layer — bundle the mover with a specific KVCache system the way Mooncake does, or keep it generic and let independent projects adopt it the way NIXL does. Neither side has published a controlled benchmark against the other, so this is a disagreement about architecture philosophy backed by adoption evidence on each side, not about measured numbers.
Purpose-built and bundled wins for the common case
Implicit in Mooncake's own design and its direct adoption by three serving engines11, cross-referenced against Module 7
Mooncake ships its Transfer Engine already supporting nine named transports — TCP, RDMA, AWS EFA, NVMe-oF, NVLink, HIP, Barex, CXL, Ascend-family11 — bundled with a KVCache-centric Store that already knows how to hash, place, and evict the specific data being moved. A team that has already adopted Mooncake's architecture for cross-node prefix caching (Module 7) gets the transfer mechanism for free, tuned to the same data model, with no separate abstraction layer to learn or debug. The implicit argument is that a KV-transfer layer's main job is moving one specific kind of data — paged KV blocks belonging to a content-addressed cache — well, and that optimizing for that one job beats optimizing for generality across jobs the deployment may never actually have.
- One component to deploy and operate instead of two
- Transports already cover AMD (HIP), Huawei Ascend, and CXL — a wider hardware surface than NIXL's current two active backends
- Production-hardened since Kimi's launch and a FAST 2025 Best Paper11
- Coupled to Mooncake's own KVCache/Store data model; moving something that is not Mooncake KVCache means stepping outside its scope
- Adopting it for transfer alone still means adopting a specific project's abstractions, not a neutral one
A generic, pluggable abstraction is the correct layer
NIXL's own stated design goal2, and its adoption as default or alternate backend by Dynamo7, TensorRT-LLM8, and SGLang9
NIXL was built explicitly to sit underneath multiple independent frameworks rather than one project's own store, unifying memory and storage behind Memory Sections and letting the Transfer Backend Interface pick UCX or GDS underneath2. Three separately engineered serving stacks now depend on the same abstraction, which means bug fixes and backend improvements amortize across all three instead of being re-solved per project. The implicit argument is the opposite bet from Mooncake's: that a cluster's transfer needs rarely stay confined to one data model forever — KV transfer today, weight streaming or storage tiering tomorrow, possibly across engines from different vendors — and that paying the cost of a thinner backend list now is worth not having to re-architect the transport layer when the second use case shows up.
- No coupling to any one KVCache data model — usable for KV transfer, weight streaming (Module 9's ModelExpress), or storage tiering alike
- Three independent, competing serving engines already default or alternate to it, evidence the abstraction is doing its job across different codebases
SGLang's own decision to ship both backends behind a configuration flag rather than pick one9 is the most honest signal available: as of 2026-09-17, no source reviewed for this module publishes a controlled, head-to-head benchmark moving identical KV blocks over identical hardware through both the NIXL-UCX path and the Mooncake Transfer Engine path, so neither position is settled by evidence rather than by which project you already run. The practical decision collapses to what else is already in the cluster: if Mooncake's KVCache-centric Store (Module 7) is already deployed for cross-node prefix caching, its Transfer Engine is the lower-friction choice because the data model is already shared. If the requirement is one transport layer serving multiple, unrelated consumers — KV transfer today, weight streaming or storage tiering tomorrow, across engines that do not share a KVCache abstraction — NIXL's genericity is the better bet, which is exactly why the NVIDIA-controlled projects in this ecosystem (Dynamo, TensorRT-LLM) standardize on it by default. What would settle the argument is the missing benchmark: identical hardware, identical KV payload, both transfer paths, measured throughput and tail latency.
What is changing
Three things in this module are most likely to be stale within a year. First, NIXL's backend surface: the architecture doc already names filesystem, block, and object-storage backends as "in development" alongside the two actively supported today2 — expect that list to grow, which would narrow the transport-list gap with Mooncake's Transfer Engine that §3 documents as of this writing. Second, LMCache's integration into Dynamo is under one year old as of this module's access date (September 2025)1; Module 9 covers where it lands inside Dynamo's KV Block Manager, and that boundary is still being drawn. Third, the release cadence itself — three NIXL versions in roughly a month, each with functional bug fixes rather than only cosmetic changes10 — means any specific version number or environment-variable default cited in this module should be re-checked against the live docs before being relied on in a deployment, not assumed to still hold. Fourth, and least likely to resolve quickly: the NIXL-versus-Mooncake choice §3 and the disagreement above document as unsettled is unsettled precisely because the deciding benchmark does not exist yet. Nothing in either project's own release notes suggests that benchmark is planned; expect this to remain a judgment call based on what a team already runs, not a data-driven decision, for the foreseeable future.
References
- repoLMCache contributors. LMCache/LMCache (README). Main branch, 2026. https://github.com/LMCache/LMCache. Accessed 2026-09-17. Used for: "AI-native knowledge" framing, full storage-backend list (CPU RAM, disk, Redis/Valkey, Mooncake, InfiniStore, S3, NIXL, GDS), vLLM V1 multimodal and Dynamo integration dates.
- officialNVIDIA (ai-dynamo). NIXL Architecture (nixl.md, main branch docs). 2026. https://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md. Accessed 2026-09-17. Used for: NIXL's design goal, the three abstractions (Memory Sections, Transfer Backend Interface, Metadata Handler), actively-supported backends (UCX, GDS) vs. in-development, transport protocol list.
- officialLMCache contributors. Example: Offload KV cache to CPU (LMCache Documentation). docs.lmcache.ai, 2026. https://docs.lmcache.ai/getting_started/quickstart/offload_kv_cache.html. Accessed 2026-09-17. Used for: offload trigger mechanism, 256-token default chunk size, quickstart backend list, vLLM KVTransferConfig example, 5.0 GB / 1.5-GB-per-10k-tokens sizing example, ~7.43x second-run speedup.
- officialvLLM contributors. Disaggregated Prefilling (vLLM Documentation). docs.vllm.ai/en/latest/features/disagg_prefill. https://docs.vllm.ai/en/latest/features/disagg_prefill/. Accessed 2026-09-17. Used for: the general Connector / kv producer / kv consumer / LookupBuffer (insert, drop_select) / Pipe abstraction that LMCache's kv_role=kv_both setting collapses onto one instance.
- paperQin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079v4, USENIX FAST 2025 Best Paper. https://arxiv.org/pdf/2407.00079. Accessed 2026-09-17. Used for: the 512-token content-hash block size, cross-referenced against LMCache's own 256-token chunk.
- repovLLM Project contributors. vllm-project/production-stack (README). Released 2026-01-22. https://github.com/vllm-project/production-stack. Accessed 2026-09-17. Used for: LMCache offload tutorial listed as a named, first-class tutorial in vLLM's own reference production stack.
- officialNVIDIA. Disaggregated Serving (NVIDIA Dynamo Documentation). docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. Accessed 2026-09-17. Used for: NIXL as Dynamo's default KV-transfer path, non-blocking VRAM-to-VRAM transfer.
- officialNVIDIA / TensorRT-LLM contributors. Disaggregated Serving (disagg-serving.md). TensorRT-LLM documentation, main branch. https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disagg-serving.md. Accessed 2026-09-17. Used for: KvCacheTransceiverV2 defaulting to NIXL, TRTLLM_NIXL_KVCACHE_BACKEND (UCX default / LIBFABRIC) selection and fallback behavior.
- officialSGLang contributors. PD Disaggregation. SGLang Documentation, docs.sglang.io/advanced_features/pd_disaggregation.html. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: NIXL as one of SGLang's two transfer backends (alongside Mooncake), pip install nixl, UCX default / LIBFABRIC via SGLANG_DISAGGREGATION_NIXL_BACKEND.
- repoNVIDIA / ai-dynamo contributors. ai-dynamo/nixl Releases. v1.4.1, 2026-09-01. https://github.com/ai-dynamo/nixl/releases. Accessed 2026-09-17. Used for: v1.3.2 EFA per-rail flush fix, v1.4.0 stride descriptors (~44 MB → ~7 KB) and nixl::trace/NVTX integration, v1.4.1 use-after-free fix and num_experts CUDA-graph fix.
- repoMoonshot AI / MADSys contributors. kvcache-ai/Mooncake (README). Main branch, updates through August 2026. https://github.com/kvcache-ai/Mooncake/blob/main/README.md. Accessed 2026-09-17. Used for: Transfer Engine's supported transport list (TCP, RDMA, EFA, NVMe-oF, NVLink, HIP, Barex, CXL, Ascend), bundling with Mooncake Store and Mooncake EP/Process Group, vLLM/SGLang/LMDeploy adoption.