Multi-Node LLM Inference: From One Node to N · Part 4: Multi-Node KV Cache Management · Module 7
Prefix Caching and Tiered KV Offload Across Nodes: Mooncake's Architecture
Every prior module in this course treated KV cache as something that lives inside one replica and gets routed around or transferred once, at the prefill/decode boundary. Mooncake treats it as a cluster-wide resource: paged blocks that are content-addressed, tiered across GPU HBM, host DRAM, local SSD, and remote nodes, scheduled by a component (Conductor) that is as central to the system as the model-parallel plan. This module works through that architecture — the four-step request workflow, the scheduling algorithm, the measured transfer bandwidth, and the production evidence — from the FAST'25 Best Paper and Mooncake's current open-source repository. After reading, you can explain why a serving cluster benefits from treating KV cache as a first-class, hash-addressed, tiered resource rather than private per-replica state, and reproduce the arithmetic behind Mooncake's own numbers.
1. KV cache as a scheduled resource, not private state
Mooncake is the serving platform for Kimi, Moonshot AI's LLM product, and the KVCache-centric architecture described in its FAST 2025 Best Paper is not a research prototype — it is the system that has run Kimi's production traffic since before the paper was written1. The paper's own framing of the problem is worth stating precisely, because it explains why Conductor (the global scheduler) is as architecturally central as the model-parallel plan: the scheduling of KVCache is central to LLM serving scheduling. To improve overall throughput, there are typically two general approaches: 1) reuse KVCache as much as possible to reduce the required computation resources; and 2) maximize the number of tokens in each batch to improve the Model FLOPs Utilization (MFU)
1. Those two goals pull in different directions — reusing a cache stored on a distant node prolongs time-to-first-token (TTFT), and a large batch increases time-between-tokens (TBT) — and the paper's stated conclusion is that "the utilization of both these throughput-oriented optimizations may lead to violations of latency-related SLOs" unless something schedules across them jointly1.
Module 2 already covered one answer to the reuse question: route the request to whichever replica's local KV cache already has the matching prefix, using approximate block hashing or precise KVEvent streaming, and never move a byte of cache at all. That works when prefill and decode run on the same GPUs. Mooncake's starting point is different: prefill and decode are already separated onto physically distinct instances (Module 3 covers why that split pays off), which means the KV cache computed by a prefill instance is never local to the decode instance that needs it — routing alone cannot solve the problem, because there is no single replica that holds both the prefill history and the decode state. Something has to move the cache, and once you have a component that moves cache between nodes anyway, the natural next step is to make that movement general enough to also serve cross-request, cross-node prefix reuse, not just the mandatory prefill→decode handoff. That generalization is what makes the KVCache pool "distributed" rather than merely "a transfer channel."
Figure 1 lays out the resulting architecture. A prefill instance's optimization goal is to maximize cache reuse subject to the TTFT SLO, a minimum MFU floor, and the constraint that its reserved KVCache footprint fit in host DRAM; a decode instance's goal is to maximize aggregate batch throughput subject to the TBT SLO and the constraint that the aggregated KVCache for its batch fit in GPU VRAM1. Table 1 makes that asymmetry explicit — it is the same prefill/decode split Module 3 derives from queueing theory, restated here as the two halves of one scheduling problem that Conductor has to solve jointly, because a decision that is locally optimal for one pool (route to the node with the longest matching prefix) can be globally wrong if that node is already overloaded.
| Pool | Objective | Constraints |
|---|---|---|
| Prefill | Maximize KVCache reuse | TTFT SLO; minimum MFU; KVCache footprint < host DRAM |
| Decode | Maximize throughput (batch size) | TBT SLO; aggregated KVCache < GPU VRAM |
The rest of this module works through how that joint problem is actually solved: the tiered pool the cache lives in (§2), how blocks are addressed and deduplicated (§3), the four-step workflow a request goes through (§4), Conductor's scheduling algorithm (§5), the multi-node prefill mechanism that keeps the prefill pool elastic under long context (§6), what happens when the cluster is overloaded (§7), the production trace evidence behind the paper's headline numbers (§8), and the Transfer Engine that moves the bytes, plus the two newer repository components built on top of it (§9).
2. The offload-tier hierarchy, concretely
Module 1 named the generic tiering hierarchy — GPU HBM, host DRAM, local SSD, remote storage — and deferred the actual bandwidth numbers to this module1. Mooncake's own architecture is worth reading precisely here, because it does not draw four cleanly separated tiers with four distinct bandwidth figures; it draws two. Figure 1 of the paper labels the per-instance layer "GPU/VRAM" (the paged KVCache actively participating in a forward pass) and everything below it "CPU/DRAM/SSD" as one box: the "Distributed KVCache Pool"1. Inside that pool, blocks are cached as paged blocks in CPU memory and evicted under one of three interchangeable policies — LRU, LFU, or a length-aware variant that behaves like LFU but favors blocks occurring later in a request1 — with SSD serving as the overflow tier for whatever does not fit in DRAM. The paper does not separately benchmark DRAM-tier versus SSD-tier bandwidth; both fall inside the same "underutilized CPU, DRAM, and SSD resources of the GPU cluster" that the disaggregated cache harvests1. That is a deliberate architectural choice, not a gap in the paper: from Conductor's point of view, the pool is one logical store with one eviction policy, and the tier boundary inside it (DRAM-resident vs. SSD-resident) is an implementation detail of the storage backend, not something the scheduling algorithm reasons about directly.
The one tier boundary Mooncake does measure precisely, because it is the one that crosses a physical machine, is the jump from a node's own KVCache pool to another node's — the "Inter-node KVCache Transfer" arrow in Figure 11. That transfer is handled by Messenger, a separate GPUDirect-RDMA-based component deployed on every node, running as its own process within each inference instance1. The repository's own benchmark of that hop, run with 40 GB of data (the size of a 128k-token KVCache on a LLaMA3-70B-class model), reports up to 87 GB/s of achieved bandwidth on a 4×200 Gbps RoCE network and up to 190 GB/s on an 8×400 Gbps RoCE network2 — the numbers this module's title promises, and the subject of §9's full treatment. The paper's own testbed used RDMA NICs supporting "up to 800 Gbps of interconnect bandwidth" per node1, roughly 100 GB/s, consistent with the 4-rail figure.
3. Content-addressed KVCache blocks
Inside the pool, KVCache is stored as paged blocks of 512 tokens each, and every block carries a hash value determined by both its own content and its prefix1. Concretely, for a prompt split into consecutive token blocks a, b, c, …, the hash of the first block is A = Hash(a), and every subsequent block's hash folds in the previous block's hash: B = Hash(A+b), C = Hash(B+c), and so on1. A request's block-hash chain is then compared, one hash at a time, against a prefill instance's cached keys to find the longest matching prefix; the paper calls this quantity prefix_len1.
The hash-of-hash chaining is the mechanism that makes this a genuinely different structure from the single-engine radix tree Aman already knows from vLLM/SGLang prefix caching: it is not that Mooncake's hashing is more sophisticated, it is that the hash itself is the cross-node addressing key. A local radix tree only needs to answer "does this node have this prefix," because the tree and the KV blocks it points to live in the same process. Mooncake's hash has to be globally comparable across every prefill instance and the shared pool, so that Conductor can ask "which instance, if any, anywhere in the cluster, has the longest matching prefix for this exact hash chain" before a single token of the request has been assigned anywhere1. The open-source request trace released with the paper stores exactly these values — each record's hash_ids field is the list of block hashes (block size 512) covering the request's input, remapped to globally unique integers for privacy1. Two records sharing the same leading hash IDs are provably sharing the same KVCache for that many blocks; the paper's own example shows two trace entries sharing their first 12 hash IDs, meaning they can share prefix caching for the first 12 × 512 = 6,144 tokens1.
4. The four-step per-request workflow
Once tokenizing is finished, Conductor selects a prefill instance (or instance group) and a decode instance and starts a workflow with four steps1.
- KVCache reuse. The prefill node receives the raw input, the block IDs of the prefix cache it can reuse, and the block IDs allocated for the full request; it loads the prefix cache from remote CPU memory into GPU memory based on those block IDs, skipped entirely if no prefix cache exists1. This selection already balances three objectives — reusing as much KVCache as possible, balancing load across prefill instances, and meeting the TTFT SLO — which §5's scheduling algorithm formalizes.
- Incremental prefill. The prefill node completes the prefill stage using the reused prefix and stores the newly generated incremental KVCache back into CPU memory. If the number of uncached input tokens exceeds
prefill_chunk— a threshold "typically larger than 1000 tokens," chosen to fully utilize the prefill GPU's computational power — the stage is split into chunks executed in a pipelined manner across the prefill node group, the mechanism §6 covers in full1. - KVCache transfer. Messenger, deployed as its own process on every node, transfers the cache to the destination decoding node's CPU memory. This step is asynchronous and overlapped with step 2: as each transformer layer's incremental KVCache finishes computing, its store-and-transfer is launched immediately rather than waiting for the whole prefill pass to complete, which is what keeps a long-context request's transfer time from simply adding on top of its compute time1.
- Decoding. Once all KVCache has arrived in the decode node's CPU DRAM, the request joins the next continuous-batching iteration. Conductor pre-selects the decode instance based on its current load to avoid violating the TBT SLO, but the local scheduler on that instance double-checks the SLO at admission time, because the anticipated load may have shifted during the prefill stage — and this double-check can itself reject the request, wasting the prefill compute already spent1. §7 covers how Mooncake tries to avoid paying that cost.
The asynchronous overlap in step 3 is implemented with launch-and-wait semantics: before a layer's attention computation begins, the model waits only for that layer's own asynchronous KVCache load to complete and immediately triggers the next layer's load; after a layer's attention finishes, its KVCache store is launched asynchronously without blocking forward progress, and the prefill instance only waits for all outstanding stores to finish once every layer's computation is done1. Measured against a serialized baseline (compute the full prefill, then store all KVCache), this layer-wise overlap cuts the added latency from storing KVCache dramatically at every tested context length from 8k to 128k tokens — the paper's own bar chart shows the layer-wise bars pinned near zero added latency across the whole range, while the serialized bars grow from roughly 0.1 s at 8k tokens to nearly 0.9 s at 128k1. The practical consequence the paper draws from this is one worth keeping: because the overlap makes the prefill instance's execution time roughly equal to either the KVCache loading/storing time or the standard prefill time (whichever dominates), Conductor's prefill scheduling can disregard available VRAM headroom entirely and consider only KVCache distribution and available DRAM capacity1 — VRAM pressure, the thing that usually dominates single-engine KV-cache scheduling, stops being the prefill pool's binding constraint.
5. Conductor: the KVCache-centric scheduling algorithm
Previous work on LLM-serving schedulers typically load-balances on the number of assigned requests alone. Mooncake's paper is explicit that this is insufficient once KVCache reuse is on the table: the selection of a prefill instance has to weigh "not just load but also the prefix cache hit length and the distribution of reusable KVCache blocks," because there is a real tension between routing to the instance with the longest prefix match (cheapest to compute, but possibly already overloaded) and routing to a different, lightly loaded instance (more balanced, but forces more incremental prefill)1. Algorithm 1 in the paper is Conductor's answer, and it is precise enough to reproduce.
Input: prefill pool P, decoding pool D, request R, cache block size B
Output: the (prefill, decode) instance pair (p, d) for R
block_keys <- PrefixHash(R.prompt_tokens, B)
TTFT <- infinity
p <- None
best_prefix_len, best_matched_instance <- FindBestPrefixMatch(P, block_keys)
for instance in P:
prefix_len <- instance.prefix_len
T_queue <- EstimatePrefillQueueTime(instance)
if best_prefix_len / prefix_len < kvcache_balancing_threshold:
# cache-aware prefill scheduling: this instance's own match is
# close enough to the cluster-wide best -- do not pay a transfer
T_prefill <- EstimatePrefillExecutionTime(len(R.prompt_tokens), prefix_len)
if TTFT > T_queue + T_prefill:
TTFT <- T_queue + T_prefill
p <- instance
else:
# cache-aware AND balancing: pull the remote best-match cache here
transfer_len <- best_prefix_len - prefix_len
T_transfer <- EstimateKVCacheTransferTime(instance, best_matched_instance, transfer_len)
T_prefill <- EstimatePrefillExecutionTime(len(R.prompt_tokens), best_prefix_len)
if TTFT > T_transfer + T_queue + T_prefill:
TTFT <- T_transfer + T_queue + T_prefill
p <- instance
d, TBT <- SelectDecodingInstance(D) # load-balancing decoding scheduling
if TTFT > TTFT_SLO or TBT > TBT_SLO:
reject R; return
if best_prefix_len / p.prefix_len > kvcache_balancing_threshold:
TransferKVCache(best_matched_instance, p) # KVCache hot-spot migration
return (p, d)
Three mechanisms are worth pulling out of that loop. First, the block-hash matching itself (PrefixHash, FindBestPrefixMatch) is the same content-addressed chain from §3, compared one block at a time against every prefill instance's cache keys; the paper notes plainly that "similar reuse logic is already implemented in vLLM, but the open-source version of vLLM only supports local KVCache caching"1 — Conductor's contribution is running that same match cluster-wide rather than per-process. Second, TTFT for a candidate instance is not just execution time; it adds the instance's current queueing time, computed by summing the estimated prefill time of every request already queued ahead of it, so a request never gets routed to an instance with the best cache match if that instance is backed up enough to blow the TTFT SLO anyway1. Third, execution-time prediction itself is a learned model: prefill duration is estimated from offline test data as a function of request length and prefix-hit length, and because transformer compute follows a regular pattern, this predictor's error bound stays small given enough offline data — transfer-time prediction is harder, the paper says, because it depends not just on payload size but on whether the sending node's network path is currently congested1.
The kvcache_balancing_threshold comparison is the load-vs-locality trade-off made explicit as a single tunable ratio: if the best available prefix match anywhere in the cluster is not meaningfully longer than what a candidate instance already has locally (the ratio stays under threshold), Conductor keeps the request local and skips the transfer; only once the gap becomes large enough does it pay to pull the remote cache across the network1. The same idea appears twice more, once for the decoding side (a separate load-balancing scheduler simply picks the least-loaded decode instance rather than doing any cache matching, because decode's KVCache is already resident wherever the prefill handoff landed it1) and once as a standing background process: cache load balancing and hot-spot migration.
5.1 Cache load balancing and hot-spot migration
Cache popularity in a real MaaS deployment is heavily skewed — a system prompt is hit by nearly every request, while a cache built from one user's long document may be read once1. A request may not always get routed to the instance with the single longest prefix match, because that instance may already be overloaded; in that case Conductor forwards both the cache's location and the request to an alternative instance, which proactively retrieves and locally stores the remote KVCache, if the estimated additional prefill time this saves is shorter than the transfer time it costs1 — exactly the TransferKVCache call at the bottom of Algorithm 1. The paper considered and rejected a straw-man alternative: collecting global per-block usage statistics and forecasting future demand with a prediction model. Because a MaaS provider's workload is highly dynamic and its user base grows quickly, accurately predicting per-block future usage was judged infeasible; Mooncake instead uses a heuristic-based automated hot-spot migration scheme that replicates hot blocks without requiring precise future-usage predictions1. Both effects — forwarding a request to a lightly loaded instance below a size-based transfer threshold, and hot-spot replication as a side effect of that forwarding — reduce prefill time for requests while automatically spreading hot caches across more machines1. In a controlled 8-prefill/8-decode-instance experiment replaying 23,000 real requests, Conductor's combined cache-aware-plus-balancing strategy produced a lower and tighter TTFT distribution than either cache-aware scheduling alone, load-balancing scheduling alone, or random assignment1.
6. Multi-node prefill: why the prefill pool needs its own parallelism
Section §5 of the paper opens by asking, directly, whether Mooncake's separate prefill pool is still necessary given chunked prefill — the same question Module 3 covers in depth from the interference-and-goodput angle. Restated narrowly here: the necessity and best practices for designing a separate and elastic prefill pool remain under debate… it is worth discussing whether this separation is still necessary with the introduction of chunked prefill
1. The paper's answer is to keep the disaggregated architecture, for two reasons, and this section is the concrete mechanism behind the first of them: prefill nodes require different cross-node parallelism settings to handle long contexts, a need that has nothing to do with interference and everything to do with how a single long-context prefill gets enough GPUs to hit its TTFT SLO1. (The second reason, VRAM savings, is the direct consequence of the four-step workflow in §4: because a prefill instance never needs to hold a completed request's KVCache once it has been transferred out, its VRAM budget is decoupled from decode's.)
Context lengths for recent models run from 8k up to 128k and beyond, and for such requests input tokens routinely outnumber output tokens by 10–100×, which makes optimizing TTFT — not TBT — the dominant concern for long-context prefill1. Using more than one 8-GPU node to prefill such a request in parallel is desirable, but the paper rules out two standard options for different reasons. Extending tensor parallelism (TP) across more than one node requires two RDMA-based all-reduce operations per layer, which significantly reduces prefill MFU1 — the cross-node TP tax Module 1 already established structurally (a TP all-reduce crossing the NVLink-to-RDMA boundary pays the bandwidth cliff on every layer, not once per request). Sequence parallelism (SP, e.g. Ring Attention or Striped Attention) partitions the input sequence itself across nodes and needs cross-node communication only once per layer rather than an all-reduce, which cuts network consumption and improves MFU relative to cross-node TP — but SP still yields worse MFU than single-node TP, and a deployment that tries to run every prefill node in one SP group forces a static parallelism choice that leaves the cluster underutilized whenever request lengths don't need it1. Dynamically resizing the SP group per request (elastic SP) is possible in principle but, the paper notes, requires establishing a global communication group in advance and complicates Conductor's design when cache-reuse utilization and SLO violations also have to factor into that same resizing decision1 — in short, elastic SP trades a scheduling problem Conductor already solves (§5) for a much harder one that couples parallelism topology to cache placement.
Mooncake's actual choice is chunked pipeline parallelism (CPP): group every X nodes in the prefill cluster into a pipelined node group, partition a request's input tokens into chunks no longer than prefill_chunk (§4), and process different chunks of the same request simultaneously on different nodes in a pipeline1. This is standard pipeline parallelism applied to a single request's prefill rather than to a training batch, and it offers two benefits the paper is explicit about: cross-node communication is needed only at pipeline-stage boundaries, which overlaps easily with computation and contends far less with the concurrent KVCache transfer traffic (§9) than SP's per-layer communication does; and CPP fits both short and long contexts without the static-partitioning overhead SP's node grouping requires, because a short request that never exceeds one prefill_chunk simply never engages the pipeline1. The paper flags this as, to its knowledge, the first application of pipeline-based prefill acceleration to the inference stage — the technique itself is borrowed from training systems, where it has been explored before, but long-context inference prefill has only recently emerged as a workload large enough to benefit from it1.
Combined with the layer-wise streaming from §4 — which lets a prefill instance's KVCache store-and-transfer run fully overlapped with its own compute — CPP is what makes "prefill nodes require different cross-node parallelism settings" a structural argument rather than a preference: a decode instance never needs more than one node's worth of parallelism for the model itself (Module 5's wide-EP and Module 3's placement discussions aside), but a prefill instance serving a 128k-token request benefits from spreading that one request's compute across several nodes in a way decode never does. That asymmetry is exactly what a colocated, non-disaggregated engine cannot express: it has one parallelism plan for the whole model, applied uniformly to both phases.
7. Overload-oriented scheduling: early rejection
Most academic work on LLM serving scheduling assumes every request eventually gets processed and optimizes throughput or latency accordingly. Mooncake's paper explicitly rejects that assumption for a production MaaS provider: cluster capacity grows far more slowly than request volume during rapid user growth, so overload during peak times is routine, and the real question is not "how fast can we serve" but "which requests do we accept at all"1. The paper's stated objective is to maximize overall throughput while meeting SLOs, with one deliberate restriction on what counts: only requests that fully complete execution count toward that throughput measure. A request rejected midway still consumed real prefill compute, and that compute is treated as wasted, not partially credited1. That framing has a direct consequence: since elastically scaling the cluster is typically infeasible on short notice, a request should be rejected as early as possible if it cannot finish under the SLO — late rejection is strictly worse than early rejection, because it wastes more sunk compute1.
Because Mooncake's prefill and decode stages run on physically separate instances, their load can be measured directly rather than inferred through the interference a colocated engine would introduce — system load is defined as SLO satisfaction itself, using the predicted maximum TTFT and TBT for an instance against the SLO thresholds \(l_{ttft}\) and \(l_{tbt}\)1. The naive policy — reject a request only once both stages have already been attempted and the decode instance's load is found to be too high — wastes the prefill compute already spent, because there is a time lag between when prefill starts and when the decode-side load is actually assessed1. Early Rejection moves that assessment forward: on arrival, Conductor evaluates whether to accept a request based on the greater of the prefill and decode pools' current load, before committing any prefill compute1.
Early Rejection on its own introduces a new failure mode: because the decode-load estimate that gates acceptance is itself stale by the time prefill actually finishes, prefill and decode instance loads start oscillating out of phase. The paper's own trace from a 20-machine cluster over 20 minutes shows this directly — when decode load is momentarily read as low, Conductor admits a burst of requests; those requests complete prefill and land on decode together, spiking decode load; Conductor then rejects a burst of new requests until decode load falls again; and the cycle repeats, with prefill and decode load visibly anti-correlated across the 20-minute window, most pronounced in clusters with fewer prefill machines1. Early Rejection Based on Prediction addresses this by predicting decode load after the prefill stage completes, rather than reading current decode load, before deciding whether to admit a request1. The paper considered request-level prediction (estimate each request's output length in advance, which would make TTFT/TBT prediction far more accurate) but rejected it as impractical — output-length prediction is either too costly or too inaccurate, especially under the scarce-resource conditions where accurate prediction matters most1. The deployed strategy instead predicts at the system level: assuming a uniform decode time \(t_d\) per request, Conductor adds requests that prefill will finish by time \(t\) into the decoding pool's projected load, removes requests whose decode is projected to finish before \(t\), and averages the resulting TBT ratio across decode instances to forecast load at \(t\)1.
| Strategy | Requests rejected |
|---|---|
| Baseline (reject after both stages attempted) | 4,183 |
| Early Rejection | 3,771 |
| Early Rejection Based on Prediction | 3,589 |
Moving from the baseline to prediction-based early rejection cuts the rejection count by \((4{,}183 - 3{,}589) / 4{,}183 \approx 14.2\%\) at the same 2× overload-replay speed1 — every one of those roughly 600 additional accepted requests is a request that, under the baseline policy, would have consumed prefill compute and then been thrown away after the decode stage rejected it. The mechanism is entirely about avoiding wasted GPU-seconds under overload, not about improving best-case latency; it only matters once the cluster is at or past capacity, which the paper treats as the normal operating condition for a fast-growing MaaS provider rather than an edge case1.
8. Production trace evidence and end-to-end results
Mooncake's SLO framing for both simulated and real experiments follows directly from the TTFT/TBT distinction already used throughout §1 and §7: a request is measured against \(\text{TTFT}_{P90}\) and \(\text{TBT}_{P90}\) — the 90th-percentile latency relative to a single, uncontended baseline request — and in the paper's own end-to-end experiments these thresholds are fixed at \(\text{TTFT}_{P90} = 10\times\) and \(\text{TBT}_{P90} = 5\times\) the single-request baseline1. Any request that exceeds either threshold counts as an SLO violation and its consumed resources are treated as wasted, consistent with §7's rejection framing.
8.1 The Kimi request trace
To ground this in real traffic rather than only synthetic benchmarks, the paper open-sources a one-hour sample of Kimi's production request stream: 23,608 entries, each carrying a relative arrival timestamp, an input length, an output length, and the block-hash chain from §31. The average input length is 7,590 tokens and the average output length is 182 tokens, giving an average input:output ratio of roughly 720 — a pattern the paper is careful to flag as representative of Kimi's long-context-heavy workload specifically, "not unanimous for all workloads"1.
Running a simple cache-policy analysis against this trace with a single global cache pool, comparing LRU, LFU, and a length-aware variant across cache capacities from 1,000 to unbounded ("Inf") blocks, the hit ratio rises from about 30% at 1,000 blocks to about 50% at 50,000 blocks, with essentially no further gain beyond that — LRU performs best on this dataset, which the paper attributes to temporal proximity in how sessions reuse cache1. The paper is explicit that the ~50% ceiling should not be read as a hard limit on KVCache reusability in general: it is a property of this one-hour sample, and "this reusability highly depends on the application scenario and can be as large as 90% for certain scenarios, such as our chat-to-paper service"1. Separately, the block hit-count distribution is heavily skewed — over 50% of cache blocks in the trace go entirely unused, while a small number are hit tens of thousands of times — which is the empirical justification for the hot-spot replication scheme in §5.11.
| Policy | 1,000 blocks | 10,000 | 30,000 | 50,000 | 100,000 | Inf |
|---|---|---|---|---|---|---|
| LRUCache | 0.30 | 0.40 | 0.48 | 0.50 | 0.51 | 0.51 |
| LFUCache | 0.30 | 0.35 | 0.43 | 0.49 | 0.51 | 0.51 |
| LengthAwareCache | 0.30 | 0.35 | 0.42 | 0.48 | 0.50 | 0.51 |
The arithmetic behind that 30%→50% jump is worth making concrete. At an average input length of 7,590 tokens per request1, a cache hit ratio of \(h\) means, on average, \(h \times 7{,}590\) input tokens per request are served from the KVCache pool instead of recomputed. Moving from \(h=0.30\) to \(h=0.50\) shifts the average reused portion from \(0.30 \times 7{,}590 \approx 2{,}277\) tokens to \(0.50 \times 7{,}590 \approx 3{,}795\) tokens per request — roughly 1,518 additional tokens' worth of prefill compute avoided per request, on average, for a 50× increase in cache capacity (1,000 to 50,000 blocks)1. That the ratio saturates near 50% despite another 2× capacity increase to 100,000 blocks is the direct empirical basis for the paper's claim that required capacity should scale with the actual reuse ceiling of the workload, not grow without bound.
8.2 End-to-end throughput and SLO attainment
Against a four-instance vLLM baseline (vLLM-[4M]) on the public ArXiv Summarization and L-Eval datasets, a Mooncake cluster split three prefill/one decode instance (Mooncake-[3P+1D]) achieves throughput improvements of 20% and 40% respectively while meeting the same SLOs1; on simulated long-context data (16k–128k-token prompts), Mooncake's throughput improvement over vLLM ranges from 50% to 525% at matched TTFT/TBT SLOs1 — the 525% figure quoted in the paper's abstract is the upper end of this simulated range, specifically for the longest-context configurations where vLLM's colocated design suffers most from prefill-decode interference. Under real workload replay — 10 prefill plus 10 decode Mooncake instances against 20 vLLM instances, both handling the same 23,000-request real trace, with a 30-second TTFT cap and a 0.1-second-per-token TBT cap — nearly 100% of Mooncake's requests meet the TBT SLO, against only 57% for vLLM, while both systems' TTFT distributions are nearly identical1. That gap is entirely a decode-side effect: vLLM's colocated engine lets a large batch or an incoming prefill degrade decode's inter-token latency for requests already in flight, while Mooncake's disaggregated decode pool, isolated from prefill entirely, holds TBT stable regardless of prefill load. The paper's headline production claim — Mooncake enables Kimi to handle 75% more requests under real workloads while adhering to SLOs1 — follows directly from this TBT gap: at a fixed SLO bar, a system whose TBT is not systematically inflated by colocated prefill traffic can simply admit more concurrent requests before anyone breaches it.
9. Transfer Engine, Mooncake Store, and Mooncake EP/PG
Everything in §2–8 depends on one component actually moving the bytes: the Transfer Engine, described in the current repository as "the core of Mooncake," a high-performance data transfer framework offering a unified interface for batched data movement across storage, network, and accelerator environments2. It supports multiple simultaneous RDMA NIC devices per transfer to aggregate bandwidth, chooses transfer paths topology-aware (accounting for NUMA affinity between source and destination), and retries automatically over alternative paths on a transient network failure2 — the engineering underneath Messenger's role in §2 and §4.
The 87 GB/s and 190 GB/s figures already used in §2 come from the repository's own benchmark: moving 40 GB of data — the size of the KVCache a 128k-token request would generate on a LLaMA3-70B-class model — the Transfer Engine sustains up to 87 GB/s on a 4×200 Gbps RoCE network and up to 190 GB/s on an 8×400 Gbps RoCE network, roughly 2.4× and 4.6× faster than plain TCP on the same links2. Those multipliers imply the TCP baseline the RDMA path is being compared against, without the repository stating it directly:
$$ \text{TCP}_{4\times200\text{G}} = \frac{87\ \text{GB/s}}{2.4} \approx 36.3\ \text{GB/s} \qquad \text{TCP}_{8\times400\text{G}} = \frac{190\ \text{GB/s}}{4.6} \approx 41.3\ \text{GB/s} $$Two things fall out of that arithmetic. First, TCP throughput barely moves between the 4-rail and 8-rail configurations (36.3 to 41.3 GB/s, a 14% gain) even though the underlying link capacity doubled, while RDMA throughput more than doubles (87 to 190 GB/s, a 2.18× gain) — consistent with TCP's per-connection overhead (kernel copies, protocol processing) becoming the bottleneck well before the wire is saturated, exactly the four-copy tax GPUDirect RDMA exists to remove (Module 1 §3). Second, the RDMA path's multi-rail aggregation is doing real work: going from 4 to 8 rails at the same per-rail rate class should double achievable bandwidth if striping is perfect, and 190/87 ≈ 2.18× is close to that ideal, meaning the Transfer Engine's multi-NIC aggregation (the first bullet point above) is capturing most of the available headroom rather than leaving it on the table.
The Transfer Engine reaches this bandwidth over a broad protocol set — TCP, RDMA, AWS EFA, NVMe-oF, NVLink, HIP, Barex, CXL, and Ascend-family transports — and, when built against the matching accelerator runtime, detects accelerator memory and routes across CUDA, MUSA, HIP, MACA, Cambricon MLU, and Ascend environments2, the multi-vendor breadth that makes it usable outside an all-NVIDIA cluster in a way Module 1's NIXL and DeepEP figures, both Hopper/Blackwell-specific, are not.
Two components have been built on top of the Transfer Engine since the original paper, and both extend the same KVCache-centric idea past request serving: Mooncake Store and Mooncake EP & Process Group (PG).
| Component | Layer | Purpose |
|---|---|---|
| Transfer Engine (TE) | transport | Unified batched data movement across storage/network/accelerator, multi-rail aggregation, topology-aware routing, automatic failover |
| Mooncake Store | storage engine | Distributed KV-cache and model-weight object store built on TE; per-object replica count, placement, soft/hard pin; decouples cache lifetime from engine restarts |
| Mooncake EP & PG | execution | Fault-tolerant, DeepEP-compatible expert-parallel dispatch/combine with rank-activeness awareness; torch.distributed process-group backend with elastic rank recovery |
Mooncake Store is, in effect, the productized and generalized version of the "Distributed KVCache Pool" box in Figure 1: it adds large-object striping and end-to-end zero-copy transfer to fully exploit TE's aggregated multi-NIC bandwidth, a multi-tier DRAM/SSD/NVMe cache design for capacity beyond what §2's blended pool description covers in the original paper, and — the part that goes beyond what the FAST'25 paper describes — programmatic, per-object control over replica count, preferred storage segment, and soft/hard pinning, so a caller can protect a specific KVCache block or model-weight object from eviction rather than relying only on the LRU/LFU/LengthAware policies of §22. Because storage nodes running Mooncake Store are decoupled from any specific inference engine process, they can be added or removed elastically, and cached data survives engine restarts, upgrades, and Conductor's own scheduling decisions2 — the KVCache pool becomes infrastructure with its own lifecycle, not a side effect of whichever engine process happened to compute it.
Mooncake EP and PG target a different failure mode entirely: wide expert-parallel MoE inference (Module 5's subject), where a single failed GPU rank can otherwise stall an entire EP group's dispatch/combine. Mooncake EP keeps its dispatch/combine API close to DeepEP's low-latency mode — deliberately, so an inference engine already integrated with DeepEP does not have to rewrite its MoE communication path — but adds active_ranks awareness so dispatch can route around a known-failed rank rather than hanging on it, while Mooncake PG registers as a standard torch.distributed process-group backend (supporting ordinary collectives like all_gather) with added primitives for peer-state polling and elastic rank recovery, letting a replacement process rejoin an existing group instead of forcing a full restart of the inference service2.
Production adoption has moved briskly since the Transfer Engine was first open-sourced in November 2024. vLLM added official Transfer Engine support for disaggregated prefilling in December 2024 and Mooncake Store support as a cluster-level shared KV-cache backend in May 20262; SGLang added Transfer Engine support for PD disaggregation in April 202532 and Mooncake Store as a hierarchical KV-caching backend (extending RadixAttention across device, host, and remote tiers) in September 20252; LMDeploy adopted Mooncake as a PD-disaggregation backend in June 20252; and TensorRT-LLM integrated the Transfer Engine for KVCache transfer in disaggregated inference in December 20252. NIXL itself (Module 8) lists Mooncake's Transfer Engine as a supported backend plugin as of May 20252 — Mooncake is not only an alternative to NIXL's transport layer, it is also reachable through NIXL, which is one reason vLLM's and SGLang's connector lists (Module 4) name Mooncake and NIXL as siblings rather than strict alternatives.
Where experts disagree
Module 3 already covers the disagreement over whether a second (decode) pool is worth its operational cost at all, including Mooncake's own §5 answer (kept, for the cross-node prefill parallelism reason worked through in §6, and the VRAM-decoupling reason from §4) — that discussion is not repeated here. Given that prefill and decode are already disaggregated, this module raises a narrower, distinct question: once KV cache has to cross a node boundary at all, is it worth building a shared, hash-addressed, cluster-wide pool with a dedicated global scheduler (Conductor), or does a simpler point-to-point handoff — move each request's cache from its prefill instance straight to its assigned decode instance, with no cross-request sharing and no global cache-aware scheduling — capture most of the benefit at far less engineering cost?
The shared pool pays for itself
Mooncake's own production deployment and evaluation12
Once KVCache exists as a transferable object anyway (forced by disaggregation), making it globally addressable and shared costs comparatively little extra machinery — the same Messenger/Transfer Engine that moves prefill→decode handoffs moves cross-request reuse too — and the measured upside is large: up to 525% higher throughput in simulated long-context scenarios, and 75% more requests served under real production traffic at matched SLOs, driven mostly by the decode-side TBT isolation and the fraction of requests that hit a shared prefix cache anywhere in the cluster rather than only on their own replica1. Production adoption across vLLM, SGLang, LMDeploy, and TensorRT-LLM within roughly a year of the paper's release2 is independent evidence that the industry judged the added complexity worth shipping, not just worth publishing.
- Directly measured, large end-to-end gains under real production traffic, not just simulation
- The transfer mechanism (Messenger/Transfer Engine) is needed for the mandatory prefill→decode handoff regardless — global sharing is an incremental cost on top of infrastructure that already exists
- Widely adopted by independent serving engines within about a year
- The 525% figure is the top of a 50–525% simulated range, concentrated in the longest-context configurations — not representative of the median workload
- Conductor's cache-aware scheduling, hot-spot migration, and early-rejection-with-prediction subsystems (§5, §7) are all additional engineering surface a point-to-point design would not need
The reuse ceiling is modest and workload-specific; the complexity is real
Mooncake's own §4.2 and §9 discussion, and the low-complexity alternative Module 2 already establishes1
On the paper's own 23,608-request production trace, cache hit ratio saturates near 50% even with effectively unlimited cache capacity, and the paper states directly that this reusability "highly depends on the application scenario," reaching 90% only for specific workloads like chat-to-paper1. For the median request in a general-purpose deployment, then, roughly half of prefill compute is never avoided by sharing regardless of pool size — and capturing even that 50% required a hash-of-hash content-addressing scheme, a learned execution-time predictor, a heuristic hot-spot replication policy tuned by a currently-manual threshold, and a whole overload-oriented early-rejection subsystem built specifically because the shared-scheduling design creates its own load-fluctuation failure mode (§7) that a simpler design would not have introduced in the first place. Module 2's prefix-aware routing achieves a meaningfully similar effect — steering requests toward wherever their prefix is already cached — using an in-memory LRU index and, at most, a KVEvent stream, without a separate global storage engine, a content-hash chain, or a dedicated scheduler process.
- Grounded in the same paper's own numbers, not a competing claim from elsewhere
- Correctly separates "does cache need to move at all" (forced by disaggregation, not in dispute) from "does it need to move into a globally shared, hash-addressed pool" (the actual design choice under debate)
- Lower operational surface: no global scheduler state, no hot-spot replication tuning, no early-rejection prediction subsystem to build and maintain
- Mooncake's own real-workload result (75% more requests) is not explained by cache-hit ratio alone — a large share comes from decode-pool TBT isolation, which a point-to-point (non-shared) transfer design would preserve just as well, so this position risks over-crediting the sharing layer specifically
- Does not have its own head-to-head production measurement against Mooncake's shared-pool design — the comparison is inferred from Mooncake's own internal numbers, not a controlled experiment
Separate two claims Mooncake's own paper does not clearly separate: that KV cache must move across a node boundary once prefill and decode are disaggregated (not in dispute — §4's four-step workflow is required infrastructure, and most of the 75%-more-requests result plausibly traces to decode-pool TBT isolation, which any disaggregated design gets), and that this cache should additionally be pooled, hash-addressed, and shared cluster-wide for cross-request reuse (the actual point of disagreement). The paper's own data supports the second claim conditionally, not universally: the ~50% hit-ratio ceiling on general Kimi traffic, against a stated 90% ceiling for long-document, high-repeat workloads like chat-to-paper, means the shared-pool machinery earns its complexity precisely for workloads with long, frequently-repeated contexts — RAG pipelines, agentic tool-use loops, multi-turn chat over a large shared system prompt or document — and earns it much less for short, largely-unique prompts, where Module 2's routing-only approach is likely to capture most of the achievable benefit at a fraction of the engineering cost. What would settle this cleanly is a controlled comparison Mooncake's own evaluation does not provide: the same production trace, same hardware, same disaggregated prefill/decode split, with cache sharing switched off (point-to-point transfer only, no cross-request pool) versus on — isolating the sharing layer's marginal contribution from disaggregation's own TBT-isolation effect. Nobody has published that ablation.
What is changing
The architecture this module describes — Conductor, the four-step workflow, hash-of-hash block addressing — is the FAST'25 paper's design and is stable; what is moving fast is the repository built around it. Mooncake Store, open-sourced in March 2025 as "a distributed KVCache based on Transfer Engine," did not exist in the original paper's evaluation and only became a first-class vLLM feature in May 20262 — the per-object replica/pin/placement controls in §9 are newer than most of the mechanism this module otherwise describes. Mooncake EP and PG are newer still, extending the same transfer infrastructure to fault-tolerant expert-parallel execution (Module 5's territory) rather than request serving, and the repository's own 2026 updates show active work on that front specifically: a declarative data-movement engine for disaggregated serving resilience and a dedicated study of surviving partial rank failures in wide-EP MoE inference are both cited as in-progress or newly published work by the same research group2. The Transfer Engine itself is also being pulled into workloads well outside inference serving — RDMA-based weight synchronization for reinforcement-learning training loops and speculative-decoding trainer/inference decoupling both now use it as a generic tensor-movement layer2 — which is worth watching because it means the transport layer's roadmap is no longer driven solely by KV-cache-serving requirements. The kvcache_balancing_threshold in Algorithm 1 (§5) is explicitly flagged in the paper itself as manually tuned today, with an adaptive algorithm left as future work1 — expect that knob to become automatic before the overall architecture changes shape.
References
- paperQin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. USENIX FAST 2025 (Best Paper Award); arXiv:2407.00079v4. https://arxiv.org/pdf/2407.00079. Accessed 2026-09-17. Used for: architecture (§3, Fig. 1), four-step workflow (§3, Fig. 3–4), content-addressed block hashing, Algorithm 1 (Conductor scheduling, §6), cache load balancing/hot-spot migration (§6.2), chunked pipeline parallelism and layer-wise prefill (§5), overload-oriented early rejection (§7), trace statistics and cache-hit-ratio table (§4, Table 1), SLO definitions (§2), end-to-end evaluation results (§8).
- repoMoonshot AI / MADSys contributors. kvcache-ai/Mooncake (README, main branch). 2026. https://github.com/kvcache-ai/Mooncake/blob/main/README.md. Accessed 2026-09-17. Used for: Transfer Engine bandwidth benchmark (87/190 GB/s, 2.4x/4.6x vs. TCP), supported transports, Mooncake Store description and features, Mooncake EP & Process Group description, production-adoption timeline (vLLM, SGLang, LMDeploy, TensorRT-LLM, NIXL backend plugin dates).
- officialSGLang contributors. PD Disaggregation. SGLang Documentation. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: confirming Mooncake Transfer Engine as one of SGLang's two supported PD transfer backends (cross-reference to Module 4/12).