Multi-Node LLM Inference: From One Node to N · Module 3
Why Disaggregation Pays: Goodput, Interference, and the Ratio Decision
A colocated engine batches prefill and decode together because they share the same weights and the same KV cache pool — but the two workloads have opposite computational profiles, and continuous batching lets one stall the other. This module works through why that stall happens, the queueing-theoretic argument DistServe used to justify splitting the phases onto separate GPU pools, the production-trace evidence Splitwise and Mooncake bring to the same conclusion, and the TCO analysis that says the win is conditional, not universal. After reading, you can state, with the actual numbers, when a second (decode) pool is worth its operational cost and when it isn't.
1. One engine, two incompatible workloads
A continuous-batching engine serving both phases treats every request the same way: it adds new prompt tokens (prefill) and in-flight generation steps (decode) to the same running batch, on the same GPUs, scheduled by the same loop. That is efficient for GPU utilization and is exactly what you already know from single-node serving. It is also the source of the entire disaggregation argument, because prefill and decode are not the same kind of work.
1.1 Compute-bound prefill, memory-bound decode
Prefill processes every token of a new prompt in parallel in one step, so its FLOP count and its memory-access count both scale with prompt length, but the FLOP count scales faster because attention and the MLP layers are dense matmuls over the whole sequence at once. DistServe's own numbers on a 13B model: computing the prefill of a 512-token sequence makes an A100 near compute-bound
1. Decode does the opposite: each step processes exactly one new token, but that one token still has to read the full set of model weights and the full KV cache for every prior token from HBM. The result, in DistServe's framing, is that decode incurs a similar level of I/O to the prefill phase
despite doing a fraction of the compute, which is why it is constrained by the GPU's memory bandwidth
1. This is the same roofline asymmetry you already know from single-GPU decode — low arithmetic intensity, bandwidth-bound — but here it collides with a second workload that sits at the opposite end of the roofline and wants the same SMs.
1.2 What a prefill job does to a running decode batch
Continuous batching does not schedule these two profiles separately; it inserts a new prefill job into whatever decode batch is currently running, because that is how it keeps GPU utilization high. DistServe measured the consequence directly: adding a single prefill job to a batch of decoding requests significantly slows down both processes, leading to a marked increase in TTFT and TPOT
1. The mechanism is simple scheduling: the decoding tasks in the batch must wait for lengthier prefill jobs to complete, thus extending TPOT
, and the reverse holds too — adding decode jobs to a prefill batch increases the prefill's own completion time, especially once the GPU is already near capacity1. Neither priority order fixes it: prioritizing decode delays prefill and inflates TTFT; prioritizing prefill delays decode and inflates TPOT. You cannot schedule your way out of it on one engine, because the interference is a property of sharing the SMs and the HBM bus between two workloads whose resource demands do not overlap in time the way continuous batching assumes.
1.3 Chunked prefill mitigates, does not eliminate
The obvious fix, and the one every serving engine you already know (vLLM, SGLang, TensorRT-LLM) ships, is chunked prefill with piggybacking: split a long prefill into smaller chunks, each sized to leave room in the batch for a handful of decode steps, so decode is never blocked for longer than one chunk's execution time instead of the whole prefill. DistServe is explicit that this is a real mitigation and an incomplete one: this technique alleviates the slowdown of the decoding job caused by the long prefill job, but it does not eliminate it
1. Two costs remain. First, the residual stall — each chunk still occupies the batch and still delays whatever decode steps are queued behind it, just for a shorter interval than the unchunked case. Second, and less obvious, chunking has its own tax: every chunk of a prefill needs the KV cache of every preceding chunk already loaded from HBM to compute its attention, so splitting one prefill into \(N\) equal chunks requires reloading \(N + (N-1) + \dots + 1 = O(N^2)\) chunks of KV cache from HBM, versus \(O(N)\) for an unchunked prefill1.
Worked arithmetic: the chunking tax at N = 4
An unchunked prefill of length \(L\) does one attention pass over \(L\) tokens: it loads its own KV cache once, so the total KV-cache traffic is proportional to \(N=1\) unit (the whole sequence, once).
Split into \(N\) equal chunks, chunk \(i\) must attend over its own tokens plus the KV cache of every chunk before it. Chunk 1 loads 1 chunk-worth of KV, chunk 2 loads 2 chunk-worths (itself + chunk 1), chunk 3 loads 3, chunk 4 loads 4:
$$ \text{total chunk-loads} = 1 + 2 + 3 + 4 = \frac{N(N+1)}{2} = \frac{4 \times 5}{2} = 10 $$against 4 chunk-loads if the same 4 chunks' worth of KV cache were read once each in an unchunked pass — a 2.5× increase in HBM traffic for the prefill's KV reads at \(N=4\), growing quadratically as the chunk count rises with longer contexts1. This is why chunked prefill is a real-time knob (smaller chunks reduce decode stall, larger chunks reduce KV re-read cost) rather than a free lunch, and it is the concrete reason a second, dedicated decode pool remains attractive even after chunked prefill ships everywhere.
2. Goodput is the target, not throughput
Raw throughput — tokens generated per second, summed across all requests — hides SLO violations: a system can push huge throughput numbers by letting tail latency blow out for a fraction of requests. The metric that matters for a serving fleet is goodput, and the field has converged on a small vocabulary for it. Time to first token (TTFT) is the elapsed time between submitting a prompt and receiving the first output token; inter-token latency (ITL), also called time per output token (TPOT), is the average time between successive output tokens for one request, computed per request as \((\text{E2E latency} - \text{TTFT}) / (\#\text{output tokens} - 1)\)2. Goodput is then defined against an SLO on both: the percentage of requests that meet every defined SLO, \(\text{Goodput} = (\#\text{requests meeting all SLOs} / \text{total requests}) \times 100\%\)2. DistServe operationalizes the same idea per GPU: per-GPU goodput is the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned
1 — higher per-GPU goodput is lower cost per query at a fixed quality bar, which is the number a capacity plan actually needs.
3. DistServe: disaggregation as a queueing-theory result
DistServe (OSDI 2024) is the paper that turned the interference observation into a placement algorithm and measured its payoff1. Its central move: once prefill and decode run on separate GPUs, each phase's batching and parallelism strategy can be optimized against only its own SLO — TTFT for prefill, TPOT for decode — instead of a shared, compromised configuration that has to satisfy both.
3.1 Why the colocated ceiling is so much lower
DistServe models a disaggregated instance's request latency with the M/D/1 queue (Poisson arrivals, deterministic service time \(D\), one server), because once a phase runs alone on its own GPU, per-request execution time is close to deterministic:
$$ \text{Avg\_TTFT} = D + \frac{RD^2}{2(1-RD)} $$where \(D\) is the fixed per-request execution time and \(R\) is the arrival rate, so \(RD\) is utilization1. The first term is just compute time; the second is queueing delay, and it is the second term that explains why goodput collapses on a shared engine. Evaluate the queueing term at two utilizations:
At \(RD = 0.8\):
$$ \frac{RD^2}{2(1-RD)} = \frac{0.8^2}{2 \times 0.2} = \frac{0.64}{0.4} = 1.6D $$Avg TTFT ≈ 2.6\(D\) — queueing already more than doubles the wait.
At \(RD = 0.95\):
$$ \frac{RD^2}{2(1-RD)} = \frac{0.95^2}{2 \times 0.05} = \frac{0.9025}{0.1} = 9.0D $$Avg TTFT ≈ 10\(D\) — the queue dominates almost entirely.
This is the standard M/D/1 blow-up as utilization approaches 1, and it is exactly what makes colocation expensive: on one engine, \(D\) itself is not fixed — it fluctuates every time a prefill job of unpredictable length gets folded into the batch, so the system either runs at low utilization to keep the queueing term small (wasting GPUs) or accepts periodic \(D\) spikes that blow out the SLO for whichever phase is waiting. Splitting the phases fixes \(D\) per pool (decode's \(D\) is now just its own memory-bound step time; prefill's \(D\) is its own compute-bound pass), so each pool can run close to its own saturation point without the other phase's variance leaking in. That is the mechanism behind the headline result on a 13B model on one A100 at 90% SLO attainment: colocated per-GPU goodput is about 1.6 requests/s, while phase-separated goodput is 5.6 requests/s for prefill-only and 10 requests/s for decode-only1 — a 3.5× and 6.25× gap respectively, all on identical hardware, from removing one source of queueing variance.
3.2 The placement algorithm: how much cross-node bandwidth you have decides the strategy
Splitting phases means moving the KV cache computed by the prefill instance to whatever decode instance serves the rest of the request — the transfer mechanics themselves belong to Module 4, but the placement question is squarely part of the "does it pay" argument, because it decides whether disaggregation needs premium interconnect or can run on whatever you already have. DistServe defines two regimes. On a high node-affinity cluster — equipped with InfiniBand, where cross-node KV transmission overhead is negligible — DistServe can place a prefill instance and a decode instance on any two nodes in the cluster without constraint, then optimize each phase's parallelism plan independently before replicating to hit the target traffic rate1. On a low node-affinity cluster — where cross-node bandwidth is limited — DistServe instead colocates the prefill and decode instance segments that belong to the same pipeline stage on one physical node, forcing the KV-cache handoff onto intra-node NVLink instead of the slower fabric, at the cost of an added placement constraint that has to be co-optimized with the parallelism search1. For very large models this constraint can bind hard: at 175B parameters (350GB of weights), one 8-GPU A100 node's aggregate HBM (80GB × 8 = 640GB) cannot even fit a single prefill-plus-decode instance pair (350GB × 2 = 700GB), so the low-affinity placement algorithm has to search jointly over model parallelism and instance placement rather than treating them separately1.
3.3 What the numbers say end to end
DistServe evaluated against vLLM and DeepSpeed-MII (the latter shipping its own chunked-prefill mitigation) across three application shapes with distinct SLOs, on OPT models from 13B to 175B, on a 4-node, 32-GPU A100 cluster with 25 Gbps cross-node bandwidth1.
| Application | Model(s) | Dataset | TTFT SLO | TPOT SLO | Rate vs. vLLM | SLO-scale vs. vLLM |
|---|---|---|---|---|---|---|
| Chatbot | OPT-13B / 66B / 175B | ShareGPT | 0.25–4.0 s | 0.1–0.2 s | 2.0–4.6× | 1.8–3.2× |
| Code completion | OPT-66B | HumanEval | 0.125 s | 0.2 s | 5.7× | 1.4× |
| Summarization | OPT-66B | LongBench | 15 s | 0.15 s | 4.3× | 12.6× |
Against DeepSpeed-MII's chunked-prefill baseline the gaps narrow but do not close: 1.6–7.4× higher rate and 1.7–1.8× more stringent SLO on chatbot, 1.6× rate and 1.4× SLO on code completion, 1.8× rate and 2.6× SLO on summarization1 — consistent with chunked prefill being a real but partial fix (§1.3).
3.4 Why the cross-node tax doesn't erase the win
Disaggregation's obvious cost is that KV cache now has to move between GPUs instead of staying resident. DistServe's own latency breakdown on OPT-175B, the largest model in its evaluation (and therefore the one with the most KV cache to move), found KV-cache transmission accounts for less than 0.1% of total request latency, and over 95% of requests see a transfer delay under 30 ms, despite a cross-node fabric limited to 25 Gbps1. The reason is the placement algorithm from §3.2: a single 512-token request's KV cache on OPT-66B is about 1.13 GB; at an average arrival rate of 10 requests/s that is roughly 11.3 GB/s of transfer demand, or about 90 Gbps of required bandwidth — small next to the 600 GB/s of intra-node NVLink that the low-affinity placement algorithm routes the transfer over1. This is the load-bearing fact behind "disaggregation pays": the transfer tax is a placement-and-engineering problem with a small, bounded cost, not a fundamental one that scales against the goodput gain. (How that transfer is actually implemented — connectors, pipes, synchronous vs. overlapped handoff — is Module 4's subject, not this one's.)
4. Splitwise: the same conclusion from production traces and hardware economics
Splitwise (ISCA 2024, Microsoft/University of Washington) arrived at phase-splitting independently and around the same time as DistServe, motivated less by a queueing derivation and more by two Azure production traces — a coding service and a conversation service, both captured over a 20-minute window on November 11, 2023, and released publicly3.
4.1 Prefill and decode are unevenly demanding, and unevenly so per workload
The two traces have very different prefill:decode ratios. The coding service has a median prompt size of 1,500 tokens against a median of only 13 generated tokens — almost entirely prefill-bound. The conversation service has a median prompt of 1,020 tokens but a near-bimodal output-length distribution with a median of 129 generated tokens — proportionally far more decode work3. This is the empirical version of the point Module 3's title makes: there is no single right prefill:decode pool ratio, because the ratio is set by the ISL:OSL shape of the traffic, and that shape is workload-specific, not a serving-stack constant.
4.2 The hardware delta the split has to be provisioned against
Splitwise's cluster-design argument is that prefill and decode benefit from different hardware, because they hit different bottlenecks (compute for prefill, memory bandwidth for decode) and A100-to-H100 improved those two axes by very different factors:
| Metric | A100 | H100 | Ratio |
|---|---|---|---|
| Peak TFLOPs | 19.5 | 66.9 | 3.43× |
| HBM capacity | 80 GB | 80 GB | 1.00× |
| HBM bandwidth | 2039 GB/s | 3352 GB/s | 1.64× |
| Power (TDP) | 400 W | 700 W | 1.75× |
| NVLink (per-lane) | 50 Gbps | 100 Gbps | 2.00× |
| InfiniBand | 200 GB/s | 400 GB/s | 2.00× |
| Cost per machine | $17.6/hr | $38/hr | 2.16× |
Note the mismatch that motivates phase splitting: H100 buys 3.43× the compute (helps prefill, which is compute-bound) but only 1.64× the HBM bandwidth (the axis decode is bound on), at 2.16× the hourly cost. A fleet that colocates both phases on H100 everywhere pays the full 2.16× premium on every GPU to get a compute upgrade that only the prefill half of the traffic actually uses efficiently; a fleet that disaggregates can put the expensive compute-heavy GPU generation where compute is the bottleneck and cheaper (or older) GPUs where memory bandwidth, not FLOPs, is the constraint. Splitwise's implementation moves state between the prompt-computation and token-generation machines over the datacenter's existing back-end InfiniBand interconnect, reporting this can be done "without any perceived performance loss"3.
4.3 The result
Built as Splitwise-HBM (prioritizing memory-bandwidth-matched hardware for decode) and Splitwise-Power (prioritizing power-matched provisioning) cluster designs, Splitwise-based clusters achieve up to 1.4× higher throughput at 20% lower cost than a current non-disaggregated design, or alternatively up to 2.35× more throughput under the same cost and power budgets3. Unlike DistServe's per-GPU-goodput framing, this is a fleet-level cost/throughput result, and it makes the same case through a different lens: not "here is the queueing mechanism," but "here is what a real production traffic mix, split onto the right hardware for each phase, actually buys you."
5. The prefill:decode ratio is a function of workload shape, not a constant
Section 4.1 already showed this qualitatively from production traces; TensorRT-LLM's own disaggregation benchmarks on GB200 make it quantitative and show how wide the range is even for the same disaggregation mechanism on the same hardware generation. Testing across configurations from 4 to 32 GPUs per instance, NVIDIA measured disaggregation speedups over an aggregated (colocated) baseline that vary by more than 4× depending purely on model and input:output length ratio4:
| Model | ISL:OSL | Configuration | Speedup range |
|---|---|---|---|
| DeepSeek R1 | 4400:1200 | MTP off | 1.4–1.8× |
| DeepSeek R1 | 4400:1200 | MTP on | 1.6–2.5× |
| DeepSeek R1 | 8192:256 | 4-GPU generation unit | up to 1.73× |
| DeepSeek R1 | 8192:256 | 8-GPU generation unit | up to 2.0× |
| Qwen3 | 8192:1024 | — | 1.7–6.11× |
With multi-token prediction (MTP) enabled, DeepSeek R1's disaggregation speedup at the 4400:1200 configuration averages 20–30% higher than with MTP off4 — a reminder that the speculative-decoding knob you already know interacts with the disaggregation decision rather than sitting orthogonal to it, because MTP changes decode's compute intensity and therefore how much headroom a dedicated decode pool has left to exploit. Read a single "disaggregation gives N×" number from any vendor benchmark as a workload-shape-dependent result, never as a hardware constant: the same mechanism on the same GB200 hardware spans 1.4× to 6.11× purely as a function of which model and which ISL:OSL ratio the request trace has4.
6. Mooncake's own reconsideration: does chunked prefill make this moot?
Mooncake (FAST 2025 Best Paper, Moonshot AI's Kimi serving platform) is the most useful source on this question precisely because it is not making a case for disaggregation from a clean-room benchmark — it is a production system's authors explicitly asking themselves whether they still need the complexity they already built. Section 5 of the paper poses the question directly: given that chunked prefill lets a single colocated engine interleave prefill chunks with decode steps and thereby reduce interference, it is worth discussing whether this separation is still necessary with the introduction of chunked prefill
5.
Their answer is yes, disaggregation stays, for two reasons that are not about interference at all:
- Cross-node parallelism for long context. A single colocated instance's prefill throughput is bounded by however much of one node's compute and memory a long-context request can use; disaggregating lets the prefill cluster scale prefill-specific parallelism (including cross-node) independently of whatever the decode cluster is doing, which matters specifically for long-context workloads where a single prefill pass is itself large enough to benefit from more GPUs than one node has.
- VRAM savings. Colocating both phases means every GPU has to reserve enough HBM for both the transient prefill working set and the long-lived decode KV cache; separating the pools lets the decode cluster dedicate its entire HBM budget to KV cache capacity (more concurrent sequences, longer contexts held) without competing against prefill's own memory footprint.
This reasoning matters more than a vendor benchmark because Mooncake is not selling disaggregation in the abstract — it is the team that runs Kimi's production traffic deciding, after chunked prefill was already available to them, that a colocated engine with chunked prefill does not substitute for a dedicated decode pool. It also narrows the justification: the case is not "interference is still bad without disaggregation" (chunked prefill has already addressed a good fraction of that, per §1.3), it is "two specific capacity constraints — cross-node prefill parallelism and KV-cache VRAM budget — are easier to solve with separate pools than with a bigger colocated one." Whether either of those two reasons applies to a given deployment (do you actually serve long-context requests at meaningful volume? is decode's KV-cache footprint actually your binding constraint?) is a more specific question than "does interference exist," and it is the right question to ask before paying disaggregation's operational cost.
7. The TCO crossover: where disaggregation stops winning
Every result so far optimizes goodput or throughput at a fixed SLO or a fixed hardware generation. SemiAnalysis's InferenceMAX benchmark asks a different question: given a real cost of ownership for the hardware, what configuration gives you the lowest dollar cost per million tokens served, across the full range of interactivity (tokens/second/user) a deployment might target6? Their framing is explicit that this, not raw throughput, is the metric that should decide a deployment: TCO per million tokens is the true north star that customers care about — performance is merely a stepping stone to calculating this metric
6.
Comparing a GB200 NVL72 rack running TensorRT-LLM with Dynamo's disaggregated prefill against single-node 8-GPU B200 servers running TensorRT-LLM aggregated (colocated), the result is a genuine crossover, not a one-sided win: below 90 tok/s/user interactivity, the GB200 NVL72 on the TRT-LLM engine using Dynamo disagg prefill decisively outperforms all single node 8-GPU servers on TCO per million tokens
, but for interactivity levels above 90 tok/s/user, the B200 on TRT-LLM beats the GB200 NVL72
6. Interactivity here is inversely related to how much per-token latency you are willing to tolerate: low tokens/s/user targets correspond to batch-heavy, throughput-oriented workloads (large concurrent batches, more tolerance for per-token latency), where the extra GPUs a disaggregated deployment devotes to prefill pay for themselves; high tokens/s/user targets correspond to latency-sensitive, small-batch workloads, where NVL72's scale-up domain advantage matters less than a single aggregated server's lower fixed cost and shorter communication path.
Two qualifiers matter for reading this result correctly. First, it is model- and workload-specific: the benchmark cited here is a summarization-shaped workload (long input, shorter output) on a specific model, and InferenceMAX's own methodology notes the advantage on B200/H200 hardware requires disaggregated prefill and wide expert parallelism to actually be implemented over the available interconnect (Spectrum-X as well as InfiniBand) to unlock the higher token-throughput-per-megawatt these parts are capable of6 — the crossover is a property of a specific, tuned software stack on specific hardware, not a universal law. Second, the crossover point itself should be read as a snapshot of the current state of Dynamo's and TensorRT-LLM's disaggregation optimization, not a hardware limit: the software work to get disaggregated prefill running efficiently across a wide interactivity range is ongoing, and where exactly the curves cross will move as that work lands.
Where experts disagree
All three positions below agree on the interference mechanism in §1 — nobody disputes that colocating prefill and decode causes measurable stalls. They disagree on how strong a conclusion that licenses: whether disaggregation is close to a default good decision, whether it is conditional on a specific operating point, or whether the second pool is frequently not worth the added complexity at all.
Disaggregation is a near-unconditional win once you optimize for goodput
Both OSDI/ISCA 2024 papers frame the result as: given an SLO-attainment target, separating the phases removes a source of queueing variance (§3.1) and lets each phase use hardware matched to its own bottleneck (§4.2), and the measured gains — 2.0–12.6× depending on metric and workload — are large enough that the cross-node communication tax (§3.4, well under 1% of latency) is not a serious counterargument. Neither paper frames the decision as conditional on an interactivity target; both present it as strictly better than colocation once you are optimizing SLO-constrained throughput per dollar of GPU.
- Grounded in a directly measured queueing mechanism, not an aggregate statistic
- Confirmed independently by two groups on different hardware and workload traces
- The gains are largest exactly where SLOs are tight, which is the regime production serving actually cares about
- Measured on A100-generation hardware where colocated per-GPU throughput headroom is lower than on B200/GB200
- Does not report a TCO-per-token figure, only SLO-constrained request rate, so it cannot be compared directly against a dollar-cost target
The win is conditional on the interactivity target; above ~90 tok/s/user it inverts
SemiAnalysis, InferenceMAX6
Measuring TCO per million tokens instead of SLO-constrained request rate, and on GB200 NVL72 vs. B200-generation hardware, disaggregated prefill only wins below roughly 90 tokens/s/user of interactivity; above that, a single aggregated 8-GPU B200 server has better economics. The crossover exists because a dedicated prefill pool's advantage is largest for throughput-oriented, latency-tolerant workloads, and shrinks as the target interactivity rises toward what a well-provisioned aggregated server can already deliver on newer hardware.
- Directly measures the dollar-denominated metric an infrastructure decision actually needs
- Uses current-generation hardware (GB200/B200) rather than A100-era hardware
- Makes the conditionality explicit instead of treating disaggregation as unconditionally correct
- Third-party analysis, not a peer-reviewed measurement, and does not publish exact TCO figures at each interactivity level (labelled analysis, not primary benchmark, per the source hierarchy)
- The disaggregated-prefill software stack it benchmarks is, by the authors' own account, still being optimized — the crossover point is a snapshot, not a fixed property of the hardware
A second pool often isn't worth the operational cost; chunked prefill on one pool captures most of it
Mooncake's own §5 discussion5, read alongside the O(N²) chunking cost DistServe itself derives1
Even a team that ships and runs a KV-centric disaggregated architecture in production explicitly asks whether chunked prefill has made the split unnecessary, and answers narrowly: keep it, but only because of two specific capacity constraints (cross-node prefill parallelism for long context, decode-side VRAM headroom), not because interference alone still justifies it. That is a materially weaker claim than "disaggregation wins" — it implies that for workloads without long-context prefill and without a binding KV-cache-capacity constraint, chunked prefill on one pool may be enough, and the operational cost of a second pool (a second autoscaler, cross-node KV transfer, xPyD reconfiguration — Module 4) is not automatically justified.
- Comes from a production system's own authors reconsidering a design they already built and operate, not a synthetic benchmark
- Narrows the justification to concrete, checkable conditions (context length distribution, KV-cache capacity pressure) rather than a blanket claim
- Mooncake's own conclusion is still to keep disaggregation — this is a narrowing of the argument, not a rebuttal of it
- Does not provide a quantitative threshold (how long is "long context," how much VRAM pressure is "binding") the way the TCO crossover does
The three positions are measuring different things and mostly do not actually contradict each other once you separate the axes. DistServe and Splitwise hold SLO attainment fixed and ask "what is the maximum request rate per GPU" — on that axis, on A100-class hardware, disaggregation's win is large and consistent across three independent workload traces (§3.3, §4). InferenceMAX holds a hardware budget fixed and asks "what is the dollar cost per token across a swept interactivity target" on GB200/B200-class hardware — on that axis, the answer inverts once the target shifts toward high-interactivity, low-batch serving, because that is exactly the regime where an aggregated server's simplicity and NVL72's per-GPU headroom close the gap that a dedicated prefill pool used to open. Neither finding is wrong; they describe the same mechanism (§1) evaluated at different operating points on different hardware generations, three years apart, with the newer benchmark's disaggregation implementation explicitly still under active optimization6. Mooncake's position is the practical synthesis: treat "does disaggregation pay" as a question about your specific traffic (context-length distribution, decode KV-cache pressure, target interactivity) rather than a yes/no verdict borrowed from someone else's benchmark. What would settle the disagreement is not more papers at either extreme but a TCO-per-token study that sweeps interactivity target the way InferenceMAX does and reports SLO attainment the way DistServe does, on the same hardware generation — nobody has published that yet.
What is changing
The InferenceMAX crossover point (§7) is the number in this module most likely to move within the next year: it is a function of how far NVIDIA's Dynamo and TensorRT-LLM teams have gotten in optimizing the disaggregated-prefill code path for high-interactivity, low-batch serving on GB200/B200-class hardware, not a property of the silicon6. As wide expert parallelism and disaggregated prefill get tuned further across a broader interactivity range6, expect the 90 tok/s/user figure to shift, most plausibly upward as the disaggregated path closes its efficiency gap at low-batch operating points. Track InferenceMAX's published results directly rather than treating this module's number as durable past 2027.
References
- paperZhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., Zhang, H. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. USENIX OSDI 2024. arXiv:2401.09670v3. https://arxiv.org/pdf/2401.09670. Accessed 2026-09-17. Used for: compute-bound/memory-bound framing (§2.1), interference mechanism and chunked-prefill limitation (§2.3, Fig. 2), per-GPU goodput definition and headline 1.6/5.6/10 rps result (§1, Fig. 1), M/D/1 queueing model (Eq. 1), high-/low-node-affinity placement algorithms (§4.1-4.2), end-to-end evaluation results (§6.2, Table 1, Fig. 8-9), communication-overhead breakdown (§3.3, §6.3).
- officialAnyscale. Understand LLM latency and throughput metrics. docs.anyscale.com, 2026. https://docs.anyscale.com/llm/serving/benchmarking/metrics. Accessed 2026-09-17. Used for: TTFT, TPOT/ITL, and goodput formula definitions (§2).
- paperPatel, P., Choukse, E., Zhang, C., Shah, A., Goiri, I., Maleki, S., Bianchini, R. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. ISCA 2024. arXiv:2311.18677v2. https://arxiv.org/pdf/2311.18677. Accessed 2026-09-17. Used for: Azure production-trace ISL/OSL medians (§III.A), A100 vs. H100 hardware-delta table (Table I), cluster-level throughput/cost result (Abstract).
- officialNVIDIA / TensorRT-LLM team. Disaggregated Serving in TensorRT LLM. NVIDIA TensorRT-LLM Tech Blog, 2026. https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.html. Accessed 2026-09-17. Used for: GB200 disaggregation speedup ranges by model, ISL:OSL, and MTP configuration.
- paperQin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. USENIX FAST 2025 (Best Paper). arXiv:2407.00079v4. https://arxiv.org/pdf/2407.00079. Accessed 2026-09-17. Used for: the §5 reconsideration of whether disaggregation is still necessary given chunked prefill, and the two stated reasons for keeping it.
- analysisSemiAnalysis. InferenceMAX: Open Source Inference Benchmarking. InferenceMAX v1, 2026. https://newsletter.semianalysis.com/p/inferencemax-open-source-inference. Accessed 2026-09-17. Used for: TCO-per-million-tokens framing, the ~90 tok/s/user crossover between GB200 NVL72 disaggregated prefill and aggregated 8-GPU B200 servers, and the note on ongoing optimization of the disaggregated-prefill implementation.