Multi-Node LLM Inference: From One Node to N · Module 15

Capacity Planning and Economics: Goodput, Tokens/s/$, and Little's-Law Reasoning at Rack Scale

Every earlier module in this course answered a mechanism question -- why disaggregate, how KV cache moves, how wide expert parallelism dispatches tokens. This module answers the question a capacity plan actually has to answer: given a target SLO and a handful of published per-node numbers, how many nodes do you provision, and what does each million tokens served actually cost. It pulls together DistServe's queueing math, DeepSeek's own disclosed economics, and the InferenceMAX TCO crossover into one sizing methodology, and is explicit about a claim it declined to use because nobody could source it.

Course 30 min read 9 sources As of 2026-09-17 Prerequisite: Module 3 Prerequisite: Module 6
The sizing pipeline: an SLO target and a published per-node throughput number flow through queueing math to a node count and a cost per million tokens SLO target TTFT / ITL, goodput % Per-node throughput (published, §2-3) Queueing math M/D/1, R→R/N (§5) Node count N Cost / M tokens feedback: at a fixed interactivity target, TCO/M-tokens (§4) tells you whether to size disaggregated or aggregated Module 3, 6, 7 supply the numbers this pipeline consumes
Figure 1. This module's structure as a pipeline: an SLO target and a published per-node throughput figure (from Modules 3, 6, and 7's worked numbers) feed a queueing-theoretic sizing step, producing a node count and a cost per million tokens -- with the TCO-per-million-tokens crossover (§4) acting as a feedback check on whether the disaggregated or aggregated topology is even the right one to size in the first place.

1. Goodput as the sizing target

Module 3 defined the vocabulary this module builds on: time to first token (TTFT), the elapsed time between submitting a prompt and receiving the first output token; TPOT/ITL, the average time between successive output tokens, computed per request as \((\text{E2E latency} - \text{TTFT}) / (\#\text{output tokens} - 1)\); and goodput, the percentage of requests meeting every defined SLO, \(\text{Goodput} = (\#\text{requests meeting all SLOs} / \text{total requests}) \times 100\%\)2. Averaged across N unequal-length requests, mean TPOT (the average of each request's own TPOT) and mean ITL (the sum of every inter-token gap divided by total output tokens) diverge whenever output lengths differ2 -- a distinction Module 3 covers in full and this module assumes.

What Module 3 used goodput for was a mechanism argument: colocating prefill and decode caps per-GPU goodput at a low ceiling because of interference. This module uses the same quantity for a different purpose -- as the thing you size against. DistServe's own per-GPU goodput framing makes the connection explicit: it is the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned1. Read backwards, a per-GPU goodput ceiling of \(g\) requests/s at a given SLO tells you directly how many GPUs a target aggregate rate \(R_\text{total}\) requires: \(N \geq R_\text{total} / g\). Every worked example in this module is some version of that inversion -- take a published per-node or per-GPU rate at a stated SLO attainment, and solve for the node count a target load requires, then price it.

Precision in the metric matters here in a way it does not for a mechanism explanation, because the inversion is only as good as the number fed into it. A published "throughput" figure that does not state its SLO-attainment bar, its TTFT/TPOT targets, or whether it is measuring mean TPOT or mean ITL (§1's divergence, restated from Module 3) is not a usable sizing input -- it tells you a system moved a lot of tokens, not whether any individual request stayed inside a latency bound you can promise a caller. Every per-node and per-GPU number this module inverts into a node count in §3, §5, and §6 is one that states, or is paired with, an explicit SLO-attainment condition: DistServe's ceilings are stated at a 90% attainment bar1, and DeepSeek's per-node throughput figures are stated as live production averages rather than best-case benchmark peaks4. A sizing exercise built on a number without that qualifier is not wrong so much as unfalsifiable -- there is no way to check after the fact whether the deployment it produced actually meets the SLO it was supposed to hit.

2. The single-node baseline every planner needs

Before sizing across nodes, two single-node facts bound what scaling out can and cannot buy, and both come from the same source: Databricks' Mosaic Research team's 2023 engineering post on LLM inference performance, still the most-cited source for this specific arithmetic3.

2.1 Batch size trades latency for throughput, and the exchange rate is not 1:1

Serving MPT-7B on an A100 at 512 input / 64 output tokens, pushing the batch size to 64 (from a batch of roughly 1) increases per-request latency by 4x while increasing aggregate throughput by 14x3. That 4x-for-14x exchange is the single-instance version of the goodput-vs-throughput distinction from §1: a batch-64 deployment posts an enormous throughput number, but any individual request now waits roughly four times longer for its tokens, which is exactly the kind of number that fails a tight TTFT/TPOT SLO even while the aggregate-throughput dashboard looks excellent. Continuous batching (the scheduling discipline every serving engine in this course ships) captures a large fraction of that throughput gain without forcing every request through a fixed, synchronized batch boundary: Databricks reports continuous batching achieving 10-20x better throughput than static or dynamic batching3, which is why static batching is not a live option for anything sized in this module -- every node-count calculation below already assumes continuous batching as the floor, not an upgrade to consider.

Schematic latency-throughput curve: relative latency and relative throughput as batch size rises from 1 to 64 relative latency (batch 1 → 64) relative throughput 1x, 1x batch 1 4x latency, 14x throughput batch 643
Figure 2. Latency and throughput both rise with batch size, but not proportionally: pushing batch size from roughly 1 to 64 on an A100 serving MPT-7B (512 in / 64 out tokens) costs 4x the per-request latency to gain 14x the aggregate throughput. Curve shape is illustrative between the two labelled, cited points3.

2.2 Doubling GPU count does not halve latency at low batch size

The second trap: it is tempting to assume that adding parallelism degree scales latency down linearly, the way it scales aggregate throughput up. It does not, at small batch sizes. Databricks measured Llama2-70B time-to-first-token at batch size 1 (a 512-token input) across GPU counts on A100-40GB: 2 GPUs is the smallest configuration the model fits in and serves as the 1x baseline (154 ms); 4 GPUs brings that down to 0.74x (114 ms); going from 4 to 8 GPUs brings it down by only a further 0.7x3. Doubling the GPU count a second time returns barely more than doubling it the first time did in absolute terms, and both fall well short of the naive expectation that 2x the parallelism degree gives 2x less latency.

Chaining the two measured ratios gives the cumulative effect of going from 2 to 8 GPUs directly: \(0.74 \times 0.7 = 0.518\) -- an 8-GPU configuration's TTFT at batch size 1 is only about half the 2-GPU baseline's, not an eighth of it3. This is the mechanism-level reason inter-op (pipeline) and intra-op (tensor) parallelism both show diminishing latency returns as you add GPUs -- communication overhead and reduced per-GPU utilization after each additional partition eat into the theoretical linear speedup1 -- and it is the reason "just add more GPUs" is not, by itself, a latency-SLO strategy. It is a throughput strategy (§2.1's other axis), and the two have to be sized separately.

Bar chart of relative TTFT at batch size 1 for Llama2-70B across 2, 4, and 8 A100-40GB GPUs, showing sublinear latency reduction 0 0.5x 1.0x 1.0x 2 GPUs (154 ms) 0.74x 4 GPUs (114 ms) 0.52x 8 GPUs (derived) relative TTFT, Llama2-70B, batch 1, 512-token input, A100-40GB3
Figure 3. Relative time-to-first-token at batch size 1 for Llama2-70B as GPU count doubles: 2→4 GPUs buys a 0.74x reduction, 4→8 GPUs a further 0.7x -- the 0.52x figure at 8 GPUs is this module's own arithmetic (0.74 × 0.7) chaining Databricks' two disclosed ratios, not a separately measured point3.

3. Tokens/s/$ as a function of N

The question this section answers directly: as a deployment grows from a handful of GPUs to hundreds, does the number of tokens served per dollar improve, degrade, or plateau? The honest answer, grounded in the only two organizations that have published matched cost and throughput numbers at scale, is: it depends which mechanism dominates, and the published data shows both directions at once.

3.1 Two real $/token numbers, and why they are not directly comparable

Module 6 established the primitives: over a 24-hour production window, DeepSeek's disclosed infrastructure cost was $87,072/day (226.75 average H800 nodes at an assumed $2/GPU-hour), and it generated 168 billion output tokens in that window4. Module 6 does not itself divide one by the other; doing so gives a blended infrastructure cost of

$$ \frac{\$87{,}072}{168{,}000\ \text{(million tokens)}} \approx \$0.518\ \text{per million output tokens} $$

SGLang's independent 96-H100 reproduction of the same wide-EP, PD-disaggregated design reports its own figure directly: $0.20 per million output tokens6, about 2.6x cheaper than the blended DeepSeek figure above. Neither the SGLang blog post nor DeepSeek's own disclosure documents the exact accounting behind its respective number in enough detail to make this a clean apples-to-apples comparison, and the gap is very plausibly an artifact of what each denominator and numerator actually include, not evidence that SGLang's stack is 2.6x more cost-efficient:

Table 1. Two published cost-per-output-token figures and what is known and unknown about each46.
FigureDeepSeek blended (derived)SGLang reproduction
Value$0.518 / 1M output tok$0.20 / 1M output tok
Source$87,072/day ÷ 168B output tok4Stated directly6
ScopeWhole fleet: all prefill + decode nodes, averaged over a live 24h window with real traffic varianceNot disclosed -- likely the 96-GPU benchmark deployment (12 nodes) under controlled load
HardwareH800 (export-limited NVLink)5H1006
GPU-hour rate$2/hr, stated as an assumption, not a quoted price4Not disclosed
Includes prefill infra cost?Yes -- numerator is the entire fleet's cost, denominator is output tokens onlyNot disclosed

The one directionally reliable takeaway is the one both figures agree on despite their different accounting: a wide-EP, PD-disaggregated DeepSeek-class deployment prices out somewhere in the low tens of cents per million output tokens on current-generation Hopper hardware, roughly a fifth of DeepSeek's own metered R1 API price6 -- both numbers are evidence the deployment strategy this course has spent five modules on is not merely reproducible mechanically (Module 6's 5.6%/6.6% throughput match), it is reproducible economically too.

3.2 Per-node throughput falls as expert-parallel degree (and node count) rises

A cleaner, better-controlled comparison sits inside Module 6's own data: three separate measurements of DeepSeek-class decode throughput per node, at three different node counts for the decode pool specifically. SGLang's EP72 reproduction across 9 nodes measured 22,282 tokens/s per node at 2K input length6; DeepSeek's own separately released profiling snapshot, run across 16 nodes, measured 18,598 tokens/s per node6; and DeepSeek's Day-6 production disclosure, averaged over its EP144/18-node configuration, measured about 14,800 tokens/s per node4. Per-node decode throughput falls monotonically as node count rises across these three points:

Bar chart of decode tokens per second per node against decode node count, for three published DeepSeek-class deployments 0 10k 20k tok/s/node 22,282 9 nodes (EP72) SGLang, H1006 18,598 16 nodes DeepSeek profile, H8006 14,800 18 nodes (EP144) DeepSeek production, H8004
Figure 4. Per-node decode throughput across three published DeepSeek-class configurations, ordered by decode node count: 22,282 tok/s/node at 9 nodes, 18,598 at 16 nodes, 14,800 at 18 nodes. Hardware (H100 vs. H800), input length, and benchmark-vs-live-production conditions differ across the three points -- see caveats in prose64.

Read this cautiously: the three points are not a controlled sweep. SGLang's 9-node figure runs on H100 (more NVLink bandwidth than the H800 the other two points use), at 2K input length under benchmark conditions; the 18-node figure is a live 24-hour production average that necessarily blends in real traffic's idle gaps, mixed request shapes, and headroom that a benchmark run does not carry. Still, the direction is consistent with a mechanism this course has already established: DeepEP's own measured bandwidth collapses as the all-to-all crosses more RDMA domains -- 726 GB/s dispatch within one 8-GPU NVLink domain (EP8), falling to 90 GB/s once the all-to-all spans two RDMA domains (EP8x2), and 61 GB/s across four (EP8x4)8. A wider decode expert-parallel degree spreads the same all-to-all traffic across more inter-node hops, and each hop pays the RDMA-not-NVLink tax Module 1 quantified as an roughly 8x cliff at the node boundary. More decode nodes does not linearly translate to more decode throughput per node for a fixed model and a fixed all-to-all mechanism -- the communication tax scales with the parallelism degree, not just the compute does.

3.3 Cache-hit ratio pulls the other way

Section 3.2's trend is specific to MoE decode's all-to-all-bound traffic. A different mechanism, also already established in this course, moves tokens/s/$ in the opposite direction as cluster size grows: cache-hit economics. Mooncake's own production-trace analysis found KV-cache hit ratio rising from about 30% at 1,000 cache blocks of capacity to about 50% at 50,000 blocks, essentially flattening beyond that -- a workload-specific ceiling the paper states can reach 90% for long-document, high-repeat traffic such as chat-to-paper9. Cluster-wide KV-cache capacity is, mechanically, a function of how many nodes' worth of HBM/DRAM/SSD Module 7's tiered hierarchy can pool -- more nodes participating in a shared cache pool means more addressable cache capacity, which (up to the workload's own reuse ceiling) means a higher fraction of prefill tokens served from cache instead of recomputed. DeepSeek's own disclosed traffic makes the dollar consequence concrete: 56.3% of its 608 billion daily input tokens were cache hits, billed at $0.14 per million tokens against $0.55 per million for a cache miss5 -- a cache hit is roughly a quarter the cost of a miss on the same input token, at DeepSeek's published API rates.

4. TCO-per-million-tokens as a planning tool

Module 3 established SemiAnalysis's InferenceMAX result in full: comparing a GB200 NVL72 rack running disaggregated prefill against single-node 8-GPU B200 servers running aggregated (colocated) serving, disaggregation wins on TCO per million tokens below roughly 90 tokens/s/user of interactivity, and the aggregated server wins above it7. This module does not re-derive that crossover -- it turns the finding into the sizing decision it implies.

Read as a planning tool rather than a benchmark result, the crossover collapses to a single question to ask before sizing anything else: what interactivity (tokens/s/user) does the product actually need to hit? A batch-oriented workload -- summarization, document processing, agentic pipelines with high tolerance for per-token latency -- sits below the crossover, and the sizing exercise should default toward a disaggregated topology with the node-count arithmetic in §3 and §6 applied to a dedicated prefill pool and a dedicated decode pool. An interactive chat product with a tight per-token latency target sits above the crossover, and the sizing exercise should default toward provisioning more aggregated 8-GPU units rather than paying for a dedicated prefill pool's fixed overhead at a scale where an aggregated server's simpler topology already wins on cost.

TCO-per-million-tokens crossover reprised as a sizing decision rule, with regions annotated for which topology to size interactivity target (tokens/s/user, low → high) TCO / M tokens (lower better) Size disaggregated (prefill + decode pools, §6) Size aggregated 8-GPU units ~90 tok/s/user7 GB200 NVL72 disagg 8×GPU B200 aggregated
Figure 5. The InferenceMAX crossover from Module 3, reprised here as a sizing rule: below about 90 tokens/s/user of interactivity, the node-count arithmetic in §3 and §6 should target a disaggregated prefill/decode split; above it, aggregated 8-GPU units are the better-costing default. Curve shape illustrative between the cited crossover point and endpoints; the 90 tok/s/user threshold is the cited number7.

Two qualifiers carry over from Module 3 unchanged and matter just as much here: the crossover is specific to the model, workload shape, and software stack InferenceMAX benchmarked, and it is a snapshot of how far NVIDIA's Dynamo and TensorRT-LLM teams have optimized the disaggregated-prefill code path at the time of measurement, not a hardware constant7. Treat 90 tokens/s/user as this module's working threshold, not a number to hard-code into a sizing spreadsheet past this year.

5. Queueing-theoretic capacity reasoning at rack scale

Module 3 derived why a colocated engine's goodput ceiling is so much lower than a disaggregated one, using DistServe's M/D/1 queueing model: with Poisson arrivals at rate \(R\) and a deterministic per-request service time \(D\) (the case once a phase runs alone on its own GPU and \(D\) stops fluctuating), average TTFT is

$$ \text{Avg\_TTFT} = D + \frac{RD^2}{2(1-RD)} $$

where the second term is queueing delay and \(RD\) is utilization1. This module does not re-derive that result. It extends it in a direction Module 3 does not cover: what the same equation says about adding replicas, which is the rack-scale version of Little's-Law-style capacity reasoning this module is scoped to provide.

5.1 Replicate, and the arrival rate each replica sees drops by a factor of N

DistServe's own practical-deployment discussion states the mechanism directly, in the paper's section on replication as a first-class alternative to model parallelism: it is worth noting that when the model can fit into the memory of a single GPU, replication is a competitive option in addition to model parallelism for both prefill and decoding instances, to linearly scale the system's rate capacity. It may also reduce the queuing delay -- as indicated by Eq. 1 -- by substituting R with R/N assuming requests are equally dispatched to N replicas, at the cost of maintaining additional replicas of the model weights in GPU memory1. This is the load-bearing sentence for this section, and it is worth being precise about what "equally dispatched" is doing: the R/N substitution is only valid if the router in front of the N replicas actually balances arrival rate evenly across them (Module 2's territory) -- a router that skews load toward a subset of replicas (a stale prefix-cache index, a hot session pinned to one pod) effectively raises that replica's local R above R/N even while the fleet-wide average sits comfortably under the SLO, which is exactly the kind of failure mode that a fleet-average utilization metric on a dashboard will not surface. Substituting \(R \to R/N\) into Eq. 1 gives the rack-scale sizing formula directly:

$$ \text{Avg\_TTFT}(N) = D + \frac{(R/N) D^2}{2\left(1 - (R/N) D\right)} $$

Two things follow immediately, and both are the reason "add more replicas" is a substantively different lever than "add more parallelism degree" (§2.2). First, the queueing term is convex in utilization \(RD/N\), not linear -- halving the per-replica arrival rate by doubling \(N\) more than halves the queueing delay, because the denominator \(1 - RD/N\) grows faster than the numerator shrinks as utilization drops. Second, this is a pure request-routing effect, independent of the parallelism-degree diminishing returns from §2.2: replication does not fight the same communication-overhead tax that intra-op/inter-op parallelism does, because each replica is a full, independent copy of the model rather than a shard, at the cost of the extra HBM those copies consume.

That convexity has a direct operational consequence: the closer a fleet runs to \(RD/N = 1\), the more expensive each additional unit of arrival-rate growth becomes in queueing-delay terms, so a sizing plan built on a per-instance ceiling (§6.1) should not provision to land exactly at that ceiling. Module 3's own evaluation of Eq. 1 at two utilization points makes the size of the effect concrete: at \(\rho=0.8\), queueing delay is already 1.6\(D\), for a total Avg TTFT of roughly 2.6\(D\); at \(\rho=0.95\), queueing delay alone is 9\(D\), for a total of roughly 10\(D\)1 -- a further 15 percentage points of utilization multiplies the queueing penalty by more than five. A fleet sized to run its replicas at 95% average utilization to save on GPU count is one traffic spike away from a queueing blow-up that no amount of additional replicas fixes quickly, because bringing a new replica online (provisioning, weight loading, warm-up) is not instantaneous the way the formula's \(N\) is. Sizing headroom -- provisioning for a target utilization meaningfully under 1, not the theoretical ceiling -- is the practical takeaway Figure 6's curve argues for, independent of which topology (§4) or which node-count arithmetic (§6) produced the ceiling in the first place.

Rack-scale M/D/1: how replica count collapses queueing delay

Utilization per replica (RD/N)
Avg TTFT

Formula: \(\text{Avg\_TTFT}(N) = D + \dfrac{(R/N)D^2}{2(1-(R/N)D)}\), DistServe Eq. 1 with the R→R/N replication substitution from §3.3 of the paper1. Default values (R=6, D=0.05s, N=1) give Avg TTFT ≈ 60.7 ms; moving N to 4 alone -- no change to R or D -- drops it to about 51.1 ms, and to about 50.3 ms at N=8, illustrating the convex collapse as utilization per replica falls.

Avg TTFT normalized by D, plotted against utilization RD, showing the convex blow-up as utilization approaches 1 and the effect of splitting arrival rate across replicas utilization ρ = RD/N Avg_TTFT / D ρ=0.8, ≈2.6D ρ=0.95, ≈10D 0 1 replicating (N↑) moves operating point left along this curve
Figure 6. Avg TTFT (normalized by service time D) as a function of utilization ρ=RD, per DistServe's Eq. 1. The curve is convex: at ρ=0.8, queueing already adds 1.6D to give ≈2.6D total; at ρ=0.95, it adds 9D to give ≈10D total, both figures from Module 3's evaluation of the same equation1. Replicating across N instances divides R by N, moving the operating point left along this curve -- the mechanism §5.1 quantifies.

5.2 Why an unsourced Little's-Law claim was excluded

The general shape of the claim -- that keeping a scheduler saturated and work-conserving matters a great deal for throughput -- is not in dispute; it is the specific, unsourced 30-70% figure that fails this course's bar. Where you need a number for how much a scheduling or replication choice is worth, derive it from Eq. 1 and your own \(R\) and \(D\), the way §5.1's calculator does, rather than importing a round number from a source that cannot show its work.

6. A worked sizing example, end to end

Two different sizing questions use two different published number types, and it is worth keeping them separate rather than forcing one formula to answer both.

6.1 Sizing from a request-rate SLO: DistServe's goodput ceilings

Suppose a target load of 50 requests/s at a 90% SLO-attainment bar, on hardware and a model class matched to DistServe's own OPT-13B/A100 evaluation (Module 3). Using the per-GPU goodput ceilings DistServe measured directly -- 1.6 requests/s/GPU colocated, 5.6 requests/s/GPU for a prefill-only pool, 10 requests/s/GPU for a decode-only pool, all at 90% SLO attainment1 -- the inversion from §1 (\(N \geq R_\text{total}/g\)) gives:

Colocated:

$$ N = \left\lceil \frac{50}{1.6} \right\rceil = 32\ \text{GPUs} $$

Disaggregated (prefill + decode pools):

$$ N_p = \left\lceil \frac{50}{5.6} \right\rceil = 9,\quad N_d = \left\lceil \frac{50}{10} \right\rceil = 5,\quad N = 14\ \text{GPUs} $$

At this target rate and SLO bar, disaggregation needs 14 GPUs against colocation's 32 -- a 2.3x reduction in provisioned GPU count to hit the identical 90%-attainment target, using nothing but the ceilings DistServe already published and the same inversion arithmetic every row of Table 2 below uses. (This is a different quantity from Module 3's 3.5x/6.25x goodput-ratio framing -- that number compares ceilings directly; this one compares the GPU count required to hit a fixed external target rate, which is the number a capacity plan actually needs.)

6.2 Sizing from a token-volume target: DeepSeek's per-node throughput

For a token-throughput target rather than a request-rate target, use a published per-node tokens/s figure and the same $2/GPU-hour assumption Module 6 uses for DeepSeek's own cost reconstruction4. At DeepSeek's Feb-2025 production average of 14,800 output tokens/s per 8-GPU node4, a target sustained output rate of 100,000 tokens/s requires \(\lceil 100{,}000 / 14{,}800 \rceil = 7\) nodes (56 GPUs), at \(7 \times 8 \times \$2/\text{hr} \times 24\,\text{h} = \$2{,}688\)/day. At that throughput and cost, tokens/s/$ works out to \(100{,}000 / (2{,}688/86{,}400) \approx 3.22\) million tokens per dollar per day of sustained serving -- or, inverted into the unit §3.1 used, about $0.311 per million output tokens, a figure that sits between the two data points in Table 1, as it should: it is built from the same per-node production throughput number as the DeepSeek blended figure, but without that figure's prefill-cost cross-subsidy diluting the denominator.

Table 2. Worked sizing examples: target SLO or volume, the published per-node/per-GPU number consumed, the derived node count, and the resulting economics14.
TargetPublished inputDerived node/GPU countCost or tokens/s/$
50 req/s @ 90% SLO, colocated1.6 rps/GPU ceiling132 GPUs--
50 req/s @ 90% SLO, disaggregated5.6 / 10 rps/GPU ceilings114 GPUs--
100k output tok/s sustained14,800 tok/s/node47 nodes (56 GPUs)$0.311 / 1M tok

Neither row substitutes for the other. The request-rate calculation answers "how many GPUs to hit an SLO-constrained arrival rate," which is the number that determines whether requests queue past their TTFT/TPOT bound at all (§5). The token-volume calculation answers "what does a sustained throughput target cost," which is the number a budget line item actually needs. A complete capacity plan runs both: size the node count against the SLO first (§5, §6.1), then price that node count against a token-volume or dollar target (§6.2) to check the SLO-driven topology is also the cost-competitive one at your interactivity target (§4).

One more asymmetry is worth stating plainly, because it is easy to lose when the two rows of Table 2 sit next to each other: the request-rate row produces a hard floor (fewer GPUs than 14 and DistServe's own measured ceilings say the 90% SLO-attainment bar is not met, full stop), while the token-volume row produces a soft target (fewer than 7 nodes and the deployment still serves tokens, just below the 100,000 tokens/s design point, with no SLO violated because none was specified). Treating the two rows as interchangeable -- for instance, rounding the request-rate-derived GPU count down because the token-volume math "looks cheaper" at a slightly smaller footprint -- silently converts a hard latency guarantee into a soft one. The ordering in this section's opening paragraph (topology from §4, node count from the SLO in §5/§6.1, price-check via §6.2) exists specifically so that the hard constraint is fixed before the soft one is allowed to trade against it.

Where experts disagree

Module 3 covered whether disaggregation pays off as a mechanism question, with three positions. This module's stake in the same underlying disagreement is narrower and more practical: which published framework should actually drive a capacity plan's node count -- a goodput-first sizing method, or a TCO-first one? The two frameworks do not always recommend the same topology at the same operating point, and a planner has to pick one to lead.

Size against per-GPU goodput ceilings first

DistServe's own framing1, and the methodology §6.1 applies directly

Goodput-first sizing asks "how many GPUs does an SLO-constrained request rate require" and answers it with a published per-GPU ceiling and a division -- the §6.1 arithmetic, 14 GPUs versus 32. It is the more conservative framework in one specific sense: it guarantees the SLO attainment target is met at the target request rate before any dollar figure is computed, so a plan built this way cannot silently under-provision below the queueing-delay cliff in Figure 6. It generalizes cleanly to the replication reasoning in §5.1: once you have a per-instance ceiling, adding replicas to hit a higher aggregate rate is direct arithmetic, not a new benchmark.

  • Directly enforces the SLO-attainment target the deployment is contractually or practically bound to
  • Composes cleanly with the M/D/1 replication result in §5, since both operate on request rate and per-instance service time
  • Ceilings are measured, not modeled -- DistServe's 1.6/5.6/10 rps figures come from a real evaluation, not a projection
  • Says nothing about dollar cost directly -- two topologies that both clear the SLO can still differ sharply in $/token, and this framework alone will not surface which one to prefer
  • The published ceilings are model- and hardware-specific (OPT-13B, A100); porting them to a different model or GPU generation requires re-measuring, not just re-scaling

Size against TCO-per-million-tokens and the interactivity crossover first

SemiAnalysis, InferenceMAX7, applied as the §4 decision rule

TCO-first sizing asks "what does this workload's interactivity target imply is the cheapest topology" before computing a node count at all, using the crossover from §4 to choose disaggregated or aggregated, then sizes node count against that topology's own per-node throughput and cost. It directly answers the budget-line-item question §6.2's worked example produces, and it is the only framework in this module that lets a genuine crossover -- disaggregation stops being the cheaper choice above roughly 90 tok/s/user -- change which topology gets sized at all, rather than assuming disaggregation (or colocation) is the answer and only asking how many units of it to buy.

  • Directly answers the dollar-cost question a budget actually needs, not just an SLO-attainment guarantee
  • Can flip the topology decision itself (disaggregated vs. aggregated), which goodput-ceiling sizing alone cannot do
  • Uses current-generation hardware (GB200/B200) rather than the A100-era ceilings goodput-first sizing is grounded in
  • Third-party analysis, and the crossover point is an explicit snapshot of an actively-optimized software stack, not a durable hardware property7
  • Optimizing dollar cost alone does not guarantee SLO attainment at the target request rate -- a TCO-optimal topology still needs the §5/§6.1 queueing check run against it before it is provisioned
Assessment

Neither framework alone is a complete sizing methodology, and the published sources do not disagree so much as answer different halves of the same question. Goodput-first sizing (DistServe) guarantees the SLO is met at a target request rate but is silent on cost; TCO-first sizing (InferenceMAX) picks the cheapest topology at a target interactivity but does not itself verify that topology clears the SLO once real arrival-rate variance (§5's queueing math) is accounted for. The defensible sequence, and the one §6's worked table follows, is: use the interactivity target to pick a topology (§4), use goodput ceilings and the M/D/1 replication result to size that topology's node count against the SLO (§5, §6.1), then verify the resulting node count against a token-volume/dollar target (§6.2). What would let a planner skip running both is a single published study that sweeps interactivity the way InferenceMAX does and reports SLO attainment the way DistServe does, on the same hardware generation -- Module 3 already noted nobody has published that; it is the same missing study this module's sizing methodology has to work around.

What is changing

Three numbers in this module are the most likely to move within a year. The EP-degree-vs-per-node-throughput trend in §3.2 is built from three snapshots on two hardware generations, not a controlled study -- a fourth data point at a fourth EP degree, especially on GB200-class hardware with a higher-bandwidth all-to-all fabric, could change the slope or flatten it entirely. The $0.518-vs-$0.20 gap in Table 1 is exactly the kind of number that a future disclosure with matched accounting methodology could either close or explain outright; right now it is evidence of an accounting difference, not a stack-efficiency difference, and that reading could reverse with better data. And the 90 tok/s/user crossover this module leans on as a sizing rule in §4 is, per Module 3, a snapshot of how far Dynamo and TensorRT-LLM's disaggregated-prefill code paths have been optimized -- track InferenceMAX's published results directly before treating that threshold as durable past 20277.

References

  1. paperZhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., Zhang, H. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. USENIX OSDI 2024. arXiv:2401.09670v3. https://arxiv.org/pdf/2401.09670. Accessed 2026-09-17. Used for: per-GPU goodput definition and 1.6/5.6/10 rps ceilings (§1, Fig. 1), M/D/1 queueing model Eq. 1 (§3.1), parallelism diminishing-returns framing (§3.2), the R→R/N replication substitution (§3.3, quoted in §5.1).
  2. officialAnyscale. Understand LLM latency and throughput metrics. docs.anyscale.com, 2026. https://docs.anyscale.com/llm/serving/benchmarking/metrics. Accessed 2026-09-17. Used for: TTFT, TPOT/ITL, and goodput formula definitions (§1).
  3. blogDatabricks Mosaic Research team. LLM Inference Performance Engineering: Best Practices. Databricks Blog, 2023-10-12. https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices. Accessed 2026-09-17. Used for: batch-size/latency/throughput trade-off (4x latency/14x throughput at batch 64), continuous-batching 10-20x gain, GPU-count TTFT ratios for Llama2-70B (2→4 GPUs 0.74x, 4→8 GPUs 0.7x).
  4. repoDeepSeek-AI. Day 6 (One More Thing): DeepSeek-V3/R1 Inference System Overview. open-infra-index, #OpenSourceWeek, 2025-02. https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md. Accessed 2026-09-17. Used for: $87,072/day cost, 226.75-node average, 14,800 tok/s/node decode throughput, 168B daily output tokens, $2/GPU-hr assumption -- consumed here to derive the blended $/1M-token figure and §6.2's worked example.
  5. officialDeepSeek-AI. DeepSeek-R1 Release (API pricing announcement). api-docs.deepseek.com/news, 2025-01-20. https://api-docs.deepseek.com/news/news250120. Accessed 2026-09-17. Used for: R1 cache-hit vs. cache-miss input token pricing ($0.14 vs. $0.55/1M), underlying the §3.3 cache-hit cost-ratio point; H800 hardware note (Table 1).
  6. blogLMSYS Org. Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs. lmsys.org, 2025-05-05. https://www.lmsys.org/blog/2025-05-05-large-scale-ep/. Accessed 2026-09-17. Used for: $0.20/1M output-token figure, EP72/9-node and 16-node-profile decode throughput (22,282 and 18,598 tok/s/node) used in §3.1-3.2.
  7. analysisSemiAnalysis. InferenceMAX: Open Source Inference Benchmarking. InferenceMAX v1, 2026. https://newsletter.semianalysis.com/p/inferencemax-open-source-inference. Accessed 2026-09-17. Used for: TCO-per-million-tokens framing and the ~90 tok/s/user crossover, reprised as a sizing decision rule (§4) and disagreement position (§7).
  8. repoDeepSeek-AI. deepseek-ai/DeepEP (README). GitHub, main branch. https://github.com/deepseek-ai/DeepEP. Accessed 2026-09-17. Used for: measured all-to-all bandwidth by configuration (NVLink EP8 726 GB/s vs. RDMA EP8x2 90 GB/s vs. RDMA EP8x4 61 GB/s), the mechanistic explanation in §3.2.
  9. paperQin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. USENIX FAST 2025 (Best Paper). arXiv:2407.00079v4. https://arxiv.org/pdf/2407.00079. Accessed 2026-09-17. Used for: the 30%→50% cache-hit-ratio-vs-capacity curve underlying the §3.3 cache-hit economics point.