Course · Computer Science

Multi-Node LLM Inference: From One Node to N

Scaling LLM inference serving from one 8-GPU node to N nodes: replication and routing, prefill/decode disaggregation, cross-node MoE, multi-node KV management, and the open-source serving stack.

18 modules7 parts1 appendix≈ 9 h readingSources

Part 1

The Multi-Node Frame

Establishes the bandwidth/latency arithmetic and the routing problem that every later part refers back to: what changes, physically and logically, the instant a second node appears.

01Interconnect and Transport: The Physical Layer of Multi-Node Serving25 min02Replication and Routing: KV-Cache-Aware, Prefix-Aware, and SLO-Aware Request Placement30 min
Part 2

Prefill/Decode Disaggregation

The queueing-theoretic case for splitting prefill and decode into separate pools, and the runtime mechanics -- KV transfer, scheduling, failure handling -- that make the split work.

03Why Disaggregation Pays: Goodput, Interference, and the Ratio Decision25 min04KV Transfer and Scheduling Across the Prefill/Decode Boundary30 min
Part 3

Cross-Node Mixture-of-Experts

Wide expert parallelism as an all-to-all problem distinct from dense-model TP/PP, and DeepSeek's own published production deployment as the grounded reference point.

05Wide Expert Parallelism: DeepEP, Placement, and Load Balancing30 min06DeepSeek's Production Inference System: EP Configurations, Economics, and Reproduction25 min
Part 4

Multi-Node KV Cache Management

Treating KV cache as a first-class, hash-addressed, tiered cluster resource instead of private per-replica state, via Mooncake's architecture and the LMCache/NIXL layer it composes with.

07Prefix Caching and Tiered KV Offload Across Nodes: Mooncake's Architecture40 min08LMCache and NIXL: The Reusable KV Layer and the Transfer-Library Primitive25 min
Part 5

The Open-Source Serving Stack

Project by project: Dynamo, llm-d, vLLM, SGLang, and TensorRT-LLM/Triton, contrasted mechanism by mechanism against the generic patterns established in Parts 1-4.

09NVIDIA Dynamo: Router, Planner, KV Block Manager, and Disaggregation Orchestration35 min10llm-d: The Inference Scheduler, Endpoint Picker, and Kubernetes Gateway API Inference Extension30 min11vLLM Multi-Node: Distributed Serving, Disaggregated Prefill, and the Production Stack30 min12SGLang: Router, PD Disaggregation, and Wide-EP Deployment Patterns30 min13TensorRT-LLM and Triton: Disaggregated Serving on GB20025 min
Part 6

Operations, Economics, and the Frontier

How the fleet is deployed and kept alive, how to size it from published numbers, and what the field has not yet settled.

14Kubernetes Deployment Topologies and Operations at Fleet Scale35 min15Capacity Planning and Economics: Goodput, Tokens/s/$, and Little's-Law Reasoning at Rack Scale30 min16The Research Frontier: Live Migration, Fused Megakernels, and What Is Still Unsettled30 min
Part 7

Contrasting Hardware: AMD and TPU

What transfers from the NVIDIA-centric course and what is a hard substitution, on AMD Instinct and Google TPU.

17AMD MI300X/MI355X Multi-Node Serving: RCCL, InfiniBand/RoCE, and vLLM-ROCm30 min18TPU Multi-Host Serving: ICI, Pathways, and JetStream35 min

Appendices

ABibliography