Multi-Node LLM Inference: From One Node to N · Module 16

The Research Frontier: Live Migration, Fused Megakernels, and What Is Still Unsettled

Parts 1-5 covered a production stack: routers, connectors, wide-EP, tiered KV caches, Kubernetes topologies. Two capabilities sit just outside that stack — cross-instance live request migration and fused-megakernel decode — and one name commonly attached to the second line of work does not exist. This module corrects that name, explains the two real research directions it should point to, and closes the course by recapping, module by module, the disagreements this course has flagged but not resolved.

Course 35 min read 20 sources As of 2026-09-17 Prerequisite: Module 4, Module 6, Module 15
Two research frontiers sit above and below the production stack surveyed in Parts 1-5: cluster-level live migration above, GPU-kernel-level fusion below. macro: cluster scheduling Live migration (Llumnix, OSDI 2024) moves a running request between instances mid-decode not implemented by Dynamo, llm-d, vLLM, SGLang, or TensorRT-LLM (§1) The production stack (Parts 1-5) Routers · connectors · wide EP · tiered KV · K8s topologies Modules 1–14, referenced not re-explained reconfigures by adding/draining workers, not by moving a live request micro: fused decode megakernel (ThunderMLA, on ThunderKittens) — §2 micro: GPU kernel
Figure 1. Two research directions frame this module: one above the production stack (Llumnix's cross-instance live migration, a cluster-scheduling capability1), one below it (ThunderMLA's fused decode megakernel, a kernel-fusion capability built on ThunderKittens2). Neither is part of the routers, connectors, or orchestration layers covered in Modules 1–14.

1. Live migration: Llumnix's virtual-usage abstraction

Llumnix is a scheduling layer, published at OSDI 2024 by Alibaba Group, implemented on top of vLLM as its backend inference engine1. Its premise: once a fleet has more than one instance, the request-to-instance assignment made at admission time is not the only scheduling decision worth making. A request already running on instance A can be moved, mid-decode, to instance B, and Llumnix's contribution is making that move cheap enough — near-zero downtime, independent of how long the sequence already is — that it becomes worth doing routinely rather than only as a last resort1.

1.1 A different move than failure recovery

Module 4 §7 covered what happens when a prefill or decode worker dies, hangs, or becomes unreachable mid-transfer: SGLang's decode-side heartbeat (5.0 s interval, 2 consecutive misses to declare a partner offline, 300 s bootstrap and waiting timeouts)3, TensorRT-LLM's startup-time coordinator-reachability check with bounded retry4, and Dynamo's worker-lifecycle registration and deregistration5. All three of those are reactive: something has already gone wrong, and the mechanism exists to detect that and stop routing new work to the failed component. None of the three engines documents what happens to a specific in-flight request's state when its partner dies mid-transfer — Module 4 flagged that gap explicitly and left it open.

Llumnix's live migration is not a response to that gap. It is a planned, healthy-instance-to-healthy-instance move, triggered by the scheduler's own policy rather than by any failure signal, for four distinct purposes the paper unifies under one abstraction1:

  • Load balancing — reduce preemptions and cross-request interference on an overloaded instance by moving some of its running requests elsewhere.
  • De-fragmentation — a cluster can have enough total free memory to admit a queuing request while no single instance has enough contiguous free memory, because load balancing itself spreads free space thin across instances; migration consolidates it.
  • Prioritization — give a high-priority request dedicated headroom on its current instance by migrating normal-priority requests away from it, without reserving a machine statically.
  • Auto-scaling — drain a terminating instance by migrating its remaining requests out, or saturate a newly launched instance by migrating requests in.

All four are absent from the production stack surveyed in Modules 9-13. Dynamo's Planner scales prefill and decode replica counts against latency targets, not individual requests6; llm-d's Endpoint Picker filters and scores which pod a new request goes to at admission time; and every engine's xPyD reconfiguration adds or drains whole workers5 — none of them relocate one already-admitted, already-decoding request's state from one running instance to another. TensorRT-LLM's own documentation is explicit about how far the closest production-stack feature is from that: dynamic node joining and leaving for a disaggregated deployment is described as "being built on top of" the existing cache-transceiver mechanism, in progress rather than generally available4 — and even once it ships, worker-level join/leave is still not per-request migration.

1.2 One rule, four purposes: virtual usage

The mechanism that unifies all four scenarios is a single scheduling policy — load balancing on a metric Llumnix calls virtual usage — rather than four separate policies. The insight: de-fragmentation, prioritization, and auto-scaling can each be expressed as "make an instance's load balancer think it is more loaded than it physically is," which then triggers the same load-balancing migration machinery1. Concretely, virtual usage of a request equals its physical memory usage, plus a priority-dependent headroom term, plus (for a queuing head-of-line request) its projected memory demand, or infinity for a synthetic request artificially injected to drain a terminating instance1.

Llumnix's virtual-usage rule feeds a single load-balancing policy that produces four different rescheduling behaviors depending on which term dominates. virtual usage = physical usage + headroom(priority) + demand(queuing) + ∞(fake) one load-balancing policy on this one number Load balancing physical usage dominates De-fragmentation queuing demand term Prioritization headroom term Auto-scaling infinity term (fake request) freeness F = (M − ΣV) / B decides who migrates where, every round
Figure 2. Llumnix's CalcVirtualUsage rule adds a headroom, demand, or infinity term to physical memory usage depending on the scenario, then a single load-balancing policy on the resulting "freeness" metric produces load balancing, de-fragmentation, prioritization, and auto-scaling as special cases of the same mechanism1.

The scheduler tracks a per-instance freeness metric \( F = (M - \sum V) / B \), where \(M\) is total instance memory, \(\sum V\) the sum of virtual usages of its requests, and \(B\) the batch size — dividing by batch size converts free memory into "how many more decode iterations this instance's current batch can run before it needs more space." Every scheduling round, Llumnix pairs the instances with the lowest and highest freeness and migrates requests from the low-freeness (overloaded, virtually or physically) instance to the high-freeness one, preferring lower-priority, shorter-sequence requests first1.

1.3 The mechanism: pipelined copy plus a handshake

Migrating a request naively — stop it, copy its entire KV cache, resume it on the destination — would stall the request for the time it takes to move however much state it has accumulated, and that time grows with sequence length. Llumnix instead exploits one property of autoregressive KV cache: it is append-only. Once a block of KV cache is written for a given iteration, it never changes in later iterations1. That means the already-generated portion of the cache can be copied to the destination instance while the source instance keeps computing new tokens, and only the last few blocks — generated after the previous copy round started — remain to be moved once computation actually has to pause.

Source instance pre-allocates request Destination instance reserves blocks or aborts handshake: PRE-ALLOC / ALLOC / ABORT Stage 0: copy completed blocks while decoding continues Stage 1..N-1: copy the few blocks made since last copy Stage N: drain the batch, copy the last iteration's blocks COMMIT decode resumes on destination
  1. Handshake. Before each stage, the source issues a pre-allocation request naming the block count it will send; the destination reserves that space and replies ALLOC, or replies ABORT if it cannot, which cancels the migration cleanly and releases any state already staged1.
  2. Stage 0. The source begins copying the KV blocks generated by all completed iterations so far. Because those blocks are append-only and will not be rewritten, this copy runs concurrently with the source continuing to decode new tokens for this and every other request in its batch.
  3. Stages 1..N-1. Each subsequent stage copies only the small number of new blocks produced since the previous stage's copy started. Because copying is faster than decoding, this catch-up set stays small, and the loop converges — Llumnix reports this pipelined portion of the transfer overlaps decoding almost entirely1.
  4. Stage N. Once the remaining uncopied set is small enough, the source suspends computation for this one request only, draining it out of its running batch, copies the last iteration's blocks, and this is the only stage that actually stalls the request — one iteration's worth of blocks, regardless of total sequence length1.
  5. Commit. The source releases its local blocks and signals the destination to resume decoding the request from its new location. If either side fails mid-protocol, the handshake's per-stage checks abort the migration and roll back cleanly rather than corrupting request state1.

The paper is explicit that naive alternatives — recomputing the prefix from scratch on the destination, or copying the whole cache in one stop-the-world transfer — introduce stalls and downtime "reaching over 50× of the decoding cost," a cost that scales with sequence length, precisely the property the staged protocol eliminates1.

Even Llumnix's own planned-migration mechanism has a reactive failure path layered underneath it, which is worth noting precisely because it shows the two categories are not mutually exclusive. The global scheduler itself can fail: Llumnix falls back to a scheduler-bypassing mode where request frontends dispatch directly to instances by simple rule, migration disabled, until the scheduler recovers. An instance — or its co-located local scheduler and migration coordinator — can also fail outright: every request running on it is aborted, including a request mid-migration whose source instance died, unless its destination was healthy enough to have already committed the transfer; Llumnix relies on Ray to automatically restart the failed actor, after which the service returns to normal scheduling1. So the planned/reactive distinction in Table 2 holds at the level of *why* a rescheduling decision is made, not as a claim that a migration-capable system needs no failure handling at all — it needs both, and Llumnix's own design keeps them as separate mechanisms rather than one subsuming the other.

Worked derivation: how much state actually has to move

Llumnix's own implementation notes give the numbers needed to see why a stop-the-world copy would be expensive enough to motivate staging in the first place. For a 16-bit LLaMA-7B model, one KV-cache block holds the key or value tensor for 16 tokens in one layer, and is fused into a 128 KB contiguous buffer before transfer1. A 1,024-token sequence over the model's 32 layers therefore has, per layer, \(1024 / 16 = 64\) key blocks and 64 value blocks — 128 blocks per layer — and across 32 layers:

$$ N_{\text{blocks}} = \frac{1024}{16} \times 2 \times 32 = 4{,}096 \text{ blocks} $$

matching the paper's own "1k tokens translates to 4k such blocks (32 layers)"1. At 128 KB per block, the total KV-cache footprint for this one request is

$$ 4{,}096 \times 128\text{ KB} = 524{,}288\text{ KB} = 512\text{ MB} $$

Copying 512 MB in one stop-the-world transfer would hold the GPU's decode loop for this request idle for the full transfer duration, and that duration grows linearly with sequence length — a 4,096-token request would need to move 2 GB before it could resume. The staged protocol sidesteps this by moving the bulk of that 512 MB during Stage 0, concurrently with ongoing decode, so only the last iteration's handful of newly appended blocks — not the full 4,096 — are on the critical path in Stage N. That is the concrete mechanism behind the paper's claim of downtime "constant to sequence lengths"1: the quantity that scales with \(L\) is moved off the blocking path, and only an \(O(1)\) slice remains on it.

Table 1. Llumnix vs. INFaaS++ on a 16-GPU LLaMA-7B/30B cluster under realistic traces1.
MetricLlumnix vs. INFaaS++Mechanism
P99 first-token latencyup to 15× betterde-fragmentation reduces queuing delay
P99 per-token generation latencyup to 2× betterload balancing reduces preemptions
High-priority request speedup1.5×prioritization headroom via virtual usage
Cost saving at similar tail latencyup to 36%tighter auto-scaling via migration-assisted draining
Table 2. Module 4's failure handling and Llumnix's rescheduling are different mechanisms solving different problems, not two versions of the same thing1345.
PropertyModule 4: partner-death detectionLlumnix: live migration
Triggera heartbeat or timeout expires; something is already wronga scheduling round; the virtual-usage policy decides to act
State of both endpointsone endpoint is dead, hung, or unreachableboth endpoints are healthy and serving
What movesrouting decisions for future requests, away from the failed workerone already-admitted request's live KV-cache state
Documented per-request outcomenot specified by any of the three engines' own docs (Module 4 §7)resume on destination after Stage N, near-zero downtime
In the Part 5 production stack?yes — SGLang, TensorRT-LLM, Dynamo all document some form of itno — not implemented by any of the five engines covered in Modules 9–13

2. The actual Hazy Research work: Tokasaurus, ThunderMLA, ThunderKittens

Before naming any project, a direct correction: there is no Stanford or Hazy Research project called "Thunder Agent." That name does not appear in Hazy Research's or the (Hazy-Research-adjacent) Scaling Intelligence Lab's published blogs, papers, or repositories728. If that name has come up elsewhere as the label for Stanford's multi-GPU serving or kernel-fusion work, it is a misremembering of one or more of three real, separately named projects: Tokasaurus, a throughput-first inference engine; ThunderMLA, a fused decode megakernel; and ThunderKittens, the tile-programming library both of the above (and much else at Hazy Research) are built on. The rest of this section describes what each of those three actually is.

2.1 ThunderKittens: the tile-kernel DSL underneath both

ThunderKittens is a framework, maintained by graduate students at Stanford's Hazy Research Lab, for writing high-performance CUDA kernels; its own repository describes it as providing "tile primitives for speedy kernels," built around three stated principles — simplicity, extensibility, speed8. It operates at the warp and warpgroup level, giving kernel authors register tiles, shared-memory tiles, and register vectors as primitives, with a minimum tile size of 16×16 values, and abstracts over tensor-core dispatch, shared-memory bank conflicts, asynchronous load/store scheduling, and distributed shared memory across thread blocks8. It is the library that ThunderMLA's megakernel is written in (§2.3) — the "interpreter template" ThunderMLA describes is implemented as a ThunderKittens construct2 — and is used well beyond that one kernel: production users cited by the project include Together AI, Jump Trading, and Cursor, and its kernels cover FlashAttention-3-style attention, GEMM, and linear-attention architectures8. As of the version visible at the time of writing, ThunderKittens 2.0 (released 2026-01-11) added Blackwell GPU support and MXFP8/NVFP4 precision, requiring CUDA 12.8 or later8.

2.2 Tokasaurus: a throughput-first engine, not a latency-first one

Tokasaurus is an inference engine built by Stanford's Scaling Intelligence Lab, explicitly optimized for "throughput-intensive workloads"7 — the total time and cost to process a large batch of sequences, rather than any single request's latency. That framing puts it in a different design space than every engine covered in Modules 9-13, all of which are built to serve interactive, per-request-latency-sensitive traffic. Two mechanisms carry the throughput claim:

  • Dynamic Hydragen grouping. Before every forward pass, Tokasaurus runs a greedy depth-first search over the current batch to find the longest shared prefixes across requests, then groups requests on a shared prefix to run Hydragen-style attention against it once rather than redundantly per request7. This is a batch-time, algorithmic answer to the same prefix-reuse problem Module 2's router-side prefix-cache scoring and Module 7's hash-addressed KV pool solve at the routing and storage layers respectively — Tokasaurus instead exploits shared prefixes inside a single forward pass's attention computation.
  • Adaptive CPU/GPU pipelining. An asynchronous CPU-side manager keeps a deep queue of prepared inputs ahead of the GPU and monitors that queue's depth; if it detects the GPU is at risk of stalling because the queue is close to empty, it starts skipping optional per-step work (stop-string checks, admitting new sequences) until queue depth recovers, trading a small amount of per-step bookkeeping precision for keeping the GPU continuously fed7.

For parallelism across GPUs, Tokasaurus makes a hardware-conditioned choice rather than a single default: on GPUs without NVLink, where tensor-parallel communication overhead is prohibitive, it uses pipeline parallelism, splitting already-large throughput-oriented batches into microbatches across pipeline stages; on NVLink-equipped GPUs (A100, H100, B200) it uses PyTorch's async tensor parallelism, which overlaps communication with compute but adds enough CPU overhead to the forward pass that it only pays off at large batch sizes (roughly 6,000+ tokens), so Tokasaurus keeps separate torch-compiled model graphs and switches TP on only once the batch crosses that size7. On eight NVLink-less L40S GPUs running Llama-3.1-70B, this pipeline-parallel path measured over 3× the throughput of vLLM and SGLang on the same hardware7.

2.3 ThunderMLA: fusing decode into one megakernel

ThunderMLA targets a narrower, deeper problem than Tokasaurus: the decode step itself, specifically for DeepSeek-style multi-head latent attention (MLA, Module 5), where DeepSeek's own FlashMLA kernel is the incumbent. FlashMLA's decode path runs as two separate kernel launches, and Hazy Research's own diagnosis is that the cost is not just the launch overhead itself but the "tail effects between kernel executions" — the second kernel cannot start filling the GPU until the first one's slowest-finishing block has drained, and decode's small batch-and-query counts (a handful of tokens across a handful of prompts) mean there often is not enough parallel work to hide that drain2. On a representative imbalanced-decode workload (batch 4, sequence lengths 4,641-1,730 tokens, 4 new tokens per prompt, 8-way tensor parallel, SXM H100), FlashMLA measured 52 µs, 144 TFLOPs, and 1,199 GB/s of memory bandwidth — well under the H100's 989 TFLOPs / 3.35 TB/s specification ceiling2.

FlashMLA's two-kernel-launch decode path leaves idle gaps between launches; ThunderMLA's fused megakernel runs continuously, with SMs pulling their next instruction from a GPU-resident instruction tensor. FlashMLA (two kernel launches) Kernel 1: attention launch + tail gap Kernel 2: combine 52 µs, 144 TFLOPs, 1,199 GB/s ThunderMLA (one fused megakernel) SMs read next instruction from instruction tensor, no launch boundary 41 µs, 183 TFLOPs, 1,520 GB/s same representative workload: batch 4, mixed sequence lengths, 4 new tokens, 8-way TP, H100 SXM
Figure 4. ThunderMLA replaces FlashMLA's attention-then-combine kernel pair with a single megakernel whose SMs pull work items from a GPU-resident instruction tensor instead of waiting on a second launch, cutting the representative-workload runtime from 52 µs to 41 µs (21% faster) on the same H1002.

The fusion mechanism: ThunderMLA builds "a simple virtual instruction set on the GPU. Instead of launching separate kernels, we fuse them into a single 'megakernel,' and then pass this megakernel its instructions at runtime through an instruction tensor"2. Within that one kernel launch, each SM reads its next instruction from the tensor, executes it, and reads the next one — attention and combine become instruction types dispatched inside one persistent kernel rather than two kernels separated by a launch boundary. Work is assigned to SMs by one of two schedulers built outside the kernel and reused across layers: a static scheduler that splits work into small pieces, builds a k-way reduction tree, and greedily assigns pieces to SMs via a priority queue; and a "makespan backwards" scheduler that starts from the known final reduction step and works backward with heuristic rollouts to choose an execution order that overlaps reduction with compute more tightly, worth roughly an additional 10% on top of the static scheduler2. Scheduling itself costs 1-2 ms, but amortizes because the same schedule is reused across dozens of layers of a large model and can be computed asynchronously on the CPU ahead of when it is needed2.

Table 3. ThunderMLA vs. FlashMLA across four decode workloads, H100 SXM, 8-way tensor parallel2.
WorkloadFlashMLAThunderMLASpeedup
B1, seq 64k, Q155.0 µs / 42 TFLOPs / 1,378 GB/s44.5 µs / 52 TFLOPs / 1,700 GB/s23.6%
B64, seq 256-1024, Q447.0 µs / 124 TFLOPs / 1,212 GB/s39.5 µs / 147 TFLOPs / 2,022 GB/s19.0%
B64, seq 512, Q239.0 µs / 59 TFLOPs / 1,092 GB/s28.6 µs / 80 TFLOPs / 1,489 GB/s36.3%
B132, seq 4k, Q4226.0 µs / 333 TFLOPs / 2,839 GB/s210.0 µs / 358 TFLOPs / 3,055 GB/s7.6%

Across the range Hazy Research reports, ThunderMLA lands 20-35% faster than FlashMLA depending on workload shape2 — the gain is largest for small, imbalanced batches (B64/Q2, 36.3%) where launch and tail-effect overhead is a proportionally larger share of a short kernel's total time, and smallest for large, compute-heavy batches (B132/Q4, 7.6%) where the two-launch overhead is already amortized against a longer-running kernel. Neither Tokasaurus nor ThunderMLA is part of the production stack surveyed in Modules 9-13: vLLM, SGLang, and TensorRT-LLM all still launch attention and combine as conventional, separately dispatched CUDA kernels.

3. State of the unresolved questions

Five modules in this course each surfaced a disagreement between credible, cited sources and did not resolve it — because the evidence needed to resolve it does not yet exist in public form. None of the five is settled as of this module's 2026-09-17 as-of date. Recapped in the order they arose:

3.1 Module 2 — approximate vs. precise KV-cache routing

llm-d's router ships two modes for scoring which replica already holds a request's prefix: an approximate mode that hashes fixed-size blocks and keeps an in-memory LRU index with no query to any model server, and a precise mode that subscribes to real KVEvents streamed over ZeroMQ from each engine, at the cost of a tokenization round trip before every routing decision9. Module 2 found no controlled benchmark comparing the two modes' throughput or TTFT under identical traffic — the choice is currently made on architectural fit (is redundant-prefill cost measured to be high enough to justify the indexer) rather than on a published number.

3.2 Module 3 — DistServe's goodput win vs. InferenceMAX's conditional TCO win

DistServe (OSDI 2024) measures SLO-constrained request rate per GPU on A100-class hardware and finds disaggregation wins by 2.0-12.6× depending on metric, framed as close to unconditional once you optimize for goodput10. InferenceMAX measures TCO per million tokens on GB200/B200-class hardware and finds the win inverts above roughly 90 tokens/s/user of interactivity, where a single aggregated 8-GPU B200 server beats GB200 NVL72 disaggregated prefill11. Module 3's reconciliation: the two studies hold different variables fixed (SLO attainment vs. dollar cost) on different hardware generations three years apart, and InferenceMAX's own disaggregation implementation was, by the analysts' account, still under active optimization at time of measurement11 — so the "crossover point" is a snapshot of one software stack's maturity, not a fixed hardware property. No study has published SLO attainment and TCO-per-token together on the same hardware generation.

3.3 Module 6 — EP320 (December 2024) vs. EP144 (February 2025) for DeepSeek decode

DeepSeek's own V3 technical report specifies a 40-node, 320-GPU decode deployment unit at EP320, one expert per GPU, with IBGDA point-to-point all-to-all12. Two months later, DeepSeek's Day-6 Open Source Week disclosure describes production decode running at EP144 across 18 nodes, two routed experts per GPU13 — a much smaller deployment unit. SGLang's independent reproduction lands at EP72 and recovers decode throughput within 6.6% of DeepSeek's own profile at under a quarter of the GPU count14. Module 6's reading: EP320 is a stated ceiling-case design, EP144 a live production average, and EPLB's own documentation frames larger EP as specifically suited to the decoding stage's "global" load-balancing policy15 without claiming a fixed EP degree — so the two artifacts are not in conflict about a fact, but the general question of how EP degree should scale with decode QPS for any DeepSeek-class model remains open; no source publishes per-expert batch-size telemetry at matched traffic across multiple EP degrees.

3.4 Module 7 — Mooncake's shared pool vs. a simpler point-to-point handoff

Given that KV cache already has to cross a node boundary once prefill and decode are disaggregated, Mooncake goes further and builds a shared, hash-addressed, cluster-wide cache pool with its own global scheduler (Conductor), reporting up to 525% higher throughput in simulated long-context scenarios and 75% more requests served under real Kimi production traffic16. The paper's own numbers cut the other way for the median case: cache-hit ratio on its 23,608-request production trace saturates near 50% even at large capacity, reaching 90% only for specific workloads like chat-to-document16 — meaning roughly half of general-purpose prefill compute is never avoided regardless of pool size, while the shared pool's own hot-spot replication and early-rejection subsystems exist specifically to handle load-fluctuation failure modes that a simpler, non-shared point-to-point handoff would not introduce. Module 7's reading: the shared pool earns its complexity for long, frequently repeated contexts (RAG, agentic tool loops, shared system prompts) and earns it much less for short, largely unique prompts. No ablation isolating the sharing layer's marginal contribution from disaggregation's own decode-isolation effect has been published.

3.5 Module 12 — engine-coupled router vs. separate orchestration layer

sglang-router ships as a single, high-performance Rust-based load balancer with multiple routing algorithms and native prefill-decode disaggregation support, run as one deployable process tightly coupled to SGLang workers only17. Dynamo and llm-d instead pull routing intelligence into a separate orchestration layer that sits above the inference engine and can front vLLM, SGLang, or TensorRT-LLM interchangeably18. Module 12's reading: the embedded-router approach has fewer moving parts and no extra network hop, and is the better default for a fleet that only ever runs one engine; the orchestration-layer approach is engine-agnostic and fits a platform team already standardized on Kubernetes' Gateway API. No published head-to-head benchmark compares sglang-router against Dynamo's Router or llm-d's EPP under an identical PD topology and identical hardware.

Table 4. This course's five open disagreements, where they were raised, and what would settle each.
ModuleQuestionPositionsWhat would settle it
2Approximate hashing or precise event-streaming for prefix-cache routing?llm-d approximate (no engine query) vs. llm-d precise (ZMQ KVEvents)9A controlled throughput/TTFT comparison of both modes under identical traffic
3Does disaggregation pay off unconditionally or only below an interactivity threshold?DistServe/Splitwise goodput win10 vs. InferenceMAX's ~90 tok/s/user TCO crossover11SLO attainment and TCO-per-token measured together, same hardware generation
6How large should decode EP degree be for a DeepSeek-class MoE model?EP320/one-expert-per-GPU12 vs. EP144/EP72, denser per-GPU layout1314Per-expert batch-size telemetry at multiple EP degrees, matched traffic
7Does a shared, hash-addressed KV pool beat simple point-to-point handoff?Mooncake's shared pool, production-adopted16 vs. routing-only reuse (Module 2)Same trace/hardware/PD split, sharing switched off vs. on
12Should routing live inside the engine or in a separate orchestration layer?sglang-router, one process17 vs. Dynamo/llm-d, engine-agnostic layer18Identical PD topology and hardware, sglang-router vs. Dynamo Router vs. llm-d EPP

Where experts disagree

This module's own disagreement is not about a number — it is about where the field's effort to close the gap between research and the production stack (Modules 1-14) should go next. Llumnix argues for smarter macro-level scheduling across instances; Tokasaurus and ThunderMLA argue, implicitly, for faster micro-level kernels within one instance; and the production stack itself, three years into disaggregation and two into wide EP, has adopted neither.

Migration is infrastructure the production stack is genuinely missing

Llumnix's own evidence1, reinforced by Module 4's undocumented mid-transfer recovery gap

A 15× P99 TTFT improvement and 36% cost saving on a 16-GPU cluster (Table 1) is a large, measured, reproducible gain from a mechanism that costs near-zero downtime to use. Every one of the four purposes it serves — load balancing, de-fragmentation, prioritization, auto-scaling — maps onto a real operational pain point the production stack still handles by cruder means: adding or draining whole workers (Modules 9, 14), not relocating individual requests. Module 4's own finding, that none of SGLang, TensorRT-LLM, or Dynamo documents what happens to a request whose transfer partner dies mid-flight, is a second, independent symptom of the same absence: a fleet that could migrate a request away from a struggling instance would have a natural answer to that gap, and today none of the three engines has one.

  • Directly measured, large, multi-dimensional gains on a real (if modest, 16-GPU) cluster
  • Addresses an operational gap (mid-transfer partner death) this course independently identified in Module 4
  • The mechanism generalizes across four distinct scheduling problems rather than solving one narrowly
  • Evaluated on A10/LLaMA-7B-scale hardware and models, two generations behind the H100/GB200-scale deployments Modules 6, 9, and 13 measure
  • Adds a cluster-wide scheduling component (global scheduler plus per-instance llumlet) that itself becomes a scaling and failure-mode concern at fleet scale

Kernel-level fusion delivers more, sooner, without new orchestration surface

Tokasaurus's and ThunderMLA's implicit bet72

Neither Tokasaurus nor ThunderMLA touches cluster scheduling at all — both work entirely within a single kernel launch or a single engine process, and both ship measured gains (over 3× throughput for Tokasaurus's pipeline-parallel path, 20-35% for ThunderMLA's megakernel) without adding a global scheduler, a migration protocol, or a new failure mode at the cluster level. If the bottleneck a deployment actually has is GPU idle time from kernel-launch overhead and tail effects — which ThunderMLA's own FlashMLA baseline measurement shows leaves an H100 running at 144 of 939 advertised TFLOPs — then no amount of smarter cross-instance scheduling fixes that; the fix has to happen inside the kernel. This is a bet that per-GPU efficiency, not cross-GPU coordination, is the more tractable and more broadly applicable lever.

  • No new cluster-level component, protocol, or failure mode introduced
  • Gains compound with whatever scheduling policy is already in place — a faster kernel helps under any router or connector
  • Directly measured against a real, current production baseline (DeepSeek's own FlashMLA)
  • Solves a different problem than Llumnix's — a perfectly fused megakernel still cannot rebalance load, de-fragment memory, or drain an instance across a cluster
  • Neither project is integrated into vLLM, SGLang, or TensorRT-LLM; adopting either means running a research codebase in production or reimplementing the technique inside a mainstream engine

Neither is necessary yet — the existing stack already covers most of the benefit

Implicit in what Modules 9, 10, and 14 already ship

Dynamo's Planner already rescales prefill and decode replica counts against measured SLA targets6; llm-d's Endpoint Picker already scores every new request against live prefix-cache and load state before admission (Module 2); and xPyD reconfiguration (Module 4) already adds and drains whole workers at runtime. None of that is migration, and none of it is a fused megakernel — but all three together address load imbalance, memory pressure, and elastic capacity at the admission boundary rather than mid-request, which is where most of a fleet's scheduling decisions already happen. A request that would benefit from migration in Llumnix's evaluation is, in a stack with good prefix-and-load-aware admission routing, less likely to land somewhere it needs to be moved away from in the first place.

  • Zero new components — uses only mechanisms this course has already covered as shipping
  • Admission-time correction is simpler to reason about and debug than mid-request relocation
  • Admission-time routing cannot fix a request that becomes a problem after it is already running — a long-context request that grows past what its instance can hold has no recourse but preemption in the mainstream stack today
  • This is an absence of counter-evidence, not a measured comparison; nobody has quantified how much of Llumnix's 15× gain a well-tuned admission-time router alone would recover
Assessment

These are not substitutes for each other, because they operate at different layers: migration is a cluster-scheduling capability, kernel fusion is a per-GPU efficiency capability, and admission-time routing is what the production stack already has instead of either. A deployment could in principle adopt all three simultaneously with no conflict. The honest read of the evidence is that neither Llumnix nor the ThunderKittens-based kernels have been shown, in a head-to-head test against the mainstream stack's own mechanisms, to be necessary rather than merely sufficient: Llumnix was evaluated against INFaaS++, not against a Dynamo-Planner-plus-llm-d-EPP baseline (Table 1), and ThunderMLA was evaluated against FlashMLA in isolation, not against a full serving stack where kernel time is one term among several in end-to-end latency. What would actually settle which investment matters more for a given deployment is the comparison neither research artifact provides: the same traffic trace, the same hardware, once through today's admission-time-routing-only stack and once with migration or fused kernels added on top, isolating each mechanism's marginal contribution rather than comparing it only against an older or narrower baseline.

What is changing

The clearest signal of what is likely to be stale within a year is not a prediction — it is the pace of change already visible in the sources this course cites. Four items are worth flagging explicitly rather than treating this course's numbers as fixed.

Table 5. Claims in this course most likely to move within a year, and why.
ClaimSourceWhy it is likely to move
Dynamo's component set and headline benchmark numbers (Module 9)v1.4.2, 2026-08-2919Patch releases have shipped roughly monthly through 2026 (v1.4.0 → v1.4.1 → v1.4.2 in weeks), with a v1.5.0-dev TensorRT-LLM container already building19
NIXL's supported backend list and transfer bug surface (Module 8)v1.4.1, 2026-09-0120v1.4.1 closed a use-after-free window; v1.4.0 changed descriptor memory footprint by three orders of magnitude (44 MB → 7 KB); v1.3.2 fixed a transfer hang introduced one version earlier20 — the abstraction is still actively settling, not stable
DeepSeek-class decode EP degree as "the" reference configuration (Module 6, §3.3 above)EP32012 → EP14413 in two monthsAlready moved once on public record; nothing in either disclosure suggests 144 is a final number rather than another snapshot of aggregate traffic at the time it was written
The ~90 tok/s/user disaggregation TCO crossover (Module 3, §3.2 above)InferenceMAX v111Measured on one hardware generation (GB200 NVL72 vs. B200) with a disaggregation software stack the analysts themselves describe as still under active optimization; a new GPU generation or a more mature disaggregated-prefill implementation shifts the crossover point, not just the absolute numbers either side of it

None of this is a reason to distrust the numbers as of 2026-09-17 — every one of them carries a citation and an accessed date for exactly this reason. It is a reason to re-check version numbers and benchmark conditions before repeating any of them a year out, particularly the two version-churn rows, which are near-certain to have moved by the time this module is read again.

References

  1. paperSun, B., Huang, Z., Zhao, H., Xiao, W., Zhang, X., Li, Y., Lin, W. Llumnix: Dynamic Scheduling for Large Language Model Serving. arXiv:2406.03243v1, USENIX OSDI 2024. https://arxiv.org/pdf/2406.03243. Accessed 2026-09-17. Used for: the four rescheduling scenarios and virtual-usage abstraction (§4.4, Figure 1, Algorithm 1), the multi-stage live-migration protocol and handshake (§4.2-4.3, Figures 6-8), the freeness metric and Gloo-based transfer implementation (§4.4, §5), the KV-cache block-fusion sizing numbers (§5, "Block fusion"), and the 16-GPU evaluation results (§6, Table 1).
  2. blogHazy Research. ThunderMLA: FlashMLA, Faster and Fused-er! Stanford University, published 2025-03-04. https://hazyresearch.stanford.edu/blog/2025-03-04-thundermla. Accessed 2026-09-17. Used for: the FlashMLA two-kernel-launch diagnosis, the megakernel/virtual-instruction-set mechanism, the static and makespan-backwards schedulers, and the four-workload benchmark table against FlashMLA.
  3. officialSGLang contributors. PD Disaggregation. SGLang Documentation, docs.sglang.io (live), 2026. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: the SGLang heartbeat/bootstrap-timeout failure-detection defaults, recapped in contrast to Llumnix's planned migration.
  4. officialNVIDIA / TensorRT-LLM contributors. Disaggregated Serving (disagg-serving.md). TensorRT-LLM documentation, main branch, 2026. https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disagg-serving.md. Accessed 2026-09-17. Used for: TensorRT-LLM's startup-time coordinator-reachability check and its explicitly not-yet-GA dynamic node joining/leaving status.
  5. officialNVIDIA. Disaggregated Serving (NVIDIA Dynamo Documentation). docs.nvidia.com/dynamo/knowledge-base, 2026. https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. Accessed 2026-09-17. Used for: Dynamo's runtime-reconfigurable xPyD worker registration/deregistration mechanism, contrasted against per-request live migration.
  6. officialNVIDIA. Planner Guide (NVIDIA Dynamo Documentation, v1.2.1). https://docs.nvidia.com/dynamo/v1.2.1/components/planner/planner-guide. Accessed 2026-09-17. Used for: the Planner's replica-count autoscaling framing, cited to show it operates on worker counts, not individual requests.
  7. blogScaling Intelligence Lab (Hazy Research-adjacent group). Tokasaurus: An LLM Inference Engine for High-Throughput Workloads. Stanford University, published 2025. https://scalingintelligence.stanford.edu/blogs/tokasaurus/. Accessed 2026-09-17. Used for: dynamic Hydragen grouping, adaptive CPU/GPU pipelining, the pipeline-parallel-vs-async-TP hardware split, and the >3× L40S throughput measurement.
  8. repoHazyResearch contributors. ThunderKittens (GitHub repository README). Stanford University, Hazy Research Lab, main branch (v2.0, released 2026-01-11). https://github.com/HazyResearch/ThunderKittens. Accessed 2026-09-17. Used for: what ThunderKittens actually is (a tile-primitive CUDA kernel framework), its warp/warpgroup abstraction, production users, and version history — fetched directly for this module to ground the "Thunder Agent" naming correction; not in the course's original source map.
  9. officialllm-d contributors. Prefix-Cache Aware Routing (llm-d docs). docs/dev (live), 2026. https://llm-d.ai/docs/dev/architecture/advanced/kv-management/prefix-cache-aware-routing. Accessed 2026-09-17. Used for: the approximate (block-hash/LRU) vs. precise (KVEvents/ZMQ) routing modes recapped from Module 2.
  10. paperZhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., Zhang, H. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arXiv:2401.09670v3, USENIX OSDI 2024. https://arxiv.org/pdf/2401.09670. Accessed 2026-09-17. Used for: the near-unconditional SLO-constrained goodput win recapped from Module 3.
  11. analysisSemiAnalysis. InferenceMAX: Open Source Inference Benchmarking. InferenceMAX v1, 2026. https://newsletter.semianalysis.com/p/inferencemax-open-source-inference. Accessed 2026-09-17. Used for: the ~90 tok/s/user TCO crossover recapped from Module 3, and its flag as likely to move as hardware and the disaggregation stack mature.
  12. paperDeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024. https://arxiv.org/pdf/2412.19437. Accessed 2026-09-17. Used for: the December 2024 EP320/40-node decode deployment unit recapped from Module 6.
  13. repoDeepSeek-AI. Day 6 (One More Thing): DeepSeek-V3/R1 Inference System Overview. open-infra-index, #OpenSourceWeek, published 2025-02. https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md. Accessed 2026-09-17. Used for: the February 2025 EP144/18-node decode configuration recapped from Module 6.
  14. blogLMSYS Org. Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs. lmsys.org, published 2025-05-05. https://www.lmsys.org/blog/2025-05-05-large-scale-ep/. Accessed 2026-09-17. Used for: SGLang's independent EP72 reproduction within 6.6% of DeepSeek's own decode profile, recapped from Module 6.
  15. repoDeepSeek-AI. deepseek-ai/EPLB (README). Main branch, 2026. https://github.com/deepseek-ai/EPLB/blob/main/README.md. Accessed 2026-09-17. Used for: EPLB's "global" load-balancing policy framing for the decoding stage, recapped from Module 6.
  16. paperQin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., Xu, X. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079v4, USENIX FAST 2025 (Best Paper Award). https://arxiv.org/pdf/2407.00079. Accessed 2026-09-17. Used for: the shared, hash-addressed cache pool's measured gains and its own cache-hit-ratio ceiling, recapped from Module 7.
  17. repoSGLang contributors. sglang-router (PyPI package page). SGLang / RadixArk, 0.3.2, 2026-01-15. https://pypi.org/project/sglang-router/. Accessed 2026-09-17. Used for: the sglang-router description recapped from Module 12's engine-coupled-router position.
  18. repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo (README). v1.4.2 (container tags), 2026. https://github.com/ai-dynamo/dynamo. Accessed 2026-09-17. Used for: Dynamo's self-description as an engine-agnostic orchestration layer fronting vLLM/SGLang/TensorRT-LLM, recapped from Module 12's separate-orchestration position.
  19. repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo Releases. v1.4.2 (2026-08-29). https://github.com/ai-dynamo/dynamo/releases. Accessed 2026-09-17. Used for: Dynamo's monthly-cadence patch release history, cited as evidence of version churn in "What is changing."
  20. repoNVIDIA / ai-dynamo contributors. ai-dynamo/nixl Releases. v1.4.1 (2026-09-01). https://github.com/ai-dynamo/nixl/releases. Accessed 2026-09-17. Used for: NIXL's recent bug-fix and descriptor-format changes, cited as evidence of version churn in "What is changing."