Multi-Node LLM Inference: From One Node to N · Module 18
TPU Multi-Host Serving: ICI, Pathways, and JetStream
Everything in Modules 1–16 assumed NVLink/NVSwitch inside a node, InfiniBand or RoCE between nodes, and an orchestrator (Dynamo, llm-d) coordinating independently-running vLLM/SGLang/TensorRT-LLM engines. TPU multi-host serving keeps the same questions — how far can one low-latency domain reach, how do you scale past it, what serves the request — but answers them with a different topology (ICI mesh/torus, not a switched fabric), a different orchestration model (Pathways' single controller, not an external scheduler over autonomous engines), and a serving-engine lineage (JetStream, now folding into vLLM's tpu-inference) that is still consolidating as of this writing. After this module you can map every mechanism in this course onto its TPU-side equivalent, structural substitute, or documented absence.
1. TPU pod anatomy and ICI: a mesh, not a switch
Module 1 built the NVLink/NVSwitch baseline on one premise: every GPU in an 8-GPU node reaches every other GPU at the same per-GPU rate, in one hop, because the NVSwitch fabric gives full bisection bandwidth2. Google's TPU Inter-Chip Interconnect (ICI) does not give you that premise. It gives every chip a fixed number of direct wires to its nearest neighbors in a grid, and lets you reach a farther chip only by routing across several of those links. The rest of this module is downstream of that one structural fact, so it is worth pinning down precisely, generation by generation.
1.1 TPU v6e (Trillium): 256-chip pod, 2D torus
A TPU v6e Pod contains 256 chips, each carrying 800 GB/s of bidirectional ICI bandwidth, 32 GB of HBM at 1,638 GB/s, and a peak of 918 TFLOPs bf16 (1,836 TOPs Int8) per chip; the pod-level BF16 peak is 234.9 PFLOPs1. The interconnect topology is a 2D torus1: each chip connects to its four nearest neighbors — north, south, east, west — and those links wrap around at the pod's edges, so the last chip in a row connects directly back to the first. That wraparound is what makes it a torus rather than a plain mesh: it removes the edge chips that would otherwise have fewer neighbors and halves the worst-case hop count between any two chips, since you can now go around the short way in either direction3.
1.2 TPU7x (Ironwood): 9,216-chip pod, 3D torus inside 64-chip cubes, OCS between cubes
TPU7x (Ironwood), Google's most recent generation, scales the same idea two dimensions further and one level higher. Each chip carries 192 GB of HBM at approximately 7.37 TB/s, 2,307 TFLOPs BF16 (4,614 TFLOPs FP8) of peak compute, two TensorCores and four SparseCores, and 1,200 GB/s of bidirectional ICI bandwidth4. Chips are wired into a 3D torus (topology strings of the form AxBxC, with dimensions that must be multiples of four once a slice exceeds 64 chips), and a pod scales up to 9,216 chips4 — comparable in chip count to the earlier v5p generation4.
The mechanism that gets from a single 3D-torus cube to a 9,216-chip pod is where TPU7x genuinely departs from anything on the GPU side. Chips inside one 64-chip (4×4×4) cube are wired directly over ICI in the 3D torus; multiple cubes are then joined by an Optical Circuit Switch (OCS) network, described by Google as "a dynamic, reconfigurable optical network"5. The official architecture reference calls out the OCS's role for fault tolerance specifically: "ICI resiliency" lets connections be "routed around OCS and optical ICI faults"3 — the optical layer can be reconfigured in the field to route around a failed link or a failed cube, something a fixed-topology electrical switch (NVSwitch) or a statically-cabled fat-tree (InfiniBand) does not do at the same layer. Module 1's NVLink/NVSwitch and InfiniBand/RoCE treatment has no OCS-equivalent component; ConnectX-7's adaptive routing sprays one flow's packets across several paths within a fixed physical topology (Module 1 §2.1) — a related but distinct idea, since it is packet-level path diversity, not a physically reconfigurable optical fabric.
1.3 Why a torus is a different kind of thing than a switched fabric
Put the two facts about Module 1's fabrics and this module's fabric side by side. NVSwitch and InfiniBand are both, structurally, switched networks: a packet goes from source to a switch (or a small number of switch tiers) to destination, and the switch's job is to give every pair of endpoints roughly the same latency and bandwidth regardless of physical position — that is the definition of full bisection bandwidth, and it is why Module 1 could say "any two of the eight GPUs" without qualification. ICI is a direct point-to-point mesh: there is no switch chip in the middle, only wires between physically adjacent chips, and the "distance" between two chips is a real, position-dependent hop count across the torus. Reaching a chip that is two hops away costs more latency and consumes ICI bandwidth on two links instead of one; reaching the diagonally opposite corner of a large torus can cost several hops. Google's own architecture documentation frames the ICI-connected set of chips within one pod as a "slice"3 precisely because a chip's position inside that mesh is a first-class fact about the deployment, not an implementation detail hidden behind a switch.
This is the single fact that should recalibrate any instinct carried over from the NVLink/NVSwitch mental model: "how many chips can I put in one interconnect domain" is not the same question on a torus as on a switched fabric, because the cost of a communication pattern on a torus depends on where in the mesh the participating chips sit, not just on whether they share a domain. All-reduce and all-to-all algorithms tuned for a torus (ring or hypercube-style schedules that follow the physical adjacency) look different from the switch-agnostic collectives Module 1 and Module 5 describe for NVLink domains — a distinction this module flags without re-deriving, since the collective-algorithm level is not covered by the sources examined for this module.
| Platform | Chips/GPUs per unit | Topology | Per-chip interconnect BW | Per-chip/GPU HBM |
|---|---|---|---|---|
| TPU v6e (Trillium) | 256 / pod | 2D torus | 800 GB/s (ICI) | 32 GB @ 1,638 GB/s |
| TPU7x (Ironwood) | 9,216 / pod | 3D torus + OCS between cubes | 1,200 GB/s (ICI) | 192 GB @ ≈7.37 TB/s |
| GB200 NVL72 | 72 / rack | NVSwitch, full bisection | 1.8 TB/s (NVLink5) | ≈186 GB @ ≈8 TB/s |
GB200 NVL72 row derived from its published rack aggregates — 130 TB/s NVLink bandwidth and 13.4 TB of HBM3e at 576 TB/s across 72 GPUs6 — divided by 72 to put it on the same per-chip basis as the TPU rows; NVIDIA does not publish these as per-GPU figures directly, but the arithmetic matches the per-GPU NVLink5 rate from Module 1 §1.2 exactly (130 TB/s ÷ 72 ≈ 1.8 TB/s).
2. The inference-optimized slice: v6e-8 as the "8-GPU node," and where the analogy breaks
Google's own terminology gives you a direct on-ramp from the GPU vocabulary: a TPU Pod is "a contiguous set of TPUs grouped together over a specialized network," and a slice is "a collection of chips all located inside the same TPU Pod connected by high-speed inter-chip interconnects"3. A slice is the unit you actually allocate and serve on — the direct analog of "one 8-GPU node" from Module 1, and the unit every later section of this module (Pathways deployments, disaggregation, checkpoint loading) is scoped to.
2.1 The 8-chip inference slice
Google calls out an 8-chip configuration specifically for serving: "slices with 8 chips (v6e-8) attached to a single VM are optimized for inference, allowing all 8 chips to be used in a single serving workload"1. That is deliberately the same chip count as an HGX 8-GPU node, and the parallel is not an accident of phrasing — both are the smallest unit that lets a serving engine treat the whole local interconnect domain as one addressable pool for tensor/expert-parallel sharding. Inside a v6e-8 slice, any two chips are within the 2D torus's 8-chip footprint, so the hop-count penalty from §1.3 stays small; that is why Google marks this particular shape, not an arbitrary 8-chip subset, as the inference-optimized one.
2.2 Scaling past one slice: 1×1 to 16×16
Multi-host configurations scale from a 1×1 topology up to 16×16 — the full 256-chip v6e pod — spread across as many as 64 VMs1. Each larger topology is still one ICI-connected slice by the §2 definition; what changes is how many separate host VMs' worth of chips participate, and therefore how many physical hosts a single-controller runtime (§3) has to coordinate to treat the slice as one serving unit.
2.3 Where the 8-GPU-node analogy breaks: chip-to-host ratio
The place the parallel stops holding cleanly is the chip-to-host ratio, and it is not the same across TPU generations either. A v6e-8 inference slice puts all 8 chips behind a single VM1 — matching HGX's 8-GPUs-per-host ratio exactly. TPU7x does not: "each host is connected to four chips"3, so a TPU7x VM is a 4-chip unit, and a 2x2x1 topology (4 chips) is explicitly the single-host case, with 2x2x2 and larger being multi-host3. A 4-chip TPU7x VM ships with 224 vCPUs and 960 GB of host RAM4 — a much larger host-CPU/host-RAM allowance per accelerator than a typical 8-GPU HGX host, reflecting a design point where the host is doing comparatively more work (input pipeline, SparseCore-adjacent preprocessing) per accelerator chip than a GPU host typically does.
The practical consequence: on TPU7x, "one node" in the GPU sense (one physical host, one PCIe/NIC domain) is a 4-chip unit, not an 8-chip one, and reaching the other 4 chips of even the smallest useful inference topology already means crossing a host boundary — still within ICI, since ICI links the chips directly rather than routing through host networking, but crossing a boundary that matters for anything host-scoped (process placement, host-local memory, VM lifecycle). This is exactly the kind of substitution the rest of this course has trained you to look for: same word ("node"), different physical referent, and the difference is load-bearing the moment you reason about failure domains or checkpoint-loading parallelism (§7).
3. Pathways: single-controller orchestration as the TPU answer to Dynamo/llm-d
Module 9 and Module 10 covered two different answers to "who decides which worker serves this request and manages this worker's lifecycle": Dynamo's Router/Planner/KVBM/Grove component set, and llm-d's Endpoint-Picker-based layered scheduler, both external to, and coordinating, independently-running vLLM/SGLang/TensorRT-LLM engine processes. Pathways is Google's answer to the same question, and it takes a structurally different position: instead of an external orchestrator issuing requests to autonomous per-GPU engine processes, Pathways puts a single controller in charge of the whole distributed computation and lets that controller schedule work across every chip in the deployment as one program.
3.1 Single-controller vs. multi-controller (SPMD)
The distinction has a name in the systems literature: Pathways "makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane, and with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns"7. Contrast this with the SPMD (single-program, multiple-data) style that the vLLM/SGLang/TensorRT-LLM stack in Modules 9–13 uses: each GPU process runs the same program independently, and coordination between them happens through explicit collective calls (all-reduce, all-to-all) that every rank must call in the same order — a multi-controller model, in the paper's terms, where there is no single point that sees the whole computation. Pathways' single controller can instead express computations that are heterogeneous across parts of the cluster — different pipeline stages, different parallelism strategies on different islands of chips — because one program, running on the controller, is deciding what every chip does, rather than each chip's process deciding for itself from an SPMD script7.
The paper reports concrete evidence this design does not sacrifice throughput to get that flexibility: Pathways achieves near-100% accelerator utilization for SPMD-style computations across 2,048 TPUs, and delivers throughput comparable to the SPMD case even for a Transformer model pipelined across 16 stages, or sharded across two separate islands of accelerators connected only by a data-center network7. That last case — two islands joined over a DCN, not a shared low-latency fabric — is the direct academic ancestor of the disaggregated-slice configuration in §5.
3.2 Pathways on Cloud: the four containers you actually deploy
The production system built on this idea, Pathways on Cloud, ships as four containers wired together by a Kubernetes LeaderWorkerSet (LWS) — the same primitive Module 14 covers as the multi-host replication unit for sharded GPU serving. Google's own multi-host inference guide names them explicitly8:
- pathways-proxy — listens on port 38681 and forwards requests to the resource manager at the LWS leader's address.
- pathways-rm (resource manager) — runs with
--node_type=resource_manageron port 38677, and manages TPU topology and instance configuration via an--instance_countparameter and an--instance_type=tpuv6e:TOPOLOGYparameter. - jax-tpu — the actual model-inference engine, running MaxText or JetStream underneath, exposing health checks on port 8000.
- jetstream-http — the HTTP server, also on port 8000, that provides the client-facing inference API.
These are deployed as an LWS group whose size matches the topology — for example, size: 5 (one leader plus four workers) for a v6e-16 slice laid out as a 4x4 topology, with a leaderworkerset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-nodepool annotation binding the group to one physical node pool8. The leader hosts the resource manager and proxy; workers connect back to the leader's resource-manager address, and the resource manager tracks every participating slice through its instance_count.
jetstream-http serves the request-facing API and calls the jax-tpu engine over ICI within the slice, while pathways-proxy and pathways-rm form the control plane that provisions and tracks the group's topology8.3.3 What is shared with Dynamo/llm-d, and what is not
The Kubernetes primitive is literally shared: Pathways on Cloud deploys over LeaderWorkerSet8, the same API Module 14 covers as the substrate llm-d and vLLM's own multi-host guides assume for gang-scheduling a sharded replica across physical hosts. That is a genuine 1:1 port — the Kubernetes-level "how do I deploy N hosts as one addressable replication unit" problem has the same answer on both stacks.
Above that layer, the models diverge. Dynamo's Planner and llm-d's Endpoint Picker both sit outside the inference engine process and make decisions — which replica gets a request, when to scale a pool — by observing metrics that engines expose (Module 9 §3, Module 10 §3), while the engines themselves remain ordinary SPMD processes unaware of the orchestrator's existence beyond the metrics endpoint. Grove, Dynamo's Kubernetes operator, does topology-aware gang scheduling as a control-plane concern layered on top of independently-running engine pods. Pathways instead puts scheduling inside the single-controller runtime that the engine itself is built on: jax-tpu is not an autonomous SPMD process that happens to expose metrics to an external scheduler, it is a worker in a Pathways-coordinated dataflow graph whose placement and gang-scheduling the resource manager already owns as part of running the program at all7. There is no Pathways-side analog to Dynamo's Router doing KV-cache-aware request-to-replica scoring (Module 9 §2) or llm-d's prefix-cache scorer (Module 2, Module 10) documented in the sources examined here — Pathways' multi-host inference documentation describes topology and lifecycle management, not cross-replica KV-aware routing, and no source surveyed for this module describes such a component.
4. JetStream, and its 2026 consolidation into tpu-inference
This module's manifest scope names JetStream as TPU's serving engine, parallel to Module 11–13's vLLM/SGLang/TensorRT-LLM. That framing is accurate for most of JetStream's life, and the mechanism is worth understanding on its own terms — but as of this writing the project has just been folded into a different, vLLM-owned repository, and the currency of that change matters enough to state plainly rather than gloss over.
4.1 What JetStream is
JetStream describes itself as "a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future)"9. Structurally it ships as an orchestrator around a pluggable "engine" interface with two reference implementations: a JAX engine, backed by Google's MaxText training/inference framework, and a PyTorch engine, backed by a separate jetstream-pytorch project9. The JAX/MaxText path is the one Google's own documentation and blog posts use for concrete numbers, and its server is launched as MaxText.maxengine_server, configured with MaxText's own sharding flags — ici_fsdp_parallelism, ici_tensor_parallelism, ici_autoregressive_parallelism — that map a model's parallelism strategy onto ICI-connected chip axes, plus max_prefill_predict_length and per_device_batch_size for the serving-specific prefill/decode shape9. That JAX engine is what Pathways' jax-tpu container (§3.2) runs underneath its HTTP surface. Concretely, a single-host launch binds parallelism directly onto ICI axes as explicit flags rather than as a separate topology descriptor:
python3 -m MaxText.maxengine_server \
MaxText/configs/base.yml \
tokenizer_path=${TOKENIZER_PATH} \
load_parameters_path=${LOAD_PARAMETERS_PATH} \
model_name=${MODEL_NAME} \
max_prefill_predict_length=1024 \
max_target_length=2048 \
ici_fsdp_parallelism=1 \
ici_tensor_parallelism=-1 \
ici_autoregressive_parallelism=1 \
per_device_batch_size=11 \
scan_layers=false \
weight_dtype=bfloat16
The three ici_* flags partition the slice's chips along FSDP, tensor-parallel, and autoregressive (decode-batch) axes; -1 tells MaxText to infer that axis's size from the others and the slice's total chip count9. There is no separate "which chips are in my ICI domain" configuration step — the flags assume the whole slice is one addressable pool, consistent with §1.3.
4.2 April 2025: Pathways integration, multi-host and disaggregated numbers
Pathways was integrated into JetStream for the first time in April 2025, and Google's own numbers from that release are the concrete evidence for why multi-host and disaggregated serving matter operationally, not just architecturally. On the multi-host side, Llama 3.1 405B — a model that does not fit on a single host at any reasonable precision — reached 1,703 tokens/s on Trillium (v6e) via Pathways-enabled multi-host inference, a figure Google reports as 3× more inference per dollar than TPU v5e10. On the disaggregated side, the same release reports, for Llama2-70B on Trillium, a 7× improvement in time-to-first-token and roughly 3× improvement in time-per-output-token from splitting prefill and decode into independently-scaled stages versus a monolithic engine10. These are Google's own headline numbers, model- and generation-specific, in the same vendor-reported category as the Dynamo and llm-d claims in Modules 9–10 — read them the same way, as an existence proof of the technique's direction rather than a portable constant.
The same April 2025 update is also where vLLM support on TPU first appears in Google's own materials, explicitly framed as "an alternative to JetStream," giving customers "choice"10 rather than a replacement. That framing did not last unchanged.
4.3 February 2026: JetStream's archival and the move to tpu-inference
As of this writing, the AI-Hypercomputer/JetStream repository carries an explicit archival notice: "In an effort to streamline TPU inference efforts in open source, we have migrated core functionality in JetStream to the new tpu-inference repository. For this reason, we will be archiving JetStream on February 1st 2026"9. The successor, vllm-project/tpu-inference, describes itself plainly: "vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path within the vLLM project"12. Concretely, tpu-inference lowers every model — including PyTorch model code, via a PyTorch-to-JAX bridge called Torchax — through one JAX→XLA compilation path, rather than maintaining a separate PyTorch/XLA path; the vLLM project reports this change alone produced roughly 20% higher throughput than the earlier direct PyTorch/XLA route, because it lets PyTorch models reuse JAX's compiler primitives11, and 3.6× (Llama 3.1-8B on v6e-1) to 2.1× (Llama 3.1-70B on v6e-8) throughput over the original, February-2025-era TPU prototype11.
The practical reading for a deployment decision today: "JetStream" as a name still appears in Google's own multi-host-inference documentation, describing what the jax-tpu container runs (§3.2, §4.1) — but the open-source repository that used to be the reference implementation is archived, and its functionality now lives inside vLLM's TPU backend. tpu-inference lists v7x, v5e, and v6e as its recommended generations and v3/v4/v5p as experimental12. This is a live consolidation, not a settled state; treat any specific feature-support claim in §5–6 as dated to the snapshot cited, not as a permanent fact.
5. Disaggregated serving on TPU: a DCN cliff, not an ICI one
Module 3 established why prefill and decode interfere when colocated on one engine, and Module 4 covered the connector abstraction GPU serving stacks use to move KV cache between separate prefill and decode pools. The interference argument — compute-bound prefill work stalling memory-bandwidth-bound decode steps — is a property of the workload, not of NVLink or ICI, so there is no reason it would not apply on TPU as well. What is worth working through carefully is how the split is implemented once ICI is the intra-slice fabric, because the natural assumption — "ICI is already one fast mesh, so disaggregation must mean something different here" — turns out to be only half right.
5.1 Aggregated one-slice vs. disaggregated two-slice deployment
Google's Pathways multi-host inference guide gives two concrete example topologies. The aggregated case is a single v6e-16 slice with a unified LeaderWorkerSet group running both prefill and decode on the same engine, the same chips, the same ICI domain8. The disaggregated case does not subdivide that one slice internally — it uses two separate v6e-8 slices, one for prefill and one for decode, configured via a Kubernetes subGroupPolicy with subGroupSize: 2 and an inference-server mode named ExperimentalMaxtextDisaggregatedServer_88 — the "Experimental" prefix in that config name is Google's own, not an editorial addition. The same documentation defines the two stages precisely: "prefill: this stage processes the input prompt and generates an intermediate representation (like a key-value cache). It's often compute intensive. Decode: this stage generates the output tokens, one by one, using the prefill representation. It is typically memory-bandwidth bound"8 — the identical interference framing Module 3 built from DistServe and Splitwise, restated in Google's own words for the TPU case.
Because each v6e-8 slice is its own ICI domain (§1.3), the two slices in the disaggregated configuration do not share an ICI mesh with each other. The KV cache produced by the prefill slice has to leave that slice's ICI domain entirely and reach the decode slice over the external network connecting the two VM groups — the data-center network (DCN), not ICI.
5.2 Quantifying the DCN cliff
This is where the "ICI dissolves the interconnect-cliff argument" intuition breaks down, and it is worth doing the arithmetic explicitly rather than asserting it. TPU7x's own spec sheet gives both numbers on the same per-chip basis: 1,200 GB/s of bidirectional ICI bandwidth, and 100 Gb/s of DCN bandwidth, per chip4.
Converting to matching units: 100 Gb/s ÷ 8 bits/byte = 12.5 GB/s. The ratio of ICI to DCN bandwidth per chip is then
$$ \frac{1{,}200\ \text{GB/s}}{12.5\ \text{GB/s}} = 96\times $$Ninety-six times is a bigger proportional cliff than Module 1's NVLink-to-InfiniBand drop (18–36×, Module 1 §2.3, per-GPU/per-chip single-rail comparison). The GPU-side story and the TPU-side story turn out to have the same shape — a large per-chip bandwidth drop the instant traffic leaves the local interconnect domain — with different constants and a different name for the fabric on the far side of the cliff (DCN, not InfiniBand/RoCE). Disaggregation on TPU therefore pays the same kind of transport tax GPU disaggregation does; it is not free just because the intra-slice fabric happens to be a mesh instead of a switch.
5.3 Where the open-source stack stands: DCN-based P/D disaggregation
The vLLM TPU blog, current as of October 2025, listed single-host prefill/decode disaggregation as experimental and multi-host dynamic disaggregation as a "What's Next" roadmap item, not yet shipped11. The live tpu-inference support-matrix snapshot cited here (2026-08-27) shows movement since: "DCN-based P/D disaggregation" is listed as Passing across both the Flax and Torchax backends12 — the feature's own name confirms the §5.2 mechanism, that the transfer path is explicitly over DCN rather than ICI. Read this alongside a second row in the same table: a separately-tracked "multi-host" feature flag is listed as Untested across all backends12 in that same snapshot. The honest reading is that DCN-based KV transfer between slices is validated, while the broader "multi-host" capability flag in the same matrix has not been independently re-verified as of that date — two different claims that should not be collapsed into one "multi-host PD works" headline.
5.4 Cross-reference: same why, different transport
Nothing about the §5.1 motivation differs from Module 3's argument: separate the compute-bound and memory-bandwidth-bound phases so neither stalls the other, and size each pool independently. What differs is entirely at the mechanism level Module 4 covers for GPU: instead of a connector plugging into a running engine process to pull KV blocks peer-to-peer (vLLM's NixlConnector/MooncakeConnector pattern, Module 4 §2), the TPU path as documented here separates prefill and decode at the level of entire Pathways slices, coordinated by the same single-controller/LeaderWorkerSet machinery from §3, with the transfer itself crossing DCN. No source examined for this module describes a TPU-side connector abstraction analogous to vLLM's pluggable Connector/LookupBuffer/Pipe interface (Module 4, Module 11) that lets a single running engine process attach to multiple named backends; the granularity of disaggregation documented here is the slice, not the connector.
6. Expert parallelism on TPU: the sharding primitive exists, the DeepEP analog does not
Module 5 and Module 6 covered wide expert parallelism on GPU in detail: DeepEP's two kernel modes, EPLB's redundant-expert placement, and DeepSeek's own EP320/EP144 production configurations, all built on a specialized all-to-all dispatch/combine kernel library tuned to NVLink-plus-RDMA. The natural question for this module is whether TPU has an equivalent. The honest answer from the sources examined here is: partially, and not yet at the same layer.
6.1 SparseCores are not a MoE-dispatch engine
TPU chips carry a component with a name that invites the wrong guess. SparseCores sound like they might be TPU's dedicated mixture-of-experts hardware, and TPU7x has four of them per chip, v6e has two31. Google's own architecture documentation is explicit about what they actually do: "SparseCores are dataflow processors that accelerate models using sparse operations. A primary use case is accelerating recommendation models, which rely heavily on embeddings"3. They are purpose-built for the irregular-memory-access pattern of large embedding-table lookups — the recommendation-system workload class — not for routing tokens to experts and running an all-to-all dispatch/combine across chips the way DeepEP does for DeepSeek-class MoE inference. No source examined for this module describes SparseCores being used for LLM MoE expert routing.
6.2 What the open-source stack actually ships: EP as a generic parallelism dimension
tpu-inference's own parallelism-technique support matrix (snapshot 2026-08-27) lists Expert Parallelism (EP) alongside pipeline, data, and tensor parallelism as a supported sharding dimension: Passing at single-host for both the Flax and Torchax backends, Untested at multi-host for both12. The same README's quantization-support tables list "Fused MoE" and "gmm" (grouped matrix multiply, the standard MoE-batched-matmul primitive) kernels as Untested across every listed weight/activation precision combination in the same snapshot12. Taken together: MoE models can be sharded across chips using EP as a parallelism strategy within a single host today, but there is no documented, validated all-to-all dispatch/combine kernel library at DeepSeek/DeepEP scale — cross-node, redundant-expert-aware, load-balanced — in the sources surveyed for this module. EPLB's redundant-expert placement algorithm and its hierarchical-vs-global policy split (Module 5 §3–4) have no documented TPU-side counterpart here.
6.3 SGL-JAX and SGL-torchtpu: a bridge, not a native mechanism
The clearest evidence that wide-EP-on-TPU is still being built, rather than already shipped, is where it appears: as a forthcoming feature of a GPU-ecosystem project being ported onto TPU, not as a Google-native capability. SGLang already runs on current TPU generations via SGL-JAX, with support for Gemma, Qwen, DeepSeek, GLM, Mimo, Kimi, Ling, MiniMax, and Grok, plus the Wan and Flux diffusion models13. RadixArk's announced follow-on, SGL-torchtpu, is a PyTorch-native TPU backend with eager execution and MPMD (multiple-program multiple-data) support, and its feature list explicitly includes "parallelism (data, tensor, expert, context, and pipeline)"13 — expert parallelism named as a roadmap item for a bridge project, not as a description of an existing Google-native TPU mechanism.
6.4 Summary: no DeepEP analog, cross-referenced
Put plainly, against Module 5 and Module 6's terms: TPU has EP as a sharding dimension12, and TPU has SGLang's own wide-EP machinery arriving via SGL-JAX/SGL-torchtpu as those ports mature13 — which will, once landed, presumably carry DeepEP-equivalent all-to-all kernels onto TPU through SGLang's own code paths rather than a TPU-native equivalent. What TPU does not have, in any source examined for this module, is a native, Google-published, DeepEP-equivalent all-to-all dispatch/combine kernel library, an EPLB-equivalent redundant-expert placement algorithm, or a published DeepSeek-scale (EP144/EP320-class) production deployment on TPU. This is a documented absence as of 2026-09-17, not a permanent architectural one — the same "still consolidating" caveat from §4.3 applies.
7. Operational realities the GPU-stack modules don't surface
Two facts from Google's own Pathways documentation don't map onto anything Modules 1–16 needed to cover explicitly, because they are specific to how Pathways-on-Cloud is currently packaged and accessed rather than to any GPU-vs-TPU hardware difference.
7.1 Checkpoint restoration time as a first-class number
Google's multi-host inference walkthrough reports concrete checkpoint-restoration times for its two worked examples: approximately 7 minutes to restore a Llama 3.1 405B (int8) checkpoint onto a v6e-16 slice, and approximately 2 minutes for a Llama 2 70B (int8) checkpoint onto two v6e-8 slices8.
No comparable published number appears in Modules 1–16 for GPU-side checkpoint loading, because none of those modules' sources reported one — but the mechanism it stands in for is not new: it is the same weight-loading step Module 9 covers under ModelExpress (Dynamo's GPU-to-GPU weight streaming over NIXL/NVLink). What is distinctive here is that Google states the number directly, as a planning input, rather than leaving it implicit. For fleet operations — the rolling-upgrade and drain-before-remove patterns Module 14 §7 covers for GPU pods — a multi-minute checkpoint-restore step per slice is a real constraint on how fast you can cycle a slice through an upgrade, and it scales with both model size and the number of hosts a restore has to fan out across.
7.2 Access model: not yet fully self-serve
Pathways on Cloud is explicitly not a fully self-serve GA product as of this writing: "to get access to Pathways, you need to contact your Google Cloud Account representative"8. This is a meaningful contrast with the rest of this course's stack, none of which gates access behind an account-representative conversation: vLLM, SGLang, TensorRT-LLM, Dynamo, and llm-d are all installable directly from their public repositories on any cluster with the right accelerators, and tpu-inference itself is a public, pip-installable, actively-developed open-source project with a public issue tracker and a live support matrix12. The practical decision this creates — Pathways' access-gated, Google-operated path versus the self-serve tpu-inference/SGL-JAX path — is substantial enough to be one of the two positions in the next section.
8. Mapping the course onto TPU: what ports, what substitutes, what's absent
Collecting §1–7 and the cross-references to Modules 1–16 into one table, the same exercise Module 17 runs for AMD's Instinct platform.
| GPU-stack mechanism | TPU-side status | Detail |
|---|---|---|
| NVLink/NVSwitch (Mod. 1) | Structural substitute | ICI mesh/torus + OCS between cubes; direct point-to-point, not switched (§1) |
| InfiniBand/RoCE (Mod. 1) | Structural substitute | Data-center network (DCN); a comparable or larger per-chip bandwidth cliff (§5.2) |
| Dynamo / llm-d orchestration (Mod. 9–10) | Structural substitute | Pathways: single-controller dataflow graph, not an external scheduler over autonomous engines (§3) |
| LeaderWorkerSet (Mod. 14) | 1:1 port | Pathways on Cloud deploys over the same LWS API (§3.2) |
| vLLM / SGLang / TensorRT-LLM (Mod. 11–13) | Partial port, consolidating | JetStream (legacy) folding into vLLM's tpu-inference; SGLang via SGL-JAX/SGL-torchtpu (§4, §6.3) |
| PD disaggregation / KV connectors (Mod. 3–4) | Structural substitute | Whole-slice separation over DCN, not a per-process connector plugin; same interference motivation (§5) |
| Wide-EP, DeepEP all-to-all, EPLB (Mod. 5–6) | No documented equivalent | EP exists as a sharding dimension only; no native all-to-all kernel library or redundant-expert placement found (§6) |
| KV-cache-aware / prefix-cache routing (Mod. 2) | Not evidenced | No TPU-side component analogous to Dynamo's Router or llm-d's prefix-cache scorer documented in sources examined (§3.3) |
"Not evidenced" and "no documented equivalent" describe the sources surveyed for this module, not a claim that no such component exists anywhere at Google — see §4.3 and the disagreement below on how much weight to put on documentation gaps for a fast-moving, partly access-gated stack.
Where experts disagree
The disagreement on TPU serving is less "which mechanism is better" than "which of two paths onto TPU should you bet a production deployment on" — Google's own Pathways-plus-JetStream-lineage stack, or the open-source vLLM/SGLang ports that carry this course's GPU-native tooling onto TPU. The two paths are not mutually exclusive today, and neither position disputes the hardware facts in §1–2; they disagree about which software layer to depend on.
Bet on Google's own Pathways/JetStream-lineage stack
Google's own positioning, most directly in its inference-updates blog10
Google describes JetStream as "based on the same inference stack used to serve Gemini models"10 — a direct claim that this is the production-hardened path, co-designed with the hardware (Ironwood's own architecture blog frames prefill/decode optimization as a chip-design decision5) and operated at Google's own scale before anyone outside Google touches it. Pathways' single-controller model is also the one substantiated by a peer-reviewed systems paper with concrete utilization numbers7, not just a vendor blog claim.
- Co-designed with the chip; Google has operated it at Gemini scale before external release10
- Single-controller model has a published, peer-reviewed performance case (near-100% utilization at 2,048 TPUs)7
- Access-gated: requires contacting a Google Cloud account representative, not self-serve8
- Does not carry this course's GPU-side tooling (KV connectors, EPP-style routers) onto TPU; you adopt a different deployment model, not a port of the one you already know
Bet on the open vLLM/tpu-inference and SGLang/SGL-JAX ports
The vLLM project's own framing of tpu-inference12 and SGLang/RadixArk's TPU announcement13
tpu-inference is public, pip-installable, and explicitly designed to "retain vLLM standardization: keep the same user experience, telemetry, and interface"12 across GPU and TPU — meaning the router, connector, and observability concepts from Modules 2, 4, and 11 carry over largely unchanged, even though the underlying kernels and parallelism plumbing are TPU-specific. SGLang's SGL-JAX bridge makes the same bet from the other open-source engine13. Neither requires an account-representative conversation to try.
- Self-serve, public, actively developed with a live support matrix12
- Preserves the operational knowledge (routing, connectors, observability) this course already built for the GPU stack
- Multi-host P/D and multi-host EP are marked Untested in the same live matrix as of 2026-08-2712 — less production mileage than Google's internal stack claims
- No published DeepSeek-scale wide-EP reproduction on TPU exists yet in the sources surveyed, unlike the GPU-side SGLang reproduction Module 6 covers
For a team already inside Google Cloud's enterprise relationship and running Gemini-adjacent workloads, Google's own Pathways stack is the lower-risk default — it is what Google itself reports running in production, and the access gate is a business-process cost, not a technical one. For a team that already operates the vLLM/SGLang stack this course spends fourteen modules on and wants one operational model across GPU and TPU fleets, tpu-inference and SGL-JAX are the more coherent bet, at the cost of adopting features (multi-host EP, at minimum) that are demonstrably less mature by the projects' own support matrices12. What would settle it: a published, reproducible, DeepSeek-or-larger-class wide-EP benchmark on TPU from either path, analogous to the GPU-side SGLang reproduction in Module 6 — none exists in the sources surveyed for this module as of 2026-09-17.
What is changing
Three things in this module are likely to be stale within a year, in order of how fast they are moving. Fastest: the tpu-inference feature-support matrix cited in §5.3 and §6.2 is a live, auto-updated dashboard, not a fixed release note12 — the specific Passing/Untested statuses quoted here are a snapshot dated 2026-08-27 and should be re-checked directly before any deployment decision. Next fastest: the JetStream-to-tpu-inference consolidation itself (§4.3) is seven months old as of this writing and the ecosystem has not finished settling — whether Pathways' own jax-tpu engine eventually runs on tpu-inference internals, stays on MaxText/JetStream permanently, or the two remain parallel paths is not resolved in any source examined here. Slowest but still worth tracking: Pathways' access-gated model (§7.2) is a business-process fact about Google Cloud's current rollout, not a technical constraint, and vendors routinely move such gates to self-serve GA as a product matures.
Module 1 opened this course by asking what changes, physically and logically, the instant a second node appears. On TPU the answer has the same shape and different constants: within one ICI-connected slice, the chip-to-chip question barely changes from the single-node case — a torus instead of a switch, but still one low-latency domain you can treat as one addressable pool. The change shows up at two boundaries instead of one: the host boundary within a slice (§2.3, where TPU7x's 4-chips-per-host ratio means "node" stops meaning what it means on an 8-GPU HGX host), and the slice boundary itself, where ICI's 800–1,200 GB/s gives way to a data-center network running two orders of magnitude slower (§5.2). Sixteen modules on NVLink, InfiniBand, and RoCE turn out to have taught the right question — where is the low-latency domain's edge, and what happens when a request crosses it — even where every specific fabric, orchestrator, and serving engine on this hardware has a different name.
References
- officialGoogle. TPU v6e (Google Cloud Documentation). 2026. https://docs.cloud.google.com/tpu/docs/v6e. Accessed 2026-09-17. Used for: 256-chip pod, 800 GB/s bidirectional ICI/chip, 32 GB HBM @ 1,638 GB/s, 918 TFLOPs bf16/chip, 234.9 PFLOPs pod peak, 2D torus, the 8-chip inference-optimized v6e-8 slice, and multi-host scaling to 16×16 across up to 64 VMs.
- officialNVIDIA. H100 GPU (product page). 2026. https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-09-17. Used for: NVLink4 900 GB/s per-GPU bandwidth, the Figure 1 and Module 1 baseline for full-bisection NVSwitch comparison.
- officialGoogle. TPU architecture (system-architecture-tpu-vm, Google Cloud Documentation). 2026. https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm. Accessed 2026-09-17. Used for: Pod/slice definitions, torus wraparound framing, SparseCore counts and purpose (embeddings, not MoE dispatch), ICI resiliency/OCS fault routing, 3D topology notation (
AxBxC, multiples of four), TPU7x's 4-chips-per-host ratio. - officialGoogle. TPU7x (Google Cloud Documentation). 2026. https://docs.cloud.google.com/tpu/docs/tpu7x. Accessed 2026-09-17. Used for: 9,216-chip pod, 3D torus, 192 GB HBM @ ≈7.37 TB/s/chip, 2,307/4,614 TFLOPs BF16/FP8, 1,200 GB/s ICI/chip, 100 Gbps DCN/chip, dual-chiplet design, 4-chip VM (224 vCPUs/960 GB RAM), two TensorCores/four SparseCores per chip.
- blogGoogle Cloud. Inside the Ironwood TPU codesigned AI stack. Google Cloud Blog, 2026. https://cloud.google.com/blog/products/compute/inside-the-ironwood-tpu-codesigned-ai-stack. Accessed 2026-09-17. Used for: 3D torus within 64-chip cubes joined by Optical Circuit Switch (OCS) as "a dynamic, reconfigurable optical network," and the compute-intensive-prefill/memory-bandwidth-intensive-decode framing.
- officialNVIDIA. GB200 NVL72 (product page). 2026. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-09-17. Used for: 130 TB/s rack-wide NVLink bandwidth and 13.4 TB HBM3e @ 576 TB/s across 72 GPUs, for the Table 1 per-chip comparison.
- paperBarham, P., Chowdhery, A., Dean, J., Ghemawat, S., Hand, S., Hurt, D., Isard, M., Lim, H., Pang, R., Roy, S., Saeta, B., Schuh, P., Sepassi, R., El Shafey, L., Thekkath, C. A., Wu, Y. Pathways: Asynchronous Distributed Dataflow for ML. MLSys 2022 / arXiv:2203.12533, 2022. https://arxiv.org/abs/2203.12533. Accessed 2026-09-17. Used for: the single-controller model definition versus SPMD/multi-controller systems, gang-scheduling across accelerator islands, and the near-100% utilization at 2,048 TPUs / 16-stage-pipeline-over-two-islands performance result.
- officialGoogle. Perform multihost inference using Pathways (AI Hypercomputer Documentation). 2026. https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/multihost-inference. Accessed 2026-09-17. Used for: the four named containers (pathways-proxy, pathways-rm, jax-tpu, jetstream-http) and their ports, LeaderWorkerSet deployment shape, the aggregated (single v6e-16 slice) vs. disaggregated (two v6e-8 slices,
subGroupPolicy,ExperimentalMaxtextDisaggregatedServer_8) examples, the prefill/decode definitions, checkpoint-restoration timings (≈7 min Llama 3.1-405B, ≈2 min Llama 2-70B), and the account-representative access requirement. - repoGoogle (AI-Hypercomputer). JetStream (README, main branch). 2026. https://github.com/AI-Hypercomputer/JetStream/blob/main/README.md. Accessed 2026-09-17. Used for: JetStream's self-description as a throughput/memory-optimized XLA inference engine, its JAX/MaxText and PyTorch reference engine implementations, the MaxText engine's ICI-mapped sharding flags (
ici_fsdp_parallelism,ici_tensor_parallelism,ici_autoregressive_parallelism) andmaxengine_serverlaunch command (docs/online-inference-with-maxtext-engine.md, same repository), and the archival notice (repository archived February 1, 2026, functionality migrated totpu-inference). - blogGoogle Cloud. AI Hypercomputer inference updates for Google Cloud TPU and GPU. Google Cloud Blog, 2025. https://cloud.google.com/blog/products/compute/ai-hypercomputer-inference-updates-for-google-cloud-tpu-and-gpu. Accessed 2026-09-17. Used for: the April 2025 Pathways-in-JetStream integration; multi-host Llama 3.1 405B at 1,703 tok/s / 3× inference-per-dollar vs. v5e; disaggregated-serving Llama2-70B (7× TTFT, ≈3× TPOT) gains; the "same inference stack used to serve Gemini models" claim; and vLLM-on-TPU's initial framing as an alternative to JetStream.
- blogvLLM Team. vLLM TPU: A New Unified Backend Supporting PyTorch and JAX on TPU. vLLM Blog, 2025-10-16. https://blog.vllm.ai/2025/10/16/vllm-tpu.html. Accessed 2026-09-17. Used for: the Torchax PyTorch-to-JAX bridge and single JAX→XLA lowering path, the ≈20% throughput gain over direct PyTorch/XLA, the 3.6×/2.1× gains over the February-2025 prototype, and the single-host-experimental/multihost-roadmap status of P/D disaggregation as of that date.
- repovLLM Project. tpu-inference (README, incl. live feature-support matrices). 2026. https://github.com/vllm-project/tpu-inference. Accessed 2026-09-17 (support-matrix snapshot dated 2026-08-27 in-repo). Used for: the "unifying JAX and PyTorch under a single lowering path" and "retain vLLM standardization" self-description; recommended (v7x/v5e/v6e) vs. experimental (v3/v4/v5p) TPU generations; the Core Features table's "DCN-based P/D disaggregation" (Passing) and "multi-host" (Untested) rows; the Parallelism Techniques table's EP row (single-host Passing, multi-host Untested); and the Fused-MoE/gmm quantization rows (Untested).
- blogLMSYS Org / RadixArk. RadixArk Joins Forces with Google to Bring Full SGLang Features to TPUs. LMSYS Blog, 2026-07-30. https://www.lmsys.org/blog/2026-07-30-sglang-google-tpu/. Accessed 2026-09-17. Used for: SGL-JAX's current TPU model support (Gemma, Qwen, DeepSeek, GLM, Mimo, Kimi, Ling, MiniMax, Grok, Wan, Flux), and the planned SGL-torchtpu backend's roadmap including expert parallelism, Radix Cache, HiCache, quantization, and speculative decoding.