AI Inference Hardware · Page 3
Google TPU v5e/v5p/v6e/v7: Pod Architecture from Trillium to Ironwood
Google's fifth-through-seventh-generation Tensor Processing Units are built from a different primitive than a GPU: a systolic-array matrix unit instead of SIMT tensor cores, and a torus fabric instead of a switched point-to-point network. This page covers the TensorCore/MXU/SparseCore building block, the per-chip compute/memory/interconnect specs across v5e, v5p, v6e (Trillium), and v7 (Ironwood), how 2D and 3D torus topologies scale to pods of up to 9,216 chips, and why Ironwood is the first TPU generation Google describes as designed for inference rather than adapted to it.
1. The TensorCore, the MXU, and the SparseCore
A TPU chip contains one or more TensorCores. Each TensorCore consists of one or more MXUs (matrix multiply units), a vector unit, and a scalar unit1. The MXU does essentially all of the FLOPs; the vector unit runs elementwise ops like activations and softmax, and the scalar unit handles control flow and address generation1. This is the TPU equivalent of a GPU SM/CU housing tensor or matrix cores, but the internal mechanism is different in a way that matters for how you reason about peak throughput: an MXU is not an array of independent multiply-accumulate lanes issuing per-thread instructions, it is a systolic array1.
This is the point where a TPU stops resembling a GPU at the mechanism level rather than just the spec sheet. NVIDIA's tensor cores and AMD's matrix cores (pages 1 and 2) are SIMT execution units: a warp or wavefront of threads issues a matrix-multiply instruction, the core reads operands from register files/shared memory for that instruction, and the next instruction can be something else entirely — control flow, a different op, a different tile shape. An MXU issues nothing per cycle in that sense; once a matmul is mapped onto the grid, every PE in the array is doing the same multiply-accumulate every cycle until the operation drains, with no per-cycle instruction fetch inside the array at all. That is the source of a TPU's efficiency on large, regular matmuls (attention QK^T and score-value products, feed-forward projections) and the source of its dependence on XLA to statically tile and schedule work onto the array ahead of time rather than dispatching it dynamically the way a CUDA or ROCm kernel would.
1.1 The MXU as a systolic array
Google's own architecture documentation is explicit: "An MXU is composed of either 256 x 256 (TPU v6e and TPU7x) or 128 x 128 (TPU versions prior to v6e) multiply-accumulators in a systolic array. As each multiplication is executed, the result is passed to the next multiply-accumulator. The output is the summation of all multiplication results between the data and parameters."1 Concretely: weights are loaded into the grid once and held stationary in each PE (processing element); activations stream in from one edge, partial sums propagate diagonally across the grid from PE to PE, and results drain out the far edge already fully accumulated. No PE ever fetches an operand from a register file or a cache line mid-computation — the only traffic in or out of the array is the activation stream in and the result stream out. That is what makes it power- and area-efficient for dense matmul and also what makes it a poor fit for anything irregular: a systolic array has no mechanism for a conditional branch, a gather, or a variable-length row, which is exactly the gap the SparseCore exists to fill (§1.2).
The size of the grid sets the array's per-cycle throughput directly, because every one of its PEs computes one multiply-accumulate (MAC) per cycle once the pipeline is full. A 128x128 array — the MXU used in every generation prior to v6e, i.e. v5e and v5p in this page's scope — has 128 x 128 = 16,384 PEs, and Google's documentation independently states "Each MXU is capable of performing 16K multiply-accumulate operations per cycle"1: 16,384 rounds to "16K" exactly, so the two statements corroborate each other for the 128x128 generation.
- Load weights. One operand matrix is loaded into the grid once and held stationary in every PE for the duration of the matmul — no weight re-fetch happens for the rest of the operation.
- Stream activations in. The other operand streams in from one edge, one column (or row) per cycle, so the array never needs the whole activation matrix resident at once.
- Propagate partial sums. Each PE multiplies its stationary weight by the incoming activation, adds the partial sum arriving from its neighbor, and passes the new partial sum onward — this is the "result is passed to the next multiply-accumulator" step Google's documentation describes1.
- Drain results. Once a partial sum has crossed every PE on its path, it emerges from the far edge already fully accumulated — no separate reduction pass is needed after the array.
Worked derivation: implied MXU clock for v5e and v5p, and where the same arithmetic breaks for v6e
v5e has one TensorCore per chip with four 128x128 MXUs2, and a stated peak of 197 TFLOPs bf16 per chip2.
$$ \text{MACs/cycle per MXU} = 128 \times 128 = 16{,}384 $$ $$ \text{FLOPs/cycle per chip} = 16{,}384 \times 2 \times 4\ \text{MXUs} = 131{,}072 $$ $$ \text{implied clock} = \frac{197 \times 10^{12}}{131{,}072} \approx 1.503\ \text{GHz} $$v5p has two TensorCores per chip, each with four 128x128 MXUs — eight MXUs total3 — and a stated peak of 459 TFLOPs bf16 per chip3.
$$ \text{FLOPs/cycle per chip} = 16{,}384 \times 2 \times 8\ \text{MXUs} = 262{,}144 $$ $$ \text{implied clock} = \frac{459 \times 10^{12}}{262{,}144} \approx 1.751\ \text{GHz} $$Both implied clocks land in the same 1.5-1.75 GHz band, which is a plausible operating point for a large systolic array on a datacenter accelerator process and is internally consistent across two different chips of the same MXU generation — a genuine cross-check, not a coincidence built into the formula.
Apply the identical formula to v6e (Trillium): one TensorCore, two 256x256 MXUs4, 918 TFLOPs bf16 per chip4.
$$ \text{MACs/cycle per MXU} = 256 \times 256 = 65{,}536 $$ $$ \text{FLOPs/cycle per chip} = 65{,}536 \times 2 \times 2\ \text{MXUs} = 262{,}144 $$ $$ \text{implied clock} = \frac{918 \times 10^{12}}{262{,}144} \approx 3.502\ \text{GHz} $$3.5 GHz is not a plausible clock for a chip this size relative to the 1.5-1.75 GHz range the same arithmetic gives v5e and v5p. The likely explanation is that the "16K multiply-accumulate operations per cycle" line in Google's general architecture documentation is legacy text scoped to the 128x128-MXU generation and was not updated when the 256x256 MXU shipped with v6e1 — the document states both array sizes in the same sentence and then gives a single "16K" throughput figure that only reconciles with the smaller one. This is a case where a vendor's own reference page is internally inconsistent rather than a case where the underlying hardware is unusual; treat the 256x256-generation peak-FLOPs figures (v6e, Ironwood) as authoritative and the "16K/cycle" line as describing only v5e/v5p-class MXUs.
MXU peak FLOPS from array geometry
Formula: \( \text{side}^2 \times \text{MXUs} \times 2 \times \text{clock} \). Set side=128, MXUs=4, clock≈1.5 GHz to reproduce v5e's stated 197 TFLOPs2; set side=256, MXUs=2 and raise the clock to see what it would take for v6e's stated 918 TFLOPs to reconcile with a 256x256 array at a plausible clock (it does not, at any clock under ~2 GHz — see the derivation above).
1.2 SparseCore: a second, different core for embeddings
A SparseCore is not a smaller or cut-down MXU. Google describes it as a dataflow processor: "SparseCores are dataflow processors that accelerate models using sparse operations. A primary use case is accelerating recommendation models, which rely heavily on embeddings."1 Where the MXU's systolic grid is efficient precisely because every PE does the same dense multiply-accumulate every cycle, embedding lookups are the opposite workload: a sparse table gather indexed by categorical IDs, followed by a scatter-add to combine looked-up rows, with no fixed access pattern the compiler can pipeline into a grid. Routing that traffic through a systolic array would stall it; the SparseCore exists as a separate execution unit specifically so the MXU is never blocked waiting on irregular memory access.
SparseCore count per chip changes across generations independently of TensorCore/MXU count: v5e ships four SparseCores per chip, v5p and TPU7x (Ironwood) ship four per chip, and v6e ships two per chip135. v6e was also the generation that introduced SparseCore as a named, documented feature at all4 — v5e and v5p already had the hardware, but Trillium's release is where Google began marketing it as a differentiator for embedding-heavy recommendation and ranking models. The XLA compiler decides how to use SparseCores; developers do not hand-schedule them directly, though XLA flags exist to tune the behavior1.
2. Four generations, one chip table
v5e, v5p, v6e, and Ironwood are not a strict speed progression — v5e and v5p shipped in the same generation with deliberately different design points, and v6e and Ironwood each replace one of them. Table 1 puts all four on one grid; the sections below explain what each chip is actually for.
| Spec | v5e | v5p | v6e (Trillium) | Ironwood (TPU7x) |
|---|---|---|---|---|
| Peak compute, BF16 (TFLOPs/chip) | 197 | 459 | 918 | 2,307 |
| Peak compute, FP8 (TFLOPs/chip) | — | 459 | — | 4,614 |
| Peak compute, Int8 (TOPs/chip) | 393 | — | 1,836 | — |
| HBM capacity (per chip) | 16 GB | 95 GiB | 32 GB | 192 GiB |
| HBM bandwidth (per chip) | 800 GiBps | 2,765 GBps | 1,638 GBps | 7,380 GBps |
| TensorCores / chip | 1 | 2 | 1 | 2 (1 per chiplet) |
| MXUs / TensorCore (array size) | 4 (128x128) | 4 (128x128) | 2 (256x256) | 2 (256x256) |
| SparseCores / chip | 4 | 4 | 2 | 4 (2 per chiplet) |
| ICI bandwidth (bidirectional, per chip) | 400 GBps | 1,200 GBps | 800 GBps | 1,200 GBps |
| ICI ports / chip | 4 | 6 | 4 | 6 |
| Interconnect topology | 2D torus | 3D torus | 2D torus | 3D torus |
| Chips per pod (max) | 256 | 8,960 | 256 | 9,216 |
| DCN bandwidth (per chip) | — | 50 Gbps | 100 Gbps | 100 Gbps |
| vCPUs (per 4-chip VM) | —† | 208 | 180 | 224 |
| RAM (per 4-chip VM) | —† | 448 GB | 720 GB | 960 GB |
† v5e's documentation reports host configuration as 8 chips/host with 512 GiB DRAM/host rather than a 4-chip-VM figure, so it is not directly comparable to the other three columns' 4-chip-VM rows2.
2.1 v5e: the cost/inference-optimized chip
v5e ships one TensorCore per chip with four 128x128 MXUs, 16 GB of HBM at 800 GiBps, and four ICI ports wired into a 2D torus2. Google positions it explicitly as a joint training-and-serving product with the caveat that the two jobs want opposite things from the scheduler: "Training jobs are optimized for throughput and availability, while serving jobs are optimized for latency. A training job on TPUs provisioned for serving could have lower availability and similarly, a serving job executed on TPUs provisioned for training could have higher latency."2 A full v5e pod tops out at 256 chips arranged as a 16x16 2D torus, giving a pod-level peak of 50.63 PFLOPs bf16, 51.2 TB/s all-reduce bandwidth, and 100 PetaOps Int82.
16 GB of HBM per chip is small next to v5p's 95 GiB or Ironwood's 192 GiB — v5e is built to be cheap per chip and scaled out, not to hold the largest possible model on a single device. Serving a large model on v5e means either sharding across chips or accepting the memory ceiling; this is the same shape of constraint page 6 works through quantitatively for GPU HBM capacity, applied here to the smallest-memory TPU in the range.
v5e VMs bundle up to eight chips per host, and the CPU-to-TPU path is non-uniform at the largest single-host size: "In 4-chip and smaller slices, all TPU chips share the same non-uniform memory access (NUMA) node. For 8-chip v5e TPU VMs, CPU-TPU communication will be more efficient within NUMA partitions."2 Each host carries 512 GiB of DRAM2 — this is host-side system memory feeding the chips over PCIe, separate from the 16 GB of on-chip HBM per chip, and it is the pool used for activation/optimizer-state offload the same way Ironwood's host memory tier works (§2.5).
2.2 v5p: the training/performance chip
v5p doubles the TensorCore count to two per chip (still four 128x128 MXUs each, eight total), roughly triples HBM to 95 GiB at 2,765 GBps, and moves to a six-port 3D torus at 1,200 GBps bidirectional ICI per chip3. The headline number is pod scale: a full v5p pod is 8,960 chips, and Google states the topology directly rather than leaving it implicit — "A TPU v5p Pod is composed of 8960 chips interconnected with reconfigurable high-speed links," with the largest documented single-slice shape at 16x16x24 (6,144 chips, 96 cubes) and Multislice extending a single training job to 18,432 chips by stitching multiple slices together over the data-center network3. A full pod comes out to 17,920 TensorCores across 8,960 chips, 2,240 hosts/VMs, and 140 cubes3.
One spec-table detail worth flagging on its own: v5p's FP8 peak (459 TFLOPs) equals its BF16 peak (459 TFLOPs) exactly3 — unlike v5e, v6e, and Ironwood, which all roughly double their throughput at FP8 or Int8 relative to bf16, v5p gets no speedup from dropping to 8-bit. v5p's MXU generation does not have an FP8-accelerated datapath the later three chips do; the 459/459 pairing in the spec table is not a rounding coincidence, it is the same number reported twice because there is only one number.
2.3 v6e (Trillium): the generational jump
v6e roughly doubles v5e's peak compute (918 vs. 197 TFLOPs bf16 — a 4.7x jump, not 2x, because v6e also moves to the larger 256x256 MXU) while cutting MXU count per TensorCore from four to two4. HBM capacity doubles to 32 GB and bandwidth roughly doubles to 1,638 GBps4; ICI bandwidth doubles from 400 to 800 GBps bidirectional, still over a 2D torus with four ports and a 256-chip pod ceiling4. Pod-level all-reduce bandwidth reaches 102.4 TB/s and bisection bandwidth 3.2 TB/s — both roughly double v5e's 51.2 TB/s and 1.6 TB/s24. v6e is also the generation where SparseCore is called out as a named "special feature" in Google's own spec table, even though the underlying hardware exists on v5e and v5p too4. Host-side provisioning grew alongside the chip: each v6e host carries 1,536 GiB of DRAM — three times v5e's 512 GiB — and four 200 Gbps NICs per host versus v5e's two 100 Gbps NICs, doubling both DCN bandwidth per pod (25.6 vs. 6.4 Tbps) and giving each host substantially more headroom for data loading and checkpoint I/O without touching HBM24.
2.4 Ironwood (TPU7x): breaking from MegaCore
Every TPU generation up to and including v5p exposed a chip as a single "MegaCore" logical device — a unified memory space and a single programming target even where the chip physically had multiple TensorCores5. Ironwood ends that: "The Ironwood programming model lets you access two TPU chiplets instead of a single logical core (also known as MegaCore) architecture used in previous generations (TPU v4 and v5p) ... Ironwood TPUs are composed of two distinct chiplets, each with its own dedicated memory space. This is a departure from the unified memory space of the MegaCore architecture."5 Each chiplet is a complete, self-contained unit: one TensorCore, two SparseCores, and 96 GB of HBM5 — meaning the chip-level totals in Table 1 (192 GiB HBM, two TensorCores, four SparseCores) are exactly two chiplets' worth, not a monolithic design that happens to have those numbers.
Google states the motivation for the split plainly, and it is not framed as a performance feature: "This change improves the cost-effectiveness and efficiency of manufacturing the chip."5 This is a yield/reticle-limit argument — the same category of reasoning behind chiplet designs on the GPU side (pages 1 and 2 cover NVIDIA Blackwell's dual-die package and AMD's XCD chiplets respectively) — not a claim that splitting the chip made it faster per se. The two chiplets are connected by a die-to-die (D2D) interface Google describes as "six times faster than a 1D inter-chip interconnect (ICI) link"5; Ironwood's 3D torus carries 200 GBps bidirectional per axis5, and if that per-axis rate is the "1D ICI link" baseline the multiplier refers to, the D2D link runs in the neighborhood of 1,200 GBps — comparable to the entire chip's external ICI bandwidth, packed into a link that only has to reach the other half of the same package.
Because each chiplet keeps its own memory space rather than sharing one, frameworks see the split directly: "Frameworks like JAX expose each Ironwood chip as two separate 'devices,' one for each chiplet ... JAX adds a fourth dimension to the topology specification to distinguish between chiplets."5 Google is explicit that this is a deliberate concession to compatibility, not an oversight: "While this represents an architectural shift, the new design ensures that you can reuse existing software models with minimal changes ... The programming model for Ironwood is similar to that of TPU generations earlier than v4, such as TPU v3."5 Section 6 covers why the chip is organized this way from the inference-workload side, not just the manufacturing side.
2.5 TPU7x memory hierarchy: HBM, VMEM, and host offload
Ironwood's own documentation describes three memory tiers, not one, and is candid that the largest tier is not automatically the answer to a bandwidth problem: "Each chip is equipped with 192 GB of HBM, with bandwidth of approximately 7.37 TB/s. The large HBM capacity enables large batch sizes, which can improve throughput. However, despite its size, HBM can still be a bottleneck, particularly for memory-bound vector operations or inefficient data access patterns."5 Underneath HBM sits VMEM, on-chip SRAM feeding the MXU directly: "VMEM is a smaller, on-chip SRAM ... with significantly higher bandwidth to the Matrix Multiply Unit (MXU) than HBM. This memory acts as a high-speed scratchpad for custom kernels. The size of this buffer is a tunable parameter."5 This is the tier a hand-written Pallas kernel has to fit its working set into — block sizes for custom kernels are constrained by available VMEM, not by HBM capacity5.
Above HBM, host DRAM is a fourth, off-chip tier reachable only over PCIe: "Each set of four TPU chips is connected to a CPU host using a PCIe network. While this connection has much lower bandwidth than HBM, the host's main memory can be used for offloading activations or optimizer states to free up HBM, a technique particularly useful for managing memory pressure in large models."5 The ordering (VMEM fastest and smallest, feeding the MXU directly; HBM large and fast but chip-local; host DRAM largest and slowest, PCIe-attached) is the same shape of hierarchy a GPU exposes as registers/shared-memory, HBM, and pinned host memory — the terminology differs (VMEM instead of shared memory/L2, no cache hierarchy in the CPU sense) but the offload strategy for managing a model too large for HBM alone is the same idea applied to a different set of names.
3. ICI and torus topology: 2D vs 3D
3.1 What wraparound links buy you
A torus is a mesh with the edges wrapped around: every chip has the same number of neighbors, including the ones at the physical boundary of the grid, because the last chip in a row is wired directly back to the first. A plain 2D mesh gives interior chips four neighbors and edge/corner chips fewer, which makes the network non-uniform and creates congestion at the edges under all-reduce-style collectives. Wrapping it into a torus makes every chip's neighbor count identical, which is why Google specifies "ICI ports per chip" as a single number per generation (four for v5e and v6e's 2D torus, six for v5p and Ironwood's 3D torus2345) rather than a range.
3D adds a third pair of wraparound links on top of the 2D grid's two pairs, at the cost of needing six ICI ports per chip instead of four and a more complex physical cabling topology. Google's v5p documentation notes the wraparound property is not automatic at every slice size: "All 4x4x4 and larger slices (one cube) have full 3D torus connectivity. Slices smaller than a full cube are 3D connected, however, they don't have wrap-around links that make them a 3D torus."3 A "cube" — a 4x4x4 block of 64 chips — is the smallest unit that gets the full torus property; below that, small slices carve out of a pod are 3D-connected (each axis routed) but not 3D-toroidal (no wraparound), which matters for collective-communication bandwidth on small allocations.
3.2 v5e/v6e (2D) vs. v5p/Ironwood (3D)
| Spec | v5e | v5p | v6e | Ironwood |
|---|---|---|---|---|
| Topology | 2D torus | 3D torus | 2D torus | 3D torus |
| Cube unit | — | 4x4x4 = 64 | — | 4x4x4 = 64 |
| Max single-slice size | 256 chips | 6,144 chips (96 cubes) | 256 chips | 9,216 chips (144 cubes) |
| Max Multislice size | — | 18,432 chips | — | —* |
| Bisection bandwidth / pod | 1.6 TB/s | — | 3.2 TB/s | — |
| All-reduce bandwidth / pod | 51.2 TB/s | — | 102.4 TB/s | — |
| DCN bandwidth / pod | 6.4 Tbps | — | 25.6 Tbps | — |
* Ironwood's documented maximum single-VM topology is 8x16x16 (2,048 chips)5; the full 9,216-chip superpod is reached through the optical fabric described in §5, not through the same Multislice-over-DCN mechanism v5p uses. Cells marked "—" are not stated in the cited documentation for that generation.
3.3 Twisted torus: trading symmetry for bisection bandwidth
A plain 3D torus slice is only symmetric — every chip sees the same neighbor pattern — when every dimension is equal. An asymmetric shape like 4x4x8 is 3D-toroidal but not uniform: chips along the long axis have a different wraparound distance than chips along the short axes, which creates load imbalance under collective communication. v5p's documentation offers a fix at the wiring level rather than the software level: "For some 3D slice shapes, you can use a twisted torus topology ... The main benefit of twisted topologies is that they change an asymmetric torus topology (for example, 4x4x8) into a symmetric one."3 The mechanism is a shifted wraparound rather than a straight one: "The difference between this twisted diagram and the untwisted one lies in the Y wrap-around edges. Instead of connecting to another TPU with the same X coordinate, these wrap-around edges shift to connect to the TPU at coordinate X+2 mod 4."3 Twisting is supported wherever "each dimension is either equal to or twice the size of the smallest dimension"3 — it is a property of specific slice shapes, not something available everywhere, and specifying it requires the Cloud TPU API directly rather than the default provisioning path3.
The payoff is measured, not just architecturally plausible: "a 4x4x8 twisted topology provides a 70% theoretical increase in bisection bandwidth compared to a non-twisted 4x4x8 slice"3, and a 4x8x8 twisted shape gets roughly 40%3 — larger relative gains on the more asymmetric shape, consistent with the mechanism (there is more imbalance to fix on 4x4x8 than on the nearer-cube 4x8x8). Measured end-to-end training impact is smaller than the bisection-bandwidth number alone would suggest: "Some experiments on the FSDP MaxText model showed 1-2 percentage point improvements in model FLOPs utilization (MFU) when using a twisted topology."3 Twisted topologies are documented as available on v5p (and v4, outside this page's scope) via the Cloud TPU API3; Table 2's Ironwood row does not carry an equivalent twisted-topology option in the cited documentation.
4. Pod scale vs. what you can actually book for serving
4.1 Peak pod size is a training number, not a serving number
Every pod-size figure in Table 1 (256, 8,960, 256, 9,216) is the maximum size a training job can span. Serving a single model replica does not get the same ceiling. For v5e, Google documents a hard, much smaller list of supported serving shapes: "The following 2D slice shapes are supported for v5e: 1x1, 2x2 and 2x4"2 — 1, 4, or 8 chips — even though the same chip's training slices scale all the way to the full 256-chip pod (16x16). Multi-host inference past that small-slice ceiling is not unsupported, but it requires a different serving layer: "Multi-host inference is supported using Sax"2.
v6e's documented slice table shows the same pattern in more granular steps — 1x1, 2x2, 2x4, 4x4, 4x8, 8x8, 8x16, and 16x16, running from 1 chip up to the full 256-chip pod4 — with an operational footnote that even the 8-chip (2x4) shape has two different mappings depending on whether it is provisioned as one VM or two, and the two-VM mapping is "only supported when using the GKE API"4. The number of chips in a slice and the number of VMs/hosts backing it are not the same axis, and getting them confused is a common source of miscounted capacity: a 2x4 v6e slice is 8 chips, but it can be either 1 VM or 2 VMs depending on configuration4.
4.2 Multi-host inference and the practical ceiling
Sax is the serving layer Google's TPU documentation points to for scaling a single model past the small serving-slice shapes in §4.1. Google's own Sax-specific documentation describes the mechanism as two components: "SAX admin server and SAX model server are two essential components that run a SAX cluster. The SAX admin server monitors and coordinates all SAX model servers in a SAX cluster ... only one of the SAX admin server is active through leader election, the others are standby servers. When the active admin server fails, a standby admin server will become active. The active SAX admin server assigns model replicas and inference requests to available SAX model servers."6 Each model server runs inference via GSPMD on a single TPU VM worker, and Google is explicit about the scaling boundary: "Single-host TPU model serving requires a single SAX model server on a single-host TPU VM. Multi-host TPU model serving requires a group of SAX model servers on a multi-host TPU slice."6
v5p's own documentation frames slices in training terms almost exclusively — "The largest job that can be scheduled is a 96 cube (6144 chip) job"3 — because v5p is not the inference-oriented sibling in this lineup; the topic list Google surfaces for it is Multislice training, not serving shapes3. Ironwood's documented topology table stops at 8x16x16 (2,048 chips, 512 hosts, 32 cubes) for the largest VM-addressable configuration Google lists explicitly5, even though the full superpod reaches 9,216 chips through the optical fabric covered in §5 — the gap between "largest documented multi-host VM topology" and "largest physical pod" is itself informative: most workloads, including most inference deployments, run well inside the documented VM-topology ceiling, and the full-superpod number is a training-scale and fabric-capability figure more than a per-job serving allocation.
The practical reading for capacity planning: use the "chips per pod" row in Table 1 to reason about how much aggregate hardware Google can physically wire together for one customer, and use the slice tables (1x1 through 16x16 for v5e/v6e, the documented single- and multi-host topology tables for v5p/Ironwood) to reason about what a single job — training or serving — can actually be allocated. They are different numbers by roughly one to two orders of magnitude for the inference-oriented chips (v5e: 8-chip serving slice vs. 256-chip pod without Sax; v6e: same pattern), and much closer for the training-oriented chips (v5p, Ironwood), where the documented ceiling is itself already in the thousands of chips.
5. Superpod fabric: Optical Circuit Switching and the cube-of-cubes
Ironwood's 9,216-chip pod is not one flat 3D torus wired in copper end to end. It is built as a hierarchy: "Each chip is connected to 6 other chips in the 3D torus ... a single rack of these hosts has 64 Ironwood chips and forms a 'cube' ... a 9,216-chip system with 144 cubes."7 64 x 144 = 9,216, so the superpod is literally 144 copies of the same 4x4x4 cube unit that appears as the minimum full-torus slice in Table 2, stitched together one level up.
The stitching between cubes uses a different physical layer than the stitching within a cube. Ironwood's ICI resiliency documentation is explicit that copper handles intra-cube links and optics handle inter-cube links: ICI resiliency "helps improve fault tolerance of optical links and optical circuit switches (OCS) that connect TPUs between cubes. (ICI connections within a cube use copper links that are not impacted [by optical-link faults])."1 Optical Circuit Switching lets Google reconfigure which cubes are wired to which at the optical layer rather than the electrical layer — useful both for fault-tolerance (routing around a failed link or a failed cube without a hardware change) and, at pod-assembly time, for composing a superpod's inter-cube topology without a fixed, fully-populated copper mesh among 144 cube-sized units.
Google frames the resulting fabric as one machine, not a federation of independent training clusters: "A full 9,216-chip Ironwood superpod leverages the OCS and ICI fabric to operate as a single, massive parallel processor."7 At that scale the aggregate figures are: 42.5 FP8 ExaFLOPS and 1.77 PB of directly addressable HBM across the full superpod7.
6. Ironwood's inference-first design philosophy
v5e, v5p, and v6e are all described by Google as flexible training-and-serving chips with an emphasis leaning one way or the other (v5e cost/serving, v5p performance/training). Ironwood is the first chip in this lineup Google frames the opposite way: as inference hardware first, with training capability as a consequence of scale rather than the design target. "Ironwood is specifically engineered for this [inference], with its large on-chip memory and compute power optimized for both the large-batch 'prefill' phase and the memory-bandwidth-intensive 'decode' phase of large generative models."7
6.1 Prefill and decode as explicit codesign targets
Prefill (processing the prompt, one forward pass over many tokens at once) is arithmetic-intensity-heavy and benefits from raw FLOPs; decode (generating one token at a time, re-reading the full KV cache every step) is bandwidth-bound and benefits from HBM bandwidth and capacity relative to compute. Table 1's HBM-bandwidth-per-chip figures let this be checked directly rather than taken on faith: dividing HBM bandwidth by peak bf16 compute gives bytes of memory bandwidth available per FLOP of compute, a chip-level (not workload-level) indicator of how memory-rich a design is relative to its own compute.
v6e has the lowest bytes/FLOP ratio of the four chips (0.00178) — it is, by this measure, the most compute-heavy relative to its own memory bandwidth, which fits a chip whose headline generational claim was raw TFLOPs (§2.3). Ironwood's ratio (0.00320) is nearly 1.8x v6e's despite Ironwood having over 2x v6e's absolute compute — Ironwood added memory bandwidth faster than it added compute, which is exactly the shape of change you would make if the target workload were shifting from throughput-bound training toward a mix that includes bandwidth-bound decode. This is a derived metric, not a number Google publishes directly, but it is arithmetic on Google's own published peak figures and it is directionally consistent with the plain-language design claim quoted above.
6.2 Perf/watt trajectory: Ironwood vs. Trillium vs. the 2018 TPU
Google states two perf/watt ratios for Ironwood directly: "Ironwood perf/watt is 2x relative to Trillium, our previous-generation TPU ... nearly 30x more power efficient than our first Cloud TPU from 2018."7 Both ratios share Ironwood as the numerator, which makes it possible to derive Trillium's own perf/watt relative to the 2018 chip, even though Google does not state that ratio directly:
$$ \frac{\text{Ironwood}}{\text{TPU v1 (2018)}} = 30\times, \qquad \frac{\text{Ironwood}}{\text{Trillium}} = 2\times \implies \frac{\text{Trillium}}{\text{TPU v1 (2018)}} = \frac{30}{2} = 15\times $$This 15x figure is a derived quantity, not a Google-published number, and it inherits whatever rounding sits inside Google's "nearly 30x" (stated as an approximation, not an exact figure7) — treat it as an estimate accurate to roughly the same precision as that qualifier, not a third significant figure.
This trajectory is presented by Google as continuous engineering progress, not a step change unique to going inference-first — the framing to hold onto is narrower: Ironwood is the generation where the two-phase inference workload (prefill, decode) became a named codesign target rather than something a general-purpose training chip happened to also run acceptably. Whether that framing is best read as a genuine inference-first architecture or primarily as a chiplet-driven manufacturing decision with inference tuning layered on top is exactly the disagreement below.
Where experts disagree
Every fact in this section comes from Google's own published material, which makes the disagreement unusual: it is not two outside analysts reading the same chip differently, it is two of Google's own documents emphasizing different, non-contradictory causes for the same design decision. That is still worth resolving explicitly, because the two framings lead to different predictions about what Google will ship next — an "inference-first" reading suggests future chips fork further by workload, while a "manufacturing economics" reading suggests the chiplet split generalizes to any future chip regardless of target workload, the same way it already has on the GPU side.
Ironwood is architected for inference, and the dual-chiplet split is part of that
Google Cloud's own Ironwood-launch engineering blog7
The blog leads with the workload, not the silicon: Ironwood exists because "Ironwood is specifically engineered for [inference], with its large on-chip memory and compute power optimized for both the large-batch 'prefill' phase and the memory-bandwidth-intensive 'decode' phase"7. Under this reading, the per-chiplet 96 GB HBM allocation, the four SparseCores per chip, and the memory-bandwidth-to-compute ratio recovering past Trillium's (§6.1) are all consequences of designing around the prefill/decode split from the start.
- Matches the measured bytes/FLOP recovery from v6e to Ironwood (§6.1) — the hardware ratio moved in the direction the stated workload target implies.
- Consistent with Google marketing Ironwood, not v6e or v5p, as the generation aimed at "the age of inference"7.
- A launch blog has an obvious incentive to frame every design choice as workload-driven rather than cost-driven; it is not a neutral source on the question of motivation.
- Does not explain why the split specifically takes the form of two independent chiplets with separate memory spaces — a structure whose stated rationale, in a different Google document, is manufacturing cost (see the other position).
The chiplet split is a manufacturing-economics decision; inference tuning is layered on top of it
Google Cloud's own TPU7x system-architecture documentation5
The technical docs state the reason for the MegaCore-to-dual-chiplet change in one sentence, and it is not about inference: "This change improves the cost-effectiveness and efficiency of manufacturing the chip."5 That is a yield/reticle-limit argument that would apply to any sufficiently large chip on a leading process node regardless of target workload — it is the same category of reasoning behind NVIDIA's Blackwell dual-die package and AMD's XCD chiplets (pages 1 and 2 cover both), neither of which is framed as inference-specific.
- Directly quoted from the primary technical documentation, not marketing copy, and stated as the explanation with no workload qualifier attached.
- Matches an industry-wide pattern (GPU chiplets on pages 1-2) that has nothing to do with inference specifically, making it the more parsimonious explanation for the chiplet decision itself.
- Does not by itself explain why each chiplet was sized at exactly 96 GB HBM and 2 SparseCores, or why the die-to-die link was built at roughly the chip's full external ICI bandwidth (§2.4) — choices that go beyond what a pure yield argument requires and do track the memory-bandwidth-recovery pattern in §6.1.
- Treats "manufacturing cost" and "inference-optimized" as though they must be alternatives, when Google's own two documents describe the same chip.
These are not actually competing explanations for the same fact — they answer different questions. Why is Ironwood built from two chiplets instead of one larger die? Google's technical documentation gives a manufacturing-economics answer, and there is no reason to doubt it: the same reticle-limit logic explains chiplet designs across the industry independent of workload (pages 1 and 2 for the GPU cases). Why, given that a chiplet split was going to happen anyway, is each chiplet's memory and SparseCore allocation shaped the way it is, and why does the aggregate chip's memory-bandwidth-to-compute ratio move back toward v5e's after v6e pushed it down (§6.1)? That is a workload question, and the bytes/FLOP arithmetic in §6.1 is real, derived evidence — not just marketing language — that the specific allocation decisions inside the chiplet split are inference-shaped. Read it as: the chiplet boundary was forced by manufacturing economics; what goes inside each chiplet was a choice, and that choice is where the "inference-first" claim is actually falsifiable and, on the bytes/FLOP evidence available, currently holds up. What would settle it further is a like-for-like decode-throughput-per-watt or decode-throughput-per-dollar benchmark against v6e and against GPU parts on real serving workloads — a number this page does not have, and which belongs with the benchmark methodology on page 5 and the cross-vendor cost data on page 6.
References
- officialGoogle Cloud. TPU system architecture. Live documentation. https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm. Accessed 2026-09-17. Used for: MXU systolic-array mechanism and array-size-by-generation, "16K multiply-accumulate operations per cycle" statement, SparseCore dataflow-processor description and per-generation SparseCore counts, TPU cube/topology/Multislice definitions, ICI resiliency and Optical Circuit Switching between cubes.
- officialGoogle Cloud. TPU v5e. Live documentation. https://docs.cloud.google.com/tpu/docs/v5e. Accessed 2026-09-17. Used for: v5e per-chip compute/HBM/ICI specs, TensorCore/MXU count, 2D torus and pod-scale figures, supported serving slice shapes and the Sax multi-host inference note.
- officialGoogle Cloud. TPU v5p. Live documentation. https://docs.cloud.google.com/tpu/docs/v5p. Accessed 2026-09-17. Used for: v5p per-chip compute/HBM/ICI specs, TensorCore/SparseCore/MXU counts, 3D torus and full-cube wraparound rule, single-slice and Multislice pod-scale figures.
- officialGoogle Cloud. TPU v6e (Trillium). Live documentation. https://docs.cloud.google.com/tpu/docs/v6e. Accessed 2026-09-17. Used for: v6e per-chip compute/HBM/ICI specs, MXU array size and count, SparseCore feature listing, pod-level bandwidth figures, supported slice shapes.
- officialGoogle Cloud. TPU7x (Ironwood). Live documentation. https://docs.cloud.google.com/tpu/docs/tpu7x. Accessed 2026-09-17. Used for: Ironwood per-chip compute/HBM/ICI specs, dual-chiplet architecture and MegaCore departure, chiplet composition and D2D interconnect multiplier, per-axis 3D torus bandwidth, documented VM topology table, consolidated v5p/v6e/TPU7x spec table.
- officialGoogle Cloud. SAX on Cloud TPU v5e. Live documentation. https://docs.cloud.google.com/tpu/docs/SAX-on-inference-converter. Accessed 2026-09-17. Used for: Sax admin-server/model-server architecture, leader election and failover, single-host vs. multi-host model-serving requirements, the multi-host-serving availability caveat.
- blogGoogle Cloud. Inside the Ironwood TPU codesigned AI stack. Google Cloud Blog, 2025. https://cloud.google.com/blog/products/compute/inside-the-ironwood-tpu-codesigned-ai-stack. Accessed 2026-09-17. Used for: superpod cube/OCS topology and aggregate FLOPS/HBM figures, explicit prefill/decode inference-era design framing, perf/watt ratios vs. Trillium and the 2018 first-generation Cloud TPU.