AI Inference Hardware · Page 2

AMD Instinct CDNA3/CDNA4/CDNA5: MI300X to MI455X and the Helios Rack

MI300X and MI325X (CDNA3) stack eight compute dies on four base dies to ship more HBM per GPU than anything NVIDIA sells; MI350X and MI355X (CDNA4) keep that layout and roughly double per-CU matrix throughput; MI455X (CDNA5) and the Helios rack are AMD's officially announced answer to NVIDIA Rubin, still eleven months from shipping. After this you can map each SKU to a memory-bound serving scenario by capacity and bandwidth alone, explain exactly what changed physically between generations, and read AMD's own MI455X-vs-Rubin comparison chart for what it is — AMD's characterization, on AMD's stated basis, pending independent measurement.

Deep dive 35 min read 13 sources As of 2026-09-17
MI300-series OAM module: eight XCDs on four base dies, shared Infinity Cache, HBM stacks, and Infinity Fabric links to seven peer GPUs One OAM module (MI300X / MI325X / MI350X / MI355X) XCD0 XCD1 XCD2 XCD3 XCD4 XCD5 XCD6 XCD7 38 CUs each (MI300X/325X) 32 CUs each (MI350X/355X) 4 base dies (I/O dies) 256 MB Infinity Cache, shared across all 8 XCDs 8x HBM3 (192GB, MI300X) / HBM3E (256GB MI325X, 288GB MI350X/355X) stacks 5.3-8 TB/s 7 peer GPUs 7x128-153.6 GB/s Host CPU (PCIe Gen5) 128 GB/s x16
Figure 1. One OAM (OCP Accelerator Module) package for the whole MI300 family: 8 accelerator compute dies (XCDs) sit on 4 base dies that hold 256 MB of shared Infinity Cache, fed by 8 HBM stacks and reaching 7 peer GPUs over dedicated Infinity Fabric links plus one PCIe Gen5 x16 host link. XCD and base-die count, CU count per XCD, and Infinity Cache size from AMD's MI300X data sheet1 and Chips and Cheese's CDNA3/CDNA4 die-layout analysis2; per-link and per-stack bandwidth ranges span MI300X through MI355X, detailed in Table 1.

1. The XCD/chiplet package (CDNA3)

MI300X and MI325X are the same multi-die package built on the same CDNA3 compute dies; the only physical difference between them is the HBM stacks. Both put eight Accelerator Compute Dies (XCDs) on top of four base dies, using AMD's mixed 3D/2.5D packaging: the XCDs are 3D-stacked directly onto the base dies, and the base dies sit on a 2.5D interposer next to the HBM stacks2. AMD's own data sheet states the arrangement directly: "Eight accelerated compute dies (XCDs) with 38 compute units (CUs), 32 KB of L1 cache per CU, 4 MB shared L2 cache shared across CUs, and 256 MB of AMD Infinity Cache shared across 8 XCDs"1. Chips and Cheese's independent teardown adds the detail AMD's data sheet doesn't spell out: the 256 MB of Infinity Cache lives in the four base dies underneath the XCDs, not on the compute dies themselves — "eight XCDs sit atop four base dies, which implement 256 MB of memory side cache"2. Every XCD reaches every base die, so any compute unit on any of the eight dies sees the same 256 MB Infinity Cache as a single shared last-level cache, not eight private slices.

1.1 Compute unit and matrix core anatomy

MI300X carries 304 compute units in total (38 per XCD x 8 XCDs), 1,216 matrix cores (4 per CU), and 19,456 stream processors, clocked up to 2,100 MHz1. Each CU keeps 32 KB of L1 vector cache, and each XCD has 4 MB of L2 shared across its 38 CUs12. Below L1 sits the Local Data Share (LDS), CDNA's software-managed scratchpad: 64 KB per CU on CDNA3, organized as 32 banks of 2 KB each, delivering up to 128 bytes per cycle2. Each CU's vector ALUs are conventional: 128 FP32 lanes delivering 256 FP32 FLOPS per cycle, unchanged in count between CDNA3 and CDNA42 — the generational gain (section 3) comes almost entirely from the matrix cores, not the vector pipeline.

The matrix cores are where AMD's own AI-peak numbers live. MI300X's data sheet lists dense/2:4-sparse peak throughput of 1,307.4 / 2,614.9 TFLOPS for FP16 and BF16, 2,614.9 / 5,229.8 TFLOPS for FP8 and INT81. AMD frames this against its own prior generation rather than against NVIDIA: "13.7x the peak AI/ML workload performance using FP8 with sparsity compared to prior AMD MI250X accelerators using FP16"1 — a same-vendor generational claim, not an independent cross-vendor benchmark (independent cross-vendor GEMM measurements are covered on page 5, not here).

Memory hierarchy from one compute unit up to HBM, CDNA3 and CDNA4 values side by side Hierarchy: one CU up to HBM Compute Unit (CU) Matrix core x4 128 FP32 lanes L1: 32 KB LDS 64→160 KB x38 (MI300X/325X) or x32 (MI350X/355X) per XCD XCD L2 cache 4 MB / XCD shared by 32-38 CUs Infinity Cache 256 MB total in 4 base dies shared by all 8 XCDs HBM3 / HBM3E 5.3-8 TB/s per GPU LDS: 32 banks x 2 KB, 128 B/clk (CDNA3) 160 KB, 256 B/clk (CDNA4, section 3.2)
Figure 2. The memory path from one compute unit's registers to HBM. L1/L2/LDS sizes and Infinity Cache placement from AMD's MI300X and MI355X data sheets15 and Chips and Cheese's die-layout and LDS analysis2; HBM bandwidth range spans MI300X (5.3 TB/s) to MI350X/MI355X (8 TB/s), see Table 1.

1.2 Infinity Cache's role relative to HBM

256 MB is small next to 192-288 GB of HBM — roughly 0.1-0.13% of capacity — so Infinity Cache is not a working-set cache for model weights the way an on-chip SRAM would need to be for that job. Its function is narrower and specific to what section 1.1's memory path implies: every XCD's L2 is only 4 MB, shared across up to 38 CUs, so any access that misses L2 and would otherwise go all the way to HBM instead has one more stop — a 256 MB last-level cache shared by all eight XCDs1 — before paying the full HBM round trip. Because the cache is shared across dies rather than private per-XCD, it also does useful work for cross-XCD traffic: data one XCD wrote can be served to a different XCD from Infinity Cache without a trip to HBM at all, which matters for the all-to-all communication pattern of mixture-of-experts inference (expert-parallel token routing among the eight dies) and for KV-cache reads shared across attention heads mapped to different XCDs. AMD's data sheets state the existence and size of Infinity Cache on every SKU in this module but do not publish a hit-rate or effective-bandwidth-multiplier figure for it15 — unlike HBM bandwidth, which is a stated peak number, Infinity Cache's practical benefit is workload-dependent and not something this module can quantify beyond the architecture itself.

2. MI300X vs MI325X: the memory-capacity refresh

MI325X is not a new compute die — it is the identical CDNA3 XCD package repackaged with more and faster HBM stacks. AMD's own MI325X data sheet restates the same compute-side numbers as MI300X line for line: "GPU compute units 304; Matrix cores 1216; Stream processors 19,456; Peak engine clock 2100 MHz"3, and the same AI-peak-theoretical table — FP16/BF16 1,307.4/2,614.9 TFLOPS, FP8 2,614.9/5,229.8 TFLOPS dense/sparse3 — matches MI300X's table1 exactly. What changes is memory: MI300X ships up to 192 GB of HBM3 at "5.3 TB/s max. peak theoretical" bandwidth1; MI325X ships up to 256 GB of HBM3E at "6 TB/s max. peak theoretical" bandwidth3 — 33% more capacity, 13% more bandwidth, same compute. Max TBP rises from 750 W to 1,000 W to power the extra HBM stacks and sustain the higher bandwidth13.

AMD's framing for the extra capacity is explicit: "An industry-leading 256 GB of HBM3E memory and 6 TB/s bandwidth enables a single accelerator to contain and process a one-trillion parameter model"3 — at INT4/FP4-class weight precision, where 1 trillion parameters need roughly 500 GB at 1 byte/parameter or roughly 250 GB at 0.5 bytes/parameter (4-bit), leaving headroom for KV cache and activations is the arithmetic behind that claim, though AMD does not spell out the precision assumption itself. The practical read: MI325X's edge over MI300X is exclusively a capacity/bandwidth edge, useful when a model's weights plus KV cache do not fit in 192 GB but do fit in 256 GB, or when decode is bandwidth-starved at the margin — it buys no additional matrix-core throughput for compute-bound prefill.

An 8-GPU MI325X platform is a drop-in replacement for an 8-GPU MI300X platform: "A platform powered by 8 AMD Instinct MI325X GPU accelerators boasts a mind-boggling two terabytes of memory... The MI325X Platform offers seamless scalability as a drop-in replacement for the Instinct MI300X Platform"3 — both use the same 7x128 GB/s Infinity Fabric mesh and the same Universal Baseboard (UBB) form factor, detailed in section 4. The "two terabytes" figure is a coherent pool, not eight separate 256 GB islands requiring explicit copies between them — section 4 covers the mechanism.

Capacity, not bandwidth, is what decides whether a model fits on one GPU at all. A 70 B-parameter model at FP8 (roughly 1 byte/parameter) needs about 70 GB just for weights — comfortable on a single MI300X's 192 GB, with well over 100 GB left for KV cache and activations. A 405 B-parameter model at FP8 needs roughly 405 GB, which does not fit on any single MI300-series GPU in this module (MI325X's 256 GB included) — it needs at least two GPUs' worth of HBM before KV cache is even considered, or a narrower weight format. A 1 T-parameter MoE model at FP4/INT4 (roughly 0.5 bytes/parameter) needs roughly 500 GB, which is why AMD frames the "one-trillion-parameter model" claim specifically around MI325X's 256 GB per GPU combined with multi-GPU capacity, not a single accelerator in isolation, despite the data sheet's single-accelerator phrasing3 — 256 GB alone is roughly half of what a 1 T-parameter model needs at 4-bit precision, before KV cache. These are illustrative, precision-dependent estimates, not a substitute for the full cross-vendor model-size-to-hardware fit worked through on this track's hardware-selection page.

Table 1. Instinct MI300-series spec comparison, per GPU. CDNA3 rows from AMD's MI300X and MI325X data sheets13; CDNA4 rows from the MI350X platform and MI355X data sheets45 (MI350X per-GPU figures divided from the 8-GPU platform data sheet, noted); CDNA5 row from AMD's MI400-series product page6.
Part (gen)Compute unitsHBM typeCapacity (GB)Bandwidth (TB/s)FP8 dense (PFLOPS)FP4/FP6 dense (PFLOPS)TBP (W)Cooling
MI300X (CDNA3)304HBM31925.32.6149n/a750Air
MI325X (CDNA3)304HBM3E2566.02.6149n/a1000Air
MI350X (CDNA4)256HBM3E≈288*8.0n/a†≈9.23*1000Air
MI355X (CDNA4)256HBM3E2888.05.033210.06631400Liquid
MI455X (CDNA5)≤256 WGP‡HBM443223.320§40n/an/a

* Divided from the MI350X 8-GPU platform data sheet's stated totals (2.3 TB HBM3E ÷ 8 ≈ 288 GB; 73.816 PFLOPS MXFP4 platform-wide ÷ 8 ≈ 9.23 PFLOPS per GPU)4, not a directly stated per-GPU figure. † MI350X's FP8-dense figure is not stated in AMD's platform data sheet. ‡ MI455X is specified in Work Group Processors (WGP) with Wave32 execution, not Compute Units (CU) with Wave64 — see section 7.1; the two units are not directly comparable from AMD's public materials. § AMD's MI455X page states "FP8 | FP6" together at 20 PFLOPS, not FP8 alone6.

3. CDNA4: MI350X and MI355X

CDNA4 keeps the eight-XCD-on-four-base-die layout from section 1 but changes what is inside each die. MI350X and MI355X both carry 256 compute units, 1,024 matrix cores, and 16,384 stream processors per GPU45 — fewer CUs than MI300X's 304, because, per Chips and Cheese, "the CDNA 4 equipped MI355X slightly cuts down CU count per XCD" to hold yields steady on a more complex die2. Each CU still runs 32 CUs' worth of 128 FP32 vector lanes at 256 FLOPS/cycle — unchanged from CDNA32 — so CDNA4's gain is concentrated entirely in the matrix cores and the memory path, not the general vector pipeline.

3.1 Matrix-core throughput and native MXFP4/MXFP6

MI355X's data sheet lists dense/2:4-sparse matrix throughput of 2.5166/5.0332 PFLOPS for FP16 and BF16, 5.0332/10.0664 POPS for INT8, 5.0332/10.0664 PFLOPS for the new MXFP8 format, and 10.0663 PFLOPS for MXFP6 and MXFP4 (no sparse figure given for the 4/6-bit formats)5. MXFP4 and MXFP6 are OCP Microscaling (MX) formats, defined by an industry group that includes AMD itself alongside Microsoft, Intel, Meta, NVIDIA, and Qualcomm: each MX block covers 32 consecutive elements sharing one 8-bit power-of-two scale (format E8M0), with the elements themselves stored as narrow floats — 4 bits for MXFP4 (1 sign, 2 exponent, 1 mantissa bit, i.e. E2M1) and 6 bits for MXFP6 (E2M3 or E3M2)7. The shared per-block scale is what makes 4-bit weights usable at all: a lone 4-bit float has almost no dynamic range, but factoring out one shared exponent per 32-element block lets each element spend its bits entirely on relative precision within that block's range, which is why the format specification frames itself as "a drop-in replacement for baseline FP32 for AI inference and training with low user friction"7. CDNA3 has no native equivalent at all; MI300X's data sheet stops at FP8/INT81. Dividing by CU count turns these GPU-level numbers into a per-CU comparison: MI300X/MI325X deliver 2,614.9 TFLOPS FP8 dense over 304 CUs, or 8.60 TFLOPS per CU; MI355X delivers 5,033.2 TFLOPS FP8 dense over 256 CUs, or 19.66 TFLOPS per CU — a 2.3x per-CU gain on the one format both generations share15. Chips and Cheese's independent read of the announcement matches this: "per-CU matrix throughput doubles in many cases, with CDNA 4 CUs matching Nvidia's B200 SMs in FP6" — though the same analysis cautions that "B200 SMs have twice as much per-clock throughput as a CDNA 4 CU across a range of 16-bit and 8-bit data types," so CDNA4 closes the per-CU gap to Blackwell in the narrowest formats without closing it everywhere2. (Blackwell's own numbers are established in this track's NVIDIA page, not re-derived here.)

Per-compute-unit dense matrix throughput, CDNA3 vs CDNA4, FP8 and MXFP4 TFLOPS /CU 0 10 20 30 40 8.60 MI300X/325X 19.66 MI355X FP8 dense (shared format) n/a (no CDNA3 FP4) 36.04 MI350X 39.32 MI355X MXFP4 dense (CDNA4-only)
Figure 3. Dense matrix throughput per compute unit, derived by dividing each data sheet's GPU-level PFLOPS/TFLOPS figure by its stated CU count. FP8: MI300X/MI325X 2,614.9 TFLOPS ÷ 304 CU = 8.60 TFLOPS/CU13; MI355X 5,033.2 TFLOPS ÷ 256 CU = 19.66 TFLOPS/CU5. MXFP4: MI350X 73,816 TFLOPS platform ÷ 8 GPUs ÷ 256 CU = 36.04 TFLOPS/CU4; MI355X 10,066.3 TFLOPS ÷ 256 CU = 39.32 TFLOPS/CU5. CDNA3 has no native MXFP4 path1.

3.2 LDS: 64 KB to 160 KB, bandwidth doubling

CDNA3's Local Data Share is 64 KB per CU, organized as "32 2 KB banks, each 32 bits wide, providing up to 128 bytes per cycle"2. CDNA4 "increases the LDS capacity to 160 KB and doubles read bandwidth to 256 bytes per clock"2 — a 2.5x capacity increase and a 2x bandwidth increase in the same scratchpad. LDS is where attention kernels hold the score matrix tile and the running softmax statistics between the QK^T and PV matmuls; a bigger, faster LDS lets a single CU carry a larger KV-cache tile or attention block per pass before it has to round-trip to L2 or Infinity Cache, which matters most for long-context decode where the attention working set, not the matmul itself, is what determines occupancy. CDNA4 also widens the `GLOBAL_LOAD_LDS` path: CDNA3 supported 32-bit transfers per lane into LDS, while CDNA4 supports "moving up to 128 bits per lane" with transfer sizes of "1, 2, 4, 12, or 16 DWORDS"2, letting a single instruction stage a wider slice of a tensor straight from global memory into the scratchpad.

3.3 Worked derivation: MI350X and MI355X are one die at two clocks

AMD sells MI350X and MI355X as separate SKUs, but every count-based spec in their data sheets is identical per GPU: 256 compute units, 1,024 matrix cores, 16,384 stream processors45 — the same silicon, confirmed by dividing the MI350X platform data sheet's totals by 8 (2,048 CUs / 8 = 256; 8,192 matrix cores / 8 = 1,024; 131,072 stream processors / 8 = 16,3844), which land exactly on MI355X's stated per-GPU numbers5. What differs is clock (2,200 MHz vs. 2,400 MHz), cooling (air vs. liquid), and Max TBP (1,000 W vs. 1,400 W)45.

Checking that throughput scales with clock alone

Using Chips and Cheese's CDNA4 CU description — 128 FP32 lanes at 256 FLOPS/cycle, unchanged between MI350X and MI355X2 — per-GPU vector FP32 throughput is CU count × FLOPS/cycle × clock:

$$ \text{MI350X: } 256 \times 256 \times 2.2\times10^9 = 144.1\ \text{TFLOPS} $$ $$ \text{MI355X: } 256 \times 256 \times 2.4\times10^9 = 157.3\ \text{TFLOPS} $$

The ratio, 157.3 / 144.1 = 1.091, matches the clock ratio 2,400 / 2,200 = 1.0909 exactly, as it should for a fixed-width pipeline. The matrix-core numbers in AMD's own data sheets confirm the same scaling independently: MI350X's per-GPU MXFP4 dense throughput (73,816 TFLOPS platform-wide ÷ 8 = 9,227 TFLOPS4) against MI355X's stated 10,066.3 TFLOPS5 gives a ratio of 1.091 — the same 1.091, to three significant figures, as the clock ratio. MI350X and MI355X are not two designs; they are one design sold at two power envelopes, and the entire throughput delta between them is explained by a 9.1% clock difference.

AMD's own platform data sheet states the practical consequence directly: the Universal Baseboard (UBB) form factor "helps ensure seamless upgrades across MI300X, MI325X, and MI350X GPUs"4 — an operator who deploys an MI300X/MI325X platform today can swap in MI350X modules later without a board redesign, addressed further in section 4.

4. Infinity Fabric scale-up mesh and UBB 2.0

Inside one 8-GPU OAM baseboard, every GPU connects directly to every other GPU — a full mesh, not a ring or a switch fabric. MI300X's data sheet specifies "7x 128 GB/s" scale-up Infinity Fabric links per GPU, with "direct connectivity of 128 GB/s bidirectional bandwidth between each GPU"1: seven links because there are seven other GPUs on the board, one dedicated link to each. CDNA4 raises the per-link rate: MI350X's platform data sheet states "each MI350X connects with its peers through seven links... over 153.6 GB/s bidirectional links"4, and MI355X's data sheet confirms the same per-GPU figure, "7x 153.6 GB/s"5 — a 20% per-link increase over CDNA3, with the topology unchanged. Each GPU additionally carries one PCIe Gen5 x16 (128 GB/s) link to the host CPU, independent of the seven peer-to-peer links15.

Full-mesh Infinity Fabric topology across an 8-GPU OAM baseboard, one GPU's seven links highlighted 7 links x 128 (MI300X/325X) or 153.6 GB/s (MI350X/355X) GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7
Figure 4. One OAM baseboard, 8 GPUs, full mesh: every GPU has a dedicated Infinity Fabric link to every other GPU (28 links total; GPU0's 7 links highlighted). Link count and topology from AMD's MI300X data sheet1; per-link bandwidth from MI300X/MI325X (128 GB/s)1 and MI350X/MI355X (153.6 GB/s)45 data sheets.

This mesh, plus a common OAM connector and power envelope, is what the Universal Baseboard (UBB) 2.0 form factor standardizes across three generations. AMD states it plainly: "the platform's UBB form factor helps ensure seamless upgrades across MI300X, MI325X, and MI350X GPUs"4 — an 8-GPU server built for MI300X can accept MI325X or MI350X modules without a baseboard redesign, because the connector, the 7-link mesh, and the PCIe host link are unchanged; only the per-link rate rises with CDNA4. MI355X's higher 1,400 W TBP and liquid-cooling requirement means it is qualified for a subset of UBB 2.0 chassis rather than every MI300X-generation server5, so "UBB 2.0-compatible" is necessary but not sufficient for a drop-in MI355X upgrade — the chassis also needs to support direct liquid cooling and the higher power draw.

The mesh's practical payoff is that HBM on any of the eight GPUs is addressable as one coherent memory space, not eight separately-managed pools. AMD's MI300X data sheet describes this directly, in a section titled "Coherent Shared Memory": "direct connectivity of 128 GB/s bidirectional bandwidth between each GPU"1 is what lets a tensor-parallel or expert-parallel serving framework treat an 8-GPU platform's combined HBM as a single addressable pool rather than requiring the application to manage explicit peer-to-peer copies for every cross-GPU access — the same mechanism behind MI325X's "two terabytes of memory" framing in section 23. This is the same class of benefit NVLink/NVSwitch coherent domains provide on the NVIDIA side (covered on this track's NVIDIA page), implemented here over the 7-link Infinity Fabric mesh instead of a switched fabric — full mesh means every GPU pair gets a dedicated, non-shared link, at the cost of link count scaling as \(n(n-1)/2\) rather than through a central switch, which is part of why AMD's scale-up domain tops out at 8 GPUs per OAM baseboard rather than growing arbitrarily within a single coherent mesh.

5. Bytes per FLOP: reading the memory-vs-compute regime

A chip's ratio of DRAM bandwidth (bytes/s) to vector compute (FLOP/s) sets where its roofline ridge point sits: the higher the ratio, the less arithmetic intensity a kernel needs before it stops being memory-bound. Chips and Cheese computes this ratio directly for CDNA3 and CDNA4: "MI300X had ~0.03 bytes of DRAM bandwidth per FP32 FLOP, which increases to 0.05 on MI355X," and, for reference, "Blackwell for comparison has ~0.10 bytes of DRAM bandwidth per FP32 FLOP"2. Both AMD figures check out against AMD's own data sheets: MI300X's HPC table states FP32 vector throughput of 163.4 TFLOPS against 5.3 TB/s of bandwidth1, giving 5.3 / 163.4 = 0.0325 ≈ 0.03 bytes/FLOP; MI355X's per-CU vector width (128 FP32 lanes, 256 FLOPS/cycle2) times 256 CUs at 2,400 MHz gives 157.3 TFLOPS against 8 TB/s5, or 8 / 157.3 = 0.0509 ≈ 0.05 bytes/FLOP — matching Chips and Cheese's independently stated figure to two significant digits.

The same arithmetic extends to the two SKUs Chips and Cheese doesn't cover directly. MI325X shares MI300X's compute die exactly (section 2), so its vector FP32 throughput is also 163.4 TFLOPS; against its 6.0 TB/s of HBM3E3, that is 6.0 / 163.4 = 0.0367 ≈ 0.037 bytes/FLOP — more bandwidth-generous than MI300X because the memory refresh didn't touch the compute die. MI350X, at 256 CUs and 2,200 MHz (section 3.3), computes to 144.1 TFLOPS vector FP32 against its 8 TB/s4, or 8 / 144.1 = 0.0555 ≈ 0.056 bytes/FLOP — the highest ratio in the CDNA3/CDNA4 lineup, because MI350X pairs MI355X's memory bandwidth with a lower clock. For a comparison point outside AMD, NVIDIA's own H100 data sheet gives FP32 (non-Tensor-Core) throughput of 67 TFLOPS against 3.35 TB/s of HBM38, or 3.35 / 67 = 0.05 bytes/FLOP — coincidentally identical to MI355X's ratio, despite the two chips using unrelated compute widths and clocks to get there.

Bytes of DRAM bandwidth per FP32 FLOP, MI300-series vs H100 and B200 bytes /FLOP 0 0.025 0.05 0.075 0.10 0.03MI300X 0.037MI325X 0.056MI350X 0.05MI355X 0.05H100 0.10B200 (NVIDIA)
Figure 5. Bytes of HBM bandwidth per FP32 vector FLOP. MI300X, MI355X, and B200 figures stated directly by Chips and Cheese2; MI325X and MI350X derived by the same method from AMD's own data sheets134; H100 derived from NVIDIA's own H100 product page8. All AMD/NVIDIA figures use vector (non-Tensor/non-matrix) FP32 throughput as the denominator, matching Chips and Cheese's methodology; matrix/Tensor-Core throughput would give a different, much smaller ratio.

The CDNA3-to-CDNA4 direction — bandwidth growing faster than vector compute, and matrix compute growing faster still (section 3.1) — continues qualitatively into CDNA5. MI455X's data sheet gives 23.3 TB/s of HBM4 bandwidth against 40 PFLOPS of FP4 matrix throughput6, a ratio of roughly 0.0006 bytes per FP4 FLOP — not directly comparable to the figures above, since it uses matrix FP4 throughput rather than vector FP32 as the denominator (AMD's public MI455X materials do not state a vector FP32 figure). The point of including it is qualitative, not a sixth bar on the same chart: every AMD generation from CDNA3 through CDNA5 has added matrix-core throughput faster than it has added bandwidth, which is exactly why the LDS, Infinity Cache, and attention-kernel design decisions in sections 1 and 3 matter more with each generation, not less — there is more compute to keep fed from the same, relatively slower-growing, pool of bytes.

6. ROCm: how much of the CUDA gap is left

AMD's own MI355X data sheet states the framework surface plainly: "the ROCm platform supports all major AI and HPC frameworks, inference engines, and model-serving systems including PyTorch, TensorFlow, JAX, ONNX Runtime, Kokkos, Raja, SGLang, Triton, and vLLM"5. That list is accurate as a list — every name on it does run on ROCm — but a framework "supporting" ROCm and a framework being at CUDA-equivalent parity on ROCm are different claims, and the gap between them is where deployability actually lives. An independent SemiAnalysis field assessment published July 25, 2026, gives the most current account of where that gap stands, and it is more textured than either "ROCm is broken" or "ROCm is solved"9.

What has genuinely improved, per that assessment: single-node performance work landed fast — "up to 18x improvement" in one reasoning model's interactivity within 30 days through AITER and vLLM fixes — and AMD's own documentation now "spans vLLM, SGLang, distributed MoRI, Mooncake, and deployment guides," with the vLLM optimization guide covering "AITER, attention-backend selection, TP/EP/DP strategies, FP8/FP4 quantization"9. AMD shifted to what the assessment calls a "developer-first posture, upstream alignment, day-0 model enablement, and faster release cadence," and "ROCm now has a much deeper public recipe layer than it did a year ago"9. SGLang in particular is described as ahead of vLLM on ROCm: "SGLang's MI355X nightly images cover DeepSeek-V4 disaggregation with DP-attention, EP8, and MTP"9.

What remains genuinely behind, per the same assessment: continuous-integration gating — the tests that actually block a broken change from merging, as opposed to tests that merely run and report — is the metric that matters, and "gating parity and gating pass rate are what truly matter" rather than raw pass counts; AMD's progress on this "has massively regressed due to AMD cluster infra stability issues"9. For Kubernetes-orchestrated inference over AMD's Pollara NICs, the assessment reports "0% parity with Nvidia's ConnectX Nightly CI," and notes that "the planned ETA of AMD reaching parity by Advancing AI 2026 was missed due to cluster issues"9. Composability across the full distributed-serving stack is explicitly called out as unproven: "AMD cannot yet assume they will work together across models, quantization formats, parallelism strategies, speculative decoding, and network topologies without special cases"9. And bring-up for the newest hardware lags the rest of the stack: for MI455X (ISA name gfx1250), "PyTorch... ships with no gfx1250 CI" and "wave32 kernels are unwritten," and the assessment states plainly that "WideEP, validation, and the wave32-tuned high-value kernels are not there yet"9 — a direct consequence of the WGP/Wave32 execution-model change noted in section 7.1, which requires new kernels rather than a recompile of CDNA4's Wave64 code.

7. CDNA5: the MI400 series and the Helios rack

AMD announced its CDNA5-generation Instinct lineup and rack platform at Advancing AI 2025: "AMD Instinct MI400 Series GPUs" paired with "Zen 6"-based AMD EPYC "Venice" CPUs and AMD Pensando "Vulcano" NICs into a rack-scale system named Helios10. At the same event AMD framed the generational target explicitly against its own prior generation, not against a specific NVIDIA part: MI400 Series GPUs are "expected to deliver up to 10x more performance running inference on Mixture of Experts models"10, while the then-shipping MI355X was claimed at "up to 40% more tokens-per-dollar compared to competing solutions"10 — the competing-solutions comparison is unnamed in AMD's own press release. Independent industry analysis frames MI400/Helios's competitive target directly: "could potentially be competitive with Nvidia's VR200 NVL144 rack scale solutions in H2 2026"11 — VR200 NVL144 being SemiAnalysis's own shorthand for NVIDIA's next-generation rack, adjacent to but not identical to the Vera Rubin NVL72 naming used in section 8 and on this track's NVIDIA page.

7.1 MI455X and MI430X

MI455X is AMD's flagship CDNA5 part. AMD's own product page states it is "built on 5th Gen AMD CDNA architecture," with "up to 256 WGP Work Group Processors with Wave32 execution," "up to 3.6 TB/s Scale-up Bandwidth per GPU," and "up to 432 GB integrated HBM4 Memory with up to 23.3 TB/s Peak Theoretical Memory Bandwidth"6 — 432 GB is 1.5x MI355X's 288 GB, and 23.3 TB/s is 2.9x MI355X's 8 TB/s65. AMD's own separate Helios page states MI455X delivers "up to 40 PFLOPS FP4" per GPU12.

AMD's MI400-series page also names a second CDNA5 part, MI430X, aimed at sovereign-AI and HPC customers who need double-precision throughput: "The AMD Instinct MI430X GPUs are expected to be available in 2027"6, a full year behind MI455X/Helios's stated 2H 2026 volume-deployment window612. Neither AMD's MI400-series page nor the Helios page in this module's source set states MI430X's FP64 throughput, memory capacity, or bandwidth; it is named here only as roadmap context, per this track's convention of not asserting specs beyond what a vendor has published verbatim.

7.2 The Helios rack

AMD's Helios product page gives the full rack-level spec sheet directly: "72x AMD Instinct MI455X GPUs, 2.9 EF FP4 FLOPS, 1.4 EF FP8 FLOPS, 31 TB HBM4 memory capacity, 23.3 TB/s Per GPU memory bandwidth, 260 TB/s Scale up bandwidth, 43 TB/s Scale out bandwidth"12. Every rack-level number here is a straightforward multiple of the per-GPU MI455X figures: 40 PFLOPS FP4 per GPU x 72 GPUs = 2,880 PFLOPS = 2.88 exaFLOPS, matching AMD's stated 2.9 EF126; 432 GB HBM4 per GPU x 72 GPUs = 31.1 TB, matching AMD's stated 31 TB126 — both checks land within rounding of AMD's own rack totals, confirming the rack figures are a direct scale-out of the per-GPU spec sheet rather than a separately measured system number.

Helios rack: 72 MI455X GPUs, Venice EPYC CPUs, Vulcano NICs, and the scale-up/scale-out bandwidth split AMD Helios rack 72x MI455X GPUs 432 GB HBM4 / GPU, 23.3 TB/s / GPU (8 of 72 shown, repeated x9) Zen 6 EPYC "Venice" CPUs host + orchestration Pensando "Vulcano" NICs scale-out fabric Scale-up: 260 TB/s (intra-rack, GPU-to-GPU) Scale-out: 43 TB/s (rack-to-rack, via Vulcano) Rack totals: 2.9 EF FP4, 1.4 EF FP8, 31 TB HBM4 -- AMD Performance Labs peak-theoretical projections, June 2026
Figure 6. AMD's Helios rack: 72 MI455X GPUs, Zen 6 EPYC "Venice" host CPUs, and Pensando "Vulcano" NICs, with separate scale-up (intra-rack) and scale-out (rack-to-rack) bandwidth figures. All figures from AMD's Helios product page12 and MI400-series product page6; AMD's own footnote states the rack-level performance figures are "calculations by AMD Performance Labs" made in June 2026, i.e. vendor peak-theoretical projections, not independent or shipped-hardware measurements12.

AMD states the Helios reference design "is being shared with partners now, with volume deployments expected in 2H 2026"12 — the same window NVIDIA states for Vera Rubin NVL72 availability on this track's NVIDIA page. Neither platform has shipped or been independently benchmarked as of this writing.

Helios's scale-up bandwidth (260 TB/s) covers GPU-to-GPU traffic inside one rack; the 43 TB/s scale-out figure is a different fabric, connecting rack to rack through the Vulcano NICs. Independent industry analysis names the scale-out mechanism directly: MI400/Helios's scale-up domain spans "72 logical GPUs," reached over what the analysis calls "UALink over Ethernet," a rename of what was previously described as "IF over Ethernet" — Infinity Fabric traffic carried over standard Ethernet transport for the links that leave a single rack11. That is a different scale-out approach from carrying the same coherent-fabric protocol across racks natively, and AMD's own product pages in this module's source set do not elaborate on the distinction beyond naming the 43 TB/s figure12.

One date in this section is not a projection: MI350 Series parts (MI350X and MI355X, CDNA4, sections 2-4) were stated as "already rolling out" with "broad availability in 2H 2025" as of AMD's June 2025 announcement10 — meaning, as of this module's September 2026 as-of date, MI350X/MI355X are shipping, deployed hardware, unlike MI455X and Helios. AMD's "up to 10x more performance running inference on Mixture of Experts models" claim for the MI400 Series is stated without naming the prior-generation baseline or the specific MoE benchmark configuration9 — treat it the same way as the rack-level exaFLOPS figures in Figure 6: AMD's own vendor-projected number for hardware that has not yet shipped in volume.

8. AMD's own comparison: MI455X vs. NVIDIA Vera Rubin

AMD's MI400-series product page carries a direct head-to-head spec table against NVIDIA's next-generation part, the only place in AMD's public materials this comparison appears in table form6. It is reproduced verbatim below. What follows is AMD's own characterization, on AMD's own stated basis — this module does not adjudicate it; the full adjudication, cross-checked against NVIDIA's own Rubin materials and any independent data that becomes available, belongs to this track's hardware-selection page (page 6).

Table 2. AMD's own comparison table, MI455X vs. "NVIDIA Vera Rubin," reproduced verbatim from AMD's MI400-series product page6. AMD does not state whether the Rubin figures are dense or sparse, training or inference peak, or their source.
MetricAMD Instinct MI455XNVIDIA Vera Rubin (per AMD)
AI performance, FP4 / OCP MXFP4 (peak PFLOPS)4035
AI performance, FP8 | FP6 (peak PFLOPS)2018
AI performance, FP16 | BF16 (peak PFLOPS)54
Memory capacity (GB)432288
Memory bandwidth (TB/s)23.322.0

AMD's own footnotes on this table: "AMD Instinct MI455X GPUs offer 15% more OCP MXFP4 peak theoretical performance compared to NVIDIA Vera Rubin GPUs with FP4 datatype... offer 50% more memory capacity and 6% more memory bandwidth vs. NVIDIA Vera Rubin GPUs"6.

Two points are independently verifiable, though. First, both parts are pre-shipment: AMD states MI455X/Helios volume deployment for "2H 2026"12, and NVIDIA states Rubin platform availability for "second half of 2026"13 — the comparison in Table 2 is vendor-projected against vendor-projected, not measured against measured, on both sides. Second, AMD's own memory-capacity and bandwidth claims (432 GB, 23.3 TB/s) are internally consistent with the rest of this module: they match MI455X's own product-page spec exactly6, and the capacity and bandwidth deltas over MI355X (1.5x and 2.9x respectively, section 7.1) are AMD's own generation-over-generation numbers, not part of the disputed cross-vendor comparison.

Where experts disagree

Every AMD data sheet in this module lists ROCm's framework coverage as if it were a solved problem: PyTorch, vLLM, SGLang, Triton, all "supported"5. Whether that coverage is now deployable at CUDA-equivalent reliability, or whether it remains a materially riskier bet than an NVIDIA/CUDA stack, is a live, currently-unsettled question among people who actually run production inference on both — not a question this module's vendor sources can answer on their own, since AMD's own data sheets have an obvious incentive to describe support as complete.

The moat is closing fast enough to matter

AMD's own product positioning5, and the more optimistic reading of independent field reports9

AMD has shifted engineering posture toward upstream-first development, day-0 model enablement on release, and a much deeper public recipe layer than a year prior; single-node interactivity fixes (AITER/vLLM) delivered up to 18x gains within 30 days of a problem being identified, and SGLang's ROCm nightly images already cover frontier-model features (DeepSeek-V4 disaggregation, DP-attention, EP8, MTP)9. On this reading, the software gap is a matter of engineering velocity that AMD has now demonstrably increased, and agentic coding tools are further shrinking the labor cost of closing remaining gaps9.

  • Concrete, dated, measurable improvements (18x interactivity gain in 30 days) rather than a roadmap promise
  • Framework coverage genuinely broad: PyTorch, vLLM, SGLang, Triton, JAX, ONNX Runtime, TensorFlow all run5
  • AMD's release cadence and upstream alignment have measurably increased9
  • "Runs" and "gating-tested, CI-enforced, production-reliable" are different bars, and the evidence for the former is much stronger than for the latter
  • Improvements are concentrated in single-node serving; the harder, newer problem (distributed multi-node inference) is explicitly unproven

The competitive frontier is moving faster than the gap closes

The more cautious reading of the same July 2026 field assessment9

Even as AMD's raw progress rate has increased, several specific, currently-blocking gaps remain unresolved as of July 2026: CI gating parity for AMD hardware has "massively regressed" due to cluster instability, Kubernetes-orchestrated inference over AMD's Pollara NICs sits at "0% parity" with NVIDIA's ConnectX tooling, and the distributed-inference composability story (multiple parallelism strategies, quantization formats, and speculative decoding working together without special-casing) is explicitly unproven9. The newest hardware generation compounds this: MI455X uses a different ISA (gfx1250) than MI355X (gfx950), and as of the same assessment "wave32 kernels are unwritten" and PyTorch ships with "no gfx1250 CI" for it at all9, meaning the software-maturity clock effectively restarts with every CDNA generation rather than compounding cleanly.

  • Grounded in the same source's own stated blocking issues, not speculation
  • Explains why AMD's own framework-support list5 is necessary but not sufficient evidence of deployability
  • Directly relevant to whether a new hardware generation (MI455X) inherits the prior generation's software maturity or starts over
  • Understates genuine, dated progress on the dimensions that did improve (single-node serving, upstream engagement)
  • Some blockers (cluster infrastructure stability) are AMD-internal-operations problems, not inherent ROCm architecture problems, and could resolve independently of the software stack itself
Assessment

For single-node inference serving on already-shipping hardware (MI300X, MI325X, MI350X, MI355X) with a mainstream model on vLLM or SGLang, the practical gap to CUDA has narrowed enough that it is a due-diligence item, not an automatic disqualifier — verify the specific model/quantization/parallelism combination has a gating-tested nightly path before committing, rather than assuming parity from the framework-support list alone. For distributed multi-node inference, Kubernetes-native orchestration over AMD's own NIC fabric, or day-one deployment on brand-new silicon (MI455X at the time of its launch), treat ROCm maturity as unproven until an independent source demonstrates it, since the same July 2026 assessment that documents AMD's real progress is also the one documenting these specific, current blockers9. What would settle this for a specific deployment: whether the target model/quantization/parallelism-strategy combination appears in a CI-gated (not just nightly-built) test on the target GPU's exact ISA — a fact AMD's own product pages do not report and that changes release to release.

References

  1. officialAMD. AMD Instinct MI300X Accelerator — Data Sheet. GD-83, 2025. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf. Accessed 2026-09-17. Used for: XCD/base-die/CU counts, cache hierarchy, AI/HPC peak-theoretical FLOPS, HBM3 capacity/bandwidth, Infinity Fabric link count/bandwidth, TBP.
  2. analysisLam, Chester / Chips and Cheese. AMD's CDNA 4 Architecture Announcement. 2025. https://chipsandcheese.com/p/amds-cdna-4-architecture-announcement. Accessed 2026-09-17. Used for: base-die/XCD stacking detail, per-CU vector-lane count, LDS capacity/bandwidth (CDNA3 vs CDNA4), CDNA4-vs-Blackwell per-CU/per-SM throughput comparison, bytes-per-FP32-FLOP ratios for MI300X, MI355X, and Blackwell.
  3. officialAMD. AMD Instinct MI325X Accelerator — Data Sheet. GD-83, 2025. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/instinct-mi325x-datasheet.pdf. Accessed 2026-09-17. Used for: MI325X compute-die specs (identical to MI300X), HBM3E capacity/bandwidth, TBP, drop-in platform compatibility, trillion-parameter framing.
  4. officialAMD. AMD Instinct MI350X Platform — Data Sheet. GD-83, PID#253461432A, 2025. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/amd-instinct-mi350x-platform-brochure.pdf. Accessed 2026-09-17. Used for: 8-GPU platform totals (CU/matrix-core/stream-processor/HBM3E/MXFP4-MXFP6 counts, divided by 8 for per-GPU figures), per-GPU bandwidth, Infinity Fabric link bandwidth, TBP, UBB 2.0 cross-generation compatibility.
  5. officialAMD. AMD Instinct MI355X GPU — Data Sheet. GD-83, LE-92703-00, 10/2025. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/amd-instinct-mi355x-gpu-brochure.pdf. Accessed 2026-09-17. Used for: MI355X CU/matrix-core counts, AI peak-theoretical FLOPS (FP16/FP8/MXFP8/MXFP6/MXFP4), HBM3E capacity/bandwidth, Infinity Fabric link bandwidth, TBP/cooling, ROCm framework-support claim.
  6. officialAMD. AMD Instinct MI400 Series GPUs (product page). 2026. https://www.amd.com/en/products/accelerators/instinct/mi400.html. Accessed 2026-09-17. Used for: MI455X WGP/Wave32 spec, HBM4 capacity/bandwidth, AMD's own MI455X-vs-Vera-Rubin comparison table and footnotes, MI430X availability.
  7. paperRouhani, B.D. et al. (Microsoft, AMD, Intel, Meta, NVIDIA, Qualcomm Technologies). Microscaling Data Formats for Deep Learning. arXiv:2310.10537v3, 2023-10-19. https://arxiv.org/abs/2310.10537. Accessed 2026-09-17. Used for: MX block-scaled format definition (block size, shared E8M0 scale, MXFP4/MXFP6 element bit-widths) underlying CDNA4's native MXFP4/MXFP6 support.
  8. officialNVIDIA. H100 GPU (product page). 2026. https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-09-17. Used for: H100 FP32 vector throughput and HBM3 bandwidth, to compute the H100 bytes-per-FLOP reference point in Figure 5.
  9. analysisSemiAnalysis. Can AMD Break the CUDA Moat? AMD Advancing AI 2026. 2026-07-25. https://newsletter.semianalysis.com/p/can-amd-break-the-cuda-moat-amd-advancing. Accessed 2026-09-17. Used for: ROCm software-maturity assessment (gating-test parity, Kubernetes/Pollara NIC CI parity, distributed-inference composability, MI455X/gfx1250 bring-up status) and the disagree section.
  10. officialAMD Investor Relations. AMD Unveils Vision for an Open AI Ecosystem, Detailing New Silicon, Software and Systems at Advancing AI 2025 (press release). 2025-06-12. https://ir.amd.com/news-events/press-releases/detail/1255/amd-unveils-vision-for-an-open-ai-ecosystem-detailing-new-silicon-software-and-systems-at-advancing-ai-2025. Accessed 2026-09-17. Used for: Helios/MI400 announcement, Venice CPU/Vulcano NIC naming, MI400 Series MoE-inference claim, MI355X tokens-per-dollar claim.
  11. analysisSemiAnalysis. AMD Advancing AI: MI350X and MI400 UALoE72, MI500 UAL256. 2025. https://newsletter.semianalysis.com/p/amd-advancing-ai-mi350x-and-mi400-ualoe72-mi500-ual256. Accessed 2026-09-17. Used for: MI400/Helios competitive framing against NVIDIA's next-generation rack, UALink-over-Ethernet naming.
  12. officialAMD. AMD Helios Rackscale Solution — Powering Frontier AI (product page). 2026. https://www.amd.com/en/products/rackscale-solutions/helios.html. Accessed 2026-09-17. Used for: Helios rack-level spec sheet (GPU count, FP4/FP8 exaFLOPS, HBM4 capacity, scale-up/scale-out bandwidth), per-GPU MI455X FP4 figure, deployment timeline, peak-theoretical-projection footnote.
  13. officialNVIDIA Newsroom. NVIDIA Kicks Off the Next Generation of AI With Rubin — Six New Chips, One Incredible AI Supercomputer. 2026-01-05. https://nvidianews.nvidia.com/news/rubin-platform-ai-supercomputer. Accessed 2026-09-17. Used for: NVIDIA's own stated Rubin GPU NVFP4 inference figure and platform availability window, placed alongside AMD's comparison chart without reconciling it.