Multi-Node LLM Inference · Module 17
AMD MI300X/MI355X Multi-Node Serving: RCCL, InfiniBand/RoCE, and vLLM-ROCm
Modules 1–15 built a mental model of multi-node serving entirely on NVIDIA parts: NVLink, NCCL, DeepEP, NIXL, Dynamo, TensorRT-LLM. This module runs the same questions against AMD's Instinct platform — MI300X and MI355X — and is explicit about which pieces are a drop-in substitution, which are a real but younger reimplementation, and which have no AMD-side answer in the sources available as of this writing. After reading, you can take any mechanism from Modules 1–15 and say what happens to it on ROCm.
1. Same problem, a different fabric
Everything Modules 1–15 established about why multi-node serving is hard — the bandwidth cliff at the node boundary, the goodput case for prefill/decode disaggregation, the all-to-all shape of cross-node MoE, KV cache as a tiered cluster resource — is hardware-agnostic. None of that gets re-derived here. What changes on AMD's Instinct platform is the layer directly underneath those ideas: the physical interconnect, the collective-communication library, the wide-expert-parallel dispatch kernel, and the maturity of the serving-engine integration that wires them together. This module works through each of those four layers in turn, then closes with a single table mapping every named NVIDIA-stack mechanism from Modules 1–15 to its AMD-side equivalent, partial equivalent, or gap.
The comparison is grounded in two AMD accelerators: MI300X, shipping since December 2023 and the subject of AMD's own Hot Chips 2024 architecture talk1, and MI355X, the next generation whose specs in this module come from a third-party aggregation source rather than an AMD-hosted product page — flagged explicitly in §2.4 because it matters for how much confidence to put in those specific numbers.
2. MI300X and MI355X: memory-first architecture
AMD's pitch for Instinct in LLM inference has always been memory capacity first, interconnect second — the opposite emphasis from NVIDIA's NVLink-centric story in Module 1. MI300X packs 153 billion transistors across a multi-die module built on TSMC 5nm/6nm FinFET, organized as eight XCD (accelerator complex die) compute chiplets plus four IOD (I/O die) memory controllers, all connected through an Infinity Fabric Advanced Package interposer1. The number that drives everything downstream in this course — how many GPUs an EP or TP group needs before it crosses a node boundary — is memory capacity: 192 GB of HBM3 per GPU, versus 80 GB on an H100 SXM12.
2.1 The chip: 304 compute units, 5.2–5.3 TB/s HBM bandwidth
Each MI300X exposes 304 Compute Units and 1,216 Matrix Cores to software, backed by 192 GB of HBM3 across eight memory stacks1. AMD's own Hot Chips slide states the per-GPU HBM bandwidth as 5.2 TB/s1; AMD's OEM-partner data sheet (reproduced verbatim in Lenovo's ThinkSystem product guide for the same part) states it as 5.3 TB/s3. That 0.1 TB/s gap is small enough to be rounding, but it is not arbitrary: multiply each figure by eight GPUs and only one of them reproduces AMD's own published 8-GPU platform aggregate.
Worked check: which per-GPU number is consistent with AMD's own platform aggregate?
AMD states the 8-GPU MI300X platform aggregate HBM bandwidth as 42.4 TB/s1. Dividing that by 8:
$$ 42.4\ \text{TB/s} \div 8 = 5.30\ \text{TB/s per GPU} $$That matches the 5.3 TB/s figure from the OEM data sheet exactly, not the 5.2 TB/s figure from the architecture talk. Going the other direction, 5.2 TB/s × 8 = 41.6 TB/s, about 1.9% below the published platform aggregate. The practical takeaway: use the platform-level number (42.4 TB/s) for capacity planning, since it is the one AMD's own materials are internally consistent about; treat the single-GPU 5.2 TB/s figure as a rounded headline number, not the one to multiply.
2.2 The 8-GPU platform: a direct mesh, not a switched fabric
Module 1 §1 established that H100/B200 GPUs inside an 8-GPU node reach any other GPU at the full per-GPU NVLink rate because NVSwitch chips provide a switched, full-bisection fabric. AMD's 8-GPU MI300X platform (the "UBB", universal baseboard) takes a structurally different approach: there is no switch chip. Each of the eight MI300X OAM modules has seven point-to-point bidirectional Infinity Fabric links running at 128 GB/s each, wired so that every GPU connects directly to every other GPU — a fully-connected mesh1. Seven links at 128 GB/s per link gives the per-GPU aggregate bandwidth figure quoted throughout AMD's materials:
$$ 7 \times 128\ \text{GB/s} = 896\ \text{GB/s per GPU} $$This is a direct, unswitched topology: a GPU-to-GPU hop never traverses a third chip the way it might on a switched fabric, but it also means the mesh does not scale past eight GPUs without adding a network hop — there is no AMD analog to NVL72's 72-GPU single-domain switch fabric (Module 1 §5) in the sources for this module. Beyond eight GPUs, AMD's own deployment guidance is to leave the mesh (Infinity Fabric "XGMI island") and use pipeline parallelism or cross-node RDMA instead of trying to extend TP4.
2.3 8-GPU MI300X platform vs. 8-GPU H100 HGX
Putting the mesh's per-GPU bandwidth together with each GPU's 192 GB of HBM3 gives the platform-level comparison AMD leads with: an 8-GPU MI300X platform against an 8-GPU H100 HGX.
| Part | HBM (GB) | HBM BW (TB/s) | Intra-node fabric | Fabric BW/GPU | TDP (W) |
|---|---|---|---|---|---|
| H100 SXM | 80 | 3.35 | NVLink 4 (NVSwitch) | 900 GB/s | 700 |
| B200 SXM (DGX/HGX) | 180 | 8.0 | NVLink 5 (NVSwitch) | 1.8 TB/s | —† |
| MI300X | 192 | 5.3 | Infinity Fabric (direct mesh) | 896 GB/s | 750 |
| MI355X analysis | 288 | 8.0 | Infinity Fabric (direct mesh) | 538 GB/s | 1,400 |
† NVIDIA's DGX B200 datasheet states ~14.3 kW for the full 8-GPU + dual-Xeon system rather than a standalone per-GPU TDP5; third-party listings commonly cite up to 1,000 W per B200 SXM, but that figure is not confirmed against an NVIDIA-hosted document in this module's research pass and is omitted from the table rather than stated as fact.
2.4 MI355X: the generational jump, at analysis-tier confidence
With that caveat stated: MI355X moves to 288 GB of HBM3e per GPU at 8 TB/s — 1.5× the capacity and 1.5× the bandwidth of MI300X — and a 5th-generation Infinity Fabric link running at 538 GB/s per chip6. If the same seven-link fully-connected topology holds (the source does not explicitly confirm the link count, only the aggregate), an 8-chip node reaches roughly 4.3 TB/s aggregate per-node fabric bandwidth (538 GB/s × 8 chips)6. Peak compute at low precision is large: 10,066 dense TFLOP/s at FP4 and 5,033 TFLOP/s at FP86. The cost of that density is power: 1,400 W TDP per chip, roughly 2.09 kW all-in once board overhead is included6 — double MI300X's 750 W3 and twice H100's 700 W2. For multi-node scale-out specifically, MI355X standardizes on RoCEv2 Ethernet via a Pollara 400GbE NIC6 — the same choice MI300X-generation platforms make, covered next.
3. Multi-node fabric: RoCEv2 via Pollara, not InfiniBand-first
Module 1 treated InfiniBand and RoCE as two options NVIDIA supports side by side, with IB the default for the highest-end fabrics (Quantum-X800) and RoCE as the Ethernet-based alternative (Spectrum-X800). AMD's own multi-node documentation flips that default: the standard scale-out fabric for both MI300X and MI355X platforms is RoCEv2 Ethernet, carried over AMD's own Pollara 400GbE NIC6. A concrete deployment example makes the shape of this concrete: AMD's own ROCm engineering blog describing a GLM-5.1-FP8 wide-expert-parallel deployment across up to 64 MI300X GPUs (§5.2) specifies eight GPU-local backend NICs per node, all carrying RDMA traffic over a RoCEv2 fabric7 — one dedicated 400 Gb/s RoCE NIC per GPU, the same rail-optimized shape Module 1 described for NVIDIA nodes.
That symmetry lets the arithmetic from Module 1 §2 transfer over unchanged. A single RoCEv2 rail at 400 Gb/s is 50 GB/s (400 ÷ 8, converting bits to bytes)6, identical to a single ConnectX-7 NDR InfiniBand rail. Summed across the standard 8-NIC-per-node configuration:
$$ 8 \times 50\ \text{GB/s} = 400\ \text{GB/s aggregate node bandwidth (all 8 rails, all-to-all traffic pattern)} $$This is the exact figure Module 1 §2 derived for an H100-generation node with 8×400 Gb/s NDR NICs. The convergence is not a coincidence: both vendors standardized on the same physical-layer generation (400 Gb/s Ethernet-class ports, whether badged InfiniBand or RoCEv2) at roughly the same point in time, and both use the same rail-per-GPU topology to reach it. The practical consequence for a wide-EP dispatch/combine kernel (§5) or a cross-node KV transfer (Module 4/7's connectors) is that the achievable aggregate bandwidth budget is the same whether the cluster is CUDA or ROCm — the difference is entirely in which library issues the RDMA verbs calls underneath, not in what the wire delivers.
RoCEv2 is a Layer-3-routable extension of the original RDMA-over-Converged-Ethernet specification, letting RDMA traffic cross IP routers the way plain InfiniBand traffic normally cannot without a gateway6. That routability is precisely why AMD's own multi-node setup guidance (§4) treats RoCE configuration as requiring more explicit operator effort than InfiniBand: IB has a well-defined subnet manager and RDMA activates automatically once the fabric is up, whereas RoCE's use of ordinary Ethernet switches means the RDMA verbs layer has to be told, by hand, which GID (Global Identifier, RoCE's IP-routable address analog) and which physical NIC to use.
4. RCCL: what transfers unchanged from NCCL, what does not
Every collective call this course has attributed to NCCL — the all-reduce inside a TP group, the all-gather feeding a decode step, the point-to-point send/recv underneath a KV-cache handoff — has a direct AMD-side counterpart: RCCL, the ROCm Collective Communication Library. RCCL is the single strongest "transfers unchanged" case in this whole module, and it is worth being precise about exactly why.
4.1 API-compatible by construction
RCCL is built to expose the same public API as NCCL: its header files (src/nccl.h.in and include/rccl/rccl.h) are generated to be a drop-in replacement, so code written against ncclAllReduce, ncclSend/ncclRecv, or any other NCCL entry point compiles and runs against RCCL without source changes8. This is why PyTorch's distributed backend, vLLM's tensor/pipeline/data-parallel plumbing, and DeepSpeed all work on ROCm with no code fork specific to the collective layer — they call NCCL's API, and RCCL answers it. RCCL implements the same collective set: all-reduce, all-gather, reduce, broadcast, reduce-scatter, gather, scatter, and all-to-all, plus point-to-point send/recv, using the same ring and tree algorithm families NCCL uses8. Transport support mirrors NCCL's shape too: RCCL optimizes for PCIe, xGMI (the technical name for the Infinity Fabric GPU-to-GPU link introduced in §2.2 — "Infinity Fabric" is the marketing name, "xGMI" is what shows up in driver and library documentation), InfiniBand Verbs, and TCP/IP sockets8.
A second dimension of the same compatibility: most of the environment variables that control NCCL's behavior work identically under the NCCL_ prefix in RCCL, not a separate RCCL_-prefixed set. NCCL_SOCKET_IFNAME, NCCL_IB_HCA, NCCL_IB_GID_INDEX, and NCCL_MIN_NCHANNELS all mean exactly what they mean under NCCL — there is no ROCm-specific renaming to learn for the variables this course already covered in Module 194. RCCL layers a small number of genuinely new, RCCL_-prefixed variables on top for AMD-specific tuning (§4.3); it does not replace the NCCL-inherited ones.
4.2 Multi-node setup: more manual steps than InfiniBand's auto-activation
Where RCCL diverges from a pure "same as NCCL" story is operational, not architectural: bringing up a multi-node RCCL job requires more explicit host-level setup than an equivalent NCCL/InfiniBand deployment, because AMD's documented reference path assumes RoCE (§3) rather than InfiniBand. AMD's own multi-node setup guide lists the required RDMA/networking package set for Ubuntu — iproute2, librdmacm-dev, rdmacm-utils, infiniband-diags, ibverbs-utils, perftest, plus the full rdma-core/libibverbs family — before any RCCL launch is attempted9. Three launch-level environment variables set the distributed topology: MASTER_ADDR (rendezvous host), NNODES (node count), and NODE_RANK (this node's rank, 0 for the master)9 — the same three variables any torchrun-launched multi-node PyTorch job already uses, unchanged by ROCm.
Below that launch layer, two RCCL-consumed variables select the physical fabric: NCCL_SOCKET_IFNAME picks the network interface for out-of-band coordination traffic (AMD's docs show both a manual example and an auto-detection one-liner built from rdma link show), and NCCL_IB_HCA lists which RDMA devices RCCL should actually move bytes over — rdma0..rdma7 for Broadcom NICs, mlx5_0..mlx5_9 for Mellanox NICs, again with an auto-detection one-liner built from ibv_devices9. The RoCE-specific step that has no InfiniBand analog is NCCL_IB_GID_INDEX=3: on RoCE, a GPU's network identity is one of several possible GIDs (RoCE's IP-routable, IB-derived addressing scheme) assigned to a NIC, and index 3 is the one AMD's guide specifies for its RoCE deployments9. AMD's documentation is explicit about the consequence of getting the RoCE path wrong: "for InfiniBand-equipped nodes, RCCL activates RDMA automatically," but for Broadcom RoCE NICs specifically, "it is crucial to install the exact same version of the RoCE library... that is installed on your host system" inside every container or environment the job runs in9 — a version-matching requirement that has no counterpart in an InfiniBand deployment, where the fabric's subnet manager handles device discovery without per-host library version pinning.
4.3 RCCL-specific additions
Layered on top of the inherited NCCL surface, AMD's own vLLM-on-ROCm performance guidance names a handful of genuinely RCCL-specific knobs: RCCL_NET_GDR_LEVEL=2 to enable GPUDirect-RDMA-class direct NIC-to-GPU transfers (paired with a note to verify NIC support, illustrated in AMD's own docs against a ConnectX-6+ card — a reminder that many ROCm clusters still run on Mellanox/NVIDIA-branded NICs rather than AMD's own Pollara parts), RCCL_DEBUG=INFO for initialization and rendezvous logging, and NCCL_MIN_NCHANNELS=112 to raise the default channel count (32–64) for tensor- and pipeline-parallel multi-GPU workloads specifically, not single-GPU inference4. AMD's multi-node checklist also flags two Kubernetes/container-level requirements this course's earlier modules did not need to name for NVIDIA: run with --distributed-executor-backend ray (vLLM's default executor for multi-node ROCm jobs, managing HIP-visible ranks and RCCL communicator creation) and ensure a shared /dev/shm across ranks — sized via Docker's --shm-size or a Kubernetes emptyDir — because RCCL's rendezvous protocol needs it4.
5. MoRI: the ROCm answer to DeepEP
Module 5 built wide expert parallelism entirely around DeepEP — NVLink-domain-to-RDMA-domain dispatch/combine kernels, FP8 dispatch with BF16 combine, EP8 NVLink bandwidth of 726/740 GB/s falling to 81–90 GB/s once dispatch crosses into the RDMA domain. The question the manifest for this module poses directly: is there a ROCm-side equivalent, or does cross-node MoE on AMD hardware have no dedicated kernel library at all? There is an equivalent, and it is young: MoRI, AMD's Modular RDMA Interface10.
5.1 What MoRI is
MoRI describes itself as a bottom-up, modular, composable framework for RDMA-and-GPU communication — explicitly positioned the way MLIR positions itself for compilers: a set of reusable building blocks rather than one monolithic library10. It splits into named sub-components, two of which map directly onto mechanisms this course has already named for the CUDA stack:
- MoRI-EP — dispatch and combine kernels for MoE all-to-all traffic, with both intra-node and inter-node (latency-optimized) variants10. This is the direct structural analog to DeepEP's normal and low-latency kernel modes from Module 5.
- MoRI-IO — point-to-point KV-cache transfer with GPU-Direct-RDMA-class low overhead10. This is what backs vLLM's
MoRIIOConnector, already named in Module 4/11's connector table as "the ROCm-only disaggregation connector" without further detail — MoRI-IO is that detail. - MoRI-SHMEM — symmetric-memory APIs for GPU-initiated communication, integrated with Triton kernels via a MoRI-IR layer10, roughly analogous in purpose to NVSHMEM on the CUDA side.
- MoRI-CCL — a lightweight collective-communication layer for latency-sensitive, customized scenarios, distinct from RCCL's general-purpose collective set10.
MoRI's dispatch/combine kernels support quantized traffic the same way DeepEP does, and the reported effect is similar in kind: AMD's own MI355X distributed-inference write-up reports FP4-dispatch-plus-FP8-combine cutting per-token dispatch/combine payload from 28,672 bytes (BF16 baseline) to 11,200 bytes, a 2.56× bandwidth reduction, with FP8-blockwise combine latency around 770 µs against 907 µs for unquantized BF1611. MoRI-EP and MoRI-IO are integrated into both SGLang and vLLM as of 2026, giving both major open-source engines a ROCm-native path for wide-EP dispatch and PD KV transfer that does not route through RCCL's general collective API1011.
5.2 A worked multi-node deployment: GLM-5.1-FP8 on up to 64 MI300X GPUs
AMD's own ROCm engineering blog documents a concrete multi-node wide-EP deployment that plays the same role for this module that Module 6 plays for DeepEP: a real, numbers-grounded production-shaped reference. The model is GLM-5.1-FP8 — 78 transformer layers (3 dense, 75 MoE), 256 routed experts with top-8 routing, Multi-head Latent Attention plus DeepSeek Sparse Attention, FP8 block-128 quantization7. The deployment scales prefill/decode-disaggregated wide-EP topologies from 2 nodes up to 8:
| Topology | Nodes | GPUs | EP degree |
|---|---|---|---|
| 1P/1D | 2 | 16 | EP8 |
| 2P/2D | 4 | 32 | EP16 |
| 2P/4D | 6 | 48 | EP16 (prefill) / EP32 (decode) |
| 4P/4D | 8 | 64 | EP32 (both) |
Peak measured throughput was 16,525 tokens/s on the 2P/4D topology at 28k input tokens and concurrency 2567. The blog's most instructive negative result is arguably more useful than the peak number: a single-instance (non-disaggregated) decode configuration collapsed to 1,405 tokens/s under the same 28k-context, 256-concurrency load — 11.8× slower than the multi-instance topologies — because KV cache capacity exhausted on one instance7. That is Module 3's disaggregation argument reproduced independently on ROCm hardware with MoRI as the transport, not RCCL alone: the deployment used a from-source vLLM build in "WideEP WRITE-mode," meaning this was a genuinely current, in-flight integration rather than a long-stable reference path7.
5.3 A maturity gap worth stating plainly
MoRI's public integration into SGLang is dated to v0.5.10+, running on ROCm 7.2, published May 202611; the GLM-5.1 wide-EP blog is dated August 20267. DeepEP, by contrast, has been in production use since DeepSeek's own December 2024 technical report (Module 6), independently reproduced by SGLang on 96 H100 GPUs (Module 6), and used across this course's entire Part 5 survey of open-source engines. Neither age nor a shorter public track record makes MoRI wrong — the GLM-5.1 numbers above are real, measured, multi-node results — but a library six months into public integration and one two years into multi-organization production validation are not the same claim of maturity — the "Where experts disagree" section below returns to this point directly.
6. vLLM-ROCm: the support-matrix view
vLLM itself is not forked for ROCm — it is the same project Modules 4, 5, and 11 already covered, with ROCm as one more backend selected by the kernels and environment variables it loads. What differs on AMD hardware is which attention/MoE kernel implementation gets selected, and which of vLLM's named multi-node connectors (Module 4/11's table) actually has a ROCm-side transport underneath it.
6.1 AITER: the kernel layer under attention and MoE
AMD's AI Tensor Engine for ROCm (AITER) supplies fused, ROCm-optimized kernels for GEMM, RMSNorm, MoE, and attention, switched on with a single master flag, VLLM_ROCM_USE_AITER=14. Underneath that flag, vLLM-ROCm selects among seven named attention backends depending on model architecture:
| Backend | Target models | Notes |
|---|---|---|
| ROCM_AITER_FA | MHA (Llama, Mistral, Qwen) | recommended default for MHA |
| ROCM_AITER_MLA | MLA (DeepSeek-V3/R1, Kimi-K2.5) | default selection for MLA |
| ROCM_AITER_MLA_SPARSE | DSA sparse-attention (DeepSeek-V3.2, GLM-5) | sparse-attention variant |
| ROCM_AITER_UNIFIED_ATTN | gpt-oss models | — |
| ROCM_ATTN | MHA fallback | Radeon / AITER-unavailable fallback |
| TRITON_MLA | MLA fallback | when AITER MLA unavailable |
| TRITON_ATTN | general fallback | — |
The measured gains from switching a MHA model onto ROCM_AITER_FA are large: 2.7–4.4× tokens-per-second over the ROCM_ATTN legacy fallback, plus a further 15–20% decode improvement at high concurrency (≥32 concurrent requests) when VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 reorders the KV cache layout for better memory access4. For MLA models, ROCM_AITER_MLA delivers 1.2–1.5× over the TRITON_MLA fallback4.
6.2 DP+EP on ROCm
The DP+EP arithmetic Module 5 and Module 11 established for CUDA — EP_SIZE = TP_SIZE × DP_SIZE, attention replicated across DP ranks at TP=1, MoE sharded across the full EP group — is unchanged on vLLM-ROCm; it is the same vLLM code path. What AMD's own guidance adds is a concurrency-dependent crossover between two configurations. At high concurrency (≥512 concurrent requests), DP+EP outperforms plain TP by 16–47%, because DP partitions the KV cache across GPUs for MLA models instead of duplicating it the way TP does4. At low concurrency (≤128 requests), the recommendation flips: TP+EP delivers 40–86% higher throughput than DP+EP, with the crossover sitting around 256–512 concurrent requests4. AMD's documented DeepSeek-R1 result at the high-concurrency end of that range is 7,114 tokens/s at 1,024 concurrent requests on an 8-GPU DP=8+EP configuration4. One further caveat AMD states explicitly: for ultra-sparse MoE models with under 1% expert activation (their example is Llama-4-Maverick-class architectures), expert parallelism should be skipped entirely — the all-to-all overhead EP adds exceeds the benefit, making EP 7–12% slower than TP alone for that model class4.
6.3 The connector list, and the one confirmed gap
Module 4 and Module 11 already listed MoRIIOConnector among vLLM's named disaggregation connectors as "the ROCm-side path," without detail on what backs it12; §5.1 supplied that detail — it is MoRI-IO. The remaining seven connectors in vLLM's list (LMCacheConnectorV1, NixlConnector, MooncakeConnector, MultiConnector, OffloadingConnector, FlexKVConnectorV1, PyNcclConnector) are, in principle, vendor-agnostic Python code paths that call into whatever collective or transfer library is installed underneath — but two of them warrant a specific caveat rather than a blanket "transfers unchanged."
NixlConnector depends on NIXL, and NIXL's own architecture documentation states its actively-supported backends as UCX and NVIDIA Magnum IO GPUDirect Storage (GDS)13. UCX itself is vendor-neutral and does support ROCm transports, but GDS is an NVIDIA-specific storage-tiering technology with no stated ROCm equivalent in NIXL's own documentation13. None of this module's sources confirm NixlConnector as a validated, supported path on ROCm clusters — which is presumably exactly why MoRIIOConnector exists as a separate, ROCm-specific connector rather than everyone simply using NixlConnector everywhere. MooncakeConnector is more clearly portable: SGLang's own PD-disaggregation documentation lists Mooncake's transfer engine as one of two selectable backends (alongside NIXL) with per-GPU InfiniBand device mapping as its only stated hardware assumption, not a CUDA-specific one14, and Mooncake's transfer engine (Module 7) is built on generic RDMA verbs rather than a CUDA-only API.
7. Economics: tokens per provisioned megawatt
Module 15's capacity-planning framework and Module 3's disaggregation-payoff disagreement both converge on the same metric being the one that actually matters for fleet sizing: not raw throughput, but throughput normalized by the power a data center actually has to provision. InferenceMAX, the same SemiAnalysis benchmark suite Module 3 and Module 15 used for the NVIDIA-side TCO crossover, publishes exactly this comparison across MI300X and MI355X on GPT-OSS 120B (MXFP4 weights) at a fixed 90 tokens/s/user interactivity target15:
Two things are worth being precise about here. First, memory capacity (§2.3's "fewer nodes fit a given MoE model" argument) and tokens-per-MW are not the same claim: MI355X's capacity/bandwidth advantage over MI300X is a modest 1.5× (§2.4), but its economic advantage at fixed interactivity is 3.4× — most of that gap comes from compute density and power efficiency at the newer process/architecture generation, not from having more HBM to hold a bigger model on fewer GPUs. Second, InferenceMAX's own broader finding from Module 3 and Module 15 — that a disaggregation TCO advantage is conditional on the interactivity target, not unconditional — is stated by the same source as depending, on the NVIDIA side, on "implementing disaggregated prefill and wide expert parallelism over Spectrum-X as well as InfiniBand"15. Nothing in this module's sources publishes an equivalent AMD-side disaggregated-prefill TCO crossover point at varying interactivity targets; the 750k/2.55M tok/s/MW figures above are single-point comparisons at one interactivity level, not a full crossover curve of the kind Module 3 §7 derived for GB200 NVL72 vs. B200.
8. What ports 1:1, what's a substitution, what's a gap
Collecting every mechanism this module and Modules 1–15 have named into one table: the middle column is the honest answer to "does this transfer," not a marketing claim in either direction.
| NVIDIA-stack mechanism | AMD/ROCm equivalent | Status |
|---|---|---|
| NVLink/NVSwitch (Module 1) | Infinity Fabric, direct 8-GPU mesh (§2.2) | substitution comparable per-GPU BW, different topology (switched vs. direct) |
| NVL72 72-GPU NVLink domain (Module 1 §5) | none in these sources | gap no >8-GPU single-fabric domain named for ROCm |
| InfiniBand NDR/XDR (Module 1) | RoCEv2 via Pollara 400GbE (§3) | substitution identical per-rail and aggregate arithmetic (§3) |
| GPUDirect RDMA (Module 1) | RDMA + RCCL_NET_GDR_LEVEL=2 (§4.3) | transfers same mechanism, different flag namespace |
| UCX (Module 1) | UCX, unchanged | transfers vendor-neutral library, no ROCm fork |
| NCCL (throughout) | RCCL (§4) | transfers API-compatible drop-in, shared NCCL_* env vars |
| DeepEP wide-EP dispatch/combine (Module 5) | MoRI-EP (§5.1) | substitution same problem shape, different library, ~2 years younger |
| NIXL (Modules 1, 8) | MoRI-IO (§5.1) for KV/PD transfer; NIXL's own GDS backend has no confirmed ROCm path | partial see “Where experts disagree” below |
| vLLM DP+EP CLI arithmetic (Modules 5, 11) | identical, same codebase (§6.2) | transfers |
| vLLM KV connectors (Modules 4, 11) | 7 of 9 vendor-agnostic + MoRIIOConnector ROCm-specific (§6.3) | transfers, with one named addition |
| Mooncake transfer engine (Module 7) | unchanged — generic RDMA verbs, no CUDA-only dependency | transfers |
| SGLang PD/router, wide-EP recipe (Module 12) | same codebase; MoRI as ROCm transfer backend (§5.2 worked example) | transfers |
| NVIDIA Dynamo (Module 9) | none in these sources | gap |
| TensorRT-LLM / Triton (Module 13) | none — CUDA-only by design | gap use vLLM-ROCm/SGLang-ROCm instead |
| llm-d / Gateway API Inference Extension (Module 10), Kubernetes orchestration (Module 14) | inference-engine-agnostic; operates above the GPU backend | transfers not verified against ROCm specifically in these sources, but architecturally backend-independent |
Where experts disagree
The material in this module supports two genuinely different readings of the same evidence, depending on which layer of the stack you weight most heavily. Both readings are grounded in real, cited numbers — they disagree about which numbers determine "production readiness."
The gap is closing fast, and the fundamentals are already equivalent
Supported by AMD's own ROCm documentation, the LMSYS/AMD MoRI announcement, and AMD's GLM-5.1 wide-EP engineering blog94117
RCCL is not an approximation of NCCL's API, it is the same API by construction, with the same NCCL-prefixed environment variables carrying over unchanged (§4.1). MoRI-EP occupies exactly DeepEP's position in the stack and reports comparable quantization-driven bandwidth reductions (§5.1). AITER's measured speedups (2.7–4.4× over legacy attention, §6.1) and DP+EP's measured 16–47% gain (§6.2) are not projected numbers, they are benchmarked ones. The GLM-5.1 deployment (§5.2) is an actual 64-GPU, 8-node, prefill/decode-disaggregated wide-EP deployment that ran and produced a throughput number — not a roadmap slide.
- Every number cited is a measured result, not a projection
- The collective-communication layer (the hardest thing to get right) is essentially solved via API compatibility
- A real multi-node wide-EP deployment exists and its failure mode (single-instance decode collapse) matches this course's own disaggregation argument
- All of the strongest evidence (MoRI, GLM-5.1 deployment) is AMD's or AMD-partner's own published material, not independently reproduced the way SGLang's DeepSeek reproduction (Module 6) independently validated DeepSeek's own numbers
- "Comparable to DeepEP" is asserted by analogy of position in the stack, not by a head-to-head benchmark against DeepEP on the same workload
The orchestration and transfer-substrate layer has a real, named gap
Grounded in what these sources do not show: no ROCm-side equivalent to NVIDIA Dynamo (Module 9) or TensorRT-LLM/Triton (Module 13) appears anywhere in this module's research, and NIXL's own architecture documentation names GDS — an NVIDIA Magnum IO technology — as one of only two actively-supported backends13
Modules 8–13 spent five modules on the fleet-orchestration and transfer-substrate layer that actually determines whether a multi-node deployment is operable at scale: SLA-driven autoscaling (Dynamo's Planner), a standardized Kubernetes Endpoint Picker protocol (llm-d/Gateway API Inference Extension), and a shared point-to-point transfer library (NIXL) that TensorRT-LLM, Dynamo, and SGLang all build on. This module found direct ROCm-side substitutes for the collective-communication and wide-EP-kernel layers, but not for the SLA-driven planning layer specifically — nothing in these sources describes an AMD-side Planner-equivalent that ingests a TTFT/ITL SLO and derives pool sizing the way Dynamo's does (Module 9). Separately, MoRI's public integration track record (ROCm 7.2, SGLang v0.5.10+, dated 2026, §5.3) is measured in months, against DeepEP's roughly two years of multi-organization production validation (Module 6).
- Grounded in a documented absence, not speculation: these specific components genuinely do not appear in the sourced material
- Distinguishes the layer that is solved (collectives, kernels) from the layer that determines actual fleet operability (SLA planning, cross-engine orchestration)
- Absence of evidence in this module's source set is not proof of absence in the field — a ROCm-side SLA planner or a NIXL ROCm backend may exist and simply not have surfaced in this research pass
- Risks overweighting orchestration-layer parity when the collective/kernel layer (§4, §5) is arguably the harder engineering problem and is the one AMD has clearly solved
Both positions are correct about different layers, and the practical read for a deployment decision is to separate them rather than average them into one "ROCm is/isn't ready" verdict. For the compute-and-collective layer — RCCL, Infinity Fabric, AITER, MoRI-EP — the evidence supports treating ROCm as a genuine substitution, not a lesser option: the numbers in §4–6 are measured, and the GLM-5.1 deployment (§5.2) proves the pieces compose end to end at real multi-node scale. For the fleet-orchestration layer — SLA-driven autoscaling, a shared cross-engine transfer substrate with GDS-class storage tiering, the kind of five-module survey Part 5 gave the NVIDIA stack — this module found no equivalent to name, which is either a genuine gap or a research-pass limitation; only a dedicated search focused specifically on ROCm orchestration tooling (outside this module's scope) would settle which. Until that search happens, the honest operational stance is: build the serving path on vLLM-ROCm or SGLang-ROCm with MoRI (both proven at multi-node scale), and expect to build SLA-driven fleet orchestration yourself rather than finding an off-the-shelf Dynamo-class layer for it.
What is changing
MoRI's SGLang integration reached v0.5.10+ on ROCm 7.2 in May 2026 and the GLM-5.1 wide-EP deployment blog followed in August 2026117 — both recent enough that a search six months from this module's as-of date is likely to find a materially larger set of validated MoRI-based multi-node deployments, and possibly vLLM's own upstream WideEP mode (currently "WRITE-mode," §5.2) reaching a stable, non-from-source release. MI355X's specs are the most likely numbers in this module to firm up: AMD's own product pages were unreachable during this research pass (§2.4), and a retry against an AMD-hosted PDF brief would either confirm or correct the analysis-tier figures used here. AMD's ROCm release cadence itself (7.2 already superseding earlier 7.x versions cited in this module's sources) means any VLLM_ROCM_* environment variable or AITER backend name in §6 should be re-checked against the ROCm version actually deployed, not assumed stable across releases.
References
- talkSmith, A., Alla, V. AMD Instinct MI300X Generative AI Accelerator and Platform Architecture. Hot Chips 2024, August 2024. https://hc2024.hotchips.org/assets/program/conference/day1/23_HC2024.AMD.MI300X.ASmith(MI300X).v1.Final.20240817.pdf. Accessed 2026-09-17. Used for: MI300X die/package specs (153B transistors, 8 XCD, 304 CUs, 192GB HBM3 @ 5.2TB/s), 8-GPU platform direct mesh topology (7×128 GB/s links, slide 13), platform-aggregate 1.5 TB @ 42.4 TB/s vs. H100 HGX's 640 GB @ 25.6 TB/s (slide 6).
- officialNVIDIA. H100 GPU (product page). 2026. https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-09-17. Used for: H100 SXM 80 GB HBM3 @ 3.35 TB/s, NVLink4 900 GB/s per GPU, 700 W TDP.
- officialLenovo. ThinkSystem AMD MI300X 192GB 750W 8-GPU Board (product guide, reproducing AMD's published MI300X specifications). https://lenovopress.lenovo.com/lp1943-thinksystem-amd-mi300x-192gb-750w-8-gpu-board. Accessed 2026-09-17. Used for: MI300X 750 W maximum TBP, 5.3 TB/s HBM bandwidth, 128 GB/s per-link/896 GB/s aggregate Infinity Fabric confirmation.
- officialAMD. vLLM V1 performance optimization. ROCm Documentation, rocm.docs.amd.com. https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html. Accessed 2026-09-17. Used for: AITER attention-backend list and measured speedups, DP+EP vs. TP+EP crossover and throughput figures, DeepSeek-R1 7,114 TPS result, RCCL_NET_GDR_LEVEL/RCCL_DEBUG/NCCL_MIN_NCHANNELS, XGMI-island TP placement guidance, multi-node checklist.
- officialNVIDIA. NVIDIA DGX B200 (datasheet). 2025. https://www.aspsys.com/wp-content/uploads/2025/05/nvidia-dgx-b200-datasheet.pdf. Accessed 2026-09-17. Used for: DGX B200 1,440 GB total GPU memory (180 GB/GPU), 64 TB/s aggregate HBM3e bandwidth (8 TB/s/GPU), 14.4 TB/s aggregate NVLink bandwidth (1.8 TB/s/GPU), ~14.3 kW system power.
- analysisSemiAnalysis / InferenceX. AMD Instinct MI355X — Specs, Pricing & AI Inference Benchmarks. 2026. https://inferencex.semianalysis.com/chips/mi355x. Accessed 2026-09-17. Used for: MI355X 288 GB HBM3e @ 8 TB/s, 538 GB/s/chip 5th-gen Infinity Fabric, 1,400 W TDP/chip, RoCEv2 via Pollara 400GbE NIC. Analysis-tier: AMD's own MI355X product pages timed out on every fetch attempt in this module's research.
- blogAMD ROCm Blogs. DI Series: Scaling GLM-5.1-FP8 to 64 MI300X GPUs. August 2026. https://rocm.blogs.amd.com/software-tools-optimization/di-glm-wideep/README.html. Accessed 2026-09-17. Used for: xPyD topology table (1P/1D through 4P/4D, 2–8 nodes), 16,525 tok/s peak throughput, single-instance decode collapse to 1,405 tok/s, 8 GPU-local RoCEv2 NICs/node, MoRI-EP/MoRI-IO usage, from-source vLLM WideEP write-mode.
- repoROCm. rccl (README.md, develop branch). 2026. https://github.com/ROCm/rccl/blob/develop/README.md. Accessed 2026-09-17. Used for: RCCL's NCCL-API-compatible header generation, supported collective set, transport support (PCIe, xGMI, InfiniBand Verbs, TCP/IP).
- officialAMD. Multi-node setup for AI workloads. ROCm Documentation, rocm.docs.amd.com. https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/system-setup/multi-node-setup.html. Accessed 2026-09-17. Used for: required RDMA/networking packages, MASTER_ADDR/NNODES/NODE_RANK launch variables, NCCL_SOCKET_IFNAME/NCCL_IB_HCA interface selection, NCCL_IB_GID_INDEX=3 for RoCE, InfiniBand auto-activation vs. RoCE library-version-matching requirement.
- repoROCm. mori (Modular RDMA Interface, README). 2026. https://github.com/ROCm/mori. Accessed 2026-09-17. Used for: MoRI's component breakdown (MoRI-EP, MoRI-IO, MoRI-SHMEM, MoRI-CCL), supported GPUs/NICs, SGLang/vLLM integration status.
- blogLMSYS Org. Win on TCO: How AMD Instinct MI355X Achieves Cost-Competitive Distributed Inference Through SGLang with MoRI. May 28, 2026. https://www.lmsys.org/blog/2026-05-28-mori/. Accessed 2026-09-17. Used for: FP4-dispatch/FP8-combine bandwidth reduction (28,672→11,200 bytes/token, 2.56×), combine latency figures, ROCm 7.2/SGLang v0.5.10+ version details, measured cost/throughput numbers.
- officialvLLM contributors. Disaggregated Prefilling. vLLM Documentation, docs.vllm.ai/en/latest/features/disagg_prefill/. https://docs.vllm.ai/en/latest/features/disagg_prefill/. Accessed 2026-09-17. Used for: the named connector list, cross-referencing MoRIIOConnector's position among the other eight connectors.
- officialNVIDIA (ai-dynamo). NIXL Architecture (nixl.md, main branch docs). 2026. https://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md. Accessed 2026-09-17. Used for: NIXL's actively-supported backends (UCX, NVIDIA Magnum IO GPUDirect Storage), the basis for the NixlConnector ROCm-gap discussion.
- officialSGLang / RadixArk. PD Disaggregation. SGLang Documentation, docs.sglang.io. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: Mooncake transfer engine as a vendor-generic RDMA-based PD backend with per-GPU InfiniBand device mapping.
- analysisSemiAnalysis. InferenceMAX: Open Source Inference Benchmarking. InferenceMAX v1, 2026. https://newsletter.semianalysis.com/p/inferencemax-open-source-inference. Accessed 2026-09-17. Used for: MI300X (750,000 tok/s/MW) vs. MI355X (2,550,000 tok/s/MW) at 90 tok/s/user interactivity on GPT-OSS 120B MXFP4; Spectrum-X/InfiniBand disaggregation framing.