Multi-Node LLM Inference: From One Node to N · Module 1

Interconnect and Transport: The Physical Layer of Multi-Node Serving

Every later module in this course — routing, disaggregation, wide expert parallelism, tiered KV offload — assumes you can compute how long it takes to move a fixed number of bytes between two GPUs, and whether those GPUs are on the same NVLink domain or not. This module builds that arithmetic from vendor-published numbers: NVLink inside one 8-GPU node, the bandwidth cliff crossing into InfiniBand or RoCE, what GPUDirect RDMA and the UCX/NIXL transport stack actually do to a byte in flight, and why a GB200 NVL72 rack is a different regime rather than nine racked 8-GPU nodes. After reading, you can take a payload size and an interconnect and produce a transfer-time estimate from first principles.

Course 25 min read 16 sources As of 2026-09-17
Bandwidth ladder from intra-node NVLink to inter-node InfiniBand, XDR, and RoCE, log scale GB/s 10 100 1,000 GB/s per link or per rail, log scale intra-node (NVLink) NVLink4, H100 (per GPU) 900 GB/s NVLink5, B200 / NVL72 (per GPU) 1.8 TB/s inter-node (scale-out fabric) — single rail InfiniBand NDR, ConnectX-7 (per port) 50 GB/s InfiniBand XDR, Quantum-X800 (per port) 100 GB/s RoCEv2, 400GbE (per port) 50 GB/s DeepEP measured, RDMA EP8×2 dispatch 90 GB/s
Figure 1. Per-link bandwidth on a log scale: two links inside an 8-GPU node (NVLink4 on H100 at 900 GB/s, NVLink5 on B200/NVL72 at 1.8 TB/s per GPU)12, against four inter-node rails — ConnectX-7 InfiniBand NDR at 400 Gb/s = 50 GB/s per port3, Quantum-X800 InfiniBand XDR at 800 Gb/s = 100 GB/s per port4, RoCEv2 over 400GbE at the same 50 GB/s per port5, and DeepEP's own measured RDMA dispatch bandwidth over two ConnectX-7 rails (EP8×2) at 90 GB/s6. A single flow that stays inside the NVLink domain runs 9–36× faster than the same flow crossing a node boundary on one rail.

You already run TP8 on 8×H100 or 8×B200 and know the mechanics of tensor-parallel all-reduce over NVLink. This section exists only to pin down the numbers every later comparison in this course is measured against — the "how fast is intra-node, exactly" baseline.

Each Hopper GPU carries fourth-generation NVLink at 900 GB/s of bidirectional GPU-to-GPU bandwidth1. An HGX H100 8-GPU baseboard reaches that bandwidth between any two GPUs through four third-generation NVSwitch chips, each switching 25.6 Tb/s (≈ 3.2 TB/s)7. The point that matters for later modules: 900 GB/s is a per-GPU figure, not an aggregate over the node, and it holds between any pair of the eight GPUs because the NVSwitch fabric gives full bisection bandwidth, not just neighbor-to-neighbor links as on pre-NVSwitch topologies.

Blackwell doubles the per-link rate: fifth-generation NVLink delivers 1.8 TB/s of GPU-to-GPU bandwidth per GPU, twice the H100 figure7. An HGX B200 8-GPU system totals 14.4 TB/s of NVLink bandwidth across the node (8 × 1.8 TB/s), switched by two NVSwitch chips2. Note the arithmetic checks out at the node level (8 × 1.8 TB/s = 14.4 TB/s) but not at the rack level once you leave the 8-GPU node — see §5 for what NVL72 changes.

Eight GPUs connected through an NVSwitch fabric inside one node, full bisection bandwidth between any pair NVSwitch fabric 4×NVSwitch3 (H100) / 2×NVSwitch5 (B200) GPU 0 GPU 1 GPU 2 GPU 3 GPU 4 GPU 5 GPU 6 GPU 7 Per-GPU bandwidth H100: NVLink4 900 GB/s B200: NVLink5 1.8 TB/s to any other GPU
Figure 2. The 8-GPU node baseline: every GPU reaches every other GPU through the NVSwitch fabric at the same per-GPU rate, 900 GB/s on H100 or 1.8 TB/s on B200127. This full-bisection property is what disappears the instant traffic has to leave the node.

2. The bandwidth cliff: leaving the NVLink domain

The instant a request needs a GPU that is not on the local NVSwitch fabric — a second replica, a decode pool on another rack, an expert that lives on another node — the transport underneath switches from NVLink to a network fabric, and the achievable bandwidth per flow drops by more than an order of magnitude. This section pins down the numbers on the other side of that boundary.

2.1 InfiniBand NDR and XDR

The standard scale-out NIC for H100/H200-generation clusters is the NVIDIA ConnectX-7, rated at a maximum total bandwidth of 400 Gb/s (= 50 GB/s) per adapter, IBTA-1.5 compliant, available in 1/2/4-port configurations, with a PCIe Gen5 host interface of up to 32 lanes3. Its RDMA message rate is 330–370 million messages per second, and it exposes hardware-based congestion control plus out-of-order RDMA with adaptive routing — both aimed at keeping a lossless fabric lossless under many-to-many collective traffic3. "Adaptive routing" and "out-of-order RDMA" are the same mechanism from two ends: the switch fabric is free to spray a flow's packets across whichever of the several physical paths between two endpoints is least congested at that instant, rather than pinning the whole flow to one static path, because the NIC on the receiving end can reassemble a message correctly even when its packets arrive out of sequence. Static routing (one flow, one path, fixed for the flow's lifetime) is simpler but lets a single congested link become a persistent bottleneck for every flow mapped onto it; adaptive routing trades a small amount of reassembly bookkeeping for load that spreads itself across the fabric automatically, which matters most exactly when many GPUs are doing an all-to-all pattern (expert dispatch, §6) that stresses every link at once.

The Blackwell-generation successor is Quantum-X800: a switch platform offering 800 Gb/s (= 100 GB/s) per port across 144 ports, for an aggregate switch capacity of 115.2 Tb/s, built for the ConnectX-8 SuperNIC generation4. NVIDIA pairs GB200 NVL72 racks with Quantum-X800 InfiniBand (or the Ethernet-based Spectrum-X800 alternative, §5) specifically for scaling a job across hundreds to thousands of Blackwell GPUs beyond the 72-GPU NVLink domain8.

2.2 RoCEv2: same PHY, different transport semantics

RoCE (RDMA over Converged Ethernet) puts the InfiniBand transport's RDMA semantics — queue pairs, kernel-bypass, zero-copy — on top of an Ethernet physical and link layer instead of a native InfiniBand one. The current, dominant revision, RoCEv2, additionally routes across Layer-3 IP networks rather than staying confined to a single Layer-2 segment, which is what makes it viable across a hyperscale data-center fabric rather than only within one rack9. NVIDIA ships ConnectX-7 in a dedicated 400GbE Ethernet SKU carrying the same silicon-level "advanced RoCE" support, accelerating NVMe-oF over both RoCE and TCP, at the same 400 Gb/s maximum bandwidth as the InfiniBand SKU5. The practical upshot: at matched NIC generation, RoCEv2 and InfiniBand deliver the same per-port bytes per second — the difference is in loss behavior, congestion-control mechanism, and operational ecosystem, not raw PHY rate (more on this in Where experts disagree).

2.3 The cliff, quantified two ways

There are two honest ways to state "how much bandwidth do you lose leaving the node," and they give very different numbers because they answer different questions.

Single flow, single rail. One GPU pair talking over one NIC each is capped at that NIC's link rate: 50 GB/s on ConnectX-7 NDR/RoCE 400GbE, 100 GB/s on Quantum-X800 XDR. Against NVLink4's 900 GB/s, that is an 18× drop; against NVLink5's 1.8 TB/s, up to 36×.

Whole node, all rails, aggregate. A rail-optimized 8-GPU node with one NIC per GPU can spread traffic across all its NICs simultaneously if the traffic pattern actually uses all eight (an all-to-all pattern does; a single point-to-point flow does not). Splitwise's own hardware-delta table gives this aggregate figure directly: an A100-generation node with 8×200Gb/s (HDR) NICs totals 200 GB/s of cross-node bandwidth, and an H100-generation node with 8×400Gb/s (NDR) NICs totals 400 GB/s10 — exactly 8 × (link-rate/8), i.e. each NIC's Gb/s figure divided by 8 to get GB/s, summed over 8 NICs. Against NVLink4's 900 GB/s per GPU, the H100-generation node aggregate of 400 GB/s is only a 2.25× drop, but that figure is only achievable when the workload actually fans traffic out across all eight rails at once (which DeepEP's all-to-all dispatch, §6, is built to do; a single KV-cache handoff between two specific GPUs is not).

Both numbers are real; which one applies depends on whether the traffic pattern is point-to-point (single-rail number) or genuinely all-to-all across the full node (aggregate number). Module 5 returns to this distinction for expert-parallel dispatch specifically.

Table 1. Interconnect generations relevant to multi-node LLM serving, per-port/per-GPU figures unless noted1234511.
InterconnectBandwidthLatency floorTypical use
NVLink4 (H100)900 GB/ssub-µs, on-package fabricIntra-node TP/PP, all-reduce
NVLink5 (B200 / NVL72)1.8 TB/ssub-µs, on-package fabricIntra-node and intra-rack (NVL72) TP/EP
InfiniBand NDR (ConnectX-7)50 GB/s≈1.7 µs (GPUDirect RDMA)H100/H200-era scale-out
InfiniBand XDR (Quantum-X800)100 GB/s≈1.7 µs (GPUDirect RDMA)GB200/GB300-era scale-out
RoCEv2 (400GbE)50 GB/s≈1.7 µs (GPUDirect RDMA)Ethernet-fabric scale-out

3. GPUDirect RDMA: bypassing the host

Every bandwidth number in §2 is a link-layer ceiling. Whether a real transfer gets close to it depends on what happens at each end of the wire — specifically, whether the payload has to detour through host (CPU) memory on its way in or out of the GPU.

3.1 The staged path vs. the direct path

Without GPUDirect RDMA, moving data from GPU A's memory to GPU B's memory over the network is a four-copy operation: GPU A copies into a pinned host staging buffer, the NIC on node A DMAs that buffer onto the wire, the NIC on node B DMAs the incoming bytes into a pinned host staging buffer, and the CPU on node B copies that buffer into GPU B's memory. Each of those four hops consumes PCIe bandwidth and adds latency, and the two host-side copies burn CPU cycles that would otherwise serve other requests. GPUDirect RDMA removes the two host copies: the NIC registers GPU memory directly (over PCIe peer-to-peer, or NVLink-C2C on Grace-Blackwell systems) and DMAs straight into or out of GPU memory, so the path becomes GPU A → NIC A → wire → NIC B → GPU B, with the CPU only involved in setting up the transfer descriptor, not moving bytes11. Both ConnectX-7 SKUs list GPUDirect RDMA as a first-class feature3, and it is the default assumption underneath NIXL, Mooncake's Messenger, and every other cross-node KV-transfer path covered later in this course.

Mechanically, GPUDirect RDMA depends on the NIC being able to issue PCIe peer-to-peer reads and writes directly against GPU memory addresses instead of only host DRAM addresses, and on those addresses being safely translatable without a host-side bounce. ConnectX-7 lists PCIe Address Translation Services (ATS) and Process Address Space ID (PASID) among its PCIe features3: ATS lets the NIC cache a device's virtual-to-physical address translations locally instead of round-tripping through the host IOMMU on every DMA, and PASID lets a single physical device (the GPU) expose multiple independent address spaces to peer devices safely. Together they are what makes "the NIC DMAs GPU memory as if it were just another PCIe-addressable region" hold up under production traffic rather than only in a point benchmark — without ATS, every peer-to-peer DMA would need a full IOMMU page-table walk, adding exactly the kind of per-transfer host-path latency GPUDirect RDMA exists to remove.

Staged (bounce-buffer) transfer path with two host copies vs. GPUDirect RDMA's direct GPU-to-GPU path Staged path (no GPUDirect RDMA) GPU A Host A pinned buf NIC A NIC B Host B pinned buf GPU B GPUDirect RDMA path GPU A NIC A NIC B GPU B no host copy on either side — NIC DMAs GPU memory directly Host-to-GPU RDMA latency floor: ≈1.7 µs (2014-era Ivy Bridge/K40 measurement)
Figure 3. The staged path copies through pinned host buffers on both ends; GPUDirect RDMA lets the NIC DMA GPU memory directly, cutting two of four hops and the associated latency11.

3.2 The latency floor and the staging crossover

NVIDIA's own microbenchmark — run on 2014-era Ivy Bridge Xeon hosts with K40m GPUs and FDR InfiniBand — measured GPUDirect RDMA host-to-GPU latency at 1.7 µs, against 1.3 µs for a plain host-to-host RDMA ping-pong, and found GPUDirect RDMA faster than the staged approach for message sizes up to roughly 400–500 KB on that hardware11. The bandwidth figures from that 2014 benchmark (single- and dual-rail GB/s, not the 50–100 GB/s per-port figures in Table 1) are roughly two orders of magnitude below what ConnectX-7/Quantum-X800-class hardware delivers today, so treat them as illustrating the mechanism, not as a current bandwidth reference. The mechanism itself still holds: GPUDirect RDMA wins on latency and CPU offload at any message size, and it wins on throughput too once you are past the point where two extra PCIe host copies would have been the bottleneck — which, at today's 50–100 GB/s per-port link rates, is almost always. For the multi-megabyte-to-gigabyte KV-cache and expert-weight payloads this course is concerned with (§8), the transfer is comfortably bandwidth-bound, and the 1.7 µs latency floor is irrelevant next to a multi-millisecond bulk transfer; it matters for small control messages (metadata handshakes, KV block-hash lookups) where a microsecond-scale RTT is the dominant cost.

4. UCX and NIXL: the transport abstraction underneath

Neither NVLink nor InfiniBand nor RoCE is something a serving engine talks to directly. Two software layers sit between the collective/point-to-point call an engine makes and the wire: UCX, a general transport library that predates LLM serving by a decade, and NIXL, a newer library purpose-built for the point-to-point transfer patterns inference serving needs. Every KV-transfer connector and every wide-EP dispatch kernel referenced later in this course is built on one or both.

4.1 UCX: UCP, UCT, UCS, UCM

UCX (Unified Communication X) is organized into four layers12:

  • UCP (Protocol) — high-level abstractions: tag-matching, streams, connection setup, multi-rail striping, and handling of different memory types.
  • UCT (Transport) — low-level primitives: active messages, remote memory access, atomic operations, mapped onto whatever hardware is present.
  • UCS (Services) — common data structures, algorithms, and system utilities.
  • UCM (Memory) — intercepts memory allocation/release events for the memory-registration cache, so repeated RDMA registration of the same buffer is cheap.

UCT's transport plugins cover RDMA (InfiniBand, RoCE, Omni-Path), Cray Gemini/Aries interconnects, GPU memory (CUDA and ROCm), several shared-memory mechanisms (POSIX, SYSV, CMA, KNEM, XPMEM), and plain TCP12 — which is precisely the set of physical layers §1–2 walked through. This is why UCX shows up underneath so many unrelated-looking projects: it is the one library that already solved "pick the fastest available path between two memory addresses, wherever they are," and everything downstream reuses that solve rather than re-implementing it per transport.

UCP's abstractions explicitly include multi-rail support12, which is the missing piece connecting §2.3's two cliff numbers. A single-rail transfer binds to one NIC and is capped at that NIC's link rate (50–100 GB/s in Table 1); reaching the node-aggregate figure (400 GB/s on an 8-rail H100-generation node) for one large transfer requires UCX to explicitly stripe that transfer's bytes across all eight rails in parallel and reassemble them at the far end, rather than relying on eight independent flows each happening to use a different rail. Whether a given transfer gets striped is a property of how the calling library (NIXL, NCCL/RCCL) issues the request, not an automatic property of having eight NICs present — which is why the single-rail number, not the aggregate, is the safer default assumption for a single point-to-point KV-cache handoff (§8) unless the calling code is known to stripe it.

NCCL itself can be pointed at UCX instead of its default ibverbs-based point-to-point transport: the NCCL-RDMA-SHARP plugin replaces NCCL's internal inter-node communication with RDMA-based transports, and setting NCCL_PLUGIN_P2P=ucx selects the UCX-backed implementation for point-to-point transport specifically, while collective transport (CollNet, including switch-based SHARP in-network reduction) is a separate code path13. RCCL (AMD's NCCL-equivalent) has an analogous UCX plugin path. The point-to-point/collective split matters: UCX and NIXL both target point-to-point transfer, which is the pattern KV-cache handoff and expert dispatch need, not the ring/tree all-reduce pattern TP/PP all-reduces need — those two traffic shapes have historically been optimized by different code paths even when they share the same underlying UCX transport layer.

4.2 NIXL: purpose-built for inference's point-to-point transfers

NIXL (NVIDIA Inference Xfer Library) is explicitly targeted at accelerating point-to-point communication in inference frameworks such as Dynamo, and it provides three abstractions on top of a transport layer like UCX14:

  • Memory Sections — unify different memory and storage types (GPU HBM, CPU DRAM, file, block, object store) behind one buffer-list primitive, so a transfer call doesn't need to know in advance whether the far end is HBM or an SSD.
  • Transfer Backend Interface — abstracts over different transfer backends and picks the one appropriate for a given source/destination pair.
  • Metadata Handler — establishes and caches agent-to-agent metadata needed to set up a transfer (addresses, registration handles) without a round-trip on every call.

NIXL's actively supported backends today are UCX and NVIDIA Magnum IO GPUDirect Storage (GDS); other filesystem, block, and object-store backends are listed as in development14. Through those backends, NIXL reaches RoCE, InfiniBand, GPUDirect RDMA, NVMe-oF, TCP, and NVLink as wire-level transport protocols14 — the same physical layers from §1–3, now exposed through one uniform transfer API instead of one API per transport. Module 8 covers NIXL's own architecture and version history in full; here the point is narrower: NIXL is the layer that makes "move this KV block from GPU HBM on node A to GPU HBM, or CPU DRAM, or an SSD, on node B" a single call regardless of which physical fabric answers it.

Layered stack: physical fabrics at the bottom, UCX as the transport-agnostic layer, NCCL/RCCL and NIXL as the two consumers above it, workloads on top NVLink · InfiniBand (NDR/XDR) · RoCEv2 · TCP · NVMe-oF UCX — UCP / UCT / UCS / UCM (transport-agnostic) NCCL / RCCL (collective: all-reduce, all-gather) NIXL (point-to-point: Memory Sections, Backend I/F) TP / PP / DP all-reduce (training, prefill) KV-cache handoff, expert dispatch, weight streaming
Figure 4. UCX abstracts every physical transport in §1–3 behind one interface; NCCL/RCCL and NIXL both build on it, but target different traffic shapes — collective reductions vs. point-to-point transfer121314.

5. NVL72: a structurally different regime

Everything in §1–4 treats "the node" as the 8-GPU unit and the cliff in §2 as the boundary at its edge. GB200 NVL72 moves that boundary from 8 GPUs to 72, and it is worth being precise about why that is a different kind of change, not just a bigger node.

5.1 Topology: 18 compute trays, 9 switch trays, one NVLink domain

A GB200 NVL72 rack is built from 18 1U compute trays, each holding 4 Blackwell GPUs, plus 9 1U NVLink switch trays — 18 × 4 = 72 GPUs total, paired with 36 Grace CPUs158. The rack-wide NVLink bandwidth is 130 TB/s8. That figure is consistent with the per-GPU NVLink5 rate from §1.2: 130 TB/s ÷ 72 GPUs ≈ 1.8 TB/s per GPU, matching the HGX B200 per-GPU figure exactly2 — the rack-scale switch fabric preserves the same per-GPU bandwidth that a single 8-GPU node gets, just across nine times as many GPUs. Memory scales with it: 13.4 TB of HBM3e across the rack at 576 TB/s aggregate bandwidth8.

5.2 Why this is not nine racked 8-GPU nodes

Nine 8-GPU nodes networked together would still have the §2 cliff at every node boundary: a GPU on node 3 reaching a GPU on node 7 pays the InfiniBand/RoCE tax (50–100 GB/s per rail) instead of the NVLink rate. NVIDIA's framing is that NVL72's switch fabric lets "all 72 GPUs act as a single GPU"7 — concretely, this means any two of the 72 GPUs communicate over the NVLink switch fabric at the full per-GPU NVLink5 rate, with no RDMA hop, regardless of which of the 18 compute trays they physically sit on. An expert-parallel or tensor-parallel group that would have needed EP/TP degree ≤ 8 to stay inside one node's NVLink domain can now use EP/TP degree up to 72 and still avoid the RDMA cliff entirely — which is exactly the condition Module 5 uses to explain why NVL72 changes viable expert-parallel degree for wide-EP MoE serving.

The cliff does not disappear; it moves to the rack boundary. Scaling a job beyond one NVL72 rack still requires leaving the NVLink domain and crossing to Quantum-X800 InfiniBand or the Ethernet-based Spectrum-X800, paired with BlueField-3 DPUs, to reach "hundreds and thousands of Blackwell GPUs"8 — i.e., the same §2 InfiniBand/RoCE numbers apply again, just one level higher in the hierarchy.

GB200 NVL72 rack: 18 compute trays of 4 GPUs each and 9 NVLink switch trays, one flat NVLink domain, with Quantum-X800/Spectrum-X800 as the boundary to the next rack GB200 NVL72 rack — one NVLink domain 4 GPU 4 GPU 4 GPU …18 trays 18×1U compute trays (4 GPU each) = 72 GPUs, 36 Grace CPUs 9×1U NVLink switch trays 130 TB/s NVLink bandwidth, rack-wide 13.4 TB HBM3e @ 576 TB/s any GPU ↔ any GPU at ≈1.8 TB/s, no RDMA hop boundary Next rack / cluster fabric Quantum-X800 InfiniBand (XDR) or Spectrum-X800 Ethernet 100 GB/s per port, §2 cliff applies again
Figure 5. Inside the rack, 72 GPUs share one NVLink switch domain at the per-GPU rate from §1; the §2 bandwidth cliff reappears only at the rack's edge, now bounding clusters of racks instead of clusters of 8-GPU nodes158.

6. DeepEP: the cliff, measured

§2's cliff numbers are link-layer ceilings. DeepEP — DeepSeek's open-source expert-parallel all-to-all communication library — publishes end-to-end measured dispatch/combine bandwidth under a realistic MoE workload, which is the closest this module gets to "what a real kernel actually achieves" rather than "what the wire is rated for." Module 5 covers DeepEP's kernel internals and placement algorithms in full; here the numbers matter only as evidence for the cliff.

6.1 Two kernel modes, two traffic regimes

DeepEP ships high-throughput, low-latency all-to-all GPU kernels for MoE dispatch and combine, with FP8-precision support6. It provides two kernel modes for two different traffic regimes6:

  • Normal kernels — optimized for asymmetric-domain bandwidth forwarding: taking data across the NVLink domain and forwarding it into the RDMA domain. This is the mode used for training and prefill, where throughput matters more than per-token latency.
  • Low-latency kernels — pure RDMA, minimizing per-call delay rather than maximizing throughput. This is the mode used for decode, where every microsecond of dispatch latency adds directly to inter-token latency.

6.2 Measured bandwidth by configuration

DeepEP's published benchmark uses 8,192 tokens per batch, 7,168 hidden dimensions, top-8-of-however-many experts routing, FP8 dispatch and BF16 combine6. Under that workload:

  • NVLink-domain dispatch/combine (SM100, EP8, intra-node): 726 / 740 GB/s at 64 SMs (the max-performance configuration).
  • RDMA-domain dispatch/combine (SM90, ConnectX-7, EP8×2 — two 8-GPU nodes' worth of experts): 90 / 81 GB/s at 12 SMs.
  • RDMA-domain dispatch/combine (SM90, EP8×4 — four nodes' worth): 61 / 61 GB/s at 6 SMs6.

726 ÷ 90 ≈ 8.1× for dispatch and 740 ÷ 81 ≈ 9.1× for combine — the manifest-level "~8× cliff" this module set out to establish, now grounded in a measured kernel rather than a link-rate ceiling, and a bit worse than the link-rate-only comparison in §2.3 because kernel and protocol overhead eat into the ceiling on both sides. Note also that EP8×4 is slower than EP8×2 in absolute GB/s despite spanning more GPUs in parallel: each rank's fixed NIC bandwidth budget has to reach more distinct remote destinations as the RDMA domain widens, so the per-destination share shrinks even though the aggregate number of experts reachable grows — a genuine trade-off between EP breadth and per-rank achieved bandwidth that Module 5's placement and load-balancing discussion returns to.

DeepEP measured dispatch and combine bandwidth by configuration: NVLink EP8 vs RDMA EP8x2 vs RDMA EP8x4 0 200 400 600 800 GB/s 726 740 NVLink EP8 (dispatch / combine) 90 81 RDMA EP8×2 (dispatch / combine) 61 61 RDMA EP8×4 (dispatch / combine)
Figure 6. DeepEP's own measured dispatch/combine bandwidth: NVLink-domain EP8 sustains 726–740 GB/s; crossing into the RDMA domain drops to 81–90 GB/s at EP8×2 and 61 GB/s at EP8×4, an approximately 8–9× cliff between the NVLink and RDMA domains6.

7. The offload hierarchy, previewed

Everything above describes moving a byte between two GPUs' HBM. Production serving systems also move KV-cache bytes to and from host DRAM, local SSD, and remote storage — a tiering hierarchy that trades capacity for bandwidth and latency at each step down: GPU HBM is fastest and smallest, host DRAM is attached over the same PCIe/NVLink-C2C link the GPU uses for everything else and is larger but far slower, local NVMe SSD is larger still and slower again, and remote storage reachable only over the network is the largest and slowest tier14. What makes this hierarchy usable at multi-node scale is exactly the machinery covered above: GPUDirect RDMA (§3) means a remote tier doesn't have to detour through a CPU copy any more than a remote GPU does, and NIXL's Memory Sections abstraction (§4.2) explicitly unifies GPU memory, CPU memory, and file/block/object storage behind one transfer call, with NVMe-oF and GPUDirect Storage (GDS) as backend protocols alongside RDMA14. This module stops at naming the tiers and the mechanism that connects them; Module 7 gives Mooncake's concrete architecture for treating the whole hierarchy as one KVCache-centric pool, and Module 8 covers LMCache's offload policy and NIXL's backend list in full, including the actual bandwidth numbers per tier.

8. Worked arithmetic: moving a KV-cache block across the fabric

Put a number on all of the above. Take a representative KV-cache transfer block using the sizing arithmetic you already use for paged KV cache — \( \text{bytes} = 2 \times L \times H_{kv} \times d \times b \times T \), for \(L\) layers, \(H_{kv}\) KV heads, head dimension \(d\), \(b\) bytes per element, and \(T\) tokens in the block. For a representative 70B-class dense model (\(L=80\), \(H_{kv}=8\), \(d=128\), BF16 so \(b=2\)) and a 2,048-token transfer block:

$$ \text{bytes} = 2 \times 80 \times 8 \times 128 \times 2 \times 2{,}048 = 671{,}088{,}640 \text{ B} \approx 671 \text{ MB} $$

A pure bandwidth-bound transfer of that block takes \( \text{bytes} / \text{bandwidth} \), ignoring the microsecond-scale GPUDirect RDMA latency floor from §3.2 (negligible against multi-millisecond bulk transfers):

Table 2. Time to move one 671 MB KV-cache block across each interconnect from this module, at the cited bandwidth12345610.
PathBandwidthTransfer time
NVLink5, single GPU pair (B200 / NVL72)1.8 TB/s0.37 ms
NVLink4, single GPU pair (H100)900 GB/s0.75 ms
DeepEP measured, NVLink EP8 dispatch726 GB/s0.92 ms
8-rail node aggregate, NDR (all-to-all only)400 GB/s1.68 ms
InfiniBand XDR, single port (Quantum-X800)100 GB/s6.71 ms
DeepEP measured, RDMA EP8×2 dispatch90 GB/s7.46 ms
DeepEP measured, RDMA EP8×4 dispatch61 GB/s11.00 ms
InfiniBand NDR / RoCEv2, single port50 GB/s13.42 ms

The spread is roughly 36× between the fastest single link (NVLink5, 0.37 ms) and the slowest single rail (NDR/RoCEv2 at one port, 13.42 ms) for the identical 671 MB payload — the same ladder as Figure 1, now expressed as wall-clock time for a payload size you'd actually see. The measured DeepEP numbers sit inside the link-rate bounds as expected, since real kernels never reach the theoretical ceiling.

Compare that to a small control message — say a 4 KB NIXL Metadata Handler exchange or a KV block-hash lookup (§4.2), not a bulk KV block. At 50 GB/s, moving 4 KB of payload takes \(4{,}096 / 5\times10^{10} \approx 0.082\ \mu\text{s}\) — roughly 21 times smaller than the ≈1.7 µs GPUDirect RDMA latency floor from §3.2. For that message, the wire time is noise and the round-trip is dominated by the fixed latency floor, not by bandwidth at all; for the 671 MB block above, the relationship inverts completely and the µs-scale latency floor is noise against a millisecond-scale bandwidth-bound transfer. Whether a given cross-node operation is latency-bound or bandwidth-bound is entirely a function of message size relative to (bandwidth × latency floor) ≈ 50 GB/s × 1.7 µs ≈ 85 KB on this hardware — below that, optimize for round-trips; above it, optimize for bytes moved. Use the calculator below to redo the bulk-transfer arithmetic for your own block size and model shape.

KV-cache block transfer time

Block size
Transfer time

Formula: \( t = \dfrac{2 L H_{kv} d\, b\, T}{\text{BW}} \). Bandwidth figures are the cited link/kernel rates from §1–6 of this module; the GPUDirect RDMA latency floor (≈1.7 µs, §3.2) is omitted as negligible at these block sizes.

Where experts disagree

Every bandwidth number in Table 1 assumes you have already decided which fabric to buy: InfiniBand or Ethernet (RoCEv2). At matched generation the two now deliver identical per-port link rates — 400 Gb/s NDR vs. 400GbE, 800 Gb/s XDR vs. 800GbE class Ethernet35 — so the argument for one over the other is no longer really about raw bandwidth. It is about transport semantics and operational fit, and the field is genuinely split.

InfiniBand is the reference fabric at extreme scale

NVIDIA's own reference architecture for GB200 NVL72 scale-out8, ConnectX-7/Quantum-X800 product positioning34

NVIDIA leads its NVL72 scale-out networking story with Quantum-X800 InfiniBand, not Spectrum-X800 Ethernet, even though it ships both8. The argument is that InfiniBand's native credit-based, lossless link-layer flow control and hardware-offloaded adaptive out-of-order RDMA routing3 avoid the packet-loss and congestion pathologies that RDMA over a fundamentally best-effort Ethernet fabric has to solve with additional mechanisms (PFC, ECN-based congestion signalling) layered on afterward.

  • Purpose-built lossless transport; no retrofit congestion-control layer
  • Hardware in-network reduction (SHARP) available for collective acceleration
  • The fabric NVIDIA itself validates first for each new GPU generation
  • Separate operational ecosystem (subnet manager, fabric tooling) from the rest of a data center's Ethernet
  • Fewer multi-vendor NIC/switch options than Ethernet

RoCEv2 Ethernet is sufficient, and preferred, at hyperscale

Meta Engineering, production GenAI training backend16

Meta built its AI training backend network on RoCEv2, not InfiniBand, and operates it at a scale of numerous clusters each with thousands of GPUs, reaching hundreds of thousands of GPUs in aggregate across data centers and regions, over 400G links with congestion-control engineering delivering measured wins such as a 40% AllReduce improvement from an enhanced ECMP scheme16. The argument is operational: a hyperscaler with an existing Ethernet fabric, tooling, and networking organization gets RDMA performance close enough to InfiniBand's without maintaining a second, InfiniBand-specific network stack end to end.

  • Reuses existing Ethernet tooling, vendors, and operational expertise
  • Demonstrated at hundreds-of-thousands-of-GPU aggregate scale in production
  • Same per-port link rate as InfiniBand at matched NIC generation
  • Congestion control (DCQCN-style ECN/PFC tuning) is harder to get right and more failure-prone under load than InfiniBand's native credit-based flow control
  • Requires more in-house networking engineering investment to reach parity
Assessment

Neither position is wrong; they answer different constraints. For a fully NVIDIA-integrated rack bought as a unit (DGX/HGX/NVL72), InfiniBand is the path of least resistance and the one NVIDIA validates first — it ships as the lead scale-out option for GB200 NVL728. For an organization that already operates a large Ethernet fabric and networking team, RoCEv2 at matched link rate is a proven production choice at very large scale, not a compromise16. The open evidence gap: there is no public, symmetric benchmark of a DeepEP-style expert-dispatch workload run identically over InfiniBand NDR/XDR and over RoCEv2 at the same link rate and cluster scale — DeepEP's own published numbers (§6) are measured on ConnectX-7 InfiniBand only6. Until that comparison exists publicly, "which fabric is faster for MoE dispatch specifically" remains an inference from separately-published numbers rather than a head-to-head measurement.

What is changing

The XDR generation is the current rollout front: Quantum-X800 switches and the ConnectX-8/800G SuperNIC generation double per-port bandwidth over NDR4, and GB200/GB300-class racks are the first to ship with it as the default scale-out option alongside NVL72's NVLink5 domain. Track the InfiniBand-vs-Ethernet balance from §Where experts disagree as a moving target rather than a settled question: it is a function of which hyperscalers publish their own backend-network choices next, not of a stable technical verdict. Within this stack specifically, expect the boundary this module draws at "8 GPUs" or "72 GPUs" to keep moving outward as NVLink domain sizes grow in future generations — each such move repeats the §5 argument (the cliff doesn't disappear, it relocates) rather than requiring a new mechanism.

References

  1. officialNVIDIA. H100 GPU (product page). n/a, 2026. https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-09-17. Used for: NVLink4 900 GB/s per-GPU bandwidth.
  2. officialNVIDIA. HGX (product page). n/a, 2026. https://www.nvidia.com/en-us/data-center/hgx/. Accessed 2026-09-17. Used for: HGX B200 NVLink5 1.8 TB/s per GPU, 14.4 TB/s node aggregate, 2×NVSwitch.
  3. officialNVIDIA. NVIDIA ConnectX-7 NDR 400G InfiniBand Adapter Card (datasheet). Apr21, 2021. https://www.nvidia.com/content/dam/en-zz/Solutions/networking/infiniband-adapters/infiniband-connectx7-data-sheet.pdf. Accessed 2026-09-17. Used for: 400 Gb/s max bandwidth, RDMA message rate (330–370M msg/s), GPUDirect RDMA, hardware congestion control, adaptive routing.
  4. officialNVIDIA. NVIDIA Quantum-X800 InfiniBand Platform (product page). 2026. https://www.nvidia.com/en-us/networking/products/infiniband/quantum-x800/. Accessed 2026-09-17. Used for: XDR 800 Gb/s per port, 144 ports, 115.2 Tb/s aggregate switch capacity.
  5. officialNVIDIA. NVIDIA ConnectX-7 400G Ethernet (datasheet). Apr21. https://www.nvidia.com/content/dam/en-zz/Solutions/networking/ethernet-adapters/connectx-7-datasheet-Final.pdf. Accessed 2026-09-17. Used for: 400GbE max bandwidth, advanced RoCE support, NVMe-oF over RoCE/TCP.
  6. repoDeepSeek-AI. deepseek-ai/DeepEP (README, main branch, v2 kernels). 2026. https://github.com/deepseek-ai/DeepEP/blob/main/README.md. Accessed 2026-09-17. Used for: normal vs. low-latency kernel modes, FP8 dispatch/BF16 combine, measured bandwidth by EP configuration (performance table).
  7. blogNVIDIA Developer Blog. NVIDIA NVLink and NVIDIA NVSwitch Supercharge Large Language Model Inference. 2026. https://developer.nvidia.com/blog/nvidia-nvlink-and-nvidia-nvswitch-supercharge-large-language-model-inference/. Accessed 2026-09-17. Used for: NVSwitch3 25.6 Tb/s per chip (4 chips, HGX H100/H200), NVLink4/NVLink5 bandwidth doubling, NVL72 "72 GPUs act as a single GPU" framing.
  8. officialNVIDIA. GB200 NVL72 (product page). n/a, 2026. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-09-17. Used for: 130 TB/s rack NVLink bandwidth, 13.4 TB HBM3e at 576 TB/s, Quantum-X800/Spectrum-X800/BlueField-3 scale-out networking.
  9. standardInfiniBand Trade Association. IBTA Announces New RoCE Specification. n/a. https://www.infinibandta.org/ibta-announces-new-roce-specification/. Accessed 2026-09-17. Used for: RoCEv2's Layer-3 routing extension over the original RoCE specification.
  10. paperPatel, P., Choukse, E., Zhang, C., Shah, A., Goiri, I., Maleki, S., Bianchini, R. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. ISCA 2024 / arXiv:2311.18677v2, 2023. https://arxiv.org/pdf/2311.18677. Accessed 2026-09-17. Used for: Table I hardware-delta figures (node-aggregate InfiniBand bandwidth, 200 GB/s A100/HDR vs. 400 GB/s H100/NDR); cross-node KV-cache transfer over back-end InfiniBand.
  11. blogNVIDIA. Benchmarking GPUDirect RDMA on Modern Server Platforms. NVIDIA Developer Blog, 2014. https://developer.nvidia.com/blog/benchmarking-gpudirect-rdma-on-modern-server-platforms/. Accessed 2026-09-17. Used for: GPUDirect RDMA mechanism, host-to-GPU latency floor (1.7 µs), staging-vs-RDMA crossover (≈400–500 KB); bandwidth figures cited only for mechanism, flagged as dated.
  12. repoopenucx/ucx. README.md. 2026. https://github.com/openucx/ucx/blob/master/README.md. Accessed 2026-09-17. Used for: UCP/UCT/UCS/UCM layer descriptions, supported transports (RDMA, TCP, shared memory, GPU).
  13. officialNVIDIA. NCCL-RDMA-SHARP Plugins (HPC-X Software Toolkit docs). 2026. https://networking-docs.nvidia.com/hpcxum/212/NCCL-RDMA-SHARP+Plugins. Accessed 2026-09-17. Used for: NCCL's UCX network plugin (NCCL_PLUGIN_P2P=ucx), point-to-point vs. CollNet/SHARP transport split.
  14. officialNVIDIA (ai-dynamo). NIXL Architecture (nixl.md, main branch docs). 2026. https://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md. Accessed 2026-09-17. Used for: Memory Sections / Transfer Backend Interface / Metadata Handler abstractions, actively supported backends (UCX, GDS), transport protocols (RoCE, InfiniBand, GPUDirect RDMA, NVMe-oF, TCP, NVLink).
  15. officialNVIDIA. Understanding Your Grace-Blackwell Systems (multi-node-nvlink-systems tuning guide docs). 2026. https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/system.html. Accessed 2026-09-17. Used for: GB200 NVL72 physical layout, 18×1U compute trays (4 GPU each), 9×1U NVLink switch trays.
  16. blogMeta Engineering. RoCE networks for distributed AI training at scale. Engineering at Meta, 2024. https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/. Accessed 2026-09-17. Used for: RoCEv2 as Meta's production GenAI training backend transport, cluster scale (thousands of GPUs per cluster, hundreds of thousands aggregate), 40% AllReduce improvement from enhanced ECMP.