Multi-Node LLM Inference: From One Node to N · Module 5

Wide Expert Parallelism: DeepEP, Placement, and Load Balancing

Tensor and pipeline parallelism move a known amount of data along a fixed pattern every step; expert parallelism moves a data-dependent amount of data to a destination the router only decides at runtime. This module works through why that distinction makes cross-node MoE an all-to-all problem, what DeepEP's two kernel modes do about it with real measured bandwidth by configuration, how EPLB decides which experts to duplicate and where to place them, and how the attention-DP + EP layout and NVL72's larger NVLink domain change the arithmetic. After it, you can reason about EP degree, placement policy, and expected dispatch bandwidth for a given cluster topology.

Course 30 min read 6 sources As of 2026-09-17 Prerequisite: Module 1
All-to-all MoE dispatch fan-out compared with ring all-reduce's fixed neighbor-to-neighbor pattern All-to-all: MoE dispatch GPU 0 GPU 1 GPU 2 GPU 3 token → whichever GPU holds its top-k expert personalized, data-dependent scatter — volume per link set by the router at runtime All-reduce: TP/PP partial sums GPU 0 GPU 1 GPU 2 GPU 3 fixed ring pattern — every rank sends/receives the same, known volume every step
Figure 1. A tensor/pipeline-parallel all-reduce moves a fixed volume along a fixed ring known at graph-compile time. MoE dispatch is a personalized all-to-all: each token's destination GPU depends on a runtime routing decision, so the bytes any given link carries change batch to batch. DeepEP implements this scatter/gather as dedicated dispatch and combine kernels1.

1. Why cross-node MoE parallelism is an all-to-all problem

Tensor parallelism and pipeline parallelism, which you already know from single-node serving, both reduce to communication patterns that are fixed by the parallelism degree and the tensor shape, not by the data flowing through the model. A TP all-reduce sums partial results that every rank already holds a same-shaped share of; a ring all-reduce over P ranks moves \( 2(P-1)/P \) times the payload per rank regardless of which tokens are in the batch, and every rank talks to the same fixed set of neighbors every step. PP send/recv is even more constrained: a fixed pair of ranks exchanges a fixed-shape activation tensor at each pipeline boundary. Both patterns are known entirely at graph-compile time — the compiler can schedule the exact communication volume and destination set before a single token arrives.

Expert-parallel dispatch does not have that property. A token's destination is the output of the router: a top-k gate computed from that token's own hidden state, decided at runtime, different for every token in the batch. If expert e lives on GPU g, every token the router sends to e has to physically move to g — and which tokens those are, and how many, is data-dependent and changes every forward pass. The result is not a reduction (many partial values combining into one) but a personalized scatter: each of the N participating ranks sends a different, batch-dependent subset of tokens to each of the other ranks, then later gathers the corresponding expert outputs back. This is the all-to-all pattern (in MPI terms, closer to Alltoallv with variable-size messages than to Allreduce), and it is why DeepSeek built DeepEP specifically as "high-throughput and low-latency all-to-all GPU kernels (MoE dispatch and combine)" rather than reusing a collective-communication library's all-reduce path1.

Two consequences follow directly from the pattern being all-to-all rather than all-reduce:

  • Volume does not shrink with degree the way all-reduce volume does. Ring all-reduce's per-rank cost falls toward \( 2 \times \text{payload} \) as P grows large and eventually plateaus; widening the ring past that point buys latency (more, smaller hops) more than it buys bandwidth relief. All-to-all dispatch has no such ceiling effect: each rank's fixed traffic budget (its own NIC/NVLink bandwidth) has to reach more distinct destinations as expert-parallel degree grows, because there are more remote GPUs each holding a piece of the expert population the router might send to. §3 quantifies this directly against DeepEP's own measured numbers.
  • The traffic pattern is irregular per step, not just per model. Even at fixed batch size and fixed EP degree, the exact byte count each link carries varies token by token and step by step, because it depends on which experts this particular batch's tokens happened to route to. A well-balanced expert population smooths this out on average; a skewed one (a handful of "hot" experts) concentrates traffic on a handful of destination GPUs. §4 covers EPLB, the mechanism DeepSeek built to keep that skew from turning into a bottleneck.

2. DeepEP: two kernel modes for two traffic regimes

DeepEP ships two distinct kernel families rather than one general-purpose all-to-all, because prefill/training dispatch and decode dispatch stress completely different things. Prefill and training move large batches and want maximum aggregate throughput; decode moves a handful of tokens per GPU per step and wants the shortest possible wall-clock time per call, even at the cost of throughput. DeepSeek's own description of the split, from the still-current documentation of the library's original (V1) architecture: normal kernels are "optimized for asymmetric-domain bandwidth forwarding, such as forwarding data from NVLink domain to RDMA domain," suitable for "both training and inference prefilling tasks," while a separate set of "low-latency kernels with pure RDMA" targets "latency-sensitive inference decoding"2. The current V2 release keeps this same two-regime design but exposes both through one unified ElasticBuffer interface rather than two separate buffer types1.

2.1 Normal kernels: NVLink-to-RDMA-domain forwarding

A normal-kernel dispatch call has to move tokens from every GPU to every GPU across a two-tier fabric: NVLink inside a node, RDMA between nodes. The naive way to do this is for the source GPU to push each token directly over RDMA to whichever remote GPU holds its destination expert — but that means, for a target node with 8 GPUs, up to 8 separate small RDMA sends per source GPU per destination node, which wastes message-rate budget on small transfers. DeepEP's normal kernels instead forward asymmetrically: a token destined for a remote node crosses the RDMA link once, to a single "gateway" rank on the destination node, then fans out to its final GPU over that node's internal NVLink fabric2. This collapses many small inter-node messages into fewer, larger ones and lets the RDMA link run closer to its rated bandwidth instead of message-rate-limited small-payload rates. Normal kernels also expose SM (streaming-multiprocessor) count as a tunable — more SMs devoted to the communication kernel raise achieved bandwidth at the cost of SMs the GEMM stages don't get to use, a trade-off §3.1 quantifies.

2.2 Low-latency kernels: pure RDMA

Decode cannot afford the gateway-forwarding hop. At a typical production decode batch (DeepSeek's own low-latency benchmark uses 128 tokens per batch, two orders of magnitude smaller than the 8,192-token prefill benchmark2), the token count routed to any single remote GPU on any given step is small enough that an extra forwarding hop adds pure latency without buying meaningful throughput headroom. Low-latency kernels instead route every token directly over RDMA, GPU to GPU, with no NVLink-domain forwarding stage. The library backs this with a hook-based communication/computation overlap mechanism: a dispatch call returns immediately with a "hook" the caller invokes only when it actually needs the received tensor, and the RDMA transfer runs in the background using zero SM occupancy — the network interface, not a GPU streaming multiprocessor, does the work of moving bytes2. That zero-SM property matters specifically because decode's compute stages (attention, the routed-expert GEMM) already run at small batch and are latency-bound; stealing SMs for communication would directly lengthen the decode step's critical path.

DeepEP normal kernels forward NVLink to RDMA via a gateway rank; low-latency kernels go direct RDMA GPU to GPU 2.1 Normal kernels (prefill/training) Node A (source) GPU src Node B (destination) gateway rank GPU dst RDMA, one hop NVLink fan-out tunable SM count — more SMs, higher bandwidth, fewer SMs left for GEMM 2.2 Low-latency kernels (decode) Node A GPU src Node B GPU dst direct RDMA, 0 SM occupancy, hook-based overlap
Figure 2. Normal kernels forward a token across the NVLink→RDMA boundary through a gateway rank on the destination node, then fan out locally over NVLink — the pattern DeepSeek describes as "asymmetric-domain bandwidth forwarding." Low-latency kernels skip the forwarding stage and send GPU-to-GPU over RDMA directly, using a hook-based callback so the transfer occupies no streaming multiprocessors2.

3. Measured bandwidth: the cliff, and what happens at small batch

Module 1 §6 already introduced DeepEP's headline normal-kernel numbers as evidence for the NVLink/RDMA bandwidth cliff. This section goes one level deeper: the SM-count trade-off behind those numbers, the separate low-latency-kernel numbers Module 1 did not cover, and a worked derivation of what those bandwidth figures mean in wall-clock time for one MoE layer's dispatch and combine step.

3.1 Normal-kernel throughput by configuration

DeepEP's own published benchmark uses a training/prefill-scale workload — 8,192 tokens per batch, 7,168 hidden dimensions, top-8 expert routing, FP8 dispatch and BF16 combine1 — and reports:

  • NVLink-domain, EP8 (single 8-GPU node), 64 SMs devoted to communication: 726 GB/s dispatch, 740 GB/s combine.
  • NVLink-domain, EP8, only 24 SMs devoted to communication: 643 GB/s dispatch, 675 GB/s combine — a 60% cut in communication SMs costs about 11–9% of achieved bandwidth1.
  • RDMA-domain, EP8×2 (two 8-GPU nodes' worth of experts), 12 SMs: 90 GB/s dispatch, 81 GB/s combine.
  • RDMA-domain, EP8×4 (four nodes' worth), 6 SMs: 61 GB/s dispatch, 61 GB/s combine1.

The 24-versus-64-SM comparison is the concrete version of the trade-off §2.1 named: a serving engine that needs those SMs for GEMM work can drop from 64 to 24 communication SMs and give up only about a tenth of dispatch bandwidth, which is often the right trade at prefill scale where compute is the bottleneck resource. DeepEP's own release notes flag this explicitly as a benefit of its second major version, reporting "up to 1.3x peak performance" over the prior version "while saving up to 4x SM count"1. The library's documentation also flags a measurement caveat worth carrying forward: the reported numbers are logical bandwidth, and "under the EP 8 x 2 case, 90 GB/s actually contains local rank traffic" — meaning part of that figure reflects the intra-node NVLink hop from §2.1's gateway-forwarding step, not purely the cross-node RDMA wire rate1.

DeepEP normal-kernel measured dispatch and combine bandwidth by EP configuration and SM budget 0 200 400 600 GB/s 726 740 NVLink EP864 SM 643 675 NVLink EP824 SM 90 81 RDMA EP8×212 SM 61 61 RDMA EP8×46 SM
Figure 3. DeepEP normal-kernel measured bandwidth (dispatch, then combine) at 8,192 tokens/batch, top-8 routing, FP8 dispatch/BF16 combine. NVLink EP8 sustains 643–740 GB/s across both SM budgets tested; crossing into RDMA drops to 81–90 GB/s at EP8×2 and 61 GB/s at EP8×4, and achieved bandwidth per rank falls further as the RDMA domain widens even though more experts become reachable1.

3.2 Low-latency-kernel latency and bandwidth by EP degree

The normal-kernel table above is a prefill-scale measurement; DeepEP's documentation of its low-latency kernels reports a separate benchmark at decode scale — 128 tokens per batch (versus 8,192 for the normal-kernel table), 7,168 hidden dimensions, top-8 experts, FP8 dispatch and BF16 combine, on H800 with ConnectX-7 400 Gb/s (≈50 GB/s per-port maximum) RDMA2. Because the traffic is small-message and latency-bound rather than throughput-bound, the library reports both a per-call latency in microseconds and an achieved bandwidth in GB/s:

Table 1. DeepEP low-latency kernel measured latency and RDMA bandwidth by EP degree, 128 tokens/batch, H800 + ConnectX-72.
EP degreeDispatch latency (µs)Dispatch BW (GB/s)Combine latency (µs)Combine BW (GB/s)
87798114127
161186319574
321554827353
641734331446
1281923936939
2561943936040

Two things stand out against the §3.1 normal-kernel table. First, dispatch latency roughly triples between EP8 and EP128 (77 µs → 192 µs), but the growth is sharply sublinear in EP degree — going from 128 to 256 ranks adds essentially nothing (192 µs → 194 µs), because at large EP degree the per-destination payload shrinks (fewer tokens per remote rank at a fixed 128-token batch) and RDMA message-rate/round-trip latency, not bandwidth, dominates. Second, achieved bandwidth falls monotonically with EP degree in both tables, for the same underlying reason §1 named structurally: a fixed per-rank NIC budget has to reach more distinct remote destinations as the domain widens, so each individual flow's share of that budget shrinks even as the aggregate reachable-expert count grows.

DeepSeek's own architecture notes tie the two kernel modes directly to the two serving phases this course's Module 3 distinguishes: normal kernels are the right tool "for both training and inference prefilling," low-latency kernels for "inference decoding"2 — the prefill/decode split is not incidental to which DeepEP kernel a serving engine calls, it determines it.

3.3 Worked derivation: dispatch and combine payload for one MoE layer

The GB/s figures in §3.1 are useful for comparing configurations, but a concrete wall-clock number for a single MoE layer's dispatch and combine step makes the FP8/BF16 asymmetry from §2 tangible. Take DeepEP's own normal-kernel benchmark parameters1: 8,192 tokens per batch, hidden dimension 7,168, top-8 routing.

Dispatch moves one hidden-state vector per (token, chosen-expert) pair, quantized to FP8 (1 byte/element):

$$ \text{dispatch bytes} = T \times k \times H \times 1\ \text{B} = 8192 \times 8 \times 7168 \times 1\ \text{B} \approx 469.8\ \text{MB} $$

Combine moves the corresponding expert outputs back, kept in BF16 (2 bytes/element) rather than FP8:

$$ \text{combine bytes} = T \times k \times H \times 2\ \text{B} = 8192 \times 8 \times 7168 \times 2\ \text{B} \approx 939.5\ \text{MB} $$

Dividing each payload by the §3.1 measured bandwidths gives the time one dispatch or combine call takes at each configuration (these are computed from the stated parameters and measured bandwidth, not numbers DeepEP publishes directly as latencies):

Table 2. Computed dispatch/combine time for one MoE layer at DeepEP's own benchmark parameters (8,192 tokens, top-8, hidden 7,168), derived from the §3.1 measured bandwidths1.
ConfigurationDispatch BWDispatch time (≈470 MB)Combine BWCombine time (≈940 MB)
NVLink EP8 (64 SM)726 GB/s0.65 ms740 GB/s1.27 ms
RDMA EP8×290 GB/s5.22 ms81 GB/s11.60 ms
RDMA EP8×461 GB/s7.70 ms61 GB/s15.40 ms

Combine consistently takes longer than dispatch at the same configuration — roughly 2× at NVLink and RDMA EP8×2, converging closer to 2× at EP8×4 — and that ratio is exactly the FP8-versus-BF16 byte-width ratio, not a difference in achieved bandwidth (combine bandwidth is close to dispatch bandwidth at every configuration in §3.1). The precision asymmetry is deliberate: dispatch delivers a raw hidden-state vector to an expert's GEMM, which is already going to run in FP8 on current DeepSeek-class deployments, so quantizing before the network hop costs no additional precision the GEMM wasn't already going to discard. Combine, in contrast, sums up to k already-independently-quantized expert outputs weighted by the router's gate values — doing that reduction in FP8 would compound rounding error across as many as 8 summed terms, so it happens in BF16 despite doubling the bytes moved12.

4. EPLB: redundant experts and placement

§1 named the second consequence of all-to-all dispatch: even a well-designed router does not guarantee an even load across experts. Some experts specialize in common patterns and get routed to far more often than others; at any fixed EP degree, if expert placement is naive (expert i always on GPU i mod N), a hot expert makes its host GPU the bottleneck for every step, regardless of how much idle capacity sits on the GPUs holding cold experts. EPLB (Expert-Parallelism Load Balancer) is DeepSeek's open-sourced answer: rather than changing the router, it changes where experts live and how many copies of each exist.

4.1 The redundant-experts strategy

EPLB's core mechanism, as its own documentation states it: "a redundant experts strategy that duplicates heavy-loaded experts," followed by heuristically packing "the duplicated experts to GPUs to ensure load balancing across different GPUs"3. Concretely: given a per-expert load estimate (the library takes the load matrix as an input and is explicit that predicting those loads — typically a moving average of recent routing statistics — is the caller's responsibility, not EPLB's3), EPLB decides how many physical replicas each logical expert gets and which GPU each replica lives on, so that the sum of load assigned to each GPU is as even as possible. A hot expert might get two or three physical copies spread across different GPUs; a cold expert gets exactly one. This only works because the router itself is unaware of the duplication — from the router's perspective it picks a logical expert, and a separate lookup resolves that to whichever physical replica the placement plan assigned for the current step.

A second mechanism compounds with replication: group-limited expert routing. DeepSeek-V3's router constrains each token to choose experts from only a limited number of expert groups rather than the full expert population, and EPLB exploits that constraint by trying "to place the experts of the same group to the same node to reduce inter-node data traffic, whenever possible"3. If a token's routing choices are confined to one or two groups, and every expert in those groups lives on the same node, then a large share of that token's dispatch traffic never has to leave the node — it stays on the §3.1 NVLink-domain bandwidth tier instead of falling to the RDMA tier.

4.2 Hierarchical versus global load balancing

EPLB ships two distinct packing policies, and which one applies is a direct function of expert-parallel degree, not a free choice:

  • Hierarchical load balancing applies "when the number of server nodes divides the number of expert groups"3. The algorithm runs in three stages: pack whole expert groups onto nodes evenly (balancing load at the node level while keeping every group intact on one node), then replicate experts within each node, then pack the replicated experts onto that node's individual GPUs. Because it preserves the group-locality property from §4.1, DeepSeek recommends this policy "in prefilling stage with a smaller expert-parallel size"3 — prefill EP degrees are typically small enough that the node-count-divides-group-count condition holds.
  • Global load balancing is the fallback "in other cases": it "replicates the experts globally regardless of expert groups, and pack[s] the replicated experts to individual GPUs"3, giving up the group-locality guarantee in exchange for a placement that works at any EP degree. DeepSeek's own guidance is that this policy is "adopted in decoding stage with a larger expert-parallel size"3 — at decode-scale EP degrees, the node-count/group-count divisibility condition hierarchical balancing needs is less likely to hold, and the priority shifts from minimizing inter-node traffic to simply keeping every GPU's load even.

4.3 A worked placement, straight from EPLB's own example

EPLB's repository ships a literal worked example that is small enough to trace by hand and shows exactly what the hierarchical policy's three stages produce. The setup: a two-layer MoE model, 12 experts per layer, 4 redundant experts introduced per layer (16 physical replicas total), placed across num_groups=4 expert groups, num_nodes=2, num_gpus=8 (4 GPUs per node)3. The call eplb.rebalance_experts(weight, num_replicas=16, num_groups=4, num_nodes=2, num_gpus=8) takes a per-expert load matrix and returns phy2log, the physical-slot-to-logical-expert mapping3.

For layer 0, with logical experts 0–11 split into four groups of three ({0,1,2}, {3,4,5}, {6,7,8}, {9,10,11}) and load weights [90, 132, 40, 61, 104, 165, 39, 4, 73, 56, 183, 86], the returned placement puts logical experts {3, 4, 5} and {6, 7, 8} — groups 1 and 2 — entirely on node 0's four GPUs, and logical experts {0, 1, 2} and {9, 10, 11} — groups 0 and 3 — entirely on node 1's four GPUs3. No group is split across the node boundary. Within that constraint, the heaviest-loaded experts get duplicated: logical experts 1 (weight 132), 4 (weight 104), 5 (weight 165), and 10 (weight 183) — the four highest-load experts in the layer — each receive a second physical copy, which is exactly the 4 redundant experts the setup specified3.

EPLB hierarchical placement: expert groups packed whole onto nodes, heaviest experts duplicated across two different GPUs within that node layer 0, 12 logical experts, 4 groups, 4 redundant — EPLB's own worked example, actual phy2log output Node 0 — groups 1 {3,4,5} + 2 {6,7,8} GPU 0 e5 (165) e6 (39) GPU 1 e5’ dup e7 (4) GPU 2 e8 (73) e4 (104) GPU 3 e4’ dup e3 (61) e5, e4 each duplicated — once per copy, on two different GPUs no expert from groups 0 or 3 appears on node 0 Node 1 — groups 0 {0,1,2} + 3 {9,10,11} GPU 4 e10 (183) e9 (56) GPU 5 e10’ dup e2 (40) GPU 6 e0 (90) e1 (132) GPU 7 e11 (86) e1’ dup e10, e1 each duplicated — once per copy, on two different GPUs no expert from groups 1 or 2 appears on node 1
Figure 4. The hierarchical policy applied to EPLB's own README example (phy2log output, layer 0): groups {1,2} pack entirely onto node 0, groups {0,3} entirely onto node 1, and the four heaviest-loaded logical experts in the layer — e1 (132), e4 (104), e5 (165), e10 (183) — each get a second physical copy placed on a different GPU from the first, spreading their load rather than doubling it on one card. A token constrained to one group by group-limited routing never needs an inter-node hop to reach any of its candidate experts3.

4.4 Memory overhead

Redundancy is not free: every duplicate expert is a full copy of that expert's weights resident in HBM on top of the GPU's regular allocation. vLLM's own EPLB integration documentation quantifies this for a concrete model: roughly 2.4 GB per redundant expert per EP rank on DeepSeek-V34. That number sets a hard planning constraint — a deployment that wants headroom for, say, 8 redundant experts per rank to absorb load skew has to budget roughly 19 GB of HBM per GPU for that headroom alone, on top of the base per-GPU expert allocation, before a single activation or KV-cache byte is counted.

5. Attention-DP + EP: the general layout pattern

§1–4 established the dispatch/combine mechanism and the placement problem it creates. This section covers the layout decision every wide-EP deployment has to make: what happens to the attention layers while the MoE layers are sharded across an EP group.

5.1 Why attention goes data-parallel

Under plain tensor parallelism, every layer — attention and MoE alike — shards across the same TP group. That is a reasonable default for a dense model, but it is a poor fit once MoE layers need to shard across an EP group that is much wider than attention needs to be: attention's compute and KV-cache footprint scale with sequence length and head count, not with expert count, so forcing attention to shard across a 32- or 64-wide EP group buys it nothing and adds all-reduce overhead on every attention block. vLLM's own documentation states the alternative directly: with tensor-parallel size set to 1, "attention weights remain replicated across all DP ranks," while the "Expert (MoE) Layers" are "sharded across all EP ranks" using the full expert-parallel group4. Each data-parallel rank runs its own complete, unsharded copy of the attention stack against its own independent slice of the request batch and its own private KV cache; only at the MoE layer boundary does a token leave its DP rank, cross the §1–3 all-to-all fabric to reach its routed expert, and come back.

The relationship between the three parallelism dimensions is arithmetic, and vLLM states it as such: expert-parallel size is computed, not set directly — "EP_SIZE = TP_SIZE × DP_SIZE"4. Setting tensor-parallel size above 1 does not disappear from the picture; it changes what attention does instead of what MoE does: at TP > 1, "attention weights are sharded using tensor parallelism across TP ranks within each DP group," while MoE layers still span the full TP × DP membership4. vLLM's own worked example is TP=2, DP=4 on 8 GPUs: "Expert layers form an EP group of size 8, with experts distributed across all GPUs. Attention layers use TP=2 within each of the 4 DP groups"4 — the same 8 physical GPUs simultaneously belong to one 8-wide EP group and to four separate 2-wide attention TP groups.

Attention-DP plus expert-parallel layout: attention TP groups nested inside DP ranks, MoE sharded across the full EP group spanning all GPUs vLLM worked example: TP=2, DP=4, 8 GPUs — EP_SIZE = 2 × 4 = 8 Attention layer: 4 DP groups, TP=2 each DP 0 g0 g1 DP 1 g2 g3 DP 2 g4 g5 DP 3 g6 g7 each GPU: full, replicated attention weights + private KV cache MoE layer: one EP group of size 8, all GPUs g0 g1 g2 g3 g4 g5 g6 g7 one expert shard per GPU, EP degree 8 all-to-all dispatch/combine crosses DP-group boundaries only at the MoE layer
Figure 5. vLLM's TP=2/DP=4/8-GPU example: attention runs as four independent 2-way tensor-parallel groups, each with its own replicated request stream and KV cache; MoE layers ignore that grouping and shard as one 8-wide expert-parallel group spanning every GPU, so EP_SIZE = TP_SIZE × DP_SIZE = 2 × 4 = 84.

5.2 vLLM's multi-node CLI arithmetic

vLLM exposes this layout entirely through flags, and a real two-node example from its own documentation makes the arithmetic concrete. The primary node launches with the total DP size, its own local share, and networking coordinates; the secondary node repeats the total but adds a start-rank offset and runs headless (no API server, worker ranks only)4:

vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --all2all-backend deepep_low_latency \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --data-parallel-address 192.168.1.100 \
    --data-parallel-rpc-port 13345 \
    --api-server-count=8
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --all2all-backend deepep_low_latency \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --data-parallel-start-rank 8 \
    --data-parallel-address 192.168.1.100 \
    --data-parallel-rpc-port 13345 \
    --headless

Both nodes agree on --data-parallel-size 16 (the global total); each reports --data-parallel-size-local 8 (its own share); node 2's --data-parallel-start-rank 8 tells the cluster its 8 local ranks are global DP ranks 8–15, "equal to the cumulative local DP size of previous nodes"4. With --tensor-parallel-size 1 on both nodes, EP_SIZE = 1 × 16 = 16: the MoE layers shard across all 16 GPUs on both nodes as one EP group, while attention runs fully replicated per DP rank — this is the low-latency, decode-oriented case from §2.2, signaled explicitly by --all2all-backend deepep_low_latency.

Table 3. vLLM's two-node DP+EP flags mapped to the resulting physical layout, from its own documented example4.
FlagNode 1 valueNode 2 valueEffect
--tensor-parallel-size11Attention fully replicated per DP rank, no attention sharding
--data-parallel-size1616Total DP ranks cluster-wide (= EP_SIZE, since TP=1)
--data-parallel-size-local88DP ranks hosted on this node
--data-parallel-start-rank0 (default)8Node 2's local ranks 0–7 map to global ranks 8–15
--headlessabsentsetOnly node 1 runs the API server; node 2 is worker-only

5.3 SGLang's --enable-dp-attention

SGLang implements the same general pattern under a differently named flag: --enable-dp-attention activates data-parallel attention so that, as in vLLM's TP=1 case, each DP rank runs attention against its own request slice and private KV cache while MoE layers dispatch across the wider expert-parallel group via a configurable all-to-all backend (--moe-a2a-backend deepep selects DeepEP specifically)5. The mechanism is the one this section already covers in full; the concrete multi-node deployment recipe — specific TP/DP degrees, separate prefill and decode master nodes, and how SGLang wires the flag into a running cluster — belongs to Module 12, not here.

6. What NVL72 changes about the EP domain

Every dispatch/combine number in §3 crosses from the NVLink domain to the RDMA domain at exactly the 8-GPU boundary, because that is where a conventional HGX/DGX node ends. Module 1 §5 established that GB200 NVL72 moves that boundary out to 72 GPUs: the rack's NVLink switch fabric delivers 130 TB/s of aggregate bandwidth across all 72 GPUs and 36 Grace CPUs, and any two of those 72 GPUs reach each other over the switch fabric at the full per-GPU NVLink5 rate with no RDMA hop, regardless of which of the rack's compute trays they sit on6. For wide expert parallelism specifically, that changes which EP degrees pay the §3.1 cliff and which do not.

On a conventional 8-GPU-node cluster, any EP degree above 8 necessarily spans a node boundary and falls onto the RDMA tier — EP16 measured at 90 GB/s dispatch, EP32 at 61 GB/s, both roughly an order of magnitude below the EP8 NVLink-domain figure of 726 GB/s1. Inside one NVL72 rack, that same EP16 or EP32 group never leaves the NVLink domain: it is still 16 or 32 GPUs out of the rack's 72, all reachable over the same switch fabric that gives any pair of GPUs the full per-GPU NVLink5 rate. An expert-parallel group that would have needed to stay at EP ≤ 8 to avoid the RDMA cliff can run at EP32, EP64, or the full EP72 and still avoid it entirely — the constraint that forces EP degree down to node size on conventional clusters simply does not apply inside one NVL72 rack.

The cliff is not eliminated, only relocated. Scaling an EP group past 72 GPUs — more experts than one rack can host, or an EP group deliberately spread across racks for other reasons — still means crossing to Quantum-X800 InfiniBand or Spectrum-X800 Ethernet at the rack's edge6, and paying a version of the same §3.1 RDMA-tier bandwidth penalty, just one level higher in the hierarchy than the conventional 8-GPU case.

Table 4. EP domain size against interconnect tier and the resulting dispatch bandwidth regime, on a conventional 8-GPU-node cluster versus inside one NVL72 rack16.
EP degreeOn 8-GPU-node clusterMeasured dispatch BWInside one NVL72 rack (72 GPUs)
EP8Intra-node, NVLink726 GB/sIntra-rack, NVLink (identical tier)
EP162 nodes, RDMA EP8×290 GB/sIntra-rack, NVLink — no RDMA hop
EP324 nodes, RDMA EP8×461 GB/sIntra-rack, NVLink — no RDMA hop
EP729 nodes, RDMA (not published at this width)—Intra-rack, NVLink — entire rack, no RDMA hop

The NVL72 column states a structural fact (no RDMA hop is required, because the destination GPUs sit inside the same 130 TB/s switch fabric6), not a measured DeepEP figure — no public DeepEP benchmark run inside an NVL72 rack at these EP degrees was found for this module; the achieved GB/s on such a rack, as opposed to the topological absence of an RDMA hop, remains unmeasured in public sources as of this writing.

Where experts disagree

§3.1 already flagged a tension in DeepEP's own numbers: RDMA-domain achieved bandwidth per rank falls as EP degree widens (90 GB/s at EP8×2 down to 61 GB/s at EP8×4), even though the aggregate number of experts a wider EP group can reach only grows. That tension sits underneath a genuine design question every wide-EP deployment has to answer: how wide should the expert-parallel group actually be?

Widen EP as far as the interconnect allows

vLLM's deployment guidance and EPLB's own policy design43

vLLM states plainly that expert parallelism "is more efficient when used in conjunction with DP"4, and its arithmetic (EP_SIZE = TP_SIZE × DP_SIZE) means scaling data parallelism for throughput automatically widens EP alongside it. EPLB's own two-policy design points the same direction for decode specifically: the global load-balancing policy, which gives up group-locality entirely, is the one DeepSeek recommends "in decoding stage with a larger expert-parallel size"3 — the library's own authors built a policy whose explicit purpose is to work well at wide EP. A wider EP group also means fewer experts resident per GPU, which at fixed total expert count leaves more HBM headroom for the §4.4 redundant-expert copies that absorb load skew, and a larger effective batch of tokens landing on each remaining expert per decode step, which is the difference between an expert's GEMM running at a batch size that uses the GPU well and one that does not.

  • Smaller per-GPU expert count leaves more HBM for redundant-expert replicas
  • Larger per-expert token batch at fixed total decode batch size, better GEMM utilization
  • The tooling (EPLB's global policy, vLLM's DP+EP arithmetic) is explicitly built for this regime
  • Requires more physical GPUs able to reach each other at usable bandwidth — only fully solved inside one NVLink domain (§6)
  • Every widening step, once past the NVLink domain, crosses into a slower RDMA tier

Widening EP has a real, measured communication cost

DeepEP's own published benchmark numbers1

Read §3.1's numbers as a trend rather than two isolated data points: RDMA-domain dispatch bandwidth drops from 90 GB/s at EP8×2 to 61 GB/s at EP8×4, a roughly one-third reduction in per-rank achieved throughput from doubling the RDMA domain width. The §3.2 low-latency table shows the same direction even more sharply — dispatch bandwidth falls from 98 GB/s at EP8 to 39 GB/s at EP128, an approximately 2.5× drop across a 16× increase in EP degree, while dispatch latency more than doubles (77 µs to 192 µs). A fixed per-rank NIC budget spread across more remote destinations is a physical constraint, not a tuning problem DeepEP's SM-count knob (§3.1) can fully compensate for once the domain has left NVLink. For a decode workload with a hard inter-token-latency SLO, that latency growth — not the GB/s figure — is often the number that matters, and it argues for capping EP degree at whatever the NVLink or lowest-hop RDMA domain will bear.

  • Grounded directly in DeepEP's own measured numbers, not a general argument
  • Latency growth is the metric that actually gates a fixed decode SLO, and it is monotonic with EP degree
  • Capping EP degree caps how many redundant-expert replicas and how large a per-expert batch a deployment can support
  • Ignores whatever a specific interconnect topology (NVL72, §6) can do to move the cliff outward
Assessment

Neither position is wrong on its own terms; they are optimizing different things. Batch-efficiency and redundancy headroom (position one) matter most when the deployment is throughput-bound and has SLO slack; the measured communication cost (position two) matters most when the deployment is a hard-latency decode path where every added microsecond of dispatch time shows up directly in inter-token latency. §6's NVL72 finding partially resolves the tension for one specific case: inside a single NVL72 rack, EP degrees up to 72 avoid the RDMA-tier penalty position two is built on, so the trade-off collapses back toward position one as long as the deployment fits inside one rack. Outside that regime — EP degrees that must cross node or rack boundaries on conventional hardware — the trade-off is real and has to be decided per deployment. Module 6 works through one production system's own resolution of exactly this trade-off, observed at two different points in time, without assuming either position was simply a mistake.

What is changing

DeepEP itself is mid-refactor as of this writing. The library's second major version unifies what were previously two separate buffer types (a standard Buffer for normal kernels, a separate low-latency-mode buffer) into a single ElasticBuffer interface covering both regimes, switches its underlying transport from NVSHMEM to what the project calls an "NCCL Gin backend," and extends supported expert-parallel scale-out to "up to EP2048"1 — two orders of magnitude past the EP8×4 configuration this module's §3.1 table measures. The same release claims up to 1.3× the prior version's peak performance while using up to 4× fewer SMs for communication1, which would shift the §3.1 SM-versus-bandwidth trade-off further in favor of leaving SMs for compute.

One regression is worth flagging precisely because it affects this module's own sourcing: the current release notes state "0 SM RDMA low-latency EP is no longer supported"1, and the project's current main documentation no longer publishes a standalone low-latency-kernel benchmark table with the microsecond-level detail §3.2's numbers came from — those numbers are sourced from the project's own archived V1 documentation, which remains hosted in the same repository but is explicitly marked legacy2. Whether the V2 ElasticBuffer interface reproduces, improves on, or regresses those specific latency numbers has not been separately published as of 2026-09-17; anyone deploying against a current DeepEP checkout should treat §3.2's figures as the last publicly documented low-latency baseline, not a live V2 measurement.

References

  1. repoDeepSeek-AI. deepseek-ai/DeepEP (README, main branch, V2). 2026. https://github.com/deepseek-ai/DeepEP/blob/main/README.md. Accessed 2026-09-17. Used for: all-to-all dispatch/combine kernel framing, FP8 dispatch/BF16 combine, normal-kernel performance table (726/740 GB/s NVLink EP8 at 64 and 24 SMs, 90/81 GB/s RDMA EP8×2, 61/61 GB/s RDMA EP8×4), "logical bandwidth" caveat, V2 ElasticBuffer unification, EP2048 scale-out, SM-count savings, "0 SM RDMA low-latency EP no longer supported."
  2. repoDeepSeek-AI. deepseek-ai/DeepEP (docs/legacy.md, V1 archived documentation). 2026. https://github.com/deepseek-ai/DeepEP/blob/main/docs/legacy.md. Accessed 2026-09-17. Used for: normal-vs-low-latency kernel mode descriptions (NVLink-to-RDMA-domain forwarding vs. pure RDMA), hook-based zero-SM communication/computation overlap, low-latency kernel benchmark table (77–194 µs latency, 39–127 GB/s by EP degree 8–256, 128 tokens/batch on H800 + ConnectX-7).
  3. repoDeepSeek-AI. deepseek-ai/EPLB (README, main branch). 2026. https://github.com/deepseek-ai/EPLB/blob/main/README.md. Accessed 2026-09-17. Used for: redundant-experts strategy, group-limited routing and node-local group placement, hierarchical vs. global load-balancing policy definitions, worked placement example (2-layer, 12 experts/layer, 4 redundant, num_groups=4/num_nodes=2/num_gpus=8, weight matrix and phy2log output), rebalance_experts interface.
  4. officialvLLM Project. Expert Parallel Deployment (vLLM Documentation). 2026. https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/. Accessed 2026-09-17. Used for: EP_SIZE = TP_SIZE × DP_SIZE, attention-DP + EP layout (TP=1 replicated attention vs. TP>1 sharded attention within DP groups), TP=2/DP=4/8-GPU worked example, two-node CLI flags and worked example (--data-parallel-size, --data-parallel-size-local, --data-parallel-start-rank, --headless), EPLB integration parameters (window_size 1000, step_interval 3000), ~2.4 GB per redundant expert per EP rank on DeepSeek-V3.
  5. officialSGLang Contributors. PD Disaggregation (SGLang Documentation). 2026. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: --enable-dp-attention and --moe-a2a-backend deepep as SGLang's implementation of the attention-DP + EP pattern.
  6. officialNVIDIA. GB200 NVL72 (product page). 2026. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-09-17. Used for: 72-GPU/36-Grace-CPU rack configuration, 130 TB/s rack-wide NVLink bandwidth, Quantum-X800/Spectrum-X800 as the rack-boundary scale-out fabric.