Multi-Node LLM Inference: From One Node to N · Module 13
TensorRT-LLM and Triton: Disaggregated Serving on GB200
TensorRT-LLM ships disaggregated serving as three separate deployment paths built around one shared mechanism, the request-ID-keyed cache-transceiver, and NVIDIA's own GB200 benchmarks show that mechanism's payoff swinging from 1.4× to 6.11× purely as a function of model and sequence-length ratio. This module extends the cache-transceiver mechanics from Module 4 with TensorRT-LLM-specific backend and scaling detail, reads those GB200 numbers against NVL72's rack-scale NVLink domain from Module 1, and closes Part 5 by placing TensorRT-LLM/Triton next to the other four stacks this part covered.
trtllm-serve orchestrator, NVIDIA Dynamo, and Triton Inference Server1.1. Context servers, generation servers, and three ways to run them
TensorRT-LLM's own vocabulary for the prefill/decode split is context servers and generation servers: a context server runs the context (prefill) phase that computes KV cache for prompt tokens, a generation server runs the generation (decode) phase that produces tokens one at a time from cached values1. Module 4 already covers the general connector pattern this maps onto (kv producer / kv consumer) and the request-ID-keyed transfer mechanism in detail; this module treats that mechanism as known and extends it with what is specific to TensorRT-LLM's own deployment surface and to GB200.
The tech blog is explicit that "there are three different approaches to do disaggregated LLM inference with TensorRT LLM, where each approach offers distinct architectural and operational characteristics suited to different deployment scenarios"1:
- trtllm-serve, native.
trtllm-serveis TensorRT-LLM's own command-line utility for launching an OpenAI-compatible server per instance. In this path you launch one server per context instance and one per generation instance, then launch an additional "disaggregated" server that acts as the orchestrator: it receives client requests and dispatches them to the appropriate context and generation servers over the OpenAI REST API1. When a context instance finishes computing KV blocks for a prompt, it returns the prompt tokens, the first generated token, and a metadata payload the blog callsctx_paramsback to the disaggregated server; generation instances use those parameters to establish communication with the context instance and pull the KV cache1. Configuration is a YAML file (disagg_config.yaml) naming the context and generation server URLs and the routing policy; the disaggregated server "supports various load balancing strategies, including round-robin and KV cache-aware routing"1. - NVIDIA Dynamo. Module 9 covers Dynamo's own component architecture (Router, Planner, KVBM) in depth. Here it is one of three interchangeable ways to run TensorRT-LLM's disaggregation, and the tech blog's framing of why you'd pick it is specific: Dynamo "introduces several advanced features not present in the other methods, including decoupled pre- and post-processing workers, which are particularly beneficial under high concurrency conditions"1. Requests are handled by pre/post-processing workers that query a smart router for the best decode worker; depending on KV-cache availability the decode worker can skip the prefill stage entirely or forward to a prefill worker, with
ctx_params-equivalent metadata again carrying the handoff1. - Triton Inference Server. Covered in full in section 6: a Triton ensemble model with a Python business-logic-scripting (BLS) orchestrator routes requests between separately-deployed context and generation TensorRT-LLM models1.
1.1 The trtllm-serve config surface, concretely
Because the native path is the one TensorRT-LLM's own GB200 benchmarks in section 4 were produced with, it is worth seeing what actually gets launched. Each context and generation instance is its own trtllm-serve process bound to a port; the disaggregated server is a separate process that reads a YAML file naming those ports and a routing policy1:
# context servers
trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8001 \
--kv_cache_free_gpu_memory_fraction 0.15 > output_ctx0 &
trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8002 \
--kv_cache_free_gpu_memory_fraction 0.15 > output_ctx1 &
# generation servers
trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8003 \
--kv_cache_free_gpu_memory_fraction 0.15 > output_gen0 &
trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8004 \
--kv_cache_free_gpu_memory_fraction 0.15 > output_gen1 &
# disaggregated orchestrator, reads disagg_config.yaml
trtllm-serve disaggregated -c disagg_config.yaml
hostname: localhost
port: 8000
context_servers:
num_instances: 2
router:
type: round_robin
urls:
- "localhost:8001"
- "localhost:8002"
generation_servers:
num_instances: 2
urls:
- "localhost:8003"
- "localhost:8004"
Two things in this file matter beyond the toy example. First, num_instances is a fixed integer set at launch — this is the config-level fact underneath the "not yet dynamic" status from section 2: changing the context:generation ratio means editing this file and relaunching, not sending a runtime command. Second, router.type is where the "round-robin and KV cache-aware routing" load-balancing choice from section 1 is actually made1; a production GB200 deployment scales this same file from two toy ports to the TEP/DEP instance counts in Table 1, with the router and instance-count fields carrying the operational weight.
2. The cache-transceiver, extended: backend selection and the road to dynamic scaling
Module 4 derives the disagg_request_id binding mechanism, the TRTLLM_NIXL_KVCACHE_BACKEND environment variable, and the GB200-specific UCX_CUDA_IPC_ENABLE_MNNVL failure mode in full — that material is not repeated here. What that module leaves for this one: which protocols the cache-transceiver actually supports, why the TensorRT-LLM team currently steers users toward two of them, and what "dynamic node joining and leaving" concretely means as a roadmap item rather than a shipped feature.
The cache-transceiver is deliberately decoupled from both the KV cache manager and the underlying communication library, and TensorRT-LLM states plainly that it supports three protocol families: "mainstream communication protocols—MPI, UCX, and NIXL—are all supported by TensorRT LLM, and the underlying communication protocols utilize RDMA / NVLink"1. That is a wider menu than the disagg-serving doc's own default (NIXL, with UCX or LIBFABRIC underneath it, from Module 4) suggests in isolation — MPI is a supported transport for the cache exchange itself, not just for process coordination.
The reason the team narrows the recommendation is stated directly: "we recommend using UCX and NIXL backends, as we are adding a dynamic scaling mechanism on top of them—specifically, dynamic node joining and leaving. This allows customers to adjust the load based on traffic demands or switch roles between context and generation dynamically"1. Two things are worth separating in that sentence. First, "dynamic node joining and leaving" means changing the number of context or generation instances in a running deployment without a restart — the same capability category as Dynamo's runtime-reconfigurable xPyD from Module 4, but TensorRT-LLM's own docs describe it as being "built on top of" the existing cache-transceiver mechanism2, i.e. future work layered on infrastructure that already exists for the static case. Second, "switch roles between context and generation dynamically" is a distinct and more unusual capability: reassigning a GPU that is currently serving as a generation server to serve as a context server (or vice versa) without redeploying it, which would let a fleet reshape its context:generation ratio in place as traffic mix shifts, rather than only adding or removing whole instances at a fixed ratio.
As of this tech blog's own "Future Work" list, neither capability is shipped: the team states it plans to "support dynamic scaling of context and generation instances based on traffic load"1 as forthcoming work, alongside overlapping KV cache communication and compute on a per-layer basis and improving performance at larger expert-parallel scales1. Read together with Module 4's framing, this is not a disagreement about design — it is one project's roadmap item, documented consistently across its tech blog and its reference docs, with a stated reason (UCX/NIXL, not MPI, are where the investment is going) for which transport will end up supporting it first.
One more property of the transceiver matters for a GB200-scale deployment specifically: when a context or generation instance spans multiple GPUs (any of the TEP/DEP configurations in section 4), "KV cache transmission between different sets of GPUs can occur in parallel"1 — the transceiver does not serialize a multi-GPU instance's transfer into one stream. TensorRT-LLM also states directly that "the optimizations required for KV cache transmission vary depending on whether it's single-node multi-GPU, multi-node multi-GPU, or different GPU models," and ships "a set of environment variables for selection in different environments"1 to tune for each case. Section 5 makes this concrete for GB200 NVL72: whether a given transfer is "single-node multi-GPU" (inside one rack's NVLink domain) or "multi-node multi-GPU" (crossing to InfiniBand) is exactly the placement question the rack-fit arithmetic below answers.
3. Measuring the payoff: rate matching and the per-GPU throughput formula
Every speedup number in section 4 — the 1.4×, the 6.11×, all of it — comes out of one measurement procedure, and it is worth understanding the procedure before trusting the numbers it produces. TensorRT-LLM's own methodology note explains why a naive comparison is not straightforward: producing a performance curve for disaggregated serving requires sweeping the full combinatorial space of parallelism strategies — TP, EP, DP, PP, and options like MTP speculative decoding — independently for context and generation stages, and "as the number of context (CTX) and generation (GEN) servers increases, the number of possible configurations grows exponentially"1.
The team's answer is a two-step process1:
- Rate matching. Measure request throughput (requests/s/GPU) for context servers across TP/EP/DP/PP mappings that satisfy the TTFT constraint, and pick the most efficient one. Separately measure total throughput (tok/s) and latency (tok/s/user) for generation servers across mappings, concurrency levels, and MTP on/off. Then find the ratio of context workers to generation workers such that the aggregated throughput of the context pool matches the aggregated throughput of the generation pool for the workload's ISL/OSL — i.e. size the two pools so neither one idles waiting on the other.
- End-to-end measurement. Benchmark actual
trtllm-serve disaggregateddeployments at the most promising configurations identified in step 1, within the practical limit of total GPUs available.
Once the ratio is fixed, throughput per GPU at a given configuration is computed as1:
$$ \text{Throughput per GPU} = \frac{\text{Total Output Tokens/s}}{\left(\dfrac{\text{NumCtxGPUs} \times \text{GenReqRate}}{\text{CtxReqRate}}\right) + \text{NumGenGPUs}} $$Read the denominator as "how many GPUs does this configuration really cost, expressed in a single common unit." NumGenGPUs counts generation GPUs directly. The first term rescales the context pool's GPU count by the ratio of generation-to-context per-GPU request rates (GenReqRate / CtxReqRate), which converts a count of context GPUs into the equivalent number of generation-GPU-units they represent once the two rates are rate-matched — the point of the rescaling is that a context GPU and a generation GPU are not interchangeable units of capacity, so a raw GPU count would misstate the true cost of the configuration. Dividing total output tokens per second by that combined GPU-equivalent count is what produces the "tokens/s/GPU" figure that a rate-matched Pareto curve plots against latency (tok/s/user)1.
This is also why a single "disaggregation gives N×" headline is the wrong unit to carry away: the speedup is read off the vertical gap between two Pareto curves — aggregated (all context+generation phases sharing GPUs) versus disaggregated (rate-matched split pools) — at a fixed latency target, and that gap changes shape with ISL:OSL ratio, MTP setting, and GPU count exactly because the rate-matching ratio itself changes with those inputs. Section 4's table is the set of gaps TensorRT-LLM has published; Module 3 already establishes the general principle that this number is workload-shape-dependent rather than a hardware constant1 — this section is the arithmetic behind why that is true, not a restatement of the conclusion.
4. GB200 benchmark numbers by model and GPU count
All performance studies in this section were run on GB200 GPUs1. Module 3 already tabulates the headline range across two models (DeepSeek R1, Qwen3) and uses it to argue that disaggregation's payoff is conditional on workload shape, not a hardware constant — that argument is not repeated here. What this section adds is the dimension Module 3 leaves aside: how the speedup moves with the number of GPUs per context/generation instance, and the TensorRT-LLM-specific parallelism notation (TEP/DEP) the source uses to label each configuration.
4.1 Reading TEP/DEP notation
Configurations in the source are labeled with attention-parallelism shorthand, e.g. CTX=1xTEP-4|GEN=2xDEP-8, meaning one context instance using tensor-parallel attention at degree 4 (with expert parallelism), paired with two generation instances using data-parallel attention (the attention-DP + EP layout from Module 5) at expert-parallel degree 81. The source states directly that "configurations with attention DP and attention TP were considered, with 4, 8, 16 or 32 GPUs per instance"1 — both parallelism strategies from Module 5 are candidates for either side of the pipeline, and instance size scales from 4 up to 32 GPUs.
The full sweep behind Table 1 is large enough that NVIDIA ships it as a reproducible artifact rather than a fixed set of numbers to take on faith: "we provide a set of scripts to reproduce the performance data presented in this paper"1. That matters for how to read the table — every row is one point NVIDIA chose to publish out of the much larger TP/EP/DP/PP × MTP × concurrency grid the rate-matching methodology in section 3 sweeps, not necessarily the single best achievable configuration at that ISL:OSL ratio.
| Model | ISL:OSL | Configuration detail | Speedup over aggregated |
|---|---|---|---|
| DeepSeek R1 | 4400:1200 | MTP off, 4–32 GPUs/instance | 1.4–1.8× |
| DeepSeek R1 | 4400:1200 | MTP on | 1.6–2.5× |
| DeepSeek R1 | 8192:256 | GEN4 (4 GPUs/generation instance) | up to 1.73× |
| DeepSeek R1 | 8192:256 | GEN8 (8 GPUs/generation instance) | up to 2.0× |
| DeepSeek R1 | 4096:1024 | MTP = 1, 2, or 3, at 50 tok/s/user | 1.7× |
| Qwen3 | 8192:1024 | attention DP/TP, 4–32 GPUs/instance | 1.7–6.11× |
4.2 Qwen3: the widest published range
Qwen3 at ISL 8192, OSL 1024 produced the widest range in this source: "the data indicate that the speedups achieved by disaggregation over aggregation range from 1.7x to 6.11x"1. The source does not publish the same GEN4/GEN8 GPU-count breakdown for Qwen3 that it does for DeepSeek R1's ISL8192:OSL256 point, only that the same 4–32-GPU-per-instance sweep and TP/DP attention-parallelism space were evaluated1. Reading Table 1 and Figure 3 together: the low end of every range (1.4×, 1.7×) clusters near what MTP-off DeepSeek R1 achieves at 4400:1200, and the high end scales up with either MTP (DeepSeek R1) or with whatever combination of GPU count and concurrency produced Qwen3's 6.11× point — the mechanism in both cases is the same rate-matching procedure from section 3, just landing at a different point on the Pareto curve.
The one clean within-model comparison the source does publish is DeepSeek R1's ISL8192:OSL256 point at GEN4 versus GEN8: growing the generation instance from 4 to 8 GPUs moves the ceiling from 1.73× to 2.0×1. That is consistent with the rate-matching arithmetic from section 3 — a larger generation instance changes NumGenGPUs and the achievable GenReqRate simultaneously, which shifts where the rate-matched ctx:gen ratio lands and therefore how much of the interference Module 3 attributes to colocated serving the disaggregated split actually removes at that concurrency. It is not evidence that "more GPUs per instance" is a general lever for a bigger speedup on its own; the source does not test a GEN16 or GEN32 point at this specific ISL:OSL ratio to confirm the trend continues.
5. Applying NVL72's topology to disaggregated placement
Module 1 establishes that a GB200 NVL72 rack is one flat 72-GPU NVLink domain at 130 TB/s aggregate rack bandwidth, and that the interconnect cliff into InfiniBand/RoCE does not disappear at NVL72 scale — it moves from the 8-GPU node boundary to the 72-GPU rack boundary3. Section 4's benchmarks all ran "on GB200 GPUs"1 without the source specifying whether each configuration stayed inside one rack. Combining the two sourced facts gives a concrete way to read the configurations in Table 1 against that boundary.
5.1 Worked arithmetic: when does a context/generation pair fit in one rack?
A GB200 NVL72 rack holds 72 GPUs3. Section 4.1 established that individual context or generation instances in TensorRT-LLM's own tests scale from 4 to 32 GPUs, and that a full serving deployment combines multiple such instances — the example notation CTX=1xTEP-4|GEN=2xDEP-8 means one 4-GPU context instance paired with two 8-GPU generation instances, for 4 + 16 = 20 GPUs total1. That configuration fits inside a single rack with 52 GPUs to spare. Scale the same shape up to the largest tested instance size: a hypothetical CTX=1xTEP-32|GEN=2xDEP-32 deployment would need 32 + (2 × 32) = 96 GPUs — more than one rack's 72-GPU capacity.
Does a context/generation deployment fit in one NVL72 rack?
Rack capacity: 72 GPUs, 130 TB/s NVLink domain3. Instance sizes 4–32 GPUs match the tested range1. When the total exceeds 72, at least one context–generation pair's cache-transceiver transfer must cross the rack boundary into InfiniBand/RoCE rather than staying on NVLink.
The implication follows directly from Module 1's cliff, not from anything TensorRT-LLM's own docs state about NVL72 specifically: when a full deployment's context and generation GPUs together fit within 72, every cache-transceiver transfer between a context server and its paired generation server can in principle stay entirely inside the NVLink switch fabric, at the full per-GPU NVLink5 rate Module 1 derives (≈1.8 TB/s)3, with no RDMA hop. Once the deployment's total GPU footprint crosses 72 — which the largest tested instance sizes (32 GPUs per instance, multiple instances per deployment) make easy to reach — some context-generation pairs necessarily span racks, and their KV transfer falls back to Quantum-X800 InfiniBand or Spectrum-X800 Ethernet, the same interconnect tier Module 1 quantifies for inter-node transfer generally3. TensorRT-LLM's cache-transceiver already supports both paths — its documentation states transfer happens "over RDMA / NVLink"1 without requiring the operator to choose one exclusively — but which path a given context/generation pair actually uses is a placement decision, not something the transceiver protocol itself determines.
6. Triton Inference Server as the third path: the BLS ensemble orchestrator
The third of the three deployment paths from section 1 builds disaggregated serving out of Triton Inference Server's existing model-composition primitives rather than a purpose-built orchestrator process. The approach uses "a Triton ensemble model, comprising a preprocessor, an orchestrator implemented as a Python business logic scripting (BLS) backend, and a post-processor. The orchestrator is responsible for routing client requests to context and generation instances, managing the flow of prompt tokens, and handling the return of generated tokens"1. This relies specifically on the Triton TensorRT-LLM backend together with TensorRT-LLM's Executor API, which the source notes "is supported only for the TensorRT backend"1 — i.e. the C++ engine path, not every backend TensorRT-LLM ships.
6.1 The disaggregated_serving_bls model repository
Triton's own worked example names the BLS model disaggregated_serving_bls. It is configured with the names of the context and generation TensorRT-LLM models it should route between; each of those models is assigned gpu_device_ids and a participant_ids range identifying which MPI ranks belong to it4. The ensemble model that a client actually calls is renamed from the usual tensorrt_llm entry point to disaggregated_serving_bls4. Launching the server requires one additional MPI rank beyond the context and generation models' own parallelism degree: "the world size should be equal to tp*pp of context model + tp*pp of generation model + 1. The additional process is required for the orchestrator"4. Multiple copies of context or generation models — i.e. data parallelism on either side — are supported by assigning each copy its own participant_ids and gpu_device_ids range and setting an instance count in the model configuration4. As of this documentation, "only the C++ version of the backend is supported"4 for this path.
disaggregated_serving_bls) that routes each request to a context model instance and then a generation model instance14.6.2 Leader mode vs. orchestrator mode, and why it matters for multi-node
Separately from the BLS routing layer, the general Triton TensorRT-LLM backend has two process-spawning modes for multi-GPU model instances, and only one of them is multi-node capable. In leader mode, "the TensorRT-LLM backend spawns one Triton Server process for every GPU. The process with rank 0 is the leader process," and the mode "is friendly with slurm deployments since it doesn't use MPI_Comm_spawn"5. In orchestrator mode, "the TensorRT-LLM backend spawns a single Triton Server process that acts as an orchestrator and spawns one Triton Server process for every GPU that each model requires" — useful when serving multiple models from one backend instance — but because it uses MPI_Comm_spawn, "it might not work properly with slurm deployments," and the documentation states plainly that "this currently only works for single node deployments"5. A multi-node, multi-GPU disaggregated deployment on Triton therefore runs in leader mode, coordinated via MPI/Slurm launch scripts rather than the backend's own orchestrator-mode process spawning5; Triton's own documentation points to a dedicated multi-node tutorial for this path rather than folding multi-node launch into either spawning mode's default behavior5.
7. Closing Part 5: five stacks, converging primitives
Part 5 has now covered five ways to run multi-node LLM inference in the open-source ecosystem: NVIDIA Dynamo (Module 9), llm-d (Module 10), vLLM (Module 11), SGLang (Module 12), and TensorRT-LLM/Triton (this module). Laid side by side, the design choices this part documented separately resolve into a small number of shared answers.
| Stack | Default transfer backend | Sync vs. overlapped transfer | Dynamic node join/leave |
|---|---|---|---|
| NVIDIA Dynamo | NIXL (VRAM-to-VRAM, non-blocking)6 | Backend-dependent (see below) | Shipped — runtime-reconfigurable xPyD6 |
| vLLM (native) | Selectable connector: NixlConnector, MooncakeConnector, LMCacheConnectorV1, others7 | Synchronous (LookupBuffer.drop_select blocks)7 | Not documented as GA in the connector docs reviewed |
| SGLang | Mooncake or NIXL (selectable)8 | Overlapped (bootstrap handshake)8 | Not documented as GA in the docs reviewed |
| TensorRT-LLM / Triton | NIXL (UCX default, LIBFABRIC option); MPI and plain UCX also supported1 | Synchronous per request, overlapped across requests1 | Explicitly in progress, not GA — built on UCX/NIXL12 |
| llm-d | Delegates to backend engine (vLLM/SGLang) connectors | Backend-dependent | Backend-dependent |
Two convergence signals stand out. First, NIXL is becoming the shared transport underneath every stack's KV-transfer path except where a project supplies its own (Mooncake, as an alternative in SGLang and as its own connector in vLLM) — NIXL's own architecture doc frames it as "targeted for accelerating point to point communications in AI inference frameworks such as NVIDIA Dynamo," built on a modular plug-in backend architecture with UCX and GPUDirect Storage as its actively supported backends9, and Module 8 covers that architecture directly; this module and Modules 9, 11, and 12 each show a different production system choosing it as a default or a first-class option. Second, every stack that documents dynamic node joining/leaving as a capability treats it the same way structurally: workers register with a discovery mechanism, publish metadata about their capacity, and drain in-flight requests before deregistering. Dynamo documents this as shipped (Module 4, Module 9); TensorRT-LLM documents the identical shape as forthcoming, built on the same NIXL/UCX transport its cache-transceiver already uses12. The gap between the two is a maturity gap on a shared design, not a disagreement about what the feature should look like.
| Stack | Orchestration model | What's shipped | What's explicitly not GA |
|---|---|---|---|
| NVIDIA Dynamo | Dedicated orchestration layer above engines | Router, SLA-driven Planner, runtime-reconfigurable xPyD6 | Rapid version churn (v1.4.x/v1.5.0-dev cadence, Module 9) |
| llm-d | Kubernetes-native EPP reference implementation | Layered filter/score/select scheduler, InferencePool CRD (GA)6 | Headline benchmarks are self-reported, scope-limited (Module 10) |
| vLLM | Connector abstraction inside the engine itself | Wide connector list, production-stack reference router7 | Disaggregated prefill labeled "experimental" in vLLM's own docs7 |
| SGLang | Bootstrap handshake inside the engine, standalone Rust router | Dual transfer backend, heterogeneous-TP staging buffer8 | Failure handling for mid-transfer partner death thinly documented (Module 4) |
| TensorRT-LLM / Triton | Three interchangeable deployment paths on one core mechanism | Cache-transceiver, three protocol backends, GB200 benchmarks across two model families1 | Dynamic node join/leave in progress2; Triton BLS path restricted to the C++ backend4 |
None of the five stacks is uniformly ahead of the others across every axis in Table 3; each is furthest along on the capability its home organization needed first — Dynamo on cross-engine orchestration, llm-d on the Kubernetes-standard routing contract, vLLM and SGLang on connector/backend flexibility inside a single engine, TensorRT-LLM on GB200-specific measured performance with three separate operational entry points. A useful way to read this part as a whole: every stack independently arrived at the same three-part shape — a producer role, a consumer role, and a pluggable transfer layer between them — and the remaining differences are in what sits above that shape (routing intelligence, autoscaling, deployment tooling) rather than in the KV-movement mechanism itself. That convergence is evidence the underlying problem has a fairly narrow set of correct solutions, even though the tooling built around it is still visibly unsettled on every axis Table 3 marks as not-GA.
Module 14 picks the story up from here at the Kubernetes deployment layer, which sits above all five stacks and is common to whichever engine is chosen underneath.
Where experts disagree
The field does not disagree about the cache-transceiver mechanism itself — all three deployment paths in section 1 share it unmodified. The live disagreement is operational: given the same underlying mechanism, which of the three deployment paths should a team actually run in production? TensorRT-LLM's own tech blog presents all three as legitimate, each "suited to different deployment scenarios"1, without ranking them — the trade-offs below are this module's synthesis of what that framing implies, not a ranking any single source states.
Native trtllm-serve: fewest moving parts
TensorRT-LLM's own default path1
Launch context and generation servers directly with trtllm-serve, add a YAML-configured disaggregated server as orchestrator. No external orchestration framework, no separate MLOps platform dependency — the entire stack is TensorRT-LLM's own tooling, and the routing policy (round-robin or KV-cache-aware) is built in1.
- Single dependency surface; easiest to reason about and debug
- Direct path to whatever GB200 benchmark numbers TensorRT-LLM publishes, since that is the tooling those benchmarks were measured with
- No autoscaling or SLA-driven scaling built in (that is Dynamo's job, Module 9)
- Dynamic node joining/leaving is explicitly in-progress on this path too, so today's instance counts are fixed at launch2
NVIDIA Dynamo: treat TensorRT-LLM as one of three backends
Dynamo's own architecture, covered in full in Module 96
Run TensorRT-LLM's context/generation servers underneath Dynamo's Router, SLA-driven Planner, and runtime-reconfigurable xPyD. The tech blog's own case for this path is decoupled pre/post-processing workers that help specifically under high concurrency1, on top of the KV-aware routing and autoscaling Module 9 covers.
- Dynamic node join/leave and SLA-driven autoscaling are already shipped here, not roadmap items
- Same orchestration layer works across vLLM and SGLang backends too, so a fleet running mixed engines gets one control plane
- An added dependency with its own fast-moving release cadence (Module 9's v1.4.x/v1.5.0-dev churn)
- Headline speedup claims for the combination are vendor-reported and backend/model-specific (Module 9)
Triton Inference Server: fold into existing MLOps investment
Triton's ensemble/BLS approach14
Use Triton's existing ensemble-model and BLS-scripting primitives to build the orchestrator, fitting disaggregated TensorRT-LLM into whatever Triton-based serving, monitoring, and multi-framework tooling an organization already runs.
- No new serving platform to adopt for teams already standardized on Triton for other models
- Multi-node support exists via leader mode and a dedicated Slurm/MPI tutorial5
- Restricted to the Executor API and the C++ backend specifically, "supported only for the TensorRT backend"1
- The worked BLS example available for review runs from an archived documentation version rather than the actively updated
llmapi-based getting-started flow in current Triton TensorRT-LLM backend docs54, a signal (not a documented statement) that this path may be receiving less active development attention than the other two
None of the three is categorically better; the choice tracks what a team already operates. A team benchmarking or reproducing TensorRT-LLM's own GB200 numbers should use native trtllm-serve, since that is the tooling behind every number in Table 11. A team that needs SLA-driven autoscaling, cross-engine orchestration, or already-shipped dynamic node join/leave should run under Dynamo (Module 9) rather than wait for TensorRT-LLM's native path to reach GA on the same capability. A team with sunk investment in Triton's model-repository tooling, multi-framework serving, and existing monitoring integrations has a real reason to prefer the BLS path despite its C++-backend restriction, provided that restriction does not conflict with a model that needs a non-TensorRT backend elsewhere in the same deployment. What would settle this more precisely than "it depends on your existing stack" is a published operational-overhead comparison — deployment complexity, mean time to recover from a worker failure, upgrade friction — across all three paths on identical GB200 hardware; none of the sources reviewed for this module publish one.
References
- officialNVIDIA / TensorRT-LLM Team. Disaggregated Serving in TensorRT LLM (Tech Blog). 2026. https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.html. Accessed 2026-09-17. Used for: context/generation server terminology, the three deployment paths (trtllm-serve, Dynamo, Triton), KV cache exchange protocol support (MPI/UCX/NIXL), rate-matching methodology and per-GPU throughput formula, GB200 benchmark numbers for DeepSeek R1 and Qwen3, TEP/DEP notation, future-work dynamic scaling statement.
- officialNVIDIA / TensorRT-LLM contributors. Disaggregated Serving (disagg-serving.md). Main branch docs, 2026. https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disagg-serving.md. Accessed 2026-09-17. Used for: dynamic node joining/leaving status ("being built on top of" the cache-transceiver mechanism), cross-reference to Module 4's disagg_request_id derivation.
- officialNVIDIA. GB200 NVL72 (product page). 2026. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-09-17. Used for: 72-GPU rack capacity, 130 TB/s NVLink bandwidth, rack-fit worked arithmetic (§5).
- officialNVIDIA. Running Disaggregated Serving with Triton TensorRT-LLM Backend. Archived documentation (Triton Inference Server release 2580). https://docs.nvidia.com/deeplearning/triton-inference-server/archives/triton-inference-server-2580/user-guide/docs/tensorrtllm_backend/all_models/disaggregated_serving/README.html. Accessed 2026-09-17. Used for: disaggregated_serving_bls ensemble/BLS mechanism, participant_ids/gpu_device_ids configuration, world_size formula, C++-backend-only restriction.
- officialNVIDIA. TensorRT-LLM Backend (README, Triton Inference Server user guide). Live docs, 2026. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/README.html. Accessed 2026-09-17. Used for: leader mode vs. orchestrator mode distinction, multi-node Slurm/MPI support, current llmapi-based getting-started flow.
- officialNVIDIA. Disaggregated Serving (NVIDIA Dynamo Documentation). 2026. https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving. Accessed 2026-09-17. Used for: Dynamo's NIXL-based non-blocking transfer, runtime-reconfigurable xPyD, cross-stack synthesis table (§7).
- officialvLLM contributors. Disaggregated Prefilling (vLLM Documentation). 2026. https://docs.vllm.ai/en/latest/features/disagg_prefill/. Accessed 2026-09-17. Used for: cross-stack synthesis table (§7) — vLLM's connector list and synchronous default, experimental status.
- officialSGLang contributors. PD Disaggregation (SGLang Documentation). 2026. https://docs.sglang.io/advanced_features/pd_disaggregation.html. Accessed 2026-09-17. Used for: cross-stack synthesis table (§7) — SGLang's dual transfer backend and overlapped bootstrap transfer.
- officialNVIDIA (ai-dynamo). NIXL Architecture (nixl.md). Main branch docs, 2026. https://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md. Accessed 2026-09-17. Used for: NIXL's role as point-to-point transport primitive and actively supported backends, cross-stack synthesis (§7).