Multi-Node LLM Inference: From One Node to N · Module 10
llm-d: The Inference Scheduler, Endpoint Picker, and Kubernetes Gateway API Inference Extension
Module 2 already worked out how a router decides which replica gets a request — approximate and precise KV-cache scoring, the filter-score-select pipeline, the plugin architecture. This module covers what is genuinely new once that decision has to run as real Kubernetes infrastructure: the InferencePool custom resource every EPP-based router is built against, the exact Envoy external-processing wire protocol that carries the picked endpoint back to the data plane, and llm-d's specific position as the project that now hosts the Gateway API Inference Extension's own reference Endpoint Picker implementation. After reading, you can read an InferencePool manifest, trace a request through the ext-proc gRPC stream, and state precisely how mature this stack is as of today.
HTTPRoute sends traffic to an InferencePool, the pool's endpointPickerRef points the gateway's ext-proc filter at the Endpoint Picker, the EPP runs Module 2's filter-score-select pipeline internally, and returns its pick as a header the gateway reads before forwarding the request to the winning pod. CRD fields per the Gateway API Inference Extension's InferencePool spec5; ext-proc wiring per the Endpoint Picker Protocol8.1. llm-d in the open-source stack: five well-lit paths and governance
1.1 What llm-d is, and what it is not
llm-d describes itself as a Kubernetes-native distributed inference stack, and its own repository organizes its capabilities into five themes it calls "well-lit paths": Intelligent Routing (prefix-cache and load-aware balancing — Module 2's subject), Advanced KV-Cache Management (tiered offloading to CPU and disk — Module 7/8's subject), Serving Large Models (prefill/decode disaggregation and wide expert-parallelism — Modules 3–6's subject), Operational Excellence (flow control and SLO-aware autoscaling), and Batch Processing (OpenAI-compatible batch APIs)2. llm-d is not itself an inference engine; it runs vLLM and SGLang as its supported engines and adds the routing, caching, and orchestration machinery around them2 — the same "orchestration layer above engines" framing Module 9 introduced for Dynamo, applied to a different set of Kubernetes-native building blocks. The project's own tagline describes it as targeting "any accelerator," and its site lists GPUs, TPUs, XPUs, CPUs, and NPUs as hardware it runs across15 — a broader hardware ambition than Dynamo's GPU-and-NVIDIA-fabric-centric framing (Module 9), though this course's own sourcing for llm-d's mechanism (Module 2 and this module) is entirely GPU-based, and the non-GPU claim is not one this course can independently verify from a fetched benchmark.
The first four of those themes are already covered from the mechanism side by earlier modules in this course. What none of those modules covered is how the routing decision from Module 2 becomes a real object a Kubernetes cluster operator deploys, upgrades, and debugs. That is this module's entire scope: the CRD, the wire protocol, and the project's own maturity.
1.2 Founding organizations and CNCF Sandbox status
llm-d was founded in May 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA2. It was accepted into the Cloud Native Computing Foundation as a Sandbox project on March 12, 2026, with the acceptance publicly announced on March 24, 2026 at KubeCon Europe4. Industry contributors named alongside the founders include AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI, with academic participation from UC Berkeley and the University of Chicago4. Section 8 comes back to exactly what "Sandbox" means for whether any of this is safe to run in production.
The project moves fast by version number: the earliest tagged GitHub release is v0.2.0 (July 2025), and the most recent as of this module's access date is v0.9.0, released August 17, 20263 — nine minor releases in about thirteen months. The v0.9.0 release notes describe themselves as "hardened for scale," listing router high availability, enhanced autoscaling, end-to-end observability, and expanded hardware support among the changes12.
1.3 A different Kubernetes-native bet than Dynamo
Module 9 covered Dynamo as a vertically integrated orchestration layer with its own named components — a Router, a Planner, a KV Block Manager — and its own backend list (vLLM, SGLang, TensorRT-LLM)13. llm-d takes a structurally different position: rather than shipping a self-contained router, it builds its scheduler as an implementation of a Kubernetes-standard API contract (sections 2–4 of this module), so that the same scheduling logic can, in principle, run behind any Envoy-based gateway that speaks that contract, not only behind llm-d's own frontend. Both projects solve overlapping problems — KV-cache-aware routing, SLO-driven scaling, disaggregated serving orchestration — from different starting assumptions about where the API boundary should sit. Section 6 returns to this contrast once the standard itself has been introduced.
2. The Gateway API Inference Extension: the standard llm-d builds on
2.1 What GAIE standardizes
The Kubernetes Gateway API Inference Extension (GAIE) is an official Kubernetes SIG project whose stated purpose is to optimize self-hosting generative models on Kubernetes by extending any Gateway API implementation that also supports Envoy's external-processing protocol into an "inference gateway" — one capable of routing on model-serving-specific signals instead of only HTTP-level load balancing1. It ships two things: an API type, InferencePool (section 3), and a wire contract, the Endpoint Picker Protocol (section 4), that any conformant EPP must speak. Module 2 already introduced the EPP as "a data-plane component that communicates via the Envoy external processing protocol, and acts as the Router"1 and covered the three-layer architecture GAIE's own proposal describes — a Routing Layer, a Flow Controller, and a Scheduling Layer9 — as the API-level description of exactly the same computation llm-d's scheduler implements concretely. Neither of those points is repeated here.
What matters for this module is that GAIE is explicitly a standard, not an implementation: gateway vendors (Envoy Gateway, kgateway, GKE Gateway, and others) implement the InferencePool and ext-proc side of the contract, and separate projects implement the EPP side. llm-d is one EPP implementation of that contract — section 5 covers exactly how central a one. GAIE's named goals go beyond the KV-cache-aware routing Module 2 covers: the project also states routing client model names to use-case-specific LoRA adapters as a goal, alongside the request-scheduling algorithm being KV-cache- and request-cost-aware1. The LoRA-routing goal is the piece that connects directly to section 3.3's multi-pool pattern — routing by model name to a specific pool is the same mechanism whether "model name" selects a different base model or a different LoRA adapter served from the same base weights.
2.2 Version history: from GA to the current release
InferencePool graduated to a stable, GA API version, v1, at project release v1.0.0, published September 9, 202565. Since then the project has shipped monthly-to-quarterly releases: v1.3.0 (January 21, 2026), v1.4.0 (March 20, 2026, adding gRPC support via the appProtocol field), v1.5.0 (April 19, 2026, the release that made endpointPickerRef optional — section 3.2), and v1.6.1, the most recent tag as of this module's access date, published September 11, 20266. That cadence — six days short of a year from a GA v1.0.0 to a v1.6.1 — is the same evidence Module 8 already used for NIXL: a young standard, moving in production-relevant ways (field defaults changing, not just patch fixes), whose exact field list is worth re-checking against the live spec rather than memorizing from any one snapshot.
3. The InferencePool CRD
3.1 The full field list
InferencePool is a namespaced custom resource. Its spec has two required fields and two optional ones, and the optional endpointPickerRef field is itself a small object with its own defaults worth knowing precisely, because one of them (failureMode) decides what happens to every request in flight if the EPP process dies.
| Field | Type | Required | Default |
|---|---|---|---|
| selector | label selector | yes | — |
| targetPorts | port array | yes | — |
| appProtocol | protocol string | no | http |
| endpointPickerRef | object reference | no (required before v1.5.0) | — |
| ↪ endpointPickerRef.group | API group | no | — |
| ↪ endpointPickerRef.kind | object kind | no | Service |
| ↪ endpointPickerRef.name | object name | yes | — |
| ↪ endpointPickerRef.port | port | no | — |
| ↪ endpointPickerRef.failureMode | enum | no | FailClose |
selector is a label selector whose labels must exactly match the labels applied to the model-server pods in the pool5 — this is what turns the CRD from a passive record into a live group membership: any pod matching the selector is a candidate for the EPP to pick, and pods leave the pool the instant their labels stop matching, independent of any explicit deregistration step. targetPorts names the port numbers on those pods the gateway should actually route to5. appProtocol, added at GAIE v1.4.0, lets a pool declare it speaks gRPC rather than plain HTTP6, which matters once a deployment puts a gRPC-based model server behind the same InferencePool contract.
3.2 endpointPickerRef and the fail-open/fail-close choice
endpointPickerRef is the field that ties an InferencePool to an actual EPP: by default it targets a Kubernetes Service object, named explicitly, on an explicit port14. Until GAIE v1.5.0 this field was required — every InferencePool had to name an EPP. Since v1.5.0 it is optional, "to allow usages of InferencePool without user-managed Endpoint Picker deployments"5, meaning a gateway can fall back to ordinary load balancing across the pool's pods when no EPP is configured, rather than treating the absence of an EPP as a configuration error.
When an EPP is configured, failureMode decides what the gateway does if that EPP stops responding. The default, FailClose, drops the request outright14 — the safer default for a scheduler whose whole job is enforcing correctness properties (decode-stage filtering, SLO-aware admission) that a blind fallback pick could violate. The alternative, FailOpen, has the gateway forward the request to a pod of its own choosing when the EPP fails14 — trading routing quality for availability. Neither GAIE's spec page nor llm-d's own documentation states a recommended choice; it is a genuine per-deployment trade-off between "never serve a request the scheduler didn't approve" and "never let a scheduler outage become a full outage."
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
name: vllm-qwen3-32b
spec:
selector:
matchLabels:
app: vllm-qwen3-32b
targetPorts:
- number: 8000
endpointPickerRef:
name: vllm-qwen3-32b-epp
port:
number: 9002
failureMode: FailOpen
Manifest per the InferencePool API reference5; this example sets failureMode explicitly to FailOpen, overriding the FailClose default.
3.3 HTTPRoute: one route, many pools
An HTTPRoute — the standard Gateway API object that maps a hostname and path to a backend — can list an InferencePool as a backendRef, and a single HTTPRoute may reference multiple InferencePools5. That is the mechanism behind multi-model and multi-tenant serving: one route (say, a public OpenAI-compatible endpoint) can fan out by model name or weighted split to several pools, each pool backed by its own EPP, its own selector, and its own model-server fleet. Module 14 covers this pattern operationally; the point to take from this section is purely structural — InferencePool occupies the same API slot a plain Kubernetes Service would occupy as an HTTPRoute's backend, except that instead of round-robin or session-affinity load balancing at the kube-proxy layer, the pool's endpointPickerRef substitutes a model-aware decision at the gateway layer.
HTTPRoute fans out to two independently scheduled InferencePools, each with its own selector, target ports, and EPP — the structural basis for multi-model serving behind one endpoint. Per the InferencePool spec's HTTPRoute relationship5.4. Wiring the EPP into Envoy: the ext-proc protocol
4.1 A bidirectional gRPC stream, not a REST callback
Module 2's Figure 4 drew the EPP's role in a request's path at the level of "the gateway calls the EPP, the EPP replies." The actual mechanism is Envoy's external-processing (ext_proc) HTTP filter: Envoy opens a bidirectional gRPC stream to an external service, sends it ProcessingRequest messages, and expects ProcessingResponse messages back7. The filter can be configured to send any combination of request headers, request body, request trailers, response headers, and response trailers as separate messages on the same stream7 — for a routing decision, only the request-headers phase (and, when a scorer needs prompt content for hashing per Module 2 section 3, the request body) needs to reach the EPP; the EPP does not need to see response data at all, since its job is finished once the gateway has a destination.
GAIE's EPP architecture proposal is explicit that this boundary is fixed, not extensible: "the Ext-Proc Server protocol is very well defined & specific, deviation could cause the EPP to become unusable or unstable"9. The request is buffered by the Ext-Proc Server component and handed to the Routing Layer9 — the three layers Module 2 already covered (Routing Layer, Flow Controller, Scheduling Layer) are everything downstream of that fixed protocol boundary; the wire format itself is not a place llm-d, or any other EPP implementation, gets to innovate.
4.2 How the EPP hands back its pick
The EPP communicates its selected pod to Envoy two ways at once, and both must agree: an HTTP header, x-gateway-destination-endpoint, set to the pod's address in ip:port form (or a comma-separated list of several addresses), and the same value duplicated in the ext-proc response's dynamic_metadata field, nested under a namespace key that defaults to envoy.lb8. Concretely, the metadata entry looks like {"envoy.lb": {"x-gateway-destination-endpoint": "10.0.4.12:8000"}}. Envoy's load-balancing layer reads that metadata to override its normal endpoint selection and send the request straight to the named pod, bypassing whatever load-balancing policy would otherwise apply to the pool. This is the concrete mechanism behind the dashed arrow in Module 2's Figure 4 ("select max" → "forward request") — a specific header plus a specific metadata key, not an unspecified internal handoff.
If the picker needs to constrain the gateway to a subset of the pool rather than a single pod — useful for a scorer that has ranked several roughly-equal candidates and wants Envoy's own connection-level load balancing to pick among them — it sets x-gateway-destination-endpoint-subset instead; if that key is absent, the EPP is expected to select from the full set defined by the InferencePool's own selector8.
failureMode fallback for when it emits nothing at all. Protocol per the Endpoint Picker Protocol spec8 and Envoy's ext_proc filter documentation7; failure-mode fallback per the InferencePool field reference14.4.3 Two layers of failure handling
There are two distinct places a request can fail on this path, and they are handled at two different layers. Inside a live ext-proc exchange, the EPP itself can explicitly signal failure with an ImmediateResponse: HTTP 503 if there are no ready endpoints in the pool at all, or HTTP 429 if the request should be dropped — for instance, because the Flow Controller's admission logic (Module 2's Request Control stage) rejected it8. Those are ordinary, expected outcomes of running the scheduler and are visible to the client as normal HTTP status codes. The failureMode field from section 3.2 covers the different case where the EPP does not respond on the stream at all — a crash, a network partition, a deployment rollout with zero available replicas of the EPP itself — and that fallback is decided once, at the CRD level, rather than per-request.
5. llm-d as the reference Endpoint Picker implementation
5.1 llm-d-router: where GAIE's own reference EPP now lives
GAIE's own project FAQ states plainly that its repository no longer hosts a full reference EPP: "Although this project previously hosted a reference Endpoint Picker (EPP) implementation (which has now been moved to the llm-d-router repository), this project now provides only a lightweight reference extension to support conformance testing"11. The FAQ adds that "individual Gateway controllers should implement their own extension or use an existing one"11 — a direct statement that GAIE itself does not ship a production EPP, and that llm-d's is the one the standard's own maintainers point to. Module 2's primary source for llm-d's scheduler, the llm-d-inference-scheduler repository10, is the same repository under an earlier name; its GitHub URL now redirects to llm-d/llm-d-router, described on its own page as "the intelligent entry point for inference requests."
This is the precise sense in which llm-d is not merely an implementation of the Gateway API Inference Extension among equals — it is the specific project the standard's own upstream repository now delegates its reference implementation to. That does not make llm-d the only conformant EPP (section 6 covers what else exists), but it does mean the mechanism Module 2 taught in detail — block hashing, KVEvents, the weighted scorer set — is not an arbitrary third-party design bolted onto GAIE; it is the implementation GAIE's maintainers migrated their own reference code into.
5.2 "Router" in llm-d's own vocabulary: Proxy plus EPP
llm-d's architecture documentation names three top-level components: a Router, an InferencePool, and the Model Server12. Confusingly for a reader coming from GAIE's own terminology, llm-d's "Router" is itself two parts: a Proxy, "a high-performance L7 proxy that conforms to the Gateway API Inference Extension standard, accepting requests and consulting the Endpoint Picker via the ext-proc protocol," and the Endpoint Picker itself, "the routing engine that scores and selects model server pods based on real-time metrics, KV-cache affinity, and configured policies"12. Mapped onto Figure 1: llm-d's Proxy is the Envoy gateway box, and llm-d's EPP is the EPP box — llm-d's own "Router" is the Proxy-plus-EPP pair together, not a separate third component. The InferencePool sits between them as the CRD-level contract, and the Model Server is any pod matching that pool's selector.
5.3 The end-to-end path, stitched together
Combining sections 2–5 with Module 2's scheduler internals gives the complete path for one request: an HTTPRoute sends it to an InferencePool (section 3); Envoy's ext-proc filter opens a gRPC stream to the pool's referenced EPP — concretely, llm-d's Proxy-plus-EPP pair (section 5.1–5.2) — and sends a ProcessingRequest (section 4.1); the EPP runs the filter-score-select pipeline Module 2 already covers in full, using whichever combination of prefix-cache, session-affinity, load, and decode-stage scorers is configured; the EPP replies with a ProcessingResponse carrying x-gateway-destination-endpoint and matching dynamic_metadata (section 4.2); Envoy forwards the request to that pod, bypassing ordinary load balancing. Nothing in this module changes what happens inside the scheduler box in Figure 1 — it only makes precise everything around that box.
6. The standard, and its other implementations
6.1 llm-d versus Dynamo: standardized contract versus vertical integration
Dynamo's own README describes its Router as one of five named components — alongside a Planner, a KV Block Manager, ModelExpress, and Grove — bundled into one orchestration layer that also names its own backend list (vLLM, SGLang, TensorRT-LLM)13. That Router is Dynamo's own component, not an implementation of GAIE's EPP contract; Dynamo does not describe itself in terms of InferencePool or ext-proc in its own README13. llm-d instead has no equivalent to a single named "Router" outside of the Proxy-plus-EPP pair described in section 5.2, both of which exist specifically to satisfy the GAIE contract rather than an internally-designed API.
The practical consequence is where each project's portability comes from. llm-d's scheduling logic is portable across any Gateway API implementation that adds ext-proc support and speaks the InferencePool contract — Envoy Gateway, kgateway, and GKE Gateway are named gateway implementations of that contract. Dynamo's Router is portable across Dynamo's own three supported engines, but is not built to be swapped for a different orchestration layer's router behind a shared standard. Table 3 makes this contrast explicit against a third project.
6.2 Other implementations of the same contract
llm-d is not the only project building against InferencePool and the Endpoint Picker Protocol. GAIE's own FAQ, in stating that "individual Gateway controllers should implement their own extension or use an existing one"11, frames llm-d-router as the default rather than the only choice. AIBrix is the other named EPP-adjacent Kubernetes-native inference stack this course covers, with its own gateway/routing, autoscaling, and distributed-KV-cache components; Module 14 covers it in the depth its own operational scope deserves. The point worth carrying forward from this module alone is definitional: Gateway API Inference Extension is the Kubernetes-standard API (InferencePool, the ext-proc contract); llm-d is the project that currently hosts the reference implementation of the EPP side of that API and the scheduler mechanism Module 2 covers in depth; other EPP implementations, AIBrix among them, can and do exist against the same standard without needing to re-implement the wire protocol from section 4.
| Property | Gateway API Inference Extension | llm-d | NVIDIA Dynamo (Module 9) |
|---|---|---|---|
| What it is | a Kubernetes API standard | the standard's reference EPP implementation | a vertically integrated orchestration layer |
| Routing API surface | InferencePool CRD + ext-proc contract | implements the CRD + contract | own Router component, not CRD-based |
| Supported engines | engine-agnostic by design | vLLM, SGLang2 | vLLM, SGLang, TensorRT-LLM13 |
| Portable across gateways | yes, by definition | yes, any GAIE-conformant gateway | no, own frontend |
| Governance | Kubernetes SIG | CNCF Sandbox (since 2026-03-12)4 | NVIDIA-led open source13 |
7. llm-d's headline numbers, read with their scope
llm-d's own README publishes several throughput and latency claims. Each is attached to a specific model, GPU count, and comparison baseline — reading the scope alongside the number is what turns a marketing figure into something usable for capacity planning.
| Claim | Model | Hardware | Comparison |
|---|---|---|---|
| 3× output throughput, 2× faster TTFT | Llama 3.1 70B | 4× AMD MI300X | prefix-cache routing vs. round-robin |
| 40% lower TTFT and ITL | not specified | NVIDIA GPUs (unspecified) | predicted-latency scheduling vs. heuristics |
| 70% higher tok/s | GPT-OSS | NVIDIA B200 (p6-b200) | disaggregation vs. standard vLLM |
| 10–30% throughput gain | GPT-OSS-120B, Llama 3.3 70B | AMD MI300X | disaggregated serving |
| 50k tok/s cluster throughput | not specified | 16×16 NVIDIA B200 (256 GPUs) | wide expert-parallelism |
| 13.9× throughput | not specified | 4× NVIDIA H100 | hierarchical KV offload, 250 concurrent users |
The 50k-tok/s cluster figure is worth converting to a per-GPU number, because "cluster throughput" scales trivially with GPU count and is not by itself comparable to any single-node figure this course has cited. At 16×16 B200 — 256 GPUs total — 50,000 tokens/s of aggregate cluster throughput works out to
$$ \frac{50{,}000\ \text{tok/s}}{256\ \text{GPUs}} \approx 195\ \text{tok/s per GPU} $$Module 6 established DeepSeek's own production decode throughput at roughly 1.85k output tokens/s per H800 GPU, derived directly from DeepSeek's published per-node figure. The two numbers are not measuring the same thing — different model, different hardware generation, an unspecified model and workload mix behind llm-d's 50k-tok/s claim, and llm-d's figure is not stated to be decode-only — so this is not a claim that one system is roughly ten times worse than the other. It is the arithmetic every headline cluster-throughput number should get before it is used for planning: divide by GPU count, and check what stage of serving (prefill, decode, or a mix) the number actually describes before comparing it to anything.
8. Is llm-d production-grade today?
CNCF's own definition of the Sandbox maturity tier, the one llm-d holds, is explicit: Sandbox projects are "experimental projects not yet widely tested in production," a category CNCF distinguishes from Incubating projects (already used in production by select adopters) and Graduated projects (stable, widely adopted, production-ready)4. That is not this module's own assessment; it is the governing foundation's structural definition of the tier the project itself applied for and holds as of this module's access date. Neither llm-d's README nor its architecture documentation contains a claim to the contrary — the v0.9.0 release notes describe hardening work (router high availability, enhanced autoscaling, broader hardware support) precisely because those properties were not yet solid in earlier releases12, and nine minor-version releases in thirteen months is the release cadence of a project still finding its stable API surface, not one that has settled it.
The Gateway API Inference Extension standard llm-d builds on is further along by one meaningful measure — InferencePool's v1 API type is GA, stable since v1.0.05 — but GA API stability is a narrower claim than production-readiness for the whole stack: it says the CRD's shape will not break compatibility, not that any given EPP behind it, llm-d's included, has been operated at scale long enough to have shaken out its failure modes. Section 4.3's two-tier failure handling (explicit 503/429 from a live EPP versus CRD-level failureMode for a dead one) is evidence of a design that has thought carefully about failure, which is a different thing from evidence that failure has been exercised in production at fleet scale.
The honest, source-grounded answer: no, not as of 2026-09-17, by the standard the project's own governing foundation applies to it. It is further along than a research prototype — it has a GA-stable CRD underneath it, five NVIDIA/AMD-hardware-backed benchmark numbers with disclosed scope, and five founding organizations with production Kubernetes experience contributing to it — but "Sandbox, not yet widely tested in production" is CNCF's own phrase, not an inference drawn from silence.
What would change this answer is not a new benchmark number but a governance event: promotion out of Sandbox to Incubating status, which CNCF reserves for projects already in production use by select adopters4. Until that happens, an operator's own risk posture — not this module's assessment — is what should decide whether a given pool runs behind llm-d's EPP with FailClose (section 3.2) or behind a vertically integrated alternative whose own maturity claims (Module 9) come with a different, but equally unresolved, evidentiary basis.
Where experts disagree
The disagreement here is not about llm-d's maturity, which the sources agree on (section 8). It is about whether building a Kubernetes-native scheduler as an implementation of an external, evolving standard (llm-d's bet) or as a self-contained, vertically integrated orchestration layer (Dynamo's bet, Module 9) is the better architecture for production inference serving on Kubernetes today.
Standardize on the CRD and the wire protocol
llm-d and the Gateway API Inference Extension's own Kubernetes SIG maintainers111
Building the scheduler behind InferencePool and the ext-proc contract means the routing logic is portable to any conformant gateway (Envoy Gateway, kgateway, GKE Gateway) and interoperable with anything else that speaks the same standard, including future EPPs this course has not covered. GAIE's own decision to move its reference EPP into llm-d-router, rather than keep maintaining one itself, is evidence the standard's maintainers consider this the right division of labor: the SIG owns the contract, llm-d owns a conformant, actively developed implementation of it.
- Portable across gateway vendors by construction, not by later integration work.
- A GA-stable CRD (
InferencePoolv1) underneath a still-evolving scheduler limits how much the API surface operators depend on can break. - Multiple independent EPP implementations (llm-d, and others per Module 14) can compete or specialize without forking the API.
- The standard itself is under a year old past GA and still changing defaults (
endpointPickerRefoptionality at v1.5.0) — operators must track two version histories, not one. - Two layers of indirection (HTTPRoute → InferencePool → EPP Service) to debug versus a single vendor-owned frontend.
Ship one vertically integrated orchestration layer
NVIDIA Dynamo, per its own README's component list and backend framing (Module 9)13
A single project that owns its Router, Planner, KV Block Manager, and weight-streaming component end to end can move faster on cross-component optimizations — the Planner's scaling decisions and the Router's KV-aware routing can share internal state without needing to pass through a standardized, generic external contract. Dynamo does not need to wait for a Kubernetes SIG's release cadence to ship a routing change.
- No dependency on an external CRD's version history or default changes.
- Tighter internal coupling between routing and autoscaling decisions (Module 9's Planner detail).
- Single, coherent versioning and support surface (one project, one release cadence) rather than a standard-plus-implementation pair.
- Router logic is not portable to a non-Dynamo frontend without reimplementation.
- No Kubernetes-SIG-level conformance guarantee comparable to GAIE's for any component.
Neither position is settled by the evidence available today, because both projects are pre-GA by their own governance's admission — llm-d explicitly (CNCF Sandbox, section 8), and Dynamo implicitly, by the same fast, patch-heavy release cadence Module 9 documents. The standardization bet pays off specifically when an operator needs gateway-vendor portability or wants to mix EPP implementations across pools on the same cluster (Table 3, section 3.3's multi-pool HTTPRoute pattern) — a real, concrete advantage the CRD structure enables today, independent of either project's overall maturity. The vertical-integration bet pays off when an operator has already standardized on one vendor's full stack (NVIDIA hardware, NVIDIA orchestration) and values a single support surface over API portability. What would settle this more generally is public, apples-to-apples throughput and TTFT comparisons between llm-d-fronted and Dynamo-fronted deployments of the same model on the same hardware — a controlled comparison neither project's own published numbers (Table 2 here, Module 9's headline figures) provides, since each publishes benchmarks against its own baseline, not against the other.
What is changing
Three things in this module are more likely to be stale within a year than the InferencePool/ext-proc mechanism itself. First, llm-d's own website already organizes its well-lit paths differently from its README's five abstract themes — the live site lists concrete named guides ("Deploy an optimized baseline," "Route Requests by Predicted Latency," "Enable Precise Prefix-Cache Aware Routing," "Configure a Tiered Prefix Cache," "Deploy Prefill/Decode Disaggregation")15 rather than the five category names this module used for Figure 4, evidence the project is converging on specific, testable deployment recipes faster than its own README text is being updated to match. Second, GAIE's field defaults have already changed once (endpointPickerRef optionality at v1.5.0) and the six-releases-in-a-year cadence in Figure 5 makes a second default change plausible within the shelf life of this module. Third, and most consequential for a production decision: CNCF Sandbox is not a terminal state. If llm-d is promoted to Incubating or Graduated status, the production-readiness assessment in section 8 changes directly, since it rests on CNCF's own tier definition rather than on any property of the code that this module can otherwise verify independently.
Module 11 picks up vLLM itself — llm-d's default supported engine alongside SGLang2 — and covers what happens inside the pods this module's InferencePool addresses: vLLM's own disaggregated-prefill connectors, its data-parallel-plus-expert-parallel CLI arithmetic, and the production-stack reference deployment vLLM ships independent of llm-d.
References
- repoKubernetes SIG contributors. kubernetes-sigs/gateway-api-inference-extension (README). Kubernetes SIGs, GA since v1.0.0, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension. Accessed 2026-09-17. Used for: GAIE's stated purpose, Endpoint Picker as a data-plane ext-proc component, engine-agnostic framing.
- repollm-d contributors. llm-d/llm-d (README). llm-d (CNCF Sandbox since 2026-03-12; Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA), v0.7-v0.9, 2026. https://github.com/llm-d/llm-d. Accessed 2026-09-17. Used for: five well-lit-path themes, supported engines (vLLM, SGLang), founding organizations, headline benchmark numbers and their hardware/model/comparison scope (Table 2), well-lit-path guide naming on the live site.
- repollm-d contributors. llm-d/llm-d Releases. llm-d, v0.9.0 (2026-08-17), 2026. https://github.com/llm-d/llm-d/releases. Accessed 2026-09-17. Used for: version history v0.2.0 (2025-07-29) through v0.9.0 (2026-08-17), release cadence (Figure 5).
- blogCNCF. Welcome llm-d to the CNCF: Evolving Kubernetes into SOTA AI Infrastructure. Cloud Native Computing Foundation, 2026-03-24. https://www.cncf.io/blog/2026/03/24/welcome-llm-d-to-the-cncf-evolving-kubernetes-into-sota-ai-infrastructure/. Accessed 2026-09-17. Used for: CNCF Sandbox acceptance date (2026-03-12), Sandbox tier definition ("not yet widely tested in production"), industry/academic contributors.
- officialKubernetes SIG contributors. InferencePool (Kubernetes Gateway API Inference Extension docs). Kubernetes SIGs, InferencePool GA since v1.0.0; endpointPickerRef optional since v1.5.0, 2026. https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencepool/. Accessed 2026-09-17. Used for: InferencePoolSpec fields, full YAML manifest example, endpointPickerRef required-then-optional history, HTTPRoute-to-multiple-InferencePools relationship.
- repoKubernetes SIG contributors. kubernetes-sigs/gateway-api-inference-extension Releases. Kubernetes SIGs, v1.6.1 (2026-09-11), 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases. Accessed 2026-09-17. Used for: exact release dates v1.0.0 (2025-09-09, GA), v1.4.0 (2026-03-20, appProtocol/gRPC), v1.5.0 (2026-04-19, endpointPickerRef optional), v1.6.1 (2026-09-11, latest).
- officialEnvoy Proxy contributors. External Processing Filter (ext_proc). Envoy Proxy, latest docs, 2026. https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter. Accessed 2026-09-17. Used for: bidirectional gRPC stream mechanism, ProcessingRequest/ProcessingResponse messages, which request/response phases can be intercepted.
- repoKubernetes SIG / Gateway API Inference Extension contributors. Endpoint Picker Protocol (004-endpoint-picker-protocol). Kubernetes SIGs, main branch proposal doc, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/004-endpoint-picker-protocol. Accessed 2026-09-17. Used for: x-gateway-destination-endpoint header and dynamic_metadata/envoy.lb format, ImmediateResponse 503/429 failure codes, x-gateway-destination-endpoint-subset.
- repoKubernetes SIG / Gateway API Inference Extension contributors. EPP Architecture Proposal (0683-epp-architecture-proposal). Kubernetes SIGs, main branch proposal doc, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/docs/proposals/0683-epp-architecture-proposal/README.md. Accessed 2026-09-17. Used for: Ext-Proc Server as a fixed, non-extensible protocol boundary; request buffering before the Routing Layer.
- repollm-d contributors. llm-d-inference-scheduler architecture.md. llm-d (CNCF Sandbox; Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA), main branch, 2026. https://github.com/llm-d/llm-d-inference-scheduler/blob/main/docs/architecture.md. Accessed 2026-09-17. Used for: cross-reference to Module 2's scheduler mechanism; noting the repository's redirect to llm-d-router.
- officialKubernetes SIG contributors. FAQ (Kubernetes Gateway API Inference Extension docs). Kubernetes SIGs, live docs, 2026. https://gateway-api-inference-extension.sigs.k8s.io/faq/. Accessed 2026-09-17. Used for: statement that the reference EPP moved to llm-d-router, and that gateway controllers should implement or reuse an existing EPP.
- officialllm-d contributors. Architecture (llm-d docs). llm-d, live docs (v0.9.0 banner), 2026. https://llm-d.ai/docs/architecture. Accessed 2026-09-17. Used for: Router = Proxy + EPP terminology, v0.9.0 "hardened for scale" release description.
- repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo (README). NVIDIA, v1.4.2, 2026. https://github.com/ai-dynamo/dynamo. Accessed 2026-09-17. Used for: Dynamo's named-component framing and supported-backend list, cited only for the brief contrast in sections 1.3 and 6.1.
- officialKubernetes SIG contributors. API Reference: InferencePool (Kubernetes Gateway API Inference Extension docs). Kubernetes SIGs, live docs, 2026. https://gateway-api-inference-extension.sigs.k8s.io/reference/spec/. Accessed 2026-09-17. Used for: full InferencePoolSpec and endpointPickerRef field table, appProtocol default, failureMode values and defaults (FailClose/FailOpen).
- officialllm-d contributors. llm-d (homepage). llm-d, live site, 2026. https://llm-d.ai/. Accessed 2026-09-17. Used for: heterogeneous-hardware tagline (GPUs, TPUs, XPUs, CPUs, NPUs), current live naming of the well-lit-path guides.