Multi-Node LLM Inference: From One Node to N · Module 10

llm-d: The Inference Scheduler, Endpoint Picker, and Kubernetes Gateway API Inference Extension

Module 2 already worked out how a router decides which replica gets a request — approximate and precise KV-cache scoring, the filter-score-select pipeline, the plugin architecture. This module covers what is genuinely new once that decision has to run as real Kubernetes infrastructure: the InferencePool custom resource every EPP-based router is built against, the exact Envoy external-processing wire protocol that carries the picked endpoint back to the data plane, and llm-d's specific position as the project that now hosts the Gateway API Inference Extension's own reference Endpoint Picker implementation. After reading, you can read an InferencePool manifest, trace a request through the ext-proc gRPC stream, and state precisely how mature this stack is as of today.

Course 35 min read 14 sources As of 2026-09-17 Prerequisite: Module 2, Module 9
The full request path from a Kubernetes HTTPRoute through an InferencePool to the Endpoint Picker over Envoy's ext-proc protocol, and back down to a chosen pod Client HTTP request HTTPRoute backendRef InferencePool selector, targetPorts Envoy gateway ext-proc filter gRPC stream Endpoint Picker (EPP) endpointPickerRef target llm-d scheduler (Module 2) filter → score → select prefix/session/load scorers x-gateway-destination-endpoint Pod A Pod B (picked) Pod C Pod D InferencePool member pods
Figure 1. The pieces this module adds around Module 2's scheduler: an HTTPRoute sends traffic to an InferencePool, the pool's endpointPickerRef points the gateway's ext-proc filter at the Endpoint Picker, the EPP runs Module 2's filter-score-select pipeline internally, and returns its pick as a header the gateway reads before forwarding the request to the winning pod. CRD fields per the Gateway API Inference Extension's InferencePool spec5; ext-proc wiring per the Endpoint Picker Protocol8.

1. llm-d in the open-source stack: five well-lit paths and governance

1.1 What llm-d is, and what it is not

llm-d describes itself as a Kubernetes-native distributed inference stack, and its own repository organizes its capabilities into five themes it calls "well-lit paths": Intelligent Routing (prefix-cache and load-aware balancing — Module 2's subject), Advanced KV-Cache Management (tiered offloading to CPU and disk — Module 7/8's subject), Serving Large Models (prefill/decode disaggregation and wide expert-parallelism — Modules 3–6's subject), Operational Excellence (flow control and SLO-aware autoscaling), and Batch Processing (OpenAI-compatible batch APIs)2. llm-d is not itself an inference engine; it runs vLLM and SGLang as its supported engines and adds the routing, caching, and orchestration machinery around them2 — the same "orchestration layer above engines" framing Module 9 introduced for Dynamo, applied to a different set of Kubernetes-native building blocks. The project's own tagline describes it as targeting "any accelerator," and its site lists GPUs, TPUs, XPUs, CPUs, and NPUs as hardware it runs across15 — a broader hardware ambition than Dynamo's GPU-and-NVIDIA-fabric-centric framing (Module 9), though this course's own sourcing for llm-d's mechanism (Module 2 and this module) is entirely GPU-based, and the non-GPU claim is not one this course can independently verify from a fetched benchmark.

The first four of those themes are already covered from the mechanism side by earlier modules in this course. What none of those modules covered is how the routing decision from Module 2 becomes a real object a Kubernetes cluster operator deploys, upgrades, and debugs. That is this module's entire scope: the CRD, the wire protocol, and the project's own maturity.

1.2 Founding organizations and CNCF Sandbox status

llm-d was founded in May 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA2. It was accepted into the Cloud Native Computing Foundation as a Sandbox project on March 12, 2026, with the acceptance publicly announced on March 24, 2026 at KubeCon Europe4. Industry contributors named alongside the founders include AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI, with academic participation from UC Berkeley and the University of Chicago4. Section 8 comes back to exactly what "Sandbox" means for whether any of this is safe to run in production.

The project moves fast by version number: the earliest tagged GitHub release is v0.2.0 (July 2025), and the most recent as of this module's access date is v0.9.0, released August 17, 20263 — nine minor releases in about thirteen months. The v0.9.0 release notes describe themselves as "hardened for scale," listing router high availability, enhanced autoscaling, end-to-end observability, and expanded hardware support among the changes12.

1.3 A different Kubernetes-native bet than Dynamo

Module 9 covered Dynamo as a vertically integrated orchestration layer with its own named components — a Router, a Planner, a KV Block Manager — and its own backend list (vLLM, SGLang, TensorRT-LLM)13. llm-d takes a structurally different position: rather than shipping a self-contained router, it builds its scheduler as an implementation of a Kubernetes-standard API contract (sections 2–4 of this module), so that the same scheduling logic can, in principle, run behind any Envoy-based gateway that speaks that contract, not only behind llm-d's own frontend. Both projects solve overlapping problems — KV-cache-aware routing, SLO-driven scaling, disaggregated serving orchestration — from different starting assumptions about where the API boundary should sit. Section 6 returns to this contrast once the standard itself has been introduced.

2. The Gateway API Inference Extension: the standard llm-d builds on

2.1 What GAIE standardizes

The Kubernetes Gateway API Inference Extension (GAIE) is an official Kubernetes SIG project whose stated purpose is to optimize self-hosting generative models on Kubernetes by extending any Gateway API implementation that also supports Envoy's external-processing protocol into an "inference gateway" — one capable of routing on model-serving-specific signals instead of only HTTP-level load balancing1. It ships two things: an API type, InferencePool (section 3), and a wire contract, the Endpoint Picker Protocol (section 4), that any conformant EPP must speak. Module 2 already introduced the EPP as "a data-plane component that communicates via the Envoy external processing protocol, and acts as the Router"1 and covered the three-layer architecture GAIE's own proposal describes — a Routing Layer, a Flow Controller, and a Scheduling Layer9 — as the API-level description of exactly the same computation llm-d's scheduler implements concretely. Neither of those points is repeated here.

What matters for this module is that GAIE is explicitly a standard, not an implementation: gateway vendors (Envoy Gateway, kgateway, GKE Gateway, and others) implement the InferencePool and ext-proc side of the contract, and separate projects implement the EPP side. llm-d is one EPP implementation of that contract — section 5 covers exactly how central a one. GAIE's named goals go beyond the KV-cache-aware routing Module 2 covers: the project also states routing client model names to use-case-specific LoRA adapters as a goal, alongside the request-scheduling algorithm being KV-cache- and request-cost-aware1. The LoRA-routing goal is the piece that connects directly to section 3.3's multi-pool pattern — routing by model name to a specific pool is the same mechanism whether "model name" selects a different base model or a different LoRA adapter served from the same base weights.

2.2 Version history: from GA to the current release

InferencePool graduated to a stable, GA API version, v1, at project release v1.0.0, published September 9, 202565. Since then the project has shipped monthly-to-quarterly releases: v1.3.0 (January 21, 2026), v1.4.0 (March 20, 2026, adding gRPC support via the appProtocol field), v1.5.0 (April 19, 2026, the release that made endpointPickerRef optional — section 3.2), and v1.6.1, the most recent tag as of this module's access date, published September 11, 20266. That cadence — six days short of a year from a GA v1.0.0 to a v1.6.1 — is the same evidence Module 8 already used for NIXL: a young standard, moving in production-relevant ways (field defaults changing, not just patch fixes), whose exact field list is worth re-checking against the live spec rather than memorizing from any one snapshot.

Version timeline of the Gateway API Inference Extension standard alongside llm-d's own release history GAIE (the standard) v1.0.02025-09-09GA v1.5.02026-04-19endpointPickerRef optional v1.6.12026-09-11latest llm-d v0.2.02025-07-29 CNCF Sandbox2026-03-12 v0.9.02026-08-17
Figure 5. Both the standard and its principal implementation are less than a year past their respective 1.0/Sandbox milestones, moving on a monthly-to-quarterly cadence. GAIE dates from the project's own GitHub release history6; llm-d dates from its GitHub release history3 and CNCF's acceptance announcement4.

3. The InferencePool CRD

3.1 The full field list

InferencePool is a namespaced custom resource. Its spec has two required fields and two optional ones, and the optional endpointPickerRef field is itself a small object with its own defaults worth knowing precisely, because one of them (failureMode) decides what happens to every request in flight if the EPP process dies.

Table 1. InferencePoolSpec and its endpointPickerRef sub-object, field by field514.
FieldTypeRequiredDefault
selectorlabel selectoryes—
targetPortsport arrayyes—
appProtocolprotocol stringnohttp
endpointPickerRefobject referenceno (required before v1.5.0)—
↪ endpointPickerRef.groupAPI groupno—
↪ endpointPickerRef.kindobject kindnoService
↪ endpointPickerRef.nameobject nameyes—
↪ endpointPickerRef.portportno—
↪ endpointPickerRef.failureModeenumnoFailClose

selector is a label selector whose labels must exactly match the labels applied to the model-server pods in the pool5 — this is what turns the CRD from a passive record into a live group membership: any pod matching the selector is a candidate for the EPP to pick, and pods leave the pool the instant their labels stop matching, independent of any explicit deregistration step. targetPorts names the port numbers on those pods the gateway should actually route to5. appProtocol, added at GAIE v1.4.0, lets a pool declare it speaks gRPC rather than plain HTTP6, which matters once a deployment puts a gRPC-based model server behind the same InferencePool contract.

3.2 endpointPickerRef and the fail-open/fail-close choice

endpointPickerRef is the field that ties an InferencePool to an actual EPP: by default it targets a Kubernetes Service object, named explicitly, on an explicit port14. Until GAIE v1.5.0 this field was required — every InferencePool had to name an EPP. Since v1.5.0 it is optional, "to allow usages of InferencePool without user-managed Endpoint Picker deployments"5, meaning a gateway can fall back to ordinary load balancing across the pool's pods when no EPP is configured, rather than treating the absence of an EPP as a configuration error.

When an EPP is configured, failureMode decides what the gateway does if that EPP stops responding. The default, FailClose, drops the request outright14 — the safer default for a scheduler whose whole job is enforcing correctness properties (decode-stage filtering, SLO-aware admission) that a blind fallback pick could violate. The alternative, FailOpen, has the gateway forward the request to a pod of its own choosing when the EPP fails14 — trading routing quality for availability. Neither GAIE's spec page nor llm-d's own documentation states a recommended choice; it is a genuine per-deployment trade-off between "never serve a request the scheduler didn't approve" and "never let a scheduler outage become a full outage."

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: vllm-qwen3-32b
spec:
  selector:
    matchLabels:
      app: vllm-qwen3-32b
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: vllm-qwen3-32b-epp
    port:
      number: 9002
    failureMode: FailOpen

Manifest per the InferencePool API reference5; this example sets failureMode explicitly to FailOpen, overriding the FailClose default.

3.3 HTTPRoute: one route, many pools

An HTTPRoute — the standard Gateway API object that maps a hostname and path to a backend — can list an InferencePool as a backendRef, and a single HTTPRoute may reference multiple InferencePools5. That is the mechanism behind multi-model and multi-tenant serving: one route (say, a public OpenAI-compatible endpoint) can fan out by model name or weighted split to several pools, each pool backed by its own EPP, its own selector, and its own model-server fleet. Module 14 covers this pattern operationally; the point to take from this section is purely structural — InferencePool occupies the same API slot a plain Kubernetes Service would occupy as an HTTPRoute's backend, except that instead of round-robin or session-affinity load balancing at the kube-proxy layer, the pool's endpointPickerRef substitutes a model-aware decision at the gateway layer.

One HTTPRoute referencing two InferencePools as backendRefs, each pool with its own selector, targetPorts, and endpointPickerRef HTTPRoute backendRefs: [A, B] InferencePool A selector: app=model-a endpointPickerRef: epp-a InferencePool B selector: app=model-b endpointPickerRef: epp-b EPP A + pods model A fleet EPP B + pods model B fleet
Figure 2. One HTTPRoute fans out to two independently scheduled InferencePools, each with its own selector, target ports, and EPP — the structural basis for multi-model serving behind one endpoint. Per the InferencePool spec's HTTPRoute relationship5.

4. Wiring the EPP into Envoy: the ext-proc protocol

4.1 A bidirectional gRPC stream, not a REST callback

Module 2's Figure 4 drew the EPP's role in a request's path at the level of "the gateway calls the EPP, the EPP replies." The actual mechanism is Envoy's external-processing (ext_proc) HTTP filter: Envoy opens a bidirectional gRPC stream to an external service, sends it ProcessingRequest messages, and expects ProcessingResponse messages back7. The filter can be configured to send any combination of request headers, request body, request trailers, response headers, and response trailers as separate messages on the same stream7 — for a routing decision, only the request-headers phase (and, when a scorer needs prompt content for hashing per Module 2 section 3, the request body) needs to reach the EPP; the EPP does not need to see response data at all, since its job is finished once the gateway has a destination.

GAIE's EPP architecture proposal is explicit that this boundary is fixed, not extensible: "the Ext-Proc Server protocol is very well defined & specific, deviation could cause the EPP to become unusable or unstable"9. The request is buffered by the Ext-Proc Server component and handed to the Routing Layer9 — the three layers Module 2 already covered (Routing Layer, Flow Controller, Scheduling Layer) are everything downstream of that fixed protocol boundary; the wire format itself is not a place llm-d, or any other EPP implementation, gets to innovate.

4.2 How the EPP hands back its pick

The EPP communicates its selected pod to Envoy two ways at once, and both must agree: an HTTP header, x-gateway-destination-endpoint, set to the pod's address in ip:port form (or a comma-separated list of several addresses), and the same value duplicated in the ext-proc response's dynamic_metadata field, nested under a namespace key that defaults to envoy.lb8. Concretely, the metadata entry looks like {"envoy.lb": {"x-gateway-destination-endpoint": "10.0.4.12:8000"}}. Envoy's load-balancing layer reads that metadata to override its normal endpoint selection and send the request straight to the named pod, bypassing whatever load-balancing policy would otherwise apply to the pool. This is the concrete mechanism behind the dashed arrow in Module 2's Figure 4 ("select max" → "forward request") — a specific header plus a specific metadata key, not an unspecified internal handoff.

If the picker needs to constrain the gateway to a subset of the pool rather than a single pod — useful for a scorer that has ranked several roughly-equal candidates and wants Envoy's own connection-level load balancing to pick among them — it sets x-gateway-destination-endpoint-subset instead; if that key is absent, the EPP is expected to select from the full set defined by the InferencePool's own selector8.

Sequence diagram of the ext-proc gRPC exchange between Envoy and the EPP, including the success path and the two failure responses Envoy (ext_proc filter) EPP ProcessingRequest (request headers[, body]) filter → score → select ProcessingResponse: success header x-gateway-destination-endpoint = 10.0.4.12:8000 dynamic_metadata.envoy.lb.x-gateway-destination-endpoint (same value) forward request to 10.0.4.12:8000 if the EPP does not respond (failureMode) ImmediateResponse 503 — no ready endpoints ImmediateResponse 429 — drop, overloaded EPP process unresponsive, no message at all FailClose: drop request — FailOpen: gateway picks its own endpoint
Figure 3. The exact wire exchange behind Module 2's abstract "select max → forward request" step: a bidirectional ext-proc gRPC stream carrying a header and matching metadata key back to Envoy, plus the two explicit ImmediateResponse failure codes the EPP can emit and the CRD-level failureMode fallback for when it emits nothing at all. Protocol per the Endpoint Picker Protocol spec8 and Envoy's ext_proc filter documentation7; failure-mode fallback per the InferencePool field reference14.

4.3 Two layers of failure handling

There are two distinct places a request can fail on this path, and they are handled at two different layers. Inside a live ext-proc exchange, the EPP itself can explicitly signal failure with an ImmediateResponse: HTTP 503 if there are no ready endpoints in the pool at all, or HTTP 429 if the request should be dropped — for instance, because the Flow Controller's admission logic (Module 2's Request Control stage) rejected it8. Those are ordinary, expected outcomes of running the scheduler and are visible to the client as normal HTTP status codes. The failureMode field from section 3.2 covers the different case where the EPP does not respond on the stream at all — a crash, a network partition, a deployment rollout with zero available replicas of the EPP itself — and that fallback is decided once, at the CRD level, rather than per-request.

5. llm-d as the reference Endpoint Picker implementation

5.1 llm-d-router: where GAIE's own reference EPP now lives

GAIE's own project FAQ states plainly that its repository no longer hosts a full reference EPP: "Although this project previously hosted a reference Endpoint Picker (EPP) implementation (which has now been moved to the llm-d-router repository), this project now provides only a lightweight reference extension to support conformance testing"11. The FAQ adds that "individual Gateway controllers should implement their own extension or use an existing one"11 — a direct statement that GAIE itself does not ship a production EPP, and that llm-d's is the one the standard's own maintainers point to. Module 2's primary source for llm-d's scheduler, the llm-d-inference-scheduler repository10, is the same repository under an earlier name; its GitHub URL now redirects to llm-d/llm-d-router, described on its own page as "the intelligent entry point for inference requests."

This is the precise sense in which llm-d is not merely an implementation of the Gateway API Inference Extension among equals — it is the specific project the standard's own upstream repository now delegates its reference implementation to. That does not make llm-d the only conformant EPP (section 6 covers what else exists), but it does mean the mechanism Module 2 taught in detail — block hashing, KVEvents, the weighted scorer set — is not an arbitrary third-party design bolted onto GAIE; it is the implementation GAIE's maintainers migrated their own reference code into.

5.2 "Router" in llm-d's own vocabulary: Proxy plus EPP

llm-d's architecture documentation names three top-level components: a Router, an InferencePool, and the Model Server12. Confusingly for a reader coming from GAIE's own terminology, llm-d's "Router" is itself two parts: a Proxy, "a high-performance L7 proxy that conforms to the Gateway API Inference Extension standard, accepting requests and consulting the Endpoint Picker via the ext-proc protocol," and the Endpoint Picker itself, "the routing engine that scores and selects model server pods based on real-time metrics, KV-cache affinity, and configured policies"12. Mapped onto Figure 1: llm-d's Proxy is the Envoy gateway box, and llm-d's EPP is the EPP box — llm-d's own "Router" is the Proxy-plus-EPP pair together, not a separate third component. The InferencePool sits between them as the CRD-level contract, and the Model Server is any pod matching that pool's selector.

5.3 The end-to-end path, stitched together

Combining sections 2–5 with Module 2's scheduler internals gives the complete path for one request: an HTTPRoute sends it to an InferencePool (section 3); Envoy's ext-proc filter opens a gRPC stream to the pool's referenced EPP — concretely, llm-d's Proxy-plus-EPP pair (section 5.1–5.2) — and sends a ProcessingRequest (section 4.1); the EPP runs the filter-score-select pipeline Module 2 already covers in full, using whichever combination of prefix-cache, session-affinity, load, and decode-stage scorers is configured; the EPP replies with a ProcessingResponse carrying x-gateway-destination-endpoint and matching dynamic_metadata (section 4.2); Envoy forwards the request to that pod, bypassing ordinary load balancing. Nothing in this module changes what happens inside the scheduler box in Figure 1 — it only makes precise everything around that box.

6. The standard, and its other implementations

6.1 llm-d versus Dynamo: standardized contract versus vertical integration

Dynamo's own README describes its Router as one of five named components — alongside a Planner, a KV Block Manager, ModelExpress, and Grove — bundled into one orchestration layer that also names its own backend list (vLLM, SGLang, TensorRT-LLM)13. That Router is Dynamo's own component, not an implementation of GAIE's EPP contract; Dynamo does not describe itself in terms of InferencePool or ext-proc in its own README13. llm-d instead has no equivalent to a single named "Router" outside of the Proxy-plus-EPP pair described in section 5.2, both of which exist specifically to satisfy the GAIE contract rather than an internally-designed API.

The practical consequence is where each project's portability comes from. llm-d's scheduling logic is portable across any Gateway API implementation that adds ext-proc support and speaks the InferencePool contract — Envoy Gateway, kgateway, and GKE Gateway are named gateway implementations of that contract. Dynamo's Router is portable across Dynamo's own three supported engines, but is not built to be swapped for a different orchestration layer's router behind a shared standard. Table 3 makes this contrast explicit against a third project.

6.2 Other implementations of the same contract

llm-d is not the only project building against InferencePool and the Endpoint Picker Protocol. GAIE's own FAQ, in stating that "individual Gateway controllers should implement their own extension or use an existing one"11, frames llm-d-router as the default rather than the only choice. AIBrix is the other named EPP-adjacent Kubernetes-native inference stack this course covers, with its own gateway/routing, autoscaling, and distributed-KV-cache components; Module 14 covers it in the depth its own operational scope deserves. The point worth carrying forward from this module alone is definitional: Gateway API Inference Extension is the Kubernetes-standard API (InferencePool, the ext-proc contract); llm-d is the project that currently hosts the reference implementation of the EPP side of that API and the scheduler mechanism Module 2 covers in depth; other EPP implementations, AIBrix among them, can and do exist against the same standard without needing to re-implement the wire protocol from section 4.

Table 3. Positioning the standard against three Kubernetes-native orchestration answers11213.
PropertyGateway API Inference Extensionllm-dNVIDIA Dynamo (Module 9)
What it isa Kubernetes API standardthe standard's reference EPP implementationa vertically integrated orchestration layer
Routing API surfaceInferencePool CRD + ext-proc contractimplements the CRD + contractown Router component, not CRD-based
Supported enginesengine-agnostic by designvLLM, SGLang2vLLM, SGLang, TensorRT-LLM13
Portable across gatewaysyes, by definitionyes, any GAIE-conformant gatewayno, own frontend
GovernanceKubernetes SIGCNCF Sandbox (since 2026-03-12)4NVIDIA-led open source13

7. llm-d's headline numbers, read with their scope

llm-d's own README publishes several throughput and latency claims. Each is attached to a specific model, GPU count, and comparison baseline — reading the scope alongside the number is what turns a marketing figure into something usable for capacity planning.

Table 2. llm-d's self-reported headline numbers, with the hardware, model, and comparison baseline each is attached to2.
ClaimModelHardwareComparison
3× output throughput, 2× faster TTFTLlama 3.1 70B4× AMD MI300Xprefix-cache routing vs. round-robin
40% lower TTFT and ITLnot specifiedNVIDIA GPUs (unspecified)predicted-latency scheduling vs. heuristics
70% higher tok/sGPT-OSSNVIDIA B200 (p6-b200)disaggregation vs. standard vLLM
10–30% throughput gainGPT-OSS-120B, Llama 3.3 70BAMD MI300Xdisaggregated serving
50k tok/s cluster throughputnot specified16×16 NVIDIA B200 (256 GPUs)wide expert-parallelism
13.9× throughputnot specified4× NVIDIA H100hierarchical KV offload, 250 concurrent users

The 50k-tok/s cluster figure is worth converting to a per-GPU number, because "cluster throughput" scales trivially with GPU count and is not by itself comparable to any single-node figure this course has cited. At 16×16 B200 — 256 GPUs total — 50,000 tokens/s of aggregate cluster throughput works out to

$$ \frac{50{,}000\ \text{tok/s}}{256\ \text{GPUs}} \approx 195\ \text{tok/s per GPU} $$

Module 6 established DeepSeek's own production decode throughput at roughly 1.85k output tokens/s per H800 GPU, derived directly from DeepSeek's published per-node figure. The two numbers are not measuring the same thing — different model, different hardware generation, an unspecified model and workload mix behind llm-d's 50k-tok/s claim, and llm-d's figure is not stated to be decode-only — so this is not a claim that one system is roughly ten times worse than the other. It is the arithmetic every headline cluster-throughput number should get before it is used for planning: divide by GPU count, and check what stage of serving (prefill, decode, or a mix) the number actually describes before comparing it to anything.

llm-d's five well-lit paths, each paired with its own headline benchmark claim and the module in this course that covers its mechanism Intelligent Routing 3× throughput 2× TTFT Module 2 Advanced KV-Cache Mgmt 13.9× throughput (hierarchical offload) Modules 7-8 Serving Large Models 70% higher tok/s 50k tok/s cluster Modules 3-6 Operational Excellence 40% lower TTFT/ITL (predicted latency) Module 15 Batch Processing OpenAI-compatible batch APIs not benchmarked This module (10) covers the InferencePool/EPP plumbing all five paths route through rather than any one path's own mechanism
Figure 4. llm-d's five well-lit paths and their headline numbers2, cross-referenced to the modules in this course that cover each mechanism in depth — this module covers none of the five mechanisms themselves, only the Kubernetes plumbing every one of them is delivered through.

8. Is llm-d production-grade today?

CNCF's own definition of the Sandbox maturity tier, the one llm-d holds, is explicit: Sandbox projects are "experimental projects not yet widely tested in production," a category CNCF distinguishes from Incubating projects (already used in production by select adopters) and Graduated projects (stable, widely adopted, production-ready)4. That is not this module's own assessment; it is the governing foundation's structural definition of the tier the project itself applied for and holds as of this module's access date. Neither llm-d's README nor its architecture documentation contains a claim to the contrary — the v0.9.0 release notes describe hardening work (router high availability, enhanced autoscaling, broader hardware support) precisely because those properties were not yet solid in earlier releases12, and nine minor-version releases in thirteen months is the release cadence of a project still finding its stable API surface, not one that has settled it.

The Gateway API Inference Extension standard llm-d builds on is further along by one meaningful measure — InferencePool's v1 API type is GA, stable since v1.0.05 — but GA API stability is a narrower claim than production-readiness for the whole stack: it says the CRD's shape will not break compatibility, not that any given EPP behind it, llm-d's included, has been operated at scale long enough to have shaken out its failure modes. Section 4.3's two-tier failure handling (explicit 503/429 from a live EPP versus CRD-level failureMode for a dead one) is evidence of a design that has thought carefully about failure, which is a different thing from evidence that failure has been exercised in production at fleet scale.

The honest, source-grounded answer: no, not as of 2026-09-17, by the standard the project's own governing foundation applies to it. It is further along than a research prototype — it has a GA-stable CRD underneath it, five NVIDIA/AMD-hardware-backed benchmark numbers with disclosed scope, and five founding organizations with production Kubernetes experience contributing to it — but "Sandbox, not yet widely tested in production" is CNCF's own phrase, not an inference drawn from silence.

What would change this answer is not a new benchmark number but a governance event: promotion out of Sandbox to Incubating status, which CNCF reserves for projects already in production use by select adopters4. Until that happens, an operator's own risk posture — not this module's assessment — is what should decide whether a given pool runs behind llm-d's EPP with FailClose (section 3.2) or behind a vertically integrated alternative whose own maturity claims (Module 9) come with a different, but equally unresolved, evidentiary basis.

Where experts disagree

The disagreement here is not about llm-d's maturity, which the sources agree on (section 8). It is about whether building a Kubernetes-native scheduler as an implementation of an external, evolving standard (llm-d's bet) or as a self-contained, vertically integrated orchestration layer (Dynamo's bet, Module 9) is the better architecture for production inference serving on Kubernetes today.

Standardize on the CRD and the wire protocol

llm-d and the Gateway API Inference Extension's own Kubernetes SIG maintainers111

Building the scheduler behind InferencePool and the ext-proc contract means the routing logic is portable to any conformant gateway (Envoy Gateway, kgateway, GKE Gateway) and interoperable with anything else that speaks the same standard, including future EPPs this course has not covered. GAIE's own decision to move its reference EPP into llm-d-router, rather than keep maintaining one itself, is evidence the standard's maintainers consider this the right division of labor: the SIG owns the contract, llm-d owns a conformant, actively developed implementation of it.

  • Portable across gateway vendors by construction, not by later integration work.
  • A GA-stable CRD (InferencePool v1) underneath a still-evolving scheduler limits how much the API surface operators depend on can break.
  • Multiple independent EPP implementations (llm-d, and others per Module 14) can compete or specialize without forking the API.
  • The standard itself is under a year old past GA and still changing defaults (endpointPickerRef optionality at v1.5.0) — operators must track two version histories, not one.
  • Two layers of indirection (HTTPRoute → InferencePool → EPP Service) to debug versus a single vendor-owned frontend.

Ship one vertically integrated orchestration layer

NVIDIA Dynamo, per its own README's component list and backend framing (Module 9)13

A single project that owns its Router, Planner, KV Block Manager, and weight-streaming component end to end can move faster on cross-component optimizations — the Planner's scaling decisions and the Router's KV-aware routing can share internal state without needing to pass through a standardized, generic external contract. Dynamo does not need to wait for a Kubernetes SIG's release cadence to ship a routing change.

  • No dependency on an external CRD's version history or default changes.
  • Tighter internal coupling between routing and autoscaling decisions (Module 9's Planner detail).
  • Single, coherent versioning and support surface (one project, one release cadence) rather than a standard-plus-implementation pair.
  • Router logic is not portable to a non-Dynamo frontend without reimplementation.
  • No Kubernetes-SIG-level conformance guarantee comparable to GAIE's for any component.
Assessment

Neither position is settled by the evidence available today, because both projects are pre-GA by their own governance's admission — llm-d explicitly (CNCF Sandbox, section 8), and Dynamo implicitly, by the same fast, patch-heavy release cadence Module 9 documents. The standardization bet pays off specifically when an operator needs gateway-vendor portability or wants to mix EPP implementations across pools on the same cluster (Table 3, section 3.3's multi-pool HTTPRoute pattern) — a real, concrete advantage the CRD structure enables today, independent of either project's overall maturity. The vertical-integration bet pays off when an operator has already standardized on one vendor's full stack (NVIDIA hardware, NVIDIA orchestration) and values a single support surface over API portability. What would settle this more generally is public, apples-to-apples throughput and TTFT comparisons between llm-d-fronted and Dynamo-fronted deployments of the same model on the same hardware — a controlled comparison neither project's own published numbers (Table 2 here, Module 9's headline figures) provides, since each publishes benchmarks against its own baseline, not against the other.

What is changing

Three things in this module are more likely to be stale within a year than the InferencePool/ext-proc mechanism itself. First, llm-d's own website already organizes its well-lit paths differently from its README's five abstract themes — the live site lists concrete named guides ("Deploy an optimized baseline," "Route Requests by Predicted Latency," "Enable Precise Prefix-Cache Aware Routing," "Configure a Tiered Prefix Cache," "Deploy Prefill/Decode Disaggregation")15 rather than the five category names this module used for Figure 4, evidence the project is converging on specific, testable deployment recipes faster than its own README text is being updated to match. Second, GAIE's field defaults have already changed once (endpointPickerRef optionality at v1.5.0) and the six-releases-in-a-year cadence in Figure 5 makes a second default change plausible within the shelf life of this module. Third, and most consequential for a production decision: CNCF Sandbox is not a terminal state. If llm-d is promoted to Incubating or Graduated status, the production-readiness assessment in section 8 changes directly, since it rests on CNCF's own tier definition rather than on any property of the code that this module can otherwise verify independently.

Module 11 picks up vLLM itself — llm-d's default supported engine alongside SGLang2 — and covers what happens inside the pods this module's InferencePool addresses: vLLM's own disaggregated-prefill connectors, its data-parallel-plus-expert-parallel CLI arithmetic, and the production-stack reference deployment vLLM ships independent of llm-d.

References

  1. repoKubernetes SIG contributors. kubernetes-sigs/gateway-api-inference-extension (README). Kubernetes SIGs, GA since v1.0.0, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension. Accessed 2026-09-17. Used for: GAIE's stated purpose, Endpoint Picker as a data-plane ext-proc component, engine-agnostic framing.
  2. repollm-d contributors. llm-d/llm-d (README). llm-d (CNCF Sandbox since 2026-03-12; Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA), v0.7-v0.9, 2026. https://github.com/llm-d/llm-d. Accessed 2026-09-17. Used for: five well-lit-path themes, supported engines (vLLM, SGLang), founding organizations, headline benchmark numbers and their hardware/model/comparison scope (Table 2), well-lit-path guide naming on the live site.
  3. repollm-d contributors. llm-d/llm-d Releases. llm-d, v0.9.0 (2026-08-17), 2026. https://github.com/llm-d/llm-d/releases. Accessed 2026-09-17. Used for: version history v0.2.0 (2025-07-29) through v0.9.0 (2026-08-17), release cadence (Figure 5).
  4. blogCNCF. Welcome llm-d to the CNCF: Evolving Kubernetes into SOTA AI Infrastructure. Cloud Native Computing Foundation, 2026-03-24. https://www.cncf.io/blog/2026/03/24/welcome-llm-d-to-the-cncf-evolving-kubernetes-into-sota-ai-infrastructure/. Accessed 2026-09-17. Used for: CNCF Sandbox acceptance date (2026-03-12), Sandbox tier definition ("not yet widely tested in production"), industry/academic contributors.
  5. officialKubernetes SIG contributors. InferencePool (Kubernetes Gateway API Inference Extension docs). Kubernetes SIGs, InferencePool GA since v1.0.0; endpointPickerRef optional since v1.5.0, 2026. https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencepool/. Accessed 2026-09-17. Used for: InferencePoolSpec fields, full YAML manifest example, endpointPickerRef required-then-optional history, HTTPRoute-to-multiple-InferencePools relationship.
  6. repoKubernetes SIG contributors. kubernetes-sigs/gateway-api-inference-extension Releases. Kubernetes SIGs, v1.6.1 (2026-09-11), 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases. Accessed 2026-09-17. Used for: exact release dates v1.0.0 (2025-09-09, GA), v1.4.0 (2026-03-20, appProtocol/gRPC), v1.5.0 (2026-04-19, endpointPickerRef optional), v1.6.1 (2026-09-11, latest).
  7. officialEnvoy Proxy contributors. External Processing Filter (ext_proc). Envoy Proxy, latest docs, 2026. https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter. Accessed 2026-09-17. Used for: bidirectional gRPC stream mechanism, ProcessingRequest/ProcessingResponse messages, which request/response phases can be intercepted.
  8. repoKubernetes SIG / Gateway API Inference Extension contributors. Endpoint Picker Protocol (004-endpoint-picker-protocol). Kubernetes SIGs, main branch proposal doc, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/004-endpoint-picker-protocol. Accessed 2026-09-17. Used for: x-gateway-destination-endpoint header and dynamic_metadata/envoy.lb format, ImmediateResponse 503/429 failure codes, x-gateway-destination-endpoint-subset.
  9. repoKubernetes SIG / Gateway API Inference Extension contributors. EPP Architecture Proposal (0683-epp-architecture-proposal). Kubernetes SIGs, main branch proposal doc, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/docs/proposals/0683-epp-architecture-proposal/README.md. Accessed 2026-09-17. Used for: Ext-Proc Server as a fixed, non-extensible protocol boundary; request buffering before the Routing Layer.
  10. repollm-d contributors. llm-d-inference-scheduler architecture.md. llm-d (CNCF Sandbox; Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA), main branch, 2026. https://github.com/llm-d/llm-d-inference-scheduler/blob/main/docs/architecture.md. Accessed 2026-09-17. Used for: cross-reference to Module 2's scheduler mechanism; noting the repository's redirect to llm-d-router.
  11. officialKubernetes SIG contributors. FAQ (Kubernetes Gateway API Inference Extension docs). Kubernetes SIGs, live docs, 2026. https://gateway-api-inference-extension.sigs.k8s.io/faq/. Accessed 2026-09-17. Used for: statement that the reference EPP moved to llm-d-router, and that gateway controllers should implement or reuse an existing EPP.
  12. officialllm-d contributors. Architecture (llm-d docs). llm-d, live docs (v0.9.0 banner), 2026. https://llm-d.ai/docs/architecture. Accessed 2026-09-17. Used for: Router = Proxy + EPP terminology, v0.9.0 "hardened for scale" release description.
  13. repoNVIDIA / ai-dynamo contributors. ai-dynamo/dynamo (README). NVIDIA, v1.4.2, 2026. https://github.com/ai-dynamo/dynamo. Accessed 2026-09-17. Used for: Dynamo's named-component framing and supported-backend list, cited only for the brief contrast in sections 1.3 and 6.1.
  14. officialKubernetes SIG contributors. API Reference: InferencePool (Kubernetes Gateway API Inference Extension docs). Kubernetes SIGs, live docs, 2026. https://gateway-api-inference-extension.sigs.k8s.io/reference/spec/. Accessed 2026-09-17. Used for: full InferencePoolSpec and endpointPickerRef field table, appProtocol default, failureMode values and defaults (FailClose/FailOpen).
  15. officialllm-d contributors. llm-d (homepage). llm-d, live site, 2026. https://llm-d.ai/. Accessed 2026-09-17. Used for: heterogeneous-hardware tagline (GPUs, TPUs, XPUs, CPUs, NPUs), current live naming of the well-lit-path guides.