System Design · Module 25
The GPU Backend Handoff: Designing the Interface to Multi-Node Serving
Module 24 narrowed every request down to one selected GPU backend. This module asks what actually crosses the line at the moment of selection: what the platform layer is allowed to assume about a backend it has never inspected, what the backend is required to expose about itself, and what has to stay true when the backend's entire internal topology changes. It is written as an interface-design problem, not a GPU-mechanics one — the multi-node serving stack itself is the subject of the separate multi-node-llm-inference course, and nothing about interconnects, PD disaggregation, or wide expert parallelism is re-derived here.
1. The boundary: what crosses it, and why this is a contract, not a mechanism
Every module up to this one has been building the platform side of a single boundary. Module 21 put an identity- and budget-checked request through a gateway; Module 22 enforced per-caller rate limits against that identity; Module 23 placed a billing hold once a request's likely cost was estimable; Module 24 narrowed an admitted, SLO-tiered request down to a candidate set of backends and then to one, using exactly two backend-state signals — KV-cache utilization and queue depth — that a generic L7 balancer cannot see2. At the instant Module 24's routing decision resolves, a request crosses from a system this course has fully specified into a system it has deliberately not touched: a stateful, multi-node GPU serving stack running some combination of prefill/decode disaggregation, wide expert parallelism, and a specific inference engine's own scheduler. That system is the entire subject of the multi-node-llm-inference course — its interconnect and transport layer, its case for why disaggregation pays off in goodput terms, its KV-transfer and wide-EP mechanics, and its per-engine implementations in Dynamo, vLLM, SGLang, and TensorRT-LLM/Triton — and none of it is re-derived here. This module's question is narrower and, on the available sources, thinner: given that the GPU backend is a black box from the platform's point of view, what exactly must the box declare about itself, and what must the platform be able to assume will keep being true no matter what changes inside the box.
That framing — declare, not explain — is what makes this an interface-design problem rather than a mechanism one. An interface is judged by what it exposes and what it hides, not by how well it describes the internals it hides. A platform team and a GPU-serving team can ship independently, on different release cadences, using different engines entirely, as long as both sides honor the same small set of exposed signals and the same request shape. Get the exposed set wrong — too thin, and the platform can't make a good routing decision; too thick, and every internal refactor on the GPU side becomes a breaking change on the platform side — and the two teams are back to being one team that happens to sit in two reporting lines.
1.1 The black-box boundary, contrasted with an ordinary service boundary
Module 10 already establishes the ordinary version of this problem: a service boundary defined by an IDL, behind which any conforming implementation is interchangeable, and Module 24's routing dimensions (model, region, tier, version) are exactly the static part of that idea applied to inference. What makes the GPU backend boundary different is not that it needs a boundary at all — every service boundary in this course needs one — but how much less the platform is entitled to assume once it has one. An ordinary stateless microservice behind an IDL can be replaced, mid-deployment, by any other implementation of the same IDL, and every in-flight request degrades gracefully because no request depends on which specific replica handled the previous one. A GPU backend that has already ingested a long prompt into its KV cache, or that already holds a specific LoRA adapter resident in HBM, is not interchangeable with an otherwise-identical replica that has not — the two are behaviorally equivalent only in the sense that both will eventually produce correct output, not in the sense that either is a free substitute for the other in the next millisecond. This module's job is to specify exactly which properties of the backend the platform is allowed to depend on (the model/version identity, the three required metrics, the request/response shape) and which it explicitly is not (which physical replica served the last token, what topology produced a given KV-cache-utilization number, how many nodes are involved in producing one response) — the second list is precisely the multi-node-llm-inference course's territory, and drawing the line there is what keeps this module from re-deriving it.
The Kubernetes Gateway API Inference Extension is the only source in this course's map that documents this exact boundary as a first-class design object, and its own scope statement is a useful anchor for what this module does and does not cover: the project extends "any gateway that supports both ext-proc and Gateway API into an inference gateway"1 — Module 21's ext-proc pattern, reused here one layer deeper — by adding an Endpoint Picker (EPP) that "fetches metrics from whichever portion of the InferencePool endpoints can best achieve the configured objectives"1. An InferencePool is defined as representing "a set of endpoints running a model server framework"1 — a set, deliberately not a single named machine, which is the first hint of the interface property Section 4 below makes precise. This module is scoped to exactly what that project's own documentation specifies about this boundary, stated as a first-principles design rather than a survey, because — as the coverage note on this module states plainly — no source in this course's map treats the platform-to-GPU-backend handoff as its own documented topic beyond what the Inference Extension project has published.
2. Why a generic health check cannot be the interface
The cheapest possible interface between a platform and a pool of backends is the one Kubernetes ships for every workload by default: a readiness probe. Kubernetes' own documentation states its job precisely — "readiness probes determine when a container is ready to accept traffic," and "if the readiness probe returns a failed state, the EndpointSlice controller removes the Pod's IP address from the EndpointSlices of all Services" that target it3. That is the entire contract: a Pod is either in a Service's rotation or it is not, and the mechanism that decides which is a single pass/fail check, run periodically, with no field anywhere in the result for how loaded, how cached, or how close to its own limit the Pod is. For a stateless CRUD service — Module 10's ordinary microservice — that binary is enough, because the whole point of that kind of backend is that every replica is interchangeable at every instant it is marked ready.
A GPU-backed model server is not that kind of backend, and the Kubernetes project's own announcement of the Inference Extension says so without qualification: "unlike typical short-lived, stateless web requests, LLM inference sessions are often long-running, resource-intensive, and partially stateful. For example, a single GPU-backed model server may keep multiple inference sessions active and maintain in-memory token caches"2. Two backends can both pass the exact same readiness probe — both process is up, both socket accepts connections, both return a 200 on the health endpoint — while one is nearly idle and the other is one request away from a queueing collapse. The probe cannot tell them apart because it was never designed to; it answers "is this backend in the rotation at all," not "how much of this backend's capacity is already spoken for."
2.1 Worked derivation: the fraction of traffic a binary signal misroutes
Make this concrete with an illustrative scenario, extending Module 24's candidate-set output (Section 3 of that module) rather than inventing a new routing dimension. Suppose Module 24's four static filters — model, region, tier, version pin — have narrowed a request to a candidate set of \(n\) backends, all of which report healthy on a readiness probe. A router with no further information (Module 1's round robin, or least-connections applied only to the readiness signal) has no basis to prefer one member of the set over another, so it spreads arriving traffic roughly evenly across all \(n\): a fraction \(1/n\) of requests to each.
Now suppose that, at a given moment, \(m\) of those \(n\) backends have already crossed the KV-cache-utilization/queue-depth threshold past which admitting one more request would push that lane's queueing delay over its stated latency budget — precisely the \(k_{\max} = B/s\) boundary Module 24 derives from a lane's latency budget \(B\) and its per-request service time \(s\). A router that only sees the readiness bit cannot identify which \(m\) of the \(n\) those are, so it keeps sending its even \(1/n\) share to all of them, meaning a fraction
$$ \frac{m}{n} $$of every arriving batch of requests lands on a backend that is already past its safe operating point — not from a bug, but because "healthy" was the only information the router had. A router with the two backend-state signals the Inference Extension names — KV-cache utilization and queue depth2 — can identify exactly those \(m\) backends and route around them, driving the misrouted fraction toward zero and redistributing that traffic across the remaining \(n - m\).
With illustrative numbers: an InferencePool of \(n = 8\) replicas behind one routing decision, of which a load spike has pushed \(m = 3\) past their safe threshold. A readiness-only router still sends \(3/8 = 37.5\%\) of arriving traffic into backends already headed for an SLO breach, because none of those three has failed its readiness probe — they are still running, still accepting connections, just already full. A router with the KV-cache-utilization and queue-depth signals sends close to \(0\%\) there, and the same traffic is absorbed by the five backends that still have room. The gap between those two numbers — not the existence of the signal, but its size — is the entire economic case for building this interface at all, and it is why the Inference Extension's own documentation attributes a latency and utilization improvement specifically to model-server-aware, metrics-driven load balancing over generic balancing1.
The illustrative fraction above is a model of the effect, not a measurement of it, but the direction is confirmed on production traffic. Google's own account of putting this exact signal pair in front of Vertex AI's model servers reports that a gateway which "scrapes real-time metrics (like KV Cache utilization) directly from the model server's Prometheus endpoints" — precisely Section 3's mechanism — cut measured Time-to-First-Token by "over 35%... for Qwen3-Coder," and improved P95 TTFT for bursty chat traffic "by 2x (52%)... for Deepseek V3.1," against the same production workload previously served by generic load balancing4. The same account attributes a further, distinct gain — prefix-cache hit rate roughly doubling from 35% to 70% — to content-aware routing that inspects a request's prefix rather than a backend's aggregate load4, which is Section 3.2's prefix-cache-status signal rather than the KV-cache-utilization/queue-depth pair this derivation covers. Both numbers are evidence of the same underlying claim in Table 2's "consumer" row: a router with backend-state signal measurably beats one without it, on real traffic, not only in an illustrative model.
3. The documented contract: the model-server protocol
The Inference Extension's own repository specifies exactly what a backend must expose to be routable by this mechanism, in a document the project calls the model server protocol — "the protocol between the EPP and the model servers"5. It is short, and its brevity is itself informative: the entire required signal vocabulary for backend state is three metrics.
3.1 The three required metrics, across four engines
The protocol document states the requirement without hedging: "the model servers MUST provide the following metrics via a Prometheus endpoint. The exact metric names don't necessarily need to be the same as the recommended names here, however the metric types and semantics MUST follow this doc"5. That sentence is the whole interface-design principle of this module compressed into one line: the semantics are the contract; the name is an implementation detail. Table 1 is the document's own metric table, and it earns its place here precisely because it shows the same three semantic metrics implemented under four different literal names across four different serving engines — the concrete proof that the contract survives an engine swap. A fourth engine the same table covers, trtllm-serve, reports the identical three semantics under a third naming convention again (trtllm_num_requests_waiting, trtllm_num_requests_running, trtllm_kv_cache_utilization)5 — a separate command-line entry point from Triton TensorRT-LLM's own metric names in the same table, and further evidence that the number of literal naming conventions in production is not the number of semantic metrics the contract has to define.
| Metric (contract semantics) | Type | vLLM | Triton TensorRT-LLM | SGLang |
|---|---|---|---|---|
| TotalQueuedRequests | Gauge | vllm:num_requests_waiting | nv_trt_llm_request_metrics{waiting} | sglang:num_queue_reqs |
| TotalRunningRequests | Gauge | vllm:num_requests_running | nv_trt_llm_request_metrics{scheduled} | sglang:num_running_reqs |
| KVCacheUtilization | Gauge | vllm:kv_cache_usage_perc | nv_trt_llm_kv_cache_block_metrics{fraction} | sglang:token_usage |
Two of these three metrics are the same pair named throughout Module 24 — KV-cache utilization and queue depth (here split into TotalQueuedRequests, waiting, and TotalRunningRequests, already being served). This module's contribution is not the signals themselves, which Module 24 already establishes as the routing basis; it is the observation that the protocol defines them as an engine-agnostic schema with a required semantic and an optional literal name, and that four unrelated serving engines — three of which are full modules in the multi-node-llm-inference course (vLLM, SGLang, TensorRT-LLM/Triton) — already implement that schema under their own native metric names. The platform-layer code Module 24 describes never has to know the difference between vllm:num_requests_waiting and sglang:num_queue_reqs; it reads TotalQueuedRequests, and the EPP's per-engine adapter is responsible for the mapping5.
3.2 Optional metrics and the request-shape half of the contract
Two further metrics are optional rather than required — BlockSize and NumGPUBlocks, both used, in the protocol's own words, "by the prefix cache scorer," with an explicit fallback if a given engine does not expose them: "if this metric is not available, [it] will be derived from the prefix plugin config" instead5. A LoRA-adapter metric, vllm:lora_requests_info, is required only of engines that support dynamic adapter serving, and it carries the running and waiting adapter lists as comma-separated label values rather than as separate metrics5. What these metrics feed into on the routing side — prefix-cache and LoRA-adapter affinity scoring — is Module 24's territory; this module's point is narrower: the protocol distinguishes a required core (three metrics, every conforming backend must have them) from an optional extension (two metrics, degrade gracefully to a config-derived estimate if absent), which is the right shape for a contract that has to keep working as new engines with different internal accounting join the pool.
The metrics table is only one side of the contract. The same document specifies the other direction — what the platform is allowed to send — with equal bluntness: "the model server MUST implement OpenAI's Completions and Chat APIs"5. This is the request-shape half of the interface, symmetric to the signal-shape half Table 1 covers: just as the platform never needs to know which literal metric name a given engine uses, the platform never needs to know which engine is on the other end of a request, because every conforming backend accepts the same two request shapes. Model identity — which model, which version — travels inside that request as data (the model field of the Completions/Chat payload), not as a property of which network address the platform happened to dial; Section 4 below is why that distinction matters.
The reason a shared request shape is worth standardizing, rather than letting each engine define its own, is the same combinatorial argument that motivates Module 21's ext-proc pattern one layer up: without it, every gateway implementation that wants to speak to every serving engine needs its own translation code for that specific pairing, an integration cost that scales with the product of the number of gateways and the number of engines rather than the sum of the two. A shared request contract collapses that product down to a sum — one Completions/Chat-shaped client per gateway, one Completions/Chat-shaped server per engine — which is the same economic logic Module 21 uses to justify reusing Envoy's ext-proc protocol instead of building a bespoke gateway for every inference platform. Table 1's per-engine metric-name column is the read side of this same argument: the EPP needs one adapter per engine for the metrics it scrapes, not one adapter per gateway-engine pair, because the metric's semantics are standardized even where its literal name is not5.
| Property | Readiness probe | Model-server protocol metrics |
|---|---|---|
| Question answered | In the Service's rotation, or not | How loaded, how cached, which model version |
| Value space | Success / Failure / Unknown | Continuous gauges (0-100% utilization, integer counts) |
| Transport | kubelet-executed exec/httpGet/tcpSocket check | Backend-exposed Prometheus /metrics endpoint, EPP-scraped |
| Consumer | EndpointSlice controller (in/out of Service) | Endpoint Picker (which member of the pool, if any) |
| Standardization | Kubernetes core API, stable for years | Project-defined schema, engine-agnostic names optional, "partially implemented" by the protocol's own status5 |
A complete inference gateway needs both rows of both signals — the readiness probe still decides whether a backend is in the pool at all (a crashed process is not "high queue depth," it is absent), and the protocol's metrics decide which member of the pool that is in gets this particular request. Neither substitutes for the other; they answer different questions, at different layers, on different cadences.
4. Model and version identity across the boundary
Module 24 established version pinning as a hard filter on the candidate set: a caller that has validated against one specific model version is excluded from every backend running a different one, including a newer version of the same alias. That section treats identity as a routing dimension. This module's job is the contract requirement underneath it: for a version pin to be enforceable at all, model and version identity have to be represented as addressable data that survives whichever physical backend is currently serving it — not as an accident of network topology.
Google's Borg cluster manager solved a structurally identical problem for a different resource, decades before an inference-specific gateway existed, and its own paper states the mechanism plainly: "Borg creates a stable 'Borg name service' (BNS) name for each task that includes the cell name, job name, and task number. Borg writes the task's hostname and port into a consistent, highly-available file in Chubby with this name"6. A caller — or, in Borg's case, another job's load balancer — resolves the BNS name, not a machine's raw address, and the mapping from name to physical location is free to change every time Borg reschedules the task, without any caller ever needing to know. The name is the interface; the physical machine behind it is an implementation detail the name was built to hide.
An InferencePool is the same idea applied one layer up: it names "a set of endpoints running a model server framework"1, and the caller-facing identity a routing decision resolves against is the pool and the model/version pin carried in the request body, not any individual endpoint's address. What Borg's BNS name does for one task on one machine, the pool-plus-pinned-model-identity pairing does for a fleet of GPU replicas: it lets the set of physical backends behind a given model version change — scale up, scale down, get replaced by a different engine entirely — while every caller's contract (the model name and version it pins to in its request) stays exactly the same string it always was. Section 6 below is what this buys the platform when the change behind that name is not just scale, but topology.
The mechanism that keeps an InferencePool's membership current is the same one Section 2 already introduced for a different purpose: a backend joins the pool's routable set when it starts passing its readiness probe, and Kubernetes' own EndpointSlice controller is what adds and removes IP addresses from that set as probes pass and fail3. Borg's BNS file in Chubby and Kubernetes' EndpointSlice objects are solving the same problem with the same shape of answer: a highly-available, continuously-updated record of where a stably-named thing currently lives, decoupled from any caller having to track that location itself. Section 2 treated the readiness signal as too coarse to route on; Section 4's point is narrower and does not contradict that — the readiness probe is exactly the right granularity for the different question of whether a backend belongs in the pool's addressable set at all, which is a precondition for the KV-cache-utilization and queue-depth signals to mean anything for it in the first place. A backend cannot be routed to by degree of load before it has been added to the pool by binary presence.
5. Backpressure: how "no room" reaches the platform
Module 24's degradation cascade (its Section 6) describes what the platform does once retry, fallback, and a bounded wait have all failed: reject. That section is about the platform's response to backpressure. This module's question is about the wire between the two systems: what signal, specifically, tells the platform that a backend — or a whole pool — has run out of room, in a form specific enough to be acted on before the cascade's later, more visible steps.
On the sources this module is scoped to, the answer is that there is no separate backpressure message at all. The model server protocol defines exactly the metrics Table 1 lists, plus the two optional and one LoRA metric Section 3.2 covers, and nothing else5 — no distinct "I am overloaded, stop sending" flag, no explicit shed-load response code defined as part of this contract. Backpressure, on this design, is not a signal the backend pushes; it is a judgment the platform derives from watching the same two continuous numbers — queue depth and KV-cache utilization — that it already reads for ordinary routing, aggregated across an entire pool rather than read one backend at a time. A single backend at high queue depth is a routing signal: send new requests elsewhere in the pool. Every backend in the pool simultaneously at high queue depth is the identical signal, read at the pool level, and it is backpressure — the same numbers, the same protocol, a different aggregate. This is the precise point where this module's contract hands off to Module 24's cascade: once the platform's own aggregation over this module's exposed signals concludes that no candidate backend has room, Module 24's queue-then-reject steps are what run next.
This is a deliberate, if implicit, design choice worth naming rather than assuming: a protocol built this way keeps the backend simple — it only ever has to report its own current state truthfully — and puts the entire judgment of "is the pool, in aggregate, out of room" on the platform side, where Module 24's SLO-tier latency budgets and queue-depth-versus-\(k_{\max}\) arithmetic already live. The alternative — a backend that pushes an explicit, self-declared "I am shedding load" signal — would require every engine to implement its own overload-detection policy and keep it consistent with whatever the platform expects that signal to mean, which reintroduces exactly the engine-specific coupling the rest of this contract is built to avoid. Reading backpressure out of the same two numbers used for routing, rather than adding a third, backend-declared one, is the smaller, more engine-portable interface — consistent with the protocol's stated three-metric floor rather than an expanding one.
This also clarifies how Section 2's binary readiness signal and this section's continuous metrics relate, rather than compete. The metrics protocol is the primary channel for ordinary backpressure — a rising queue depth or KV-cache utilization across a pool is the normal, expected way the platform learns it is running out of room, well before anything is actually broken. The readiness probe is the channel of last resort, for the case the metrics protocol cannot cover at all: a backend so overloaded, or so wedged, that it stops answering its own health check. A backend that is merely full still answers "yes, I am here, I am just reporting a high number" — that is the metrics protocol working as designed. A backend that has stopped answering anything, including the cheap binary check, has left the domain this module's contract can describe and fallen back to the one guarantee every Kubernetes-scheduled workload already has: Section 2's EndpointSlice removal, pulling it out of rotation entirely rather than routing around it by degree. The two channels are not redundant; one handles the graded case, the other catches the case where grading itself has stopped being possible.
6. What the interface must guarantee across a backend upgrade
The strongest test of any interface is what happens when one side changes everything about its internals while the other side changes nothing about its code. For this boundary, the concrete version of that test is a backend upgrade that changes prefill/decode topology — collapsing a disaggregated deployment back into a monolithic one, or the reverse, exactly the trade-off the multi-node-llm-inference course's own goodput-versus-interference analysis works through in depth. This module does not re-derive when that trade-off favors disaggregation; it only asks what the platform layer is entitled to assume stays true while someone else makes that call.
Three properties have to hold across that change for the platform layer to need zero code changes. First, the metric semantics Table 1 defines — queued, running, KV-utilization — have to keep meaning the same thing whether one replica type does both prefill and decode or two pool types split the work, which is exactly why the protocol specifies semantics as the required contract and literal metric names as optional5: a disaggregated deployment's prefill pool and decode pool each still report their own queue depth and KV-cache utilization under the same three field names, and the EPP scores each endpoint the same way regardless of which half of the pipeline it represents. Second, the request-shape half of the contract — OpenAI Completions/Chat APIs5 — has to keep being what every endpoint in the pool accepts, so a caller's pinned model/version identity resolves identically whether it lands on a monolithic replica or a prefill-then-decode pair of them. Third, the identity the caller pins against — the model/version string, resolved the way Section 4 describes through the pool rather than through any one endpoint's address — has to keep resolving to the same InferencePool even though the set of endpoints behind that pool, and the internal handoff between them, is completely different than it was before the upgrade.
None of those three properties depends on knowing anything about why the upgrade happened or what it bought — that is precisely the multi-node-llm-inference course's territory, not this module's. What this module adds is the observation that a protocol specified as engine-agnostic semantics over a required, deliberately small field set, plus a request-shape contract every conforming server implements identically, is what makes "the GPU team changed the entire serving topology and the platform team's code did not need a single-line diff" a true statement rather than an aspiration.
"Did not need a single-line diff" is a claim, not a guarantee, and Module 11's observability triad — metrics, logs, traces — is how a platform team checks it rather than assumes it. The check is narrow and specific to this interface: after a backend upgrade, the same three required metrics (Table 1) should still be arriving, at the same field names, from every endpoint in the affected InferencePool, and Module 24's SLO-tier latency budgets should still be met at whatever rate they were before the upgrade. If either check fails — a metric goes missing, or the same routing logic starts producing worse SLO attainment against an unchanged set of exposed signals — the failure has located itself precisely: a missing or renamed field is a contract violation on the backend side (Section "What is changing" below is exactly this failure mode, caught once already), while unchanged fields with worse outcomes means the new topology's behavior under those fields differs enough that the platform's routing logic, tuned against the old topology, needs retuning — a Module 24 concern, not a sign that the interface itself broke. Keeping those two failure modes distinguishable is a second, quieter benefit of specifying the contract explicitly rather than leaving it implicit: without a written contract, "something changed and now it's worse" is one undifferentiated incident; with one, it is immediately either a broken promise or a parameter that needs re-tuning.
Where experts disagree
Module 24's disagreement section covers how a platform should decide admission — hard quota, optimistic estimate, or telemetry-driven placement — and that argument does not repeat here. The disagreement specific to this module's boundary is different: given that a backend-state contract has to exist at all, should it be one standardized, engine-agnostic protocol that every serving engine implements, or should the platform instead integrate directly with each serving engine's own native router, forgoing a cross-engine contract altogether?
One standardized, engine-agnostic contract
The Kubernetes Gateway API Inference Extension project's own design15
A fixed, minimal, versioned metric schema — three required gauges, two optional, one LoRA metric — lets any conforming gateway route to any conforming backend5, which is exactly what Section 6's backend-upgrade scenario needs: the platform layer never changes when the GPU team swaps engines or topology.
- One platform-layer implementation serves every engine that implements the protocol, evidenced concretely by the same three metrics already shipping under different names in vLLM, Triton TensorRT-LLM, trtllm-serve, and SGLang5.
- Engine-agnostic by construction: swapping the serving engine behind an InferencePool is not a platform-layer change.
- Deliberately minimal — only three required metrics — so it cannot expose engine-internal state a purpose-built router might use; the protocol's own status note that a pluggable EPP architecture means this contract "can, by definition, not be as strict"5 concedes the same point.
- Standardization lags implementation: the metric a backend reports under this contract has itself changed name within one release (Section "What is changing" below).
Let the serving engine's own router own the decision
The native router/scheduler each major serving stack ships on its own — NVIDIA Dynamo's Router, Planner, and KV Block Manager; SGLang's own router; vLLM's production-stack router — covered in depth in the multi-node-llm-inference course rather than re-derived here
A router built inside the same project as the serving engine can react to whatever internal state that engine's own maintainers choose to track, on that engine's own release cadence, without waiting for a cross-vendor specification to add a field for it.
- Not bounded by a lowest-common-denominator metric set; can use any internal state the engine's own team decides is useful.
- Iterates on the engine's own schedule, with no cross-project coordination cost.
- Locks the platform's routing code to one engine's semantics; a platform that runs one engine in one region and a different one elsewhere, or migrates between engines, cannot reuse the same routing integration.
- Reintroduces exactly the platform-to-GPU coupling a stable interface (Section 6) exists to avoid.
Which position wins depends on whether the platform's job is to stay engine-portable over the system's lifetime or to extract maximum routing quality from one engine it has already committed to permanently. A platform that expects to run more than one engine — even sequentially, across a migration, rather than simultaneously — should pay the minimality cost of the standardized contract, because engine portability is precisely the property Section 6's backend-upgrade scenario depends on, and it is not a property a native, engine-specific router provides by design. A platform that has fully committed to one engine, is not migrating, and wants every last bit of routing quality that engine's own internal state can provide should prefer the native router and accept the coupling, because there is no portability requirement left to trade against. What would settle this in a specific deployment is not a preference but a measurement: run the fixed three-metric contract and the native router against the same traffic and compare SLO-attainment rate; if the standardized contract's routing quality is close to the native router's, the portability it buys is close to free, and the standardized contract should be the default rather than the exception.
What is changing
The model server protocol's own status line reads simply "Partially implemented," with the stated reason that "with the creation of the pluggable architecture this protocol can, by definition, not be as strict"5 — a project-level admission that the contract this module treats as fixed is itself mid-stabilization, consistent with Module 24's separate observation that the extension's broader API surface still carries both a v1 and a v1alpha1 reference side by side. A concrete instance of that churn reached the metric names Table 1 lists directly: the project's v1.3.0 release, published 2026-01-217, includes the changelog line "use the correct vllm metric gpu_cache_usage_perc --> kv_cache_usage_perc"7 — the literal Prometheus name for the KVCacheUtilization metric in vLLM changed within a single release, months after the protocol itself was published. Because the protocol's own stated principle is that semantics are the contract and literal names are not5, this rename is exactly the kind of change the interface is designed to absorb without breaking a conforming EPP implementation — but it is also a concrete, dated reminder that even the "stable" side of this boundary has moved once already, less than a year into this course's as-of date, and should be expected to move again.
References
- officialkubernetes-sigs contributors (Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA). Introduction (Kubernetes Gateway API Inference Extension). Kubernetes SIGs, live docs, 2026. https://gateway-api-inference-extension.sigs.k8s.io/. Accessed 2026-09-17. Used for: the InferencePool and Endpoint Picker (EPP) definitions; the ext-proc extension pattern; the claimed latency/utilization benefit of model-server-aware balancing.
- blogKubernetes project contributors. Introducing Gateway API Inference Extension. Kubernetes (kubernetes.io/blog), published 2025-06-05. https://kubernetes.io/blog/2025/06/05/introducing-gateway-api-inference-extension/. Accessed 2026-09-17. Used for: partially stateful, long-running inference sessions with in-memory token caches; KV-cache utilization and queue depth named as routing signals.
- officialKubernetes. Pod Lifecycle. kubernetes.io documentation, live docs, 2026. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/. Accessed 2026-09-18. Used for: the readiness probe's definition and its binary effect on Service EndpointSlice membership.
- blogFeyisetan, F., Yuan, Y. How GKE Inference Gateway improved latency for Vertex AI. Google Cloud Blog, published 2026-02-07. https://cloud.google.com/blog/products/containers-kubernetes/how-gke-inference-gateway-improved-latency-for-vertex-ai. Accessed 2026-09-18. Used for: measured Time-to-First-Token reductions (over 35% for Qwen3-Coder; 2x/52% P95 for Deepseek V3.1) and prefix-cache hit-rate improvement (35% to 70%) from adopting load-aware and content-aware routing in production.
- repokubernetes-sigs/gateway-api-inference-extension contributors. Model Server Protocol (design proposal 003). GitHub repository,
docs/proposals/003-model-server-protocol/README.md, main branch. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/docs/proposals/003-model-server-protocol/README.md. Accessed 2026-09-18. Used for: the required-metrics table (TotalQueuedRequests, TotalRunningRequests, KVCacheUtilization) and per-engine metric names for vLLM, Triton TensorRT-LLM, trtllm-serve, and SGLang; optional BlockSize/NumGPUBlocks metrics; the LoRA-adapter metric and its labels; the OpenAI Completions/Chat API requirement; the "Partially implemented" protocol status. - paperVerma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., Wilkes, J. Large-scale cluster management at Google with Borg. EuroSys '15. https://research.google.com/pubs/archive/43438.pdf. Accessed 2026-09-17. Used for: the Borg Name Service (BNS) as a stable-name-over-changing-physical-location precedent (§2.6).
- repokubernetes-sigs/gateway-api-inference-extension. Release v1.3.0. GitHub releases, published 2026-01-21. https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/tag/v1.3.0. Accessed 2026-09-18. Used for: the changelog entry renaming the vLLM metric
gpu_cache_usage_perctokv_cache_usage_perc(PR #1905).