Multi-Node LLM Inference: From One Node to N · Module 2

Replication and Routing: KV-Cache-Aware, Prefix-Aware, and SLO-Aware Request Placement

The moment you run N copies of a TP8 engine behind one endpoint, each copy accumulates its own private paged-KV-cache state, and a request routed to the wrong copy pays for prefill it didn't have to. This module works through the two mechanisms production routers use to see that state — approximate block-hashing with an LRU index, and precise event-streamed indexing over ZeroMQ — the layered filter-score-select pipeline that combines them with load and session signals, and where the four major open-source routers (llm-d, vLLM production-stack, sglang-router, and the Gateway API Inference Extension they build against) land on the cost/accuracy trade-off. After reading, you can trace exactly how a request's replica gets chosen, mechanism by mechanism, in any of these stacks.

Course 30 min read 6 sources As of 2026-09-17 Prerequisite: Module 1
A single router in front of N independent TP8 replicas, each holding a different, private slice of cached KV blocks client requests Router (Endpoint Picker) Replica 0 TP8 engine, 8× GPU private KV cache warm: 3 blocks of prefix A Replica 1 TP8 engine, 8× GPU private KV cache warm: mixed prefix A/B blocks Replica N−1 TP8 engine, 8× GPU private KV cache cold: newly joined pod
Figure 1. Each replica behind the router is a complete, independent TP8 engine with its own private KV cache — not a shared pool. Replica 0 has cached prefix A, replica 1 has a mixed set overlapping A and a different prefix B, and a newly joined replica N−1 starts cold. A router that cannot see this state (plain round-robin or least-connections) cannot tell these three apart. Endpoint Picker as the routing component sitting in front of a pool of model-server replicas, per the Kubernetes Gateway API Inference Extension1.

1. Replication turns placement into a stateful decision

1.1 The replica model: N private KV caches behind one entry point

Module 1 established the physical layer: NVLink inside a node, an ~8× bandwidth cliff crossing to InfiniBand/RoCE between nodes, and NVL72 as a structurally different 72-GPU NVLink domain. This module starts from what that physical layer is carrying once you scale a serving fleet out rather than up: N independent copies of the same TP8 engine, each occupying its own 8-GPU node (or fraction of an NVL72 domain), each running its own continuous-batching loop, and — the fact that matters for this module — each holding its own paged KV cache in its own HBM. There is no shared cache across replicas unless something is built to make one (that something is Part 4 of this track). A request that lands on replica 3 can reuse only the blocks replica 3 happens to have resident; it gets no benefit from an identical prefix sitting warm on replica 7.

This is a different situation from everything upstream of it. Inside one TP8 engine, KV-cache management, paged allocation, and continuous batching are all solved problems Aman already has working code for — a single scheduler owns a single cache and every request in flight sees the same allocator. The instant a second replica exists, the cache stops being one object. It becomes N objects, privately owned, diverging from the moment they boot, and the component that decides which object a given request talks to — the router, called the Endpoint Picker (EPP) in the Gateway API Inference Extension's terminology — is now making a decision with real consequences for that request's latency, not just a load-spreading decision1. The EPP is a data-plane component that sits behind the ingress gateway, communicates with it over Envoy's external-processing protocol, and is responsible for picking which of the pods behind an InferencePool a request goes to1. The CRD wiring that connects a gateway to an EPP (the InferencePool object, the Envoy ext-proc plumbing) is Kubernetes machinery covered in Module 10; what this module covers is the decision logic inside the EPP itself, independent of how it got invoked.

1.2 What a cache-blind router costs

A stateless policy — round-robin, least-connections, random — treats the three replicas in Figure 1 as interchangeable. They are not: replica 0 can serve a request continuing prefix A without recomputing it, replica 1 can serve part of a request touching prefix A or B, and replica N−1 must recompute everything from scratch because it just joined the pool. A stateless policy sends roughly one-third of prefix-A traffic to each of the three replicas, so two-thirds of that traffic re-executes prefill for tokens that are already resident somewhere in the fleet. The router does not need to know the exact FLOPs or HBM-bandwidth cost of that redundant prefill — Aman already has that arithmetic from single-node serving — it needs a cheap way to know where a given prefix is likely to be cached before it picks a replica. Sections 2 and 3 cover the two mechanisms production routers use to get that information: one approximate and cheap, one precise and more expensive.

2. Approximate prefix-cache scoring

llm-d's default mechanism for guessing which replica has a request's prefix cached does not ask the model server anything. It infers cache state entirely from the router's own routing history, using a fixed-size block hash and an in-memory index — cheap, stateless with respect to the model server, and, by construction, approximate.

2.1 Block hashing and the rolling hash chain

The approx-prefix-cache-producer plugin splits an incoming prompt into fixed-size blocks — llm-d's own documentation gives 16 tokens per block as the example granularity — and builds a rolling hash chain over them: each block's hash is a function of that block's content and the hash of the block before it, so block k's hash changes if any earlier block in the prefix changes2. This is the same chaining idea used for content-addressed prefix caching inside a single engine (block hash depends on prefix, not just local content), extended here to become the router's own view of prefix identity, independent of what any individual model server's internal PagedAttention block table looks like.

2.2 The in-memory LRU index of recently-routed prefixes

The router does not query replicas for their cache contents. Instead, the Endpoint Picker keeps an in-memory LRU index that records, for each prefix hash it has seen, which pod that prefix was last routed to2. When a new request arrives, the router hashes its prompt into the same block chain and looks each hash up in the LRU index: a hit means "the last time this router saw this prefix, it sent it to pod X, so pod X plausibly still has it cached." This is an inference from routing history, not a measurement of the pod's actual cache state — the index has no way to know if pod X evicted those blocks five minutes ago under memory pressure. That gap is exactly what precise indexing (section 3) closes, at a cost.

2.3 The prefix-cache-scorer: matched/total block ratio

Given the LRU index, the prefix-cache-scorer plugin scores each candidate pod by the fraction of the prompt's blocks whose hash maps to that pod in the index: matched blocks divided by total blocks in the prompt2. A pod that (as far as the index knows) has all of a prompt's prefix blocks scores 1.0; a pod with none scores 0.0; a pod with a partial prefix overlap scores somewhere in between. This score is one input to the weighted scorer set described in section 5 — it does not, by itself, decide the winning pod, because load and session signals are layered on top of it (section 6).

Approximate indexing (rolling hash chain plus LRU lookup) versus precise indexing (KVEvents subscription) side by side Approximate (section 2) Prompt → 16-token blocks rolling hash chain h1, h2, h3… EPP in-memory LRU index hash → last pod routed prefix-cache-scorer score = matched / total blocks no query to the model server can be stale after eviction Precise (section 3) vLLM/SGLang/TensorRT-LLM engine block added/evicted in real KV cache KVEvent emitted over ZeroMQ (ZMQ) pub/sub, per model server KV-Cache Indexer exact block → pod map, all pods correct as of last delivered event needs tokenization round-trip (§3.2)
Figure 2. Both mechanisms answer the same question — which pod has this prefix cached — from opposite ends: the approximate path infers it from the router's own routing history, the precise path is told directly by the engine every time its cache changes. Mechanism per llm-d's prefix-cache-aware routing documentation2.

Prefix-cache score for two candidate pods

Score A (matched / total)
Score B (matched / total)

Formula: score = matched blocks ÷ total prompt blocks, from the rolling hash chain's LRU lookup2. With an 8-block prompt (default above), a pod matching 6 blocks (0.75) outscores one matching 2 (0.25); the prefix-cache-scorer's output feeds into the weighted scorer set in section 5, so a higher prefix score does not automatically win if a competing pod is overloaded (section 6).

3. Precise KV-cache indexing: KVEvents over ZeroMQ

3.1 What triggers an event, what the indexer maintains

The precise path removes the inference step entirely. Model servers — llm-d's documentation names vLLM specifically — are configured to emit a KVEvent over ZeroMQ every time their internal KV cache changes: a block added, a block evicted2. A separate component, the KV-Cache Indexer, subscribes to these event streams from every pod in the pool and maintains a precise, globally consistent view of exactly which token blocks reside on which pods2. Where the approximate path's LRU index is a memory of what the router told pods to do, the precise path's index is a record of what the pods actually did to their own cache — it reflects real evictions, not just the passage of time since the last routing decision.

This is a straightforward publish/subscribe architecture: each model-server process is a ZMQ publisher, the KV-Cache Indexer is a subscriber aggregating across all of them. It is not synchronous — the indexer's view is only as current as the last event it has processed, a point that matters for section 8's staleness discussion.

3.2 Tokenization round-trip: the render endpoint

Block hashing, whether approximate or precise, has to operate on the same token boundaries the model server itself uses, or the hashes will not agree with what is actually cached. To get exact token IDs for an incoming prompt, llm-d's precise-mode token-producer plugin sends the prompt to vLLM's HTTP render endpoint, /v1/completions/render, before hashing it2. That is an extra network round trip to a model server — specifically, to some model server, since the router does not yet know which pod to send the request to when it needs the tokenization — on the way to deciding which pod to send the request to for actual generation. The approximate path avoids this entirely by hashing an approximation of the prompt (llm-d's own example: 16 tokens approximated as characters2), trading exact token-boundary alignment for zero extra round trips.

3.3 Cost/accuracy trade-off, summarized

Table 3 (section 7) compares the routers themselves; the table below isolates just this one design choice, since it recurs across every router in this module.

Table 1. Approximate versus precise prefix-cache indexing, as implemented in llm-d's scheduler2.
PropertyApproximate (block hash + LRU)Precise (KVEvents + ZMQ)
Source of truthrouter's own routing historyengine's actual cache state
Query to model servernonetokenization round-trip (render endpoint)
Reflects evictionsno — only reflects time/LRU eviction of the index itselfyes — event fires on real eviction
Extra subsystemnone beyond the EPP processZMQ pub/sub, KV-Cache Indexer component
Token-boundary accuracyapproximate (character-block heuristic)exact (real tokenizer)
Consistency windowstale as soon as a pod evicts without the router knowingbounded by event delivery latency, not by staleness of a guess

4. Session affinity vs. prefix affinity

4.1 Sticky routing by session ID

Prefix affinity, as built in sections 2 and 3, routes on the content of the prompt: any two requests that share a prefix should land on the same pod, regardless of which client sent them or in what order. Session affinity is a different, simpler idea: route on an opaque session or conversation identifier, so that every request tagged with the same session ID lands on the same pod for the life of that session, independent of what the prompt content actually is. llm-d implements this as its own scheduler plugin, session-id-producer, which supplies session-affinity data as one more signal the scorer set can weight3. vLLM's production-stack router implements the same idea at the router level directly: its request router supports session-ID-based routing as one of its three routing modes, alongside round-robin and Kubernetes service discovery4.

4.2 When they agree, when they conflict

For a single-turn request, session affinity and prefix affinity are nearly indistinguishable: a session's first request has no cache anywhere, so whichever pod session affinity happens to pick becomes the pod prefix affinity would also pick for every subsequent turn in that conversation — the two signals converge because the session is the growing prefix. This is the common case multi-turn chat routing optimizes for, and it is why session-ID routing is a defensible default even without any block-hashing machinery at all (vLLM production-stack's default routing modes do not require the LRU index or KVEvents subsystem of sections 2–34).

The two signals diverge in at least three situations a pure session-ID router cannot see. First, a shared system prompt or a shared retrieved document used across many independent sessions: prefix affinity would route all of them to whichever pod already has that shared prefix cached, while session affinity, keyed on a per-conversation ID, scatters them across the fleet exactly as a stateless policy would, gaining nothing from the shared prefix. Second, a session that resumes after the routing pod has been drained or has failed: session affinity has no fallback signal once its target pod is gone, whereas prefix-aware scoring can still find the next-best pod by content match. Third, load imbalance: a session pinned to a pod that happens to be overloaded stays pinned to it under pure session affinity, whereas a scorer set that layers load signals on top of affinity (section 6) can route the session elsewhere at the cost of a cold cache, trading a known recompute cost against an unknown queueing delay.

5. The layered scheduling pipeline: filter, score, select

5.1 llm-d's scheduler: Request Control, Filtering, Scoring, Selection

llm-d's inference scheduler documents its own pipeline as a sequential chain: a Request Control stage handles flow-control admission, where a global Screener component does preliminary endpoint filtering; a Filtering stage runs pods through a sequential chain of filters that can eliminate candidates outright (the decode-filter plugin, for example, filters out pods not currently in the decode stage); a Scoring stage runs the surviving pods through a weighted set of scorers — prefix-cache-scorer and any load- or session-affinity scorers configured — each contributing a weighted term; and a Selection stage where the highest-scored pod is chosen, via the max-score-picker plugin3. This is a filter-then-rank pipeline, not a single formula: filtering removes pods that are categorically wrong for this request (wrong lifecycle stage, admission-rejected) before scoring ever runs, and scoring only has to rank pods that already passed every hard constraint.

The data underneath every scorer follows what llm-d calls a Source → Extract → Attribute lifecycle: raw signals (KVEvents, load metrics, session IDs) are sourced from wherever they originate, extracted into a normalized form, and attributed to specific endpoints in a shared datastore that the scoring stage reads from3. The scheduler's stated design principle is pluggability: new scorers or filters can be added without changing the pipeline's core code, which is exactly what lets a deployment mix and match the approximate producer, the precise indexer, session affinity, and load signals from sections 2–4 and 6 as separately configurable plugins rather than a monolithic policy3.

5.2 The Gateway API Inference Extension's EPP contract

llm-d's scheduler is one implementation of a pattern the Kubernetes Gateway API Inference Extension (GAIE) standardizes at the API level for any EPP, not just llm-d's. GAIE's own architecture proposal describes requests flowing through three layers: a Routing Layer that applies per-model routing rules and enriches the request with metadata; a Flow Controller that handles priority, fairness, and queueing; and a Scheduling Layer, described as "the load balancing algorithm that intelligently routes requests based on the current state of the InferencePool"5. The CRD-level plumbing that connects this to an actual Kubernetes gateway — the InferencePool object's fields, how an HTTPRoute references it, Envoy's ext-proc wiring — is Module 10's territory; what matters here is that GAIE's three-layer architecture and llm-d's four-stage pipeline describe the same computation from two altitudes.

5.3 How the two map onto each other

GAIE's Flow Controller (priority, fairness, queueing) corresponds to llm-d's Request Control stage (flow-control admission, the Screener). GAIE's Scheduling Layer (load-balancing on current pool state) corresponds to llm-d's combined Filtering + Scoring + Selection stages — GAIE names the mechanism at the level of "an algorithm that routes based on pool state," llm-d specifies what that algorithm actually is: sequential hard filters, then a weighted soft-scoring pass, then arg-max selection. GAIE's Routing Layer (per-model rules, request enrichment) sits logically upstream of both, since it decides which InferencePool a request even reaches before any pod-level filter or scorer runs.

llm-d's four-stage scheduler pipeline mapped onto the Gateway API Inference Extension's three-layer EPP architecture llm-d scheduler stages Request ControlScreener, admission Filteringdecode-filter, etc. Scoringprefix-cache-scorer+ load/session scorers Selectionmax-score-picker GAIE EPP layers (same computation, three levels) Flow Controllerpriority, fairness,queueing Scheduling Layer"load balancing algorithm that routesbased on current InferencePool state"
Figure 3. llm-d's Filtering, Scoring, and Selection stages are one concrete instantiation of what the Gateway API Inference Extension's architecture proposal calls, at the API level, the Scheduling Layer; Request Control maps to the Flow Controller. Pipeline per llm-d's scheduler architecture3; layer definitions per GAIE's EPP architecture proposal5.
Sequence diagram of one request's path from client to selected replica Client Gateway EPP Candidate pods Selected pod HTTP request Envoy ext-proc call score request (filter) filtered candidates + scores select max forward request tokens (streamed) relay response HTTP response
Figure 4. The EPP is invoked by the gateway over Envoy's external-processing protocol; it runs filtering and scoring against the InferencePool's candidate pods, selects the highest-scored one, and the gateway forwards the request there directly — the EPP itself does not sit on the token-streaming data path. Per GAIE's description of the Endpoint Picker as a data-plane component using Envoy ext-proc1 and llm-d's filter/score/select pipeline3.

6. Load- and SLO-aware scoring layered on prefix affinity

6.1 decode-filter and inflight-load-producer

A pod with a perfect prefix-cache score is still a bad choice if it is already saturated: routing every request for a hot shared prefix to the one pod that happens to hold it would recreate, inside the cache-aware router, exactly the hot-spotting problem stateless load balancing was supposed to prevent. llm-d's scheduler ships two plugins that address this directly. decode-filter runs in the Filtering stage (section 5.1) and removes pods that are not currently in the decode stage from consideration — relevant once prefill/decode disaggregation (Module 3 onward) is in play, since a request needing decode service should never be filtered onto a prefill-only pod3. inflight-load-producer supplies a load signal — in-flight request count or queue depth — into the same data layer that feeds the Scoring stage, so it becomes one more attribute a load-aware scorer can weight alongside the prefix-cache score3.

6.2 Weighted scorer combination

Because llm-d's Scoring stage runs a weighted set of scorers rather than a single formula (section 5.1), prefix affinity and load are combined, not chosen between: a pod's final score is a weighted sum across whichever scorers are configured — prefix-cache-scorer, a load-based scorer built on inflight-load-producer's data, and optionally a session-affinity scorer — and max-score-picker selects the pod with the highest combined score3. The scheduler's own documentation does not publish default weight values for this combination, so the practical effect of raising or lowering any one scorer's weight is a deployment-specific tuning decision, not a fixed constant this module can give a number for; what is fixed is the mechanism — every scorer contributes an independent, weighted term, and the pluggable architecture (section 5.1) means new signals can be added as new scorer plugins without touching the combination logic itself3.

The practical effect is that a pod with a high prefix-cache score does not automatically win: a heavily loaded pod holding a perfect prefix match can still lose to a lightly loaded pod holding a partial match, if the load scorer's weighted contribution outweighs the prefix scorer's. This is the mechanism that keeps the approximate/precise indexing of sections 2–3 from creating its own hot-spotting problem — the same weighted-sum architecture that lets prefix affinity and session affinity coexist (section 4.2) is what lets prefix affinity and load-awareness coexist here.

The arithmetic itself is nothing more than a weighted sum, but it is worth writing out once because the crossover point — where a load penalty overtakes a prefix-cache advantage — is where the thundering-herd failure mode of section 8 actually originates. With a prefix-cache score \(s_{pfx} \in [0,1]\) (section 2.3), a normalized load score \(s_{load} \in [0,1]\) (1 = idle, 0 = saturated), and scorer weights \(w_{pfx}, w_{load}\) set by the deployer, a pod's combined score is

$$ s = w_{pfx}\, s_{pfx} + w_{load}\, s_{load} $$

Table 2 below plugs in illustrative weights (llm-d's documentation does not publish defaults3, so treat these as worked numbers, not a citation) to show the crossover concretely: at equal weights, a fully idle cold pod (\(s_{pfx}=0\), \(s_{load}=1\)) ties a fully warm but half-loaded pod (\(s_{pfx}=1\), \(s_{load}=0.5\)) at \(s = 0.5\), and beats any pod running above half load regardless of how much of the prefix it holds.

Table 2. Illustrative combined scores at equal weights (\(w_{pfx}=w_{load}=0.5\)), showing where an idle cold pod overtakes a busier warm one — not sourced defaults, a worked instance of the section 6.2 formula.
PodPrefix scoreLoad scoreCombined score
Warm, lightly loaded1.000.800.90
Warm, half loaded1.000.500.75
Cold, idle (newly joined)0.001.000.50
Warm, nearly saturated1.000.100.55

Read down the table: a fully warm pod only loses to the cold pod once its own load score drops below the cold pod's load score by more than its full prefix-score advantage divided by the weight ratio — at equal weights, once the warm pod is busier than the cold pod is idle. That crossover is unremarkable for one request. It becomes the thundering-herd failure mode of section 8 when many concurrent requests each independently cross it against the same cold pod at once, because the load scorer has no way to account for requests still in flight to that pod from other, concurrent scoring decisions within the same routing instant.

7. Three routers compared: llm-d, vLLM production-stack, sglang-router

The mechanisms above are llm-d's; they are not the only production router design in the open-source stack. vLLM's own reference deployment, production-stack, ships a simpler router, and SGLang ships a standalone Rust router built specifically for prefill/decode-aware routing. All three sit behind the same architectural slot — a request-routing layer in front of a pool of stateful engine replicas — but differ in how much KV-cache-awareness they build in versus assume the deployer will add separately.

vLLM's production-stack README describes a three-component reference architecture — the serving engine (the vLLM instances themselves), a request router, and an observability stack (Prometheus and Grafana) — explicitly designed to scale from a single vLLM instance to a distributed deployment without application-code changes4. Its router supports round-robin routing, session-ID-based routing, and automatic Kubernetes service discovery4; KV-cache offloading is available as a separate integration via LMCache rather than as router-native prefix scoring4 (LMCache itself is Module 8's subject). sglang-router is described on its package page as "a high-performance Rust-based load balancer for SGLang with multiple routing algorithms and prefill-decode disaggregation support," currently at version 0.3.2, released 2026-01-156 — its distinguishing design goal, unlike either llm-d's Python/Go scheduler plugin chain or production-stack's router, is PD-aware routing built into a single dependency-light Rust binary. SGLang's own PD-disaggregation documentation (Module 12's subject) describes the deployment topology this router targets, not the router's internal scoring formula, which sglang-router's own package page does not itemize at the level of detail llm-d's scheduler documentation does.

Table 3. Router feature comparison across the three routers this module can source in detail3246.
Featurellm-d scheduler / EPPvLLM production-stack routersglang-router
ImplementationGo/Python plugin chain behind an EPPPython request routerRust binary
Prefix-cache awarenessapproximate (hash+LRU) and precise (KVEvents) modesvia separate LMCache integration, not router-nativenot itemized in package metadata6
Session affinitysession-id-producer pluginsession-ID routing, native modenot itemized in package metadata6
Load awarenessinflight-load-producer + weighted scorernot a named routing mode4"multiple routing algorithms," unspecified in package metadata6
PD-disaggregation awarenessdecode-filter pluginnot nativenative, stated design goal6
Service discoveryInferencePool (Module 10)automatic Kubernetes service discovery4not itemized in package metadata
Extensibility modelexplicit plugin architecture, no core changes to add scorers3fixed set of three routing modesfixed set of routing algorithms

"Not itemized in package metadata" means sglang-router's PyPI page does not describe the property at the level of detail this table needs6 — not that the feature is absent. SGLang's PD-disaggregation documentation (Module 12) covers its transfer-backend and deployment mechanics in more depth than the router package page alone.

8. What learning KV state costs: staleness and failure modes

Every mechanism in this module buys accuracy at some operational price, and every one of those prices shows up as a specific failure mode rather than a uniform "sometimes wrong." Three are worth naming precisely, because each has a different root cause and a different mitigation, and conflating them leads to fixing the wrong layer.

Stale index after eviction. The approximate path's LRU index (section 2.2) records where the router sent a prefix, not what the pod's cache still holds. If a pod evicts blocks under memory pressure — ordinary paged-KV-cache behavior Aman already knows from single-node serving — the router's index has no signal that the eviction happened until enough time passes for its own LRU policy to age the entry out independently. In the interim, the prefix-cache-scorer keeps awarding a high score to a pod that will, in fact, have to recompute the prefix it appears to have cached. This is precisely the gap precise indexing (section 3) closes: a real KVEvent fires the moment the engine evicts a block, so the KV-Cache Indexer's view corrects itself as fast as the event pipeline delivers the message, not as fast as an independent LRU-aging heuristic happens to catch up2.

Event-delivery staleness window. Precise indexing is not instantaneous either. ZMQ pub/sub (section 3.1) delivers events asynchronously; the KV-Cache Indexer's global view is correct as of the last event it has processed, not as of the current instant2. Under normal operation this window is small, but it is nonzero by construction — there is no synchronous barrier between a model server mutating its cache and the router's copy of that fact updating. A router that treats the precise index as a real-time oracle rather than an eventually-consistent replica is making the same category of mistake the approximate path makes, just with a much smaller error window.

Thundering herd on a newly-warm replica. Figure 1's replica N−1 illustrates the third failure mode. A pod that just joined the pool has an empty prefix-cache footprint by construction — the prefix-cache-scorer correctly scores it near zero for essentially every incoming prefix. But inflight-load-producer (section 6.1) also correctly reports that pod as having near-zero in-flight load, since it has served nothing yet. If the load scorer's weight is high relative to the prefix scorer's, the weighted combination (section 6.2) can rank the empty pod as the best choice for many requests simultaneously — each one individually a reasonable "route to the least-loaded pod" decision, collectively a burst of full-prefill traffic landing on one pod at once, the opposite of the gradual warm-up a newly joined replica actually needs. Neither scorer is wrong in isolation; the failure is an interaction between two correct signals under a specific weighting.

Table 4. Three routing failure modes, their root cause, and how the mechanisms in this module bound them23.
Failure modeRoot causeBounded by
Stale prefix index after evictionLRU index tracks routing history, not real cache stateprecise KVEvents indexing (§3)
Event-delivery staleness windowZMQ pub/sub is asynchronous, not a synchronous barrierevent-pipeline latency (small, not zero)
Thundering herd on a cold-but-idle replicaload scorer and prefix scorer disagree, load wins by weightscorer-weight tuning; no plugin eliminates it structurally

Where experts disagree

The disagreement in this module is not about whether KV-cache-aware routing helps — every source agrees it does — but about how much machinery is worth building to get it, and whether "cache-aware" should mean an approximate, cheap signal or a precise, more expensive one. Three positions show up across the sources this module draws on.

Approximate hashing is the pragmatic default

llm-d's own scheduler documentation, describing its default prefix-cache-aware mode2

Block hashing plus an in-memory LRU index requires no query to any model server, no extra subsystem beyond the EPP process itself, and no tokenization round trip. It is a router-local, stateless-with-respect-to-the-engine mechanism that scales to any fleet size the router itself can handle, and it is the mode llm-d documents first and treats as the baseline before introducing precise indexing as an alternative.

  • Zero extra network calls to model servers per request.
  • No additional service (ZMQ broker, indexer process) to operate or fail independently.
  • Scales with router capacity alone, not with per-pod event throughput.
  • Can be stale after an eviction the router has no way to observe (§8).
  • Character-approximated block boundaries (llm-d's own 16-token example) do not guarantee alignment with the engine's real tokenizer.

Precise event-streaming is worth the cost for exact correctness

llm-d's precise-mode documentation, describing KVEvents/ZMQ indexing as the alternative to approximate scoring2

An index built from real eviction events, sourced directly from the engine, is correct as of the last delivered event rather than correct as of an inference about routing history. For deployments where a stale routing decision is expensive — large models where a wrong guess means re-executing a long, compute-heavy prefill — the tokenization round trip and the ZMQ subsystem are a small price for removing an entire class of silent mis-routing.

  • Reflects the engine's actual cache state, not a heuristic proxy for it.
  • Staleness is bounded by event-delivery latency, not by an independent LRU-aging policy that has no relationship to real evictions.
  • Adds a tokenization round trip to vLLM's render endpoint before every routing decision (§3.2).
  • Requires an additional always-on component (the KV-Cache Indexer) subscribing to every pod's event stream.

Skip cache-awareness in the router; use session or round-robin

vLLM production-stack's default routing modes (round-robin, session-ID, Kubernetes service discovery)4, and sglang-router's framing around PD-aware rather than prefix-aware routing as its distinguishing feature6

Neither of the two other major open-source routers this module can source in detail ships llm-d's block-hashing-plus-LRU or KVEvents machinery as a router-native feature. vLLM production-stack routes on session ID or round-robin and treats prefix-cache reuse as LMCache's job (Module 8), a separate integration rather than router logic. sglang-router's stated distinguishing feature is PD-aware routing, not prefix-cache scoring. Read together, this implies at least part of the field does not consider router-level prefix-cache scoring worth the added complexity relative to simpler affinity signals plus a separate caching layer.

  • Far less router-internal complexity; no scorer weights to tune.
  • Session-ID routing captures most of the benefit for the common multi-turn case (§4.2).
  • Loses the shared-prefix-across-sessions case entirely (§4.2) unless a separate caching layer compensates.
  • No native load-aware combination with cache affinity; scaling decisions and cache decisions are made by different components.
Assessment

These are not really competing claims about the same fact; they are different bets about where in the stack cache-awareness should live. llm-d's position is defensible when the router is the natural place to make this decision because it already sits on every request's path and the fleet is large enough that redundant prefill is a real cost. Production-stack's simpler routing is defensible when LMCache or an equivalent caching layer (Module 8) already captures most of the reuse benefit independently of router sophistication, and the deployment values fewer moving parts over marginal routing precision. Within llm-d's own two modes, the approximate/precise choice is a genuine cost-accuracy trade-off rather than one option dominating the other: approximate is the safer default until a workload's redundant-prefill cost (measured, not assumed) exceeds the tokenization round-trip and indexer-operation cost of going precise. None of the sources in this module's scope publish a controlled throughput or TTFT comparison between the three routers under identical traffic, so the honest answer is that the choice is currently made on architectural fit and operational preference, not on a published benchmark that would let you rank them numerically; a study holding traffic pattern, model, and hardware fixed across all three routers would settle it.

What is changing

The Gateway API Inference Extension itself lists "prefix-cache aware load balancing" as a roadmap goal at the API-standard level, alongside KV-cache- and request-cost-aware scheduling and LoRA-adapter-aware routing1 — meaning the pattern this module describes as llm-d's implementation choice is on a path toward becoming a Kubernetes-standard capability any EPP-based router can claim, not just llm-d's own scheduler. All three routers in Table 3 are under active development on the same 2026-09-17 snapshot this module cites; sglang-router's 0.3.2 release date (2026-01-15) is eight months old relative to that snapshot, and llm-d's scheduler documentation carries no version tag at all, describing itself only as the main branch3 — both signals that the specific plugin names, default weights, and API surfaces in this module are more likely to shift than the underlying filter-score-select pattern itself.

References

  1. repoKubernetes SIG contributors. kubernetes-sigs/gateway-api-inference-extension (README). Kubernetes SIGs, GA since v1.0.0, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension. Accessed 2026-09-17. Used for: Endpoint Picker as a data-plane component using Envoy ext-proc; KV-cache- and request-cost-aware scheduling goals; prefix-cache-aware load-balancing roadmap item.
  2. officialllm-d contributors. Prefix-Cache Aware Routing (llm-d docs). llm-d, docs/dev (live), 2026. https://llm-d.ai/docs/dev/architecture/advanced/kv-management/prefix-cache-aware-routing. Accessed 2026-09-17. Used for: approximate block-hashing/rolling-hash-chain/LRU mechanism, precise KVEvents/ZMQ mechanism, tokenization round-trip via the render endpoint.
  3. repollm-d contributors. llm-d-inference-scheduler architecture.md. llm-d (CNCF Sandbox; Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA), main branch, 2026. https://github.com/llm-d/llm-d-inference-scheduler/blob/main/docs/architecture.md. Accessed 2026-09-17. Used for: filter-score-select pipeline stages, named plugins (prefix-cache-scorer, decode-filter, max-score-picker, session-id-producer, inflight-load-producer), Source-Extract-Attribute data layer, pluggability design principle.
  4. repovLLM Project contributors. vllm-project/production-stack (README). vLLM Project, released 2026-01-22. https://github.com/vllm-project/production-stack. Accessed 2026-09-17. Used for: three-component reference architecture, session-ID/round-robin/service-discovery routing modes, LMCache integration.
  5. repoKubernetes SIG / Gateway API Inference Extension contributors. EPP Architecture Proposal (0683-epp-architecture-proposal). Kubernetes SIGs, main branch proposal doc, 2026. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/docs/proposals/0683-epp-architecture-proposal/README.md. Accessed 2026-09-17. Used for: three-layer EPP architecture (Routing Layer, Flow Controller, Scheduling Layer).
  6. repoSGLang contributors. sglang-router (PyPI package page). SGLang / RadixArk, 0.3.2, 2026-01-15. https://pypi.org/project/sglang-router/. Accessed 2026-09-17. Used for: Rust-based load balancer description, PD-disaggregation support, version number.