System Design · Module 21
Edge and API Gateway for AI Inference Platforms
Every inference platform Aman builds against sits behind a gateway before it ever reaches a GPU, and that gateway does five specific jobs in a specific order: check identity and budget, check rate limits, pick a backend, call the provider, and record what happened without slowing the response down. This module derives that lifecycle from the gateways that have published it -- LiteLLM, the Kubernetes Gateway API Inference Extension, Kong AI Gateway, Azure API Management, Cloudflare AI Gateway -- and gives every later module in this part a numbered stage to attach to.
1. The gateway's place in the request path
Module 1 established the L7 reverse-proxy vocabulary: TLS termination, routing on host/path, and load balancing by round robin, least-connections, ip-hash, or weight across an interchangeable pool of backends2. An AI gateway sits in exactly that position in the topology -- the first hop inside your trust boundary that terminates the client's connection -- but it is answering a different question. A classical L7 balancer asks "which of these identical backends should handle this request." An AI gateway asks "which model, under whose budget, at what priority, and did this key already burn its quota this minute" before it asks anything about backend health at all. The identity and quota questions are Module 22's subject in depth; this module's job is to fix the lifecycle those questions sit inside, so Module 22 can cite a stage number instead of re-deriving the whole request path.
The gateway is also a trust boundary in Module 10's sense: it is the point where an external, unauthenticated bearer token becomes an internal, authorized, budgeted principal. Everything downstream of the gateway -- router, provider adapter, backend fleet -- can assume that boundary has already been crossed correctly. That assumption is why the gateway has to get identity and budget checks right on every request rather than sampling or deferring them, even though (as section 6 shows) the bookkeeping those checks produce is deliberately allowed to lag.
There is a second, quieter continuity with Module 20: the router's job of translating a caller's request into whatever wire format a specific provider expects is the same problem Airbnb's Service IDL solved for internal service boundaries -- a declared contract that lets the caller and the callee change independently, as long as both honor the contract1. Module 20 covered that contract for services one company owns end to end; an AI gateway's router enforces the same discipline across provider boundaries it does not control at all, which is why the translation logic in stage 2.3 below has to be maintained continuously against a moving target rather than agreed once and left alone.
2. The request lifecycle, stage by stage
LiteLLM's own architecture documentation is the most precise public description of this lifecycle available, because LiteLLM is a proxy whose entire job is being this lifecycle, and it names each stage explicitly1. The stages generalize past LiteLLM specifically: Kong AI Gateway's plugin chain, Azure API Management's policy pipeline, and the Kubernetes Gateway API Inference Extension's Endpoint Picker all implement the same ordering under different names, which section 4 compares directly.
Coverage note: OpenAI and Anthropic have not published a gateway architecture document comparable to LiteLLM's or the Inference Extension's. What each has published publicly is the caller-visible behavior of usage tiers, spend limits, and rate limits -- the input to Module 22's identity and rate-limit design, not the internal lifecycle of their own edge tier. This module does not infer or fabricate either company's internal gateway mechanics beyond what their own documentation states about caller-visible behavior.
2.1 Auth and budget check
The request arrives with a bearer token. The gateway looks up whether that token (a "virtual key" -- section 5 covers the concept in depth) is valid and under its budget. LiteLLM's documentation is explicit about the lookup path: a cache lookup first, with a database fallback only on a cache miss1. That ordering matters mechanically, not just for latency: a database round trip on every request would put a synchronous write-adjacent read in the critical path of every single call, which is the exact anti-pattern Module 8 and Module 11 already flagged for high-throughput systems. A cache-first check means the enforcement is only as fresh as the last cache write, which reappears as a real consistency problem in section 6.
2.2 Parallel-request rate limiter
Once the key is authenticated and within budget, a separate check asks whether it is within its rate limit -- requests per minute and tokens per minute, evaluated at the key, user, and team level simultaneously1. This is a distinct mechanism from the budget check in stage 2.1: a key can be well within its dollar budget and still get a 429 for sending too many requests in too short a window. Module 22 derives the token-bucket, fixed-window, and sliding-window algorithms that implement this stage and the disagreement about which one is correct; this module only fixes where the check sits in the pipeline -- after identity, before routing -- and notes that "parallel" in LiteLLM's own naming reflects that the limiter runs as an independent check next to, not serialized inside, the budget check.
2.3 Router
The router is where an AI gateway earns its name over a generic reverse proxy. LiteLLM's router performs load balancing across configured deployments of a model, applies fallbacks when a deployment errors, retries transient failures, and translates the caller's request into the target provider's SDK format1. Section 3 derives why this stage cannot be satisfied by Module 1's round-robin or least-connections algorithms alone once the backends are stateful GPU-backed model servers rather than interchangeable stateless web servers.
The SDK-translation half of the router's job is not cosmetic. A caller writing against one schema (commonly the OpenAI Chat Completions shape, since it is the de facto lingua franca) expects the gateway to speak whatever the chosen backend actually requires. Envoy AI Gateway's own usage-based rate-limiting documentation gives a concrete instance of why this matters beyond convenience: token-usage accounting itself depends on the response having a recognizable shape, so the gateway "automatically captures token usage through its request/response transformer" for a provider like AWS Bedrock whose native response format is not OpenAI-shaped10. If the translation layer is wrong or incomplete for a given backend, stage 2.5's accounting silently degrades along with it -- the two concerns are coupled even though they sit in different stages of the lifecycle.
2.4 Provider call
The gateway makes the actual call to the model backend -- an internal GPU fleet, or an external provider's API. This is the boundary Module 25 designs in detail: what signals the backend exposes back across this call (KV-cache utilization, queue depth) so the router in stage 2.3 can make a better decision next time, and what a stable contract across a backend upgrade must guarantee. This module treats the call itself as a single edge in the lifecycle diagram; Module 25 opens it up.
2.5 Async, off-critical-path accounting
After the response is already on its way back to the client, LiteLLM's documentation states plainly that spend logging, rate-limit accounting, and any configured logging callbacks all run as background tasks -- "no database write sits in the request path"1. Section 6 treats this as more than an implementation detail: it is a deliberate consistency trade, and it creates a window during which the system's view of a key's spend and rate-limit state is stale relative to reality.
| Stage | Checks / does | Timing | Failure mode |
|---|---|---|---|
| 1. Auth + budget | Bearer token valid, key/user/team under max spend | Sync, cache-first | 401 (invalid), 429/402-style rejection (over budget) |
| 2. Rate limiter | rpm/tpm at key, user, team level | Sync, parallel to stage 1 | 429 Too Many Requests |
| 3. Router | Load balance, fallback, retry, SDK translation | Sync | Fallback to next deployment or provider |
| 4. Provider call | Actual inference call | Sync | Timeout, error surfaced to router for retry |
| 5. Accounting | Spend logging, rate-limit counters, callbacks | Async, off critical path | Delayed enforcement, not request failure |
3. Why generic L7 balancing breaks for inference traffic
The Kubernetes project's own announcement of the Gateway API Inference Extension states the problem as plainly as any source in this course states anything: "unlike typical short-lived, stateless web requests, LLM inference sessions are often long-running, resource-intensive, and partially stateful. For example, a single GPU-backed model server may keep multiple inference sessions active and maintain in-memory token caches"3. Every assumption behind Module 1's load-balancing algorithms breaks against that sentence.
3.1 Stateful, long-running, partially cached sessions
Round robin and least-connections both assume that any backend in the pool can serve any request equally well, so the only question worth optimizing is which backend is currently least busy. That assumption holds for a fleet of stateless web servers reading from a shared database. It does not hold for a fleet of model replicas: a request that hits a replica already holding a matching prefix in its KV cache is cheaper to serve there than anywhere else in the fleet, and a request that hits a replica with a cold cache pays the full prefill cost regardless of that replica's current connection count. Least-connections has no way to see cache state at all -- it counts connections, not bytes of reusable KV cache -- so it will happily send a request to an idle-but-cold replica over a busier-but-warm one, forfeiting exactly the reuse the platform is paying GPU memory to hold.
Session length compounds the problem. A stateless web request completes in milliseconds; an LLM generation can run for seconds to minutes while holding GPU memory the whole time. A load balancer that treats connection count as a proxy for load will systematically under-weight a backend serving a few long, expensive generations and over-weight one serving many short, cheap ones, because "number of active connections" was never the right unit of load for this workload in the first place.
3.2 The signals a generic balancer does not have
The same announcement names the two things a classical balancer is missing: "traditional load balancers focused on HTTP path or round-robin lack the specialized capabilities needed for these workloads. They also don't account for model identity or request criticality (e.g., interactive chat vs. batch jobs)"3. The Gateway API Inference Extension's own architecture doc names the two routing signals it adds instead: KV-cache utilization and queue depth on each candidate model server4. Module 25 designs the full contract for how a GPU backend exposes those two signals across the platform boundary and what a stable interface must guarantee across a backend topology change; this module only needs the fact that they exist and that a generic L7 balancer has no field to carry them in.
"Model identity" is the more basic gap. A classical balancer routes on host and path -- fine when every backend behind a path serves the same content. An inference gateway's router must route on model name, and frequently on model version, because two backends behind the same logical endpoint may be running different checkpoints, different quantizations, or entirely different model families with different latency and cost profiles. Path-based routing degenerates into a proxy for model identity only if the operator is disciplined about giving every model its own path -- the Inference Extension instead makes model identity a first-class field the router reads directly, via an InferencePool object that represents a set of endpoints running a given model-server framework, with the Endpoint Picker (EPP) fetching live metrics from that pool's endpoints to choose the one that "best achieve[s] the configured objectives"4.
"Request criticality" is the newer gap, and it is the one this module leaves narrowest, on purpose. The Inference Extension's own architecture documentation claims that model-server-aware, metrics-driven load balancing "has been proven to reduce the serving latency and improve accelerator utilization" relative to generic L7 balancing4, but a routing decision informed by KV-cache utilization and queue depth still has to decide what to do when every candidate backend is saturated -- whether an interactive chat request should preempt a batch job's slot, and what a caller is told when nothing is available at all. That is a scheduling and admission-control question, not a routing-signal question, and Module 24 designs it: SLO-tiered service classes, fallback and graceful-degradation ordering, and the point at which the gateway stops trying to route and starts queueing or rejecting.
4. The ext-proc pattern: extend an existing gateway instead of building one
The Gateway API Inference Extension's own architecture documentation states its design choice directly: it works by "leveraging Envoy's External Processing Protocol to extend any gateway that supports both ext-proc and Gateway API into an inference gateway"4. That sentence is the whole idea. Instead of writing a new data plane that terminates TLS, parses HTTP/2, pools connections, and retries on your behalf -- all of which Envoy (or any comparable proxy) has already solved and hardened over years of production traffic -- you leave the data plane alone and insert a hook into it that lets an external process see and modify each request.
4.1 The ext_proc mechanism
Envoy's External Processing filter (ext_proc) is a bidirectional gRPC stream between Envoy and an external processor: Envoy sends the processor ProcessingRequest messages and the processor replies with ProcessingResponse messages5. The filter can be configured to send events at any subset of four points in a message's lifecycle -- request headers, request body, response headers, response trailers -- and the processor can change which of those points it wants to see on a message-by-message basis, which lets a handler subscribe only to the events it actually needs and avoid unnecessary round trips5. The processor can examine and modify headers, body, and trailers, or short-circuit the whole exchange by returning a brand-new response directly5. For an inference gateway, this is exactly the hook a routing decision needs: intercept the request headers (to read the model name and the caller's identity), call out to a process that has live KV-cache and queue-depth telemetry from the backend fleet, and get back a modified request pointed at the chosen replica -- all without Envoy itself knowing anything about models, KV caches, or queues.
The per-message event subscription matters for exactly the latency reason the design guide for this stack calls out: a routing decision only needs the request headers, since model name and caller identity both travel there, so a well-built processor subscribes to header events only and skips the request-body callback entirely, avoiding the added round trip of streaming the full prompt body through the external processor before making a decision it did not need the body for5. A processor that also wants to inspect or transform the body (to rewrite a prompt, or to compute a request's true token count before the provider call) pays for that visibility with an additional callback in the same gRPC stream -- the mechanism supports it, but every subscribed event is one more round trip against the request's total latency budget.
The Endpoint Picker (EPP) in the Gateway API Inference Extension is that external processor. It is described as "an implementation of an Inference Router that will fetch metrics from whichever portion of the InferencePool endpoints can best achieve the configured objectives"4 -- in ext_proc terms, it is the gRPC server on the other end of the stream, and Envoy (or any Gateway-API-and-ext-proc-capable proxy) is the unmodified data plane in front of it.
4.2 Three implementations of the same idea
The pattern is not unique to Kubernetes. Kong AI Gateway's second major version replaces the older, AI-specific "AI Proxy" plugin with an "AI Model" entity that "defines the upstream provider, model name, and routing behavior," while attaching AI-specific policies -- AI Semantic Cache, AI Prompt Guard, AI Rate Limiting Advanced -- on top of Kong's existing, plugin-centric request-processing pipeline6. The AI capability is new; the pipeline it plugs into is the same one Kong has run non-AI traffic through for years.
Azure API Management states the relationship even more directly for its own AI gateway feature set: "the AI gateway, including MCP server capabilities, extends API Management's existing API gateway; it's not a separate offering"7. The token-limit policy that enforces a tokens-per-minute quota (llm-token-limit), the semantic-cache lookup/store policies, and the load-balancing and circuit-breaker configuration on backends are all additions to the same policy pipeline that has always processed API Management's REST traffic -- not a parallel system7.
The Gateway API Inference Extension project also separates who configures each side of this split into two named roles: an "Inference Platform Admin," who "creates and manages the infrastructure necessary to run LLM workloads," including the gateway configuration itself, and an "Inference Workload Owner," who owns one or more model-serving workloads and is responsible for defining that workload's request priorities and rollout of model/adapter changes4 (Concepts: Roles and Personas). That split matches the extension pattern structurally: the platform admin operates the shared, extended gateway (the thing sections 4.1-4.2 describe), while the workload owner's decisions -- what counts as high priority, which adapter is live -- are exactly the configuration the EPP reads at request time rather than something baked into the gateway itself. Module 24 is where request priority, defined by the workload owner in this framing, becomes a scheduling and admission decision rather than just a label attached to a request.
Three organizations, three different underlying proxies (Envoy via the Gateway API, Kong's own nginx-based core, Azure's own gateway runtime), and the same architectural answer: keep the hardened data plane, add an AI-aware policy or processing layer beside it. The disagreement section below contrasts this extension pattern with the alternative of not reusing an existing gateway at all.
| Gateway | Deployment model | Extension mechanism | Inference-aware signal | Caching |
|---|---|---|---|---|
| LiteLLM | Self-run Python proxy | Application code (not a data-plane extension) | Model/deployment config, not live KV-cache telemetry1 | Optional response cache, app-level |
| K8s Gateway API Inference Extension | Self-run, on top of an existing Envoy/Gateway-API proxy | ext_proc gRPC hook (Endpoint Picker)4 | KV-cache utilization, queue depth3 | None specified |
| Kong AI Gateway | Self-run, on top of Kong's existing core | AI Model entity + policy plugins6 | Routing behavior per AI Model config6 | AI Semantic Cache plugin6 |
| Azure API Management | Managed PaaS, existing APIM instance | AI-gateway policies on existing policy pipeline7 | Round-robin / weighted / priority / session-aware backend LB7 | llm-semantic-cache-lookup/store7 |
| Cloudflare AI Gateway | Fully managed edge service | Vendor-hosted proxy in front of provider APIs12 | None (routes to named provider) | Exact-match, disabled by default13 |
5. Virtual keys: identity as a spend and rate-limit hierarchy
Stage 2.1's auth/budget check is a lookup against a "virtual key" -- a credential the platform issues to a caller, distinct from the underlying provider API key it maps to. LiteLLM's virtual-key documentation gives the most complete public description of how this hierarchy is structured, and it generalizes to every gateway that separates "who is calling" from "which upstream credential actually pays the provider."
5.1 Key, user, team: rollup and inheritance
A virtual key's owner is whoever its user_id field points at, and a key can additionally carry a team_id8. Spend is tracked at all three levels independently: per-key spend is retrievable directly, and "if the key has an attached user_id or team_id, the spend for that user is tracked in the LiteLLM_UserTable, and team in the LiteLLM_TeamTable"1. Every completion, chat completion, and embeddings call updates these three running totals in USD, computed from the provider's own per-token pricing8.
Rate limits (tpm_limit, rpm_limit) and spend caps (max_budget) follow an inheritance model rather than three independently-set numbers: a key without its own explicit limit inherits its owner's limit -- "the owner's tpm_limit and rpm_limit whenever they are set, and the owner's max_budget for keys with no team"8. An administrator can also cap what any key-creation request is allowed to set, via upperbound_key_generate_params, which prevents a delegated key-issuer from minting a key with a higher limit than the org intends to allow at all8.
5.2 Hard limits, soft limits, and scope restrictions
Beyond the numeric budget, a virtual key can carry hard limits on tokens, requests, or upstream billing, soft limits that only trigger an alert rather than a rejection, model allow-lists, and its own per-key rate limits, independent of what its owner would otherwise inherit1. A hard limit is enforced at stage 2.1 of the lifecycle -- the request is rejected before it reaches the router. A soft limit does not touch the request path at all; it is a monitoring signal that fires off the critical path, in the same async category as stage 2.5's accounting. Module 22 derives the specific algorithms (token bucket, fixed window, sliding window) that enforce the numeric side of this; this module's job is only to place the virtual key at the correct point in the lifecycle and show that it is a hierarchy, not a flat list of unrelated credentials.
The market has not converged on a single shape for this hierarchy outside LiteLLM's key/user/team/org model. Portkey originally exposed "Virtual Keys" as a single flat mapping -- one Portkey API key giving access to many providers' credentials without exposing them in application code -- and has since deprecated that model in favor of a "Model Catalog," described as offering "fine-grained budgets, rate limits, and model allow-lists" at what is now a provider-and-model routing-table level rather than a per-caller identity level9. The direction of that migration is informative: a flat, single-purpose virtual key (one key, one provider's credentials) is being replaced by something closer to LiteLLM's structured hierarchy, because a flat mapping has nowhere to attach a team-level budget rollup at all.
| Level | Spend this month ($) | max_budget ($) | Under limit? |
|---|---|---|---|
| Key sk-1 (alice) | 400 | 1,000 (inherited from alice) | Yes |
| Key sk-2 (alice) | 240 | 1,000 (inherited from alice) | Yes |
| User alice | 640 | 1,000 | Yes |
| Key sk-3 (bob) | 610 | 1,000 (inherited from bob) | Yes |
| User bob | 610 | 1,000 | Yes |
| Team eng-infra | 1,250 | 2,000 | Yes, but at 62.5% with two users only |
Every individual key and every individual user in Table 3 is comfortably under its own budget. The number that actually constrains the team's next hire's key is the $2,000 team ceiling minus the $1,250 already spent by two users -- an aggregate the per-key and per-user checks in stage 2.1 cannot see unless the check is explicitly performed at the team level too, which is exactly why LiteLLM tracks spend at all three levels independently rather than deriving the team total lazily from its members at check time1.
6. Async accounting and the staleness window it creates
Stage 2.5 runs after the response has already left the gateway: "after the response is returned to the client, spend logging, rate-limit accounting, and logging callbacks all run as asynchronous background tasks -- no database write sits in the request's critical path"1. This is the right engineering call for latency, and it is exactly the kind of trade-off Module 5 named in the abstract: choosing to serve every request without paying a synchronous write's latency, in exchange for a bounded period during which the gateway's enforced state (what it checked against at stage 2.1/2.2) lags the true state (what stage 2.5 will eventually record).
6.1 The window is real, not theoretical
Two published gateways sit at different points on this trade-off, and the difference is a genuine, cited disagreement rather than a matter of taste. Envoy AI Gateway's usage-based rate limiting checks, on request arrival, "if processing this request would exceed the configured token limit. If the limit would be exceeded, the request is rejected with a 429 status code. If within the limit, the request is processed and its token usage is counted towards the total"10 -- an optimistic-admission design: the check happens against whatever total has already landed, and the request's own eventual usage is folded in only after it completes. A burst of concurrent requests can each pass that pre-check against the same stale total before any of their usage has been counted, because none of them has finished yet.
Anthropic's rate-limit documentation describes the more conservative alternative for its own tokens-per-minute limits: input-token throughput is "estimated at the beginning of a request and adjusted during it to reflect actual input tokens used," and output-token throughput counts only tokens actually generated, in real time11. That is a pessimistic-then-corrected design -- reserve a worst-case estimate up front, adjust downward once the true count is known -- rather than an optimistic one. Module 22 derives both algorithms and the general problem they are solving (you cannot know an LLM request's true token cost until generation finishes) in full; this module's point is narrower: the choice of where in the lifecycle you reconcile is inseparable from stage 2.5's async design, because any reconciliation has to happen off the critical path by construction once you have decided the critical path performs no blocking write.
6.2 Sizing the overspend window
The size of the staleness window is a function of three things: how many requests the affected key can issue per second, the average dollar or token cost per request, and how long it takes the async accounting path to land a write that the next admission check will actually see (whether that is a cache invalidation delay, a batched-write interval, or simply the in-flight duration of concurrent requests that haven't completed yet). Multiplying request rate by cost by window length gives the worst-case overspend a single key can achieve before enforcement catches up -- not a probabilistic estimate, a hard ceiling on how bad an optimistic-admission design can be for a given key's throughput.
Worst-case overspend during the async-accounting window
Formula: \( \text{overspend} = \text{rps} \times \frac{\text{win}}{1000} \times \text{cost} \). At 20 req/s, five cents/request, and a 500 ms reconciliation lag, a single key can push roughly $0.50 past its limit before the next admission check reflects it1. Scale rps to a burst of concurrent requests during an incident and the same arithmetic explains why an optimistic-admission gateway needs a hard cap on concurrent in-flight requests per key, independent of its steady-state rate limit.
6.3 The async write still has to be idempotent
Stage 2.5 running off the critical path does not exempt it from the retry problem Module 8 and Module 18 already established: a router that retries a failed provider call (stage 2.3's job) or a client that retries a dropped connection can cause the same logical request to be accounted for twice unless the accounting write itself is keyed the same way Stripe keys a payment retry -- a client- or gateway-generated identifier whose second write is recognized as a duplicate of the first rather than a second charge1. Airbnb's Orpheus library (Module 20) and Stripe's Idempotency-Key header (Module 18) solved this exact problem for two different kinds of financial writes; an AI gateway's async spend log is a third instance of the same problem, and Module 23 designs the credit ledger that has to get it right for token-metered billing specifically, including the harder case of a request that is retried mid-stream rather than cleanly before or after.
7. Why the edge tier of an AI gateway caches almost nothing
Module 1 and Module 2 established that a classical CDN's whole reason for existing is caching: an edge PoP holds a copy of the response and serves it without a round trip to origin, and Cloudflare's own reference architecture describes tiered caching -- certain data centers acting as reverse proxies to origin for other data centers -- specifically to raise the cache-hit rate further14. An AI gateway sits in the same topological position and, on most traffic, does almost none of that.
Cloudflare AI Gateway's own caching documentation is explicit about why: caching is "disabled by default" and, when turned on, a cache hit requires an exact match -- the cache key is the SHA-256 hash of the concatenated provider, endpoint, model, provider authentication header, and full request body, so "any difference in the body -- including messages, tools, or model parameters -- will result in a separate cache entry"13. A classical HTTP cache keys on a URL and a handful of headers because many different clients request the identical resource; an LLM request's cache key is effectively its entire prompt, and most prompts sent to a shared endpoint are unique strings that will never recur byte-for-byte. TTLs when caching is enabled range from a minimum of 60 seconds to a maximum of one month, with a 5-minute default when neither a per-request nor a gateway-level TTL is configured13 -- numbers that only matter for the minority of traffic where the exact same request really does repeat, which Cloudflare's own documentation frames narrowly: it names "a support bot that asks 'How can I help you?' and lets the user select an answer from a limited set of options" as the kind of workload this benefits, not general chat traffic13.
Cloudflare AI Gateway itself is positioned as a unified front door across providers -- it "sits in front of multiple AI model providers" including Workers AI, Anthropic, Google Gemini, OpenAI, and Replicate, adding rate limiting, retries, model fallback, and analytics on top of whichever provider the request is routed to12 -- but the caching feature is opt-in precisely because exact-match caching captures only a narrow slice of real traffic. This is the same conclusion Module 2's cache-placement analysis would predict once you note that an LLM prompt is closer to a unique query than to a static asset: the cache-hit ratio a CDN reports for images and scripts (routinely well above 90% for popular content) has no LLM-traffic analogue, because there is no equivalent of "the same JPEG requested by ten thousand different clients."
The workaround the market has converged on is not exact-match caching at all, but semantic caching: compare a new prompt's embedding to previously-cached prompts' embeddings and serve a cached completion if the vector proximity clears a threshold, rather than requiring byte-identical input. Kong AI Gateway ships this as its AI Semantic Cache policy, and Azure API Management implements the same idea via llm-semantic-cache-store and llm-semantic-cache-lookup policies backed by Azure Managed Redis or another RediSearch-compatible external cache7. Semantic caching only helps when a meaningful fraction of traffic really does cluster around a small number of recurring intents -- FAQ-style bots, support triage, structured extraction over a narrow domain -- which is a materially smaller share of an AI gateway's traffic than the share of a classical CDN's traffic that a URL-keyed cache successfully serves.
Semantic caching also reintroduces a correctness question that exact-match caching never had to answer. A classical cache hit is a proof of identity: byte-for-byte the same request gets byte-for-byte the same response, so serving from cache is never observably different from recomputing it. A semantic-cache hit is a proof of similarity above a configured threshold, not identity -- two prompts that a vector-similarity score judges "close enough" can still differ in exactly the clause that changes the correct answer (a negation, a changed constraint, a different number in an otherwise identical sentence). Neither Kong's nor Azure's documentation states a default similarity threshold or a false-hit rate, so an operator turning this feature on is trading some unmeasured rate of wrong-but-plausible cached answers for the token-cost reduction, and has to set the threshold empirically against its own traffic rather than a documented default7.
Where experts disagree
Every gateway this module cites has already chosen one of three answers to a question none of them frames as a single decision, but which is the same decision every time: given that Envoy-class data planes already solve TLS termination, HTTP/2, connection pooling, and retries, should the AI-specific logic live inside that data plane as an extension, beside it as a purpose-built application proxy you run yourself, or entirely outside your infrastructure as a vendor-hosted service?
Extend an existing L7 gateway (ext-proc / plugin)
Kubernetes Gateway API Inference Extension, Kong AI Gateway, Azure API Management467
Keep the hardened, already-operated data plane (Envoy, Kong's core, Azure's gateway runtime) and add AI-specific routing/policy logic through a defined extension point -- ext_proc's gRPC hook, Kong's plugin/entity model, or Azure's policy pipeline. You inherit years of production hardening on the parts of the problem that have nothing to do with LLMs.
- No need to reimplement TLS, HTTP/2, connection pooling, or retry logic
- Kubernetes-native deployments get InferencePool/EPP integration with existing Gateway API tooling for free4
- Operationally, it's one gateway to run, not two
- Every AI-aware decision costs an extra network hop (the ext_proc gRPC round trip) that a purpose-built proxy avoids by making the decision in-process5
- Constrained by the host gateway's request/response model and configuration surface
A purpose-built application-layer proxy
LiteLLM1
Run a proxy whose entire codebase is AI-gateway logic -- virtual keys, per-provider SDK translation, cost calculation, router fallbacks -- as ordinary application code rather than as an extension to someone else's data plane.
- Every check (auth, budget, routing) happens in-process with no extra network hop
- Fastest to change: adding a new provider or a new cost model is an application-code change, not a data-plane redeployment
- Full visibility into and control over LLM-specific state (token counts, provider quirks) that a generic ext_proc processor has to reconstruct from raw bytes
- You own connection handling, TLS, HTTP/2, and retry semantics yourself, or depend on whatever your web framework gives you for free -- reimplementing what Envoy has already hardened
- Scaling the proxy itself becomes your operational problem rather than an existing gateway's already-solved one
Fully managed, vendor-hosted gateway-as-a-service
Point client traffic at a vendor's edge instead of operating any gateway infrastructure at all; the vendor handles routing to providers, rate limiting, retries/fallback, and (optionally) caching.
- Zero infrastructure to run or scale -- "it only takes one line of code to get started"12
- Provider-agnostic unified interface across OpenAI, Anthropic, Gemini, and others out of the box12
- Another network hop and another party's incident history between your client and the model provider
- Routing and caching logic limited to what the vendor exposes as configuration, not arbitrary application code
- Vendor lock-in on the identity/budget model the gateway imposes
The right answer tracks where you already have sunk infrastructure and how much custom AI-specific logic you need, not any inherent superiority of one pattern. If you already run Kubernetes with a Gateway-API-capable ingress (or Kong, or Azure API Management) for non-AI traffic, the ext-proc/plugin extension is close to free marginal cost and gets you inference-aware routing without a second system to operate -- this is the right default for a platform team bolting AI traffic onto existing infrastructure. If your AI-specific logic (custom cost models, unusual provider SDKs, a bespoke virtual-key hierarchy) is complex and changes weekly, a purpose-built proxy like LiteLLM is faster to iterate on precisely because it is ordinary application code with no extension-point ceremony, and the extra latency of doing everything in-process is usually smaller than the ext_proc round trip it avoids. If you have no existing gateway investment and want to ship quickly with minimal operational surface, the vendor-hosted option is rational, with the accepted cost of an additional hop and reduced customization. No source in this map publishes a head-to-head latency or feature-velocity benchmark across all three; that comparison, run on comparable traffic, is what would actually settle which pattern wins at a given scale rather than which one a given team happened to already have running.
What is changing
Azure API Management's "unified model API" is in preview as of 2026-09-18: a single OpenAI-compatible endpoint in front of multiple providers' backends, with format translation and governance policy applied once across all of them rather than once per provider integration7. That collapses part of what a router (stage 2.3) does today into a declared API shape instead of custom routing code, and is worth re-checking once it leaves preview.
Portkey's move from a flat "Virtual Keys" credential model to a "Model Catalog" routing table9 is a live data point that the identity layer of these gateways is still being redesigned, not settled; LiteLLM's key/user/team hierarchy (section 5) is the most structured public example today, but it is one design among several still in motion.
MCP server governance is entering the same gateways this module describes: both Kong AI Gateway and Azure API Management now list MCP server registration and policy enforcement alongside their LLM-proxying features67. Whether MCP traffic ends up going through the same auth/budget/rate-limit lifecycle this module derives, or through a materially different one, is unresolved in the public documentation as of 2026-09-18 and is out of this module's scope.
References
- officialBerriAI / LiteLLM contributors. Life of a Request (LiteLLM Proxy architecture). docs.litellm.ai live docs, 2026. https://docs.litellm.ai/docs/proxy/architecture. Accessed 2026-09-17. Used for: the five-stage request lifecycle (auth/budget check, parallel rate limiter, router, provider call, async accounting), cache-first key lookup, virtual-key hard/soft limits and key/user/team spend rollup.
- officialNGINX. Using nginx as HTTP load balancer. NGINX / F5, live documentation, 2026. https://nginx.org/en/docs/http/load_balancing.html. Accessed 2026-09-17. Used for: round robin, least-connections, and ip-hash as the classical L7 balancing algorithms contrasted against inference-aware routing.
- blogKubernetes project contributors. Introducing Gateway API Inference Extension. kubernetes.io/blog, published 2025-06-05. https://kubernetes.io/blog/2025/06/05/introducing-gateway-api-inference-extension/. Accessed 2026-09-17. Used for: why LLM inference sessions are long-running/stateful/partially cached, why generic L7 balancers lack model-identity and criticality awareness, and the naming of KV-cache utilization and queue depth as routing signals.
- officialkubernetes-sigs contributors (Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA). Introduction (Kubernetes Gateway API Inference Extension). Kubernetes SIGs, live docs, 2026. https://gateway-api-inference-extension.sigs.k8s.io/. Accessed 2026-09-17. Used for: the InferencePool/Endpoint Picker (EPP) architecture, the statement that it extends any ext-proc-and-Gateway-API-capable gateway rather than replacing it, the claimed latency/utilization benefit, and (Concepts: Roles and Personas page) the Inference Platform Admin / Inference Workload Owner role split.
- officialEnvoy Proxy project. External Processing Filter (ext_proc). envoyproxy.io, latest docs, 2026. https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter. Accessed 2026-09-18. Used for: the bidirectional gRPC ProcessingRequest/ProcessingResponse stream, the four interception points (request/response headers/body/trailers), and per-message dynamic subscription to events.
- officialKong Inc. Kong AI Gateway. developer.konghq.com, live docs, 2026. https://developer.konghq.com/ai-gateway/. Accessed 2026-09-18. Used for: the AI Model entity replacing the AI Proxy plugin, and AI-specific policies (AI Semantic Cache, AI Prompt Guard, AI Rate Limiting Advanced) layered on Kong's existing plugin pipeline.
- officialMicrosoft. AI gateway capabilities in Azure API Management. Microsoft Learn, updated 2026-06-25. https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities. Accessed 2026-09-18. Used for: "extends API Management's existing API gateway; it's not a separate offering," the llm-token-limit policy, semantic caching via llm-semantic-cache-lookup/store, backend load balancing (round-robin/weighted/priority/session-aware) and circuit breaker with Retry-After-based trip duration, and the unified model API preview.
- officialBerriAI / LiteLLM contributors. Virtual Keys. docs.litellm.ai live docs, 2026. https://docs.litellm.ai/docs/proxy/virtual_keys. Accessed 2026-09-18. Used for: key-to-owner mapping via user_id/team_id, budget/rate-limit inheritance from owner to key, and upperbound_key_generate_params.
- officialPortkey. Virtual Keys. portkey.ai/docs, live docs, 2026. https://portkey.ai/docs/product/ai-gateway/virtual-keys. Accessed 2026-09-18. Used for: the deprecated flat Virtual Keys model and its migration to a Model Catalog routing table with fine-grained budgets/rate limits/model allow-lists.
- officialEnvoy AI Gateway project (rebranded "Agent Router" in v0.1 docs). Usage-based Rate Limiting. v0.1 docs, 2026. https://aigateway.envoyproxy.io/docs/0.1/capabilities/usage-based-ratelimiting/. Accessed 2026-09-17. Used for: the optimistic-admission enforcement design (check against current total, admit, count actual usage afterward).
- officialAnthropic. Rate limits. Claude API docs, live documentation, 2026. https://platform.claude.com/docs/en/api/rate-limits. Accessed 2026-09-17. Used for: the pessimistic estimate-then-adjust design for input-token throughput limits, contrasted with Envoy AI Gateway's optimistic admission.
- officialCloudflare. AI Gateway. developers.cloudflare.com, live documentation, 2026. https://developers.cloudflare.com/ai-gateway/. Accessed 2026-09-18. Used for: AI Gateway as a unified front door across providers (Workers AI, Anthropic, Gemini, OpenAI, Replicate), rate limiting, retries/fallback, and one-line-of-code deployment.
- officialCloudflare. Caching. AI Gateway features docs, live documentation, 2026. https://developers.cloudflare.com/ai-gateway/features/caching/. Accessed 2026-09-18. Used for: caching disabled by default, the SHA-256 exact-match cache key over provider/endpoint/model/auth header/full body, and the 60-second-to-one-month TTL range with a 5-minute default.
- officialCloudflare. Content Delivery Network (CDN) Reference Architecture. Cloudflare, live documentation, 2026. https://developers.cloudflare.com/reference-architecture/cdn-reference-architecture. Accessed 2026-09-17. Used for: tiered caching as the classical CDN pattern contrasted against an AI gateway's near-absence of caching.
- officialCloudflare. Rate limiting. AI Gateway features docs, live documentation, 2026. https://developers.cloudflare.com/ai-gateway/features/rate-limiting/. Accessed 2026-09-17. Used for: Cloudflare AI Gateway named as the "Where experts disagree" example of a fully managed, vendor-hosted gateway.