Multi-Node LLM Inference: From One Node to N · Module 14

Kubernetes Deployment Topologies and Operations at Fleet Scale

Every stack in Part 5 — Dynamo, llm-d, vLLM, SGLang, TensorRT-LLM — eventually has to answer the same four Kubernetes questions: how do you schedule a model that is sharded across physical nodes as one unit, how do you scale that unit without lying to yourself about GPU utilization, how do you route more than one model or tenant through the same fleet, and how do you upgrade or fail one node out of a group without taking the whole group down. This module answers those four questions at the mechanism level, using LeaderWorkerSet, KServe, KEDA, Ray Serve, and AIBrix as the concrete implementations. After it, you can specify a Kubernetes topology for an N-node deployment and place any orchestration project you encounter on the landscape it belongs to.

Course 35 min read 13 sources As of 2026-09-17 Prerequisite: Module 9, Module 10
Full deployment stack: gateway routes to InferencePools, each pool is a set of LeaderWorkerSet groups spanning physical nodes, KEDA scales the pool on SLO metrics, an orchestration layer sits above coordinating both HTTPRoute Gateway API InferencePool A model / tenant 1 (§4) InferencePool B model / tenant 2 LeaderWorkerSet: replicas = 2 groups, size = 2 (§1) group 0 (1 replica) leader node A worker node B group 1 (1 replica) leader node C worker node D TP/PP shard spans leader+worker — gang-scheduled as one unit Orchestration Dynamo / llm-d (Mod. 9, 10) or Ray Serve LLM / KServe / AIBrix (§5) Prometheus: TTFT/ITL, queue depth, KV-cache occupancy (§3, §6) KEDA ScaledObject → HPA → LWS replicas
Figure 1. The four layers this module covers: a Gateway API HTTPRoute fans out to one InferencePool per model/tenant (§4, cross-referencing Module 10), each pool is backed by LeaderWorkerSet replica groups that gang-schedule a leader and worker pod across physical nodes as one unit (§1), an orchestration layer (either an inference-specific control plane or a general-purpose Kubernetes/Ray project, §5) sits above making placement and scaling decisions, and Prometheus-fed autoscaling adds or removes whole groups rather than resizing one (§3). Structure per the LWS and Gateway API Inference Extension specifications1.

1. LeaderWorkerSet: the multi-host replication primitive

A dense 70B-class model sharded with tensor parallelism across two 8-GPU nodes is not, from Kubernetes' point of view, sixteen independent things. It is one thing that happens to occupy sixteen GPU slots on two machines: the sixteen ranks must start together, discover each other's addresses before the first forward pass, and — critically — if any single rank dies, the other fifteen are no longer a working replica and have to restart together too. None of Kubernetes' native workload controllers model this. A Deployment assumes every pod is independently replaceable; a StatefulSet gives pods stable identity and ordered rollout but still treats each pod as its own unit of readiness and failure, with no notion that pod 3 and pod 7 belong to the same logical replica and must live or die together.

LeaderWorkerSet (LWS) is the Kubernetes SIGs API built to close that gap: "an API for deploying a group of pods as a unit of replication," aimed specifically at "multi-host inference workloads where the LLM will be sharded and run across multiple devices on multiple nodes"1. It is co-designed with llm-d (Module 10) as one of the two projects driving its API surface1, and it is the substrate vLLM's own multi-node deployment guides, KServe's multi-node predecessor design, and SGLang's multi-node recipes (Module 12) all assume sits underneath a production router.

1.1 The leader/worker group model

An LWS resource's spec.replicas field is not a pod count — it is a count of groups, where each group is "a single leader and a set of workers"2. The group's actual pod count is a separate field, leaderWorkerTemplate.size: "the total number of pods in each group"3, with one pod running the leaderTemplate and the rest running the workerTemplate3. Every pod in a group carries a unique index from 0 to size − 1, with index 0 reserved for the leader2, and — the property that makes gang scheduling work — "pods in the group will have the same lifecycle and be created in parallel"2, with an all-or-nothing restart policy on failure3: if one pod in the group dies, LWS restarts the whole group rather than leaving a half-formed shard limping along.

Two templates instead of one matters mechanically, not just cosmetically. The leader pod typically runs the API server and is the one thing an external router or InferencePool endpoint points at; the worker pods run headless, contributing GPUs and layers to the shard but exposing no serving endpoint of their own. This maps directly onto how vLLM structures a multi-node launch: --headless on every non-leader rank, so exactly one process in the group answers HTTP requests while the rest exist purely as distributed-execution participants.

1.2 Group discovery and the two startup paths

Once Kubernetes has scheduled a group's pods, they still need to find each other and agree on ranks before a collective operation can run. LWS injects three environment variables into every pod for exactly this: LWS_LEADER_ADDRESS (the leader pod's network address), LWS_GROUP_SIZE (the group's total pod count, i.e. leaderWorkerTemplate.size), and LWS_WORKER_INDEX (0 for the leader, 1..size−1 for workers). vLLM's own reference deployment wires these three variables directly onto its distributed-launch flags: --pipeline-parallel-size $(LWS_GROUP_SIZE), --nnodes $(LWS_GROUP_SIZE), --node-rank $(LWS_WORKER_INDEX), and --master-addr $(LWS_LEADER_ADDRESS), with the leader serving on --port 8080 and every worker adding --headless2. A two-pod group (one leader, one worker) becomes, with zero YAML beyond the environment-variable substitution, a working two-node pipeline-parallel shard: the leader's vllm serve process is rank 0, the worker is rank 1, and both learn the leader's address from LWS rather than from any hand-written service-discovery step.

vLLM documents a second startup path built on Ray instead of raw multiprocessing, and the distinction matters because it recurs in section 5's orchestration-layer landscape. Before vllm serve ever runs, a helper script forms a Ray cluster across the group: the leader runs multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE) to start the Ray head, and each worker runs multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS) to join it2. Only once the Ray cluster has formed does the leader invoke vllm serve --distributed-executor-backend ray, handing tensor- and pipeline-parallel execution to Ray's own actor placement rather than to raw torchrun-style rank assignment2. LWS's job in both paths is identical and stops at the same boundary: get the right pods scheduled together, on time, with known addresses. What runs inside the group once it is up — plain multiprocessing collectives or a Ray cluster — is the inference engine's choice, not LWS's.

1.3 Group-level rollout

LWS extends the same group-as-a-unit logic to upgrades. Its rolloutStrategy is presently fixed to RollingUpdate, configured with a rollingUpdateConfiguration carrying maxUnavailable and maxSurge3 — the same two knobs a Deployment exposes, but evaluated in units of whole groups rather than individual pods. The project's own framing is explicit: "groups are upgraded one by one as a unit (i.e. the pods within a group are updated together)"2. A rolling image bump on a fleet of eight two-node groups with maxUnavailable: 1 takes one group fully offline (both its leader and worker pods), replaces both, waits for the new group to pass readiness, and only then moves to the next group — never partially updating a group's leader while its worker still runs the old image, which would otherwise break the two ranks' agreement on model weights and parallel-execution code paths. Section 7 returns to this mechanism directly, contrasted against the pool-level drain-before-remove pattern Module 4 already established for prefill/decode reconfiguration.

LWS also exposes topology-placement and networking options relevant to why a group exists as one object instead of two independent pods with matching labels: it supports "opt-in support for pods in the same group to be co-located in the same topology"2 (keeping a shard's cross-node traffic on fast interconnect, the same NVL72-domain reasoning Module 1 introduced), a configurable subdomainPolicy controlling how the per-group headless Service is created for pod-to-pod DNS3, and a scale subresource so an external HPA or KEDA object (section 3) can drive replicas directly without a custom controller in between2.

LeaderWorkerSet: one replica is a group of a leader pod and worker pods spanning two physical nodes, with environment variables carrying rank information LeaderWorkerSet: replicas = N groups — this figure shows one group physical node A leader pod — index 0 LWS_WORKER_INDEX=0 serves API on :8080 rank 0 / node-rank 0 physical node B worker pod — index 1 LWS_WORKER_INDEX=1 --headless (no API) rank 1 / node-rank 1 LWS_LEADER_ADDRESS restartPolicy: all-or-nothing either pod dies → the whole group restarts together
Figure 2. One LWS replica group: a leader pod (index 0, serving) and a worker pod (index 1, headless) on separate physical nodes, discovering each other via LWS-injected environment variables and restarting as a single unit on failure. Field and variable names per the LWS README and vLLM's LWS deployment guide12.

2. KServe's parallelism arithmetic and multi-node ServingRuntime

KServe (a CNCF project) is a Kubernetes-native model-serving control plane with its own InferenceService CRD, predating and independent of LWS: its multi-node ServingRuntime, kserve-huggingfaceserver-multinode, is "built on vLLM, which supports multi-node/multi-GPU feature"4, but its own documentation never mentions LeaderWorkerSet — it defines its own head-pod/worker-pod model directly, requiring "exactly one head pod" per deployment4 and using a plain Kubernetes Deployment resource underneath rather than an LWS object. It is a useful second data point precisely because it solves the same gang-scheduling problem section 1 described without building on LWS at all.

2.1 The GPU-count formula

KServe's workerSpec exposes two fields that are easy to misread as literal GPU counts but are not: tensorParallelSize "is only used to configure tensor parallelism — that is, how the model weights are sharded across GPUs," and pipelineParallelSize "is only used to configure pipeline parallelism — how the model layers are split across the GPUs"4. The actual GPU accounting is a separate, explicit mechanism KServe's own documentation spells out as four formulas4:

$$ \text{Total GPUs} = \text{tensorParallelSize} \times \text{pipelineParallelSize} $$ $$ \text{Ray Node Count} = \left\lceil \frac{\text{Total GPUs}}{\text{workerSpec.resources.requests.gpu}} \right\rceil $$ $$ \text{Worker GPU Count} = \frac{\text{Total GPUs}}{\text{Ray Node Count}} - 1, \qquad \text{Head GPU Count} = \text{Total GPUs} - 1 $$

Read plainly: total GPU demand is TP × PP, the number of physical nodes needed is that total divided by however many GPUs a single worker pod requests (rounded up), and — the detail that trips people up — the head pod itself only claims Total GPUs − 1 because it counts as one of its own node's GPU-holding ranks. If Total GPUs equals workerSpec.resources.requests.gpu exactly, the division degenerates to one node and "only the head node will be deployed"4 — there is no separate worker pod at all, and the multi-node machinery is silently a no-op.

2.2 Worked example: 16 GPUs across 2 nodes

KServe's own reference environment is 16 GPUs split 8-per-node, worked three ways for the same total demand4:

Table 1. KServe's three documented ways to reach 16 GPUs across 2 nodes for the same model, and the vllm serve flags each one produces4.
CasepipelineParallelSizetensorParallelSizeGPU / podResulting topologyResulting vllm serve flags
1 — explicit PP2881 head + 1 worker pod, 2 nodes--tensor-parallel-size 8 --pipeline-parallel-size 2
2 — TP only1 (default)168KServe derives PP=2, TP=8 internally to fit the 8-GPU pod request — same 2-node result--tensor-parallel-size 16
3 — single node188Total GPUs (8) = GPU/pod (8) → only the head pod deploys, no multi-node--tensor-parallel-size 8

Case 1 and Case 2 land on the identical physical topology — two 8-GPU pods, one head and one worker — because KServe internally reduces a bare tensorParallelSize=16 request down to a PP×TP factorization that fits the declared per-pod GPU budget4. Case 3 is the reminder that the formulas are unconditional: plug in numbers where Total GPUs already equals one pod's GPU request, and the "multi-node" ServingRuntime deploys nothing but a single pod.

KServe 16-GPU, 2-node worked example: pipelineParallelSize 2 times tensorParallelSize 8 equals 16 total GPUs split across a head pod and a worker pod Total GPUs = tensorParallelSize (8) × pipelineParallelSize (2) = 16 head pod — node 1 tensor-parallel-size 8 Head GPU Count = 16 − 1 = 7 claimed + 1 self → 8 GPUs / pod worker pod — node 2 Worker GPU Count = 16/2 − 1 = 7 claimed + 1 self → 8 GPUs / pod Ray cluster Ray Node Count = ceil(16 / 8) = 2 — autoscalerClass must be none
Figure 3. KServe's 16-GPU/2-node worked example (Case 1 of Table 1): the head pod and worker pod each request 8 GPUs, with the formula's −1 term accounting for the pod's own rank inside its GPU count. Numbers per KServe's multi-node documentation4.

2.3 What the formulas cost you

KServe's multi-node path carries three explicit, non-negotiable constraints its own documentation states directly. First, "the Autoscaler must be configured as none"4 — a multi-node InferenceService sets serving.kserve.io/autoscalerClass: none and does not participate in KServe's usual scale-to-N-replicas behavior at all; section 3 returns to why, and to how other projects answer the same problem differently. Second, resizing pipelineParallelSize in production is explicitly discouraged: it "is not an autoscaling concept; rather, it determines how many nodes will be used to run the model," and changing it live "can impact the existing service due to the default behavior of the Deployment resource" — the documented workaround is deleting and recreating the InferenceService rather than patching it in place4. Third, the choice between RollingUpdate and Recreate deployment strategy is a real resource trade-off, not a style preference: RollingUpdate "ensures that the existing service remains operational until the new service becomes Ready" but "requires more than twice the resources to function effectively," since a full second 16-GPU topology must exist before the first one is torn down4 — expensive enough that KServe's own guidance recommends Recreate during development and reserves RollingUpdate for production paths that can actually absorb the doubled GPU footprint.

3. Autoscaling on SLOs, not utilization

Section 2.3 flagged that KServe disables autoscaling outright for multi-node deployments. That is a real constraint, but it is not evidence that multi-node fleets cannot be autoscaled — it is evidence that the metric Kubernetes' default HorizontalPodAutoscaler (HPA) reaches for, CPU or GPU utilization, is the wrong signal for an LLM engine in the first place, multi-node or not. Fixing that is a prerequisite this section works through before returning to how a pool of LWS groups actually gets scaled.

3.1 Why GPU utilization is a broken scaling signal

A continuous-batching engine keeps its GPU busy by design: it packs whatever requests are in flight into the largest batch the KV-cache budget allows, on every decode step. That means GPU utilization sits near saturation whether the engine is serving 4 requests or 40 — the metric conflates "the GPU is doing work" with "the GPU has no more headroom," and an HPA driven by it reacts to the wrong condition or reacts too late. llm-d's own autoscaling documentation states the failure mode directly: "during active token generation, GPU utilization can remain high whether a model server is lightly loaded or saturated," which makes DCGM-style utilization metrics unable to distinguish an underloaded replica from an overloaded one7. AIBrix's paper frames the same problem from the deployment-latency side: LLM request complexity and I/O size vary widely enough to overwhelm a utilization-based autoscaler before it reacts, and large model images with slow model distribution already add a 2–3 minute delay before a newly scaled pod is useful — compounding a signal that was already lagging12. The llm-d workload-autoscaling guide adds a second reason traditional resource metrics fail structurally, not just numerically: "resource utilization metrics (CPU/GPU) are... lagging indicators — they only reflect saturation after it has already occurred"8, whereas the quantity that actually determines whether new requests will meet their latency SLO is queueing and cache pressure building up right now, upstream of the compute engine even noticing.

3.2 The demand-driven metric vocabulary

Modules 9 through 12 each documented a Prometheus surface for their own stack; none of those modules built a scaling trigger out of them, because that was explicitly this module's job. Pulled together, the metrics that actually predict SLO breach before it happens cluster into four categories, present in some form in every stack this course has covered:

Table 2. The demand-driven metric vocabulary used as autoscaling triggers, by project, synthesized from the Prometheus surfaces Modules 9–12 already documented plus llm-d's EPP-specific additions578.
SignalvLLM metricllm-d EPP metricWhat it predicts
Queue depthvllm:num_requests_waitingllm_d_epp_flow_control_queue_sizerequests buffered, not yet admitted — direct backlog
Running requestsvllm:num_requests_runningllm_d_epp_request_runningin-flight concurrency, a proxy for near-term saturation
KV-cache occupancyvllm:kv_cache_usage_perc— (upstream of EPP, engine-native)headroom before admission/eviction pressure hits
Pool saturation—llm_d_epp_flow_control_pool_saturationnormalized 0.0–1.0+ demand ratio, portable across pool sizes
TTFT / ITLvllm:time_to_first_token_secondspredicted/measured TTFT, TPOT histogramsthe SLO itself, measured or estimated directly

Module 11 already built a four-series minimal SLO dashboard from vLLM's own metrics (TTFT on the prefill pool, ITL on the decode pool, KV-cache occupancy on both, phase-split histograms for diagnosis) — that same set is exactly what an autoscaling trigger consumes, just read as a live scalar rather than a plotted histogram. Queue depth and running-request count answer "is demand exceeding capacity right now"; KV-cache occupancy and pool saturation answer "how much headroom is left before it does"; TTFT/ITL histograms against the SLO answer "has it already failed." A trigger built on any single one of these is a reasonable starting point; llm-d's own guidance recommends combining at least two — one measuring unfulfilled demand (queue size) and one measuring active concurrency (running requests) — and taking the larger of the two resulting replica counts, "enabling holistic demand representation" rather than under-reacting to a spike that shows up in only one signal7.

3.3 Mechanism comparison: how four projects turn these signals into replica counts

The mechanism that consumes Table 2's metrics differs by project in exactly the way section 5's landscape map will generalize: some are thin, portable glue on top of stock Kubernetes primitives, others are integrated performance models specific to one control plane.

KEDA + Prometheus is llm-d's recommended path and the most portable of the four. A ScaledObject — not a second, competing HPA — points at a PromQL query over any of Table 2's metrics; KEDA evaluates it directly against Prometheus (no Prometheus Adapter required) and creates/manages the underlying HPA against the model-server Deployment or the LWS group's scale subresource on the writer's behalf7. This is purely reactive — it scales on the current value of a live metric — but it composes cleanly with scale-to-zero, since the EPP can buffer requests during a cold-start pod's warmup rather than dropping them7. It also supports an explicit "SLO-aware" formulation: a Prometheus recording rule computes measured-latency ÷ SLO-target as a ratio, and KEDA scales on that ratio directly via its scalingModifiers expression support8 — which is the cleanest way to fold TTFT/ITL directly into a scaling decision without leaving Kubernetes-native tooling.

Dynamo's Planner (Module 9, §4) sits at the other end: rather than reacting to a live metric, its sla mode runs a performance model — a Rust engine-performance shim backed by analytical (AIC) estimates or an observed-data regression fallback — to predict the replica count a given traffic forecast will need to hold TTFT and ITL at their configured targets, 500 ms and 50 ms by default9. This module does not re-derive that mechanism; the point worth carrying forward here is that it is predictive rather than reactive, trading the simplicity of "read a metric, scale on it" for a model that has to be bootstrapped with profiling data before it is trustworthy.

Ray Serve LLM uses Ray Serve's own generic autoscaler, configured per LLMConfig deployment with an autoscaling_config block (min_replicas, max_replicas, and Ray Serve's own request-concurrency target)10 — reactive like KEDA, but scoped to Ray's own replica abstraction rather than a Kubernetes HPA, which matters directly for the Ray-vs-Kubernetes-native question section 5 returns to.

AIBrix's autoscaler is the fourth point on this spectrum: it explicitly bypasses the Kubernetes custom-metrics path and "maintains sliding window metric aggregation directly in the autoscaler for real-time load reporting," built on two selectable algorithms — Knative Pod Autoscaler (KPA) and AIBrix's own Pod Autoscaler (APA)12. AIBrix's own measured comparison against native HPA on these algorithms reports an 11.5% latency reduction, an 11.4% token-throughput increase, and a 33% reduction in scaling oscillations12 — the oscillation number is the concrete cost of a lagging, noisy trigger that this section's argument predicts: fewer, better-timed scaling decisions instead of a controller thrashing up and down on a signal that does not track real demand.

Table 3. Four autoscaling mechanisms compared on trigger source and reactive-vs-predictive character791012.
ProjectTrigger sourceCharacterScales
llm-d + KEDAPrometheus query over EPP/engine metricsreactive (or SLO-ratio via recording rule)standard HPA on Deployment/LWS scale subresource
Dynamo Planner (sla mode)performance model + traffic forecastpredictive, bootstrapped from profilingPlanner-managed prefill/decode replica counts
Ray Serve LLMRay Serve's own concurrency targetreactiveRay Serve deployment replicas
AIBrix (KPA/APA)sliding-window aggregation, bypasses custom-metrics pathreactive, tuned for reduced oscillationAIBrix Pod Autoscaler-managed pods

What all four share, and what section 2.3's KServe constraint should be read against, is that none of them resize the inside of a multi-host group — none changes a running replica's TP or PP degree. Every one of them scales the count of replicas (or replica groups), which is exactly the unit LWS's replicas field and scale subresource already expose. That distinction is the crux of this module's disagree section below.

Autoscaling pipeline: engine and EPP metrics feed Prometheus, a scaler evaluates a trigger, and the resulting replica-count decision adds or removes whole LWS groups Engine + EPP queue, KV%, TTFT/ITL Prometheus scrape + store (§3.2) KEDA ScaledObject reactive PromQL (§3.3) Planner / APA predictive model (§3.3) replica count +/− LWS groups
Figure 4. The common shape underneath every mechanism in Table 3: engine and EPP metrics feed Prometheus, a reactive (KEDA) or predictive (Planner/APA) scaler evaluates a trigger against Table 2's demand signals, and the output is always a whole-group replica-count decision, never an in-place resize of one group's TP/PP shape.

4. Multi-model and multi-tenant serving

Module 10 established the structural mechanism: an HTTPRoute "may reference multiple InferencePools," so one public endpoint can fan out by model name to several independently scheduled pools, each with its own selector, endpoint picker, and model-server fleet. This section covers what running that pattern operationally involves — how a request's declared model name actually reaches the right pool, how a base model's LoRA adapters get treated as first-class routable targets rather than static attachments, and how tenancy interacts with a fleet spanning more than one GPU type.

4.1 LoRA-adapter routing as a tenancy mechanism

Serving twenty fine-tuned variants of the same base model as twenty separate multi-GPU deployments wastes the memory duplicated base weights would otherwise share — the standard fix is multi-LoRA serving, where one base-model replica holds many small adapter deltas in GPU memory and switches between them per request. That only works as a fleet-level pattern if the routing layer can direct a request to a replica that already has the requested adapter loaded, which is why the Gateway API Inference Extension lists "routing client model names to use-case-specific LoRA adapters" as a named goal of the standard, alongside its KV-cache- and request-cost-aware scheduling goals6 — the same InferencePool/HTTPRoute machinery Module 10 covers for base-model routing extends unchanged to adapter-level routing: "model name" in a request simply resolves to an adapter identifier instead of a different base model.

AIBrix implements this concretely as "high-density LoRA management": a LoraController watches a LoRA custom resource, issues a load_model_command into the AI Engine Runtime sidecar to dynamically load or unload an adapter inside a running vLLM container, and republishes the adapter's availability through Kubernetes' native Service and EndpointSlice objects so the gateway's routing layer sees it as just another routable target12 — the mechanism the paper frames as fixing "unreliable evictions" and "suboptimal failure handling" that ad hoc adapter management produces when LoRA is treated as a static, offline-configured attachment rather than a dynamically scheduled resource12.

4.2 Heterogeneous hardware as a second tenancy axis

Multi-tenancy is not only about serving multiple models behind one endpoint — a fleet mixing GPU generations (older L20s alongside newer A10s or V100s, in AIBrix's own experiment) forces a placement decision that ordinary Kubernetes scheduling does not make well, because request cost-efficiency varies by GPU type even for an identical model and workload shape12. AIBrix's GPU Optimizer addresses this as an integer-linear-programming assignment problem, informed by offline per-GPU-type throughput profiling (Mélange-style12), choosing which accelerator class should serve a given request-size distribution to balance cost against SLO adherence rather than routing purely on availability. In AIBrix's own reported comparison, a heterogeneous fleet (mixing GPU types) against a homogeneous, single-type fleet serving the same workload traded a latency increase of up to 20% — while staying within the declared SLO — for roughly a 10% cost reduction12. That trade-off is the concrete shape of what "multi-tenant" costs once tenants are allowed to land on cheaper hardware: it is not free, and the number that should gate the decision is the SLO headroom left after the latency increase, not the cost saving alone.

5. The orchestration-layer landscape

Sections 1–4 named five different projects solving overlapping pieces of the same problem: raw LWS, KServe, Ray Serve LLM, AIBrix, and — by cross-reference — Dynamo and llm-d from Modules 9 and 10. None of these compete on identical terms; they sit at different points on two independent axes worth making explicit before comparing their feature lists.

5.1 Positioning the five projects

Raw LWS is a primitive, not a platform: it gang-schedules a group of pods and upgrades them as a unit, and does nothing else — no routing, no autoscaling policy, no observability. Every other project in this section either builds on it or reimplements its job independently.

KServe is Kubernetes-native by construction — its own CRD (InferenceService), its own head/worker pod model for multi-node (section 2, built without LWS), and, internally, Ray as the distributed-execution layer inside a multi-node ServingRuntime (the "Ray Node Count" terminology in section 2.1's formulas is not incidental — vLLM's Ray-based executor backend is what KServe's worker pods actually run). It is general-purpose model serving with LLM-specific runtimes bolted on, not an LLM-native control plane.

Ray Serve LLM inverts that emphasis: Ray owns placement and distributed execution natively, and Kubernetes (via KubeRay, outside this module's scope) is only the layer that gets a Ray cluster running in the first place. Its LLMConfig/build_openai_app/serve.run interface accepts a list of llm_configs for multi-model serving in one application and a per-deployment autoscaling_config for replica scaling10 — multi-node and multi-model are both native Ray Serve concepts here, not Kubernetes CRD fields.

AIBrix names the trade-off between the previous two explicitly and builds a hybrid to resolve it: "Kubernetes operators offer coarse-grained resource management but can be complex for fine-grained scheduling, whereas Ray excels in distributed communication but lacks holistic resource control"12. Its answer is what the paper calls Mix-Grain orchestration — a RayClusterFleet and RayClusterReplicaSet layer that gives Kubernetes-style rolling updates, self-healing, and scalable desired-state semantics to a collection of underlying Ray clusters, each Ray cluster internally handling fine-grained rank placement (including, in its disaggregated variant, separate prefill and decode Ray clusters coordinating directly)12. It is, structurally, Kubernetes wrapping Ray wrapping the inference engine — a third point that is neither pure-Kubernetes (KServe) nor pure-Ray (Ray Serve LLM).

Dynamo and llm-d (Modules 9, 10) sit on the second axis, not really the first: both are inference-specific control planes — KV-aware routing, disaggregation-aware scheduling, SLA-driven replica sizing — that need a scheduling substrate underneath them (Grove for Dynamo, LWS for llm-d) rather than replacing it. They compete with each other on how much inference-specific intelligence to build in-house versus standardize at the Kubernetes API level (Module 9's own disagree section covers that split directly); they do not compete with KServe or Ray Serve LLM on the Kubernetes-native-vs-Ray-native axis at all, because neither one is a general-purpose model-serving platform.

Orchestration landscape map: Kubernetes-native versus Ray-native on one axis, generic model-serving platform versus inference-specific control plane on the other K8s-native Ray-native inference-specific generic platform raw LWS (§1) KServe (§2, generic + LLM) Ray Serve LLM (§3.3, §5.1) AIBrix (hybrid, §5.1) llm-d (Mod. 10, on LWS) Dynamo (Mod. 9, on Grove)
Figure 5. Two independent axes place these six projects: Kubernetes-native versus Ray-native scheduling substrate on the horizontal, generic model-serving platform versus inference-specific control plane on the vertical. AIBrix's own stated design rationale is the source for its position as a deliberate hybrid12.

5.2 AIBrix's component list against what it replaces or complements

AIBrix is the widest single project surveyed in this module, and its component list is a useful checklist for what a fleet built purely from generic Kubernetes primitives (raw LWS, KServe, KEDA, InferencePool) would otherwise need to assemble separately.

Table 4. AIBrix's named components and their closest generic-stack equivalent, per AIBrix's own README and paper1112.
AIBrix componentWhat it doesClosest generic-stack equivalent
LLM Gateway and RoutingEnvoy-Gateway-based routing on token patterns, prefix-cache availability, compute overhead; five policies (random, throughput, least-request, least-kv-cache, least-latency, prefix-cache-aware)12InferencePool + EPP (Module 2, 10) — same scoring problem, different implementation
App-Tailored AutoscalerSliding-window metric aggregation, KPA/APA algorithms bypassing custom-metrics path (§3.3)KEDA + Prometheus (llm-d path)
Distributed KV Cache PoolCross-engine KV reuse pool with scan-resistant eviction; combined with prefix caching, measured at approximately 50% higher peak throughput and approximately 70% lower P99 TTFT/ITL versus vLLM's own built-in prefix caching12LMCache / Mooncake (Module 7, 8) — same problem, project-specific implementation
High-Density LoRA ManagementDynamic LoRA CR, controller-driven load/unload (§4.1)GAIE's stated LoRA-routing goal — a standard, not yet a shipped generic implementation6
GPU Hardware Failure DetectionAutomated fault diagnostics plus a failure mock-up/injection tool, Nvidia GPUs and Ascend 910B NPUs supported12No standardized generic-stack equivalent (§7.3)

Reading this table against the landscape map: everything in the "closest equivalent" column exists somewhere else in this course as a project-specific solution to the same problem, which is the general pattern an integrated platform like AIBrix (or Dynamo, per Module 9) is selling — one coherently designed answer to five problems, versus assembling five separately governed generic-stack pieces yourself. Only GPU hardware-failure detection lacks a real generic-stack alternative as of this module's access date, which is why section 7.3 uses AIBrix as the concrete worked example for that specific capability rather than as one option among several.

6. Observability: what actually matters at fleet scale

Modules 9 through 12 each documented their own project's Prometheus surface in full — Dynamo's dynamo_*-prefixed metrics, vLLM's vllm:* gauges and histograms, SGLang's smg_* metrics spanning over 40 series across HTTP, router, worker, and circuit-breaker layers, and llm-d's EPP-level flow-control metrics this module has already used as autoscaling triggers. None of those modules asked which subset of that surface a fleet operator should actually put on a dashboard with an alert threshold attached, versus which subset exists purely to debug a specific incident after the fact. That question does not change by stack — the same answer applies whether the engine underneath is vLLM, SGLang, or TensorRT-LLM.

6.1 What gates an alert

Module 11 already worked out the minimal set for a single vLLM disaggregated deployment: TTFT scraped from the prefill pool, ITL from the decode pool, KV-cache occupancy on both, and the phase-split histograms to distinguish which stage regressed. Generalized across every project this course covers, that same four-signal shape recurs because it maps directly onto Table 2's demand vocabulary from section 3: TTFT and ITL percentiles are the SLO itself, and queue depth / pool saturation / KV-cache occupancy are the leading indicators that predict an SLO breach before it shows up in the latency histograms. A fleet-level dashboard needs these per pool, not fleet-aggregated — an aggregate p99 TTFT across ten prefill pools can look fine while one pool is badly degraded and the other nine are idle, exactly the failure mode a per-pool breakdown catches and a single global number hides.

GPU power draw and utilization belong on the same dashboard, but as a secondary, diagnostic signal rather than an alert trigger — section 3.1 established why: utilization saturates under continuous batching regardless of whether the pool is meeting its SLO, so an alert on "GPU utilization above 90%" fires constantly and means nothing. It is still useful for capacity planning and for confirming a pool is not idling unexpectedly, just not for answering "is this pool currently failing its SLO."

6.2 What is diagnostic-only

The rest of any given stack's metric surface is real, and useful during an incident, but does not earn a standing alert threshold. Module 11's own framing of vLLM's surface is the template: the LoRA-adapter gauge, the static cache-config-info gauge, and the KV-block-lifetime/idle-before-evict histograms are "diagnostic detail for debugging a specific incident, not something worth an alert threshold on its own." The same split applies to SGLang's circuit-breaker metrics (smg_worker_cb_*) and service-discovery counters (smg_discovery_*) — they tell you why a router made a particular decision after something has already gone wrong, not whether something is currently wrong. Deprecated v0-era vLLM metrics (vllm:num_requests_swapped, vllm:cpu_cache_usage_perc, vllm:time_in_queue_requests) belong on neither tier; Module 11 already flagged that a dashboard built against an older metrics reference will alert on gauges that no longer move at all.

Dashboard sketch grouping metrics into an alert tier per pool and an informational tier for incident debugging alert tier — per pool (§6.1) TTFT p50/p90/p99 vs. SLO ITL p50/p90/p99 vs. SLO KV-cache occupancy / pool saturation queue depth / running requests these four drive KEDA/Planner triggers (§3.3) informational tier (§6.2) GPU power draw / utilization phase-split (prefill/decode time) histograms KV-block lifetime / eviction histograms circuit-breaker / discovery counters pulled up only during incident review
Figure 6. A fleet-level dashboard sketch, generalized from Module 11's vLLM-specific minimal SLO dashboard: per-pool latency and demand signals gate an alert and drive autoscaling triggers (§3), everything else is pulled up only to explain an already-detected incident.

7. Upgrade and failure handling at fleet scale

7.1 Rolling upgrade of a stateful multi-host sharded unit

Section 1.3 already described LWS's mechanism directly: rollingUpdateConfiguration's maxUnavailable/maxSurge operate on whole groups, and a group's leader and worker pods are always replaced together — "groups are upgraded one by one as a unit." Section 2.3 gave KServe's version of the same trade-off from the resource-cost side: RollingUpdate keeps the old topology serving until the new one is ready, at the price of needing more than twice the GPU footprint during the transition, which is exactly why KServe's own guidance reserves it for production and defaults development environments to the cheaper, disruptive Recreate strategy.

This is the same shape of trade-off Module 4 already worked through for xPyD reconfiguration — drain-before-remove, where a worker publishes a RuntimeConfig on join and drains its active requests before deregistering on removal — just applied one level up. Module 4's mechanism operates at the granularity of one prefill or decode worker joining or leaving a pool at runtime, driven by the orchestration layer's own scheduling decisions. This section's mechanism operates at the granularity of one multi-host group being replaced wholesale during a planned image or config upgrade, driven by a rollout controller. Both exist for the same underlying reason: neither an individual GPU rank nor half a sharded model can be swapped in isolation without either corrupting the shard's collective state or leaving a partially-upgraded, version-mismatched group serving traffic.

7.2 Drain, cordon, and PodDisruptionBudget underneath both

Both LWS's group-level rollout and any node-level maintenance operation ultimately reduce to two generic Kubernetes primitives. kubectl drain performs a cordon (marking the node unschedulable so no new pods land there) followed by eviction of the node's existing pods, respecting each pod's graceful-termination period13. A PodDisruptionBudget (PDB) sets a floor — minAvailable or a ceiling, maxUnavailable — on how many pods matching a selector may be simultaneously unavailable, and the eviction API checks every proposed eviction against it before allowing it to proceed13. For an ordinary stateless Deployment this is enough on its own: a PDB with minAvailable: 2 on a 3-replica set permits exactly one pod to be drained at a time.

Applied naively to a multi-host LWS group, a plain pod-level PDB is the wrong granularity: it would happily permit draining the worker pod of a group while leaving the leader pod of that same group still marked available, which satisfies the PDB's counting rule while producing exactly the half-formed shard section 1.1's all-or-nothing restart policy exists to prevent. The correct application of PDB-style reasoning to a gang-scheduled group has to operate at group granularity, not pod granularity — either by selecting whole groups as the disruption-budget unit, or, as Module 9 covers for Dynamo's Grove component, by having a purpose-built gang scheduler (Grove's PodGang object, read and enforced by the KAI Scheduler) reject any partial-group disruption outright rather than relying on a PDB's per-pod counting to happen to preserve group integrity. This module does not re-derive Grove's mechanism — the point to carry forward is that plain Kubernetes PDBs are a necessary but not sufficient primitive once the unit of availability is a multi-pod group rather than a single pod.

7.3 GPU hardware failure detection

A GPU that degrades silently — ECC errors climbing, a fabric link intermittently dropping, a card overheating under sustained load without crashing outright — is a harder failure mode than a pod simply dying, because nothing about the pod's own liveness/readiness probes necessarily notices. Table 4 already flagged this as the one AIBrix component without a standardized generic-Kubernetes-stack equivalent as of this module's access date: AIBrix's Accelerator Diagnostic tooling "automates fault detection, helping users proactively identify and resolve GPU-related performance issues before they impact workloads," paired with a failure mock-up tool that deliberately simulates hardware failures so a fleet's recovery path can be exercised under controlled conditions rather than discovered for the first time during a real outage12. Support currently spans Nvidia GPUs and Ascend 910B NPUs, with additional accelerator support planned12.

Once a fault is detected, the remediation path is the same drain-before-remove mechanism sections 7.1 and 7.2 already established: the affected node is cordoned so no new pods schedule onto it, the pod (or, for a multi-host group, the whole group) is evicted respecting whatever group-level disruption constraint applies, and — for an orchestration layer with its own worker-registration lifecycle, as Module 9 covers for Dynamo's RuntimeConfig publish/deregister path — the control plane routes around the missing capacity automatically rather than requiring a human to manually rebalance. What AIBrix's tooling adds on top of that generic remediation path is the detection step itself: catching a degrading GPU before it produces a failed or, worse, silently-corrupted forward pass, rather than only reacting once the pod has already crashed and Kubernetes' own liveness probe has caught it.

Where experts disagree

Section 3.3 closed on a claim worth stating as a real disagreement rather than a settled fact: that every autoscaling mechanism this module covers scales the count of multi-host replica groups, never the internal shape of one group. Section 2.3 showed KServe going further than that and disabling autoscaling outright for multi-node deployments. Those two positions look contradictory on the surface — one treats a gang-scheduled group as a fine autoscaling unit, the other treats "multi-node" and "autoscaled" as close to mutually exclusive — and the disagreement is real enough that it is worth separating what each side is actually claiming.

Multi-node topology is not autoscaling's job

KServe's own multi-node documentation4

KServe requires autoscalerClass: none for every multi-node InferenceService and states directly that pipelineParallelSize "is not an autoscaling concept" — changing it in production is discouraged strongly enough that the documented path is deleting and recreating the deployment rather than patching it live. Nothing in KServe's own multi-node ServingRuntime exposes a scale-the-group-count path at all; the entire multi-node feature is framed as fixed-topology by design.

  • Honest about a real constraint: TP/PP degree genuinely cannot be resized inside a running shard without breaking collective-communication assumptions, and pretending otherwise invites a worse failure than just not autoscaling.
  • Simpler operationally — no risk of an autoscaler racing a rollout or fighting a manual scale command on a topology that cannot safely change shape.
  • Conflates "cannot resize a group's internal shape" with "cannot change how many groups exist," which are different claims — the second one is exactly what section 3.3's four mechanisms do successfully.
  • Leaves an operator manually sizing a multi-node fleet to peak demand, since KServe's own multi-node path offers no automated alternative.

Scale the pool of groups, not the group itself

Dynamo's Planner (Module 9) and llm-d's KEDA+EPP autoscaling path79

Both treat a multi-host group — an LWS replica, or the equivalent unit under Grove — as the atomic thing that gets added or removed, driven continuously by TTFT/ITL-derived demand signals rather than fixed at deploy time. Neither one ever resizes a running group's TP or PP degree; the Planner's sla mode and llm-d's KEDA ScaledObject both output a target replica count, which LWS's scale subresource or Grove's PodCliqueSet replica field then satisfies by adding or removing whole, unchanged-shape groups.

  • Matches what section 3.1 established: utilization-based signals are broken for LLM serving, and a fleet that cannot scale its group count at all is stuck over-provisioning for peak demand or accepting SLO breaches at the trough.
  • Grounded in the same TTFT/ITL SLO vocabulary this module and Module 15 both use, rather than a separate ad hoc metric.
  • Both mechanisms need a working worker-registration/deregistration path underneath them (Module 4's drain-before-remove, Module 9's RuntimeConfig publish) that a simpler stack may not have implemented; KServe's own multi-node path, notably, does not have one.
  • Predictive modes (Dynamo's sla) carry the bootstrap-accuracy caveat Module 9 already flagged — a performance model with no accuracy guarantee is not obviously safer than KServe's "don't autoscale at all" position during a deployment's early life.
Assessment

These are not actually opposed once "autoscale" is disambiguated into two separate claims. KServe is correct that a running group's internal TP × PP shape cannot be elastically resized — vLLM's own guidance against changing PipelineParallelSize live agrees, and no project surveyed in this module claims otherwise. Dynamo and llm-d are correct that the group as a whole is a perfectly good autoscaling unit, and Table 3's four mechanisms are existence proofs, not proposals. KServe's blanket "no autoscaling for multi-node" position is better read as an honest statement of its own current implementation gap — it has not built a group-level HPA-friendly scale path for its multi-node ServingRuntime — rather than as evidence that group-level autoscaling is unsafe or unworkable in general. What would settle this cleanly is KServe shipping its own group-count autoscaling path for multi-node InferenceServices (there is no committed timeline for this in the documentation available as of this module's access date) and, on the other side, a published incident report of a Planner- or KEDA-driven group-count scaling decision going wrong in production — neither exists yet to falsify the position it would bear on.

References

  1. repoKubernetes SIG contributors. kubernetes-sigs/lws (LeaderWorkerSet README). Main branch. https://github.com/kubernetes-sigs/lws/blob/main/README.md. Accessed 2026-09-17. Used for: LWS's purpose statement, group/leader/worker model, all-or-nothing restart and rollout framing ("groups are upgraded one by one as a unit"), co-design with llm-d, topology co-location and scale-subresource support.
  2. officialvLLM contributors. LWS (vLLM Documentation, deployment/frameworks/lws). https://docs.vllm.ai/en/stable/deployment/frameworks/lws/. Accessed 2026-09-17. Used for: LWS_LEADER_ADDRESS/LWS_GROUP_SIZE/LWS_WORKER_INDEX environment variables, the multiprocessing vLLM-flag mapping, and the Ray-based multi-node-serving.sh startup path.
  3. officialKubernetes SIG contributors. LeaderWorkerSetSpec API Reference (LWS documentation). https://lws.sigs.k8s.io/docs/reference/leaderworkerset.v1/. Accessed 2026-09-17. Used for: exact field names and semantics — replicas, leaderWorkerTemplate.size, leaderTemplate/workerTemplate, restartPolicy, subGroupPolicy, rolloutStrategy.rollingUpdateConfiguration (maxUnavailable/maxSurge), networkConfig.subdomainPolicy.
  4. officialKServe contributors. Multi-node/Multi-GPU Inference (KServe docs). https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node. Accessed 2026-09-17. Used for: the tensorParallelSize x pipelineParallelSize = Total GPUs formula set, the 16-GPU/2-node worked cases (Table 1), the autoscalerClass: none requirement, exactly-one-head-pod constraint, and the RollingUpdate-vs-Recreate resource trade-off.
  5. officialvLLM contributors. Metrics (vLLM Design Documentation, v1 engine). https://docs.vllm.ai/en/latest/design/metrics/. Accessed 2026-09-17. Used for: named vLLM Prometheus gauges/histograms (num_requests_waiting, num_requests_running, kv_cache_usage_perc, TTFT/ITL histograms) reused in Table 2's demand-signal vocabulary.
  6. repoKubernetes SIG contributors. kubernetes-sigs/gateway-api-inference-extension (README). https://github.com/kubernetes-sigs/gateway-api-inference-extension. Accessed 2026-09-17. Used for: the standard's stated goal of routing client model names to LoRA adapters, grounding section 4.1's multi-tenant LoRA-routing pattern.
  7. officialllm-d contributors. KEDA with EPP Metrics (llm-d docs). https://llm-d.ai/docs/dev/architecture/advanced/autoscaling/keda-epp. Accessed 2026-09-17. Used for: why GPU utilization fails as a scaling signal, the EPP-exposed metric set (queue size, pool saturation, running requests, inflight tokens), the KEDA ScaledObject-not-a-second-HPA architecture, and the dual-trigger (queue + running-requests) recommendation.
  8. repollm-d contributors. workload-autoscaling guide (llm-d/llm-d repository, main branch). https://github.com/llm-d/llm-d/blob/main/guides/workload-autoscaling/README.md. Accessed 2026-09-17. Used for: the "lagging indicator" critique of CPU/GPU-based autoscaling, the vllm:num_requests_waiting/kv_cache_usage_perc trigger examples, and the SLO-aware recording-rule scaling path via KEDA scalingModifiers.
  9. officialNVIDIA. Planner Guide (NVIDIA Dynamo Documentation). v1.2.1. https://docs.nvidia.com/dynamo/v1.2.1/components/planner/planner-guide. Accessed 2026-09-17. Used for: default SLA targets (TTFT 500 ms, ITL 50 ms) cited in section 3.3, cross-referencing Module 9's fuller treatment of the Planner's sla mode without re-deriving it.
  10. officialRay / Anyscale contributors. Serving LLMs (Ray Serve Documentation). Ray 2.58.0. https://docs.ray.io/en/latest/serve/llm/index.html. Accessed 2026-09-17. Used for: LLMConfig/build_openai_app/serve.run interface, multi-model llm_configs list, per-deployment autoscaling_config, and Ray-native placement as the basis for section 5.1's Ray-native positioning.
  11. repoAIBrix contributors. vllm-project/aibrix (README). v0.7.0 (2026-06-16). https://github.com/vllm-project/aibrix. Accessed 2026-09-17. Used for: AIBrix's top-level component list (Table 4) and current release version.
  12. paperThe AIBrix Team. AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure. arXiv:2504.03648, 2025. https://arxiv.org/abs/2504.03648. Accessed 2026-09-17. Used for: LLM-specific autoscaling policies and KPA/APA measured gains (§3.1, §3.3), high-density LoRA management mechanism (§4.1), GPU Optimizer ILP heterogeneous-serving trade-off (§4.2), Mix-Grain Ray+Kubernetes orchestration rationale (§5.1), distributed KV cache benchmark numbers (Table 4), and accelerator diagnostic/failure mock-up tooling (§7.3).
  13. officialKubernetes contributors. Safely Drain a Node (Kubernetes Documentation). https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/. Accessed 2026-09-17. Used for: the cordon-then-evict mechanism of kubectl drain and the PodDisruptionBudget minAvailable/maxUnavailable eviction-blocking rule underlying section 7.2.