System Design · Module 31
Multi-Tenancy, Isolation, and Capacity/Cost Management
A GPU inference platform serving more than one paying customer has exactly two structural answers to "how do I stop one tenant's traffic from hurting another's": give every tenant its own hardware, or share the hardware and enforce fairness in software. This module grounds that fork in Borg's admission-by-quota mechanism, in Google's own measurements of what segregating tenants actually costs in machines, and in the two commercial GPU-inference platforms that have published real numbers on each side of the trade.
1. Two designs for tenant isolation
Every earlier module in Part 3 assumed a single logical serving fleet behind the gateway and asked how one request moves through it: which lane it queues in (Module 24), which backend signals route it (Module 25), how its cost is tracked (Module 23) and later attributed (Module 30). This module asks a question those modules deliberately deferred: when the fleet serves more than one paying tenant, is "the fleet" one shared pool of GPUs at all, or is it, from a given tenant's point of view, a private pool that happens to be operated by the same company? Both are real, shipping designs, and the choice is not a scheduling detail -- it determines whether a noisy tenant can ever affect a quiet one, and it determines how much of the platform's GPU fleet sits idle at 3 a.m.
1.1 Dedicated GPU clusters per tenant
Together AI's Dedicated Endpoints, launched with an on-demand pricing update on 2025-03-13, are the clearest vendor-documented instance of the dedicated design. The product's own framing draws the isolation line explicitly: dedicated deployments give "full control over a single-tenant GPU instance to deploy any model, without sharing compute resources," and the company's own head-to-head comparison table lists the reason a customer upgrades from its serverless tier as needing "a single-tenant deployment with no risk of ‘noisy neighbors’"1. That phrase -- noisy neighbors -- is the industry's shorthand for exactly the failure mode dedicated hardware is built to make structurally impossible: one tenant's request pattern (a burst, a pathological prompt shape, a runaway retry loop) changing another tenant's latency or throughput, because the two tenants' requests never touch the same GPU, the same KV-cache pool, or the same batching queue in the first place.
Fireworks AI is reported to run the same design at a larger claimed scale. A third-party summary of Fireworks' public claims, published by GMI Cloud and not independently verified against a Fireworks-authored engineering source, states that Fireworks "runs dedicated GPU clusters for each model and customer tier, eliminating the ‘noisy neighbor’ performance variations common in shared infrastructure," and that the platform processes 200,000 queries per second and more than 10 trillion tokens per day at peak across a custom inference engine2. Section 6 returns to exactly why this claim is marked unverified rather than presented as fact, and why the Together AI numbers above are treated as the load-bearing evidence for the dedicated design instead.
Structurally, a dedicated cluster removes an entire category of platform mechanism rather than adding one. There is no per-request admission check against a shared budget, because there is nothing to be admitted into that another tenant could also claim -- the GPUs are a static allocation, sized (ideally) to the tenant's own peak. Together AI's own scaling knobs make this concrete: a dedicated deployment scales vertically by increasing the GPU count per replica (2, 4, or 8 GPUs), and horizontally by defining "a minimum and maximum replica count," with "additional replicas spin[ning] up on demand" as that one tenant's traffic grows1. Every scaling decision is local to the tenant. The cost is equally local: the tenant (or the platform, if it absorbs the cost) pays for that hardware whether or not it is busy, because dedicated hardware billed by the hour is billed by the hour regardless of utilization.
1.2 Shared pool with per-key budget enforcement
The alternative treats the GPU fleet as one undifferentiated resource pool and pushes isolation entirely into software, at the gateway, before a request ever reaches a GPU. LiteLLM's own architecture documentation describes the request path this way: a virtual key is checked "to ensure it is valid and under its budget," then a parallel-request rate limiter checks requests-per-minute and tokens-per-minute "at global/key/user/team levels," and only after both checks pass does the request reach the router that picks a backend3. Nothing about the GPUs themselves is tenant-specific; the same physical fleet serves every key, and the only thing standing between one tenant's traffic and another's is the rate limiter's arithmetic.
Envoy AI Gateway's usage-based rate limiting implements the same idea with an explicitly two-phase check: "when a request is received: AI Gateway checks if processing this request would exceed the configured token limit. If the limit would be exceeded, the request is rejected with a 429 status code. If within the limit, the request is processed and its token usage is counted towards the total"4. (The project has also been rebranded "Agent Router" in its v0.1 documentation, a naming change worth knowing if you go looking for it directly.) The gateway's own example configuration -- limits such as "1000 total tokens per hour per user" for one model and "5000 total tokens per hour per user" for another4 -- is the software-isolation equivalent of Together AI's per-tenant hardware carve-out: a ceiling, just implemented as a counted quantity in a rate limiter instead of a physical GPU boundary.
Module 22 already covers how these per-key, per-workspace, and per-org budget structures are built (token buckets, fixed vs. sliding windows, the RPM/TPM dimension set), and Module 24 already covers the admission-vs-scheduling split these checks sit inside. This module does not re-derive either. What matters here is the consequence of choosing this design over Section 1.1's: isolation is probabilistic and enforced continuously by a rate limiter reacting to measured usage, not structural and guaranteed by a hardware boundary decided once at provisioning time. A rate limiter that is misconfigured, or a burst that arrives faster than usage counters update, can let one tenant's traffic degrade another's before the limiter catches up -- a failure mode that is, by construction, impossible in Section 1.1's design and, by construction, the tax Section 1.2's design pays for its utilization advantage.
The two designs also put different demands on the GPU-backend interface Module 25 covers. A dedicated cluster's backend only ever sees one tenant's traffic, so the KV-cache-utilization and queue-depth signals Module 25 describes are used purely for that tenant's own scaling and load-balancing decisions -- there is no cross-tenant routing decision to make, because there is only one tenant's requests to route among that tenant's own replicas. A shared pool's backend, by contrast, is exactly the setting those signals were built for: Module 24's routing layer has to pick among backends serving a mix of tenants, so KV-cache occupancy and queue depth are the aggregate-load signals that let the platform keep one tenant's burst from silently starving another's requests for a shared resource neither the rate limiter nor the admission-quota layer can see directly. Isolation-by-hardware and isolation-by-software differ, in other words, not just in where the boundary sits, but in which of this course's earlier mechanisms are even doing meaningful work on either side of it.
None of this is strictly binary in commercial practice, either. Together AI's own product line lists a third tier alongside serverless and Dedicated Endpoints -- "Provisioned Throughput," described in its own site navigation as "token-based capacity with SLAs"1 -- sitting between a fully shared, per-token serverless pool and a fully dedicated, per-GPU-hour deployment: a token-volume commitment large enough to earn an SLA, without necessarily reserving whole GPUs to one tenant. This module does not chase that tier's mechanics further, because the underlying source used here documents it only as a labelled product option, not with the level of technical detail Section 1.1 and 1.2 rely on for the two poles -- but its existence is worth stating plainly: the dedicated-vs-shared fork this module builds around is the two ends of a spectrum real platforms sell, not the only two points on it.
| System | Isolation unit | Enforced at | Pricing model | Evidence quality |
|---|---|---|---|---|
| Borg shared cell5 | None by default -- 98% of machines in shared cells run prod and non-prod together | Priority + quota vector, checked at job admission | Internal cost accounting, not customer-billed | Primary paper, quantified (§5.2) |
| Together AI Dedicated Endpoints1 | Whole GPU(s), 2/4/8 per replica | Provisioning time, per tenant | Hourly, per GPU-hour ($3.36/hr for one H100, as of the source's pricing table) | Vendor's own dated blog post, quoted directly |
| Fireworks AI (claimed)2 | Whole cluster, per model/customer tier | Provisioning time, per tier | Not disclosed in the source used | Third-party summary, not vendor-authored, unfetched primary |
| LiteLLM + Envoy AI Gateway34 | None -- virtual key inside a shared pool | Request time, per token/RPM budget | Whatever the underlying serverless per-token rate is | Vendor's own architecture/API docs |
2. What isolation actually buys: Borg's own evidence
Every dedicated-cluster pitch rests on a claim -- sharing hardware degrades performance for the tenants forced to share it -- that is stated far more often than it is measured. Google measured it directly, at Google's scale, more than a decade before "noisy neighbor" became inference-platform vocabulary, and the paper's own numbers are a better empirical anchor than any single vendor's marketing copy, dedicated or shared.
2.1 The cost of segregation: Borg's cell-sharing experiment
Nearly all of Borg's machines run mixed workloads: "98% of the machines in shared Borg cells" and "83% across the entire set of machines managed by Borg" run production and non-production tasks side by side, with only "a few dedicated cells for special uses"5. Because that is an unusual choice relative to how many other organizations operate -- separate clusters for user-facing and batch work -- the paper's authors ran the counterfactual directly: take a real cell's workload and ask how many machines it would need if production and non-production work were segregated into separate cells instead of sharing one. The answer, reported as a 90th-percentile figure across fifteen representative cells, is 20-30% more machines in the median cell to run the same workload segregated5. The paper's own explanation of why is the whole mechanism in one sentence: "prod jobs usually reserve resources to handle rare workload spikes, but don't use these resources most of the time. Borg reclaims the unused resources... to run much of the non-prod work, so we need fewer machines overall"5. Section 4.1 returns to that reclamation mechanism directly; here, the point is only the size of the number it buys back.
The methodology behind that number is worth knowing before trusting it, because "how many machines does a workload need" is not self-evidently a single number. Borg's authors reject average utilization as their metric -- "we needed a more sophisticated metric than ‘average utilization’" -- and instead use cell compaction: given a real workload, repeatedly repack it from scratch into progressively smaller synthetic cells, removing machines "until the workload no longer fitted," across eleven trials per cell with different random seeds5. They report the 90th-percentile machine count across those trials as the result specifically because "a system administrator wants to be reasonably sure that the workload would fit," not merely that it fits on average5. Every overhead percentage in this section -- 20-30% for prod/non-prod segregation, 20-150% for per-user segregation -- is the additional 90th-percentile machine count this repacking procedure finds, applied to real production constraints, placement rules, and actual (not modeled) usage data pulled from live Borg checkpoints5. That is a materially stronger methodology than a back-of-envelope utilization ratio, and it is part of why this module treats these numbers as the load-bearing evidence for the trade-off curve in Section 5 rather than as an illustrative estimate.
Borg ran the same experiment at tenant, not workload-class, granularity, and this is the closer analogue to a dedicated-cluster-per-customer design. The authors "split off a user's workload into a new cell if they consumed at least 10 TiB of memory (or 100 TiB)" -- i.e., gave each sufficiently large user a private cell instead of a shared one -- and found that "even with the larger threshold, we would need 2-16× as many cells, and 20-150% additional machines. Once again, pooling resources significantly reduces costs"5. Read plainly: giving your largest tenants their own dedicated capacity, sized so each doesn't run out, costs you a factor of two to sixteen more clusters and up to one and a half times the machine count of a single shared pool serving everyone. That is the utilization tax a dedicated-per-tenant design pays, measured, not asserted -- and it is the single strongest piece of grounding available anywhere in this course's source map for the "dedicated wins on isolation, loses on utilization" side of Section 5's trade-off curve.
2.2 Is shared compute actually worse? CPI interference data
Segregation costing more machines only justifies dedicated hardware if sharing actually hurts the tenants forced to share. Borg checked this too, using cycles-per-instruction (CPI) as a proxy for interference -- "a doubling of CPI doubles the runtime of a CPU-bound program"5 -- sampled from roughly 12,000 randomly selected production tasks over a week. The regression result is a genuine surprise if you expect noisy-neighbor effects to dominate: "adding a task to a machine increases the CPI of other tasks by 0.3%... increasing machine CPU usage by 10% increases CPI by less than 2%," and even though both correlations are statistically significant, "they only explain 5% of the variance... other factors dominate, such as inherent differences in applications and specific interference patterns"5. Co-location itself is a weak predictor of interference; what an application is matters far more than how many neighbors it has.
Comparing shared cells to a small sample of dedicated ones directly, the paper found "a mean CPI of 1.58 (σ = 0.35) in shared cells and a mean of 1.53 (σ = 0.32) in dedicated cells -- i.e., CPU performance is about 3% worse in shared cells"5. A second measurement, isolating the Borglet agent process that runs identically on every machine in both cell types (removing the confound that different applications might be selected into dedicated cells), found a CPI of 1.20 in dedicated cells versus 1.43 in shared ones -- "1.19× as fast in a dedicated cell as in a shared one" -- though the authors flag that this measurement "over-weights the effect of lightly loaded machines, slightly biasing the result in favor of dedicated cells"5. Table 2 lays out both experiments side by side.
| Experiment | Shared | Segregated / dedicated | Overhead of segregating |
|---|---|---|---|
| Prod vs. non-prod, separate cells | 1× baseline machines | 1.2-1.3× machines | 20-30% |
| Per-user private cells (≥10 TiB threshold) | 1× baseline machines | 1.2-2.5× machines, 2-16× cells | 20-150% |
| CPI, sampled prod tasks | 1.58 (σ=0.35) | 1.53 (σ=0.32) | ~3% worse shared |
| CPI, Borglet agent only | 1.43 (σ=0.45) | 1.20 (σ=0.29) | ~19% worse shared |
3. Borg's quota-checked-at-admission, reapplied at tenant granularity
Module 24 already builds the general admission-control mechanism this section reuses: Borg's quota is "a vector of resource quantities (CPU, RAM, disk, etc.) at a given priority, for a period of time," "quota-checking is part of admission control, not scheduling," and a job with insufficient quota is "immediately rejected upon submission" without Borg ever consulting live machine state5. Module 24 applies that rule to a single job or request: does this unit of work fit inside its owner's already-negotiated allocation. This module applies the same rule one level up the hierarchy, to a question Module 24 does not ask: how large should a tenant's entire allocation be in the first place, and does that allocation take the form of dedicated hardware or a budget inside a shared pool?
The distinction matters because the two designs from Section 1 answer the sizing question differently, even though both are still, structurally, a quota vector decided in advance and checked cheaply at the boundary.
- Dedicated path. The tenant's quota vector is the hardware: a fixed count of GPUs, replicas, and a scaling ceiling, decided at provisioning time. There is no shared ledger to check at request time -- Together AI's own scaling model exposes exactly two knobs, GPU count per replica and min/max replica count1, both fixed until an operator changes them. "Admission" degenerates to a Boolean: is there an idle replica or headroom to autoscale into. If not, the request queues or is rejected, but it never competes against a different tenant's quota, because no such shared quota exists.
- Shared path. The tenant's quota vector is a budget -- tokens or requests per minute, checked against a per-key counter -- inside a pool whose total physical capacity is shared across every tenant. LiteLLM's virtual-key budget check and parallel rate limiter run exactly this check before a request reaches the router3; Envoy AI Gateway's token-limit check is the same idea with an explicit two-phase estimate-then-reconcile shape4. Because the underlying GPUs are shared, admitting one tenant's request against its own budget says nothing about whether the pool as a whole has spare capacity -- that is a separate, aggregate check Module 24's Section 2 already covers for the single-tenant case, and which Section 5 below extends across tenants.
Borg's own "infinite quota at priority zero, frequently hard to exercise because resources are over-subscribed"5 pattern has a direct reading at tenant granularity in the shared design: a platform's lowest-tier customers (free trial, evaluation tier) can be sold a nominally unlimited or very generous token budget precisely because that budget sits behind every paying tenant's budget in scheduling priority, and is rarely fully exercisable in practice once the shared pool is under real load. A platform offering a "generous free tier" on a shared pool is doing the same oversell-at-the-bottom trick Borg's priority-zero quota does, not offering something structurally different.
3.1 Worked derivation: sizing a tenant's quota vector against peak and average demand
Take a tenant whose measured request pattern has a peak throughput \(P\) (tokens/minute at its busiest observed interval) and an average throughput \(\bar{D}\) over a representative window, with a burstiness ratio \(\beta = P / \bar{D} \ge 1\). Both designs from Section 1 must provision for \(P\), because provisioning below peak means either dropped requests (dedicated) or budget-exceeded rejections (shared) at the tenant's busiest moments -- but the two designs differ in what happens to the gap between \(P\) and \(\bar{D}\) the rest of the time.
In the dedicated design, the tenant's hardware allocation is sized to \(P\) (plus headroom), and that capacity is idle -- unreachable by any other tenant, per Figure 1's Path A -- for the fraction of time the tenant runs below peak. The wasted capacity-hours per unit time is proportional to \((1 - 1/\beta)\): a tenant with \(\beta = 4\) (peak four times its average) leaves roughly three-quarters of its provisioned dedicated capacity idle on average. Borg's own cell-sharing numbers from Section 2.1 are exactly a measurement of this effect aggregated across many tenants: dedicating capacity per user costs 20-150% more machines than pooling the same tenants' demand into one cell5, precisely because each tenant's own \(\beta\) creates idle capacity that only a shared pool can reclaim for someone else's use.
In the shared design, the tenant's quota vector is still sized to \(P\) as a ceiling (the platform still needs to protect itself and other tenants from that tenant exceeding its committed rate), but the physical capacity behind that ceiling is not reserved to the tenant -- it is drawn from the same pool every other tenant draws from. The platform's total physical capacity only needs to cover the sum of tenants' average demand plus a much smaller aggregate burst margin, not the sum of every tenant's individual peak, because it is statistically unlikely that every tenant peaks simultaneously. This is the same law of large numbers that lets an electricity grid serve millions of households without provisioning for every household's peak draw at once, and it is the formal reason Section 5's utilization curve places the shared design closer to 100% utilization than the dedicated one for the same aggregate demand.
4. Capacity planning: burstiness, idle reclamation, and peak-provisioning cost
Section 3.1's derivation showed why a bursty tenant is expensive under the dedicated design and cheap under the shared one; this section covers how a platform actually claws back the idle capacity a bursty tenant leaves behind, and how a platform decides, per tenant, which side of the trade-off it should even offer.
4.1 Resource reclamation as the general mechanism
Borg's answer to "a task requested more resources than it uses most of the time" is not to shrink the request -- shrinking a live allocation risks killing the task if it later needs the resources back -- but to estimate actual usage continuously and lend the unused difference to lower-priority work that can tolerate losing it. The paper's own description: "we estimate how many resources a task will use and reclaim the rest for work that can tolerate lower-quality resources, such as batch jobs. This whole process is called resource reclamation. The estimate is called the task's reservation... The initial reservation is set equal to the resource request (the limit); after 300s, to allow for startup transients, it decays slowly towards the actual usage plus a safety margin. The reservation is rapidly increased if the usage exceeds it"5. High-priority latency-sensitive tasks never rely on reclaimed resources and are scheduled purely against limits, so they are never exposed to another task's estimate being wrong5 -- reclamation only ever lends spare capacity to work that can afford to lose it, never borrows capacity a protected tenant is depending on.
The scale of what this buys back is Borg's own headline number for the mechanism: "about 20% of the workload runs in reclaimed resources in a median cell"5, and disabling reclamation entirely would require substantially more machines across the fifteen cells the paper measured -- roughly 5% to 45% additional machines depending on the cell5. This is the mechanism a shared-pool inference platform's capacity planning is implicitly leaning on every time it sizes a fleet below the sum of every tenant's peak: it is betting that tenants' idle periods are not simultaneous, and that a reservation that decays toward measured usage (rather than staying pinned to a worst-case request) can safely find and reallocate the resulting slack.
How aggressively to run that estimate is itself a tuned parameter with a measured downside, not a free lunch, and Borg's own operators ran the experiment on a live cell: they "picked a live production cell and adjusted the parameters of its resource estimation algorithm to an aggressive setting for a week... and then to a medium setting... and then reverted to the baseline"5. Reservations tracked usage more tightly in the aggressive week, which is the whole point -- less headroom between what a task reserved and what it actually used means more of the gap is available to reclaim -- but "the rate of out-of-memory (OOM) events increased slightly in weeks 2 and 3"5, because a tighter reservation estimate is more likely to be wrong on the occasions a task's usage spikes unexpectedly. Google's own conclusion after running the comparison was that "the net gains outweighed the downsides," and it deployed the medium setting -- not the most aggressive one -- fleet-wide5. The general lesson transfers directly to sizing a shared inference pool: the more tightly a platform estimates a tenant's steady-state resource use in order to reclaim the rest for other tenants, the more exposed it is to that tenant's genuine, if infrequent, spikes -- and the correct operating point is not the most aggressive one the mechanism supports, but the one where a measured failure-rate increase (OOMs, here; queuing delay or throttling, on an inference platform) is still acceptable against the utilization it buys.
4.2 Worked derivation: the dedicated-vs-shared breakeven
Together AI's own published pricing turns Section 3.1's idle-capacity argument into a specific number a tenant (or a platform pricing a tenant) can compute. As of the source's pricing table, an H100 80GB GPU costs $3.36/hour on a Dedicated Endpoint1, and the company's own worked comparison prices a two-GPU H100 dedicated deployment against serverless (shared-pool, per-token) pricing for four popular models, reporting the throughput at which dedicated becomes the cheaper option1:
| Model | Breakeven (tokens/min) |
|---|---|
| Meta Llama 3.3 70B Instruct Turbo | 127,273 |
| Mixtral-8x7B Instruct v0.1 | 186,667 |
| Mistral-7B-Instruct-v0.2 | 560,000 |
| Meta Llama 3.1 8B Instruct Turbo | 622,222 |
Both the hourly dedicated price and the breakeven throughput are Together AI's own published numbers, which means the implied serverless per-token price is not a separate assumption -- it falls out of the two numbers the company already discloses. A two-GPU deployment costs \(2 \times \$3.36/\text{hr} = \$6.72/\text{hr}\), or \(\$6.72 / 60 = \$0.112\) per minute. If serverless is priced at \(c\) dollars per token, the breakeven condition is the point where the shared-pool bill per minute equals the dedicated bill per minute:
$$ c \times T_{\text{breakeven}} = \frac{H}{60} $$where \(H\) is the dedicated deployment's hourly cost and \(T_{\text{breakeven}}\) is throughput in tokens/minute. Solving for the implied serverless price using Llama 3.3 70B's own published breakeven of 127,273 tokens/min1:
$$ c = \frac{H/60}{T_{\text{breakeven}}} = \frac{\$0.112}{127{,}273} \approx \$8.8 \times 10^{-7}/\text{token} \approx \$0.88 \text{ per million tokens} $$The same H100-pair hourly rate applies to every row of Table 3 -- Together AI's own comparison holds the hardware fixed and varies the model -- so the same formula run against Mixtral-8x7B's published breakeven of 186,667 tokens/min1 gives \(c = \$0.112 / 186{,}667 \approx \$6.0 \times 10^{-7}/\text{token} \approx \$0.60\) per million tokens, a lower implied serverless price than Llama 3.3 70B's. That ordering is consistent with the two models' relative sizes and expected serving cost -- a cheaper-to-serve model needs a higher token volume before the flat dedicated rate pays for itself, exactly what a larger breakeven threshold means -- and it is a useful sanity check on the arithmetic itself: solving the same two-variable equation against two different rows of a single published table should, and here does, produce internally consistent implied prices rather than contradictory ones.
The general form, \(T_{\text{breakeven}} = H / (60c)\), is the capacity-planning rule a platform actually needs: a tenant's average sustained throughput, not its peak, decides which side of the breakeven it sits on, because the dedicated hourly bill is paid whether the GPU is busy or idle while the shared-pool bill is paid only for tokens actually generated. A tenant whose average throughput is below \(T_{\text{breakeven}}\) is, by this arithmetic, being overcharged by a flat-rate dedicated deployment relative to what the same traffic would cost on a shared pool -- which is exactly why Together AI's own comparison table frames serverless as the right choice for "unpredictable or highly variable" traffic and dedicated for "sustained traffic [that] justifies reserved capacity costs"1: sustained, not peak, throughput is the deciding variable, because it is average utilization of the reserved hardware that the flat hourly rate is being paid for.
4.3 Cost attribution as the capacity feedback signal
Section 4.2's breakeven arithmetic is only actionable if a platform actually knows each tenant's sustained throughput, not just its instantaneous rate-limit consumption. That per-tenant, per-request cost signal is Module 30's subject: cost attribution per request and tenant is described there as "the economic close of the observability loop," building on the per-token spend tracking Module 23's ledger and Module 22's virtual-key budgets already produce as a side effect of billing. This module's dependency on that signal runs one direction only -- capacity planning consumes an attributed-cost time series as an input, it does not generate one. Concretely: a platform tracking each tenant's attributed spend and token volume over time has, for free, the \(\bar{D}\) and \(\beta\) inputs Section 3.1's derivation needs, and can re-run Section 4.2's breakeven calculation on a rolling basis to decide, per tenant, whether a dedicated allocation, a shared-pool budget, or a change in tier is the cheaper way to serve that tenant's actual measured traffic -- rather than a one-time decision made at sales time and never revisited as a tenant's usage pattern shifts.
LiteLLM's own spend tracking already carries this rollup structure: spend for a key "rolls up to a user_id and team_id" in the platform's own tables3, which is the raw material a capacity-planning process would aggregate across a longer window than any single rate-limit check ever looks at. A rate limiter answers "is this request, right now, within budget"; capacity planning answers a slower question over the same underlying data -- "given this tenant's last month of attributed usage, is its current isolation tier still the cheapest one that meets its needs."
A genuinely new tenant has no such history to consult, which is a real gap in the feedback loop rather than an edge case: a platform onboarding a large customer has to make an initial dedicated-vs-shared decision before Module 30's attribution signal has accumulated anything to re-run Section 4.2's breakeven against. In practice this initial decision is made on a declared or contracted volume estimate -- a sales conversation's projected tokens/month, or a minimum-commitment figure in an enterprise agreement -- treated as a provisional value for \(\bar{D}\) until measured usage replaces it. The asymmetry in the cost of being wrong favors starting a new tenant on the shared path whenever the projection is uncertain: an under-provisioned shared-pool budget is a rate-limit ceiling that can be raised the moment measured usage justifies it, while an over-provisioned dedicated deployment is committed hardware spend that was, by Section 4.2's own arithmetic, never justified in the first place. Only once attributed usage confirms sustained throughput above the breakeven point does migrating that tenant to a dedicated allocation stop being a bet and start being the arithmetically cheaper option.
5. The utilization-vs-isolation trade-off curve
Sections 2 and 4 produced two families of numbers that live on the same pair of axes: how much worse a shared pool's CPU performance measures relative to a dedicated one (Borg's CPI overhead, Section 2.2), and how much aggregate utilization a platform gives up by dedicating capacity per tenant instead of pooling it (Borg's segregation-overhead numbers, Section 2.1, read as their utilization-relative inverse). Figure 3 places both families on one chart, alongside the two commercial reference points from Section 1 -- Together AI's verified design and Fireworks' unverified one -- kept visually and evidentially separate from Borg's measured axis rather than plotted as if they carried the same kind of number.
Read as a whole, Figure 3 makes the shape of the trade-off concrete rather than schematic: Borg's own fleet did not find a large interference cost to sharing (3% on the primary, less-biased comparison) but did find a large utilization cost to segregating (up to 150% more machines)5. That asymmetry is why Google runs 98% of its shared-cell machines mixed5 and why the shared design is the default in every commercial inference gateway this course's source map documents (LiteLLM, Envoy AI Gateway). Dedicated hardware remains the right answer for a specific tenant profile -- Together AI's own guidance names it directly: "sustained traffic justifies reserved capacity costs"1 -- but the burden of proof, on Borg's own numbers, sits with the dedicated design, not the shared one.
6. Source confidence: the Fireworks problem
Section 1.1 introduced Fireworks AI's claimed dedicated-cluster design and flagged it as unverified rather than citing it as fact, and it is worth stating precisely why, because the distinction is the difference between citing a company's engineering and citing a summary of a company's marketing. The claim available in this course's source map -- that Fireworks "runs dedicated GPU clusters for each model and customer tier" and processes "200,000 queries per second and 10+ trillion tokens per day at peak"2 -- comes from a blog post published by GMI Cloud, a different company, summarizing Fireworks' own public claims. No Fireworks-authored engineering post at a comparable level of technical detail was locatable to corroborate it directly2. That does not make the claim false. It makes it a claim about Fireworks, made by someone other than Fireworks, with no independent measurement backing it -- structurally weaker evidence than a vendor's own dated blog post making the same kind of claim about its own product, which is exactly what Together AI's Dedicated Endpoints post is1.
The comparison matters beyond just this module's two examples, because it is a recurring failure mode in how "everyone knows platform X does Y" claims propagate through the AI-infrastructure commentary ecosystem: a vendor makes a claim in a sales conversation or a conference talk, a third-party blog or newsletter summarizes it without the original technical detail, and later citations point to the summary rather than tracking down whether the vendor ever published the underlying mechanism. Table 1's "evidence quality" column exists specifically to keep this distinction visible rather than letting a widely-repeated number quietly become treated as settled fact.
It is worth separating the two different kinds of claim bundled inside the Fireworks summary, because they carry different confidence even within the same weak source. The architectural claim -- that Fireworks runs dedicated clusters per model and customer tier2 -- is a design choice a company would plausibly disclose accurately even secondhand, because it is the kind of positioning statement a vendor repeats consistently in sales material, talks, and interviews; a third party summarizing it is unlikely to invert or badly distort it. The throughput claim -- 200,000 queries per second and more than 10 trillion tokens per day2 -- is a different kind of number entirely: a specific, large, and easily rounded or misattributed figure of the sort that drifts across secondary sources (a peak number becomes an average, a company-wide total becomes a per-model total, a projection becomes a measured result) without anyone downstream having a way to catch the drift. This module's citations of the Fireworks source lean on the architectural claim, which is lower-risk to relay incorrectly, and mark the throughput figures specifically inside <mark class="unverified"> rather than treating the whole source as uniformly untrustworthy or uniformly fine.
Where experts disagree
The disagreement this module is built around is not abstract: it is the actual design fork between the two families of commercial and internal systems covered above, and the evidence for each side comes from parties with a direct interest in their own answer being right. Reading the two positions against Borg's independently-measured numbers is the closest this module can get to an outside check.
Dedicated per-tenant GPU pools are worth the utilization cost
Together AI's Dedicated Endpoints product, and Fireworks AI's claimed architecture (flagged, Section 6)12
Physical isolation is the only way to make a noisy-neighbor guarantee structural rather than probabilistic: "single-tenancy, ensuring your traffic is never impacted by other users"1 is true by construction once no other tenant's requests can reach the hardware, whereas any software-enforced budget can, in principle, be misconfigured, race a burst, or fail to account for a cost dimension the platform didn't anticipate. For a tenant whose traffic is sustained above the breakeven throughput derived in Section 4.2, dedicated hardware is also the cheaper option, not just the safer one.
- Isolation is a hardware fact, not a policy that can be misconfigured or race a burst.
- Cheaper than shared, per Section 4.2's own arithmetic, once a tenant's sustained throughput clears the breakeven point.
- Full control over hardware generation, optimization flags (e.g., enabling speculative decoding1), and placement.
- Borg's own measurement of the identical structural choice found 20-150% more machines needed to segregate tenants this way5 -- the utilization cost is not hypothetical.
- Below the breakeven throughput, the tenant (or the platform absorbing the cost) pays for idle GPU-hours regardless.
- Fireworks' own claimed numbers for this design are not independently verifiable from a vendor-authored source2.
Shared pools with per-key budgets maximize utilization and are the industry default
LiteLLM's proxy architecture, Envoy AI Gateway's usage-based rate limiting, and Borg's own operating practice at Google345
Google runs 98% of its shared-cell machines with mixed workloads specifically because segregating them costs more hardware than the interference it avoids is worth5, and every commercial inference gateway this course's map documents defaults to the same shared-pool-plus-budget shape. A rate limiter that checks a virtual key's spend and RPM/TPM before routing34 bounds cross-tenant interference to whatever a misconfigured or slow-reacting limiter allows through, which in practice is a narrow, engineerable window, not an open-ended risk.
- Borg's own fleet-wide CPI comparison found only about 3% worse CPU performance from sharing5 -- a small, bounded cost.
- Avoids paying for idle reserved hardware; capacity is drawn from a common pool sized to aggregate, not per-tenant peak, demand.
- The same mechanism (per-key budgets) already exists for billing (Module 22, 23) -- no separate isolation infrastructure to build.
- Isolation is enforced continuously by software that can be misconfigured, lag a burst, or have a dimension of cost it doesn't account for.
- A single-process (Borglet) CPI comparison found a considerably larger 19% gap, biased toward dedicated cells but not zero5 -- the interference cost is small on average, not nonexistent for every workload.
- Cannot offer a tenant with strict compliance or SLA requirements a hardware-level guarantee, only a policy-level one.
On the only independently-measured evidence available -- Borg's own decade of production data -- the shared design wins by default: a 20-150% utilization cost is large and certain, while the interference cost of sharing measured out to about 3% on the primary, less-biased comparison5. That is why Google runs almost everything shared, and it is why every commercial AI-inference gateway this course documents defaults to shared pools with per-key budgets rather than dedicated-by-default. The dedicated design is the right answer for a specific, checkable condition, not a generally superior one: a tenant whose sustained throughput clears the Section 4.2 breakeven, or whose compliance/SLA requirements need a hardware-level guarantee a software budget cannot provide regardless of cost. What would settle this more precisely than Borg's decade-old, general-purpose-cluster numbers is a published interference measurement from an actual multi-tenant LLM-inference platform -- CPI or an equivalent proxy measured on GPU-bound decode workloads specifically, comparing a shared vLLM/SGLang-style continuous-batching pool against a dedicated one under realistic bursty tenant traffic. Neither Together AI's nor Fireworks' public material contains that measurement; until one of them (or a comparable platform) publishes it, this module's numbers are the best available evidence, borrowed from a different, if closely analogous, domain.
What is changing
Every specific number in this module's commercial sourcing is a pricing-page or blog-post snapshot, not a fixed specification. Together AI's own post states its 2025-03-13 update cut Dedicated Endpoint prices by "up to 43%"1 -- a company actively repricing a product line this aggressively should be expected to move the specific hourly and breakeven numbers in Table 3 again, likely downward as GPU supply and competing inference engines improve, without necessarily changing the underlying breakeven mechanism Section 4.2 derives. Envoy AI Gateway's own project has already renamed itself "Agent Router" at the v0.1 documentation stage referenced here4, which is the kind of naming churn expected of a project this young. The Fireworks claim flagged throughout Sections 1 and 6 is, in principle, resolvable: if Fireworks publishes its own engineering account of its multi-tenancy architecture, it should replace the third-party summary used here as this module's citation for that platform, and the "unverified" mark should be removed accordingly. Borg's own numbers, by contrast, are unlikely to move -- they describe a decade of production operation already closed out at the time of publication -- but they are also the one dataset in this module not drawn from an AI-inference platform at all, and a directly comparable measurement from a GPU-serving system, when one is published, should be preferred over borrowing Borg's CPU-cluster numbers by analogy.
References
- blogTogether AI. On-demand dedicated endpoints: run inference with unmatched price-performance & control at scale. Published 2025-03-13. https://www.together.ai/blog/on-demand-dedicated-endpoints. Accessed 2026-09-18. Used for: the "single-tenancy... never impacted by other users" and "no risk of noisy neighbors" claims; per-GPU hourly pricing table (H100 80GB $3.36/hr); dedicated-vs-serverless breakeven throughput table by model; vertical/horizontal scaling mechanics; when-to-choose-each guidance.
- blogGMI Cloud (summarizing Fireworks AI's public claims). Fireworks AI Deep Dive: Inference at Scale. 2026. https://www.gmicloud.ai/en/blog/fireworks-ai-inference-at-scale. Accessed 2026-09-17. Used for: Fireworks' claimed dedicated-GPU-cluster-per-customer design and throughput figures, explicitly flagged throughout as a third-party-relayed, unverified secondary source (Sections 1.1, 6, disagreement).
- officialBerriAI / LiteLLM contributors. Life of a Request (LiteLLM Proxy architecture). docs.litellm.ai, 2026. https://docs.litellm.ai/docs/proxy/architecture. Accessed 2026-09-17. Used for: virtual-key budget check and parallel rate limiter preceding the router; per-key/user/team spend rollup.
- officialEnvoy AI Gateway project (rebranded "Agent Router" in v0.1 docs). Usage-based Rate Limiting. v0.1 docs, 2026. https://aigateway.envoyproxy.io/docs/0.1/capabilities/usage-based-ratelimiting/. Accessed 2026-09-17. Used for: the two-phase check-then-process-then-count token-budget admission design; example per-hour token-limit configuration.
- paperVerma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., Wilkes, J. Large-scale cluster management at Google with Borg. EuroSys '15. https://research.google.com/pubs/archive/43438.pdf. Accessed 2026-09-17. Used for: quota-as-vector and quota-checked-at-admission (§2.5), reapplied at tenant grain rather than re-derived; cell-sharing segregation-cost experiment and per-user dedicated-cell experiment (§5.2, Fig. 5-6); CPI interference measurements, fleet-wide and Borglet-only (§5.2); resource-reclamation mechanism and the ~20%-of-workload figure (§5.5, Fig. 10); shared-cell prevalence (98%/83%, §5.2).