System Design · Module 22
Identity, Rate Limits, and Quotas: API Keys, Orgs, and Token-Bucket Enforcement
Every request to an inference API carries a bearer key that resolves through an organization/workspace hierarchy to two independent gates — a spend limit measured in dollars per month and a rate limit measured in requests or tokens per minute — and the two gates fail in different, deliberately distinguishable ways. This module reads OpenAI's and Anthropic's own rate-limit documentation for what it reveals about that hierarchy, applies Module 10's fixed-window/sliding-window/token-bucket vocabulary to the specific problem of a request whose true cost is unknown until generation finishes, and applies Module 10's BeyondCorp/Zanzibar vocabulary to org-level authorization on an API platform. After this module you can read either platform's 429 response and say exactly which limit tripped, why, and whether retrying is even possible.
1. The identity hierarchy incidentally revealed by rate-limit docs
Neither OpenAI nor Anthropic has published a document titled "API key architecture" or "how our OAuth/service-account model works." The identity hierarchy each platform actually runs is visible only where a different document — the rate-limit page — has to state it in passing, because a limit is meaningless without saying at what level it is enforced. That is a real gap in what is publicly documented, not a stylistic choice by this module: no primary source in this course's source map describes either platform's identity system as its own subject, and the hierarchy below is reconstructed entirely from what each platform's rate-limit documentation says about where a limit lives.
1.1 OpenAI: organization, project, and cumulative spend
OpenAI's rate-limit documentation states its scope directly: "rate limits are defined at the organization level and at the project level, not user level"2. That single sentence fixes two levels of the hierarchy — organization above, project below — and rules out a third: there is no per-human-user rate limit, only per-organization and per-project ones, which means every API key inside a given project shares that project's budget regardless of which individual created the key. OpenAI's six rate-limit dimensions (RPM, RPD, TPM, TPD, IPM, and audio minutes per minute; detailed in §3.3) are each tracked at both levels, and the response headers reflect this explicitly with a parallel project-scoped set — x-ratelimit-limit-tokens alongside x-ratelimit-limit-project-tokens2 — so a client inspecting a single response can see both its project's remaining budget and (implicitly) that a higher, organization-wide ceiling exists above it.
The other half of OpenAI's identity-adjacent structure is the usage tier, and it is keyed to a number that has nothing to do with request volume: cumulative dollars paid. An organization starts in the Free tier (available only in allowed geographies) and moves to Tier 1 after paying $5, Tier 2 after $50, Tier 3 after $100, Tier 4 after $250, and Tier 5 after $1,000 — cumulative payment history, not a subscription plan a customer selects2. Each tier carries its own monthly usage cap, independent of the per-minute rate-limit dimensions: $100/month for Free and Tier 1, $500 for Tier 2, $1,000 for Tier 3, $5,000 for Tier 4, and $200,000 for Tier 52. The mechanism is a graduated trust ladder that an organization climbs automatically by paying more over time, not a plan it negotiates — a shape that makes sense for a platform whose primary abuse vector is a large number of low-trust, low-history signups rather than a small number of enterprise accounts.
1.2 Anthropic: organization, workspace, and named plan tiers
Anthropic's rate-limit documentation states its hierarchy just as directly, using a different pair of nouns: "the API enforces service-configured limits at the organization level, but you may also set user-configurable limits for your organization's workspaces"1. Organization is the same top level OpenAI uses; workspace replaces OpenAI's project as the second level, and — unlike OpenAI's projects, which the documentation does not describe as independently configurable by the customer — Anthropic states plainly that workspace-level limits are something the organization itself sets, below the org-wide ceiling, specifically "to protect Workspaces in your Organization from potential overuse"1 (the mechanics of this are worked through with real numbers in §4). A response header makes the resolution concrete for any given request: anthropic-workspace-id "carries the ID of the Workspace that your API key or access token resolved to"1 — direct textual confirmation that an API key is not itself the unit of account; it resolves to a workspace, which resolves to an organization.
Anthropic's tier structure differs from OpenAI's in kind, not just in name. Where OpenAI's tiers are unlabeled numbers keyed to cumulative payment, Anthropic's are named plans — Start, Build, Scale, and Custom — each carrying an explicit monthly spend cap set by the plan rather than accumulated by history: $500 for Start, $1,000 for Build, $200,000 for Scale, and no cap at all for Custom, where limits are negotiated directly with an account team1. Anthropic's documentation also names a state OpenAI's does not: new organizations, or organizations without much usage history, may start in an Evaluation tier below the standard tiers' published limits, "as part of how Anthropic prevents fraud and abuse," with limits that increase automatically as usage history accumulates1 — functionally similar to OpenAI's payment-gated tier ladder, but framed as a probation period rather than a rewards ladder, and applied before an organization is even assigned to Start/Build/Scale rather than as one more rung within that ladder.
1.3 The documented gap, and the one fully-documented reference hierarchy
Both platforms' rate-limit pages agree that a two-level customer hierarchy exists (organization above a second unit — project or workspace — below), and both agree that unit-of-account is tracked per key resolving through that hierarchy. Neither page says anything about how a key is issued, rotated, or scoped to a service account; neither describes an OAuth flow, a machine-identity credential, or how a human user's console login relates to the keys that user can mint. That is not an oversight in this module — it is the actual state of what either lab has published, and a reader who needs deeper API-key or service-account mechanics has nowhere in either platform's own documentation to look beyond the rate-limit and API-key-management console pages, which describe the mechanics of creating a key but not the architecture behind it.
The one identity hierarchy in this space that is fully documented at an architectural level belongs to neither frontier lab but to LiteLLM, the open-source LLM gateway/proxy Module 21 introduces as a reference implementation of the gateway request lifecycle. LiteLLM's own architecture document specifies virtual keys as the unit a client actually holds, and states the rollup explicitly: "spend for that user is tracked... team in the LiteLLM_TeamTable"3 — a three-level hierarchy of key, user, and team that a proxy operator configures directly, with "hard limits (tokens, requests, upstream-billing caps), soft limits with alerts, model whitelists, and rate limits" attached at the key level and rolled up at each level above it3. LiteLLM's model is worth holding next to OpenAI's and Anthropic's precisely because it is the mirror image of the gap: it is a fully specified, code-visible hierarchy for a self-hosted gateway sitting in front of either lab's API, built by an operator who needed exactly the identity mechanics neither lab documents for its own hosted product.
| Level | OpenAI | Anthropic | LiteLLM (self-hosted gateway) |
|---|---|---|---|
| Top level | Organization | Organization | Team |
| Second level | Project | Workspace | User |
| Unit a client holds | API key (per project) | API key (resolves to a workspace) | Virtual key |
| Tier naming | Free / Tier 1–5, keyed to cumulative $ paid | Evaluation / Start / Build / Scale / Custom, named plans | operator-defined, no fixed tiers |
| Example monthly cap | $200,000 at Tier 5 | $200,000 at Scale | configured per key/user/team |
| Second-level limit configurability | not described as customer-configurable | customer sets workspace limits below org ceiling1 | fully configurable per level3 |
| Dedicated architecture document | none | none | yes3 |
2. Spend limits vs rate limits: two mechanisms, two error signatures
Anthropic's documentation draws the distinction as cleanly as any source in this course draws any distinction: "spend limits set a maximum monthly cost an organization can incur for API usage. Rate limits set the maximum number of API requests an organization can make over a defined period of time"1. One bounds a dollar total accumulated over a calendar month; the other bounds a rate — requests, tokens, images, or audio-minutes per minute or per day. A caller can be well under its rate limit and still blocked because the org has spent its monthly cap, and can be far from the spend cap while still throttled because it sent 200 requests in the same second a 60 RPM limit allows only one. The two mechanisms answer unrelated questions and are checked independently, which is exactly what Figure 1's two-gate structure shows: passing the spend gate says nothing about the rate gate, and vice versa.
2.1 Why the error has to look different
Both failures return the same HTTP status and, on Anthropic's API, literally the same error.type string: rate_limit_error1. That shared label is not sloppy naming — a rate limit and an exhausted spend cap are, from the caller's point of view, both instances of "you cannot make this request right now" — but it creates a real problem for any client that wants to react correctly: a client that retries a genuine rate limit after backing off is doing the right thing; a client that retries an exhausted monthly spend cap in a loop will burn every retry for the rest of the calendar month for nothing. Anthropic's documentation resolves the ambiguity with two separate signals rather than a third top-level error type: the retry-after header is present on an ordinary rate limit and absent on the spend-cap response, and the spend-cap response additionally carries error.details.error_code: enforced_spend_limit_reached, which the documentation states exists specifically so a client can "tell this response apart from a rate limit"1. "Retrying, including the SDKs' automatic retries, fails until access resumes" is stated explicitly for the spend-cap case — a caller is told plainly that no amount of backoff helps, only waiting for the next billing month or requesting a higher limit1.
There is a third distinct signature that is easy to miss, because it looks similar to the other two but is neither: a self-imposed spend limit — one an organization sets below its tier's cap voluntarily, covered in §1.2's mention that Anthropic lets an org "set your own spend limit below your tier's cap" — returns HTTP 400 with error.type: invalid_request_error, not HTTP 429 at all, and a message beginning "You have reached your specified API usage limits"1. The distinction matters operationally: HTTP 400 conventionally means "this specific request is malformed and will never succeed as sent," which is the wrong signal for "your organization is out of self-imposed budget for the month," but Anthropic's documentation is explicit that this is the actual status code returned, and a client that treats every 4xx identically will misclassify a self-imposed budget cap as a request bug rather than a billing state. One further variant sits alongside these three: the Claude Code workspace's limit, when exceeded, can instead return a 429 that does carry a retry-after header even though it is enforcing what is functionally a spend-type limit1 — a reminder that the three-signature taxonomy below is what the general Messages API returns, not an inviolable law that every limit type on the platform obeys without exception.
error.type: rate_limit_error and HTTP 429, distinguished only by the presence of retry-after and the enforced_spend_limit_reached error code; a self-imposed spend limit uses a different status code and error type entirely1.| Case | HTTP status | error.type | retry-after | Distinguishing field | Correct client action |
|---|---|---|---|---|---|
| Ordinary rate limit | 429 | rate_limit_error | present | — | back off and retry |
| Org spend cap reached | 429 | rate_limit_error | absent | error_code: enforced_spend_limit_reached | stop retrying; wait for next billing month or request a higher tier |
| Self-imposed spend limit | 400 | invalid_request_error | absent | message text ("specified API usage limits") | raise or remove the self-imposed limit |
OpenAI's documentation makes the same rate-limit-vs-abuse distinction in spirit, though from the opposite direction: rather than a separate spend-cap error code, it warns that "unsuccessful requests contribute to your per-minute limit, so continuously resending a request won't work" and instructs clients to "treat this value as a minimum: wait at least that long and add a small random delay"2 before retrying — a statement this module returns to in §3.5, because it has direct consequences for retry design that go beyond simply reading a header correctly. OpenAI's documentation states that "OpenAI sets an approved monthly usage limit for each organization" tied to the tier ladder in §1.1, but — unlike Anthropic's explicit enforced_spend_limit_reached error code and no-retry-after signal — it does not state what error a caller actually receives on reaching that monthly cap. That is a second, narrower instance of §1.3's documentation gap: OpenAI's rate-limit page is precise about the six per-minute/per-day dimensions and silent about the error signature for the one limit that is closest in kind to Anthropic's spend cap.
3. Enforcing a limit whose true cost is known only after generation
Module 10 §4 established three rate-limiting algorithms — fixed window, sliding window, token bucket — and worked a derivation showing the identical 20-request burst produces three different accept counts (20, 10, and 12 of 20) depending on which algorithm measures it4. That derivation assumed every request costs the same one unit of "rate." An LLM API cannot make that assumption: the quantity that actually matters — tokens generated — is not known when the request arrives, only after the model finishes producing output, and a request that turns out to generate 4,000 output tokens looks, at admission time, identical to one that will generate 40. This section applies Module 10's three algorithms to that harder problem rather than re-deriving them.
3.1 Fixed window vs sliding window, applied to an API key's RPM limit
Cloudflare's AI Gateway — a product built specifically to sit in front of LLM API traffic rather than a generic web backend — documents exactly the boundary-straddling failure mode Module 10 derived, using the same style of example: with a 10-requests-per-10-minutes limit, "sending 10 requests at 12:09 and 10 requests at 12:11... all 20 requests would be successful" under fixed-window enforcement, because the first batch lands in the window ending at 12:10 and the second in the fresh window that opens immediately after4. Anchored to a sliding window instead, "there would be no more than x requests in the last ten minutes... they would fail in a sliding window strategy since there were more than ten requests in the last ten minutes"4 — the second batch is rejected outright because the trailing 10-minute interval already contains the first batch. The mechanism is identical to Module 10's; what is new here is only the object being counted: an API key's RPM allocation rather than a generic backend's admission counter, and Cloudflare's own AI Gateway product is the direct evidence that this exact boundary problem is something LLM-platform operators have had to document and solve, not a hypothetical.
3.2 Token bucket applied: continuous replenishment, and the sub-window smoothing caveat
Anthropic's Messages API is the token-bucket case Module 10 §4.3 introduced in general terms1, and its documentation adds a detail Module 10 did not need: the nominal bucket capacity is not always the effective burst capacity a caller can actually extract, because the platform can smooth enforcement to a finer sub-window than the limit's stated unit. Anthropic states this directly: "you might hit rate limits over shorter time intervals. For instance, a rate of 60 requests per minute (RPM) might be enforced as 1 request per second. Short bursts of requests can exceed the limit and trigger rate limit errors"1. Read literally against the pure token-bucket model — capacity 60, refill 1 token/second, starting full — a caller that has been idle should be able to spend all 60 tokens in the first second of a burst; Anthropic's own caveat says that in practice, enforcing "1 request per second" as the effective ceiling means a burst of, say, 20 requests inside one second can trigger a 429 even though the nominal 60-per-minute bucket has plenty of capacity left over the full minute. The nominal number on the rate-limit page (60 RPM) and the number that actually predicts whether a burst succeeds (roughly 1 per second, i.e. a much smaller effective bucket over a much shorter interval) are two different quantities, and only the documentation — not the stated limit — tells you which one governs a specific burst pattern.
This is the general lesson a token-bucket implementation can hide from a naive reading of its parameters: "continuously replenished... rather than reset at fixed intervals" describes the refill rule1, but says nothing about whether the operator additionally caps how much of that capacity can be spent inside a sub-interval shorter than the nominal window — a second bucket, effectively, nested inside the first, with its own tighter capacity. A caller tuning its request pacing against the published 60 RPM number alone, without accounting for this sub-window smoothing, will see intermittent 429s that a naive token-bucket model says should not happen.
3.3 Per-dimension limit sets: six axes vs three
Both platforms answer "how much can this key do per minute" with more than one number, because a single combined counter cannot distinguish a caller sending many small requests from one sending few enormous ones. OpenAI enforces six independent dimensions — RPM, RPD, TPM, TPD, IPM, and audio minutes per minute for streaming audio models — and states that whichever one is reached first triggers the 4292. Anthropic enforces three — RPM, ITPM, OTPM — and states the same "any one" logic: "if you exceed any of the rate limits you will get a 429 error describing which rate limit was exceeded"1. The difference is not that OpenAI's platform is more heavily used or Anthropic's less; it is that OpenAI's product surface spans more modalities requiring their own independent ceiling (images, audio) while Anthropic's is scoped to text-in/text-out on the Messages API, and — the detail worth dwelling on — Anthropic further splits tokens into input and output rather than combining them into one TPM figure the way a naive reading of "tokens per minute" would suggest.
That input/output split matters mechanically, not just cosmetically: input tokens are countable at request time (you can tokenize the prompt before sending it), while output tokens are not known until the model has finished generating them. Anthropic's documentation states this distinction as two different enforcement timings rather than one: ITPM "is estimated at the beginning of each request, and the estimate is adjusted during the request to reflect the actual number of input tokens used," while OTPM "is evaluated in real time as output tokens are produced, counting only the actual tokens generated" — and explicitly, "the max_tokens parameter does not factor into OTPM rate limit calculations, so there is no rate limit downside to setting a higher max_tokens value"1. A combined TPM dimension would have to either estimate the unknown output length pessimistically at admission (over-reserving capacity against requests that turn out short) or under-count it (letting a long generation blow past a limit that was fine at admission time); splitting into ITPM and OTPM sidesteps the estimation problem for outputs entirely by only ever counting what was actually produced, in real time, as it is produced.
| Dimension | OpenAI | Anthropic | Known at request time? |
|---|---|---|---|
| Requests / minute | RPM | RPM | yes |
| Requests / day | RPD | not a separate dimension | yes |
| Input tokens / minute | combined into TPM | ITPM (cache-excluded, most models) | yes (tokenizable) |
| Output tokens / minute | combined into TPM | OTPM (real-time, actual tokens only) | no — known only as generated |
| Tokens / day | TPD | not a separate dimension | — |
| Images / minute | IPM | not a separate dimension | yes |
| Audio minutes / minute | for streaming audio models | not a separate dimension | partially |
Anthropic's own published tables give a concrete sense of how far these three numbers move across tiers for one model family. Claude Sonnet 5's limits scale roughly 2x from Start to Build and roughly another 2x from Build to Scale on every dimension simultaneously1 — the tiers are not just a dollar-cap ladder (§1.2) but a matched rate-limit ladder rising alongside it.
| Tier | RPM | ITPM | OTPM |
|---|---|---|---|
| Start | 1,000 | 2,000,000 | 400,000 |
| Build | 5,000 | 5,000,000 | 1,000,000 |
| Scale | 10,000 | 10,000,000 | 2,000,000 |
Both platforms also carve out rate limits for specific resources that sit outside the six-vs-three framing entirely, because a resource that isn't a chat completion doesn't fit naturally into RPM/TPM-shaped counters. OpenAI meters vector-store ingestion separately at 300 requests per minute per vector store2, and its Batch API is not rate-limited by RPM/TPM at all but by a queue depth: "Batch API queue limits are calculated based on the total number of input tokens queued for a given model"2 — a caller can queue an enormous batch job without touching its interactive RPM/TPM budget, but the aggregate size of everything still waiting to run is its own separate ceiling. Anthropic carves out comparably specific limits for its Message Batches API (a combined RPM plus a cap on batch requests in the processing queue — 200,000 at Start, rising to 500,000 at Scale — plus a 100,000-request-per-batch ceiling at every tier), its Managed Agents endpoints (300 requests per minute to create an agent/session/environment, 1,200 per minute for read operations), and its Files API (a separate per-organization limit shared across upload, list, retrieve, download, and delete)1. The general lesson is that "six dimensions" and "three dimensions" describe only the interactive-completion surface; any resource whose cost shape doesn't match request-count-or-token-count gets its own bespoke limit instead of being forced into the existing six or three, and a platform's true rate-limit surface is the union of all of them, not just the headline table.
3.4 Cache-aware ITPM: a worked derivation of effective throughput
Anthropic excludes most models' cache_read_input_tokens from the ITPM count entirely: "only uncached input tokens count toward your ITPM rate limits"1 (prompt caching's own mechanics — breakpoints, TTLs, pricing multipliers — belong to Module 28; this section needs only the rate-limit consequence). That single exclusion turns a fixed ITPM number into a variable effective throughput ceiling that depends on how much of a workload's input is repeated context served from cache. Let \(L\) be the stated ITPM limit and \(h\) the fraction of input tokens served from cache (the cache hit rate). If a caller sends \(X\) total input tokens per minute, a fraction \(h X\) is cache-read (excluded) and the remaining \((1-h)X\) is uncached and counts toward the limit:
$$ (1-h)X \le L \quad\Rightarrow\quad X \le \frac{L}{1-h} $$
At \(h = 0\) (no caching), effective throughput equals the stated limit, \(X_{\max} = L\), exactly as a combined, cache-blind TPM dimension would behave. At \(h = 0.8\) and \(L = 2{,}000{,}000\), \(X_{\max} = 2{,}000{,}000 / 0.2 = 10{,}000{,}000\) — Anthropic's own worked example states this exact number: "with a 2,000,000 ITPM limit and an 80% cache hit rate, you could effectively process 10,000,000 total input tokens per minute"1, which is precisely the arithmetic above. The relationship is not linear in \(h\) — it diverges as \(h \to 1\): at a 95% cache hit rate the same 2,000,000 ITPM limit supports 40,000,000 total input tokens per minute, and at 99% it supports 200,000,000. A workload's effective rate-limit ceiling is therefore not a fixed platform number at all once caching is in play; it is a function of that workload's own cache discipline, which is the direct reason Anthropic's documentation recommends monitoring cache hit rate on the usage page specifically "to tune your caching strategy"1 rather than treating the published ITPM number as the ceiling to plan against.
Effective ITPM throughput under cache-aware rate limiting
Formula: \( X_{\max} = L / (1 - h) \), derived above. At the default 2,000,000 ITPM / 80% inputs, this reproduces Anthropic's own stated 10,000,000 tokens/minute example1.
3.5 Why a failed request still counts against the limit
A rate limiter that only decremented its counter on successful requests would be trivially exploitable: a caller denied for exceeding its limit could immediately retry, and because the denied attempt didn't count, the retry would face exactly the same limit state as the original request — an infinite-retry loophole around any cap. Both platforms close this by counting the attempt, not the outcome. OpenAI states it plainly: "unsuccessful requests contribute to your per-minute limit, so continuously resending a request won't work"2. The design consequence is that a client cannot treat a 429 as free to immediately retry — doing so both fails again and burns another unit of whatever budget is already exhausted, which is strictly worse than doing nothing. OpenAI's own guidance follows directly from this: "treat this value as a minimum: wait at least that long and add a small random delay" before retrying2 — the exponential-backoff-with-jitter pattern Module 11 covers in general terms for any retried operation, applied here to the specific case where the "operation" is itself the thing consuming the budget being retried against. The same logic underlies OpenAI's traffic-ramping guidance for a caller scaling up from a cold start: once traffic reaches 1,000,000 input TPM, increase it by no more than 50% every 15 minutes2 — a deliberately slow ramp precisely because a burst of requests that trips a limit is not merely rejected for free; every rejected request in that burst has already spent capacity that a well-paced ramp would not have wasted.
3.6 Response headers: how a client actually inspects its remaining budget
A well-behaved client should not have to guess where it stands relative to a limit by counting its own requests — both platforms return the live state on every response, not just on a 429. OpenAI's headers are scoped in a pair per dimension, one for the effective (often project-level) limit and one explicitly project-scoped: x-ratelimit-limit-requests, x-ratelimit-remaining-requests, and x-ratelimit-reset-requests, mirrored for tokens, plus a parallel x-ratelimit-limit-project-tokens set2. Anthropic's headers are scoped per rate-limit type rather than per request/token pair, with a separate triple (limit/remaining/reset) for requests, combined tokens, input tokens, and output tokens, plus a fifth set gated behind Priority Tier access1 — and Anthropic states explicitly which limit's numbers you are looking at when more than one could apply: "the anthropic-ratelimit-tokens-* headers display the values for the most restrictive limit currently in effect. For instance, if you have exceeded the Workspace per-minute token limit, the headers will contain the Workspace per-minute token rate limit values"1 — a direct answer to the §4 question of whether a request's throttling came from its workspace cap or the org ceiling above it, readable from a single response without querying a console. A separate anthropic-workspace-id header, already introduced in §1.2, resolves which workspace a given key's traffic counted against1. Reading these headers, rather than tracking client-side counters that can drift from the server's true state, is the mechanism both platforms actually expect a production client to use to stay under a limit rather than discover it via a 429.
| What it tells you | OpenAI header | Anthropic header |
|---|---|---|
| Request limit / remaining / reset | x-ratelimit-limit-requests / -remaining-requests / -reset-requests | anthropic-ratelimit-requests-limit / -remaining / -reset |
| Token limit / remaining / reset | x-ratelimit-limit-tokens / -remaining-tokens / -reset-tokens (combined) | separate input and output triples: anthropic-ratelimit-input-tokens-*, anthropic-ratelimit-output-tokens-* |
| Project/workspace-scoped variant | x-ratelimit-limit-project-tokens and remaining/reset equivalents | same header names reused; "most restrictive limit currently in effect" rule1 disambiguates org vs. workspace |
| Which workspace/project a request counted against | not exposed as a header | anthropic-workspace-id |
| Retry timing on an ordinary rate limit | Retry-After | retry-after (absent on spend-cap responses, §2.1) |
| Premium-tier-only variant | none documented | anthropic-priority-input-tokens-* / -output-tokens-* (Priority Tier only) |
4. Workspace/org limit hierarchy and over-ceiling allocation
§1.2 established that Anthropic lets an organization configure a workspace's limits below the org-wide ceiling. The documentation gives one concrete real number to anchor the pattern: "if your Organization's limit is 40,000 input tokens per minute and 8,000 output tokens per minute, you might limit one Workspace to 30,000 input tokens per minute. This protects other Workspaces from potential overuse and ensures a more equitable distribution of resources across your Organization"1. That single workspace, capped at 30,000 of the org's 40,000 ITPM, still leaves 10,000 ITPM of headroom the org has not pre-allocated to anyone. The interesting design question — and the one the manifest for this module asks directly — is what happens once a second and third workspace are also configured with their own caps, and those caps sum to more than the org's total.
4.1 Extending the example past the ceiling
Take Anthropic's real numbers as given (org ceiling 40,000 ITPM / 8,000 OTPM; Workspace A capped at 30,000 ITPM) and add two further workspaces, illustratively, to make the over-allocation concrete: Workspace B configured at 25,000 ITPM, and Workspace C configured at 20,000 ITPM. Summed, the three workspace-level caps total 75,000 ITPM — nearly double the organization's 40,000 ITPM ceiling. Anthropic's documentation states directly that this is not an error state and does not need to be prevented at configuration time: "organization-wide limits always apply, even if Workspace limits add up to more"1. The mechanism that makes this safe is headroom sharing, not reservation: a workspace's configured cap is a ceiling on how much of the shared org pool that one workspace may draw at once, not a guaranteed slice carved permanently out of the total. "The remaining unused tokens per minute (or more, if that Workspace doesn't use the limit) are then available for other Workspaces to use"1 — so if Workspace A, B, and C are all quiet simultaneously, none of them is blocked by the others' configured caps; if all three try to burst to their individual caps at the same instant, the organization-level enforcement is the one that actually gates total throughput to 40,000 ITPM, admitting requests from whichever workspaces arrive first (or by whatever internal tie-break the platform applies, which the documentation does not specify) until the shared ceiling is reached, independent of any single workspace's own configured number.
This is the same headroom-sharing shape Module 10 §3.1 described for Borg's best-effort priority tier, where every user's nominally unlimited low-priority quota is still bounded by oversubscribed shared capacity underneath it — except here the outer ceiling is a hard org-wide number rather than Borg's physical cluster capacity, and the workspace caps are a voluntary further restriction an organization applies to itself, not a market-priced purchase. The practical consequence for anyone configuring workspace limits is that summing configured workspace caps and comparing the total to the org ceiling is not a validation step the platform performs or needs to perform — over-provisioning workspace allocations is a deliberate, safe, and in fact ordinary way to set these numbers, because the org-level check is what actually bounds aggregate usage regardless of how generously any individual workspace was configured.
5. Zero trust and relationship-based authorization applied to an API platform
Everything above assumes a request's API key is genuinely the party it claims to be and asks only how much of the platform that party may use. Module 10 §5–6 built two mechanisms for the prior question — should this party reach this resource at all, and does it hold the specific permission it is claiming — using BeyondCorp's device/identity trust tiers and Zanzibar's relation-tuple authorization56. Neither OpenAI's nor Anthropic's rate-limit documentation states that either mechanism runs inside their platforms, and no source in this course's map claims it does; this section applies Module 10's general vocabulary to the org/workspace/key domain established in §1 rather than reporting an attested feature of either named platform.
5.1 What a bearer key lacks that BeyondCorp's model has
A typical inference-API request authenticates on exactly one fact: possession of a bearer token — the API key itself, sent in an HTTP header, checked against a database of valid keys and their associated org/workspace. BeyondCorp's model, applied to the same request, would ask a structurally richer question: not just "is this credential valid" but "what do we currently know about the device and context this credential is being used from, and does that context clear the minimum trust tier this specific resource requires"5. Under BeyondCorp's own framing, a credential presented from a device with a stale security patch, a flagged infection signal, or a known-stolen status would be downgraded or denied even if the credential itself is technically valid5 — a second, independent axis of trust layered on top of "is this the right secret."
A bearer-key-only model has no equivalent second axis: once a key is valid, it carries its full authority from any device, any network, any client software, indefinitely until revoked. This is a real, structural gap relative to BeyondCorp's design, not a minor stylistic difference — the entire point of BeyondCorp's Trust Inferer is that identity alone (a valid credential) is not sufficient for access, because a valid credential can be stolen, and continuously re-evaluated device/context state is what limits the blast radius of that theft5. An inference platform whose only gate is bearer-key possession inherits the older perimeter-style failure mode BeyondCorp was built to retire: a leaked key is exactly as powerful as the legitimate key, usable from anywhere, until someone notices and rotates it. Extending anything like a Trust Inferer to API traffic would mean scoring the calling context — client IP reputation, TLS client-certificate identity, anomalous usage-pattern signals — the same way BeyondCorp scores a corporate laptop's patch level, and gating a resource's minimum required tier against that score rather than against key validity alone. Whether either lab does anything resembling this internally is not stated in any source available to this module, which is why this paragraph is phrased as what the mechanism would add, not what either platform has built.
5.2 Zanzibar's tuple algebra applied to org/team/key permissions
Zanzibar's data model represents every permission and every group membership as one uniform relation tuple, \(\langle object \rangle \# \langle relation \rangle @ \langle user \rangle\), composable through usersets so that nested group membership resolves without ever materializing a flat per-user access-control list6. That algebra generalizes cleanly to the org/workspace/key hierarchy §1 established, even though no source states that either lab implements authorization this way. An illustrative (not attested) tuple set for the identity hierarchy in this module reads:
org:acme#admin@user:11
workspace:prod#owner@org:acme#admin
key:sk_live_abc#issued_by@workspace:prod#owner
key:sk_live_abc#can_call@model:sonnet-5
A check for "can user 11 call model sonnet-5 through key sk_live_abc" does not match any single tuple directly, in exactly the shape Module 10 §6.1 worked through for a document-viewer check6: it resolves by walking the chain — user 11 is an org:acme admin; org:acme admins are workspace:prod owners; workspace:prod owners are who key sk_live_abc was issued by; and that key is separately granted can_call on model:sonnet-5. Revoking access at any single link — demoting user 11 from admin, for instance — invalidates every check that depended on that link without touching the other tuples, which is the same audit and revocation property Module 10 attributes to Zanzibar's design generally6. The illustrative label matters here specifically because §1.3 already established that neither platform documents its authorization internals at this level; what is genuinely transferable from Module 10 is the algebra — relations compose, groups nest, and a single revoked tuple cuts every check that chained through it — not a claim about which frontier lab, if any, implements exactly this data model underneath its console's team and permission settings.
Where experts disagree
Two disagreements from Module 10 sharpen once the request being rate-limited is an LLM generation rather than a generic API call. The first — fixed window vs. sliding window — is the same trade-off Module 10 §4 already worked through in general terms, now anchored to a product built specifically for LLM traffic. The second is new to this domain entirely: an LLM request's true resource cost (output tokens generated) is not known at admission time, and the systems in this course's source map disagree about whether to estimate that cost pessimistically up front or admit optimistically and reconcile afterward.
Fixed window is acceptable for a coarse RPM ceiling
Implicit in Cloudflare's own presentation of fixed window as the plainer of its two documented AI Gateway modes4; consistent with Module 10's general assessment that fixed window survives mainly where O(1) simplicity matters more than boundary-burst risk.
A per-key or per-workspace RPM ceiling exists mainly to catch gross misuse — a runaway loop, a misconfigured retry storm — rather than to enforce a precise physical resource constraint. For that purpose, a counter that resets on the clock is cheap to implement across a fleet of gateway replicas and easy for an operator to reason about, and the up-to-2x boundary burst Cloudflare documents4 is a rare, bounded annoyance rather than a resource-safety failure.
- trivial to shard across gateway replicas without a shared timestamp log
- predictable, easy to explain to a customer disputing a 429
- a caller that understands the boundary can extract a sustained 2x burst deliberately4
- weak as the sole guard against a genuine resource-exhaustion attack on a scarce backend
Sliding window (or an equivalent continuously-evaluated bound) is necessary once tokens, not requests, are the scarce resource
Cloudflare's own sliding-window mode, offered to close exactly this gap4; reinforced by Anthropic's move away from fixed-interval reset toward continuous replenishment for its own production limiter1.
GPU capacity backing an LLM API is not elastic on the timescale of a rate-limit window the way a stateless web backend's CPU headroom might be; a 2x admitted burst at a boundary is not merely an inconvenience, it is GPU-seconds actually consumed that a continuously-evaluated bound would have refused. Once the thing being protected is a physically scarce, expensive backend rather than a cheap stateless handler, the boundary-burst gap is a resource-safety problem, not a cosmetic one.
- no boundary at which a caller can extract more than the stated rate4
- matches the actual failure mode being protected against (backend saturation, not just abuse-counting)
- exact sliding-window state is proportional to traffic, not O(1) per caller4
- Anthropic's own token bucket is not a sliding window either — it is a third option, discussed next, that gets most of the same benefit at O(1) state
Estimate pessimistically at admission, adjust during the request
Anthropic's Messages API: ITPM "is estimated at the beginning of each request, and the estimate is adjusted during the request to reflect the actual number of input tokens used," while OTPM counts only tokens actually generated in real time1.
Because the platform controls the model doing the generating, it can meter output as it is produced rather than guessing at admission — there is no window in which a request's true output cost is unknown to the enforcement layer, only a window in which it is not yet finished accumulating. Splitting ITPM (estimable, since the prompt is fully known before generation starts) from OTPM (only knowable in real time) lets each half of the cost be metered by the mechanism actually suited to it, rather than forcing one combined estimate to cover both.
- no separate reconciliation step after the fact — the limit's live state is always the true state
- cache-aware ITPM exclusion (§3.4) composes cleanly with this because uncached input tokens are exactly counted, not estimated
- only works when the enforcement layer is co-located with, or has real-time visibility into, the model doing the generating
- Anthropic's own sub-window smoothing caveat (§3.2) shows the nominal number still under-describes true burst behavior in practice
Admit optimistically against current state, reconcile after the fact
Envoy AI Gateway: "AI Gateway checks if processing this request would exceed the configured token limit. If the limit would be exceeded, the request is rejected... If within the limit, the request is processed and its token usage is counted towards the total"7.
A gateway sitting in front of a third-party model provider (rather than operating the model itself, as Anthropic does) cannot always get real-time token counts mid-generation from every backend it might route to — Envoy AI Gateway explicitly supports non-OpenAI-schema backends like AWS Bedrock through response transformers7 — so it checks the limit against the best information available at admission time and reconciles the running total only once the response actually returns.
- works uniformly across heterogeneous backends without needing mid-generation visibility into every provider's internals
- simpler mental model: one check at admission, one update on completion, no mid-flight adjustment
- a burst of concurrent requests can each pass the pre-check before any of their usage lands, over-admitting relative to the true limit — the same race-condition shape Module 10 §4.5 describes for any shared counter updated by more than one process
- the gap between admission and reconciliation is exactly proportional to how long a generation takes, so it is worst for the longest, most expensive requests
On fixed vs. sliding window, Module 10's general verdict carries over unchanged: sliding window is the only one of the two that actually enforces the stated rate at every instant, and it is worth its higher per-caller state whenever the resource behind the limit is expensive enough that a 2x boundary burst is a real cost rather than a rounding error — which a GPU-backed inference API almost always is. Token bucket, as Anthropic implements it, gets most of sliding window's smoothness at fixed-window's O(1) state cost, which is why it is the more common production choice for LLM APIs specifically rather than either of Module 10's two window strategies in pure form. On estimate-then-adjust vs. admit-then-reconcile, the right choice is determined by where the enforcement point sits relative to the model doing the generating: a lab that operates its own inference stack (Anthropic, OpenAI) has no excuse not to meter output in real time, so estimate-then-adjust dominates there and the residual risk is only the sub-window smoothing gap already covered in §3.2. A gateway that fronts arbitrary third-party backends (Envoy AI Gateway, and by extension any multi-provider router) genuinely cannot always get that real-time signal, so admit-then-reconcile is the pragmatic fallback — but it should be paired with a conservative per-request ceiling (a hard cap on max_tokens, enforced before admission) precisely because the over-admission window is proportional to generation length and otherwise unbounded. What would settle the second disagreement more precisely than either vendor's documentation currently does is a published number: how large is the actual over-admission rate Envoy-style optimistic admission produces under realistic concurrent bursty traffic, measured against Anthropic-style real-time metering on the identical workload. No source in this course's map publishes that comparison.
What is changing
Every specific number in this module is a live-documentation snapshot, not a fixed specification, and the surrounding structure is visibly still moving. Anthropic's Evaluation tier and acceleration-limit warning1 exist specifically because new-organization fraud patterns keep shifting, which means the thresholds that gate an org out of that tier are the kind of number likely to change without a version bump. Anthropic's fast-mode Priority Tier tokens are explicitly labeled a research preview with their own dedicated headers1 — a fourth rate-limit dimension layered on top of the three this module treats as the stable set, and one that may be renamed, merged into OTPM, or dropped before it leaves preview. The model-family bucketing itself is not static: Anthropic's own footnotes describe combined rate-limit buckets shared across specific model version ranges (for instance, several Opus point releases sharing one limit while the newest Opus generation gets its own)1, and that bucketing has visibly been reshuffled before and will be again as new model versions ship and old ones are retired. OpenAI's guidance to increase traffic by no more than 50% every 15 minutes once above 1,000,000 input TPM2 is an operational recommendation tied to current capacity-provisioning practice, not a protocol guarantee, and should be expected to move as the underlying serving capacity does. None of this affects the mechanisms this module actually teaches — spend limits vs. rate limits as distinct error signatures, cache-aware ITPM's \(L/(1-h)\) relationship, the org/workspace headroom-sharing model — which are architectural choices unlikely to be reversed even as the specific dollar figures and per-model numbers around them keep shifting.
References
- officialAnthropic. Rate limits (Claude API). platform.claude.com documentation, 2026. https://platform.claude.com/docs/en/api/rate-limits. Accessed 2026-09-17. Used for: organization/workspace hierarchy and response headers (§1.2), spend-limit vs. rate-limit mechanism and error signatures including the three-signature taxonomy (§2), token-bucket definition and sub-window smoothing caveat (§3.2), the three rate-limit dimensions and per-tier RPM/ITPM/OTPM tables for Claude Sonnet 5 (§3.3, Table 4), the Batches/Managed-Agents/Files API resource-specific carve-outs (§3.3), cache-aware ITPM exclusion and the 2M/80%/10M worked example (§3.4), the anthropic-ratelimit-* and anthropic-workspace-id response headers and the "most restrictive limit" disambiguation rule (§3.6, Table 5), the workspace-limit example (40,000/8,000/30,000) and the "org limit always applies" rule (§4.1), and the Evaluation tier, fast-mode Priority Tier headers, and combined-model-bucket footnotes ("What is changing").
- officialOpenAI. Rate limits (OpenAI API). developers.openai.com documentation, 2026. https://developers.openai.com/api/docs/guides/rate-limits. Accessed 2026-09-17. Used for: organization/project scope statement and project-scoped response headers (§1.1), the six rate-limit dimensions and cumulative-payment usage-tier table (§1.1, Table 3), the undocumented monthly-cap error signature (§2.1), the vector-store and Batch API queue-based limits (§3.3), the x-ratelimit-* response headers (§3.6, Table 5), and the unsuccessful-requests-still-count and traffic-ramping guidance (§3.5, "What is changing").
- officialBerriAI / LiteLLM contributors. Life of a Request (LiteLLM Proxy architecture). docs.litellm.ai live documentation, 2026. https://docs.litellm.ai/docs/proxy/architecture. Accessed 2026-09-17. Used for: the fully-documented virtual-key/user/team identity hierarchy contrasted against OpenAI's and Anthropic's undocumented internals (§1.3, Table 1).
- officialCloudflare. Rate limiting (Cloudflare AI Gateway). developers.cloudflare.com, 2026. https://developers.cloudflare.com/ai-gateway/features/rate-limiting/. Accessed 2026-09-17. Used for: the fixed-window vs. sliding-window 20-request boundary burst re-anchored to an API key's RPM limit (§3.1, Figure 3) and the fixed-vs-sliding disagreement position ("Where experts disagree").
- paperOsborn, Barclay, Justin McWilliams, Betsy Beyer, and Max Saltonstall. BeyondCorp: Design to Deployment at Google. USENIX ;login:, vol. 41, no. 1, Spring 2016, pp. 28-33. https://research.google.com/pubs/archive/44860.pdf. Accessed 2026-09-17. Used for: the device/identity trust-tier mechanism applied illustratively to bearer-key-only API authentication (§5.1).
- paperPang, Ruoming et al. Zanzibar: Google's Consistent, Global Authorization System. USENIX ATC '19. https://www.usenix.org/system/files/atc19-pang.pdf. Accessed 2026-09-17. Used for: the relation-tuple algebra applied illustratively to an org/workspace/key permission chain (§5.2).
- officialEnvoy AI Gateway project. Usage-based Rate Limiting. aigateway.envoyproxy.io v0.1 documentation, 2026. https://aigateway.envoyproxy.io/docs/0.1/capabilities/usage-based-ratelimiting/. Accessed 2026-09-17. Used for: the admit-optimistically-then-reconcile enforcement design contrasted with Anthropic's estimate-then-adjust approach ("Where experts disagree").