System Design · Module 10

Services, APIs, and the Trust Boundary: Microservices, Service Mesh, Backpressure, Rate Limiting, and Zero Trust

Every case study in Part B runs as many independently-deployed services instead of one program, and every one of those services has to decide what to do when a caller sends more traffic than it can handle and whether to trust the caller at all. This module gives you the vocabulary for both problems — service boundaries and RPC, backpressure and rate limiting, zero trust and relationship-based authorization — grounded in the systems that actually named these mechanisms: Airbnb's SOA migration, Vitess, Borg, Cloudflare and Anthropic's rate limiters, BeyondCorp, and Zanzibar. After this module you can read any of those systems' request path and say precisely what happens when it is overloaded or asked for something it doesn't own.

Course 40 min read 8 sources As of 2026-09-17 Prerequisite: Module 1, Module 6
One deployable unit versus many, and what crossing that boundary costs Monolith Search Booking Payments Messaging in-process function calls one Rails codebase, one datastore one deploy pipeline — one blast radius Service-oriented Search svc Booking svc Payments svc Messaging svc Thrift-over-HTTP RPC own datastore per service (typical) deploy A deploy B deploy C each service ships on its own schedule migrate
Figure 1. Airbnb's own framing of its move off a single Rails monolith: independently-deployed services replace in-process function calls with Thrift-over-HTTP RPC calls, trading one shared blast radius for N independent ones1.

1. From one deployable unit to many

Everything in Module 1 through Module 9 — the request path, the caches, the replicated and partitioned stores, the consensus underneath them, the file layer, the message log, the batch and stream frameworks — assumed a boundary already existed between "this machine" and "that machine." This module is about a boundary one level up: the one between "this program" and "that program," drawn deliberately by the people who own them. A monolith puts every capability of a product — search, booking, payments, messaging — inside one deployable artifact, talking to itself through in-process function calls. A service-oriented architecture (SOA) draws lines inside that artifact and gives each piece its own process, its own deploy pipeline, and a network call where a function call used to be.

Airbnb's own account of this move is the clearest primary source available for what the trade actually costs, because it names the challenge without romanticizing it: moving from a monolithic Rails service toward an SOA while simultaneously shipping new products and features is, in the company's own words, "not without its challenges"1. That sentence is doing real work — it is a company that made the move telling you the move was not free, written after the fact rather than as a pitch for making it.

1.1 What the migration actually costs

The cost is not abstract. Two things change the moment a function call becomes a network call:

  • Failure becomes partial. In a monolith, a bug in the payments module can still crash the whole process, but a slow payments module cannot silently degrade only payments while search keeps working at full speed — everything shares one thread pool, one garbage collector, one deploy. In an SOA, the payments service can be slow, wedged, or down while search stays fast, but now every caller of payments needs an opinion about what to do when it doesn't answer (a problem Module 11 treats in full, with circuit breakers and bulkheads; this module only needs you to see that the problem exists once you cross a network boundary).
  • Coordination moves from the compiler to the wire. A monolith's internal calls are type-checked and versioned by a single build. Across a service boundary, the caller and callee ship independently, so the interface between them — the schema of the request and response — becomes a contract that has to survive both sides changing without the other one's knowledge.

What the migration buys back is exactly the thing it costs: independent deployability. A team can ship the messaging service ten times a day without coordinating a release train with the team that owns payments, and a spike in messaging traffic no longer risks an out-of-memory crash in the process that also handles bookings. Airbnb frames this trade explicitly as the reason it built new service infrastructure at all, even while under pressure to ship product features on the old monolith in parallel1.

A cost that does not show up in either bullet above, but that every team crossing this boundary discovers within the first quarter, is operational multiplication: one monolith has one process to restart, one set of logs to read, one dashboard to watch, and one on-call rotation; N services have N of each, and the question "why is checkout slow right now" now requires knowing which of N services is the one actually responsible before you can even open the right dashboard. Airbnb's own account is candid that the SOA migration was a genuine engineering undertaking layered on top of ordinary product work, not a side effect that happened for free alongside it1 — and the specific machinery for answering "which service, and why" once you have many of them (metrics/logs/traces, SLOs, and the circuit-breaker/bulkhead patterns that contain a slow service's blast radius) is exactly what Module 11 builds next, on top of the service boundaries this section establishes.

1.2 Designing a service IDL instead of adopting one

The interesting engineering decision in Airbnb's account is not "we split the monolith" — it is what they chose to build the RPC layer out of. By 2017, mature RPC frameworks already existed (gRPC, Thrift's own native transport, Finagle). Airbnb instead built a customized Thrift-based interface-definition system it calls the Service IDL, and ran it over HTTP rather than switching the company's existing HTTP-based service stack to something else, because replacing that stack outright "means a big infrastructure overhaul that would require a large amount of engineering resources and cause major disruptions to productivity"1. Thrift's IDL gives them strongly-typed request/response schemas and code generation across languages — the same benefit a shared codebase's compiler gave for free inside the monolith — while HTTP as the wire protocol lets every existing piece of infrastructure that already understood HTTP (load balancers, proxies, logging, the reverse-proxy layer from Module 1) keep working unmodified.

This is the general shape of the "build vs. adopt" decision every SOA migration faces for its inter-service contract layer: adopt an existing RPC framework wholesale and inherit its transport, or keep the transport you already operate at scale and add just the typed-schema layer on top of it. Airbnb picked the second option specifically because the first one's blast radius — replacing a battle-tested HTTP stack under live traffic — was larger than the problem they were trying to solve. The tooling built around the IDL mattered as much as the IDL itself: a "make-me-a-service" scaffolding tool took several new Java services from inception to handling production traffic in three weeks, an estimated saving of two to three weeks of engineering time per service compared to Airbnb's prior process1. That number is a direct measure of how much of the "cost" side of the monolith-to-SOA trade is really a tooling problem: once bootstrapping a new service is a solved, scripted path, the marginal cost of adding the Nth service drops close to the cost of writing its business logic.

Table 1. What changes when a capability moves from an in-process module to an independently-deployed service, based on Airbnb's account of its own migration1.
DimensionMonolithService-oriented
Call between modulesin-process function call, compiler-checkednetwork RPC (Thrift-over-HTTP at Airbnb), schema-checked at the IDL layer
Deploy unit / blast radiusone artifact; any change risks every capabilityone artifact per service; a bad deploy is contained to that service's callers
Release cadenceshared release train across all teamsindependent per service, bounded by how disciplined the interface contract is
New-capability costadd a module to the existing buildbootstrap a new deployable unit — Airbnb's scaffolding tool cut this to about 3 weeks end to end
Failure modeall-or-nothing crash of one shared processpartial failure: one service degrades while its callers must decide how to react

There is a reason the interface layer, not the network transport, is where Airbnb chose to invest custom engineering effort: once two teams deploy independently, the interface between their services is the only thing standing in for the compile-time contract a monolith gave them for free. A monolith's caller and callee are recompiled together, so a signature change that would break a caller is caught before either side ships. Across a service boundary, the caller's next deploy and the callee's next deploy are decoupled in time — the callee's team can add a field, but removing or renaming one can break every caller that has not yet updated, silently, in production, with no compiler in the loop to catch it. A typed IDL like Airbnb's Service IDL does not eliminate this problem, but it makes the contract explicit and machine-checkable at the boundary instead of implicit and checked only by whichever caller happens to notice a broken response first1. This is why "just POST some JSON" is a smaller step than it looks for a company crossing this boundary for the first time: JSON-over-HTTP has no schema at all unless a team layers one on top, so the very problem the migration is supposed to solve — decoupling teams without silently breaking each other — has to be solved again, by hand, at every service pair, unless the interface layer enforces it centrally.

2. A proxy that already looks like a mesh: Vitess's VTGate

Once a system is a graph of services instead of one process, something has to sit between a caller and the right backend instance and make the topology of the backend invisible to the caller. A general-purpose service mesh does this for arbitrary RPC calls between arbitrary services. Vitess does the identical job for one very specific, very common case — a caller that wants to talk to "the database" when the database is actually many shards of MySQL — and its documentation describes the mechanism in enough concrete detail to serve as a fully worked example of the pattern before the next module needs the general version.

Vitess grew out of YouTube's need to keep scaling a sharded MySQL fleet without every application team having to know which shard a given row lives on, and it served all of YouTube's database traffic for over five years2 (Module 13 covers YouTube's use of Vitess as a case study in its own right; this module only needs Vitess's mechanism). The architecture has two moving parts. VTGate is a proxy that speaks the MySQL wire protocol to the application — so from the caller's point of view it looks exactly like talking to one big MySQL server — and it routes each incoming query to the shard (or shards) that actually own the relevant rows. VTTablet is an agent that runs alongside every real MySQL instance and manages that instance's queries, its connection pool, and its replication state2. A separate topology service — Vitess supports etcd, ZooKeeper, or Consul (all three coordination kernels from Module 6) — holds the authoritative map of which shard covers which key range and which VTTablet fronts which MySQL instance, and every VTGate consults that map to route2.

Client to VTGate to shard-routed VTTablet to MySQL, with the topology service feeding routing state Application(MySQL client) MySQL protocol VTGateroutes by shard key Topology service etcd / ZooKeeper / Consul cluster state VTTabletshard 0 VTTabletshard 1 VTTabletshard 2 MySQL MySQL MySQL every raw MySQL connection costs 256 KB – 3 MB of server memory; VTGate pools connections instead of the app holding thousands open
Figure 2. Vitess's client-facing proxy (VTGate) routes each query to the VTTablet agent that fronts the owning MySQL shard, consulting a topology service for cluster state — the same "one logical endpoint, many physical backends" shape a service-mesh sidecar provides for general RPC2.

2.1 Sidecars, service mesh, and mTLS as the generalized version of the same idea

VTGate solves one problem — routing MySQL protocol traffic to shards — with a purpose-built proxy the application talks to directly. A general service mesh generalizes the same idea to arbitrary service-to-service RPC and changes where the proxy sits: instead of the application dialing a separate proxy process, a mesh runs a small proxy — a sidecar — next to every single service instance, and transparently redirects that instance's outbound and inbound traffic through it. Istio, whose data-plane sidecar is an Envoy proxy instance per workload, documents the mechanism plainly: "Istio re-routes the outbound traffic from a client to the client's local sidecar Envoy," which then negotiates directly with the destination's local sidecar; "the server side Envoy authorizes the request. If authorized, it forwards the traffic to the backend service through local TCP connections"3. Neither the client's nor the server's application code has to know the mesh exists — the redirection happens at the network layer, which is the whole point: it turns "every service needs to implement retries, timeouts, encryption, and authorization" into "the sidecar does that once, uniformly, for every service."

The other piece a mesh sidecar typically adds beyond routing is mutual TLS (mTLS): both the calling side's sidecar and the receiving side's sidecar present certificates and authenticate each other before any request bytes flow, so every hop is encrypted and both ends can cryptographically verify which service — not just which IP address — they are talking to. Istio issues each workload "a strong identity representing its role" and automates the certificate lifecycle so the encryption is not something a service author has to remember to configure: "Istio agents, running alongside each Envoy proxy, work together with istiod to automate key and certificate rotation at scale"3. Compare that to VTGate's world: Vitess doesn't need per-hop mTLS between every application and every shard because it interposes one proxy at the client edge rather than one sidecar per backend instance — a reasonable simplification when the "many backends" are all MySQL shards under one team's control rather than services owned by dozens of independent teams that don't trust each other by default. That difference in threat model — one owner's shards vs. many teams' services — is exactly why Zero trust (§5) and relationship-based authorization (§6) become necessary machinery once an organization is large enough that "which team owns the caller" stops being something you can assume from the network it's on.

The sidecar shape itself is a deliberate rejection of the alternative: a shared client library, linked into every service, that implements retries, timeouts, encryption, and load balancing in-process. Envoy — the proxy Istio uses as its per-workload sidecar — states its own founding rationale as making "the network... transparent to applications," and the practical case for doing that out-of-process rather than in a library is twofold: a library has to be reimplemented, or at least rebuilt and relinked, in every language a service fleet uses, while a sidecar proxy is "language agnostic" and can sit next to a Java service exactly as it sits next to one written in Go or Python; and upgrading a library's behavior across hundreds of services means hundreds of independent rebuild-and-redeploy cycles, while upgrading a sidecar is one rollout of one binary that every service picks up without a code change3. That second property — network-layer upgrades without touching application code — is the direct mesh-scale analogue of what Airbnb's scaffolding tooling did for service creation in §1.2: it turns a cross-cutting concern that used to require coordinating N teams into a change made once, centrally, by the team that owns the mesh.

The control plane matters as much as the sidecar data plane, and it is easy to conflate the two. Envoy sidecars are the data plane — the thing that actually touches request bytes — but they need to be told which certificates to present, which upstream services exist, and which routing and retry policy applies to a given request. In Istio, a separate control-plane process, istiod, is what "work[s] together with" every Envoy sidecar "to automate key and certificate rotation at scale"3 and pushes configuration down to the data plane. VTGate collapses this distinction because it is simultaneously the routing decision-maker and the thing the traffic flows through; a general mesh separates the two because the number of sidecars (one per service instance, potentially thousands) makes a centralized control plane the only tractable way to keep all of their configuration consistent.

3. Backpressure: bounded queues, load shedding, and admission by quota

A service that accepts more concurrent work than it has resources for does not fail gracefully by default — it fails by getting slower for every request in flight, including the ones that arrived first and would otherwise have succeeded quickly. Backpressure is the general name for mechanisms that make a system say "no" to some work explicitly, instead of accepting all of it and degrading silently. Two of its most direct forms are a bounded queue (accept work only up to a fixed queue depth; once full, reject or block new arrivals rather than growing the queue without limit) and load shedding (deliberately drop or refuse a fraction of incoming work, usually the lowest-priority fraction, once a resource is saturated). Both trade a controlled, visible failure now for the alternative — an uncontrolled, often invisible failure (unbounded memory growth, thrashing, eventual crash) later.

3.1 Borg's admission-control-by-quota: rejecting at the door, not the back of the queue

Google's Borg — the cluster manager that schedules both Borg's long-running "prod" services and its batch jobs across cells that run up to tens of thousands of machines, with a median cell size around 10,000 machines4 — makes a specific, load-bearing design choice about where backpressure happens: quota is checked at admission time, not at scheduling time. "Quota-checking is part of admission control, not scheduling: jobs with insufficient quota are immediately rejected upon submission"4. That single sentence separates two questions that are easy to conflate: "is this request allowed to exist at all, given who's asking and how much they've been allocated" (admission control) versus "given that it's allowed to exist, when and where does it actually run" (scheduling). Borg answers the first question the instant a job is submitted, before the scheduler ever spends a cycle deciding placement.

Why this ordering matters: if quota were checked only at scheduling time, an over-quota job would sit in a pending queue — consuming memory in the scheduler's data structures, showing up in metrics as "queued" rather than "rejected," and potentially delaying the scheduler's evaluation of jobs that do have quota — before eventually being told no. Checking at admission instead means the rejection is immediate, cheap, and legible: the caller gets a clear signal the moment they ask, rather than an ambiguous delay followed by a late failure. This is precisely the same shape of decision a rate limiter makes (§4) and the same shape BeyondCorp's gateways make (§5): reject as early in the pipeline as the information needed to reject is available, rather than doing expensive work first and discovering the rejection reason afterward. Borg's representative cell data makes the stakes of getting this wrong concrete: prod (latency-sensitive) jobs are allocated about 70% of a cell's CPU and represent about 60% of CPU usage, while consuming about 55% of memory allocated against about 85% of memory usage4 — a cell running that close to its memory ceiling has essentially no headroom to absorb work that admission control should have rejected outright. The general pattern — bound the queue, reject early, make the rejection explicit and cheap rather than late and expensive — is the same pattern Module 11 revisits when it covers circuit breakers and bulkheads as the next layer of defense once a dependency, rather than a whole cell, is the thing running out of headroom.

Quota is not Borg's only backpressure lever, and the second one shows up only after admission, once the scheduler has already accepted work it now cannot fully honor. Every Borg job carries a priority — "a small positive integer" — and Borg groups priorities into non-overlapping bands, in decreasing order: monitoring, production, batch, and best-effort (also called testing or free)4. "A high-priority task can obtain resources at the expense of a lower-priority one, even if that involves preempting (killing) the latter"4 — this is backpressure applied reactively, at the machine, rather than proactively, at the door: instead of refusing to admit a job because a quota check failed, Borg admits it, schedules it, and later evicts it if a higher-priority job needs the same resources. Left unconstrained this could cascade — a high-priority task bumping a slightly-lower one, which bumps another slightly-lower one, and so on — so Borg specifically disallows tasks within the production band from preempting one another, cutting off the cascade at the one boundary where it would otherwise be most damaging4.

Quota and priority are coupled through price and oversubscription in a way that gives the whole mechanism a market-like shape: "higher-priority quota costs more than quota at lower priority," and production-priority quota is capped at the cell's actual physical resources, so a job that fits within its purchased production quota can expect to run4. Below production, Borg deliberately oversells: "every user has infinite quota at priority zero," but that quota is frequently impossible to exercise in full because the underlying resources are oversubscribed across every user with access to that tier — "a low-priority job may be admitted but remain pending (unscheduled) due to insufficient resources"4. That sentence is the precise mechanical distinction between admission and scheduling this section opened with: a best-effort job can pass the admission-control quota check (trivially, since that tier's quota is unlimited) and still never actually run, because passing admission only means "you are allowed to wait in line," not "capacity exists for you now." Preemption and oversubscribed low-priority quota are two sides of the same backpressure design: push the pain of contention onto the cheapest, most tolerant tier first, and reserve guaranteed capacity for the tier that paid for it.

Table 2. Borg's priority bands, in decreasing priority order, and how each interacts with quota and preemption4.
BandQuota behaviorPreemption behavior
Monitoringpurchased, priced highestcan preempt any lower band
Production (prod)capped at actual cell resources — what you buy is what you can runcan preempt batch/best-effort; cannot preempt other production tasks, to block cascades
Batchpurchased at a lower price than productioncan be preempted by monitoring or production
Best-effort (testing/free)unlimited ("infinite") quota at priority zero, but resources are oversubscribedadmitted freely, but frequently stays pending — admission is not a promise of scheduling

4. Rate limiting: three ways to bound a rate

Admission control by quota answers "how much total capacity does this caller get." Rate limiting answers the narrower, time-shaped version of the same question: "how many requests may this caller make per unit time." Three algorithms dominate production rate limiters, and they differ in exactly one place — how they define "the window" a request is measured against — which turns out to produce meaningfully different accept/reject decisions on identical traffic.

4.1 Fixed window

A fixed-window limiter divides time into non-overlapping intervals — say, ten-minute blocks aligned to the clock — and counts requests within the current block against a limit; the counter resets to zero the instant the next block begins. Cloudflare's own documentation states the mechanism and its consequence directly with a 10-request-per-10-minute example: because "the window is based on time," a caller who sends 10 requests at 12:09 and then 10 more at 12:11 sees all 20 requests succeed, even though the nominal limit is 10 per 10 minutes, because the first 10 land in the window that resets at 12:10 and the second 10 land in the fresh window that just opened5. The algorithm is cheap — one integer counter per caller, reset on a timer or lazily on the next request past the boundary — which is exactly why it remains common despite this failure mode.

4.2 Sliding window

A sliding-window limiter evaluates the same nominal limit against a continuously moving trailing interval instead of a fixed clock-aligned block: at any instant, count requests in the last N minutes, not "requests since the top of the current block." Run the identical 20-request scenario against a sliding 10-per-10-minute limit and the outcome inverts: "there would be no more than x requests in the last ten minutes... they would fail in a sliding window strategy since there were more than ten requests in the last ten minutes"5. The 10 requests at 12:11 are evaluated against the trailing window [12:01, 12:11], which already contains the 10 requests sent at 12:09 — so the limit is already exhausted before any of the second batch is admitted, and they are rejected. Whether a limiter enforcing this uses an exact log of every timestamp (correct but memory-proportional-to-traffic) or an approximation that blends two adjacent fixed-window counters (cheaper, slightly imprecise at the boundary) is an implementation detail; the defining property is that the window itself moves with the clock rather than resetting on a fixed schedule. A caller that exceeds the configured rate under either algorithm receives an HTTP 429 Too Many Requests, and the request is not processed5 — the difference between the two algorithms is entirely in which requests trigger that 429, not in what a rejection looks like.

4.3 Token bucket

A token bucket takes a third approach that avoids the clock-boundary question altogether: instead of counting requests against a window, it maintains a bucket of tokens with a fixed capacity, drains one token per accepted request, and refills continuously at a fixed rate rather than resetting at an interval boundary. Anthropic's own rate-limit documentation states this framing precisely: "the API uses the token bucket algorithm to do rate limiting. This means that your capacity is continuously replenished up to your maximum limit, rather than being reset at fixed intervals"6. (Anthropic's specific per-dimension limits — requests per minute, input and output tokens per minute, and how cached tokens are excluded from the input-token count — are a platform-specific application of this general mechanism and belong to Module 22, which builds directly on this section rather than re-deriving it.) A caller that has been idle accrues tokens up to the bucket's capacity and can spend them in a single burst; a caller that has been running near its limit has few tokens available and gets throttled smoothly as it drains the bucket faster than it refills. There is no "reset instant" at which the entire budget snaps back to full, which is exactly the property that eliminates the fixed-window boundary-doubling problem — but it introduces a design choice fixed and sliding windows don't have: how large a burst (the bucket's capacity) to allow relative to the steady-state rate (the refill rate), since those are now two independently tunable numbers instead of one.

4.4 Worked derivation: the same burst under all three

Reuse Cloudflare's exact scenario — a nominal limit of 10 requests per 10 minutes, a burst of 10 requests at minute 9 and another 10 at minute 11 — and run it through a token bucket parameterized to match the same steady-state rate: capacity \(C = 10\) tokens, refill rate \(r = 1\) token/minute (10 tokens per 10 minutes, the same nominal rate as the window examples), starting full. This parameterization is illustrative — chosen here to make the three algorithms comparable on identical inputs, not a specific vendor's published configuration.

  • Fixed window (two 10-minute blocks, e.g. [12:00,12:10) and [12:10,12:20)): 10 requests at minute 9 land in the first block (count 10, at the limit, all accepted); 10 requests at minute 11 land in the fresh second block (count 10, all accepted). Result: 20 of 20 accepted5.
  • Sliding window (trailing 10-minute interval, evaluated continuously): the 10 requests at minute 9 are the first traffic seen and are accepted (trailing count goes from 0 to 10). At minute 11, the trailing window [1,11] still contains all 10 requests from minute 9, so the count is already at the limit before any new request is evaluated; every one of the 10 requests at minute 11 is rejected. Result: 10 of 20 accepted5.
  • Token bucket (capacity 10, refill 1/min, starts full): at minute 9 the bucket is still full (idle time before minute 9 only tops it back up to its 10-token cap, it can't overflow), so all 10 requests drain it from 10 tokens to 0. Between minute 9 and minute 11, two minutes elapse, refilling \(2 \times 1 = 2\) tokens. At minute 11 the bucket holds 2 tokens: 2 of the 10 new requests succeed and drain the bucket to 0, the remaining 8 are rejected. Result: 12 of 20 accepted.

Three algorithms, three different accept counts — 20, 10, and 12 — on exactly the same input traffic and the same nominal 10-per-10-minute rate. That spread is the entire practical stakes of the choice: it is not a stylistic preference, it is a factor-of-two difference in how much burst a caller can extract from the system at a window or refill boundary.

The same 20-request burst evaluated by fixed-window, sliding-window, and token-bucket rate limiting Limit: 10 requests / 10 minutes — burst of 10 @ minute 9, 10 @ minute 11 Fixed window t=10 (reset) 10 req @9 10 req @11 20/20 accepted Sliding window trailing 10-min window 10 req @9 — OK 10 req @11 — reject 10/20 accepted Token bucket (cap 10, refill 1/min) 0 10 full → drained to 0 @9 refilled to 2 @11 10 req @9 — OK 2 OK, 8 reject @11 12/20 accepted minute 9 → minute 11, same traffic, same nominal rate, three different outcomes
Figure 3. Cloudflare's fixed-vs-sliding-window burst example5, extended with a token bucket parameterized to the same nominal rate6 — worked in §4.4. Fixed window accepts the full 20-request burst; sliding window rejects the entire second batch; token bucket admits a partial second batch sized by how much has refilled.
Table 3. Rate-limiting algorithms compared on the mechanism that produces their different burst behavior56.
AlgorithmState keptBoundary burstSmoothness
Fixed windowone counter, resets on a clock boundaryup to 2x nominal rate at a boundarynone — a step function that snaps to zero at reset
Sliding windowtimestamp log or blended adjacent-counter approximationnone — trailing window always reflects true ratecontinuous, but requires more state per caller
Token bucketone counter (tokens) + one continuous refill rulebounded by bucket capacity, independent of clock alignmentcontinuous; burst and steady-state rate are independently tunable

4.5 Where the counter actually lives

Every description above quietly assumed a single counter per caller, updated atomically on every request. That assumption is easy to satisfy on one machine and hard to satisfy on the fleet of gateway replicas a real API-facing service runs behind a load balancer (Module 1): if caller X's requests can land on any of ten gateway replicas, each replica cannot keep its own local counter without the limiter becoming ten independent limiters, each enforcing a tenth of the intended rate against whichever fraction of X's traffic it happens to see. The rate limiter's counter has to be shared state, updated consistently across replicas — which is precisely the replication and coordination problem Modules 3 and 6 already built the vocabulary for. A common resolution is to push the counter into a fast shared store (an in-memory key-value store reachable by every gateway replica) and accept that the update to it is a small, focused instance of exactly the same read-modify-write race every replicated counter faces: two replicas can both read "9 of 10 used" at the same instant, both decide there is room for one more request, and both admit — an over-admission by exactly the margin of the race window. Making that race window small (a single fast round trip to one shared store) rather than eliminating it outright is usually the pragmatic choice, because a limiter that is occasionally off by one or two requests under contention is a far smaller problem than one that adds the latency of a consensus round to every request it gates. The general lesson carries forward past rate limiting specifically: any backpressure mechanism enforced by more than one process needs an explicit answer to "where does the shared counter live and how consistently is it updated," and that answer is a smaller-scale replay of the same trade-offs Module 5's PACELC framing makes for data storage generally.

5. Zero trust: gating on device and identity state

Everything so far in this module assumes the caller is already a known, legitimate party and asks only "how much of the system may this party use, and how fast." Zero trust asks the prior question: "should this party be allowed to reach this resource at all, right now, given what we know about the device and the user making the request" — and it answers that question without reference to where the request came from on the network. Google's BeyondCorp initiative is the origin of this pattern in the form it is deployed today, and its own framing states the departure from the older model directly: "BeyondCorp doesn't gate access to services and tools based on a user's physical location or the originating network; instead, access policies are based on information about a device, its state, and its associated user. BeyondCorp considers both internal networks and external networks to be completely untrusted"7. The old model — a corporate VPN or firewall perimeter, inside which everything is trusted and outside which everything is not — collapses the moment employees work from arbitrary networks and devices; BeyondCorp replaces "which network is this on" with "what do we currently know about this specific device and user."

5.1 The Trust Inferer and tiered access

BeyondCorp's architecture has six named components, and the one that does the actual trust decision is the Trust Inferer: "a system that continuously analyzes and annotates device state. The system sets the maximum trust tier accessible by the device and assigns the VLAN to be used by the device on the corporate network. These data are recorded in the Device Inventory Service. Reevaluations are triggered either by state changes or by a failure to receive updates from a device"7. Every resource — an internal wiki, a financial database, a code repository — is separately tagged with the minimum trust tier required to reach it, and "in order to access a given resource, a device's trust tier assignment must be equal to or greater than the resource's minimum trust tier requirement"7. The tiers are not a fixed universal ladder with official names in the paper, but the examples it gives make the shape concrete: a device that is missing some noncritical operating-system patches gets downgraded "to an intermediate tier, allowing access to some business applications but denying access to others"; a device missing a critical security patch or reporting an antivirus infection "may only be allowed to contact remediation services"; and "a known lost or stolen device can be denied access to all corporate resources"7. That is a four-rung ladder in practice — full access, intermediate/limited access, remediation-only access, and denied — even though BeyondCorp's own text treats "tiers" as a general concept rather than enumerating a fixed, named list.

The Trust Inferer's decision only becomes an actual access-control action at a Gateway — "resources are accessed via Gateways, such as SSH servers, Web proxies, or 802.1x-enabled networks. Gateways perform authorization actions, such as enforcing a minimum trust tier or assigning a VLAN"7 — which is architecturally the same role a fixed-window or token-bucket limiter plays in §4: a checkpoint that every request must pass through, making a binary allow/deny decision from a piece of continuously-updated state, before any real work happens on the far side. The scale of that continuously-updated state is not small: Google's Device Inventory Service — the pipeline that feeds the Trust Inferer — had, at the time of the paper, ingested billions of deltas from more than 15 data sources at a typical rate of about three million per day, totaling over 80 terabytes7.

Device signals feeding a Trust Inferer that sets a device's maximum trust tier, checked at a gateway against a resource's minimum tier Device signals OS patch level certificate identity infection signal lost / stolen flag Trust Inferercontinuousreevaluation Trust tier assigned Denied Remediation only Intermediate Full / privileged Gatewaytier >= resourcemin tier? Device Inventory Service 15+ sources, ~3M deltas/day, 80+ TB7
Figure 4. BeyondCorp's Trust Inferer continuously scores device state and sets a maximum reachable trust tier; a Gateway enforces that a device's tier meets a resource's minimum before granting access7.
Trust Inferer
continuously analyzes device state, sets the max trust tier and VLAN7
Device Inventory Service
continuously collects/publishes device state changes7
Access Control Engine
centralized service each gateway calls for a binary allow/deny decision7
Access Policy
programmatic mapping of resources to required trust tiers7
Gateways
SSH servers, web proxies, 802.1x networks — enforce the tier check7
Resources
anything gated: wikis, databases, code repos, lab networks7

5.2 Deploying zero trust without breaking every existing workflow

A zero-trust redesign is not something an organization can flip on for every application at once, because the old perimeter model is what every existing workflow was built against, and Google's own account of the rollout treats that migration problem as seriously as the architecture itself. The initial phase deliberately did not start from the target end state: Google "integrated a subset of gateways with an interim meta-inventory service" fed mostly by prescribed (manually-maintained) data, and applied "an access policy that mirrored Google's existing IP-based perimeter security model" to the new gateways — leaving access unchanged for devices already on privileged networks while the new system was brought online underneath them, so the components could be validated under real traffic before a single existing workflow was asked to change7. Only after a higher-scale, lower-latency Device Inventory Service replaced the interim one did Google "gradually replace IP-based policies with trust tier assignments," and even then it proceeded tier by tier, applying fine-grained restrictions "to higher trust tiers, proceeding to our ultimate goal of retroactively increasing trust tier requirements for devices and corporate resources over time"7 — an incremental tightening rather than a single cutover.

The paper is explicit that this staged approach exists specifically to avoid disrupting users: the team built a simulation pipeline that classified real corporate-network traffic against the proposed policy before enforcing it, which surfaced services that would have broken silently on an unprivileged network, "such as services that had supposedly been decommissioned but were still running with no clear owner"7 — a reminder that any access-policy tightening exercise doubles as an inventory audit, because you cannot correctly gate a resource you don't know still has live traffic. The paper also treats mobile as a special case worth calling out on its own: because mobile access is "almost all... exclusively HTTP-based" and Android devices already carry cryptographically secured device identification, a tiered trust model turned out to be easier to deploy on mobile than on some legacy desktop workflows, not harder7 — the opposite of the intuition that mobile devices are the hard case for a zero-trust rollout.

6. Relationship-based authorization: Zanzibar's relation tuples

BeyondCorp answers "should this device reach this resource at all." A separate question sits underneath it once the device is allowed through: "does this specific user have permission to view, edit, or comment on this specific object" — and that question, at Google's scale, has to be answered consistently across Calendar, Cloud, Drive, Maps, Photos, and YouTube, "hundreds of client services," while storing more than two trillion access-control entries and serving millions of checks per second8. Zanzibar is Google's answer, and its data model is deliberately narrow: every permission, and every group membership, is represented as one uniform kind of object called a relation tuple.

Zanzibar's model is worth contrasting directly with plain role-based access control (RBAC), because the difference is exactly what makes the nested-group check in §6.1 possible at all. RBAC assigns a user to one of a fixed set of roles (admin, editor, viewer) and grants that role a fixed bundle of permissions; the mapping from role to permission is defined once, centrally, independent of any specific object. Zanzibar's relations are per-object and composable: "the language allows clients to define arbitrary relations between users and objects, such as owner, editor, commenter, and viewer. It includes set-algebraic operators such as intersection and union for specifying potentially complex access control policies in terms of those user-object relations. For example, an application can specify that users granted editing rights on a document are also allowed to comment on the document, but not all commenters are given editing rights"8. That last clause is the tell: the relationship between "editor" and "commenter" is not two independent role definitions that happen to overlap in a permissions table — it is a directed implication (editor implies commenter, not the reverse) expressed once, in the configuration language, and then automatically true for every document rather than something an administrator has to remember to replicate object by object. This relationship-based approach to authorization (sometimes abbreviated ReBAC) is what Module 22 assumes when it applies the same tuple model to org/workspace/API-key permissions on an inference platform, rather than re-deriving relation tuples from scratch.

6.1 Resolving a nested-group check

A relation tuple has the form \(\langle object \rangle \# \langle relation \rangle @ \langle user \rangle\)8 — object, then a relation name, then a user. The "user" slot can hold either a concrete user id or a userset, itself written as \(\langle object \rangle \# \langle relation \rangle\), which is what lets one tuple point at "everyone who holds some relation to some other object" instead of a single individual — Zanzibar's own description calls this feature out explicitly: "one feature worth noting is that a userset allows a client to refer to groups and thus supports representing nested group membership"8. The paper's own example table gives the exact tuples needed to see this resolve end to end8:

  • group:eng#member@11 — user 11 is a member of group:eng.
  • doc:readme#viewer@group:eng#member — members of group:eng are viewers of doc:readme.

A check for "does user 11 have the viewer relation on doc:readme?" does not match either tuple directly — there is no tuple doc:readme#viewer@11 stored anywhere. Zanzibar resolves it by walking the userset: the second tuple says the viewer relation on doc:readme is satisfied by anyone who is a member of group:eng; the first tuple says user 11 is exactly such a member. The check is satisfied by composing the two, and because groups can themselves contain other groups, "evaluating whether a user belongs to a group can entail following a long chain of nested group memberships"8 — the same two-tuple resolution repeated recursively at whatever depth the group hierarchy actually has. This is the general mechanism that lets a Zanzibar-style system answer "can this org's engineering team, three levels of sub-team down, view this document" without ever materializing a flat per-user access-control list — it stores the relationships and resolves membership at check time instead.

A nested-group check resolved by composing two relation tuples Check(doc:readme#viewer@11) does user 11 view doc:readme? doc:readme#viewer@group:eng#member members of group:eng are viewers of doc:readme is 11 a member? group:eng#member@11 user 11 is a member of group:eng ALLOW
Figure 5. Zanzibar's own example tuples8 resolving a check that matches neither tuple directly: the viewer relation on doc:readme is defined in terms of a userset (members of group:eng), and a second tuple supplies the group membership that closes the check.

Zanzibar's consistency model matters as much as its data model, because a permission system that returns stale answers is not a minor bug — it either denies access that was just granted or, worse, grants access that was just revoked. Zanzibar stores its tuples in Spanner (Module 4's externally-consistent transactional store) and attaches a client-supplied consistency token — a "zookie" — to checks so a client can force a check to be evaluated against ACL data no older than a specific prior change8, which is what lets it report 95th-percentile check latency under 10 milliseconds while still guaranteeing that an access revocation is visible to every subsequent check8.

The nested-group walk in Figure 5 is cheap for a two-tuple example, but groups can contain other groups, and Zanzibar's own paper is candid that "evaluating whether a user belongs to a group can entail following a long chain of nested group memberships," which it calls out as one of the central challenges the system has to survive at scale8 — a check on a deeply-nested organizational hierarchy is not one tuple lookup but a recursive fan-out across however many group layers separate the user from the object. Zanzibar's answer is not a smarter algorithm so much as aggressive reuse of work already done: it "handles hot spots on normalized data by caching final and intermediate results, and by deduplicating simultaneous requests. It also applies techniques such as hedging requests and optimizing computations on deeply nested sets with limited denormalization"8. Caching intermediate results means a popular group's membership resolution, once computed for one check, is reused by every other check that touches the same group instead of being recomputed from scratch; deduplicating simultaneous requests means two checks that arrive for the same object at nearly the same moment share one evaluation instead of doing the walk twice; hedging requests — issuing the same request to more than one replica and taking whichever answer returns first — trades a small amount of duplicated work for protection against the one slow replica that would otherwise set the tail latency for every check that happens to land on it. None of these techniques changes the correctness story in §6.1; they are what makes millions of checks per second at sub-10-millisecond p95 latency compatible with a data model whose worst case is an unbounded recursive graph walk.

Where experts disagree

The rate-limiting choice in §4 is not settled by theory — fixed window, sliding window, and token bucket are all in wide production use today, and the systems that document them (Cloudflare and Anthropic, in the sources this module draws on) each made a specific choice for a specific product rather than declaring one algorithm universally correct.

Fixed window is good enough, and its cost is the point

Implicit in Cloudflare's own presentation, which describes fixed window first as the plainer of its two documented modes5, and in the wide default use of fixed-window counters across rate-limiting middleware generally.

A fixed window needs one counter per caller and one reset timer; there is no per-request timestamp to store, no log to prune, and no arithmetic beyond incrementing an integer and comparing it to a threshold. At high request volume across millions of distinct callers, that constant-memory-per-caller property is not a rounding error — it is the difference between a rate limiter that fits in a shared in-memory counter store and one that needs a timestamp log per caller.

  • O(1) state per caller, trivial to shard and replicate
  • simple to reason about and debug ("how many since the top of the window")
  • allows up to roughly 2x the nominal rate in a burst straddling a window boundary, as the exact 20-vs-10 example shows5
  • the boundary effect is worst precisely when it matters most — a caller deliberately timing requests around the reset can extract sustained above-limit throughput

Sliding window is the correct enforcement of the stated limit

Cloudflare's own sliding-window mode, offered specifically to close the fixed-window gap5.

If a limit is stated as "N requests per T," the only algorithm that actually enforces that statement at every instant, not just at window boundaries, is one that evaluates a trailing window of length T continuously. Anything else is a limit that is only true on average across whole windows and can be exceeded locally by design.

  • no boundary burst under any traffic pattern — the limit means what it says
  • an exact timestamp log makes the enforcement auditable after the fact
  • an exact implementation needs state proportional to traffic volume within the window, not O(1)
  • common approximations (blending two adjacent fixed-window counters) trade some of that precision back for lower memory, reintroducing a smaller version of the same boundary error

Token bucket is the better fit when bursts are expected and desirable

Anthropic's production rate limiter for the Claude API, chosen explicitly over a fixed-interval reset6.

Real clients are not uniform-rate traffic generators — a client that has been quiet for a while has legitimately earned the right to send a burst without being penalized for the algorithm's window alignment, and a token bucket's continuously-replenishing capacity is designed around exactly that pattern rather than treating every window boundary as a hard reset.

  • O(1) state per caller (a token count plus a last-refill timestamp), same memory profile as fixed window
  • burst capacity and steady-state rate are two independently tunable numbers, letting an operator permit bursts on purpose rather than as an accident of window alignment
  • still allows a full-capacity burst at any moment the bucket happens to be full, which is a real burst even though it isn't tied to a clock boundary
  • choosing capacity and refill rate well requires knowing the traffic pattern you want to permit, which is one more parameter to tune than a bare window limit
Assessment

These are not three attempts at the same goal with one winner — they answer subtly different questions. If the requirement is "never allow more than N requests in any T-length interval, full stop," sliding window is the only one of the three that is actually correct, and the memory cost of exact enforcement is usually affordable once you accept it scales with traffic rather than with caller count. If the requirement is "protect the backend from sustained overload while letting quiet, well-behaved callers burst occasionally," token bucket is the better fit, because it makes burst tolerance an explicit, tunable knob instead of an artifact of clock alignment — this is why Anthropic chose it for an API where request patterns are naturally bursty (an agent making several rapid calls, then idling)6. Fixed window survives in practice mainly where its O(1) simplicity matters more than either property — extremely high-cardinality caller counts, or limits that are advisory/coarse rather than a hard SLA — and where the operator has either measured that the boundary-burst risk is acceptable or layered a coarser fixed-window limit underneath a finer-grained token bucket, which is a common combination rather than a forced choice between the two. What would settle this for any specific system is not more theory but a traffic trace: replay real caller request timestamps against all three algorithms and measure how often the fixed-window boundary case actually triggers in practice — for many caller populations it is rare enough that the simplicity wins, and for adversarial or bursty populations it is common enough that it doesn't. LLM inference APIs sharpen this disagreement rather than settling it, because the true cost of a request — the number of output tokens actually generated — is not known until after the request has already consumed GPU time, which is a constraint none of Cloudflare's or Anthropic's request-counting examples above have to deal with; Module 22 takes up that sharper version of the same three-way disagreement directly.

References

  1. blogGuo, Liang. Building Services at Airbnb, Part 1. Airbnb Tech Blog, published 2017-12-12. https://medium.com/airbnb-engineering/building-services-at-airbnb-part-1-c4c1d8fa811b. Accessed 2026-09-17. Used for: the monolith-to-SOA migration framing, the Service IDL / Thrift-over-HTTP design decision, and the "make-me-a-service" tooling's measured 3-week/2-3-week-saved figures.
  2. officialVitess contributors. What is Vitess?. Vitess (CNCF graduated project) documentation, 2026. https://vitess.io/docs/overview/whatisvitess/. Accessed 2026-09-17. Used for: VTGate/VTTablet architecture, the topology service, YouTube's five-plus-year usage, and the 256KB-3MB per-connection MySQL memory overhead.
  3. officialIstio contributors. Istio / Concepts / Security. istio.io documentation, 2026. https://istio.io/latest/docs/concepts/security/. Accessed 2026-09-17. Used for: the sidecar traffic-interception mechanism and mTLS identity/certificate-rotation description used to generalize VTGate's pattern to a service mesh.
  4. paperVerma, Abhishek et al. Large-scale cluster management at Google with Borg. EuroSys '15. https://research.google.com/pubs/archive/43438.pdf. Accessed 2026-09-17. Used for: median cell size, admission-control-by-quota checked at submission not scheduling time (§2.5), and prod CPU/memory allocation-vs-usage figures (§2.1).
  5. officialCloudflare. Rate limiting (Cloudflare AI Gateway). developers.cloudflare.com, 2026. https://developers.cloudflare.com/ai-gateway/features/rate-limiting/. Accessed 2026-09-17. Used for: the fixed-window vs. sliding-window 10-per-10-minute burst example and the 429 rejection behavior, reused verbatim in the worked derivation and Figure 3.
  6. officialAnthropic. Rate limits (Claude API). platform.claude.com documentation, 2026. https://platform.claude.com/docs/en/api/rate-limits. Accessed 2026-09-17. Used for: the general token-bucket definition (continuous replenishment vs. fixed-interval reset), used only for the mechanism, not Anthropic's specific per-dimension limits.
  7. paperOsborn, Barclay, Justin McWilliams, Betsy Beyer, and Max Saltonstall. BeyondCorp: Design to Deployment at Google. USENIX ;login:, vol. 41, no. 1, Spring 2016, pp. 28-33. https://research.google.com/pubs/archive/44860.pdf. Accessed 2026-09-17. Used for: the six-component architecture (p.28), the Trust Inferer and tiered-access mechanics with worked examples (p.29), the Device Inventory Service ingestion scale (p.30), and the phased-rollout and mobile-platform sections (pp.31-32).
  8. paperPang, Ruoming et al. Zanzibar: Google's Consistent, Global Authorization System. USENIX ATC '19. https://www.usenix.org/system/files/atc19-pang.pdf. Accessed 2026-09-17. Used for: relation-tuple grammar and nested-group userset support (§2.1, p.34), Table 1's example tuples (p.35), zookie-based external consistency, and the trillions-of-ACLs / sub-10ms-latency scale figures (§1).