System Design · Module 2
Caching at Every Layer: Placement, Coherence, and Invalidation
Every rung of the request path you mapped in Module 1 also holds a copy of something. A cache is a bet that the copy you already have is still the right answer; this module is about what breaks when that bet is wrong — how the four write-path strategies differ, why TTL expiry on a popular key can take a database down, and how a fixed amount of fast storage gets allocated across a catalog by popularity rather than recency alone.
1. The cache placement ladder
Module 1 walked the hop-by-hop path a request takes from a client's DNS lookup to an origin server: anycast edge, CDN point of presence, load balancer, reverse proxy. Five of those hops (and one more below the origin that Module 1 didn't need to open) are also caches. A cache, for the rest of this module, is anything that answers a read from a copy instead of from the system of record, on the bet that the copy is still correct. The bet is priced in two currencies: how much latency you save when it pays off, and how wrong an answer you might serve when it doesn't. Figure 1 lays the five rungs out in the order a request meets them, and the rest of this section is about what makes each rung's validity mechanism different, even though the hit/miss shape is identical everywhere.
There is no single document that describes this ladder end to end — no company has published a "caching architecture" paper that spans browser to disk block. What follows is stitched from the primary source for each rung: the IETF's HTTP caching standard for the browser, Cloudflare's own CDN reference architecture for the edge, and Google's Bigtable paper for the database rung, with Kleppmann's synthesis in Designing Data-Intensive Applications (DDIA) supplying the cross-cutting frame7.
1.1 Browser and HTTP caching
The browser cache is governed entirely by HTTP, and HTTP caching is an IETF standard (RFC 9111, obsoleting RFC 7234), not a vendor convention. A cache computes a response's freshness lifetime by checking, in order, the shared-cache s-maxage directive, then max-age, then an Expires header, then a heuristic if none of those are present1. The max-age response directive states directly how long a response is good for: "the response is to be considered stale after its age is greater than the specified number of seconds"1 (§5.2.2.1). Once a stored response is older than its freshness lifetime it is stale, defined as a response that "either has explicit expiry information or is allowed to have heuristic expiry calculated, but is not fresh"1 (§4.2.4). Staleness does not mean the cache throws the entry away; it means the cache cannot use it without first validating it.
Validation is the mechanism that lets a cache reuse bytes it already has instead of re-downloading them. RFC 9111 defines two validators: a Last-Modified timestamp, and an entity tag carried in ETag1 (§4.3.1). A cache holding a stale response sends a conditional request — If-None-Match against the stored ETag, or If-Modified-Since against the stored timestamp — and if the origin agrees nothing changed, it returns 304 Not Modified, at which point "the stored response can be updated and reused"1 (§4.3.3) without re-sending the body. This is the cheapest possible cache refresh: one round trip, no payload, and a reset freshness clock. It is also the pattern every other rung in the ladder reinvents in its own vocabulary — a check against the source of truth that is far cheaper than a full re-fetch.
Two more directives matter enough to name here and use later. no-cache means a stored response cannot satisfy a request "without forwarding it for validation"1 (§5.2.2.4) — despite the name, this still caches the bytes, it just forces a freshness check every time. no-store is the actual "do not cache this" directive: a cache "MUST NOT store any part of either the immediate request or the response"1 (§5.2.2.5). A separate extension, stale-while-revalidate, lets an origin tell a cache it may keep serving a response for a bounded window past its freshness lifetime while it revalidates in the background, "hiding latency (both in the network and on the server) from clients"2 (§1). That single directive is the browser/CDN-tier answer to a problem this module returns to at length in §4: what happens when a lot of clients hit a stale entry at once.
1.2 CDN edge and tiered cache
Module 1 already covers how a request finds its way to an edge point of presence and what a general-purpose CDN and a purpose-built one like Netflix's Open Connect each do with that placement. What Module 1 does not cover, and what belongs here, is what a CDN does once a response is stale at one edge location but might still be fresh at another. Cloudflare's reference architecture calls this Tiered Cache: "certain data centers act as reverse proxies to the origin for other data centers, increasing cache-hit rate and reducing origin load"3. A miss at a small edge PoP does not necessarily reach the origin; it can be satisfied by a regional tier that already fetched the object for a different, nearby PoP. This is the CDN's version of the fall-through arrow in Figure 1 having an extra rung hidden inside it.
Cloudflare also documents a second, explicitly higher tier called Cache Reserve, built on R2 object storage, whose entire purpose is longer retention than the normal edge cache allows3. Critically, Cache Reserve does not change the freshness rules: "the TTL policy still affects the content's freshness which means when content cache TTL expires inside of Cache Reserve, the content will need to be revalidated by checking the origin for any updates"3. This is worth sitting with, because it is the general lesson of this entire module in miniature: adding a bigger, longer-lived cache tier does not remove the invalidation problem, it just moves the moment of truth further out in time. A tier that never revalidates is not a cache anymore; it is an uncoordinated second copy of the data, which is a correctness problem, not a performance win.
1.3 Reverse proxy and application-layer object caches
Below the CDN sits the generic key-value object cache: Memcached or Redis, usually reached from application code or through a reverse proxy such as Nginx or Varnish. Facebook's own account of building a Memcached-based key-value store at social-network scale describes exactly this rung: "we use memcache as a demand-filled look-aside cache... If the item addressed by that key is not cached, the web server retrieves the data from the database or other back-end service and populates the cache with the key-value pair"5 (§2). That is the cache-aside pattern by name, and §2 of this module works through it and its three siblings in full. For now, the placement point is simpler: this rung is the first one in the ladder where the application, not a protocol or a vendor's edge network, decides what gets cached, for how long, and how it gets invalidated. Everything below §1 is either a browser/CDN convention (HTTP headers) or a database internal (block cache); this rung is where system design as a discipline actually has design freedom, which is why §§2–4 spend the most time here.
1.4 Database buffer / block cache
The last cacheable rung before you are reading off disk is inside the storage engine itself. Bigtable stores its data in an immutable, ordered file format called SSTable, and each SSTable "is a persistent, ordered immutable map from keys to values... each block is 64 KB in size"6 (§4). A tablet server keeps an in-memory index over these blocks so that a single disk seek, guided by the index, finds the right 64 KB block; the block itself is then cached in memory once read, so a second access to a nearby key in the same block is free. This is a cache in the exact sense used everywhere else in this module — a copy, in a faster medium, that is trusted until something tells the tablet server otherwise — except that here the "something" is simpler than anywhere higher in the ladder: Bigtable's storage layer is log-structured and immutable, so a cached block never goes stale under a concurrent write the way a mutable row would; new writes go to a different, newer SSTable, and compaction (not invalidation) is what eventually reconciles versions6 (§4).
The general version of this rung, independent of Bigtable specifically, is the storage engine's buffer pool or the operating system's page cache sitting between a database and its disk. Kleppmann's synthesis in DDIA frames caches, indexes, and materialized views as a single category — derived data, meaning data that is reconstructable from a system of record and exists only to make reads cheaper7. That framing is the reason this module and the next one (Module 3, on replication and partitioning) stay strictly separated: replication and partitioning decide how many independent copies of the system of record exist and who owns which slice of it; caching decides how many derived, disposable copies exist on top of that, and how they get told they are wrong. A replica that falls behind is a consistency problem for Module 3. A cache that falls behind is an invalidation problem for this module. The failure modes look similar — a reader sees an old value — but the recovery story is different: you resynchronize a replica from its leader; you simply evict or refill a cache, because by construction it can always be rebuilt from the source of record7.
| Rung | Example implementation | Validity mechanism | Who triggers invalidation |
|---|---|---|---|
| Browser | HTTP cache in the user agent | max-age freshness, ETag/Last-Modified validation1 | Origin server response headers |
| CDN edge / tiered cache | Cloudflare edge PoP + regional tier | TTL, with Cache Reserve requiring revalidation on expiry3 | Origin, or a cache-purge API call |
| Reverse proxy cache | Nginx, Varnish | TTL plus request coalescing on miss4 | Upstream response headers |
| Application object cache | Memcached, Redis | TTL, explicit delete on write5 | Application code, on every write |
| DB buffer / block cache | Bigtable's 64 KB SSTable block cache6 | Immutability of SSTables; compaction, not invalidation | Storage engine's own compaction process |
2. Cache-aside, read-through, write-through, write-back
Everything in §1 was about where a cache sits. This section is about the one design decision every cache at the application-object rung actually forces you to make: who is responsible for keeping the cache and the database from disagreeing, and when. There are four named strategies, and cloud vendors don't even agree on names for all of them — AWS calls the first one "lazy loading," Azure calls it "cache-aside," Facebook's engineers call it a "demand-filled look-aside cache." The mechanism is identical across all three names; the vocabulary just never converged.
2.1 Cache-aside (lazy loading, look-aside)
Facebook's account of Memcache at scale describes the read path plainly: "we use memcache as a demand-filled look-aside cache... When a web server needs data, it first requests the value from memcache by providing a string key. If the item... is not cached, the web server retrieves the data from the database or other back-end service and populates the cache with the key-value pair"5 (§2). AWS's version of the same mechanism, which it calls lazy loading, spells out the miss penalty precisely: a miss costs three trips — "initial request for data from the cache," "query of the database for the data," "writing the data to the cache" — versus one trip on a hit8. Azure's cache-aside write path adds the detail that finishes the picture: "when an application updates information, it writes the change to the data store and then invalidates the corresponding item in the cache"9.
The word "invalidates" is doing real work there, and it is worth asking why cache-aside deletes a stale entry rather than overwriting it with the new value in place — both would seem to fix the staleness. Facebook's answer is explicit: "We choose to delete cached data instead of updating it because deletes are idempotent. Memcache is not the authoritative source of the data and is therefore allowed to evict cached data"5 (§2). A delete applied twice, or a delete that races with a slightly older write, always ends in the same state: no entry, which forces the next reader to refill from the database. An update applied out of order can leave the cache holding an older value than the one it just "fixed" — exactly the stale-set hazard this module comes back to in §4. Deleting instead of updating trades a small amount of extra miss traffic for the removal of an entire class of reordering bugs.
Cache-aside's principal cost is the miss penalty and the staleness window between a write and the next read. Its principal benefit, repeated in both the AWS and Azure sources, is that a cache-aside cache is optional infrastructure: if it disappears, the application still functions, just slower, because "node failures aren't fatal for your application. When a node fails and is replaced by a new, empty node, your application continues to function, though with increased latency"8. That property — a fully replaceable, disposable copy — is the whole reason DDIA groups caches with derived data rather than with the system of record7 (§1).
2.2 Read-through
Read-through moves the miss-fill logic in Figure 2's top half out of the application and into the cache client itself. AWS's Builders' Library draws the line between the two as "inline vs. side caches": an inline cache embeds "cache management into the main data access API," with Amazon DynamoDB Accelerator (DAX) given as the example, so that "caching can be added, removed, or tweaked without any changes to client logic"4. A cache-aside store like Memcached or Redis, by contrast, is a "generic object store" where "the application code directly manipulates the cache before and after calls to the data source"4 — exactly Figure 2's five explicit steps.
The convenience has a cost that is easy to miss until an incident makes it obvious: an inline cache is now load-bearing for availability, because "there is no opportunity for the client to compensate for a temporarily unavailable cache"4. Cache-aside's application code can, at least in principle, fall back to querying the database directly if the cache is down; a read-through cache client that fails closed has no such escape hatch built in, because the application never learns the difference between "cache miss" and "cache unreachable" unless the client library is written to expose it.
2.3 Write-through
Write-through keeps every write operation entangled: "the write-through strategy adds data or updates data in the cache whenever data is written to the database"8. Unlike cache-aside's delete, this actually puts the new value into the cache as part of the write, so the cache is never observed to be stale by a reader — Azure's phrasing is that write-through "updates the data store and the cache in the same write operation so that readers see the new value immediately after a successful write"9, which is the property that most directly distinguishes it from cache-aside's brief post-write miss window.
The price is paid on every write, not just the rare miss: "every write involves two trips: a write to the cache [and] a write to the database... which adds latency to the process"8. AWS's own assessment is that this is usually an acceptable trade because "end users are generally more tolerant of latency when updating data than when retrieving data"8. The other cost is specific to scaling events: a freshly added or replaced cache node has no data in it, and write-through by itself does nothing to fill that gap — "if you spin up a new node... there is missing data. This data continues to be missing until it's added or updated on the database"8. In practice this means write-through is almost always paired with a read-through or cache-aside miss path for the cold-node case, and used on its own only when every key is written far more often than a node's lifetime, which is rare.
2.4 Write-back (write-behind)
Write-back inverts the ordering completely: the write is acknowledged from the cache alone, and the database write is deferred. This pattern is old enough to be a first-class configuration option in distributed caching products; Hazelcast's MapStore documents it directly through a single tunable, write-delay-seconds: "if this value is zero, the MapStore is write-through, otherwise it is write-behind," and the delay is defined as the "number of seconds of delay before the MapStore.store(key, value) method is called"10. Azure's architecture documentation names the same family in passing when describing commercial caching systems that provide "read-through and write-through or write-behind operations. In these systems... the system automatically writes any changes made to cached data back to the data store"9 — write-behind is the case where "automatically" means "later, in the background," not "as part of this call."
The mechanism directly creates the failure mode the manifest for this module calls out explicitly: a data-loss window. Between the moment the cache acknowledges a write and the moment write-delay-seconds elapses and the deferred store call actually reaches the database, the new value exists in exactly one place. If that cache node is lost — process crash, host failure, eviction under memory pressure before the flush — the write is gone, and the database never saw it, with no error raised anywhere, because the application was already told the write succeeded. Write-through and cache-aside do not have this failure mode at all: in both, the database call is synchronous and on the critical path of the write, so by the time the application gets an acknowledgment, the database already has the value. Write-back is the only one of the four strategies where "the write returned success" and "the database has the value" can be false at the same time.
write-delay-seconds MapStore configuration10; general naming and "automatic" write-back framing per Azure's architecture documentation9.| Strategy | Read path | Write path | Staleness window | New/failed node | Data-loss risk |
|---|---|---|---|---|---|
| Cache-aside | App checks cache; miss reads DB, then fills cache5 | App writes DB, then deletes cache key5 | Brief: miss-to-refill gap only | Degraded latency, still correct8 | None — DB is authoritative |
| Read-through | Cache client fetches on miss; app never sees the DB call4 | Paired with write-through or cache-aside independently | Depends on paired write mode | Cache outage removes the app's fallback path4 | None (read path only) |
| Write-through | Same look-aside miss path as cache-aside | App writes DB and cache in one logical operation8 | None — readers see the new value immediately9 | Missing data until each key is next written8 | None — DB write is synchronous |
| Write-back | Same look-aside path; cache usually holds the freshest copy | App writes cache only; DB write deferred by write-delay-seconds10 | None from the reader's view; DB itself is stale until drained | N/A (writes go to cache first) | Real — unflushed writes are lost on cache-node failure |
3. Invalidation: TTL expiry vs explicit events
Every rung in §1 and every strategy in §2 eventually has to answer one question: how does a cache find out that the value it is holding is no longer correct? There are exactly two mechanisms, and every system in this module is built from one, the other, or a combination.
The first is time-based expiry: attach a TTL to the entry and stop trusting it once the TTL elapses, whether or not the underlying value actually changed. AWS's ElastiCache documentation frames the trade directly: "lazy loading allows for stale data but doesn't fail with empty nodes. Write-through ensures that data is always fresh, but can fail with empty nodes... By adding a time to live (TTL) value to each write, you can have the advantages of each strategy"8. A TTL is a knob traded against two costs in opposite directions: "don't make the expiration period too short because premature expiration can cause applications to continually retrieve data from the data store... don't make the expiration period so long that the cached data becomes stale"9. Nothing about a TTL knows whether the value actually changed; it is a blind, calendar-based guess calibrated to how often the data typically changes.
The second is explicit invalidation: the system of record tells the cache, at the moment of the write, that a specific key is now wrong. Cache-aside's delete-on-write (§2.1) is the simplest version of this, and it composes with a TTL rather than replacing it — the delete handles the common case where the same process that wrote the database also fronts the cache; the TTL is the backstop for every other way the database could have changed (a different service, a batch job, a direct database edit) that never goes through the cache-invalidating code path at all.
3.1 The ordering hazard: update-then-delete, not delete-then-update
Explicit invalidation has exactly one place it is easy to get backwards, and Azure's documentation calls it out directly. The correct order is: write the database first, then delete the cache key. Getting this backwards creates a race: "if you remove the cached item first, there's a small window of time when a client might fetch the item before the data store is updated. In this situation, the fetch results in a cache miss because the item isn't in the cache. The cache miss causes the application to retrieve the outdated item from the data store and add it back to the cache. This sequence leads to stale data in the cache"9. Deleting first opens a window in which a concurrent reader repopulates the cache with the value the write is about to overwrite — the cache ends up holding data that is now permanently wrong until the next write or TTL expiry, because nothing else will invalidate it. Writing first and deleting second closes that window: any reader who misses during the gap either sees the new value already committed, or misses and refills with it a moment later.
3.2 Pushing invalidation from the source of a record: DDIA's dual-writes critique
Both mechanisms above assume the same process that wrote the database also remembers to touch the cache. That assumption breaks the moment more than one writer exists, which is close to universal at any real scale — a batch job, an admin console, a second microservice, all writing the same row without necessarily running the same cache-invalidation code. Kleppmann's treatment of this problem in DDIA frames caches, search indexes, and materialized views as derived data: data whose correctness depends entirely on staying synchronized with a system of record it was computed from7. The failure mode he describes for keeping two such systems in sync by having the application issue two separate writes — one to the primary store, one to the derived store — is that the two writes can be reordered, or one can fail while the other succeeds, with no mechanism to detect or repair the divergence; the standard replacement is to make the primary store's write the single source of change events (a write-ahead log, or change-data-capture stream) and have every derived system, including caches, consume that stream to decide what to invalidate, rather than trusting every writer to remember to call `cache.delete()`7.
This is the fully event-based end of the invalidation spectrum: no TTL at all, because the cache is told about every relevant write as it happens, from the one place all writes must pass through. It removes the "did every writer remember" problem completely. What it does not remove is the cold-miss problem — a key nobody has read yet still has to be filled from the database on first access, by some cache-aside-shaped path, event stream or not — and it adds a hard infrastructure dependency (a durable, ordered log with every writer hooked up to it) that most applications reaching for a cache do not already have for other reasons. In practice this is why TTL-plus-explicit-delete, not event-sourced invalidation, is what AWS's and Azure's own caching documentation describes as the default: the log-based approach is the more correct answer to "how do I never miss an invalidation," but it is also the more expensive one to build, and the disagreement over when that expense is worth paying is taken up directly in this module's closing section.
4. Cache stampede and thundering herd
TTL-based expiry has a failure mode that explicit invalidation does not: because many independent readers can be relying on the exact same cached entry, and that entry expires at one single moment, a popular key's expiry can turn one cache miss into hundreds arriving at the origin at once. AWS's Builders' Library names this directly: "many clients make requests that need the same uncached downstream resource at approximately the same time... [which] results in a large number of requests... which can lead to throttling/brownout"4. The mechanism that makes this specifically dangerous, rather than merely inefficient, is that the database or backend being hit was sized for the cache's steady-state hit rate, not for the full uncached request rate arriving in a single burst.
The size of the burst is not fixed; it scales with how long the origin takes to answer. Every request that arrives after the TTL expires and before the first regenerated value is back in the cache is a fresh miss, so the number of colliding regenerate attempts scales roughly with the product of the key's request rate and however long regeneration takes. This is what makes the failure mode self-reinforcing: a database that is already slow (because it is receiving an unexpected burst) takes longer to answer each of those queries, which widens the window in which more requests arrive uncoalesced, which enlarges the burst further.
4.1 Leases: Facebook's mechanism, and the measured effect
Facebook's Memcache paper introduces leases specifically to address this, alongside a second, related problem of stale sets from reordered writes: "a thundering herd happens when a specific key undergoes heavy read and write activity. As the write activity repeatedly invalidates the recently set values, many reads default to the more costly path. Our lease mechanism solves both problems"5 (§3.2.1). The mechanism is a token, not a lock: "a memcached instance gives a lease to a client to set data back into the cache when that client experiences a cache miss. The lease is a 64-bit token bound to the specific key the client originally requested. The client provides the lease token when setting the value in the cache. With the lease token, memcache can verify and determine whether the data should be stored"5 (§3.2.1). Only the client holding the current lease for a key is allowed to refill it; every other concurrent miss on that key is told to wait briefly rather than being sent to the database itself: "each memcached server regulates the rate at which it returns tokens. By default, we configure these servers to return a token only once every 10 seconds per key. Requests for a key's value within 10 seconds of a token being issued results in a special notification telling the client to wait a short amount of time"5 (§3.2.1). In effect, one request per key per 10 seconds is allowed through to regenerate the value; everyone else that arrives in that window waits instead of independently re-querying the database.
Facebook measured the effect directly, on the specific set of keys they identified as most susceptible to thundering herds, over a full week: "without leases, all of the cache misses resulted in a peak database query rate of 17K/s. With leases, the peak database query rate was 1.3K/s"5 (§3.2.1). That is a reduction of roughly \( 17{,}000 / 1{,}300 \approx 13.1\times \) in the peak load actually reaching the database, for the same client-side traffic — the lease mechanism did not reduce the number of requests arriving at memcache at all, only the number that were allowed to fall through to the database when a value was missing.
4.2 Request coalescing, one layer up the ladder
The same idea reappears at the reverse-proxy rung under a different name. AWS's remedy for the general problem is "request coalescing, where the servers or external cache ensure that only one pending request is out for uncached resources," and it notes that "some external inline caches (such as Nginx or Varnish) do as well"4. This is the same mechanism family as a lease — one request goes through, the rest are held or coalesced onto its result — implemented at whichever rung of Figure 1's ladder happens to own the cache in question, rather than only inside the object-cache client. The label in the wild for this specific implementation pattern is often "single-flight": deduplicate concurrent callers asking for the same not-yet-available key into a single in-flight fetch, then fan the one result out to every waiter.
4.3 Serving stale instead of waiting
A different family of mitigation avoids making anyone wait at all: instead of coalescing requests behind a lock or a lease, serve the last known-good value immediately, and refresh it in the background. This is exactly what stale-while-revalidate (§1.1) does at the HTTP layer2. AWS's Builders' Library documents the equivalent idea at the application-cache tier as a soft/hard TTL split: "the client will attempt to refresh cached items based on the soft TTL, but if the downstream service is unavailable... the existing cache data will continue to be used until the hard TTL is reached," citing the AWS IAM client as a production example of the pattern4. Nobody waits on the origin at all in the common case; the origin is consulted asynchronously, and only an unusually long outage (past the hard TTL) forces a caller to actually block or fail. §5 of this module's closing "Where experts disagree" section returns to exactly this contrast: coalesce-and-wait versus serve-stale-and-revalidate are two different, both well-attested answers to the same miss-at-scale problem, and they are not interchangeable.
One more mitigation is worth naming because it costs nothing beyond slightly more code: jittered TTL. If every reader that filled a key did so at the same wall-clock moment, they all expire together and the herd is synchronized by construction. Adding a small random offset to each entry's TTL — a few percent of the base TTL, applied independently per fill — spreads expirations across a window instead of a point, so that a burst of \(K\) simultaneous misses becomes a trickle of roughly \(K\) misses spread over the jitter window, at a rate the origin was already sized to handle. This does not fix the worst case (a genuinely cold, never-before-seen key still produces a real burst on first popularity), but it removes the self-inflicted case where the cache's own fill behavior manufactured the synchronization.
5. Popularity-aware placement: memory vs NVMe
Every rung in Figure 1 has a fixed budget of its fastest medium and a much larger, cheaper medium behind it — DRAM in front of NVMe or disk, an edge PoP's local cache in front of a regional tier. The remaining design question this module has not yet addressed is: given that budget, which fraction of the catalog goes in the fast tier? Module 1 covers Netflix's Open Connect appliances as a purpose-built CDN generally; the placement decision inside a single appliance is this module's content specifically, because it is a caching decision, not a topology one.
Netflix's own engineering account of pushing a single Open Connect appliance to 100 Gbps is explicit about the split: "10-20% of the most popular titles [are] being served from memory" with the remainder served from NVMe-based solid-state storage11. The reasoning is a direct application of content popularity following a long tail — a small fraction of titles account for a disproportionate share of requests, so putting that fraction in the fastest medium captures most of the latency and throughput benefit without needing enough DRAM to hold the entire catalog, which at Netflix's catalog size is not an option regardless of budget.
What makes this worth a dedicated figure rather than a one-line fact is a result that runs against the instinct that "memory is always faster, so more memory-resident content is always better." Netflix measured the opposite when they tried serving everything from memory instead of NVMe: "the performance actually dropped from being CPU limited at 40 Gbps to being CPU limited at only 22 Gbps"11. Copying data out of the page cache into a socket buffer costs CPU cycles per byte regardless of whether the source was DRAM or NVMe; at the throughput Netflix was targeting, that fixed per-byte copy cost dominates, and NVMe's own DMA-driven read path turned out to spend fewer CPU cycles per byte served than treating memory as the source, freeing up CPU for the encryption and TCP work the appliance also has to do. The lesson generalizes past Netflix's specific numbers: past a certain throughput target, "put everything in the fastest medium" can be a worse design than "put the popular fraction in the fastest medium and let the rest use a medium that costs the serving path less CPU," because the bottleneck has moved from medium latency to the CPU cost of moving bytes at all.
The same rank-not-recency principle appears, in miniature, at the database rung of Figure 1: Bigtable's in-memory block index6 is itself a fixed, small, fast-tier budget sitting in front of a much larger set of 64 KB SSTable blocks on disk, and which blocks stay resident is governed by an ordinary least-recently-used-style policy rather than by any per-key configuration. The difference at Netflix's tier is that the 10-20% split is a deliberate, popularity-informed placement decision made ahead of time from known catalog statistics, not a policy that discovers popularity reactively the way an LRU cache does. Both are answers to the identical constraint — a fast tier that is far smaller than the data it fronts — and the choice between "let an eviction policy discover what's hot" and "decide what's hot in advance and place it" recurs throughout system design wherever a fast/slow storage split exists.
Where experts disagree
§4 laid out three genuinely different answers to the same moment: a hot key's cached value is gone or about to be, and the traffic that was reading it has to go somewhere. None of the three sources below are arguing that the other two are wrong in general; each is optimizing for a different failure cost, and the disagreement is really about which cost a given system can least afford.
Block-and-coalesce: leases, single-flight, request collapsing
Facebook's Memcache design (leases)5; AWS's Builders' Library, naming the same pattern at the proxy tier via Nginx/Varnish4
Admit exactly one regenerating request per key per interval; make every other concurrent caller wait a short, bounded time or explicitly retry, rather than letting them independently reach the origin. Correctness is preserved exactly: no caller ever observes data older than what the single regenerating request fetched.
- Measured, large effect: peak database load fell roughly 13× on Facebook's most stampede-prone keys5.
- No caller reads data staler than the fill it is waiting on.
- Adds real latency to every waiting caller, up to the regeneration time.
- Needs a coordination primitive (a lease token, a lock, or a proxy that tracks in-flight keys) that a plain cache client does not provide for free.
Serve stale, revalidate in the background
IETF's stale-while-revalidate extension, adopted across CDN and HTTP caching tiers2; AWS's soft/hard TTL pattern for the same idea at the application-cache tier4
Never make a caller wait on the origin at all. Return the last known value immediately, kick off exactly one background refresh, and only fall back to blocking if the background refresh itself has been failing for long enough to cross a second, harder deadline.
- Zero added latency for every caller in the common case — strictly better on the metric that usually matters most to users.
- No waiting-request bookkeeping; a background timer is enough.
- Callers can read data that is staler than the nominal TTL implies, by however long the background revalidation takes.
- Wrong choice outright for data where "slightly old" is a correctness bug, not a UX nuisance (an authorization decision, an account balance).
Remove the synchronized-expiry moment: event-driven invalidation
Kleppmann's derived-data framing in DDIA: dual writes are racy, so push invalidation from a durable change log instead of relying on TTL polling7
Don't give every copy of a key the same expiry clock in the first place. Have the system of record emit an event on every write; every cache consumes that stream and invalidates precisely the keys that actually changed, whenever they change, instead of guessing with a TTL.
- No synchronized expiry, so no herd caused by the cache's own fill behavior.
- Correctness does not depend on every writer remembering to call a delete.
- Requires a durable, ordered log with every writer already hooked up to it — infrastructure most caches reaching for a TTL do not otherwise need.
- Does not remove the cold-miss case: a key nobody has read before still needs a fill path, herd-prone or not.
These are not competing beliefs about the same workload; they are correctly matched to different ones. Serve-stale-and-revalidate wins whenever the data is read far more often than it changes and a few seconds of extra staleness costs nothing measurable — CDN assets, catalog pages, most of what sits at the top three rungs of Figure 1 — because it is strictly cheaper than coalescing when staleness is free. Leases and request coalescing win when the origin behind the cache is a shared, expensive, easily-overloaded resource and the hot set is small enough that a single-digit-millisecond wait is a rounding error next to the alternative — Facebook's own numbers are the case study: a database, not a CDN, sitting behind memcache. Event-driven invalidation wins only when the durable log already exists for an unrelated reason (replication, audit, event sourcing); building one solely to avoid a stampede is rarely the cheapest fix available, since a jittered TTL plus coalescing solves the same problem with far less new infrastructure. What would settle a specific case: measure the peak concurrent-miss rate on your hottest key at the current TTL. If it is a small multiple of steady-state load, jitter alone is enough. If it is an order of magnitude or more above steady-state — Facebook's 13× is the calibration point — the origin needs either coalescing or a staleness budget it can afford, and which of those two is cheaper depends entirely on whether that specific data can tolerate being a few seconds wrong.
What is changing
Cache Reserve's design — a cache tier backed by durable object storage (R2) rather than ordinary edge RAM/SSD, still governed by the same TTL-and-revalidate rules as every other tier3 — is one instance of a broader move: cache tiers that used to be purely disposable (lose them, warm them back up, nothing is lost forever) are being built on storage durable enough that losing them looks more like a real recovery event than a warm-up period. That is the same direction DDIA's derived-data framing already points7: the practical difference between "a cache" and "a system of record" was never the data model, only how casually you were allowed to treat data loss, and that line is shifting as vendors add durability underneath tiers that used to have none. Whether that makes such a tier a cache at all, in the sense this module has used the word, is a naming question the field has not settled.
References
- standardIETF. RFC 9111: HTTP Caching. R. Fielding, M. Nottingham, J. Reschke, eds., 2022. https://www.rfc-editor.org/rfc/rfc9111.html. Accessed 2026-09-17. Used for: freshness lifetime and max-age (§4.2, §5.2.2.1), staleness (§4.2.4), validation via ETag/Last-Modified and 304 (§4.3, §4.3.1, §4.3.3), no-cache/no-store (§5.2.2.4, §5.2.2.5).
- standardIETF. RFC 5861: HTTP Cache-Control Extensions for Stale Content. M. Nottingham, 2010. https://www.rfc-editor.org/rfc/rfc5861.html. Accessed 2026-09-17. Used for: stale-while-revalidate and stale-if-error semantics and rationale (§1, §3).
- officialCloudflare. Content Delivery Network (CDN) Reference Architecture. Live documentation, 2026. https://developers.cloudflare.com/reference-architecture/cdn-reference-architecture. Accessed 2026-09-17. Used for: Tiered Cache mechanism, Cache Reserve and its revalidation-on-expiry behavior.
- blogAmazon Web Services (Builders' Library). Caching Challenges and Strategies. Live documentation, n.d. https://aws.amazon.com/builders-library/caching-challenges-and-strategies/. Accessed 2026-09-17. Used for: inline vs. side cache framing, request coalescing (Nginx/Varnish), soft/hard TTL, negative caching, thundering herd definition.
- paperNishtala, R. et al. Scaling Memcache at Facebook. Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI '13), pp. 385–398, 2013. https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf. Accessed 2026-09-17. Used for: demand-filled look-aside cache definition (§2), delete-not-update rationale (§2), leases mechanism and measured 17K/s→1.3K/s peak database query rate (§3.2.1).
- paperChang, F. et al. Bigtable: A Distributed Storage System for Structured Data. Proceedings of OSDI '06, 2006. https://www.usenix.org/legacy/event/osdi06/tech/chang/chang.pdf. Accessed 2026-09-17. Used for: SSTable 64 KB block size and in-memory block index (§4).
- textbookKleppmann, M., Riccomini, C. Designing Data-Intensive Applications. 2nd edition, O'Reilly Media, 2026. https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/. Accessed 2026-09-17. Used for: derived-data framing of caches/indexes/materialized views, and the dual-writes-are-racy critique motivating log/CDC-based invalidation. Chapter numbers not cited individually: the publisher's table of contents returned HTTP 403 to automated fetch, and 2nd-edition chapter numbering is unconfirmed against the well-known 1st-edition structure.
- officialAmazon Web Services. Caching strategies for Memcached. Amazon ElastiCache User Guide, live documentation. https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Strategies.html. Accessed 2026-09-17. Used for: lazy loading (cache-aside) and write-through mechanics, advantages/disadvantages, TTL rationale.
- officialMicrosoft. Cache-Aside pattern. Azure Architecture Center, Microsoft Learn, updated 2025-12-09. https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside. Accessed 2026-09-17. Used for: cache-aside pattern definition, write-through/write-behind naming, update-then-delete ordering hazard.
- officialHazelcast. Building a Cache with MapStore and Configuring a MapStore. Hazelcast Documentation, v5.7, live documentation. https://docs.hazelcast.com/hazelcast/5.7/mapstore/configuration-guide. Accessed 2026-09-17. Used for: write-delay-seconds configuration distinguishing write-through (0) from write-behind (>0), and the deferred MapStore.store() call.
- blogNetflix Technology Blog. Serving 100 Gbps from an Open Connect Appliance. 2017-09-29. https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99. Accessed 2026-09-17. Used for: 10-20% memory-resident title fraction, memory-only throughput regression from 40 to 22 Gbps.