System Design · Module 15
Netflix Resilience and Personalization: Chaos Engineering and the Offline/Nearline/Online Split
Netflix runs two disciplines that look unrelated but solve the same problem: what to do when a resource — a thread, a millisecond of latency budget, an uptime guarantee — is finite and something you don't control keeps trying to take more of it than you have. Chaos engineering and Hystrix are how Netflix decided, deliberately and in production, whether its systems actually survive that scarcity; the offline/nearline/online split is how Netflix decided which of its recommendation algorithms get to spend a request's worth of that same scarce latency and which have to work within a much longer, cheaper budget instead. This module reads both directly from Netflix's own engineering accounts.
1. Chaos engineering: building confidence by breaking things on purpose
Module 11 §6.1 already gives chaos engineering's formal definition and its four advanced principles as a general verification discipline. This section applies that discipline to the company that invented it, with the specific tools Netflix built and the specific outage classes those tools were designed to surface — the worked example Module 11 deliberately left for this module.
1.1 The four advanced principles, restated for what Netflix actually built
Chaos Engineering is, in the canonical definition co-authored by the Netflix engineers who built it, "the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production"1. The baseline method is to state a hypothesis about a system's steady-state behavior — "focus on the measurable output of a system, rather than internal attributes of the system"1 — and then check whether an injected failure changes that output. Four further principles, all drawn directly from the same document, describe how Netflix runs that experiment credibly rather than as a one-off drill: vary the injected events to reflect real-world failure classes, prioritized by likely impact or frequency, rather than arbitrary faults; run the experiment directly against production traffic wherever possible, because a staging environment's traffic pattern, cache state, and dependency graph are never quite the production ones; automate experiments so they run continuously rather than as an occasional exercise; and, non-negotiably, minimize and contain the blast radius of whatever fallout the experiment causes1.
That last principle is the one Netflix's own practice took the longest to formalize, and reading its two chaos documents side by side against their publication dates makes the direction of travel explicit. The 2011 post introducing the Simian Army describes Chaos Monkey's method in blunt terms: "a tool that randomly disables our production instances"2 — no stated targeting, no stated scope beyond "production." The 2019 Principles of Chaos document, written after nearly a decade of Netflix (and by then, the wider industry's) chaos-engineering practice, states blast-radius minimization as an explicit, load-bearing principle in its own right1 — the same company's account of the same discipline, eight years later, making an implicit constraint an explicit one.
1.2 The Simian Army: from Chaos Monkey to Chaos Gorilla
Netflix's own framing for why any of this was necessary is a physical-world analogy stated directly in the post that introduced it: "since no single component can guarantee 100% uptime... we have to design a cloud architecture where individual components can fail without affecting the availability of the entire system... we have to be stronger than our weakest link"2. Designing for that is necessary but not sufficient — a fault-tolerant architecture that has never actually been exercised against a real failure is, in Netflix's own words, like keeping a spare tire in the trunk without knowing if it is inflated or whether you remember how to change it2. Chaos Monkey is the automated version of poking a hole in that tire on a Sunday afternoon instead of finding out it was flat on the freeway at night: it "randomly disables our production instances to make sure we can survive this common type of failure without any customer impact," run deliberately "in the middle of a business day, in a carefully monitored environment with engineers standing by," so that "next time an instance fails at 3 am on a Sunday, we won't even notice"2.
Chaos Monkey's success led Netflix to generalize the idea into a set of purpose-built tools, each targeting one specific class of failure or unhealthy state rather than one generic "kill something" action. Latency Monkey injects artificial delay into the RESTful client-server layer rather than killing an instance outright, which lets Netflix simulate a dependency's full unavailability — including cases where the failure is timing-based rather than an outright crash — "without physically bringing these instances down"2. Conformity Monkey and Security Monkey target configuration risk rather than runtime failure: Conformity Monkey shuts down instances that don't follow best practice (the stated example is an instance outside an auto-scaling group, which the post calls "trouble waiting to happen"2), and Security Monkey extends the same idea to security posture, finding "improperly configured AWS security groups" and expiring SSL/DRM certificates2. Doctor Monkey and Janitor Monkey handle degraded-but-not-failed and wasted-but-not-failed states respectively — Doctor Monkey removes instances that fail health checks or show abnormal signals like elevated CPU load, and Janitor Monkey disposes of unused cloud resources before they become clutter2. 10-18 Monkey (a numeronym for localization-internationalization) checks for configuration problems specific to serving customers across regions, languages, and character sets2. Chaos Gorilla scales Chaos Monkey's method up one level of granularity: instead of one instance, it "simulates an outage of an entire Amazon availability zone," specifically to verify that "services automatically re-balance to the functional availability zones without user-visible impact or manual intervention"2 — the same steady-state-hypothesis method from Section 1.1, applied at the zone level instead of the instance level.
2. Surviving what chaos finds: Hystrix on Netflix's own dependency graph
Chaos experiments are diagnostic — they tell you which dependency is a weak point, not what to do about it once you know. Module 11 §4 and §5 already build the general vocabulary for the answer: a circuit breaker's CLOSED/OPEN/HALF-OPEN state machine, and the choice between thread-pool and semaphore isolation as a bulkhead around each dependency3. This section does not re-derive that machinery. It applies it to the traffic numbers and the specific configuration Netflix's own engineering team published when Hystrix's predecessor — internally called the DependencyCommand pattern — first went into production, because the numbers are what make "circuit breaker" stop being an abstract state machine and start being a concrete answer to a concrete, measured problem.
2.1 The arithmetic that made fault tolerance mandatory
By 2012, the Netflix API was receiving more than 1 billion incoming calls per day, fanning out at an average ratio of 1:6 into several billion outgoing calls to dozens of underlying subsystems, with peaks exceeding 100,000 dependency requests per second, all running across thousands of EC2 instances4. At that volume, "intermittent failure is guaranteed... even if every dependency itself has excellent availability and uptime"4 — a claim that is not rhetorical but arithmetic, and Netflix's own engineering post states the calculation directly:
$$ 0.9999^{30} \approx 0.997 $$Thirty dependencies, each independently available 99.99% of the time, compound multiplicatively rather than additively, because the whole request fails if any one of the thirty fails. The result — roughly 99.7% combined availability — sounds close to the 99.99% each individual dependency promises, but translated into downtime the gap is large: "30 dependencies each with 99.99% uptime would result in 2+ hours downtime/month"4. Compare this to the allowed-downtime arithmetic Module 11 §1.2 builds for a single SLO: one dependency at 99.99% availability over a 30-day month allows about 4.3 minutes of downtime; thirty independent dependencies at that same individual target allow over 2 hours, a roughly 30-fold degradation purely from the multiplication, before any single dependency has gotten measurably worse. This is the reason Netflix's own post frames fault tolerance the way it does: "it is a requirement of high volume, high availability applications to build fault tolerance into their architecture and not expect infrastructure to solve it for them"4 — no amount of individually well-run dependencies fixes a multiplicative problem; only isolating each dependency's failure from the others does.
The failure mode this arithmetic predicts is specific, and Netflix's post names it precisely: "when a single API dependency fails at high volume with increased latency (causing blocked request threads) it can rapidly (seconds or sub-second) saturate all available Tomcat... request threads and take down the entire API"4. This is exactly the cascading-failure mechanism Module 11 Figure 6 draws for a shared thread pool with no bulkhead — Netflix's account is the production incident that mechanism is describing, not a hypothetical.
2.2 One breaker, one bulkhead, per dependency
Netflix's DependencyCommand wraps every call to an external system in the same combination of mechanisms Module 11 §3-§5 names generically: network-level timeouts and retries, a per-dependency thread pool (or, for cheap in-process calls, a semaphore), and a circuit breaker4. What Netflix's own numbers add is the specific tuning behind each layer. The circuit trips "when a DependencyCommand has passed a certain threshold of error (such as 50% error rate in a 10 second period) and will then reject all requests until health checks succeed"4 — the same 50% error-rate figure Module 11 Table 3 cites as Hystrix's later published default, here stated as Netflix's own production configuration a version earlier, evaluated over a 10-second rolling window rather than a raw request count. A tripped breaker "release[s] the pressure on underlying systems (i.e. shed load) when they are having issues and reduce[s] the user request latency by failing fast... instead of making every user request wait for the timeout to occur"4 — the same fail-fast argument Module 11 §4 makes in the abstract, restated here as an explicit design goal rather than a side effect.
The thread-pool sizing example Netflix published shows how conservative a working configuration actually looks in practice: "the threadpool is sized at 10 to handle a burst of 99th percentile requests, but when everything is healthy this threadpool will typically only have 1 or 2 threads active at any given time to serve mostly 40 ms median calls"4. Ten threads is not a capacity plan for the dependency's normal load — it is headroom for the tail, sized so that a burst of slow p99 requests does not exhaust the pool while the median case, which is what actually happens almost all the time, uses almost none of it. Moving each dependency's calls onto its own thread turned out to have a second, unrelated benefit Netflix's post is explicit about: once a call runs on a separate thread anyway, nothing stops several such calls from executing concurrently instead of serially, and Netflix used exactly that opening to add "request collapsing (automatic batching)," merging what would otherwise be several nearly-simultaneous calls to the same dependency into one, "to increase overall efficiency and reduce user request latencies"4 — the isolation mechanism paying for its own thread-handoff overhead by also being the thing that makes concurrent dispatch possible in the first place. Semaphores, in the same architecture, are reserved for dependency calls known not to touch the network — in-memory cache lookups, for instance — "since the overhead of a separate thread is too high for these types of operations"4; Module 11 §5.2 already explains why that trade only makes sense when a hung call is not a realistic failure mode, and an in-process cache lookup is exactly that case. Netflix's post adds one further, easy-to-miss use of the same mechanism: a fallback function itself runs on the calling thread by contract, and rather than trusting every fallback implementation to honor that contract, "it too is protected by a semaphore so that if an implementation is done that involves a network call and becomes latent, the fallback itself won't be able to take down the entire app"4 — the bulkhead pattern applied recursively to the thing that is supposed to be the safety net.
Netflix's post also gives a worked example of how the network-level and thread-level timeouts in Module 11 §3.1 are meant to compose rather than duplicate each other, and it is worth reproducing because the arithmetic is not obvious from either layer alone. Take a dependency whose calls have no legitimate reason to sit near the tail of their own latency distribution: "cuts it short at the network timeout layer and immediately retries with the expectation to get median latency most of the time, and accomplish this all within the 300 ms thread timeout"4 — a short network timeout, one retry, and a longer thread-level backstop, all sized so the retry itself still finishes inside the outer bound. Compare that to a dependency where hitting the tail is a legitimate, expected outcome — "cache miss with lazy generation" is Netflix's own example — and the configuration inverts: "the network timeout will be set higher than it, such as at 325 ms with 0 or 1 retries and the thread timeout set higher (350 ms+)"4. The lesson generalizes past Netflix's specific numbers: a network timeout should be set from the dependency's own latency distribution (Module 11 §3.1's percentile heuristic), a retry should only be attempted if the round trip plus the retry still fits under the thread-level timeout, and the thread-level timeout is the one number the calling application actually controls regardless of how the dependency's own client library is configured — "the timeouts at the DependencyCommand threading level are the first line of defense regardless of how the underlying dependency client is configured or behaving"4.
2.3 The fallback hierarchy: cache, eventual consistency, stubbed data, fail silent
A breaker that fails fast still has to return something. Netflix's DependencyCommand routes every failure — "timeout, rejection, short-circuited etc" — through the same fallback step regardless of which specific mechanism caught it, "to give a DependencyCommand the opportunity to do something other than 'fail fast'"4, mirroring the unified fallback path Module 11 §4 already describes for Hystrix's later, public version. What is new here is Netflix's own explicit ranking of what a fallback should try, ordered by impact on the user experience from least to most degraded4:
| Fallback | Mechanism | What the user sees |
|---|---|---|
| Cache | Retrieve from a local or remote cache, even if the data ends up stale | Near-normal result, possibly a little out of date |
| Eventual consistency | Queue the write (e.g. via SQS) to persist once the dependency recovers | Action appears to succeed immediately; write lands later |
| Stubbed data | Revert to a default, non-personalized value | Generic result in place of a personalized one |
| Empty response ("fail silent") | Return null or an empty list for the UI to ignore | The feature is simply absent from the screen |
The ordering encodes a real design principle, not just a list of options: "our goal is to have the fallbacks deliver responses as close to what the actual dependency would deliver"4, so a stale cached value is preferred over a generic stub, and a generic stub is preferred over showing nothing — but showing nothing is still preferred over blocking the page, or the whole API, on a dependency that has already demonstrated it cannot answer in time. Read against Table 1 in Module 11 (the SLI/SLO/SLA distinction), this hierarchy is what an SLO looks like once it has actually been missed: not a single binary success/failure outcome, but four different degrees of "we did not get the ideal answer," chosen so that the least important features degrade first and checkout, playback, and the rest of the core experience degrade last, if at all.
3. The other constraint on resilience: freshness vs. latency in recommendations
Sections 1 and 2 protect a request that is already committed to running: something has agreed to answer, and the question is how to fail gracefully if a dependency cannot deliver. Netflix's recommendation architecture faces a different version of the same scarcity one step earlier — before any request has committed to an answer, someone has to decide which algorithm is even allowed to try, because the more sophisticated an algorithm is, the longer it usually takes to run, and a recommendation that has not been computed by the time the page needs to render is not a recommendation at all.
3.1 Why one computation mode cannot serve both goals
Netflix's own account of its recommendation architecture states the tension directly: "one of the key issues in a personalization architecture is how to combine and manage online and offline computation in a seamless manner"5. Purely online computation "can respond quickly to events and use the most recent data," but is "subject to an availability and response time Service Level Agreements (SLA)," which "can make it harder to fit complex and computationally costly algorithms in this approach"5 — the same SLA-as-contract framing Module 11 §1.3 builds in the abstract, here appearing as the specific reason an algorithm gets vetoed from the request path, not because it produces a worse answer, but because it cannot produce any answer inside the time budget. Purely offline computation inverts every one of those constraints: it "allows for more choices in algorithmic approach... and less limitations on the amount of data that is used," and if a new experimental algorithm runs slower, the fix is simply to "deploy more Amazon EC2 instances to achieve the throughput required," rather than spend engineering time optimizing it5. The price of that freedom is staleness: offline computation, by construction, "does not have strong latency requirements," so "it will not react quickly to changes in context or new data," which "can lead to staleness that may degrade the member experience"5. Neither mode dominates the other — one buys speed and freshness at the cost of algorithmic complexity, the other buys complexity and cheap experimentation at the cost of staleness — which is exactly why Netflix runs a third mode between them rather than choosing one. Netflix's own examples make the contrast concrete rather than abstract: an online example is "a gallery of action movies sorted for the member using the current context," recomputed against whatever the member is doing right now; an offline example is periodically aggregating "statistics from millions of movie play events to compile baseline popularity metrics"5 — a computation over a dataset far too large to touch inside a request, and one where nothing about "current context" matters at all.
3.2 Offline, nearline, online
Nearline computation is Netflix's own name for that middle option: "computation is performed exactly like in the online case," but "we remove the requirement to serve results as soon as they are computed and can instead store them, allowing it to be asynchronous"5. It runs in response to individual events rather than on a fixed schedule, which "opens the door for potentially more complex processing to be done per event" than a hard online SLA would allow5 — Netflix's stated example is updating a member's recommendations immediately after they begin watching a movie, rather than waiting for the next scheduled offline batch job to notice5. Because nearline computation writes its result to storage instead of returning it directly to a waiting request, it inherits none of the online tier's response-time SLA, but it reacts far faster than an offline batch job scheduled on a fixed cadence — a genuine third point on the trade-off, not merely an average of the other two.
Netflix's own post is explicit that the three modes are not a menu to choose one from, but layers meant to be combined: "the choice of online/nearline/offline processing is not an either/or question. All approaches can and should be combined"5. The stated combination patterns include using an offline result as the fallback when an online computation cannot meet its SLA5 — precisely the "cache" and "stubbed data" fallback levels from Table 1, applied here to a stale precomputed recommendation list instead of a stale API response — and splitting a single model's training into a large, offline global-model phase plus a lighter, online or nearline per-user update phase, which Netflix's post gives Matrix Factorization and clustering as concrete examples of: some factors precomputed in batch, others updated in real time to keep the result fresh5.
| Tier | Triggered by | Latency bound | Algorithmic freedom | Staleness risk |
|---|---|---|---|---|
| Offline | Scheduled batch job | None — relaxed SLA | Highest — arbitrary complexity, more EC2 instances instead of tuning | Highest — stale until the next batch run |
| Nearline | An individual event (e.g. a play started) | Asynchronous — no hard response deadline | Moderate — richer than online per-event, less than a full batch job | Moderate — reacts per event, not per request |
| Online | The user's request itself | Hard — availability + response-time SLA5 | Lowest — must fit inside the SLA | Lowest — uses the freshest signals available |
3.3 What an algorithm in any tier actually consumes: models, data, and signals
Every tier in Table 2 runs some machine-learning algorithm, and Netflix's own post draws a three-way distinction for what that algorithm reads, independent of which tier it runs in: "models are usually small files of parameters that have been previously trained offline"; "data is previously processed information that has been stored in some sort of database, such as movie metadata or popularity"; and signals are "fresh information we input to algorithms... obtained from live services," such as "what the member has watched recently, or context data such as session, device, date, or time"5. The three-way split maps directly onto the three tiers without being identical to them: a model is almost always the offline tier's output, consumed by all three; data is typically read from whichever result store Section 4.2 lands the offline pipeline's output in; and a signal is available only where a live service can still be reached synchronously, which by definition excludes anything computed further upstream than the online tier itself. This is also precisely why a full historical replay cannot exactly reproduce an online computation's original result even given the same model and the same data — an online decision that consumed a live signal has no equivalent signal available to a job replaying that same time window after the fact, only whatever signal value happened to be logged as data alongside it, if it was logged at all.
4. Connecting the tiers: events, data, Manhattan, and Hermes
Table 2's three tiers only work as one system if something actually moves a result from wherever it was computed to wherever it is needed next — an offline batch job's output has to reach the online serving path, and a nearline computation triggered by one user action has to reach whatever subscriber cares about that result. Netflix names two specific internal systems for this, and draws a line between two kinds of thing that flow through them.
4.1 Events vs. data: two distribution paths
Netflix's own architecture post draws a working distinction, admitted to be "blurry" at the edges, between events and data: "we think of events as small units of time-sensitive information that need to be processed with the least amount of latency possible... routed to trigger a subsequent action or process, such as updating a nearline result set," versus data, "more dense information units that might need to be processed and stored for later use," where "latency is not as important as the information quality and quantity"5. A single user action — starting to watch a movie — can be both at once, "sent to both flows"5: a low-latency event that should immediately nudge a nearline computation, and a data point that also belongs in the large aggregate dataset an offline job will eventually process. This split matters because it means Netflix does not route every user interaction through one pipeline sized for the strictest requirement — the event path is built to be fast and can afford to be lossier or less complete, and the data path is built to be complete and can afford to be slower.
4.2 Manhattan and Hermes
Manhattan is Netflix's name for the system that carries the event path: "our near-real-time event flow is managed through an internal framework called Manhattan. Manhattan is a distributed computation system that is central to our algorithmic architecture for recommendation. It is somewhat similar to Twitter's Storm, but it addresses different concerns and responds to a different set of internal requirements"5. Netflix's post does not spell out exactly which concerns differ — the comparison is offered as an orientation for a reader who already knows Storm's stream-processing model, not as a full specification — but the placement is precise: Manhattan is the mechanism that turns "a movie has been watched" into an actual nearline recomputation of that member's recommendations, the concrete implementation behind the nearline row of Table 2.
The data path runs differently. Offline jobs process refined data "usually generated by running a database query," and because these queries run over large volumes, Netflix runs them "in a distributed fashion... on Hadoop via either Hive or Pig jobs"5, fed in turn by event and interaction logs collected "through Chukwa to Hadoop for the initial steps of the process"5. Once a batch query finishes, something has to tell every interested consumer that a fresh result exists, and to do that reliably Netflix built Hermes: "an internal tool named Hermes that provides all of these capabilities and integrates them into a coherent publish-subscribe framework. It allows data to be delivered to subscribers in near real-time"5. Hermes's own stated requirements are worth reading as a short spec for what a notification layer sitting downstream of a batch system needs: notify subscribers when a query result is ready, support multiple backing repositories rather than one ("not only HDFS, but also S3 or Cassandra"), and handle errors, monitoring, and alerting transparently5. Netflix's own comparison for Hermes is as pointed as the Manhattan-to-Storm one, and cuts the opposite way: "in some sense, it covers some of the same use cases as Apache Kafka, but it is not a message/event queue system"5 — Module 8's log-structured broker model is explicitly not what Hermes is; Hermes is a notification layer over a query result, not a durable, offset-addressed stream of individual messages.
The choice of result store tracks the same trade-offs Module 4 already built for the general relational-vs-NoSQL split, applied to this specific problem: Netflix states plainly that "MySQL allows for storage of structured relational data... [but] the generality comes at the cost of scalability issues in distributed environments," while "Cassandra and EVCache both offer the advantages of key-value stores," with EVCache preferred specifically "in cases where we need intensive and constant write operations"5. None of the three is universally right — the post's own summary is that "the key issue... is not so much where to store them as to how to handle the requirements in a way that conflicting goals such as query complexity, read/write latency, and transactional consistency meet at an optimal point for each use case"5, which is Module 5's PACELC framing in different words: a store is chosen for what it gives up, not for what it claims to guarantee.
Where experts disagree
Netflix's offline/nearline/online split is, structurally, the same shape as the Lambda Architecture that Nathan Marz popularized for general big-data systems: a batch layer for completeness and reprocessing, a fast layer for freshness, and a serving layer that merges both. Section 3 treats that three-way split as settled, because that is how Netflix itself describes its own system. It is not settled in the wider stream-processing literature — the most direct challenge to the entire shape comes from inside the same company whose messaging model (Module 8) this course already leans on.
Keep the batch and speed layers separate, and add an online tier on top
Netflix's own recommendation architecture, as published5
Netflix's stated reason for three tiers instead of one is that offline and online computation are subject to genuinely different constraints, not just different latencies: offline has no SLA and can absorb arbitrary algorithmic complexity by adding EC2 instances, while online is contractually bound to respond inside a fixed window regardless of how much compute is thrown at it5. Nearline exists because neither extreme serves a per-event update well: too slow for a batch job's cadence, too complex for a hard-SLA request path. Every tier is kept because Netflix has evidence that collapsing any two of them costs something specific — algorithmic freedom, freshness, or a response-time guarantee.
- Each tier's constraints are independently real and independently documented — the response-time SLA is not fungible with the batch layer's throughput
- Fallback composition across tiers (Section 3.2) reuses the same graceful-degradation pattern as Table 1's dependency fallbacks, giving the whole architecture one consistent failure-handling idea
- Lets a team keep two very different engineering cultures — throughput-oriented batch and latency-oriented serving — cleanly separated by system boundary
- Three tiers mean three codebases (or at least three deployment and monitoring surfaces) that all have to agree, eventually, on what "the recommendation" is
- The nearline tier is where correctness bugs are hardest to see, because it is neither request-synchronous (so you notice immediately) nor a scheduled batch job (so you can diff against a known-good historical run)
Replace the batch layer with a replayable stream and one codebase
Jay Kreps, co-creator of Apache Kafka, arguing for the Kappa Architecture6
Kreps's objection is not to freshness-versus-completeness as a real trade-off — it is to solving that trade-off with two separate systems that are each supposed to compute the same thing: "the problem with the Lambda Architecture is that maintaining code that needs to produce the same result in two complex distributed systems is exactly as painful as it seems like it would be," and he adds that the resulting operational complexity "is the one thing that seems to be universally agreed on by everyone doing it"6. His proposed alternative, the Kappa Architecture, keeps only a stream-processing layer backed by a log with enough retention to replay history: "when you want to do the reprocessing, start a second instance of your stream processing job that starts processing from the beginning of the retained data"6, and crucially, "unlike the Lambda Architecture, in this approach you only do reprocessing when your processing code changes"6 — not on every fixed batch cadence. His underlying claim is that a mature stream processor does not need a separate batch system bolted on to get completeness: "stream processing systems already have a notion of parallelism; why not just handle reprocessing by increasing the parallelism and replaying history very, very fast?"6
- One codebase computing one answer removes the entire class of bug where batch and streaming versions of the same feature quietly diverge
- Reprocessing is triggered only by a code change, not on every scheduled batch run, which is strictly less wasted computation when nothing has changed
- A sufficiently parallel replay can reprocess historical data fast enough that "batch" becomes an implementation detail of the streaming engine rather than a separate system
- Requires a log with enough retention to hold the full reprocessing window, which for Netflix's multi-year viewing history is a materially larger and more expensive retained log than a typical Kafka retention window
- Says nothing about the third, request-time-bound tier: a Kappa-style stream still has to hand its result to something that answers within an online SLA, which is a different problem than unifying batch and streaming
- Assumes the same processing logic is genuinely reusable across a live event and a full historical replay, which is not automatic when the live path also depends on request-time signals (Section 3.3) that do not exist in historical replay
The two positions are arguing about different halves of Netflix's own triangle. Kreps's critique lands squarely on the offline-versus-nearline boundary: both are, in effect, computation over the same events at different latencies, and if the nearline path is already a stream processor (Manhattan), the case for also maintaining a wholly separate Hadoop/Hive batch pipeline computing an overlapping result is exactly the duplicated-logic problem he describes. It does not touch the online tier at all, because the online tier's constraint is not staleness-versus-completeness but a contractual response-time SLA enforced against whatever request arrives right now (Module 11 §1.3) — no amount of stream-processing unification changes the fact that something has to answer inside a fixed budget or fall back. Read this way, Kreps is right that Netflix's own account gives no principled reason the offline and nearline tiers must be two different systems rather than one stream processor doing both incremental and replayed computation — and Netflix's post itself, published a year before Kreps's, already hedges here, presenting the three tiers as points on a spectrum to be "combined" rather than as three necessarily separate architectures. What would settle it: whether Manhattan (or its modern successor) can replay historical events fast enough, and cheaply enough, to match what the Hive/Pig batch jobs compute — a comparison neither publicly available Netflix source in this map states.
References
- officialprinciplesofchaos.org (Netflix Chaos Team and community). Principles of Chaos Engineering. v1, 2019. https://principlesofchaos.org/. Accessed 2026-09-17. Used for: the definition of chaos engineering, the steady-state-hypothesis baseline method, and the four advanced principles (real-world events, production-traffic experiments, continuous automation, blast-radius minimization).
- blogIzrailevsky, Y., Tseitlin, A. (Netflix). The Netflix Simian Army. Netflix Technology Blog, 19 Jul 2011. https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116. Accessed 2026-09-17 (retrieved via an archived mirror of the Netflix Technology Blog's Medium-hosted publication after the live page returned an automated-access block). Used for: Chaos Monkey's original description and motivation, and the Latency Monkey, Conformity Monkey, Doctor Monkey, Janitor Monkey, Security Monkey, 10-18 Monkey, and Chaos Gorilla tools.
- repoNetflix. Hystrix Wiki: How it Works. v1.5.18. https://github.com/Netflix/Hystrix/wiki/How-it-Works. Accessed 2026-09-17. Used for: pointing to the general circuit-breaker/bulkhead state machine this module applies rather than re-derives (built out fully in Module 11 §4-5).
- blogChristensen, B. (Netflix). Fault Tolerance in a High Volume, Distributed System. Netflix Technology Blog, 29 Feb 2012. https://netflixtechblog.com/fault-tolerance-in-a-high-volume-distributed-system-91ab4faae74a. Accessed 2026-09-17 (retrieved via an archived mirror of the Netflix Technology Blog's Medium-hosted publication after the live page returned an automated-access block). Used for: the Netflix API's traffic scale (1B+ calls/day, 1:6 fan-out, 100k req/s peaks), the 30-dependency compounding-uptime arithmetic, the DependencyCommand thread-pool/semaphore design, the 50%-error/10-second trip condition, thread-pool sizing (10 threads, typically 1-2 active), and the ranked fallback hierarchy (cache, eventual consistency, stubbed data, fail silent).
- blogAmatriain, X., Basilico, J. (Netflix). System Architectures for Personalization and Recommendation. Netflix Technology Blog, 27 Mar 2013. https://netflixtechblog.com/system-architectures-for-personalization-and-recommendation-e081aa94b5d8. Accessed 2026-09-17 (retrieved via an archived mirror of the Netflix Technology Blog's Medium-hosted publication for full article text). Used for: the offline/nearline/online definitions and trade-offs, the events-vs-data distinction, Manhattan (compared to Storm), Hermes (compared to Kafka), Chukwa/Hadoop/Hive/Pig as the batch pipeline, and Cassandra/EVCache/MySQL as result stores.
- blogKreps, J. Questioning the Lambda Architecture. O'Reilly Radar, 2 Jul 2014. https://www.oreilly.com/radar/questioning-the-lambda-architecture/. Accessed 2026-09-17. Used for: the Kappa Architecture critique of maintaining separate batch and streaming codebases, in "Where experts disagree."