System Design · Module 39

Where the Field Still Disagrees and What Is Changing: Consistency, Multi-Tenancy, Async RL, and the Next Edition

This is the last module of a 39-module course, and it introduces nothing new. It gathers four disagreements this course already documented — strong versus eventual consistency, dedicated versus shared GPU multi-tenancy, synchronous versus asynchronous RL trainer/rollout coupling, and pessimistic versus optimistic rate-limit enforcement — sets them side by side, and asks the same question of each: is this still open because the field hasn't finished arguing, or because the question itself has no single answer. It closes by treating the course's own foundational textbook as a data point, not an authority: Designing Data-Intensive Applications shipped a second edition while this course was being written, which is exactly the kind of fact a "settled" claim needs to survive.

Course 30 min read 16 sources As of 2026-09-18 Prerequisite: Module 5 Prerequisite: Module 31 Prerequisite: Module 38
The course's five parts as an arc, with the four recapped forks anchored at the module where each first appeared A. Foundations (1-11) B. Case studies (12-20) C. Inference platform (21-31) D. Training infra (32-38) E. This module Fork 1: Consistency Spanner vs. Dynamo Modules 3, 5 Fork 2: Multi-tenancy Dedicated vs. shared Module 31 Fork 4: Rate limits Estimate vs. reconcile Modules 22, 24 Fork 3: Trainer/rollout Sync vs. async RL Modules 37, 38 All four converge here — none of the four is resolved by this module; each is only restated
Figure 1. The four forks this module recaps, placed at the point in the course where each was first raised. Fork 1 (consistency) surfaces in Modules 3 and 5; Fork 2 (multi-tenancy) in Module 31; Fork 3 (trainer/rollout coupling) in Modules 37 and 38; Fork 4 (rate-limit enforcement) in Modules 22 and 24. Sections 1-4 restate each fork's evidence with citations to the same primary sources those modules used; no new disagreement is introduced here, consistent with this module's own scope.

1. Consistency: strong synchronous replication vs. eventual availability

Module 3 introduced Bigtable and Dynamo as flagship examples of two opposite bets, and Module 5 formalized the trade-off as PACELC. Neither module resolved it, because the two systems were not answering the same question wrong — they were answering different questions correctly for different workloads, and that is the reason this fork is still open more than fifteen years after both papers shipped.

Spanner's own justification for building two-phase commit over Paxos-replicated groups, synchronized by the TrueTime API, is that Bigtable's eventual, best-effort cross-datacenter replication and single-row-only transaction model generated real pain: application teams built Megastore on top of Bigtable specifically for stronger consistency, and Spanner's authors describe this directly as their reason for existing1. Dynamo's team, working from the opposite side of the same industry, reached the mirror-image conclusion from the same kind of production experience: "experience at Amazon has shown that data stores that provide ACID guarantees tend to have poor availability," so Dynamo "targets the design space of an ‘always writeable’ data store," sacrificing consistency under certain failure scenarios rather than ever rejecting a write2. Both teams had access to the same distributed-systems literature. Both concluded the other side's default was wrong for their own workload.

PACELC, as Daniel Abadi's original framing states it, makes the shape of the disagreement precise rather than settling it: "if there is a partition (P) how does the system tradeoff between availability and consistency (A and C); else (E) when the system is running as normal in the absence of partitions, how does the system tradeoff between latency (L) and consistency (C)"3. Spanner answers both questions the same way (favor consistency, pay the cost every time); Dynamo answers both the other way (favor availability and latency, pay the reconciliation cost only on conflict). A third quadrant — PNUTS-style systems that sacrifice availability during a partition but still trade consistency for latency day-to-day — exists in principle and is documented in Module 5, but is rare in practice, mostly because record-level mastership is harder to operate than either pure extreme.

Module 5's own verdict is the reason this module treats the fork as still open rather than merely under-researched: "the deciding question is not ‘how much scale’ but ‘does this data have exactly one natural owner most of the time, and does a merge function exist for the cases where it doesn't.’" A funds transfer has no acceptable merge function and no natural single owner during a partition — Spanner's cost is cheap insurance there. A shopping cart has an obvious merge (union the items) — Dynamo's cost of paying for global consensus on every write would be waste with no offsetting benefit. Neither system is more "correct" in the abstract; each is correct for a data shape the other one handles badly. That is a structural property of the question, not a temporary gap in benchmarking, which is why no single future paper is likely to close it the way Raft's understandability study (Module 6) settled its narrower question. A workload-by-workload audit — does this table's conflicts have a well-defined merge, a well-defined single owner, or neither — is the only thing that has ever moved a specific system's classification, and that audit has to be redone per table, not once per industry.

2. Multi-tenancy: dedicated GPU clusters vs. shared pools with per-key budgets

Module 31 built this fork around the same structural question Section 1 just recapped, but at a different layer: does a piece of infrastructure have exactly one owner (a tenant's own hardware) or does it get shared and reconciled through software (a rate-limited pool)? The evidence, unusually for this course, points one way harder than the framing suggests it should.

Together AI's Dedicated Endpoints and Fireworks AI's claimed architecture both promise structural, hardware-level isolation — "single-tenancy, ensuring your traffic is never impacted by other users," in the vendor-confirmed case, and a similar claim for Fireworks relayed only through a third party and explicitly flagged as unverified rather than confirmed in Module 314. LiteLLM and Envoy AI Gateway instead push isolation into a rate limiter that runs before a request ever reaches a GPU: a virtual-key budget and RPM/TPM check56. Google's own Borg paper ran the counterfactual experiment neither commercial vendor has published for its own fleet: segregating tenants into private cells instead of sharing one costs "2-16× as many cells, and 20-150% additional machines," while a fleet-wide comparison of shared versus dedicated cells found CPU performance only "about 3% worse in shared cells"7. A large, certain utilization cost against a small, noisy interference cost is why Google runs 98% of its shared-cell machines mixed, and it is the strongest evidence available anywhere in this course's source map for either side of this specific fork.

What keeps the fork open despite that lopsided evidence is that Borg's numbers describe a general-purpose CPU cluster, not a GPU-bound, continuously-batched LLM-inference pool, and no inference vendor has published the equivalent interference measurement for its own hardware. Module 31's own verdict names exactly this gap: "what would settle this more precisely... is a published interference measurement from an actual multi-tenant LLM-inference platform... comparing a shared vLLM/SGLang-style continuous-batching pool against a dedicated one under realistic bursty tenant traffic. Neither Together AI's nor Fireworks' public material contains that measurement." That is a research gap, and research could close it. But it is not the only thing keeping the fork open — the other half is economic, and economics, not a new isolation mechanism, is what is actually likely to move it.

2.1 Worked extension: why cost curves, not a new architecture, settle this one

Module 31 derived a breakeven throughput for choosing dedicated hardware over a shared pool from two numbers a vendor already publishes: a dedicated hourly rate \(H\) and an implied shared per-token price \(c\), giving

$$ T_{\text{breakeven}} = \frac{H}{60c} $$

in tokens per minute — the sustained throughput above which a tenant's dedicated GPU-hour bill is cheaper than the same traffic priced per token on a shared pool. Nothing about that formula depends on which architecture is intrinsically better; it depends only on the current values of \(H\) and \(c\), and both are prices, not physical constants. If dedicated hourly rates \(H\) fall — through newer GPU generations, denser packing, or plain price competition, all of which are ordinary hardware-market dynamics rather than a change in the isolation mechanism itself — the breakeven throughput falls in direct proportion, and dedicated hardware becomes the cheaper choice for tenants with progressively smaller, less bursty workloads. If the shared-pool's effective per-token price \(c\) falls at a comparable or faster rate — through better continuous-batching utilization, cheaper serving engines, or the same hardware-market dynamics reaching the shared fleet first — the ratio holds still or moves the other way, and dedicated stays a niche choice for only the largest, most sustained tenants. Together AI's own repricing history is the concrete instance of the first half of this argument: the company's 2025-03-13 update cut Dedicated Endpoint prices by "up to 43%" in one update4 — a swing large enough, on Module 31's own formula, to move a specific model's breakeven threshold by a comparable fraction without anyone touching the underlying multi-tenancy mechanism at all.

This is the sense in which this fork is the sharpest of the four: it is not waiting on an architecture paper the way the trainer/rollout fork in Section 3 might be. Both positions already agree on the mechanism trade-off — Borg's numbers are not in dispute, and no one is claiming a third isolation design that beats both. What is genuinely unresolved is a pair of prices that move independently, on their own market schedules, and the fork's practical answer for any specific tenant changes every time either price moves — which, on Together AI's own evidence, happens on the order of months, not years.

3. Trainer/rollout coupling: synchronous vs. asynchronous RL infrastructure

Modules 37 and 38 built this fork from the same underlying fact: RL post-training's rollout-generation step is an inference workload wearing a training hat, and it dominates wall-clock time — a 32B model generating 32,768 tokens per rollout across 512 sequences at roughly 1,200 tokens/second on a single H100 takes on the order of 3.7 hours, against a training step that is far shorter8. Every framework in the ecosystem is answering the same question — how much can generation and training overlap before a stale sample corrupts the policy update — and a survey spanning sixteen of them found no single converged answer, only three orthogonal knobs8.

verl, the open-source name for the HybridFlow architecture, represents the more tightly coupled end: single-controller coordination between nodes for flexible cross-node data resharding, multi-controller execution within a node for efficiency, and a 3D-HybridEngine that reshapes actor-model parameters between training and generation phases with zero memory redundancy9. AReaL represents the fully decoupled end: version 2.0 refactored into independent training, inference, agent, and weight-update microservices, with staleness bounded by a single tunable parameter, max_head_offpolicyness, that recovers a synchronous algorithm exactly at zero and reports a measured 2.77× speedup over synchronous systems at comparable or better final performance10. Both are production systems with published throughput numbers. Neither has driven the other out of use.

The wider sixteen-framework survey found the ecosystem is not converging toward either pole; it is fragmenting across independent orchestration substrates entirely: eight frameworks build on Ray, four use native Python asyncio/threading, one (Atropos) uses HTTP microservices, and one (TorchForge) uses the Monarch actor model8. Weight-synchronization latency spans a comparable range for the same underlying reason — standard NCCL broadcast at roughly 100-500ms, bucketed variants near 20ms, and PipelineRL's outlier design swapping weights between forward passes at a 1-10ms gap without ever interrupting an in-flight sequence8. That spread is itself evidence against convergence: if one design were simply better, the field's fastest-moving, most heavily benchmarked corner — sixteen actively maintained frameworks in roughly a year — would be expected to show it by now.

What makes this fork different from Section 2's is that the three staleness-management categories the survey documents — hard version-rejection, depth-bounded queueing, and importance-sampling correction — are not mutually exclusive positions a framework must pick one of. AReaL's own staleness parameter demonstrates this directly: it is a dial, not a switch, with full synchrony and full asynchrony as its two endpoints and every intermediate value a legitimate, currently-used operating point10. A fork framed as "sync vs. async" is, on the ecosystem's own evidence, better described as a continuum multiple frameworks are independently exploring at different points, which is a structurally different kind of disagreement from Section 1's workload-conditional split or Section 2's two-price economic question — this one may simply not resolve into two camps at all.

The practical consequence for anyone choosing among these frameworks today, rather than debating the fork in the abstract, is that the first decision is not "sync or async" at all — it is which orchestration substrate to commit to, because that choice constrains which staleness knob is even available. Committing to a Ray-based framework (verl, SkyRL, NeMo-RL, and five others of the sixteen surveyed) buys the operational tooling and cluster-scheduling lineage Module 34 already covers, at the cost of Ray's own actor-scheduling overhead sitting between every generation and training step. Committing to a native-async design (AReaL, PipelineRL, ART) buys the tighter weight-sync latencies Section 3's own numbers show, at the cost of building cluster orchestration the framework does not get from Ray for free8. Neither the HTTP-microservice path (Atropos) nor the actor-model path (TorchForge) has enough production deployment history in this survey to compare against either mainstream option on equal footing. That is a genuinely unresolved engineering trade-off, not a preference, and it is a large part of why sixteen teams built sixteen different answers instead of converging on Ray's or asyncio's terms alone.

4. Rate-limit enforcement: pessimistic estimate-and-adjust vs. optimistic admit-and-reconcile

Modules 22 and 24 raised this fork from a specific problem general rate limiting (Module 10) does not have: an LLM request's true resource cost — the number of tokens it will actually generate — is unknown at the moment a platform has to decide whether to admit it. Two production systems in this course's source map answer that problem in opposite directions.

Anthropic's Messages API splits the estimate by direction and re-measures continuously: input-tokens-per-minute "is estimated at the beginning of each request, and the estimate is adjusted during the request to reflect the actual number of input tokens used," while output-tokens-per-minute counts only tokens actually generated, in real time11. Because Anthropic operates the model doing the generating, there is no window in which the enforcement layer is blind to true cost — only a window in which that cost has not finished accumulating. Envoy AI Gateway, sitting in front of arbitrary third-party backends it does not operate, cannot assume that same real-time visibility, and its own documentation states the alternative plainly: it "checks if processing this request would exceed the configured token limit... If within the limit, the request is processed and its token usage is counted towards the total" only afterward6 — admit against the best information available now, reconcile against the truth later.

Module 22's own verdict traces the split to where each system's enforcement point physically sits, not to which philosophy is generally superior: "a lab that operates its own inference stack... has no excuse not to meter output in real time, so estimate-then-adjust dominates there... A gateway that fronts arbitrary third-party backends... genuinely cannot always get that real-time signal, so admit-then-reconcile is the pragmatic fallback." That verdict also names the specific measurement that would move this fork past architecture-tracing and into evidence: "how large is the actual over-admission rate Envoy-style optimistic admission produces under realistic concurrent bursty traffic, measured against Anthropic-style real-time metering on the identical workload. No source in this course's map publishes that comparison." As of this course's as-of date, that comparison still does not exist in public form, for either side.

This fork resembles Section 2's more than Section 1's or Section 3's: it is not workload-conditional (a given platform's architecture, not its traffic mix, decides which side it can even choose) and it is not a continuum multiple designs are simultaneously exploring (a gateway either has real-time backend visibility or it does not). It is closer to a product-architecture constraint dressed up as a philosophy — which means, like Section 2's economics, it is more likely to be resolved by infrastructure convergence (more gateways gaining real-time backend telemetry, the way Module 25's KV-cache-utilization and queue-depth signals already give a platform visibility it didn't have before) than by either side conceding the other's approach is wrong in principle.

A platform team building a new gateway today, rather than choosing between two already-shipped products, does not have to pick one philosophy outright. The two are composable rather than mutually exclusive: a hard per-request ceiling on max_tokens, checked pessimistically at admission the way Anthropic's ITPM path checks input tokens, bounds the worst case an optimistic admit-and-reconcile design can ever be wrong by; the optimistic path then only has to reconcile the gap between that ceiling and the request's actual output length, not an unbounded one. That combination is not documented as a named pattern in either Anthropic's or Envoy's own material — it follows from applying Module 22's own verdict literally, pairing the cheap, universal check with the precise, model-colocated one wherever the second is available, rather than treating the fork as a single either-or choice a platform has to fully commit to on day one.

5. The four forks side by side

Read together, Sections 1-4 are not four instances of the same phenomenon — the four forks stay open for four different structural reasons, and confusing those reasons is the fastest way to misjudge which ones a future paper, a future gateway release, or a future GPU generation might actually close.

Table 1. The four forks this course could not resolve, why each stays open, and what specifically would move it, per the source modules' own verdicts12746811.
ForkPosition APosition BWhy it stays openWhat would move it
1. Consistency (Modules 3, 5)Strong, synchronous replication (Spanner)1Always-writeable eventual consistency (Dynamo)2Workload-conditional by nature — different data shapes have different correct answersA per-table audit (merge function? natural owner?), not a new benchmark
2. Multi-tenancy (Module 31)Dedicated per-tenant GPU clusters4Shared pool + per-key budget56A pair of independently-moving prices (\(H\), \(c\)), plus a missing GPU-specific interference measurementCost-curve convergence, or a published LLM-inference-specific CPI-equivalent7
3. Trainer/rollout coupling (Modules 37, 38)Fully async, microservice-decoupled (AReaL)10Ray-orchestrated, resharded coupling (verl)9A continuum, not two camps — 16 frameworks span the space independently8Ecosystem consolidation, if it happens at all — no sign of it as of this survey
4. Rate-limit enforcement (Modules 22, 24)Estimate pessimistically, adjust live (Anthropic)11Admit optimistically, reconcile after (Envoy)6Determined by whether the enforcer co-locates with the model — a constraint, not a preferenceGateways gaining real-time backend token telemetry
Four fork boxes, each annotated with what would settle it, drawn from each source module's own stated verdict Fork 1: Consistency Spanner (PC/EC) vs. Dynamo (PA/EL) Settled per-table, never per-industry: "merge function or natural owner?" — Module 5's verdict Fork 2: Multi-tenancy Dedicated vs. shared + per-key budget A two-price economic question, not an architecture question — Section 2.1 plus a missing GPU interference number Fork 3: Trainer/rollout coupling AReaL (async) vs. verl (coupled) 16 frameworks, no convergence in sight — a continuum, not a binary choice — Hugging Face's own survey conclusion Fork 4: Rate-limit enforcement Anthropic (estimate) vs. Envoy (reconcile) Tracks enforcement-point location, not philosophy — a measured over-admission rate would still sharpen it — Module 22's verdict
Figure 2. The four forks, restated with each source module's own account of what would settle it — Module 5 for consistency, Module 31 §4.2 for multi-tenancy, the sixteen-framework survey8 for trainer/rollout coupling, and Module 22 for rate-limit enforcement.

6. DDIA's second edition: the foundational text is still being rewritten

Part A of this course leaned on Designing Data-Intensive Applications as the synthesizing textbook for replication, partitioning, consistency, and batch/stream processing12. That choice deserves the same scrutiny this module has applied to everything else, because the book itself is not a fixed reference: Martin Kleppmann published the first edition through O'Reilly on 2017-03-2713, and a second edition, co-authored with Chris Riccomini, followed on 2026-03-24 — 670 pages, ISBN 9781098119065 (paperback) and 9781098119027 (ebook)14. Nine years is not a long time for a textbook to need a second edition; it is a short one, and it is worth asking what changed in nine years that a book built explicitly around "principles that endure" felt it had to add.

The Pragmatic Engineer's Gergely Orosz obtained an advance excerpt directly from Kleppmann and published it alongside his own account of what the second edition covers that the first did not15. The changes cluster around exactly the technologies that did not exist, or did not matter, in 2017: a new opening chapter, "Trade-Offs in Data Systems Architecture," that treats cloud-versus-self-hosting as the first decision a systems designer now makes rather than an implementation detail; explicit coverage of vector databases and DataFrames as the data-systems substrate underneath production AI systems; local-first software, an area Kleppmann has worked in directly since the first edition; formal methods, motivated by Orosz's own framing that "the advent of AI-generated code means this topic is getting more attention industry-wide"; and regulation and legal context, specifically the EU's General Data Protection Regulation, treated as material a systems engineer now needs rather than a legal department's problem15. The book closes with an entirely new chapter, "Doing the Right Thing," on the societal consequences of the systems this course has spent thirty-eight modules describing how to build15.

What receded is as informative as what was added. In a podcast conversation with Orosz, Kleppmann named a specific casualty directly: "topics that have become less relevant over time, like details on MapReduce"15. Module 9 of this course covers MapReduce's map/reduce type contract and failure-recovery asymmetry in full, exactly as the field's own foundational textbook has now decided to de-emphasize it in its most recent edition. That is not a contradiction this course needs to resolve — Module 9's own framing already treats MapReduce as a historical lineage point (the ancestor Spark's lineage-based fault tolerance replaced for interactive and iterative workloads) rather than a live design recommendation — but it is a clean, small-scale instance of the exact phenomenon this entire module is about: a technology that mattered enough to anchor a chapter in 2017 has, by the field's own most authoritative synthesizer's own account, mattered less every year since.

Table 2. What DDIA's second edition added and de-emphasized, relative to the first, per the publisher's own edition metadata and the author-sourced excerpt/commentary1215.
Area1st edition (2017)2nd edition (2026)
Deployment modelSelf-hosted / on-premises framing defaultNew Ch. 1, "Trade-Offs in Data Systems Architecture" — cloud vs. self-hosting as the first decision
AI-adjacent data systemsNot covered — predates the current LLM/RAG stackVector databases, DataFrames for training data, batch processing of large training corpora
Local-first softwareNot coveredNew coverage, an area the author works in directly outside the book
Formal methodsNot coveredAdded, motivated explicitly by AI-generated code increasing the need for it
Regulation / legal contextNot coveredGDPR and adjacent regulation, framed as engineer-relevant knowledge
Closing chapterEnds on stream processing / the future of data systemsNew closing chapter, "Doing the Right Thing" — engineering ethics
Batch processing depthDetailed MapReduce treatmentDe-emphasized — the author's own stated reason: "less relevant over time"
Timeline from DDIA's first edition to its second, with what changed annotated at each end 2017-03-27 1st edition Kleppmann, O'Reilly 2026-03-24 2nd edition Kleppmann & Riccomini, 670pp ~9 years De-emphasized: MapReduce depth "less relevant over time" — the author's own words Added: cloud framing, AI-adjacent systems, local-first, formal methods, GDPR, ethics
Figure 3. Nine years between editions of the field's own synthesizing textbook1314, with the specific additions and the one named removal per the author-sourced excerpt15. Note: this figure uses the dates confirmed directly from Kleppmann's own publication metadata rather than the course's earlier source-map estimate (February 2026, 650 pages), which was self-flagged there as unfetched and inferred; the verified figures are March 2026 and 670 pages.

7. What's moving: a matrix across this course

Placing every module this course covered against two axes — which part of the stack it belongs to, and whether its central mechanism is settled or actively contested as of this course's as-of date — makes a pattern visible that no single module could show on its own.

Classical, inference-platform, and training-infra modules placed against settled vs. actively contested Settled (broad practitioner agreement) Actively contested Classical Modules 1-20 Replication mechanics, consensus, partitioning, indexing (Modules 3, 4, 6, 7) how to build either side is settled Which consistency model a new system should default to (Fork 1) Modules 3, 5 Inference platform Modules 21-31 Gateway request lifecycle, idempotent billing, streaming protocol (Modules 21, 23, 26) reapplications of Part A primitives Multi-tenancy isolation (Fork 2), rate-limit philosophy (Fork 4) Modules 31, 22, 24 Training infra Modules 32-38 Checkpoint/restart, storage fabric, Borg/K8s scheduling lineage (Modules 33-35) Trainer/rollout coupling (Fork 3) Modules 37, 38 — least converged of all
Figure 4. This course's own thirty-eight preceding modules, placed by part (rows) and by whether their central mechanism has broad practitioner agreement or is an open fork (columns). The four recapped forks each land in the contested column; nothing else in the course does, which is itself the finding — everything settled in this course settled because it reduces to a mechanism Part A already closed out, and everything still open involves a genuine two-sided trade a mechanism alone cannot resolve.

The pattern in Figure 4 is not an accident of which topics this course happened to choose. Classical storage's replication and consensus mechanics (Modules 3, 4, 6, 7) are settled because the field spent decades and multiple independently-converging systems (Raft, Paxos, Chubby, ZooKeeper) arriving at compatible answers to "how do machines agree despite failure" — Module 6's own disagreement, Raft versus Paxos, is about which implementation is easier to teach, not about whether consensus itself works, and that narrower question is exactly the kind a controlled user study8 can settle. Fork 1 sits one level up from that mechanism and stays open because it is not a mechanism question at all — it is a data-modeling question the mechanism cannot answer for you.

Most of Part C's inference-platform mechanics — the gateway request lifecycle (Module 21), idempotent billing (Module 23), the SSE streaming protocol (Module 26) — are settled for a specific reason worth stating plainly: they are almost entirely reapplications of primitives Part A had already closed out (idempotency keys from Module 8's exactly-once semantics, backpressure from Module 10, observability from Module 11) to a new domain, not new open questions. The two genuinely contested corners of Part C, multi-tenancy and rate-limit philosophy, are exactly the two places this course found a fork rather than a settled answer — which is evidence, not coincidence, that reapplying a settled primitive to a new domain tends to stay settled, while a genuinely new trade-off (a GPU is not fungible the way a CPU core is; true cost is not known until generation finishes) tends to open a new fork even inside an otherwise-mature layer.

Training infrastructure is the newest part of the stack this course covers, and it shows: the scheduling and storage lineage (Modules 33-35) inherited a settled answer wholesale from Borg and Kubernetes, but the layer built specifically for RL post-training — rollout generation coupled to a training loop — has no comparable lineage to inherit from, because nothing quite like it existed before frontier-scale RL post-training did. Sixteen frameworks built independent answers in roughly a year8, which is the signature of a genuinely new problem, not an unsettled old one.

The settled side of that same row is worth grounding as concretely as the contested side, because "settled" should not just mean "this course did not find a disagreement." Meta's own account of Llama 3 pre-training reports 466 job interruptions across 54 days, roughly 78% hardware-attributed, alongside a maintained effective training time above 90% despite that failure rate16 — a concrete, measured outcome from checkpoint/restart and heartbeat-based anomaly detection, the mechanisms Module 35 covers in full. No comparable disagreement exists over whether checkpoint/restart is the right recovery primitive for a training cluster; the open questions there are tuning parameters (checkpoint frequency, heartbeat interval) within an agreed mechanism, not competing mechanisms the way Fork 3's orchestration substrates are. That is the difference between a settled row and a contested one in Figure 4: not the absence of hard problems, but the absence of a live fork over which mechanism to use to solve them.

Where experts disagree

This section does not introduce a fifth disagreement. It restates the four already argued in Sections 1-4, compressed to their essential claim, precisely because the point of this closing module is to show them side by side rather than to relitigate any one of them from scratch.

Fork 1 — Pay for consistency everywhere, unconditionally

Spanner's design team1; Section 1

If an application cannot enumerate every invariant weak consistency might silently violate, the only safe default is to violate none of them, ever, and pay the commit-wait and cross-group consensus cost on every write.

  • One mental model for every read and write, forever.
  • No later audit ever discovers a silently-violated invariant.
  • Unconditional latency and availability cost, even for data with a trivial merge function.
  • Wrong for the workloads (carts, counters, sessions) that dominate most services' request volume.

Fork 1 — Never reject a write; reconcile later

Dynamo's design team2; Section 1

Most operations against most keys have an obvious merge function or no genuine cross-key invariant at all, so paying Spanner's tax on every one of them is waste with no correctness benefit.

  • Meets 99.9th-percentile SLAs by construction — no operation waits on a slow quorum member.
  • Availability-first design costs nothing for genuinely mergeable data.
  • Every genuine cross-key invariant becomes the application's permanent, unassisted responsibility.
  • Wrong for ledgers, unique-registration, and anything else with no acceptable merge.

Fork 2 — Dedicated hardware is worth the utilization cost

Together AI, and Fireworks AI's claimed (unverified) design4; Section 2

Physical isolation makes a noisy-neighbor guarantee structural rather than probabilistic, and for a sufficiently sustained tenant it is also the cheaper option once throughput clears the breakeven point.

  • Isolation is a hardware fact, immune to rate-limiter misconfiguration.
  • Cheaper than shared for tenants above the Section 2.1 breakeven.
  • Borg's own measurement of the same structural choice found 20-150% more machines needed7.
  • Below breakeven, idle GPU-hours are paid for regardless.

Fork 2 — Shared pools with per-key budgets maximize utilization

LiteLLM, Envoy AI Gateway, and Borg's own operating practice567; Section 2

Google runs 98% of its shared-cell machines mixed because segregation costs more hardware than the interference it avoids is worth, and every commercial AI gateway this course documents defaults to the same shape.

  • Only about 3% worse measured CPU performance, fleet-wide, on Borg's own numbers7.
  • No idle reserved hardware; capacity tracks aggregate, not per-tenant peak, demand.
  • Isolation is continuous and software-enforced, not structural.
  • No hardware-level guarantee for a tenant with strict compliance or SLA needs.

Fork 3 — Fully asynchronous, microservice-decoupled RL infra wins

AReaL10, PipelineRL8; Section 3

Decoupling generation and training into independent services, with staleness bounded by a tunable parameter, delivers large measured speedups without sacrificing final model quality.

  • 2.77× measured speedup over synchronous systems at comparable or better performance10.
  • The staleness dial spans sync to async in one framework, not two incompatible ones.
  • Correctness depends on a tuned bound (max_head_offpolicyness) that can be set wrong.
  • Harder to debug than a single coupled dataflow graph.

Fork 3 — Bounded-queue, Ray-orchestrated coupling is simpler to reason about

verl/HybridFlow9, and 7 of 16 surveyed frameworks on Ray8; Section 3

Explicit resharding between training and generation phases, orchestrated through Ray's task/actor primitives, keeps the dataflow a single graph a team can reason about end to end.

  • 3D-HybridEngine resharding eliminates memory redundancy across phase transitions9.
  • One dataflow graph, not four independently-scaled services to keep synchronized.
  • Leaves more of the async speedup on the table than a fully decoupled design.
  • Still coupled enough that a slow generation step can stall the training step behind it.

Fork 4 — Estimate pessimistically, adjust in real time

Anthropic's Messages API11; Section 4

A platform that operates the model doing the generating has no excuse not to meter output as it happens, closing the estimate-vs-truth gap continuously rather than after the fact.

  • No reconciliation step — the limit's live state is always the true state.
  • Composes cleanly with cache-aware token accounting (Module 28).
  • Only works when the enforcer is co-located with, or has real-time visibility into, the model.
  • Sub-window smoothing still under-describes true burst behavior (Module 22 §3.2).

Fork 4 — Admit optimistically, reconcile after the fact

Envoy AI Gateway6; Section 4

A gateway fronting arbitrary third-party backends cannot always get real-time token counts mid-generation, so it checks the best information available at admission time and corrects the ledger afterward.

  • Works across heterogeneous, non-OpenAI-schema backends via response transformers.
  • The pragmatic-fallback choice for any multi-provider router, not just a single-vendor API.
  • Over-admission is proportional to generation length and otherwise unbounded.
  • No published measurement yet quantifies how large the resulting over-admission actually is.
Assessment

These four forks are not one disagreement wearing four costumes, and treating them as if they were is the mistake this closing module is built to prevent. Fork 1 is workload-conditional in a way no future paper closes — it will still be open in another fifteen years, because the right answer changes per table, not per era. Fork 2 and Fork 4 are constraint-conditional: Fork 2 tracks two independently-moving prices and a missing GPU-specific measurement, and Fork 4 tracks whether an enforcement point physically co-locates with the model doing the generating — both are the kind of fork that infrastructure maturity (better shared-pool utilization, gateways gaining real-time backend telemetry) plausibly narrows over the next few years without either side conceding a design error. Fork 3 is the newest and least settled precisely because it has the least lineage to inherit from: sixteen frameworks built sixteen answers in about a year, and nothing in that evidence points toward consolidation yet, only toward a continuum multiple teams are exploring independently. What would move each: a per-table consistency audit for Fork 1 (this will never produce one global answer, only many local ones); a published GPU-specific interference measurement for Fork 2; either ecosystem consolidation or a second survey a year from now showing the same fragmentation for Fork 3; and a published over-admission-rate comparison under real bursty load for Fork 4. None of those four pieces of evidence exists yet in this course's source map, and that absence, not a lack of engineering effort, is the actual reason this module ends without resolving anything.

What is changing

Every specific number cited across these thirty-nine modules is a snapshot, not a specification, and this course has tried to say so at the point each number appeared rather than only here at the end. Together AI's dedicated-endpoint pricing has already moved once by "up to 43%" in a single update4; Envoy AI Gateway's own project renamed itself mid-development6; the Kubernetes Gateway API Inference Extension was, as of Module 24's own writing, still moving fields from an alpha surface to a stabilized v1 one; the sixteen-framework RL survey this module leaned on twice is itself a single dated snapshot of an ecosystem adding and retiring frameworks faster than any textbook chapter could track8. None of that is a flaw in how this course was built. It is the reason every header in every one of its thirty-nine modules carries an explicit as-of date instead of presenting a number as a permanent fact.

The clearest evidence that this discipline is not paranoia is the one artifact this course leaned on for its own foundations. Designing Data-Intensive Applications is not an AI-infrastructure blog post or a vendor's pricing page; it is the standard graduate-level synthesis of exactly the material Part A of this course covers, written by an author whose stated goal was principles that endure past the technologies that illustrate them12. It still needed a second edition nine years after the first, adding cloud-first framing, AI-adjacent data systems, local-first software, formal methods, and regulatory context, and quietly retiring the depth of coverage MapReduce once had131415. If the field's own most careful synthesizer revises his account of what endures every nine years, a course built from primary sources dated 2026 should be read the same way: as the best available account as of a stated date, not as a settled record that stops needing revision the moment it is published.

That is the note this course closes on, not as a hedge but as the actual finding of this module. Four forks — consistency, multi-tenancy, trainer/rollout coupling, rate-limit philosophy — survived thirty-eight modules of primary-source grounding without collapsing into a single answer, and the reasons they survived were four different reasons: one workload-conditional, two constraint-conditional, one still too new to have a lineage. A fifth kind of claim survived alongside them without ever being named a disagreement at all — the assumption that a foundational textbook, once written, stays foundational without revision. It does not. Read every "settled" claim in this course, including this one, as time-stamped.

References

  1. paperCorbett, J.C., Dean, J., Epstein, M., et al. Spanner: Google's Globally-Distributed Database. OSDI '12. https://research.google.com/archive/spanner-osdi2012.pdf. Accessed 2026-09-17. Used for: recap of Spanner's two-phase-commit-over-Paxos design and its own stated reason for existing (§1, §2.3), reused from Modules 3 and 5.
  2. paperDeCandia, G., Hastorun, D., Jampani, M., et al. Dynamo: Amazon's Highly Available Key-value Store. SOSP '07. https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf. Accessed 2026-09-17. Used for: recap of Dynamo's "always writeable" design and its abstract/§2.3 justification, reused from Modules 3 and 5.
  3. blogAbadi, D.J. Problems with CAP, and Yahoo's little known NoSQL system. DBMS Musings, 2010-04-23. http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html. Accessed 2026-09-17. Used for: the PACELC framework's own statement of the partition/else, availability-consistency/latency-consistency split, reused from Module 5.
  4. blogGMI Cloud (summarizing Fireworks AI's public claims); Together AI. On-demand dedicated endpoints and Fireworks AI Deep Dive: Inference at Scale. Together AI published 2025-03-13; GMI Cloud, 2026. https://www.together.ai/blog/on-demand-dedicated-endpoints, https://www.gmicloud.ai/en/blog/fireworks-ai-inference-at-scale. Accessed 2026-09-18. Used for: the dedicated-cluster claims, the "up to 43%" price cut, and the Fireworks claim's unverified flag, reused from Module 31.
  5. officialBerriAI / LiteLLM contributors. Life of a Request (LiteLLM Proxy architecture). docs.litellm.ai, 2026. https://docs.litellm.ai/docs/proxy/architecture. Accessed 2026-09-17. Used for: the virtual-key budget check and rate limiter preceding the router, reused from Module 31.
  6. officialEnvoy AI Gateway project (rebranded "Agent Router" in v0.1 docs). Usage-based Rate Limiting. v0.1 docs, 2026. https://aigateway.envoyproxy.io/docs/0.1/capabilities/usage-based-ratelimiting/. Accessed 2026-09-17. Used for: the check-then-process-then-count admission design, reused from Modules 22 and 31.
  7. paperVerma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., Wilkes, J. Large-scale cluster management at Google with Borg. EuroSys '15. https://research.google.com/pubs/archive/43438.pdf. Accessed 2026-09-17. Used for: the 20-150% cell-segregation-cost figures and the ~3% shared-vs-dedicated CPI comparison (§5.2), reused from Module 31.
  8. blogHugging Face. Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries. hf.co/blog, 2026. https://huggingface.co/blog/async-rl-training-landscape. Accessed 2026-09-17. Used for: the 16-framework orchestration-substrate split, the three staleness-management categories, weight-sync latency spread, and the 3.7-hour generation-bottleneck arithmetic, reused from Modules 37 and 38.
  9. paperSheng, G., Zhang, C., Ye, Z., et al. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256, EuroSys '25. https://arxiv.org/pdf/2409.19256. Accessed 2026-09-17. Used for: verl's single-controller/multi-controller split and 3D-HybridEngine resharding, reused from Module 38.
  10. repoinclusionAI. AReaL README. GitHub, main branch (AReaL 2.0), 2026. https://github.com/inclusionAI/AReaL. Accessed 2026-09-17. Used for: the microservice refactor, max_head_offpolicyness staleness parameter, and the 2.77× measured speedup, reused from Module 38.
  11. officialAnthropic. Rate limits (Claude API). platform.claude.com, 2026. https://platform.claude.com/docs/en/api/rate-limits. Accessed 2026-09-17. Used for: the ITPM estimate-and-adjust and real-time OTPM metering design, reused from Module 22.
  12. textbookKleppmann, M., Riccomini, C. Designing Data-Intensive Applications. O'Reilly Media, 2nd edition, 2026. https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/. Accessed 2026-09-17. Used for: DDIA's role as Part A's synthesizing textbook, reused from the course's source map. Note: this listing's own metadata (February 2026, 650 pages) could not be fetched directly (403) and was self-flagged in the source map as an inferred estimate; refs 13-14 below are the directly-fetched, author-confirmed correction (March 2026, 670 pages).
  13. officialKleppmann, M. Designing Data-Intensive Applications [1st edition]. martin.kleppmann.com, O'Reilly Media, 2017-03-27. https://martin.kleppmann.com/2017/03/27/designing-data-intensive-applications.html. Accessed 2026-09-18. Used for: confirmed first-edition publication date (citation_publication_date meta tag: 2017-03-27), fetched directly to ground the nine-year gap to the second edition.
  14. officialKleppmann, M., Riccomini, C. Designing Data-Intensive Applications, 2nd Edition [author's own publication page]. martin.kleppmann.com, O'Reilly Media, 2026-03-24. https://martin.kleppmann.com/2026/03/24/designing-data-intensive-applications-2e.html. Accessed 2026-09-18. Used for: confirmed second-edition publication date (citation_publication_date meta tag: 2026-03-24), 670-page count, ISBN 9781098119065 (paperback) / 9781098119027 (ebook), and co-author credit; fetched directly, superseding the source map's unfetched estimate (ref-12).
  15. blogOrosz, G. Designing Data-Intensive Applications: The Cloud & Doing the Right Thing (book excerpt, reprinted with publisher permission, plus commentary). The Pragmatic Engineer, 2026-05-05. https://newsletter.pragmaticengineer.com/p/designing-data-intensive-applications-book-excerpt. Accessed 2026-09-18. Used for: the specific list of what the second edition added (cloud/self-hosting framing, vector databases and DataFrames, local-first software, formal methods, GDPR/regulation, the closing "Doing the Right Thing" chapter) and what receded (MapReduce detail, per Kleppmann's own podcast remark quoted in the piece); fetched directly, not in the original source map.
  16. paperMeta (Grattafiori, A. et al.). The Llama 3 Herd of Models. arXiv:2407.21783, 2024. https://arxiv.org/abs/2407.21783. Accessed 2026-09-17. Used for: the 466-interruption/54-day taxonomy and the maintained >90% effective training time, reused from Module 35, grounding the "settled" side of Figure 4's training-infra row.