System Design · Module 35
Fault Tolerance and Elasticity: Checkpoint/Restart, Stragglers, Node Health, and Incident Patterns
A 16,000-GPU synchronous training job failed 466 times in 54 days and still spent over 90% of the elapsed time doing useful work. This module is grounded entirely in the two production accounts that document how: Meta's Llama 3 infrastructure paper's reliability section, and ByteDance's MegaScale paper on the heartbeat, diagnosis, and checkpoint machinery that turns constant hardware failure into a survivable, boring event. It is also, by design, this course's only treatment of incident patterns in AI infrastructure — a coverage decision stated explicitly below, not left implicit.
1. Why synchronous training breaks the classical fault-tolerance playbook
Incident patterns for AI infrastructure have no dedicated module anywhere in this course's source map — no company has published a standalone taxonomy of LLM-serving or LLM-training incidents at the depth of, say, Google's SRE book chapters. This module is where that gap is folded, and it is folded narrowly: everything below traces to what Meta's Llama 3 paper and ByteDance's MegaScale paper actually disclose about their own reliability engineering, not a generic incident-response framework retrofitted onto AI infrastructure. Both are the only public, numbers-bearing accounts of running training at this scale that either company has released.
The reason this needs its own module, rather than a paragraph inside Module 34's scheduling material, is that training breaks an assumption every classical cluster-management system in this course relies on. Module 11's circuit breakers and bulkheads, and Module 34's Borg/Kubernetes lineage, both assume that one task's failure is cheap and containable: a request fails, a retry or a fallback absorbs it; a batch task fails, the scheduler reschedules it on another machine while its thousands of siblings keep running. Borg's own numbers reflect this design point directly — quota is checked once, at admission, not repeatedly at scheduling time, so a resource-starved job is rejected before it ever displaces a running one2, and when Borg does preempt a task to make room for a higher-priority one, it delivers a graceful SIGTERM roughly 80% of the time before resorting to SIGKILL2. The implicit assumption is that any one task's disappearance is a local event.
Synchronous data- and model-parallel pre-training inverts that assumption. Every step ends in a collective — an all-reduce or reduce-scatter across the data-parallel group, a point-to-point handoff across the pipeline stages — and a collective blocks until every participant arrives. Meta states the consequence plainly: "the complexity and potential failure scenarios of 16K GPU training surpass those of much larger CPU clusters that we have operated... the synchronous nature of training makes it less fault-tolerant — a single GPU failure may require a restart of the entire job"1. A cluster an order of magnitude smaller than Google's median 10,000-machine Borg cell2 can be harder to keep running than that far larger cell, because Borg's tasks are independent and training's ranks are not. MegaScale frames the same fact as a design principle rather than a complaint: "failures and stragglers are the norm rather than the exception for LLM training," and a single straggler "not only affects its own work, but slows down the entire job involving tens of thousands of GPUs"3.
Both companies respond by building dedicated machinery rather than reusing what a general-purpose cluster manager already provides. Table 1 lays the two systems side by side; the rest of this module works through each row.
| Dimension | Llama 3 405B pre-training (Meta) | MegaScale (ByteDance) |
|---|---|---|
| Cluster scale | Up to 16,384 of 24,576 H100 GPUs on the RoCE cluster1 | Largest production cluster >10,000 GPUs; benchmark runs on 12,288 GPUs3 |
| Job scheduler | MAST, Meta's global-scale training scheduler1 (named scope only; Module 34) | A custom framework layered on Kubernetes for pod allocation3 |
| Anomaly signal | NCCLX watchdog/heartbeat timeouts feeding PyTorch's NCCL flight recorder1 | Purpose-built heartbeat messages carrying host state, process status, and RDMA traffic metrics3 |
| Diagnosis tooling | Flight-recorder ring buffer of collective metadata/stack traces, dumped on watchdog timeout1 | Self-check test suite (Loopback, RNIC-to-RNIC, NCCL all-to-all/all-reduce) plus a CUDA-event heat-map and a 3D dependency-visualization tool3 |
| Recovery primitive | Checkpoint/restart; per-GPU state 1 MB–4 GB on Tectonic (Module 33)1 | Checkpoint/restart via a two-stage pipeline: GPU→host memory in seconds, async host→HDFS in the background3 |
| Node replacement | Not detailed in the paper beyond automated cluster maintenance | Kubernetes evicts the faulty pod and replenishes with "an equivalent number of healthy ones"3 |
| Effective training time | >90%, with at least one interruption daily1 | >90% effective training-time rate3 |
| Manual intervention | 3 of 419 unexpected interruptions1 | >90% of faults auto-detected/located/recovered3 |
Meta's diagnosis tooling row deserves its own mechanism, since the paper is explicit about why a generic profiler was not enough. Training runs on a mixed NVLink-and-RoCE fabric, and NVLink data transfer happens through load/store instructions issued directly by CUDA kernels rather than through an explicit send/receive call — so a failure in the remote GPU or in NVLink connectivity itself "often manifest[s] as stalled load/store operations within CUDA kernels without returning a clear error code"1. There is no exception to catch; the kernel just never returns. Meta's fix is NCCLX, a fork of NVIDIA's NCCL tuned for the higher latencies of a multi-hop network1, co-designed tightly enough with PyTorch that PyTorch can read NCCLX's internal state directly: the system watches the communication library and automatically times out on a detected stall, then dumps a snapshot of the failing collective's internal state — which ranks had finished their part of the data transfer and which were still pending1. That snapshot is what a human (or, per section 4, an automated loop) actually reads to localize the fault. Because this tracing has a cost, Meta can turn the more expensive collection on selectively, live, "through online configuration changes without needing a code release or job restart"1 — the diagnostic depth is a runtime dial, not a build-time decision.
2. The 54-day interruption census: what actually breaks
Meta's Table 5 is the most granular public breakdown of what fails during frontier-scale pre-training. Table 2 reproduces it in full, unedited: eighteen root-cause categories, each tagged with a coarser category label, covering all 419 unexpected interruptions recorded in the 54-day snapshot1.
| Component | Category | Count | % of interruptions |
|---|---|---|---|
| Faulty GPU | GPU | 148 | 30.1% |
| GPU HBM3 memory | GPU | 72 | 17.2% |
| Software bug | Dependency | 54 | 12.9% |
| Network switch/cable | Network | 35 | 8.4% |
| Host maintenance | Unplanned maintenance | 32 | 7.6% |
| GPU SRAM memory | GPU | 19 | 4.5% |
| GPU system processor | GPU | 17 | 4.1% |
| NIC | Host | 7 | 1.7% |
| NCCL watchdog timeouts | Unknown | 7 | 1.7% |
| Silent data corruption | GPU | 6 | 1.4% |
| GPU thermal interface + sensor | GPU | 6 | 1.4% |
| SSD | Host | 3 | 0.7% |
| Power supply | Host | 3 | 0.7% |
| Server chassis | Host | 2 | 0.5% |
| IO expansion board | Host | 2 | 0.5% |
| Dependency | Dependency | 2 | 0.5% |
| CPU | Host | 2 | 0.5% |
| System memory | Host | 2 | 0.5% |
Three things stand out. First, the distribution is heavily right-skewed: the top two rows (faulty GPU, GPU HBM3 memory) alone account for 47.3% of all unexpected interruptions, and the bottom eleven rows together account for barely 10%. A fault-tolerance system that only handled the top two categories well would already be most of the way to the observed >90% effective-training-time result discussed in section 3.
Second, the paper's own summary sentence states that "GPU issues are the largest category, accounting for 58.7% of all unexpected issues"1. Summing every row Table 2 itself tags with category GPU — faulty GPU, HBM3 memory, SRAM memory, system processor, silent data corruption, and the thermal interface/sensor row — gives 148+72+19+17+6+6 = 268 of 419, or 64.0%, about 5 points higher than Meta's own headline figure. The paper does not reconcile the two numbers; the most likely explanation is that Meta's 58.7% figure excludes rows it treats as "suspected" rather than "confirmed" hardware (silent data corruption is explicitly called out elsewhere in the same paragraph as a suspected rather than confirmed category1), but the paper does not say so explicitly. Both numbers are Meta's own; this module reports both rather than picking one.
Third, approximately 78% of unexpected interruptions trace to confirmed or suspected hardware issues1 — consistent with summing the GPU (268), Host (21), and Unplanned-maintenance (32) category rows in Table 2, which gives 321 of 419, or 76.6%, close enough to Meta's rounded "approximately 78%" to treat the two figures as the same measurement reported at different precision.
The eleven smallest categories in Table 2 (NIC, SSD, power supply, server chassis, IO expansion board, dependency, CPU, and system memory, each ≤7 events) are not eleven separate engineering problems worth eleven separate fixes. Each is a component that fails at some small, roughly constant rate in any sufficiently large fleet of servers — the same long tail Module 33 and Module 34 both assume exists underneath any cluster-scale system, training or otherwise. No single fix moves the needle on any one of them; that is precisely why the tooling built in sections 4 and 5 targets the categories large enough to justify dedicated engineering — faulty GPUs, HBM3 memory, software bugs, and network hardware — rather than chasing the tail.
3. Effective training time: the metric, and the arithmetic behind >90%
Neither paper reports mean-time-between-failures or mean-time-to-recovery as the headline number. Both report a single ratio instead. Meta defines it directly: "the effective training time measures the time spent on useful training over the elapsed time"1. It is a fraction, not a count, and it folds together failure frequency and recovery latency into one number a scheduler or a finance team can budget against.
The number itself is identical in both papers, to two significant figures, despite describing two different clusters at two different companies: Meta reports "higher than 90% effective training time" across the 54-day snapshot1; MegaScale reports maintaining "over 90% effective training time rate" on its production runs3. Neither paper claims the other's number as evidence — this is an independent convergence, not a shared measurement — but it is still the strongest available signal that >90% is close to what a well-engineered 2024-generation H100-class cluster running synchronous 4D/3D parallelism actually achieves, rather than a number either company rounded up for a paper.
The definition is worth pausing on for what it does not measure. It is not an SLO in the Module 11 sense — there is no external customer with a contract, and missing 90% on a given day has no defined consequence beyond a slower run. It functions more like an internal SLI for a single long-lived job: a single number a scheduler, a finance team, or a paper's own authors can budget against, that folds together how often something breaks and how expensive each break is into one ratio, without requiring either paper to disclose a full incident log to make its reliability claim legible.
Worked derivation: what >90% effective training time implies about the cost of a single interruption
Meta's 54-day snapshot recorded 466 total interruptions (47 planned, 419 unexpected)1 and >90% effective training time1. Converting the window to minutes and applying the >90% floor bounds the average cost per interruption:
$$ E = 54 \times 24 \times 60 = 77{,}760 \text{ minutes elapsed} $$ $$ \text{time lost} \le (1 - 0.90) \times E = 7{,}776 \text{ minutes} $$ $$ \text{average cost per interruption} \le \frac{7{,}776}{466} \approx 16.7 \text{ minutes} $$That upper bound — about 16.7 minutes of lost time per interruption, averaged over detection, diagnosis, node replacement, and resumption — is consistent with, though independently derived from, MegaScale's own directly-stated numbers: an average failure-detection-and-diagnostic-test time under 10 minutes, and a system that "can catch up to the training progress prior to the crash within 15 minutes from the latest checkpoints"3. The two figures describe different clusters and different companies; the arithmetic does not prove they are the same system, only that both land in the same 10–20-minute-per-incident regime, which is the order of magnitude a 90%-effective-training-time target actually requires once you have 400+ interruptions to amortize it over.
Scaling that same 7,776-minute upper bound by the cluster size gives a sense of what is actually at stake. At up to 16,384 GPUs1, 7,776 minutes (129.6 hours) of lost time is also an upper bound on 129.6 × 16,384 ≈ 2.12 million lost GPU-hours across the 54-day run — the quantity every mechanism in sections 4 and 5 exists to keep small, and the reason a fast diagnosis loop is worth the engineering cost of building NCCLX or a custom Kubernetes controller in the first place.
The other number worth deriving is the daily interruption rate implied by the same snapshot. Meta's own prose emphasizes only a floor — automated cluster maintenance "resulted in at least one training interruption daily"1 — but dividing the count in Table 2 by the window gives the actual average: 419 unexpected interruptions over 54 days is ≈7.8 unexpected interruptions per day, an order of magnitude above the "at least one" floor the prose highlights. The floor and the average are both true; the floor is about planned maintenance specifically, the average is about everything in Table 2.
4. MegaScale's closed-loop recovery: heartbeat, diagnosis, eviction, resume
MegaScale is the more mechanistically detailed of the two papers on this specific question: it describes a complete, named control loop rather than a set of disconnected tools. The paper calls it the "robust training workflow"3, and it runs as a driver process that submits the job, allocates one Kubernetes pod per node via a custom controller, and starts one training daemon per executor3.
- Heartbeat and anomaly signal. Each executor's training daemon sends the driver a periodic heartbeat carrying its IP, pod name, hardware info, the current status of the training process, filtered
stdout/stderrkeywords, and RDMA traffic metrics; the traffic-metric channel matters because some anomalies degrade throughput without producing an explicit error, so a step-to-step deviation in expected RDMA traffic is itself treated as a signal3. Monitoring runs at two granularities: second-level for overall health and configuration drift (ECN/PFC/QoS settings, link flapping), millisecond-level for detecting network congestion and whether the data-parallel/pipeline-parallel communication has hit its physical bandwidth limit3. - Diagnosis. A missed heartbeat within a predefined window, or an explicitly abnormal status, triggers the driver to suspend the job across every executor and run a suite of lightweight self-check diagnostics3: a Loopback test measuring full-mesh RDMA-NIC bandwidth within a host, an RNIC-to-RNIC test checking connectivity and bandwidth between RNICs on different hosts, an intra-node NCCL all-to-all test, and an inter-node NCCL all-reduce test against neighboring machines under the same top-of-rack switch3. The tests trade completeness for speed deliberately — long diagnostics protect accuracy but cost effective training time directly, and the paper reports tuning this trade-off through iterative experimentation3: "extended diagnostic duration can adversely affect the effective training time, while high false positive rates can lead to unnecessary exclusion of machines that are actually functional"3 — a test suite that is too thorough burns the same effective-training-time budget it exists to protect, and one that is too eager quarantines healthy hardware the job still needed.
- Checkpoint-based recovery. Once the driver has the IPs of the faulty nodes, it submits them to the custom Kubernetes controller, which evicts the faulty pods and replenishes the job with "an equivalent number of healthy ones which pass our diagnostic tests"3. The loop is not exclusively automatic even in MegaScale's own account: the driver also exposes a user interface for manual eviction, specifically for nodes identified through the section-5 tooling rather than the automated self-checks3 — the "fully automated" framing in Table 1 describes the common path, not the only one. Either way, the job's parallel shape (its TP/PP/DP layout) is preserved; only the physical nodes filling it change. The resumed set of workers then reloads the latest checkpoint. Recovery reads are the bottleneck here, so MegaScale designates one worker per data-parallel group to read the group's shared state partition from HDFS and broadcast it to the other workers in that group, which "reduces the load linearly" on HDFS bandwidth instead of having every worker hit the file system independently3. (HDFS's own throughput ceiling and Meta's equivalent Tectonic fabric — 240 PB, 2 TB/s sustained, 7 TB/s peak — are Module 33's subject; the point here is what the checkpoint pipeline does to the recovery-critical path, not the storage fabric's raw capacity.)
- Resumed training. Writing checkpoints is optimized the same way recovery reads are, and for a stated reason: "it is critical to ensure that the latest checkpoint is as close as possible to the state of training progress when the faults happened... this requires us to increase the frequency of checkpointing during training. However, we also want to reduce the latency introduced by the checkpointing process, especially the time on the critical path which blocks the training progress"3 — checkpointing more often bounds how much progress a failure can erase, but only if each checkpoint is cheap enough that writing it does not itself become the interruption. MegaScale's answer is the same two-stage pipeline as the recovery read path: first copy each GPU's on-chip state to host memory using pinned memory and an optimized serialization path, a step the paper reports reducing "to several seconds," after which a separate background process asynchronously transfers the host-memory copy to HDFS without blocking the resumed training loop3. Over a multi-week production run of a several-hundred-billion-parameter model on more than 10,000 GPUs, this loop repaired and resumed the job over 100 times while the training loss continued to converge3 — the same >90% effective-training-time result Section 3 quantified, now attached to the mechanism that produces it.
5. Diagnosing what a heartbeat can't see: stragglers and the 3D visualization tool
The self-check tests in section 4 catch faults that fail cleanly: a link that will not pass a bandwidth test, an NCCL call that errors out. They do not catch a GPU that keeps running, just slower, or one that hangs a collective without ever returning an error code. MegaScale built two additional tools specifically for that gap, and both are worth understanding on their own mechanism, not just their name.
The first is a CUDA-event performance analyzer with two visualization modes3. It times critical code segments on every rank using CUDA events rather than CUDA synchronization, which avoids the performance hit of forcing a sync just to measure one, so it can run continuously in production3. The first mode aggregates this into a per-machine heat-map of computation-phase latency; MegaScale reports that this reveals a small fraction — approximately 0.5% — of machines running substantially slower than the rest, and that training efficiency is "predominantly determined by the slowest machine's performance," which is why MFU is inconsistent run to run even with identical configurations until those specific outlier machines are excluded3. A related investigation using the same event-timer data found specific hosts consistently taking about 10% longer to execute the same forward computation across multiple experimental setups; isolating and removing those hosts from the cluster produced roughly a 0.7 percentage-point MFU improvement3 — a small number in isolation, but it came from a handful of machines out of thousands, which is the entire point of a straggler: its cost is disproportionate to its share of the fleet.
The second mode is a distributed trace view that stitches per-rank CUDA event timelines into one unified, cross-machine timeline, showing pipeline bubbles, communication gaps, and synchronization structure across ranks that a single-node profiler like the standard PyTorch profiler cannot represent3. Selecting an event surfaces the dependency that produced it, which is exactly the primitive the 3D visualization tool builds on.
The mechanism behind why this tool is necessary at all is specific and worth stating precisely. When a GPU is defective in a way that probabilistically blocks an NCCL communication call rather than erroring immediately, that call hangs on the faulty GPU's rank; every other rank that is waiting on a collective involving that rank hangs too, and the hang cascades outward through dependent ranks until the entire job times out3. From outside, this looks like one generic, cluster-wide blockage — every rank is equally stuck. MegaScale's fix is a targeted logging asymmetry: each GPU worker logs its own ongoing operation whenever it exits on a communication timeout, so every rank that was waiting on the faulty one produces a log entry describing what it was waiting for3. The faulty rank itself, still hung inside the blocked call, produces no such log — it is the one silent node in a sea of nodes that all logged the same complaint. The 3D visualization tool reconstructs the dependency graph from those logs and error messages and surfaces the silent node directly, rather than requiring an engineer to manually cross-reference which of ten thousand ranks never checked in3.
This is the same pathology section 1 quoted from the Llama 3 paper in different words — a stalled load/store instruction with no error code1 — and it is worth noting that Meta and ByteDance converge on the same class of fix despite building unrelated tools: neither waits for the faulty component to report its own failure, because at this layer it usually cannot. Meta's NCCLX watches the communication library from outside and snapshots state on a detected stall; MegaScale exploits the fact that every other rank still logs normally, and infers the faulty one by its silence. Both are answers to the same underlying fact: once a GPU or a link fails badly enough to hang a collective, the failure is legible only from everything around it, never from the thing itself.
Two further production incidents in the paper illustrate the same "diagnose by elimination" pattern applied to problems that are not a single faulty part. Training efficiency gradually declined over the course of one job even though per-phase (forward/backward/optimizer) computation time stayed flat; tracing the per-step time increase to its last collective call identified the data-parallel reduce-scatter as the point where some ranks launched later than others, and the resulting time-skew grew across steps rather than staying fixed, eventually traced to irregular garbage-collection pauses and certain PyTorch operations sitting on the critical path — removing those code segments stopped the decline3. Separately, intermittent training stalls traced to network-interface flapping — brief but repeated link up/down cycles, during which every packet in transit is dropped for several seconds at a time. The first lesson MegaScale reports is that the NCCL retransmit timeout has to be set explicitly higher than the library default, because the default returns a completion error to the training job before the interface has a chance to come back up; the second is that the root cause is usually bad physical-layer signal quality between the network card, the AOC cable, and the switch, fixed by tightening that signal quality directly and by enabling the NIC's adap_retrans feature, which retransmits at a shorter interval and recovers faster specifically when the flapping period itself is short3. Neither problem would show up in Table 2's clean root-cause categories; both were found only because the tooling in this section made the cluster's internal state legible enough to chase a symptom back to its cause.
Some of MegaScale's network engineering is preventative rather than diagnostic, and it earns its place here because congestion that looks like a stall is otherwise indistinguishable from a real fault until someone has manually ruled it out. All-to-all collective traffic at this scale produces "fat" flows that traditional equal-cost multi-path (ECMP) routing balances poorly, so MegaScale redesigns the network topology and scheduling specifically to reduce ECMP hashing conflicts — splitting each downlink port into multiple lower-bandwidth links with distinct cabling, and scheduling data-intensive nodes from the same job under the same top-of-rack switch — which lowers the chance that ordinary collective traffic gets misrouted into a congested path and manifests as exactly the kind of transient slowdown section 5.1's straggler-hunting tooling would otherwise have to spend time ruling out3.
6. Checkpoint/restart as the universal primitive, and what "elasticity" means when the shape is fixed
Strip away the tooling and both papers land on the same recovery primitive: reload the last checkpoint, resume the job. That is a narrower move than it might sound next to the rest of this course. Module 9's MapReduce re-executes only the specific map task whose output lived on a now-dead machine, because map tasks are independent and idempotent by construction; the other thousands of tasks in the job never notice. Module 11's retry-with-backoff absorbs a single request's failure with a blast radius of exactly one request. Training has neither luxury: because a step ends in a synchronous collective across every participating rank, there is no unit smaller than "the whole job" to re-execute once a rank has dropped out mid-collective. Checkpoint/restart is not one option among several here — given the constraint stated in section 1, it is close to the only option, which is why both an American frontier lab and a Chinese hyperscaler independently converged on it as the recovery primitive and instead spent their engineering effort on the two levers that actually move the needle inside that constraint: how fast you can detect and localize a fault (sections 4 and 5), and how little state you lose and how fast you can reload it (section 4.4's two-stage checkpoint).
This also clarifies what "elasticity" means in this specific setting, and it is a narrower claim than the word usually carries elsewhere in this course. An autoscaling web tier (Module 10, Module 11) is elastic in the sense that its shape changes: more replicas under load, fewer at night, and the system's logical structure does not depend on exactly how many replicas exist at a given moment. A synchronous training job's shape is fixed by its 4D or 3D parallel configuration — a specific TP×PP×DP×CP grid, chosen ahead of time and baked into which rank talks to which1 — and that shape cannot flex mid-run without changing the parallelism plan itself, which neither paper describes doing. What MegaScale calls elasticity is narrower and more precise: when a node fails, Kubernetes evicts it and "replenishes the cluster with an equivalent number of healthy ones which pass our diagnostic tests"3 — a like-for-like replacement that restores the same grid with different physical hardware, not a grid that grows or shrinks. The elastic resource is spare capacity in the fleet, held in reserve so that a replacement is available on demand; the job itself stays a fixed size throughout.
Why the shape cannot flex for free is itself measurable in Meta's own numbers, not just asserted. Table 4 of the Llama 3 paper reports 43% BF16 MFU on 8,192 GPUs with a data-parallel degree of 64, versus 41% on 16,384 GPUs with a data-parallel degree of 128 — a real, measured drop from doubling the GPU count on the same job1. Meta attributes it directly to keeping the global tokens-per-batch constant while doubling the data-parallel degree, which forces a smaller batch size per data-parallel group and leaves each GPU doing proportionally less useful work per synchronization round1. That is the mechanism a naive "just add more GPUs mid-run" version of elasticity would have to solve for every single node addition, not just the two data points in Table 4 — which is exactly why neither paper attempts it, and why the elasticity both papers do build is scoped to holding the GPU count fixed and only swapping which physical GPUs occupy it.
One boundary worth being explicit about, in the spirit of this module's own coverage-honesty framing: neither paper says how large that spare-capacity buffer is, how it is provisioned, or what happens if no equivalent healthy node is available when one is needed. That is a real gap in the public record, not a claim this module can fill from Meta's or ByteDance's own words. What both papers do establish, precisely, is the loop that runs once a spare is available: detect, diagnose, evict and replace, reload, resume — and that loop, run enough times in the background, is what an >90% effective-training-time number is actually made of.
Where experts disagree
Neither Meta nor ByteDance frames its own paper as a rebuttal of the other, and the researcher's source map records no pre-existing disagreement for this specific topic. But the two papers make visibly different engineering bets on the same underlying question — given that a single-node fault can stall an entire synchronous job, where should the engineering investment go? — and a third position, raised by MegaScale itself only to reject it, sharpens the contrast. All three are grounded in direct statements from the two source papers, not inferred motive.
Automate the loop; do not over-diagnose
MegaScale / ByteDance3
Treat individual nodes as interchangeable. Build a fast, generic detect→evict→replace→resume loop that acts on an IP address and a pass/fail diagnostic result, without first fully root-causing every fault. Reserve deep tooling (the CUDA-event heat-map, the 3D visualization) for the residual cases the fast loop cannot resolve — stragglers and silent hangs that pass a binary self-check.
- Recovery latency does not scale with diagnostic depth; most faults never need a human or a slow investigation.
- >90% of faults were auto-detected, located, and recovered on MegaScale's production runs3.
- A fault that passes every self-check (a probabilistic hang, a silent straggler) defeats the fast path entirely and falls through to the slow, tool-heavy path anyway.
- Requires spare, pre-validated capacity on hand to "replenish" into — a cost the paper does not quantify.
Invest in diagnosis instrumentation; accept rare manual triage
Meta / Llama 3 405B infrastructure team1
Build deep, always-on visibility into the communication library itself — NCCLX co-designed tightly enough with PyTorch that a flight recorder can capture every collective's metadata and stack trace into a ring buffer, diagnosing hangs in minutes even on a mixed NVLink/RoCE fabric where failures "often manifest as stalled load/store operations... without returning a clear error code"1. Treat the rare case that instrumentation cannot resolve as an acceptable, bounded cost rather than a gap to be automated away at any price.
- Only 3 of 419 unexpected interruptions needed manual intervention over the 54-day window1 — a lower manual-intervention share than MegaScale's own stated ">90% automated" figure implies.
- Deep tracing generalizes to problems that were never in the diagnostic-test catalog to begin with.
- Requires owning and modifying the collective-communication library itself (NCCLX is a fork of NCCL) — a much larger engineering investment than a self-check test suite.
- The paper does not describe an automatic node-eviction-and-replace loop comparable to MegaScale's; recovery mechanics beyond diagnosis are left unstated.
Remove the coupling instead of curing it
Cited by MegaScale as related work, and rejected3
If a single rank's failure stalls the whole job because every step ends in a blocking collective, the more fundamental fix is to stop requiring every rank to arrive at the same collective at the same time. MegaScale's own related-work section notes that "some works introduce fixed staleness in the training pipeline for full overlapping communication and computation," which would shrink a single node's blast radius by design rather than recovering from it faster.
- Attacks the root cause identified in section 1 (synchronous coupling) rather than the symptom.
- Already the accepted trade-off elsewhere in this course's training material — rollout generation in RL post-training (Module 37) tolerates stale actors precisely because it is not gradient-synchronous.
- MegaScale's own verdict: "the staleness may degrade the model performance"3 — for pre-training, where matching a synchronous baseline's loss curve at a given token count is the entire point, this cost is not obviously acceptable.
- Neither Llama 3 nor MegaScale's production pre-training runs adopt it; it remains a literature position, not a demonstrated production one at this scale.
For frontier pre-training specifically, where the entire point is to match a known-good loss curve at a fixed token budget, positions one and two dominate position three: both Meta and ByteDance ship synchronous training and invest in making failure recovery fast rather than making failure survivable-by-design, and neither reports adopting a staleness-tolerant pipeline for its flagship pre-training run. Between positions one and two, the public record does not cleanly separate them — both land on the same >90% effective-training-time outcome by different routes (a generic automated loop versus deep, library-level instrumentation with rare manual fallback), and each paper reports a number that looks favorable by its own preferred measure (MegaScale's ">90% automated," Meta's "3 of 419 manual"). What would settle it is a controlled comparison on the same hardware and the same failure mix, holding the cluster constant and varying only the recovery architecture; neither paper provides that, because each describes only its own company's fleet. The safer reading is that they are not actually competing solutions to the same problem so much as two points on a spectrum every operator at this scale eventually occupies both ends of — cheap automated eviction for the faults that fail cleanly, and library-level tracing for the ones that do not.
References
- paperLlama Team, Meta AI. The Llama 3 Herd of Models. arXiv:2407.21783, 2024. https://arxiv.org/abs/2407.21783. Accessed 2026-09-17. Used for: cluster/hardware specs (§3.3.1), the 54-day interruption census and full root-cause Table 5, the effective-training-time definition and >90% result, diurnal throughput/power-swing observations, and the NCCLX flight-recorder diagnosis approach (all §3.3.4).
- paperVerma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., Wilkes, J. Large-scale cluster management at Google with Borg. EuroSys '15, 2015. https://research.google.com/pubs/archive/43438.pdf. Accessed 2026-09-17. Used for: quota-checked-at-admission and the ~80% graceful-SIGTERM preemption figure, as the general-purpose cluster-management baseline this module's failure model breaks from (§2.3, §2.5).
- paperJiang, Z., Lin, H., Zhong, Y., et al. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs. USENIX NSDI '24, 2024. https://www.usenix.org/system/files/nsdi24-jiang-ziheng.pdf. Accessed 2026-09-17. Used for: the robust training workflow (§4.1), heartbeat payload and monitoring granularity (§4.2), self-check diagnostic tests (§4.3), the two-stage checkpoint/recovery pipeline (§4.4), the CUDA-event heat-map/trace tools and 3D visualization tool (§5.1-5.2), production recovery experience and MFU/effective-training-time figures (§6.1-6.3), and the related-work mention of staleness-tolerant pipelines (§7).