System Design · Module 34
Cluster Scheduling and Orchestration: Slurm, the Borg/Kubernetes Lineage, Ray, and MAST
A training job asking for several thousand GPUs at once is a scheduling problem before it is anything else: something has to decide whether the job is even allowed to ask, where its tasks land relative to the network topology Module 33 already fixed, and what happens when two jobs want the same GPUs at the same time. This module reads four different answers to that problem -- Google's Borg, SchedMD's Slurm, Ray, and Meta's MAST -- against a real multi-tenant GPU cluster trace (Microsoft's Philly) that shows exactly which generic-scheduler assumptions break under gang-scheduled, locality-sensitive DNN training.
1. What a training-cluster scheduler actually decides
Strip away vendor names and a cluster scheduler for GPU training answers exactly three questions, in a fixed order. First: is this job even allowed to exist, given who submitted it and how much of the cluster they are entitled to -- admission. Second: given that it is admitted, which physical machines does it actually run on, respecting the interconnect topology Module 33 already built (rack, pod, cluster; full bisection bandwidth within a pod, 1:7 oversubscription across pods) -- placement. Third: when two admitted jobs want overlapping resources, who wins -- arbitration under contention. Every system in this module answers all three questions; they differ in where the logic lives, how centralized it is, and how much they know about GPUs specifically.
That last qualifier matters more than it sounds like it should. A CPU core running a web server can be time-sliced across many small requests, which is exactly the shape Module 10's backpressure and admission-control material assumes. A GPU training job does not look like that. Microsoft's own framing of the problem, from the trace study this module leans on most heavily in Section 6, states the difference plainly: "unlike CPUs, accelerators do not typically have proper hardware support for fine-grained sharing," and "machine learning frameworks typically require that tasks on each GPU be scheduled at the same time, i.e., gang scheduled"5. Two consequences follow directly. A GPU is effectively an indivisible unit for the duration of a job -- there is no cheap equivalent of an inference gateway's token-bucket rate limiter parceling out fractional GPU-seconds to many tenants at once. And a distributed training job's workers must all start together or not at all: gang scheduling is not an optimization a scheduler can skip, it is a correctness requirement, because a training step that only half its workers joined will simply hang waiting for an all-reduce that never completes.
This module's scope ends exactly where a job successfully starts running. What happens when an already-scheduled, already-running job hits a hardware fault mid-step -- checkpoint/restart, straggler detection, the actual failure taxonomy of a large training run -- is Module 35's job, not this one; Section 6 below touches Philly's failure data only insofar as it changes how a scheduler should treat a newly submitted job, not how a running one recovers. Four systems carry this module's main comparison: Borg (Section 2), the general-purpose Google cluster manager and Kubernetes' direct architectural ancestor1; Slurm (Section 3), the dominant HPC batch scheduler; Ray (Section 4), a distributed-Python framework whose task/actor/object-store primitives increasingly sit underneath training and RL infrastructure; and MAST (Section 5), Meta's own scheduler, named at the one sentence of detail Meta has published. Section 6 adds a fifth, different kind of evidence: Philly, a two-month production trace from a real multi-tenant GPU cluster, fetched and read directly for this module rather than taken from a cached summary, which shows concretely what generic cluster-scheduling logic gets wrong when the resource being scheduled is a GPU instead of a CPU core.
2. Borg: cells, priority, quota, and admission control specialized to a training job
A Borg cell is "a set of machines, managed as a unit"1; a cell's architecture is a logically centralized Borgmaster -- replicated five times, with a single elected master handling every state mutation via a Paxos-based store on the replicas' local disks -- plus a Borglet agent running on every machine in the cell1 (§3.1). The Borgmaster itself splits into two processes: the main process, which handles client RPCs and holds the authoritative state of every object in the system, and a separate scheduler process that scans the pending queue asynchronously, high priority to low, "modulated by a round-robin scheme within a priority to ensure fairness across users and avoid head-of-line blocking behind a large job"1 (§3.2). Cells run up to tens of thousands of machines, with a median cell size around 10,0001. This is a general-purpose cluster manager, not a training-specific one: Borg's own workload description names Gmail, Google Docs, web search, and BigTable as its latency-sensitive tenants, and "batch jobs that take from a few seconds to a few days" as the other1 -- nothing in the paper mentions GPUs or gang scheduling, because Borg predates Google's reported use of it for accelerator-heavy ML training at this scale. What carries forward to GPU clusters is the admission-control mechanism, not any GPU-specific feature.
That mechanism is priority plus quota, and Module 10 (§3.1) and Module 24 (§2) have already built the general version of it in full: every job carries a small positive-integer priority, grouped into non-overlapping bands -- in decreasing order, monitoring, production, batch, and best-effort1 -- and quota is "a vector of resource quantities (CPU, RAM, disk, etc.) at a given priority, for a period of time (typically months)," with Borg's own worked example being "20 TiB of RAM at prod priority from now until the end of July in cell xx"1. The load-bearing sentence, quoted identically in both of those modules because it is the single design decision this whole comparison turns on, is: "Quota-checking is part of admission control, not scheduling: jobs with insufficient quota are immediately rejected upon submission"1. A job can be admitted and still fail to run -- "a low-priority job may be admitted but remain pending (unscheduled) due to insufficient resources"1 -- but a job that fails the quota check never reaches the scheduler at all. This module does not re-derive that separation; it specializes it.
2.1 Worked derivation: quota headroom for a training job's GPU count
Borg's own quota example is denominated in RAM. Nothing about the admission algorithm cares what fills the vector -- "CPU, RAM, disk, etc." is explicitly open-ended1 -- so substitute a GPU count for RAM and the arithmetic Module 24 (§2.1) already worked through for an inference platform's memory quota carries over unchanged. Suppose a training organization holds a quota of \(Q\) GPUs at production priority in a cluster, its currently running production jobs commit \(C\) GPUs against that ceiling, and a newly submitted job requests \(N\) GPUs. Admission control does not ask whether any rack has \(N\) physically idle, topologically adjacent GPUs -- that question belongs to the scheduler's feasibility-and-scoring step, run only after admission succeeds, and it is exactly the question Section 6 shows a generic scheduler answering badly for GPU jobs. Admission only asks:
$$ C + N \le Q $$
If the inequality fails, the job is rejected in the time it takes to read and compare two integers, before a single machine is examined -- the same property Module 24 (§2.1) derives for a token-budget quota check, here with GPUs instead of RAM as the unit. A rejection at this step tells the submitter nothing about cluster load, congestion, or fragmentation; it says only that their organization's own ledger has no room left, which is precisely why it can be checked without touching the scheduler's much more expensive feasibility search.
Feasibility checking itself, once a job clears admission, "finds machines on which the task could run" -- machines meeting the task's constraints with enough available resources, "which includes resources assigned to lower-priority tasks that can be evicted"1 -- and scoring then picks among the feasible set using criteria that include "picking machines that already have a copy of the task's packages, spreading tasks across power and failure domains, and packing quality"1. None of Borg's built-in scoring criteria mention interconnect locality or gang-scheduling atomicity, because Borg's general-purpose scheduler was not built around either concern. That gap -- a generic feasibility-and-scoring algorithm asked to place a job whose performance depends heavily on which specific GPUs it lands on, all at once -- is exactly what Section 6's trace data measures.
2.2 Resource reclamation: a mechanism GPUs cannot use the way CPUs can
Borg has one more admission-adjacent mechanism worth carrying into this comparison, because it draws a distinction quota and priority alone do not: the gap between what a task reserves and what it actually uses. Borg tracks a task's limit -- the resource request quota was checked against -- separately from a continuously re-estimated reservation computed by the Borgmaster from the Borglet's fine-grained usage telemetry: "the initial reservation is set equal to the resource request (the limit); after 300s, to allow for startup transients, it decays slowly towards the actual usage plus a safety margin"1 (§5.5). Whatever gap remains between limit and reservation is handed to lower-priority batch work that can tolerate the risk of being evicted if the original task's usage spikes back up, and the effect at Google's own scale is substantial: "about 20% of the workload runs in reclaimed resources in a median cell"1 (§5.5).
That mechanism exists only because Borg's resources -- CPU, RAM, disk -- are fine-grained and reclaimable at runtime without destroying the task holding them: CPU can be throttled, RAM reclaimed down to a safety margin, both while the original task keeps running undisturbed. Section 1 already quoted the reason this does not transfer to GPUs: "unlike CPUs, accelerators do not typically have proper hardware support for fine-grained sharing"5. Section 6.2's finding that roughly 47.7% of in-use GPU cycles go to waste is, in Borg's own vocabulary, an enormous limit-minus-usage gap -- almost exactly the shape resource reclamation exists to close for CPU and RAM -- but with no hardware-level mechanism to reclaim the unused fraction of one job's GPU without preempting the job outright. The mechanism Borg built for one resource class simply has no GPU-shaped equivalent yet, which is one concrete reason this module's GPU-utilization numbers sit so far below what a CPU-oriented cluster manager's operators would consider normal.
2.3 Preemption notice, and why it matters for a checkpoint-friendly training job
Priority-based preemption is only survivable if the preempted task gets warning before it dies. Borg sends a SIGTERM ahead of the SIGKILL that actually reclaims the machine, giving the task time to "clean up, save state, finish any currently-executing requests, and decline new ones"1 -- though "the actual notice may be less if the preemptor sets a delay bound," and in practice "a notice is delivered about 80% of the time"1. For a stateless web request, a missed notice costs one dropped request. For a large gang-scheduled training job it costs far more: a preempted worker that never receives its SIGTERM either loses whatever gradient computation has accumulated since the last checkpoint, or -- because every worker synchronizes at every step -- stalls the entire job waiting on an all-reduce that one missing rank will never join. Borg's admission-and-priority mechanism is what produces the preemption in the first place; what happens after that signal arrives, or fails to, is squarely Module 35's territory (checkpoint cadence, restart latency), not this one's, but the 80% figure is a concrete reason that module's recovery machinery cannot assume every eviction arrives with a clean warning.
3. Slurm: slurmctld, slurmd, and the pending-work queue
SchedMD's own description of Slurm names three functions, stated as plainly as Borg's paper states admission control: it "allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work"; it "provides a framework for starting, executing, and monitoring work" on those allocated nodes; and it "arbitrates contention for resources by managing a queue of pending work"2. Each of those three is a direct analog of one of Section 1's three questions -- allocation is admission plus placement bundled into one step, the monitoring framework is what makes gang-scheduled multi-node jobs coherent once running, and the pending-work queue is arbitration under contention.
The architecture that implements this is deliberately simple and centralized. "Slurm has a centralized manager, slurmctld, to monitor resources and work"2, typically with an optional backup instance for failover. Every compute node runs a slurmd daemon, described in the documentation with a specific analogy: it "can be compared to a remote shell: it waits for work, executes that work, returns status, and waits for more work"2. That comparison is precise and worth taking literally -- slurmd does not participate in any decision about what to run or where; it is a passive executor that reports status upward and accepts dispatch instructions from slurmctld. Two optional daemons round out the picture: "an optional database daemon stores accounting data across multiple clusters, while a REST API daemon enables remote interactions with Slurm through standard HTTP protocols"2 -- slurmdbd for accounting, slurmrestd for programmatic access outside Slurm's native command-line tools.
Notice what Slurm's own documentation does not claim, because the omission is the whole comparison. There is no mention of a quota vector, a priority band, or an admission-control step separate from scheduling -- Slurm's allocation model is a single step: a user asks for a duration and a resource shape, and slurmctld either grants exclusive or non-exclusive access to a set of compute nodes for that duration, or the request waits in the pending queue2. Exclusive access hands a job every resource on a node for the duration of its allocation, with no other job's tasks landing there regardless of how idle any single GPU on it might be; non-exclusive access lets slurmctld pack multiple jobs' tasks onto the same node's remaining capacity. A gang-scheduled, multi-node training job almost always wants the exclusive form -- it needs every GPU on every node it touches, and sharing a node with an unrelated job reintroduces exactly the interference Section 6.2's ResNet-50 experiment measures. That is a coarser-grained, simpler decision than Borg's two-phase admission-then-feasibility-then-scoring pipeline, and it maps naturally onto the shape of a single large, synchronous, gang-scheduled pretraining job: one job, one duration, one block of nodes, allocated as a unit. It maps far less naturally onto a cluster serving many small, short, differently-prioritized jobs at once, which is precisely the shape Borg's priority bands and quota vector exist to arbitrate. Slurm's simplicity is a design choice suited to its origin -- HPC supercomputing centers running a queue of large, mostly homogeneous batch jobs -- not an oversight relative to Borg's richer model.
The slurmctld/slurmd split is structurally the same master/agent pattern as Borg's Borgmaster/Borglet split and Ray's driver/worker split below: a centralized brain and a dumb executor on every machine is the one architectural idea every system in this module reuses, regardless of which camp it belongs to in Section 7's fork. What differs is what the brain does with the information the agents report, and how often it re-evaluates its own decisions -- a distinction Section 7's control-loop comparison makes explicit.
4. Ray: tasks, actors, and the distributed object store
Ray takes a third approach, and its own documentation frames the design goal directly: distributed Python scaling with "a small set of essential primitives (tasks, actors, and objects) for building and scaling distributed applications"3. A task is a stateless, remotely-executed function call -- the caller gets back a future immediately and the actual computation runs asynchronously somewhere in the cluster. An actor is the stateful counterpart: "Ray actors allow you to create stateful workers that maintain their internal state between method calls," and critically, "the actor executes method calls serially in the order it receives them, preserving consistency"3. That serial-execution guarantee is what makes an actor usable as, for instance, a model-serving process holding weights in GPU memory across many calls, or a parameter server accumulating gradients -- without it, concurrent method calls into the same actor could interleave and corrupt whatever state it holds.
Neither tasks nor actors would scale past a single machine's memory without a third primitive: the object store. Ray's documentation describes it as managing "data across your cluster" through three mechanisms -- "automatic storage when tasks/actors return values, explicit placement via ray.put(), and passing references between distributed components to avoid unnecessary data copying"3. The distinction between the first two matters for how a Ray program is actually written: a task's return value is stored in the object store automatically, with no call the programmer has to make, while a value the programmer wants to share deliberately -- a large array, a model checkpoint fragment, a dataset shard -- is placed there explicitly via ray.put(). Both paths converge on the same object store, and both hand the caller back the same kind of thing: a reference, not the underlying bytes.
ray.put() -- and hands every consumer a reference rather than a copy, so passing a large object between distributed components does not duplicate its bytes3.Ray does have a first-class answer to gang scheduling, separate from the task/actor/object-store trio: placement groups. A placement group lets a caller "atomically reserve groups of resources across multiple nodes, a concept commonly known as gang scheduling"6. The unit of reservation is a bundle -- a set of resource quantities such as {"CPU": 1, "GPU": 4} that "must be able to fit on a single node on the Ray cluster"6 -- and creation is atomic in exactly the all-or-nothing sense Section 6.1's Philly locality algorithm is not: "if a bundle can't fit in any of the current nodes, Ray reserves no resources for the placement group" at all, rather than partially granting it6. Four placement strategies control locality directly: STRICT_PACK forces every bundle onto a single node; PACK tries the same but falls back to other nodes if one node cannot fit everything; STRICT_SPREAD forces each bundle onto a separate node; SPREAD attempts the same on a best-effort basis6. STRICT_PACK is Ray's version of Philly's intra-server, single-RDMA-domain preference; SPREAD is what a deployment wants when it needs to keep one job's workers out of a single failure domain. What Ray's placement groups do not offer, unlike Philly's algorithm, is a documented timeout-then-relax fallback: a bundle set either fits atomically at request time or the whole reservation is refused outright, with no equivalent of Philly's bounded wait followed by constraint relaxation56.
The rest of Ray's scheduling behavior is different in kind from Borg's or Slurm's. Outside of an explicit placement group, task and object placement run through no quota vector and no pending-work queue in the sense Sections 2 and 3 described; Ray's driver process and its cluster-wide metadata service (the GCS) schedule tasks and actors onto workers directly, continuously, as work arrives, rather than allocating a fixed block of nodes to one job for a stated duration the way Slurm does. That continuous-scheduling shape is one reason Ray sits on the cloud-native side of Section 7's fork rather than the HPC-batch side, and it is also the reason Ray shows up underneath a specific, narrower slice of training infrastructure: reinforcement-learning post-training pipelines that need to run a generation (rollout) process and a training process as separate, independently-scaled services exchanging weights and trajectories, rather than as one monolithic gang-scheduled job. A survey of sixteen open-source RL training libraries found exactly this pattern: eight of the sixteen -- verl, SkyRL, NeMo-RL, SLIME, MILES, ROLL, OAT, and open-instruct -- use Ray for orchestration, while the other eight use native Python asyncio/threading, HTTP microservices, or an actor model of their own7. Module 38 covers what those frameworks actually do with Ray's primitives; this module's point is narrower -- Ray's task/actor/object-store model, plus placement groups for the minority of cases that need atomic gang scheduling, is a real, adopted answer to "who runs where" for a workload shape (heterogeneous, long-running, communicating services) that neither Borg's priority bands nor Slurm's exclusive node blocks were designed around.
5. MAST: named at exactly the scope Meta discloses
Meta's Llama 3 paper gives MAST one sentence: "Training jobs are scheduled using MAST, Meta's global-scale training scheduler"4. Everything else the paper discloses about the training run MAST scheduled is about the run, not the scheduler -- up to 16,000 H100 GPUs, each at 700 W TDP with 80 GB HBM3, on Meta's Grand Teton platform, eight GPUs and two CPUs per server connected via NVLink within a server4, running on the three-layer Clos network topology and Tectonic storage fabric Module 33 already covers in full. No source in this course's map -- and none that a search for a dedicated MAST architecture paper turned up as fetchable within this module's scope -- describes MAST's internal admission logic, whether it uses a quota vector, a priority scheme, or anything resembling Borg's or Slurm's control plane.
That silence is itself informative, and worth stating rather than filling in. Meta operates at a scale where Borg's own general-purpose lineage exists inside the company (Borg is a Google system, not Meta's, but the broader point -- that hyperscalers with sufficient internal engineering investment tend not to run either off-the-shelf HPC batch schedulers or off-the-shelf cloud-native orchestrators unmodified for their largest training runs -- is directly evidenced by Meta building and naming a bespoke system rather than describing its pretraining infrastructure as "we run vanilla Slurm" or "we run vanilla Kubernetes"4. MAST's name recurs in Section 7's disagreement as exactly this kind of evidence: a purpose-built scheduler is a real, adopted third path, but one that presupposes an engineering budget only a handful of organizations have.
6. Philly: what a real multi-tenant GPU cluster trace shows about generic scheduling
Borg, Slurm, and Ray are all described from their own documentation -- authoritative about design intent, silent about what actually happens when the design meets a real, contentious, multi-tenant workload. Philly closes that gap. It is a production GPU-cluster scheduling service built at Microsoft Research on top of Apache YARN's resource manager and HDFS for storage5 (§2.2), and the paper analyzes a 75-day trace (Oct-Dec 2017) of 96,260 jobs across 14 virtual clusters and hundreds of users5 (§2.4) -- the same kind of large-N, real-workload evidence Module 9 and Module 32 already lean on for MapReduce/Spark and Llama 3's data pipeline respectively, applied here to scheduling instead.
6.1 Virtual clusters as GPU quota, and locality-aware gang scheduling
Philly's admission-control layer is the concrete, production-grade version of Section 2.1's GPU-quota derivation. "To support multiple production groups we create a virtual cluster for each group and associate a resource share or quota in terms of number of GPUs to each virtual cluster. Each virtual cluster has a separate allocation queue in Apache YARN and we use the Fair Scheduler to manage these queues"5 (§2.3). That is Module 10's and Module 24's quota-checked-at-admission concept, instantiated with GPUs as the quota unit exactly as Section 2.1 generalized it from Borg's RAM example -- except Philly is not running Borg at all; it is a Fair-Scheduler queue per group, built independently, arriving at the same design because the problem (bound how much of a scarce, unshareable resource one tenant can claim) forces a similar answer regardless of which general-purpose scheduler sits underneath.
Where Philly goes beyond a generic quota check is locality-aware gang scheduling with a bounded-timeout, relax-after-retry fallback, which none of Borg's, Slurm's, or Ray's own documentation specifies as an algorithm -- Ray's placement groups (§4) come closest, with atomic reservation and explicit packing strategies, but refuse outright rather than waiting and relaxing. Distributed training requires all of a job's GPUs to be available simultaneously, and training speed depends heavily on whether those GPUs sit close together on the network: "workers in a distributed training job must either be colocated on the same machine or preferably communicate over a higher-speed network"5 (§2.3) -- inside a rack, machines share a 100-Gbps RDMA (InfiniBand) domain; across racks, traffic goes over plain Ethernet5 (§2.2). Philly's scheduler ranks racks and servers by increasing occupancy and tries to pack a job's GPUs onto the fewest servers within one RDMA domain, waiting for a pre-specified timeout (2-3 minutes in their deployment) to acquire all the GPUs the locality constraint calls for5 (§2.3). If the timeout expires, partially-acquired resources are released and the job retries after a backoff (2 minutes); after a fixed number of retries, locality constraints are relaxed entirely to prevent starvation5 (§2.3). None of Borg's built-in scoring criteria (package locality, failure-domain spreading, packing quality) implement this specific trade -- wait for topological locality, but only up to a bounded, retried timeout -- because Borg's general-purpose workload does not depend on it the way gang-scheduled training does.
6.2 Fragmentation delay dominates, and allocated GPUs still sit mostly idle
Trading locality for queueing delay has a measurable cost, and Philly's paper separates that cost into two causes. Fair-share delay happens when a virtual cluster has simply used up its GPU quota; fragmentation delay happens when a job is within quota but the scheduler cannot find enough topologically-close GPUs to satisfy its locality constraint5 (§3.1). The paper's own measurement shows fragmentation, not quota exhaustion, dominates as job size grows.
| GPU request | Fair-share delay | Fragmentation delay |
|---|---|---|
| 2-4 GPU | 40.6% (5,168) | 59.4% (7,567) |
| 5-8 GPU | 25.8% (3,793) | 74.2% (10,928) |
| >8 GPU | 2.1% (66) | 97.9% (3,117) |
Across all jobs, fragmentation accounts for roughly 80% of total queueing time -- "fair-share delays are easy to mitigate with preemption, but fragmentation delays are much harder to overcome in our current design"5 (§3.1.1). That asymmetry is a direct consequence of Section 2's admission-control framing: a fair-share (quota) rejection is a cheap, deterministic ledger comparison a scheduler can act on immediately, exactly as Figure 2 shows; a fragmentation delay is a bin-packing problem against live machine and topology state, the same feasibility-and-scoring step Borg's paper describes but gives no GPU-specific algorithm for, and it is inherently harder to bound.
Underneath both delay types sits Section 1's third question -- arbitration under contention -- and Philly's YARN-based scheduler answers it in a way worth naming because it is a real, work-conserving alternative to Borg's strict-priority-with-round-robin scan. Rather than leave GPUs idle while a large job waits for its locality constraint, the scheduler lets smaller, later-arrived jobs run out of submission order whenever idle capacity exists that the waiting large job cannot yet use: out-of-order scheduling occurred in 38.1% of all scheduling decisions in the trace, rising to 100% of decisions for jobs requesting 5-8 or more than 8 GPUs5 (§3.1.1). The reordering is not free of risk -- Section 2's Borg treatment already named the failure mode this could cause, head-of-line blocking behind a large job, which is exactly why Borg's own scheduler round-robins within a priority band -- but Philly's data shows the risk did not materialize badly in practice: for large jobs, 85.0% of out-of-order occurrences corresponded to idle GPUs being used productively without measurably prolonging the waiting job's own delay5 (§3.1.1). Work-conserving backfill, in other words, bought utilization at close to zero cost to the jobs it stepped in front of -- a data point a purely priority-ordered scheduler like Borg's cannot produce, because it never lets a lower-priority job run ahead of a higher-priority one waiting in the same band.
Locality and packing choices then feed straight into a second, separate cost: how efficiently the GPUs a job did get are actually used. Philly measured aggregate per-minute GPU utilization across its full workload and found "around 47.7% of in-use GPUs' cycles are wasted across all jobs," with mean utilization of 52.32% overall and as low as 40.39% for 16-GPU jobs specifically -- the largest job-size bucket the paper reports and the one Philly's scheduler is least able to pack with high locality5 (§3.2). A controlled ResNet-50 experiment isolates why: a job confined to one server averaged 57.7% GPU utilization at 114.8 images/sec; the identical job split across two RDMA-connected servers dropped to 49.6% utilization and 98.0 images/sec purely from the added synchronization cost of distribution, before any other job shared either machine5 (§3.2, Table 4). Adding interference from a colocated, unrelated job on the same server dropped utilization further to 37.5% (75.6 images/sec); interference from a job sharing the RDMA network dropped it to 36.5% (74.1 images/sec)5 (Table 4).
Worked derivation: what "47.7% of in-use GPU cycles wasted" costs at cluster scale
Philly's headline utilization loss is a percentage, not a GPU-hour count; putting it against a round, illustrative cluster size makes the cost concrete. This is a generic illustration only, not a claim about any specific real cluster's own measured occupancy -- no source in this course's map discloses a Philly-style GPU-occupancy utilization figure for a Slurm-, Ray-, or MAST-orchestrated cluster, and this derivation is not a statement about Llama 3's own reported 38-43% BF16 Model FLOPs Utilization4, which measures achieved-vs-peak compute throughput on GPUs that are already scheduled and running, a different quantity from whether a GPU is scheduled at all. Take a hypothetical 10,000-GPU cluster running continuously:
$$ 10{,}000 \text{ GPUs} \times 24 \text{ h/day} = 240{,}000 \text{ GPU-hours/day of allocated capacity} $$
At Philly's measured mean utilization of 52.32%5, roughly \(240{,}000 \times 0.5232 \approx 125{,}568\) of those GPU-hours perform useful work per day; the remaining 114,432 GPU-hours -- just under half the cluster's allocated capacity, every day -- sit idle on GPUs that are neither running another job nor available to be handed to one, because Section 2.2 already established that GPUs, unlike CPU or RAM, have no runtime mechanism to reclaim that gap without preempting the job holding it. That arithmetic is why Philly treats locality-driven fragmentation, not raw quota scarcity, as the harder problem for a next-generation scheduler to solve.
One further finding belongs in a scheduling module rather than Module 35's fault-tolerance treatment, because it is a submission-time, not a runtime, conclusion: 30.7% of jobs in the trace finished killed or unsuccessful rather than passed, and that 30.7% of jobs consumed roughly 55% of all GPU-time used during the collection period5 (§4, Table 6) -- failed jobs are, on average, disproportionately expensive, not cheap to discard. Philly's own design guidance draws the scheduling-relevant conclusion directly: "many failures ought to be caught early, well before they are scheduled on a larger shared cluster. This can be achieved by scheduling each incoming job on a small dedicated pool of servers or even using a single GPU"5 (§5) before admitting it to the full quota-and-locality machinery Sections 6.1-6.2 describe. That guidance, plus prioritizing locality even at a queueing-delay cost and isolating jobs on dedicated servers to limit interference5 (§5), are the paper's three design recommendations for next-generation DNN cluster schedulers -- recommendations aimed squarely at whichever scheduler sits underneath, not at replacing it.
7. Five systems, one axis: heartbeat loop vs. declarative reconciliation
Line the five systems up and the real fork is not "which one is best" -- it is a difference in what the control plane's loop actually does on every iteration. A Slurm-style controller polls or receives a heartbeat from every agent on a fixed cadence, updates its view of who is alive and what they are running, and dispatches newly-admitted work against that snapshot; the loop's unit of work is the heartbeat interval. A Borg-, Kubernetes-lineage-, or Ray-style controller instead holds a persistently stored desired state (Borg's Paxos-backed store; an analogous replicated store in Kubernetes; Ray's GCS) and runs a continuous reconciliation process that compares desired state against observed state and issues corrective actions -- schedule, reschedule, evict -- whenever the two diverge, with no fixed interval gating when a correction can happen.
| System | Origin | Control-plane shape | Admission unit | Native gang scheduling |
|---|---|---|---|---|
| Borg1 | Google, general-purpose | Continuous reconciliation (Borgmaster/Borglet) | Quota vector (any resource) at a priority | Not documented |
| Slurm2 | SchedMD, HPC batch | Periodic heartbeat (slurmctld/slurmd) | Exclusive/non-exclusive node-block, per duration | Implicit in exclusive allocation |
| Ray36 | Anyscale, distributed Python | Continuous reconciliation (driver/GCS) | Task/actor request against cluster resources | Placement groups: atomic, no relax-after-timeout |
| MAST4 | Meta, internal/proprietary | Not disclosed | Not disclosed | Not disclosed |
| Philly (YARN + custom placement)5 | Microsoft Research, GPU-specific layer | Continuous (YARN Fair Scheduler) | Per-virtual-cluster GPU quota | Explicit: timeout, backoff, relax-after-retry |
Philly's row is the one worth sitting with, because it does not fit cleanly into either camp. It is built on YARN, which is architecturally on the continuous-reconciliation side of the fork (a central resource manager negotiating with per-application masters, not a periodic-heartbeat batch allocator), yet it is the one system in this table with an explicit, documented, timeout-and-relax GPU locality algorithm -- a fallback neither Borg's scoring criteria nor Ray's atomic placement groups specify. That is the strongest evidence in this module for the claim the disagreement below develops fully: which side of the HPC-batch/cloud-native fork a scheduler sits on predicts almost nothing about whether it handles GPU training workloads well. What predicts that is whether someone built gang scheduling, locality-awareness, and a GPU-denominated quota on top of whichever control plane they started with -- and every real, production-documented answer in this module (Philly's custom layer over YARN, and by inference MAST's undisclosed-but-named-as-training-specific design over whatever Meta's internal substrate is) got there by adding exactly those features to a general-purpose base, not by picking a side of the fork and stopping.
Where experts disagree
The disagreement this module's source map names explicitly is HPC-batch scheduler versus cloud-native orchestrator for GPU training clusters213. Section 7's Philly evidence complicates a clean two-sided reading of it, so this section states three positions rather than two: the two the map names, plus the position Philly's own trace data forces onto the table.
A centralized heartbeat controller is the right shape for training
SchedMD's own architectural framing of Slurm2; the HPC supercomputing-center tradition Slurm grew out of
A large pretraining job is one synchronous, gang-scheduled unit that needs exclusive access to a fixed block of nodes for a fixed duration. Slurm's allocation model matches that shape exactly: one job, one allocation, monitored by a simple periodic heartbeat, arbitrated by one pending-work queue. There is no quota vector, no priority-band bookkeeping, and no continuous reconciliation loop to reason about -- the control plane does one thing and the failure modes are correspondingly easy to reason about.
- Minimal control-plane complexity; a heartbeat interval is trivial to monitor and alert on
- Exclusive-access allocation is a natural match for a single, monolithic, gang-scheduled job
- Decades of HPC operational maturity behind the same daemon model
- No documented general mechanism for fine-grained priority/quota arbitration across many concurrent tenants at once
- Heartbeat-interval-gated updates are structurally slower to react to a change than an event-driven reconciliation loop
Declarative reconciliation is the right shape, because training infrastructure is heterogeneous
Borg's cell/priority/quota architecture as Kubernetes' direct ancestor1; Ray's task/actor/object-store model, adopted by 8 of 16 surveyed RL training frameworks73
Modern training infrastructure is not one job type. A cluster runs pretraining, hyperparameter sweeps, RL rollout-and-train pipelines, and ad hoc debugging jobs side by side, at different priorities, with different resource shapes, some needing multi-service coordination (a rollout generator and a trainer exchanging weights) that a single exclusive-block allocation cannot express. A continuously-reconciling control plane with a real priority-and-quota model, or an actor-based scheduler like Ray's, handles that heterogeneity as a first-class case rather than a workaround.
- Priority bands and quota vectors handle many concurrent, differently-important tenants without a bespoke queueing scheme per case
- Continuous reconciliation reacts to state changes without waiting for a fixed interval
- Ray's actor model is the documented substrate under most surveyed RL post-training frameworks
- More moving parts: a persistent replicated store, a scheduler process, feasibility-and-scoring logic all have to be built and kept consistent
- Borg's own scoring criteria specify no GPU-aware locality algorithm; Ray's placement groups do gang-schedule atomically but refuse outright rather than waiting and relaxing the way Philly's algorithm does
The fork is second-order; ML-specific mechanisms are what actually determine outcomes
Philly's own production trace and design guidance5
Philly is built on YARN -- architecturally on the cloud-native side of the fork -- and still lost roughly half its allocated GPUs' cycles to waste, with fragmentation delay, not quota exhaustion, causing about 80% of all queueing time5. The generic scheduler underneath was never the bottleneck; the missing piece was gang scheduling, locality-aware placement with a bounded timeout and backoff, and a GPU-denominated quota per virtual cluster -- none of which came for free from YARN's Fair Scheduler, and none of which either Borg's or Slurm's own documentation specifies either. Whichever side of the fork an organization starts from, it still has to build these three mechanisms explicitly, and the evidence for how much they matter is a trace study, not an architectural argument.
- Directly evidenced by a real 96,260-job, 75-day production trace, not an architectural claim
- Explains why Philly (cloud-native lineage) and MAST (undisclosed, but purpose-built) both needed bespoke additions rather than an off-the-shelf answer
- Philly is one cluster, one company, one 2017 snapshot -- it does not by itself show whether Slurm-based clusters have the same fragmentation profile
- Doesn't resolve which base control-plane shape is cheaper to add these mechanisms to, only that both need them
For a single flagship pretraining run -- one job, exclusive access, no elasticity needed mid-run -- Slurm's simplicity is the right default, and it is telling that Meta's own MAST, built for exactly this workload at Llama-3 scale, is a bespoke system rather than a description of "we run declarative Kubernetes-style scheduling"4; the shape of the workload (one gang-scheduled unit) is closer to Slurm's allocation model than to Borg's multi-tenant priority arbitration. For a cluster running heterogeneous, concurrent, differently-prioritized training and RL workloads at once -- which is the normal state of any shared GPU fleet, not the exception -- the cloud-native camp's richer admission and priority model, or Ray's actor-based decoupling for RL specifically, earns its added complexity, which is why 8 of 16 surveyed RL frameworks build on Ray rather than shelling out to Slurm per job7. Philly's finding sits underneath both answers rather than choosing between them: whichever control plane a cluster runs, the GPU-specific mechanisms (gang scheduling, locality-bounded placement, GPU-denominated quota) have to be added deliberately, and Philly's own measurements are the best evidence available in this course's source map for how much is lost when they are missing or under-tuned -- about half the allocated GPU-cycles, and four-fifths of all queueing delay attributable to fragmentation rather than fair-share exhaustion5. What would settle the first question more decisively is a comparable trace study from a large Slurm-based training cluster, published with the same level of queueing-delay and utilization detail Philly discloses; no source in this course's map provides one.
References
- paperVerma, Abhishek, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, John Wilkes. Large-scale cluster management at Google with Borg. EuroSys '15. https://research.google.com/pubs/archive/43438.pdf. Accessed 2026-09-17. Used for: cell/Borgmaster/Borglet architecture (§3.1), scheduler feasibility and scoring (§3.2), priority bands and quota-checked-at-admission (§2.5), median cell size and workload description (§2.1-§2.2).
- officialSchedMD. Slurm Workload Manager Overview. Live documentation. https://slurm.schedmd.com/overview.html. Accessed 2026-09-17. Used for: Slurm's three key functions, the slurmctld/slurmd architecture and "remote shell" analogy, and the optional slurmdbd/slurmrestd daemons.
- officialAnyscale (Ray project). Ray Core Walkthrough. docs.ray.io, live documentation. https://docs.ray.io/en/latest/ray-core/walkthrough.html. Accessed 2026-09-17. Used for: tasks/actors/objects as Ray's essential primitives, actor state and serial execution ordering, and the object store's automatic-storage/ray.put()/reference-passing model.
- paperLlama Team, Meta AI. The Llama 3 Herd of Models. arXiv:2407.21783, 2024. https://arxiv.org/pdf/2407.21783. Accessed 2026-09-17. Used for: MAST named as Meta's global-scale training scheduler and the 16K-H100/Grand Teton training-run context (§3.3.1) -- the one sentence of MAST detail this module cites.
- paperJeon, Myeongjae, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, Fan Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC '19. https://www.usenix.org/system/files/atc19-jeon.pdf. Accessed 2026-09-18. Used for: Philly's YARN/HDFS architecture and virtual-cluster GPU quota (§2.2-§2.3), gang scheduling and locality-aware placement with timeout/backoff (§2.3), fair-share vs. fragmentation queueing delay (§3.1, Table 2), GPU utilization and the ResNet-50 colocation experiment (§3.2, Table 4), job-outcome/GPU-time split (§4, Table 6), and design guidance for future schedulers (§5). Fetched directly this session; the course's source map had recorded this source as fetched:false with zero facts.
- officialAnyscale (Ray project). Placement Groups. docs.ray.io, live documentation. https://docs.ray.io/en/latest/ray-core/scheduling/placement-group.html. Accessed 2026-09-18. Used for: placement groups as atomic gang scheduling, the bundle definition, and the STRICT_PACK/PACK/STRICT_SPREAD/SPREAD strategies.
- blogHugging Face. Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries. hf.co/blog, 2026. https://huggingface.co/blog/async-rl-training-landscape. Accessed 2026-09-17. Used for: the count of 8 of 16 surveyed RL training frameworks orchestrating with Ray.