System Design · Module 7

Distributed File and Object Storage: GFS, HDFS, S3, and Tectonic

Every database in this course — Bigtable's tablets, HDFS's warehouse tables, a training job's checkpoint — ultimately writes its bytes onto something that looks like GFS: a single metadata authority that never touches the data path, and a large number of dumb, replicated data servers that do. This module derives that file-layer design from GFS's own paper, follows it into HDFS and today's object stores, and ends at Tectonic, the disaggregated storage fabric that gives Meta's 24K-GPU training clusters 2 TB/s of sustained throughput.

Course 40 min read 8 sources As of 2026-09-17 Prerequisite: Module 3
GFS architecture: single master holding metadata only, chunkservers holding replicated 64MB chunks, clients on the data path only to chunkservers ApplicationGFS client Master namespace, file→chunk map, chunk locations (in memory only) op log + checkpoint on disk Chunkserver A64MB chunk, Linux FSplain local files Chunkserver B64MB chunk, Linux FSsame chunk, 2nd replica Chunkserver C64MB chunk, Linux FSsame chunk, 3rd replica metadata RPC chunk handle + replica locations data HeartBeat: state, chunk lists, instructions Clients never read or write file data through the master — only chunkservers carry bytes control (metadata) data (chunk bytes)
Figure 1. GFS's single master holds three kinds of metadata in memory and answers only metadata queries; the client then talks directly to chunkservers for every byte of actual data. Chunkservers exchange HeartBeat messages with the master to report their chunk inventory and receive instructions1.

1. The file layer beneath every database

Module 3 introduced GFS as the example of leader-based, lease-ordered replication and walked through its write path in detail. This module returns to GFS as a different kind of object: not an instance of a general replication pattern, but the specific file-storage layer that a distributed filesystem's own concerns — namespace, chunking, metadata scale, garbage collection — sit on top of, and that every structured store in this course (Bigtable's tablets, Module 3/4) and every AI-training checkpoint (Module 33) ultimately writes bytes onto.

GFS's own numbers frame the scale this design targets: "the largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients"1 — from one paper, one cluster, in 2003. The architectural choice that makes a thousand-machine cluster manageable by one metadata authority is stated directly in the paper: "having a single master vastly simplifies our design and enables the master to make sophisticated chunk placement and replication decisions using global knowledge. However, we must minimize its involvement in reads and writes so that it does not become a bottleneck. Clients never read and write file data through the master. Instead, a client asks the master which chunkservers it should contact"1. Figure 1 draws exactly that split: a metadata RPC to the master (dashed, control-only) followed by a direct data connection to whichever chunkserver holds the bytes (solid, on the actual data path). Everything in this section follows from taking that split seriously — what has to live in the master's memory to make it fast, and what the master can safely never touch.

1.1 What the master keeps in memory, and what it never persists

"The master stores three major types of metadata: the file and chunk namespaces, the mapping from files to chunks, and the locations of each chunk's replicas. All metadata is kept in the master's memory"1. Only two of those three are ever written to disk. The namespace and the file-to-chunk mapping "are also kept persistent by logging mutations to an operation log stored on the master's local disk and replicated on remote machines"1 — durable, because losing them loses the filesystem. Chunk locations get no such treatment: "the master does not keep a persistent record of which chunkservers have a replica of a given chunk. It simply polls chunkservers for that information at startup"1, and keeps itself current afterward "because it controls all chunk placement and monitors chunkserver status with regular HeartBeat messages"1.

The paper's own justification for that asymmetry is worth reading exactly, because it is a load-bearing systems-design argument, not an implementation detail: "a chunkserver has the final word over what chunks it does or does not have on its own disks. There is no point in trying to maintain a consistent view of this information on the master because errors on a chunkserver may cause chunks to vanish spontaneously... or an operator may rename a chunkserver"1. Persisting a fact that a remote machine can silently invalidate at any time buys nothing but a stale copy and a reconciliation protocol to keep it fresh; polling the one machine that actually knows the truth removes the reconciliation problem by removing the redundant copy of the fact. Table 1 lays out the resulting split.

Table 1. The master's three metadata types and where each one actually lives1.
MetadataIn master's memoryPersisted on disk (op log)How the master gets it
File & chunk namespaceyesyesclient mutations, logged
File → chunk mappingyesyesclient mutations, logged
Chunk replica locationsyesnopolled at startup, refreshed via HeartBeat

Everything in memory is cheap to keep small. The namespace is not a per-directory listing structure at all: "GFS logically represents its namespace as a lookup table mapping full pathnames to metadata... with prefix compression, this table can be efficiently represented in memory"1, and each node in that lookup table carries a read-write lock so that, for example, creating /home/user/foo and snapshotting /home/user to /save/user serialize correctly against each other without either operation needing a full-tree lock1.

1.2 The operation log, checkpointing, and recovery

"The operation log contains a historical record of critical metadata changes. It is central to GFS. Not only is it the only persistent record of metadata, but it also serves as a logical time line that defines the order of concurrent operations"1. Because every mutation must survive a crash before a client is told it succeeded, the master "replicate[s] it on multiple remote machines and respond[s] to a client operation only after flushing the corresponding log record to disk both locally and remotely," and batches several log records together before each flush to keep that cost off the critical path of every single operation1.

Replaying an ever-growing log from the beginning would make every master restart slower than the last, so the master checkpoints: "the master checkpoints its state whenever the log grows beyond a certain size so that it can recover by loading the latest checkpoint from local disk and replaying only the limited number of log records after that. The checkpoint is in a compact B-tree like form that can be directly mapped into memory and used for namespace lookup without extra parsing"1. Checkpointing itself never blocks incoming mutations: "the master's internal state is structured in such a way that a new checkpoint can be created without delaying incoming mutations. The master switches to a new log file and creates the new checkpoint in a separate thread"1, and a failure mid-checkpoint is harmless because "the recovery code detects and skips incomplete checkpoints"1. Recovery time is therefore bounded by one checkpoint load plus a bounded tail of log records — for "a cluster with a few million files" the paper reports a checkpoint takes about a minute to build1. GFS also runs shadow masters that replay the same operation log to serve stale, read-only metadata during a master outage, which the paper mentions only in passing since a stale read is exactly the kind of relaxed guarantee GFS's applications were built to tolerate.

2. The 64MB chunk-size decision

"Chunk size is one of the key design parameters. We have chosen 64 MB, which is much larger than typical file system block sizes"1 — a typical Linux filesystem block is 4KB, so GFS's chunk is roughly 16,000× larger than the unit most engineers think of as "a block." Each chunk replica is stored as an ordinary Linux file, extended only as needed, and "lazy space allocation avoids wasting space due to internal fragmentation, perhaps the greatest objection against such a large chunk size"1 — a 1KB file does not actually consume 64MB of disk; it consumes 1KB plus the metadata overhead of one chunk.

2.1 Three advantages of choosing it large

The paper gives three concrete reasons, and each is a different kind of cost avoided. First, client-master traffic: "it reduces clients' need to interact with the master because reads and writes on the same chunk require only one initial request to the master for chunk location information... the client typically asks for multiple chunks in the same request and the master can also include the information for chunks immediately following those requested"1. Second, connection reuse: "since on a large chunk, a client is more likely to perform many operations on a given chunk, it can reduce network overhead by keeping a persistent TCP connection to the chunkserver over an extended period of time"1 — the cost of a TCP handshake is amortized over many operations instead of paid per small read. Third, and the one that determines everything else in Section 1: "it reduces the size of the metadata stored on the master. This allows us to keep the metadata in memory, which in turn brings other advantages"1 — chunk size and in-memory metadata are not two separate design choices; the second is a direct consequence of the first.

2.2 The cost: hot spots on small files, and how GFS mitigated one in production

The paper names the failure mode directly: "a small file consists of a small number of chunks, perhaps just one. The chunkservers storing those chunks may become hot spots if many clients are accessing the same file"1, and then gives a real production incident rather than a hypothetical one: "hot spots did develop when GFS was first used by a batch-queue system: an executable was written to GFS as a single-chunk file and then started on hundreds of machines at the same time. The few chunkservers storing this executable were overloaded by hundreds of simultaneous requests. We fixed this problem by storing such executables with a higher replication factor and by making the batch-queue system stagger application start times"1. This is a genuinely important pattern to recognize outside GFS's own history: any storage system that treats replica count as fixed per file, rather than adaptive to fan-in, will reproduce this failure whenever one small object becomes suddenly popular — the fix is always some version of "add more copies of the hot thing," whether that is GFS raising a namespace region's replication factor, a CDN's edge cache (Module 2) absorbing the read fan-out instead, or an inference platform replicating a hot model shard across more GPUs.

2.3 Worked derivation: how much storage a byte of master RAM buys

Section 1.1 established the number that makes the whole design work: "the master maintains less than 64 bytes of metadata for each 64 MB chunk. Most chunks are full because most files contain many chunks, only the last of which may be partially filled"1. That ratio — 64 bytes of metadata per 64 MB of actual storage — is worth turning into a concrete number, because "the metadata fits in memory" is an abstract claim until you compute what it actually buys.

Take one gigabyte of master RAM devoted to chunk metadata. At under 64 bytes per chunk record:

$$ \text{chunks addressable} = \frac{1\ \text{GiB}}{64\ \text{B}} = \frac{2^{30}\ \text{B}}{2^{6}\ \text{B}} = 2^{24} \approx 16.8\times10^{6}\ \text{chunks} $$

and each of those chunks is worth up to 64 MB of storage, so:

$$ \text{storage addressable} = 2^{24}\ \text{chunks} \times 2^{26}\ \text{B/chunk} = 2^{50}\ \text{B} = 1\ \text{PiB} $$

One gigabyte of master memory buys roughly one petabyte of addressable chunk storage — a million-to-one leverage ratio, before namespace metadata (also "typically less then 64 bytes per file... because it stores file names compactly using prefix compression"1) is even counted. This is exactly the calculation behind the paper's own closing remark on the topic: "if necessary to support even larger file systems, the cost of adding extra memory to the master is a small price to pay for the simplicity, reliability, performance, and flexibility we gain by storing the metadata in memory"1. It is also the calculation that eventually breaks: a single master's memory is finite, and Section 7 covers what Meta's Tectonic does once a single filesystem instance needs to address exabytes rather than petabytes.

3. Replication factor, placement, rebalancing, and garbage collection

Module 3 covers the lease mechanism that orders mutations to a chunk's replicas — the master grants one chunkserver a renewable lease, and that primary assigns the serial order every replica follows1. This section covers the master's other, non-ordering jobs over those same replicas: deciding how many copies of a chunk should exist, where they should live, when to make more of them, and how to reclaim space once a file is deleted. None of this is Module 3's topic; all of it is specific to GFS as a file-storage system rather than as an instance of leader-based replication in the abstract.

3.1 Default replication factor, and the per-region override

"For reliability, each chunk is replicated on multiple chunkservers. By default, we store three replicas, though users can designate different replication levels for different regions of the file namespace"1. Three is a default, not a constant — the batch-queue executable hot-spot fix in Section 2.2 was exactly an application raising that per-region override above three for one specific, unusually popular file.

3.2 Placement: spreading replicas across racks, deliberately

Where those three replicas physically land is not left to chance. "The chunk replica placement policy serves two purposes: maximize data reliability and availability, and maximize network bandwidth utilization. For both, it is not enough to spread replicas across machines, which only guards against disk or machine failures... We must also spread chunk replicas across racks. This ensures that some replicas of a chunk will survive and remain available even if an entire rack is damaged or offline"1. The explicit trade-off the paper accepts: cross-rack placement means "write traffic has to flow through multiple racks, a tradeoff we make willingly," in exchange for read traffic — the far more common operation — being able to "exploit the aggregate bandwidth of multiple racks"1.

3.3 Re-replication and rebalancing

"The master re-replicates a chunk as soon as the number of available replicas falls below a user-specified goal. This could happen for various reasons: a chunkserver becomes unavailable, it reports that its replica may be corrupted, one of its disks is disabled because of errors, or the replication goal is increased"1. Re-replication is not first-come-first-served across all under-replicated chunks; the master prioritizes: "we give higher priority to a chunk that has lost two replicas than to a chunk that has lost only one. In addition, we prefer to first re-replicate chunks for live files as opposed to chunks that belong to recently deleted files... we boost the priority of any chunk that is blocking client progress"1. To copy a chunk, "the master picks the highest priority chunk and 'clones' it by instructing some chunkserver to copy the chunk data directly from an existing valid replica"1, throttled per-cluster and per-chunkserver so that repair traffic cannot itself overwhelm the machines being repaired. Independently of failures, "the master also rebalances replicas periodically: it examines the current replica distribution and moves replicas for better disk space and load balancing... gradually fill[ing] up a new chunkserver rather than instantly swamp[ing] it with new chunks and the heavy write traffic that comes with them"1.

3.4 Garbage collection: lazy, not immediate

Deleting a file in GFS does not free the underlying chunks synchronously. "When a file is deleted by the application, the master logs the deletion immediately... However instead of reclaiming resources immediately, the file is just renamed to a hidden name that includes the deletion timestamp. During the master's regular scan of the file system namespace, it removes any such hidden files if they have existed for more than three days (the interval is configurable). Until then, the file can still be read under the new, special name and can be undeleted by renaming it back to normal"1 — a built-in undelete window, essentially free, as a side effect of choosing to reclaim lazily rather than synchronously. The chunk-level cleanup follows the same lazy pattern and closes the loop back to Section 1's decision not to persist chunk locations on the master: "we can easily identify all references to chunks: they are in the file-to-chunk mappings maintained exclusively by the master. We can also easily identify all the chunk replicas: they are Linux files under designated directories on each chunkserver. Any such replica not known to the master is 'garbage'"1. A regular HeartBeat exchange — the same mechanism that keeps the master's chunk-location knowledge current — is what actually deletes the orphaned bytes: the chunkserver reports what it has, the master replies with the identities of chunks no longer in its metadata, and "the chunkserver is free to delete its replicas of such chunks"1. The paper's own assessment of trading a harder real-time delete protocol for this lazier one: "although distributed garbage collection is a hard problem that demands complicated solutions in the context of programming languages, it is quite simple in our case"1 — because there is exactly one authority (the master) for what should exist, and exactly one authority (each chunkserver) for what does exist, reconciling the two lazily is strictly simpler than trying to keep them synchronously consistent.

4. What the write pipeline costs once the lease is granted

Module 3 derives GFS's closed-form estimate for pushing a write of B bytes to R replicas, \( t_{\text{ideal}} = B/T + RL \), and works the arithmetic for a 1MB write across three replicas on GFS's 100 Mbps links to get "1 MB can ideally be distributed in about 80 ms"1. What that module's stepper diagram does not show is where the time actually goes once you stop treating the pipeline as an instantaneous hop count and instead draw it against a clock. Figure 2 does that: three replicas, one client, and the moment each one starts and finishes receiving the same 1MB write.

Timeline of a pipelined 1MB write across 3 replicas: each hop adds one latency L, then all three transmissions overlap once the pipeline is full time (ms) — not to scale near t=0, L < 1 ms while B/T ≈ 80 ms 0 ≈80 Client→R1 push B bytes at rate T ≈ 80 ms R1→R2 +L R2→R3 +L total ≈ B/T + 2L (2 forwarding hops for R=3) ≈ 80 ms + <2 ms R1 starts at t=0 R2 starts ~L later R3 starts ~2L later, all finish within ~80 ms of each other
Figure 2. The three hops in a chain overlap almost completely: each replica starts receiving one latency L after the previous one, but all three are transmitting for nearly the full 80 ms because the transfer itself (B/T) vastly outlasts the staggered start. GFS's own network figures — 100 Mbps links, sub-millisecond L — make the second term of B/T + RL negligible next to the first1; see Module 3 for the full derivation.

The practical upshot for anyone designing a replicated write path rather than reading about one: pipelining a chain only pays off when B/T dominates RL, which is exactly the bandwidth-bound regime a 64MB-chunk, large-sequential-write filesystem lives in. A system moving small, frequent, latency-sensitive writes — a coordination service's log entries, for instance — sits in the opposite regime, where RL dominates and a flatter, parallel fan-out beats a chain. Dynamo's quorum write (Module 3, Section 3.1) makes exactly that opposite choice, fanning out to N−1 replicas in parallel rather than chaining through them.

5. Atomic record append: the write primitive for concurrent producers

Module 3 introduced record append in one paragraph and explicitly deferred its full mechanics to this module — this is that deferred treatment.

5.1 Why a traditional write cannot serve concurrent appenders

"In a traditional write, the client specifies the offset at which data is to be written. Concurrent writes to the same region are not serializable: the region may end up containing data fragments from multiple clients"1. If two clients both believe they are writing at the current end-of-file offset — because both read that offset a moment ago and neither has heard about the other — their writes race for the same bytes, and whichever one's data lands last simply overwrites the other's, or the two interleave into corrupted fragments. A distributed lock manager could serialize the offset assignment itself, but GFS's actual producer/consumer workloads — "many clients on different machines append to the same file concurrently... to serve as multiple-producer/single-consumer queues or contain merged results from many different clients"1 — do not need a particular relative order between two concurrent appends, only a guarantee that each one lands somewhere, intact, exactly once as one continuous unit. That weaker requirement is what makes a much cheaper primitive than general write serialization possible.

5.2 The mechanism: GFS picks the offset, pads at chunk boundaries, and may duplicate on retry

"GFS provides an atomic append operation called record append. In a record append, the client specifies only the data. GFS appends it to the file at least once atomically... at an offset of GFS's choosing and returns that offset to the client"1. Mechanically, record append reuses almost all of the lease-ordered write path Module 3 already covers — data is still pushed to every replica first, the write request still goes only to the primary — with one extra piece of logic at the primary: "the client pushes the data to all replicas of the last chunk of the file[,] then sends its request to the primary. The primary checks to see if appending the record to the current chunk would cause the chunk to exceed the maximum size (64 MB). If so, it pads the chunk to the maximum size, tells secondaries to do the same, and replies to the client indicating that the operation should be retried on the next chunk. (Record append is restricted to be at most one-fourth of the maximum chunk size to keep worst-case fragmentation at an acceptable level.) If the record fits within the maximum size, which is the common case, the primary appends the data to its replica, tells the secondaries to write the data at the exact offset where it has, and finally replies success to the client"1.

The failure case is the part worth being precise about, because it is the reason record append's guarantee is "at least once" rather than "exactly once": "if a record append fails at any replica, the client retries the operation. As a result, replicas of the same chunk may contain different data possibly including duplicates of the same record in whole or in part. GFS does not guarantee that all replicas are bytewise identical. It only guarantees that the data is written at least once as an atomic unit"1. Figure 3 draws the two-client version of this: both clients append concurrently to the same file's last chunk, each gets back a distinct, GFS-chosen offset for its own record, and one client's retry after a partial failure leaves a duplicate and some padding behind that a reader has to know how to skip.

Two concurrent clients calling RecordAppend on the same file; GFS assigns each a distinct offset, and a retried append leaves padding and a duplicate Client P Client Q Primary (last chunk) append("recP") append("recQ") Chunk fits both: primary serializes the two, assigns each an offset off 12,400: recP off 12,600: recQ chunk has room — no padding needed Client Q's write fails on one secondary — Q retries, GFS assigns a new offset off 12,400: recP recQ (partial) padding off 12,940: recQ duplicate Reader skips the failed partial fragment and the padding using per-record checksums and unique IDs1
Figure 3. Two clients append concurrently; GFS's primary serializes the two requests and returns each client a distinct offset it did not choose. Bottom: a secondary-write failure forces client Q to retry, which leaves a partial fragment, padding, and a duplicate record behind — the price of "at least once" append semantics1.

5.3 What "defined" actually means: the region-state table

GFS states its consistency guarantee per mutation type, and the paper's own table is precise enough to reproduce directly rather than paraphrase:

Table 2. File region state after a mutation, by outcome and by mutation type1.
OutcomeWriteRecord append
Serial successdefineddefined, interspersed with inconsistent
Concurrent successesconsistent but undefineddefined, interspersed with inconsistent
Failureinconsistent

"A region is consistent if all clients will always see the same data, regardless of which replicas they read from. A region is defined after a data mutation if it is consistent and clients will see what the mutation writes in its entirety"1. Record append's "defined, interspersed with inconsistent" cell is the table's most important entry: even when every append individually succeeds, the padding and any partial fragments left behind by retries (Figure 3) sit in the file as inconsistent bytes between the well-defined records. GFS pushes the burden of dealing with that onto the application deliberately: "each record prepared by the writer contains extra information like checksums so that its validity can be verified. A reader can identify and discard extra padding and record fragments using the checksums"1 — the filesystem guarantees atomicity of each record, not cleanliness of the file as a byte stream, and the two are different guarantees with different costs to provide.

6. HDFS: GFS's open-source descendant

The Hadoop Distributed File System reproduces GFS's master/data-plane split almost exactly, with the names changed: "HDFS has a master/slave architecture. An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients... There are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on... [and are] responsible for serving read and write requests from the file system's clients"2 — NameNode for master, DataNode for chunkserver. This module names that correspondence rather than re-deriving it: everything Sections 1–5 established about why a single metadata authority stays off the data path, why the unit of storage is large, and why replica location is not treated as durable ground truth, carries over to HDFS's NameNode/DataNode split with the numbers below adjusted.

The numbers themselves have moved on from GFS's 2003 defaults. HDFS's own documentation states its default unit size grew past GFS's original figure: "a typical block size used by HDFS is 128 MB. Thus, an HDFS file is chopped up into 128 MB chunks"2 — double GFS's 64MB, the same underlying trade-off (Section 2) pushed further as disks, networks, and file sizes all grew. The default replication factor is unchanged: "for the common case, when the replication factor is three, HDFS's placement policy is to put one replica on the local machine if the writer is on a datanode... another replica on a node in a different (remote) rack, and the last on a different node in the same remote rack"2 — the same cross-rack-diversity logic Section 3.2 described for GFS, expressed as a concrete three-replica placement recipe: one local, two on a second rack (so a single rack failure still leaves one surviving copy off-rack), rather than three racks each holding one.

What HDFS added that this module's scope does not need to re-derive is the ecosystem built around that same file layer: YARN for resource scheduling, the Hadoop MapReduce runtime (Module 9's topic), and HBase as an HDFS-hosted structured store analogous to Bigtable riding on GFS. The single-NameNode design is also the specific limitation Section 8's disagreement returns to directly — it is the reason Meta needed "tens of HDFS clusters per datacenter" for its data-warehouse tenant before consolidating onto Tectonic.

7. Object storage's consistency evolution: S3 from eventual to strong

Everything through Section 6 shares one structural assumption: a hierarchical namespace of directories and files, each file made of ordered chunks or blocks. Object storage drops that structure entirely. Amazon S3 "is an object storage service"3 in which "an object is a file and any metadata that describes the file. A bucket is a container for objects... Each object has a key (or key name), which is the unique identifier for the object within the bucket"3. There is no directory tree the storage system itself understands — a key like photos/puppy.jpg is one flat string, and any hierarchy a client perceives is a convention in how keys happen to be named, not a structure S3 walks the way GFS walks its prefix-compressed namespace table (Section 1.1). Durability is engineered independently of that flat key space: S3 Standard "is designed to exceed 99.999999999% (11 nines) data durability" and "stores data redundantly across a minimum of 3 Availability Zones by default"5 — the same replicate-across-failure-domains idea as GFS's rack diversity (Section 3.2), just applied at the datacenter-region scale rather than the rack scale.

7.1 What changed on December 1, 2020

Object storage's consistency story has a specific before-and-after, not just a spectrum. AWS's own announcement states the change directly: "effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent"4 — the word "now" marking a transition, since the reason this needed announcing at all is that S3 had previously run with a materially weaker model for overwrite PUTs and DELETEs, weak enough that customers built and depended on external workarounds — AWS's own EMRFS Consistent View and the open-source S3Guard among them — specifically to paper over the gap between "the write succeeded" and "every reader now sees it"4. AWS's stated motivation for closing that gap ties directly back to this course's batch-processing lineage (Module 9): "customers are using big data analytics applications like Apache Spark and Apache Hadoop that often require access to an object immediately after a write"4 — an object store sitting underneath a Spark job behaves badly if the job's own reducer cannot reliably see the mapper's output the moment the mapper's write call returns.

What makes this a genuinely unusual transition, rather than the ordinary consistency-for-availability trade every PACELC-framed system (Module 5) makes, is the claimed price: "there's no impact on performance, you can update an object hundreds of times per second if you'd like, and there are no global dependencies"4. Today's documentation states the resulting guarantee plainly: "Amazon S3 provides strong read-after-write consistency for PUT and DELETE requests of objects in your Amazon S3 bucket in all AWS Regions... Updates to a single key are atomic. For example, if you make a PUT request to an existing key from one thread and perform a GET request on the same key from a second thread concurrently, you will get either the old data or the new data, but never partial or corrupt data"3. Two edges of that guarantee are worth stating precisely because they are easy to over-generalize from "S3 is strongly consistent": updates are key-based only — "there is no way to make atomic updates across keys"3 — and two simultaneous PUTs to the same key still resolve by last-writer-wins with no locking, so "if this is an issue, you must build an object-locking mechanism into your application"3. Bucket-level configuration (creating/deleting a bucket, first enabling versioning) is explicitly carved out as still only eventually consistent3 — the strong guarantee is about object data and its per-object metadata, not about the bucket namespace itself.

7.2 Why this is a different move than Dynamo's

Dynamo (Module 3) chose eventual consistency deliberately and permanently, as an availability trade its designers judged worth the cost for an always-writeable shopping cart. S3's history looks superficially similar — it, too, ran eventually consistent for years — but the 2020 change is not S3 discovering a new trade-off; it is S3's operators concluding that the specific technique they had been using to achieve availability and low latency could be replaced by a different internal mechanism (an index design AWS has not published in the depth GFS's or Dynamo's papers were published in) that removes the eventual-consistency window without giving back the availability or latency it originally bought. That is a genuinely different claim from PACELC's "you must choose": it says the earlier choice was an implementation limitation, not a law of the trade-off, for this particular access pattern (single-key reads and writes, no cross-key transactions). Section 9's disagreement returns to whether that claim generalizes.

8. Tectonic: disaggregated metadata and a flat chunk store at exabyte scale

Before Tectonic, Meta ran separate, specialized storage systems for its two largest tenants: "blob storage was spread across Haystack and f4... data warehouse was spread across many HDFS instances"7. Each specialized system stranded resources the other tenant could have used — "Haystack became IOPS-bound... the spare disk capacity resulted in Haystack's effective replication factor increasing to 5.3×. In contrast, f4 had an abundance of IOPS from storing a large volume of less-frequently-accessed data"7 — and the warehouse side hit a ceiling this module's Section 6 already named: "HDFS clusters are limited in size because they use a single machine to store and serve metadata. As a result, we needed tens of HDFS clusters per datacenter to store analytics data. This was operationally inefficient... [and] distributing datasets among the HDFS clusters created a two-dimensional bin-packing problem"7.

8.1 Chunk Store, Metadata Store, and why metadata is hash-partitioned rather than range-partitioned

Tectonic's answer is to separate data storage from filesystem logic entirely and scale each independently. "To scale metadata, Tectonic disaggregates the filesystem metadata into independently-scalable layers, similar to ADLS. Unlike ADLS, Tectonic hash-partitions each metadata layer rather than using range partitioning. Hash partitioning effectively avoids hotspots in the metadata layer. Combined with Tectonic's highly scalable chunk storage layer, disaggregated metadata allows Tectonic to scale to exabytes of storage and billions of files"7. Concretely, the Metadata Store sits as three layers — Name, File, and Block — on top of a general-purpose, horizontally-scalable key-value store, and the Chunk Store beneath it is "a flat, distributed object store for chunks, the unit of data storage in Tectonic. Chunks make up blocks, which in turn make up files"7. "Flat" is doing the same job here that a hash-partitioned ring did for Dynamo (Module 3): "the number of chunks stored grows linearly with the number of storage nodes... it is oblivious to higher-level abstractions like blocks or files; these abstractions are constructed by the Client Library using the Metadata Store"7.

Tectonic architecture: a Client Library orchestrating a layered Metadata Store over a key-value store, and a flat Chunk Store of storage nodes ClientLibrary API calls → RPCs Metadata Store Name layer File layer Block layer Key-value store hash-partitioned, not range Chunk Store flat object store storage nodes grows linearly with node count no file/block concept Per-block durability is tunable Reed-Solomon RS(r,k) encoding (space-efficient) or straight replication (fast, more space) — picked per tenant
Figure 4. Tectonic separates the filesystem logic (Name/File/Block layers over a hash-partitioned key-value store) from the raw storage layer (a flat Chunk Store), and lets each tenant tune per-block durability independently7.

Durability is a per-block, per-tenant choice rather than a filesystem-wide constant: "blocks are either Reed-Solomon (RS) encoded or replicated for durability. For RS(r,k) encoding, the block data is split into r equal chunks... and k parity chunks are generated from the data chunks. For replication, data chunks are the same size as the block and multiple copies are created. Chunks in a block are stored in different fault domains (e.g., different racks) for fault tolerance"7. That is GFS's rack-diversity placement policy (Section 3.2) generalized to a system where the cost of extra durability is chosen per workload instead of fixed at three whole copies for everyone.

8.2 Tectonic at training-cluster scale

Meta's Llama 3 training infrastructure paper gives Tectonic's scale numbers in the specific context this course's Part D returns to: "Tectonic, Meta's general-purpose distributed file system, is used to build a storage fabric for Llama 3 pre-training. It offers 240 PB of storage out of 7,500 servers equipped with SSDs, and supports a sustainable throughput of 2 TB/s and a peak throughput of 7 TB/s"6. The workload that stresses that fabric is not steady streaming but a sharp, synchronized burst: "checkpointing saves each GPU's model state, ranging from 1 MB to 4 GB per GPU"6, and every GPU in a training job checkpoints at nearly the same moment, so the fabric has to absorb a short, cluster-wide spike rather than a smooth average rate — the difference between the 2 TB/s sustainable figure and the 7 TB/s peak figure is exactly the headroom that spike needs. Module 33 builds directly on this: the checkpoint-storage design question there is how a training job's I/O pattern (synchronized, bursty, occasionally huge single objects) maps onto the tunable durability and flat scaling this section just established, not a re-derivation of what Tectonic is.

9. Layering: chunks and blocks, structured stores, applications

Every system this module has covered sits at the bottom of a stack, not the whole stack. Bigtable stores its data "on GFS using the immutable SSTable file format"8 — a tablet server never manages raw disk blocks itself; it manages sorted, immutable SSTable files, and those files are just GFS files subject to everything in Sections 1–5: 64MB chunks, a lease-ordered primary per chunk, three-way rack-diverse replication. Module 3 and Module 4 cover what a tablet and an SSTable actually are; this module's business is only the layer underneath them.

Three layers: chunk/block/file storage at the bottom, a structured store built from immutable sorted files in the middle, applications on top File / chunk / object layer GFS · HDFS · Tectonic Chunk Store · S3 (this module) Structured store Bigtable tablets / SSTables on GFS-style files (Module 3, Module 4) — out of scope here Application: search index, Gmail, training checkpoint reader
Figure 5. Structured stores are a layer, not a replacement, on top of a file or chunk store — Bigtable's SSTables ride on GFS files exactly as this module describes them8; the application layer never sees a chunk or a lease directly.

Object storage collapses that picture slightly differently: S3 has no equivalent of a tablet server sitting between the application and the object store in the way Bigtable sits between an application and GFS — an application talks to S3's flat key space directly, and anything resembling a structured, sorted, indexable layer (a data lake table format, a search index) is built by the application or by a separate service reading and writing S3 objects, not by S3 itself. Tectonic sits in between: its Metadata Store gives the Client Library enough structure (a real file hierarchy, not just flat keys) that Meta's data-warehouse tenant can keep using file-oriented compute engines — Presto, Spark, and the training pipelines this course's Part D covers — with only the storage backend swapped from HDFS to Tectonic, while its blob-storage tenant uses that same fabric with a flatter, blob-oriented access pattern closer to S3's. Table 3 places the four systems this module covered side by side.

Table 3. GFS, HDFS, Tectonic, and S3 compared on the axes this module derived12763.
SystemMetadata modelStorage unitDefault replicationNamespaceConsistency
GFSsingle master, in-memory64 MB chunk3×, per-region overridehierarchical, prefix-compressed tabledefined / consistent / inconsistent per Table 2
HDFSsingle NameNode, in-memory128 MB block3×, rack-awarehierarchical (POSIX-like)write-once, same GFS-style regions
Tectonichash-partitioned KV, 3 layerschunk → block → file3× replication or RS(r,k), per tenanthierarchical, multitenantstrong within Metadata Store's KV layer
S3internal, undisclosedobject (any size), flat key≥3 AZs, 11 nines durabilityflat key spacestrong read-after-write since 20204

Where experts disagree

This module's sources do not disagree about mechanism — GFS, HDFS, S3, and Tectonic all publish the numbers they publish without contradicting each other. The disagreement is about which of three structural choices is correct, and the answer this module's own sources give is explicitly conditional on scale and workload shape, which is worth stating plainly rather than picking a single winner.

A single master with a hierarchical namespace is the right default

GFS's and HDFS's own design position12

GFS's paper argues directly for centralizing metadata in one place: "having a single master vastly simplifies our design"1, and backs the claim with the arithmetic in Section 2.3 — a single machine's memory addresses petabytes of chunk storage at under 64 bytes of metadata per chunk. HDFS kept the identical single-NameNode design for its entire multi-decade production life at Meta, Yahoo, and elsewhere. A hierarchical namespace also gives applications real semantics — atomic rename, cheap recursive listing, POSIX-adjacent tooling — that a flat key space does not.

  • One authority for metadata means no distributed-consensus protocol is needed just to answer "which chunkserver has this data."
  • In-memory metadata makes every master operation fast, and the memory cost per unit of storage is genuinely tiny (Section 2.3).
  • A real directory hierarchy supports atomic rename and cheap subtree operations that flat object stores cannot offer natively.
  • A single machine's memory and request-handling capacity is a hard ceiling; Meta hit it directly and needed "tens of HDFS clusters per datacenter" to keep operating past it7.
  • Multitenancy across many independent single-master clusters recreates exactly the stranded-resource problem Section 8 describes for pre-Tectonic Meta.

Metadata must be disaggregated and hash-partitioned past exabyte, multitenant scale

Meta's Tectonic team, against both single-master GFS/HDFS and range-partitioned alternatives7

Tectonic's own stated design goal was explicitly to remove the single-master ceiling: "to scale metadata, Tectonic disaggregates the filesystem metadata into independently-scalable layers"7. It goes one step further than "disaggregate" alone by choosing hash partitioning specifically over the range-partitioned alternative it names in its own paper (Azure Data Lake Storage, ADLS): "unlike ADLS, Tectonic hash-partitions each metadata layer rather than using range partitioning. Hash partitioning effectively avoids hotspots in the metadata layer"7 — the same range-vs-hash trade-off Module 3 covers for Bigtable versus Dynamo, applied one layer down, to the metadata service itself rather than to user data.

  • No single machine's memory or CPU bounds the whole filesystem's metadata capacity; each layer scales by adding nodes.
  • Hash partitioning distributes metadata load evenly regardless of key patterns, avoiding the specific hot-region failure mode range partitioning can hit under skewed access.
  • One multitenant fabric replaces many single-tenant clusters, eliminating the stranded-resource and bin-packing problems Section 8.1 describes.
  • Disaggregating metadata into a key-value store plus three logical layers is substantially more moving parts than one master process — more services to operate, monitor, and keep consistent with each other.
  • Hash partitioning gives up the cheap, index-order range scans a range-partitioned or single-master hierarchical namespace supports natively (the same cost Module 3 attributes to Dynamo's hash ring versus Bigtable's range-partitioned tablets).

A flat object namespace beats a hierarchical filesystem for blob workloads

S3's own architecture, implicit in its design and explicit in its 2020 consistency announcement34

S3 does not merely disaggregate metadata the way Tectonic does — it dispenses with a directory hierarchy altogether. There is no per-directory data structure to shard, rebalance, or make consistent; a key is a flat string, and "the number of chunks stored grows linearly with the number of storage nodes"-style scaling (Tectonic's phrasing, but the same idea applies to S3's object index) never has to contend with hierarchy-induced hot paths like a single popular parent directory. The 2020 move to strong read-after-write consistency at "no impact on performance"4 is easier to believe for a flat key space than it would be for a system that also has to keep a directory tree's aggregate listings consistent.

  • No directory-hierarchy hot spots, no recursive-rename problem, no tree-structured lock contention (Section 1.1's namespace locking, GFS-style, simply does not exist as a concern).
  • Scaling is close to embarrassingly parallel across keys, which is exactly why S3 could add strong consistency without giving back availability or latency.
  • No atomic cross-key operations and no atomic rename — an application that needs "move this whole directory" has to simulate it with per-key copies, at the cost of both time and consistency during the simulated move.
  • Workloads that genuinely need file-system semantics (POSIX-like appends, byte-range random writes to a single large file, directory listings sorted by a hierarchy) still have to build or rent that semantics on top of a flat store — which is exactly what Tectonic's Name/File/Block layers exist to provide for Meta's HDFS-descended data-warehouse tenant.
Assessment

None of the three positions is wrong for the workload it was built for; the disagreement resolves on scale and access pattern, not on engineering quality. A single master with a hierarchical namespace (GFS's and HDFS's choice) is the simplest correct answer up to roughly the point where one machine's memory and request rate stop being able to track the metadata working set — Section 2.3's arithmetic puts that ceiling in the petabyte range per gigabyte of master RAM, and it is a real, not theoretical, ceiling: Meta hit it directly with HDFS and needed tens of clusters to route around it7. Past that point, and especially under genuine multitenancy where several large, differently-shaped workloads want to share one storage fabric, disaggregated and hash-partitioned metadata (Tectonic) is not an optimization but a requirement — the alternative is the bin-packing and stranded-resource problems Section 8.1 describes, which do not have a good fix within the single-master model. For workloads that are fundamentally write-once, read-many, addressed by an opaque key rather than needing a file-system's rename/append/byte-range semantics — blob storage being the canonical case — flat object storage wins outright and is not really competing on the same axis at all; it trades away semantics most blob workloads never needed in exchange for a scaling story with no hierarchy-induced hot spots to design around. What would settle the disagreement between positions one and two more precisely is a published account of exactly where an HDFS-style single NameNode starts to degrade under real production load (request rate and working-set size both matter, not just total files) — none of this module's sources give that inflection point in numbers, only Meta's own post-hoc statement that it was hit.

What is changing

Object storage is fragmenting into more storage classes with narrower, more specific latency and locality guarantees rather than staying one undifferentiated tier: AWS's S3 Express One Zone gives up the multi-AZ redundancy Section 7 describes as S3's default in exchange for co-locating storage with compute for "consistent, single-digit millisecond data access," at up to 10× lower latency and 80% lower request cost than S3 Standard3 — a direct trade of Section 3.2's rack/AZ-diversity durability argument for raw speed, made explicit and opt-in rather than baked into one storage tier for everyone. Whether that pattern — many narrow storage classes instead of one general-purpose tier — spreads to file-oriented systems the way it already has to S3 is unresolved in this module's sources.

Disaggregated, independently-scalable metadata (Section 8's Tectonic argument) is displacing the single-master model as the default for any new system built past petabyte scale, not just as a fix Meta needed for its own historical HDFS ceiling; this module's sources do not cover whether a comparable disaggregation has happened inside HDFS itself (HDFS Federation and Router-based federation exist but are outside this source map) or whether the field has converged on hash- over range-partitioned metadata as broadly as Tectonic's paper argues for its own workload. Per-block, per-tenant tunable durability (Reed-Solomon versus straight replication, chosen per workload rather than fixed cluster-wide) is the other piece of Tectonic's design this module expects to see more of elsewhere, precisely because it is the mechanism Module 33's training-checkpoint storage problem needs: a checkpoint's 1MB-to-4GB-per-GPU write burst6 has a very different cost-durability trade-off than a permanently-retained warehouse table, and a storage fabric that can express that difference per object rather than per cluster is the direction all three of Section 9's disagreement positions are converging on, even while disagreeing about how far to push disaggregation to get there.

References

  1. paperGhemawat, S., Gobioff, H., Leung, S-T. The Google File System. SOSP 2003. https://research.google.com/archive/gfs-sosp2003.pdf. Accessed 2026-09-17. Used for: single-master architecture (§2.3-2.4), chunk size rationale (§2.5), in-memory metadata and persistence (§2.6-2.6.3), consistency model and region-state table (§2.7, Table 1), namespace locking (§4.1), replica placement/re-replication/rebalancing (§4.2-4.3), garbage collection (§4.4), leases and record append mechanics (§3.1, §3.3), and the B/T+RL transfer-time arithmetic (§3.2).
  2. officialApache Software Foundation. HDFS Architecture Guide. Apache Hadoop documentation, live. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html. Accessed 2026-09-17. Used for: NameNode/DataNode roles, 128MB default block size, default 3x rack-aware replica placement.
  3. officialAmazon Web Services. What is Amazon S3? Amazon S3 User Guide, live documentation. https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html. Accessed 2026-09-17. Used for: object/bucket/key model, current strong read-after-write consistency model, key-based atomicity and its limits, S3 Express One Zone latency/cost figures.
  4. blogBarr, J. Amazon S3 Update: Strong Read-After-Write Consistency. AWS News Blog, 2020-12-01. https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/. Accessed 2026-09-17. Used for: the exact eventual-to-strong consistency transition, "no impact on performance" claim, motivation tied to Spark/Hadoop analytics workloads.
  5. officialAmazon Web Services. Amazon S3 Storage Classes. Live documentation. https://aws.amazon.com/s3/storage-classes/. Accessed 2026-09-17. Used for: 99.999999999% (11 nines) durability figure and default 3-Availability-Zone redundancy for S3 Standard.
  6. paperLlama Team, Meta AI. The Llama 3 Herd of Models. arXiv:2407.21783, 2024. https://arxiv.org/pdf/2407.21783. Accessed 2026-09-17. Used for: Tectonic's training-cluster scale figures (§3.3.1) — 240PB across 7,500 SSD servers, 2TB/s sustained / 7TB/s peak throughput, per-GPU checkpoint size range.
  7. paperPan, S., Stavrinos, T., Zhang, Y., et al. Facebook's Tectonic Filesystem: Efficiency from Exascale. 19th USENIX Conference on File and Storage Technologies (FAST '21), 2021, pp. 217-231. https://www.usenix.org/system/files/fast21-pan.pdf. Accessed 2026-09-17. Used for: pre-Tectonic Haystack/f4/HDFS motivation (§1-2), disaggregated hash-partitioned metadata vs. ADLS's range partitioning, Chunk Store / Metadata Store architecture (§3.1-3.2), per-block tunable durability (RS encoding vs. replication).
  8. paperChang, F., Dean, J., Ghemawat, S., et al. Bigtable: A Distributed Storage System for Structured Data. OSDI 2006. https://www.usenix.org/legacy/event/osdi06/tech/chang/chang.pdf. Accessed 2026-09-17. Used for: the single sentence that SSTables are stored on GFS (§4), grounding the layering figure only — full Bigtable mechanics are Module 3/4's scope.