System Design · Module 17

WhatsApp, Twitter, and Instagram: Messaging and Feed Architectures at Scale

Three companies, each solving a different scaling bottleneck, each documented in exactly one detailed public account: WhatsApp pushed a single Erlang server to 2 million concurrent TCP connections in 2012, Twitter described its Redis-cached fan-out-on-write timeline in a 2013 talk, and Instagram published its hand-rolled Postgres sharding and 64-bit ID scheme the same year. After this module you can name the specific mechanism each company chose, do the arithmetic behind each one's headline number, and place all three on the same three-way spectrum of "who decides how data is split" that reappears throughout this course.

Course 45 min read 10 sources As of 2026-09-17 Prerequisite: Module 3, Module 8
Three companies, three scaling bottlenecks, one dated public record each WhatsApp bottleneck: connection count 1 Erlang node 2,000,000+ TCP connections1 source: company blog, 2012 Twitter bottleneck: read fanout 1 tweet N follower timelines 300K : 6K read : write QPS23 source: talk + analysis, 2013 Instagram bottleneck: write partitioning logical shards 41 + 13 + 10 bits: time / shard / seq4 source: company blog, 2012 All three: one detailed public account, published 2012-2013, no comparably detailed update since
Figure 1. Three unrelated systems, each solving a different scaling axis — how many live connections one process can hold, how many copies of one write a read-heavy fanout must produce, and how one write stream is split across machines. Each is documented in a single detailed public account from 2012 or 20131234, and none of the three companies has published a comparably detailed architectural update since.

1. Why three companies share one module

Every other case-study module in this course leans on a company that keeps publishing: Google's Spanner and Bigtable papers span 2006-2012 and are still the basis of production systems described in later posts; Netflix's engineering blog produces a steady stream of infrastructure detail; Twitch published a named, current routing architecture in Module 16. WhatsApp, Twitter (as it operated through 2013), and Instagram do not fit that pattern. Each has exactly one moment of genuine public engineering depth, and each moment is over a decade old at this module's as-of date:

  • WhatsApp's engineering team posted "1 million is so 2011" on 2012-01-06, reporting a single production server pushed past 2 million concurrent TCP connections1. No comparably detailed WhatsApp infrastructure post has followed.
  • Twitter's Raffi Krikorian, then Senior Director of Applications Services, gave "Timelines at Scale" at QCon San Francisco; the talk is dated 2012 on the conference program and was recorded and published by InfoQ on 2013-04-032. InfoQ's page carries only an abstract, not a transcript, so the specific numbers in this module are corroborated through a third-party analysis, High Scalability, published 2013-07-08, which attributes every figure it quotes to that same talk3. Treat every Twitter number below as one step removed from the primary talk.
  • Instagram Engineering published "Sharding & IDs at Instagram" in 2012, describing its Postgres-schema sharding scheme and 64-bit ID generator4. The origin domain now blocks automated fetches; the facts below are cross-checked across two independently hosted, character-for-character reproductions of the same post's code and constants, and that caveat is repeated where it matters (§4).

None of the three companies' current (2026) production architecture is documented at this depth anywhere in this course's source map. Twitter has since been acquired and renamed; Instagram is now run inside Meta's infrastructure; WhatsApp is run inside Meta's infrastructure as well. Whatever each actually runs today is almost certainly different in specifics — more machines, different storage engines, different languages in places — but no public account at the level of detail these three posts provide has replaced them. This module treats each source as what it is: an accurate description of a real system at a specific point in time, not a claim about what runs in production now. Instagram is widely discussed online as also having run Cassandra for some data types; no source available to this module documents that at any level of mechanism, so this module says only what Instagram's own 2012 post says: Postgres, sharded by schema (§4.3 returns to this explicitly).

The reason to read all three together despite the thinness is that each demonstrates a different answer to the same underlying question every system in this course eventually faces: where does the deciding logic for "how is load spread across machines" live? WhatsApp's answer is inside a single language runtime's process model. Twitter's answer is a background daemon that pre-computes the answer before anyone asks. Instagram's answer is baked directly into the primary key. Module 3 covered the general replication and partitioning vocabulary (consistent hashing, range vs. hash partitioning, leader-based replication); this module supplies three concrete, load-bearing instances of that vocabulary, then closes by placing Instagram's approach against two systems from elsewhere in this course — Vitess (Module 13) and Spanner (established across Modules 3-5) — that made different choices about the same problem.

2. WhatsApp: one Erlang node, two million sockets

WhatsApp's 2012 post is a follow-up: "a few months ago we published a blog post that talked about our servers doing 1 million tcp connections on a single box," and the new post's point is that the same architecture, on the same class of hardware, now sustains twice that1. The number that matters is not "2 million" as a round headline but the precise reading the team published from the box while it was holding that load, because the precision is what makes the claim checkable and the arithmetic in §2.2 possible.

2.1 The measurement

The server ran FreeBSD 8.2-STABLE (amd64) and Erlang R14B03 (erts-5.8.4), on an Intel Xeon X5675 at 3.07 GHz with 24 CPU cores and 103,062,118,400 bytes of physical memory — 103 GB by a decimal (109) count, 96 GiB by a binary (230) count1. At the moment of measurement, the kernel reported 2,277,845 open sockets1 — the source of the "over 2 million" headline. A `top`-style snapshot from the same post gives the full CPU breakdown: 37.9% user, 0.0% nice, 13.6% system, 6.6% interrupt, 41.9% idle1 — five categories that sum to exactly 100.0%, with 41.9% of the box's CPU time doing nothing at all while holding 2.28 million live connections. WhatsApp's own framing of the result is that they can do this "with plenty of CPU and memory to spare and do it sustainably"1 — the headroom, not just the connection count, is the claim.

The post also lists the FreeBSD kernel-tuning parameters needed to get there: kern.ipc.maxsockets=2400000, kern.maxfiles=3000000, kern.maxfilesperproc=2700000, and net.inet.tcp.tcbhashsize=5242881, alongside TCP host-cache tuning (net.inet.tcp.hostcache.hashsize=4096, net.inet.tcp.hostcache.cachelimit=131072, net.inet.tcp.hostcache.bucketlimit=120)1. None of these are Erlang settings — they are the operating system's own accounting structures for open file descriptors and TCP control blocks, raised well above their stock defaults so the kernel itself does not become the bottleneck before the application does. tcbhashsize in particular sizes the hash table FreeBSD uses to look up a TCP control block by connection tuple; a table sized for a few thousand connections turns every packet's connection lookup into a long chain walk once millions of sockets are open, so this parameter has to move in lockstep with the socket ceiling. The host-cache parameters govern a related structure: FreeBSD's TCP host cache remembers recent round-trip-time and congestion-window estimates per remote peer so a new connection to an already-seen host can start from a better initial estimate than a blind default, and like the TCB hash table, its own hash table and per-bucket limits need raising before the connection count that feeds it grows past what the stock configuration was sized for1.

The post does not state whether the 2.28 million connections were live production chat traffic or an internal load test standing in for it; it presents the kernel and runtime metrics without characterizing the traffic behind them1. Read the number as "this hardware and this kernel configuration can sustain 2.28 million simultaneous open TCP connections with headroom to spare," not as a claim about concurrent active WhatsApp users on one box.

WhatsApp single-server topology: one Erlang/BEAM node terminating 2.28 million TCP connections, with hardware spec and CPU utilization breakdown clients (illustrative sample of 2,277,845 open sockets1) Erlang / BEAM node R14B03, smp:24:24 2,277,845 open TCP sockets1 1 process per socket (§2.2) FreeBSD 8.2-STABLE amd64 hardware1 Xeon X5675 @ 3.07 GHz 24 CPU cores 103 GB RAM (96 GiB) kernel: maxsockets=2.4M1 tcbhashsize=524,2881 maxfiles=3,000,0001 CPU utilization at 2,277,845 connections1 user 37.9% sys 13.6% int 6.6% idle 41.9% nice: 0.0% (omitted, zero width) — five categories sum to 100.0%1
Figure 2. One Erlang node terminates 2,277,845 TCP connections on a single 24-core, 103 GB server, with 41.9% of CPU time still idle — WhatsApp's own point being headroom, not just the raw connection count1.

2.2 Why Erlang's process model makes the arithmetic work

Holding 2.28 million live TCP connections on one machine is a question about where the per-connection state lives, and Erlang's answer is to give every connection its own preemptively scheduled, garbage-collected process, managed entirely by the BEAM virtual machine rather than by the operating system. This is not the same object as an OS thread: Erlang's official documentation states that a newly spawned process starts at 327 words of memory, of which 233 words back the process's own heap and stack5. On a 64-bit build, a word is 8 bytes, so 327 words is 2,616 bytes — under 2.6 KB for everything the runtime needs to track one process's execution state before it does any work5. Erlang's documentation is explicit that this default is deliberately conservative "to support Erlang systems with hundreds of thousands or even millions of processes"5 — the same order of magnitude as WhatsApp's connection count, and not a coincidence: a one-process-per-connection design is only viable at this scale because the per-process floor is measured in kilobytes, not megabytes.

The arithmetic behind WhatsApp's headroom follows directly from that floor. At 2,277,845 connections and 2,616 bytes of minimum process memory each:

$$ 2{,}277{,}845 \times 2{,}616\ \text{bytes} \approx 5.96\ \text{GB} $$

against 103 GB of physical memory1 — under 6% of the box's RAM spent on the theoretical minimum footprint of every connection's process state, before counting the message queue, socket buffers, and any per-user session data each process actually holds. That gap is consistent with, though not identical to, the 41.9% idle CPU and the "plenty of... memory to spare" the post reports1: a process model whose baseline unit costs kilobytes leaves most of a 103 GB machine for actual work rather than for the bookkeeping of staying connected. An OS-thread-per-connection design on the same hardware would not have this headroom. Linux's own thread-creation documentation states that a new thread's default stack size is set by the process's RLIMIT_STACK soft limit at the time the program started, falling back to a fixed 2 MB per-architecture default on most architectures if that limit is "unlimited"6 — and a typical Linux distribution's default soft RLIMIT_STACK is itself commonly set to 8 MB (8,192 KB)6. Either figure, 2 MB or 8 MB per thread, is three to four orders of magnitude above Erlang's roughly 2.6 KB process floor computed above, which is the concrete reason thread-per-connection servers hit practical ceilings in the tens of thousands of connections while a one-process-per-connection Erlang node reaches millions on the same class of hardware: 2.28 million threads at even the smaller 2 MB default would need 4.56 TB of stack space alone, dwarfing the box's entire 103 GB of RAM16, while 2.28 million Erlang processes need under 6 GB, as computed above.

The other half of the design is preemption: the BEAM scheduler is what lets 24 physical cores serve 2.28 million logically concurrent processes at all, by time-slicing each core's attention across many more runnable processes than there are cores to run them on, so that one connection's handler blocking or looping cannot starve the other 2.28 million indefinitely. Module 6 covers the general vocabulary of preemptive scheduling and fairness this section leans on informally; the point specific to this module is only that the unit being scheduled (a process) and the unit whose memory floor was just computed (also a process) are the same object — Erlang's concurrency primitive is simultaneously the scheduling unit and the connection-state container, which is what lets one design decision (make processes cheap) pay off on both axes (memory and fair scheduling) at once.

2.3 Contrast: state in the process vs. state at the edge

WhatsApp's design keeps each connection's session state inside a live, addressable Erlang process for as long as the socket is open: the process is the connection, and the server holds 2.28 million of them simultaneously in memory, one per open socket1. Module 8 covers Kafka's architecture in full; the relevant contrast here is only the placement of state. Kafka's broker is deliberately stateless with respect to any individual reader: "the information about how much each consumer has consumed is not maintained by the broker, but by the consumer itself"7, and a consumer can disconnect, restart, and resume days later by presenting its own last-read offset — the broker does not need to have kept anything about that consumer while it was away. Kafka's brokers scale the number of independent readers without a floor proportional to connection count, because the state a WhatsApp-style connection process would hold (where is this reader up to?) is pushed out to the client instead of held in server memory for the connection's lifetime.

These are not competing solutions to the same problem — WhatsApp is terminating live, bidirectional, low-latency chat sessions that must be addressable at any instant, and a Kafka-style client-tracked-offset design would not deliver a chat message to a currently open socket any faster than the socket's own process already does. The general point worth keeping past this module is that "how many connections can one machine hold" and "how many independent readers can one machine serve" are different questions with different answers depending entirely on where the per-connection or per-reader bookkeeping lives: inside a lightweight in-process actor (WhatsApp), or pushed to the far side of the wire and re-presented on every request (Kafka).

The same state-placement choice shows up again in what each design keeps warm across a restart. A WhatsApp-style Erlang process's state lives inside that process's own heap; kill and restart the node, and every one of its 2.28 million processes — and the session state each was holding — is gone, which is why systems built this way lean on the client reconnecting and re-authenticating rather than on the server preserving anything across the crash. Kafka inverts this by design: rather than build an application-level cache for its log segments, "we rely on the underlying file system page cache," specifically so that "a restarted broker keeps a warm cache" rather than starting cold7. Because the data a Kafka broker serves lives in ordinary files rather than in any single process's private heap, the operating system keeps recently touched pages resident independently of whether the broker process itself is still running — a second, related consequence of keeping the broker stateless with respect to any individual connection: the broker's own restart does not have to mean discarding what was cached.

3. Twitter: fan-out-on-write and the 50-to-1 read/write economy

At the time of Krikorian's talk, Twitter served roughly 150 million worldwide active users, took in tweets at a daily average of about 5,000/sec (peaking past 12,000/sec during major live events), and answered 300,000 queries per second for home timelines3. The core design decision the talk describes is when to pay the cost of computing "what does this specific user's home timeline look like right now" — at write time, once, when a followed account tweets, or at read time, every time a follower opens the app.

3.1 The write path: a daemon that pre-computes every follower's timeline

Twitter's answer is fan-out-on-write. A tweet's arrival triggers a fanout daemon that looks up the tweet's author's followers and writes the tweet into each of those followers' cached timelines — before any follower has asked for anything3. Two named systems sit underneath this: Flock maintains the follower and following lists (the social graph the daemon queries), and Gizzard sits in front of the disk-backed MySQL storage behind Flock, abstracting away SQL transactions and providing global replication across it3. The daemon does not write one follower at a time — it pipelines the writes in batches of 4,000 destinations at once3, and what it writes into each follower's timeline is small and fixed-width: the tweet's own ID, the ID of the tweet's author, and 4 bytes of flag bits marking whether the tweet is a retweet, a reply, or another special case3 — not the tweet's text, which is fetched separately (by Tweetypie, the tweet-object service3) only when a client actually renders the timeline.

Where the daemon writes to is a Redis cluster: each user's home timeline is a Redis list capped at a maximum of 800 entries, replicated three times across three different machines, including across datacenters3. Only users active within the preceding 30 days are kept resident in this way — "every active user is stored in RAM to keep latencies down"3 — and the cluster holding all of them runs to a couple of terabytes of RAM in aggregate3. A home-timeline read, in the common case, is then a single fetch from an in-memory list rather than a live join across the follower's entire following set: the read path is deliberately made as cheap as possible by having already done the expensive part (finding every follower) at write time.

Author tweets Fanout daemon wakes on write Flock follower list lookup Pipelined writes, 4,000 destinations/batch3 payload: tweet ID + author ID + 4-byte flags3 Follower A timeline Follower B timeline ... N followers
  1. Tweet arrives. The write triggers the fanout daemon3.
  2. Daemon queries Flock for the author's current follower list3.
  3. Daemon pipelines writes to Redis in batches of 4,000 destinations, each write carrying only the tweet ID, author ID, and 4 bytes of flags3.
  4. Every follower's 800-entry, 3x-replicated Redis timeline now contains the new tweet ID, targeted to complete in under 5 seconds3.
Figure 3. The write path pre-computes every follower's timeline update before any follower asks for anything; Twitter's stated target for the whole sequence is under 5 seconds, against a naive worst case of up to 5 minutes for an account with 31 million followers3.

3.2 Why the asymmetry makes fan-out-on-write pay off

Krikorian's talk gives the specific traffic split that justifies paying the fanout cost at write time: 300,000 requests per second are spent reading timelines, and only 6,000 requests per second are spent on writes3. That is a ratio of exactly:

$$ \frac{300{,}000}{6{,}000} = 50 $$

fifty read requests for every one write request. This ratio is the entire argument for fan-out-on-write: any cost paid once per write is amortized across fifty times as many reads as there are writes to pay for it. A design that instead computed each timeline at read time — fetching the follower's full following list and merging the N most recent tweets from each on every request — would put the expensive part of the work (finding and merging every source) on the side of the ratio that receives fifty times the traffic. Fan-out-on-write does the reverse: it puts the one genuinely expensive operation (finding every follower, which can number in the millions for a large account) on the side of the ratio that only has to happen 6,000 times a second, and turns every one of the 300,000-per-second reads into the cheapest possible operation, a fetch of an already-assembled, bounded-size (800-entry) list from RAM3.

The talk's own numbers leave one thing unreconciled that neither source available here resolves: tweet-ingest rate is reported separately as roughly 5,000/sec average, 7,000/sec peak, and over 12,000/sec during large events3, which does not equal the 6,000/sec "writes" figure the read/write ratio uses. The two are plausibly measuring different points in the same pipeline — raw tweets arriving versus write-side requests against the timeline-serving tier, which is not necessarily one write per tweet if, for instance, retweets or other write-triggering events are counted separately from original tweets — but the talk as transcribed does not say so explicitly, and no primary transcript is available to check23. Treat "6,000/sec writes" as the figure Twitter's own talk paired directly with "300,000/sec reads" for the 50:1 ratio, and the 5,000-12,000/sec tweet-ingest figures as a related but distinct measurement of a different point in the system.

Twitter read versus write QPS: 300,000 timeline reads per second against 6,000 writes per second QPS 0 60K 120K 180K 240K 300K 300,000 timeline reads/sec3 6,000 writes/sec3 ratio: 50 reads for every 1 write
Figure 4. Twitter's home-timeline read traffic outweighs write traffic by 50:1 (300,000 vs. 6,000 requests per second)3 — the reason it is worth paying the fanout cost once, at write time, rather than on every read.

3.3 The exception: merging at read time for the highest-follower accounts

Fan-out-on-write has a worst case that scales with follower count, not with traffic: Krikorian's talk gives the concrete example that it can take up to 5 minutes for a tweet to flow from a 31-million-follower account to all of its followers under the naive version of the fanout, against a target of under 5 seconds3. Writing one tweet ID into 31 million Redis lists is a real, non-trivial amount of work no matter how well pipelined, and unlike the read path, that cost cannot be amortized across other requests — it has to complete once per tweet, in full, regardless of how many followers there are.

Twitter's documented response is a hybrid: for its highest-follower accounts, the system stops fanning out at write time altogether and instead merges that account's tweets into a follower's timeline at read time, alongside the pre-computed fanned-out portion of the timeline3. This is fan-out-on-read applied narrowly, only to the small set of accounts whose follower count makes the write-time cost disproportionate, while every other account still uses fan-out-on-write. The stated effect is to balance the read and write paths and save "10s of percents of computational resources"3 — a materially cheaper system than either pure strategy applied uniformly. The general lesson is that fan-out-on-write and fan-out-on-read are not mutually exclusive architectural commitments; the choice can be, and in Twitter's own account was, made per account based on which side of the write-cost-vs-read-cost trade a given account's follower count puts it on.

3.4 Gizzard, Flock, and an eventual-consistency resemblance to Dynamo

Gizzard's stated job is to abstract away SQL transactions and provide global replication over the disk-backed MySQL shards underneath Flock3. No source in this course's map documents Gizzard's internal replication protocol at the level of detail Dynamo's own paper documents its algorithms (Module 3, §3.2-3.3, covers Dynamo's consistent hashing and vector clocks in full), so this section states only what is actually known and where the resemblance is real versus asserted. What is known: Gizzard replicates a follower/following graph globally, across datacenters, on top of a storage engine (MySQL) that does not itself support that kind of geographic replication3 — the same problem Dynamo's paper frames as its starting point, an "always writeable" store that replicates across failure domains and reconciles afterward rather than blocking a write on a synchronous cross-region round trip8. Both systems are solving "keep writing during a replica or region failure, and let replicas catch up asynchronously" rather than "block until every replica agrees," which is the shared design lineage the resemblance is actually about.

What is not known from any source available here: whether Gizzard uses vector clocks, read-repair, Merkle-tree anti-entropy, or any of the other specific mechanisms Dynamo's paper documents8 for reconciling concurrent writes to the same record. Treat "Gizzard resembles Dynamo" as a statement about shared design philosophy — asynchronous, multi-region replication over an always-writeable store — and not as a claim that Gizzard implements Dynamo's specific algorithm.

The fanout daemon, Flock, and Gizzard sit inside a larger named constellation of services the talk describes without going into each one's internals: Timeline, Tweet, User, and Social-Graph services divide the object model along those lines; Gizmoduck serves user objects and Tweetypie serves tweet objects, which is where the actual tweet text a rendered timeline needs is fetched from once the Redis-cached list of IDs (§3.1) has been read; Early Bird is a modified version of the Lucene search library used for Twitter's own search; and Zipkin, explicitly modeled on Google's Dapper paper, provides distributed tracing across this whole service graph3. None of these are documented at the mechanism level this module has given the fanout daemon and the Redis timeline cache — they are named, not explained — but the names are worth having, because "Tweetypie" and "Gizmoduck" recur in later, better-documented accounts of Twitter/X's architecture that this module's source map does not include.

4. Instagram: several thousand logical shards over a Postgres schema

Instagram's problem is different in kind from WhatsApp's or Twitter's: not how many live connections one process can hold, and not when to pay a fanout cost, but how to keep inserting rows once no single database server can hold the whole table, while still being able to generate a globally unique, roughly time-sortable primary key for every new row without any server having to ask another server "what's the next ID." Instagram's 2012 post covers both halves of that problem together, because they are the same problem: partitioning the data and generating IDs that already encode which partition a row lives in4.

4.1 Logical shards, mapped onto far fewer physical servers

Instagram's account states that the team considered NoSQL alternatives but chose to keep PostgreSQL and shard the data themselves, splitting it across "several thousand" logical shards that are mapped, in code, onto a much smaller number of physical database servers4. The mechanism for this mapping is Postgres's own schema feature: each logical shard is implemented as one Postgres schema, and every sharded table — the post's own example is "likes on our photos" — exists once inside each schema4. A single physical Postgres server hosts many schemas (many logical shards) at once; growing the fleet means moving whole schemas onto new physical servers, not re-partitioning any individual table's rows.

The reason this beats a naive auto-incrementing primary key is direct: a single `SERIAL` column only produces a globally unique, strictly increasing sequence when there is exactly one database inserting into that table. The moment rows for the same logical table are being inserted into several thousand independent schemas at once, each schema's own sequence starts again from wherever it left off, and two different shards will mint the same integer for two different rows. Decoupling "which shard is this row in" from "which physical server currently hosts that shard" is what lets Instagram move a logical shard between physical machines as load grows, without having to renumber or re-key any row that shard already contains — the exact separation-of-concerns Module 3 introduced for GFS's chunks and Bigtable's tablets, applied here to a stock relational engine rather than a purpose-built distributed store.

Which logical shard a given row lands in is decided by a plain modulo over the user ID: the post's own worked example takes user ID 31,341 and computes shard ID as \(31{,}341 \bmod 2{,}000 = 1{,}341\)4. This is hash partitioning in Module 3's vocabulary, with the hash function reduced to the simplest one that works — the identity function, then modulo — rather than a cryptographic or otherwise well-mixed hash; it inherits the same property Module 3's consistent-hashing treatment cares about, that partitioning is a deterministic function of a key rather than a lookup, at the cost of the same property consistent hashing exists to fix: changing the shard count (2,000 in the example) changes almost every user's assigned shard, rather than reassigning only the minimal set consistent hashing's virtual nodes achieve. Sharding by user ID, specifically, also has a consequence that follows directly from the scheme even though the post does not spell it out: any table whose rows are assigned to a shard using that same user's ID keeps all of that user's rows in one logical shard together, so a query scoped to one user's own data never has to fan out across shards to answer it. The post does not state whether every sharded table (its own example is photo likes) uses the photo owner's user ID, the acting user's ID, or some other key for this same modulo — only that the mechanism for computing a shard ID, once a sharding key is chosen, is the one shown above.

4.2 The 64-bit ID: anatomy and a worked example

Instagram generates every row's primary key with a PL/pgSQL function running inside the owning shard, so no cross-shard coordination is needed to mint an ID4. The resulting 64-bit integer packs three fields, most-significant first:

  • 41 bits — milliseconds elapsed since a custom epoch, not the Unix epoch4.
  • 13 bits — the logical shard ID that generated this row4.
  • 10 bits — a per-shard sequence number, taken modulo 1,024 from a Postgres sequence local to that shard4.

The generating function shifts and OR's these three fields together: the millisecond delta is shifted left by 23 bits (making room for the 13+10 bits below it), then OR'd with the shard ID shifted left by 10 bits, then OR'd with the raw sequence value4. Instagram's own origin domain now blocks automated fetches, so the exact function body below is cross-checked against two independently hosted, character-for-character-identical reproductions of the same post, both showing the same custom epoch constant and the same bit-shift structure4:

CREATE OR REPLACE FUNCTION insta5.next_id(OUT result bigint) AS $$
DECLARE
    our_epoch bigint := 1314220021721;  -- custom epoch, ms since Unix epoch
    seq_id bigint;
    now_millis bigint;
    shard_id int := 5;                  -- this shard's 13-bit ID
BEGIN
    SELECT nextval('insta5.table_id_seq') % 1024 INTO seq_id;
    SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
    result := (now_millis - our_epoch) << 23;
    result := result | (shard_id << 10);
    result := result | (seq_id);
END;
$$ LANGUAGE PLPGSQL;
Instagram's 64-bit ID: 41 bits of time, 13 bits of shard ID, 10 bits of sequence, with a worked example 64-bit ID layout, most-significant bit at left4 41 bits ms since custom epoch 13 bits shard ID 10 bits sequence bit 63 ... bit 23 bit 22 ... bit 10 bit 9 ... bit 0 Worked example epoch = 1,314,220,021,721 ms → 2011-08-24T21:07:01.721Z4 now − epoch = 5,000 ms shard_id = 5 seq_id = 12 result = (5000 << 23) | (5 << 10) | 12 = 41,943,040,000 | 5,120 | 12 result = 41,943,045,132
Figure 5. Instagram's 64-bit ID reserves 41 bits for a millisecond timestamp against a custom epoch, 13 bits for the generating shard's ID, and 10 bits for a per-shard sequence4. The worked example (delta = 5,000 ms, shard 5, sequence 12) applies the documented shift-and-OR formula by hand; the epoch constant and formula structure are Instagram's own, the specific delta/shard/sequence values here are illustrative.

The bit widths are not arbitrary, and they cross-check against the "several thousand" figure the post states in prose. Forty-one bits of milliseconds gives:

$$ 2^{41}\ \text{ms} = 2{,}199{,}023{,}255{,}552\ \text{ms} \approx 69.7\ \text{years} $$

of headroom from the custom epoch before the time field overflows — comfortably past any realistic system lifetime, which is presumably why the post chose a custom 2011 epoch rather than the 1970 Unix epoch: every bit of the 41 spent on years before 2011 is a bit not available for years after it. Thirteen bits of shard ID gives a ceiling of \(2^{13} = 8{,}192\) distinct shard IDs — comfortably above the "several thousand" logical shards the post describes using4, which is the point of choosing 13 rather than, say, 10 or 16: enough headroom to grow the logical shard count without a second migration, without wasting bits that could extend the time field's or the sequence field's range instead. Ten bits of sequence gives \(2^{10} = 1{,}024\) IDs a single shard can mint within the same millisecond before it has to wait for the clock to advance — the ceiling on how many rows one shard's `next_id()` function can generate per millisecond, independent of every other shard, since each shard runs its own sequence and its own copy of the function.

That per-shard ceiling is also a per-shard risk: the sequence field is generated by nextval(...) % 1024, a plain modulo with no check against the millisecond boundary4, so a shard that is asked to insert more than 1,024 rows inside the same millisecond wraps the sequence back to 0 and mints a colliding ID rather than raising an error or blocking. The scheme's actual safety margin is therefore not "1,024 IDs per shard per millisecond" as an abstract ceiling but "however far below 1,024 a single shard's real write rate stays," which is a property of the sharding scheme's fan-out (how evenly several thousand logical shards divide the total write load) rather than of the ID format itself.

Multiplying the per-shard ceiling by shard count gives a sense of how much headroom the design carries in aggregate, in the same spirit as WhatsApp's idle-CPU headroom (§2.1): at the "2,000 logical shards" scale the post's own worked example uses, 1,024 IDs/ms/shard across 2,000 shards is a theoretical aggregate ceiling of

$$ 2{,}000 \times 1{,}024 \times 1{,}000\ \text{ms/s} = 2{,}048{,}000{,}000\ \text{IDs/sec} $$

— on the order of two billion inserts per second if every shard were simultaneously saturated, a number with no realistic relationship to Instagram's actual 2012 write volume (which this module's sources do not state) but a useful sense check: the bit widths were chosen with room to spare, the same posture every headline number in this module turns out to have once its arithmetic is done by hand.

5. Three philosophies for who owns the shard map

Instagram's scheme, Vitess (Module 13's subject), and Spanner (established across Modules 3-5) all answer the same question — once a dataset outgrows one machine, what decides which machine owns which row, and what happens when that decision has to change — with three structurally different answers about where that decision lives.

Instagram's answer lives in application code and the primary key itself: a client that already has a row's ID can extract the 13-bit shard field directly (§4.2) and go straight to the owning schema, with no separate lookup service consulted on the read path at all4. Vitess's answer is a proxy tier that stands between the application and an unmodified MySQL fleet: VTGate accepts ordinary MySQL-protocol connections and itself resolves which shard a query belongs to, consulting a topology service (etcd, ZooKeeper, or Consul) that tracks the current shard map, while VTTablet runs alongside each MySQL instance managing that instance's own queries, connections, and replication9 — Module 13 covers VTGate's Vindex-based routing and the topology service's replication in full; the point here is only that the application never sees a shard ID, because VTGate resolves it on every query. Spanner's answer removes the decision from both the application and any middleware operator: data is organized into directories, contiguous key ranges that are the unit of geographic placement, and the system itself can move a typical 50MB directory between Paxos groups in a few seconds10 as load or a placement policy requires, with each of a zone's hundred-to-several-thousand spanservers hosting 100 to 1,000 tablets, each tablet backed by its own Paxos state machine with long-lived, time-based leader leases10.

Three sharding philosophies: Instagram's ID-encoded routing, Vitess's VTGate proxy, and Spanner's automatic directory movement Instagram: routing in the key App id >> 10 shard # schema per logical shard Postgres server(s) no proxy, no lookup service4 Vitess: routing in a proxy App MySQL proto VTGate topology service (etcd/ZK/Consul)9 VTTablet+MySQL VTTablet+MySQL app never sees a shard ID9 Spanner: routing moves itself Client lib spanserver directory A directory B moves in seconds10 Paxos group per tablet10
Figure 6. Three places to put the sharding decision: baked into the primary key and read directly by application code (Instagram4), resolved on every query by a proxy consulting a topology service (Vitess9, detailed in Module 13), or moved automatically by the storage engine itself between Paxos-replicated groups (Spanner10, established in Modules 3-5).
Table 1. Three sharding philosophies compared on who decides, how resharding happens, and what it costs4910.
PropertyInstagram (Postgres schemas)Vitess (VTGate/VTTablet)Spanner (Paxos directories)
Who resolves the shardApplication code, from bits in the IDVTGate proxy, per queryThe storage engine itself
Storage engineUnmodified PostgreSQLUnmodified MySQLPurpose-built (Spanner)
Unit of reshardingA whole logical shard (schema)A whole keyspace shardA ~50MB directory10
Resharding triggerManual operator migrationOperator-initiated, app-transparentAutomatic, load- or policy-driven
Extra runtime componentNoneVTGate fleet + topology service9None (built into spanserver)
Cross-shard transactionsNot supported by this schemeSupported, at a stated cost (Module 13)Externally consistent, via 2PC over Paxos10
Documented production runInstagram, 2012 account4100% of YouTube's DB traffic, 5+ years9Google-internal + Cloud Spanner10

A related but separate question — not who decides how data is split, but who is trusted to mint a durable identifier and how a system stays correct when that identifier's minting is retried — recurs in Module 18's treatment of Stripe's idempotency keys. Instagram's ID scheme (§4.2) solves "generate a unique key with no cross-shard coordination"; Stripe's idempotency key solves a different problem with a similarly load-bearing identifier: making a client-generated key the basis for detecting and safely no-op'ing a retried request. The mechanisms differ, but both are answers to the same underlying question this course keeps returning to — what has to be true about an identifier for a distributed system to trust it.

Where experts disagree

Table 1 laid out three positions side by side; the actual disagreement is about which one is the right default once a relational workload has outgrown a single machine, and the three systems in this module make three different bets. None of the sources below frame the comparison as a live debate between named people — each is one company or project explaining the choice it made — but the underlying trade-off is the same one Module 3's Dynamo-vs-Bigtable material and Module 13's Vitess-vs-Spanner disagreement both already surface: is the sharding decision cheapest to make in application code, in a middleware layer, or inside the storage engine itself?

Hand-roll it: encode the shard in the key, keep the database engine untouched

Instagram's own account of its 2012 architecture4

Instagram considered NoSQL and rejected it, choosing instead to shard an unmodified PostgreSQL fleet using the database's own schema feature and an ID scheme that encodes the shard directly in every primary key4. No proxy, no topology service, and no new runtime component sits between the application and the database; a client with a row's ID already knows which shard to query.

  • Zero new infrastructure: no proxy fleet, no topology service to keep consistent, nothing new that can fail.
  • Read latency for by-ID lookups is minimal — the shard is a bit-shift away, not a network round trip to a router.
  • A small team can implement and fully understand the entire mechanism (§4.1-4.2 is close to the whole of it).
  • Every piece of application code that queries by anything other than ID has to be sharding-aware from day one.
  • Moving a logical shard to new hardware, or changing the bit widths once chosen, is a manual, whole-schema migration with no documented automatic mechanism4.
  • No cross-shard transactions: the scheme says nothing about atomically updating rows that live in two different logical shards.

Put a proxy in front: keep the engine, add a routing tier the app never sees

Vitess's documented architecture and its five-plus years as 100% of YouTube's database traffic9 (mechanism detail in Module 13)

Keep MySQL exactly as it is and add VTGate, a proxy that speaks the unmodified MySQL wire protocol and resolves shard routing on every query by consulting a topology service, while VTTablet manages each shard's own connections and replication9. The sharding decision moves out of application code and into infrastructure that many applications can share.

  • Application code is sharding-agnostic; the same query pattern that worked on one shard keeps working after a resharding operation.
  • One deployment of the pattern serves an entire organization's many services, not just one team's schema.
  • Documented to run transparently under YouTube's full database traffic for over five years9, and adopted since by other operators (Slack, Square, JD.com, per Module 13's sourcing).
  • A whole new tier (VTGate fleet plus a topology service) is now a load-bearing part of every query's path and its own operational surface.
  • Every MySQL connection VTTablet manages still carries MySQL's own 256KB-to-almost-3MB per-connection memory overhead9 — Vitess controls this cost, it does not eliminate the underlying constraint.
  • Cross-shard transactions are supported but not free (Module 13 gives the specific cost).

Replace the engine: let the storage layer reshard itself automatically

Spanner's own design, established in Modules 3-510

Spanner shards data across many Paxos state machines and moves the unit of placement — a directory, typically around 50MB — between Paxos groups automatically, in a few seconds, as load or an explicit placement policy dictates10. Neither the application nor an operator decides when a directory moves; the system does.

  • Resharding is not a scheduled migration or an operator-run script — it is a normal, automatic, seconds-scale operation10.
  • Cross-directory transactions get the same external-consistency guarantee as single-directory ones, via 2PC over Paxos and TrueTime (Module 5), with no separate proxy tier to reason about.
  • No MySQL/Postgres-style per-connection memory tax, because the engine was built for this from the start rather than adapted to it.
  • Requires abandoning whatever relational engine (and its tooling, staff expertise, and existing schema) an application already runs — a much larger commitment than adding a proxy in front of the existing one.
  • Externally consistent commits pay TrueTime's clock-uncertainty cost on every cross-group transaction, a latency tax neither of the other two designs incurs in the same form.
  • Only available as Google's own operated system or as the managed Cloud Spanner product — not something most teams stand up themselves the way they can install Vitess or write Instagram's PL/pgSQL function.
Assessment

Which of the three wins depends almost entirely on what already exists and how often the application needs a transaction that spans more than one shard. Instagram's approach is close to strictly right for a small team, a schema that is already ID-centric, and a workload where cross-shard transactions are rare or unnecessary — it is the cheapest possible answer precisely because it adds nothing, and Instagram's own 2012 account describes a team that had already rejected a heavier NoSQL migration for this reason4. It stops being the right answer the moment enough different services need to query the sharded data by something other than ID, because every one of those call sites now has to carry sharding logic by hand. Vitess's proxy tier is the right answer once that condition is met but the organization is still unwilling or unable to migrate off its existing relational engine — YouTube's actual 2010 position, and the reason Vitess exists at all: it buys transparency for the application at the cost of running an extra, shared piece of infrastructure, which is a good trade exactly when many teams would otherwise duplicate Instagram-style sharding logic independently. Spanner's automatic directories win only when the application's correctness requirements genuinely need cross-shard transactions with external consistency and the organization can accept both a storage-engine migration and TrueTime's per-transaction latency cost — a bet that pays off at Google's scale and for Cloud Spanner customers who need exactly that guarantee, and does not pay off for a workload, like Instagram's photo-and-like counters, that never needed cross-shard atomicity in the first place. No source in this module's map publishes a head-to-head benchmark of the three approaches on identical workloads; that comparison — same schema, same query mix, same failure injection, run against a hand-rolled scheme, a Vitess deployment, and a Spanner-class system — is the evidence that would actually settle which one is fastest rather than merely which one is architecturally cleanest for a given starting position.

References

  1. blogWhatsApp (Rick Reed). 1 million is so 2011. WhatsApp Engineering blog, 2012-01-06. https://blog.whatsapp.com/1-million-is-so-2011. Accessed 2026-09-17. Used for: the 2,277,845-open-socket figure, the full CPU utilization breakdown (37.9% user / 0.0% nice / 13.6% system / 6.6% interrupt / 41.9% idle), hardware spec (Xeon X5675, 24 cores, 103,062,118,400 bytes RAM), FreeBSD/Erlang version, and the kernel-tuning sysctl values (maxsockets, maxfiles, maxfilesperproc, tcbhashsize, hostcache parameters).
  2. talkKrikorian, Raffi. Timelines at Scale. QCon San Francisco 2012, recorded/published by InfoQ 2013-04-03. https://www.infoq.com/presentations/Twitter-Timeline-Scalability. Accessed 2026-09-17. Used for: the talk's existence, subject, speaker, and date, as the primary source that ref-3 (High Scalability) transcribes and attributes numbers to.
  3. analysisHoff, Todd. The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds. High Scalability, 2013-07-08. https://highscalability.com/the-architecture-twitter-uses-to-deal-with-150m-active-users/. Accessed 2026-09-17. Used for: 150M active users, 300K QPS reads vs. 6,000/sec writes, tweet-ingest rates, the 800-entry/3x-replicated Redis timeline, the fanout daemon's 4,000-destination pipelined batches and per-entry byte layout, the 5-minute-worst-case/5-second-target figures, the celebrity read-time-merge hybrid and its "10s of percents" savings claim, and the Gizzard/Flock/Tweetypie system names — all attributed by the article to Krikorian's talk (ref-2).
  4. blogInstagram Engineering. Sharding & IDs at Instagram. instagram-engineering.com, 2012. https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c. Accessed 2026-09-17. Used for: several-thousand-logical-shards-over-fewer-physical-servers claim, the Postgres-schema-per-shard mechanism, the 41/13/10-bit ID layout, the custom epoch constant (1,314,220,021,721 ms) and the PL/pgSQL next_id() function's shift-and-OR structure. The origin domain blocks automated fetches; content is cross-checked across two independently hosted, character-for-character reproductions of the same post.
  5. officialEricsson AB / Erlang OTP team. Processes. Erlang System Documentation, erlang.org. https://www.erlang.org/doc/system/eff_guide_processes.html. Accessed 2026-09-17. Used for: the 327-word (2,616-byte on 64-bit) newly spawned process memory floor, the 233-word default heap/stack allocation, and the stated design intent to support hundreds of thousands to millions of processes.
  6. officialKerrisk, Michael. pthread_create(3) — Linux manual page. man7.org, Linux man-pages project. https://man7.org/linux/man-pages/man3/pthread_create.3.html. Accessed 2026-09-17. Used for: the RLIMIT_STACK-governed default thread stack size (2 MB per-architecture default when RLIMIT_STACK is "unlimited"; commonly 8 MB / 8,192 KB under a typical Linux distribution's soft ulimit), contrasted with Erlang's per-process memory floor (§2.2).
  7. paperKreps, Jay, Narkhede, Neha, Rao, Jun. Kafka: a Distributed Messaging System for Log Processing. NetDB '11 workshop, 2011. https://notes.stephenholiday.com/Kafka.pdf. Accessed 2026-09-17. Used for: the stateless-broker/consumer-tracked-offset design and the page-cache/warm-restart property (§2.3), contrasted with WhatsApp's stateful per-connection process model; full Kafka mechanics are Module 8's subject, not re-derived here.
  8. paperDeCandia, Giuseppe, Hastorun, Deniz, Jampani, Madan, et al. Dynamo: Amazon's Highly Available Key-value Store. SOSP '07, 2007. https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf. Accessed 2026-09-17. Used for: the "always writeable," asynchronously-reconciled design philosophy cited as the shared lineage with Gizzard's global-replication role (§3.4); full Dynamo mechanics (consistent hashing, vector clocks) are Module 3's subject.
  9. officialVitess contributors. What is Vitess? Vitess (CNCF graduated project) documentation, 2026. https://vitess.io/docs/overview/whatisvitess/. Accessed 2026-09-17. Used for: VTGate/VTTablet roles, the topology-service description, YouTube's 5+-year, 100%-of-traffic production run, and the 256KB-to-almost-3MB per-MySQL-connection memory overhead figure, in Table 1 and §5's comparison; full VTGate/Vindex routing mechanics are Module 13's subject.
  10. paperCorbett, James C., Dean, Jeffrey, Epstein, Michael, et al. Spanner: Google's Globally-Distributed Database. OSDI '12, 2012. https://research.google.com/archive/spanner-osdi2012.pdf. Accessed 2026-09-17. Used for: sharding data across Paxos state machines (§1), the 100-to-several-thousand-spanservers-per-zone and 100-to-1000-tablets-per-spanserver figures, 10-second default Paxos leader leases, and the 50MB-directory-moves-in-a-few-seconds figure (§2.1-2.2), used here only for the §5/disagree comparison; full TrueTime and transaction mechanics are established across Modules 3-5.