System Design · Module 8

Messaging and Streams: Logs, Queues, Exactly-Once, and Idempotency

Kafka's designers rejected the smart-broker, per-consumer-queue model that every messaging system before it used, and replaced it with a dumb, append-only, offset-addressed log that the OS page cache does most of the work of serving. This module derives why that trade paid off, what "exactly once" actually means once you read past the marketing sentence, and the idempotency-key state machine that gives you exactly-once effects on top of a network that only promises at-least-once delivery — the vocabulary Module 18 and Module 23 assume rather than re-derive.

Course 35 min read 10 sources As of 2026-09-17 Prerequisite: Module 6
One partition, one log, three independent readers at three different offsets A partition is a log, not a mailbox Producer append partition 0 — one append-only log 0 4 6 9 17 writable tail group A @ offset 4 group B @ offset 6 group C @ offset 9 Each consumer group owns its own offset. The broker tracks none of them.
Figure 1. Kafka does not push messages to subscribers and does not delete a message once it is read. A partition is one ordered, append-only log; every consumer group independently remembers how far into that log it has read, and the broker's only job is to keep appending and keep serving reads at whatever offset is asked for12.

1. The log as the interface

Every messaging system before Kafka — JMS brokers, IBM MQ, early AMQP implementations — modeled a queue the way a real-world mailbox works: the broker holds each message until a consumer has confirmed it processed it, then throws the message away. That model requires the broker to track, for every message, whether it has been delivered, and increasingly, whether it has been acknowledged, so it can safely delete it and never redeliver a message that already succeeded2. Kafka's designers, building the system at LinkedIn to move activity-stream and operational-metrics data at web scale, decided that bookkeeping was the wrong thing for a broker to be good at. Their design note is explicit that Kafka is "more akin to a database log than a traditional messaging system"2 — a partition is not a mailbox that empties as it is read, it is an immutable, ordered append log that consumers scan.

This is a different storage abstraction from Module 7's chunked, byte-addressed files: a GFS or HDFS file is a sequence of fixed-size chunks a client can read or overwrite at an arbitrary byte offset, replicated for durability, with a single master tracking chunk placement. A Kafka partition is a sequence of immutable, monotonically-growing segment files that are only ever appended to, never overwritten, and whose only unit of navigation is the logical offset of a record, not a byte range. Kafka happens to store those segments as ordinary files on a local disk (or, since Kafka 3.6, optionally on a remote object store — see "What is changing" below) rather than on a GFS-style chunked filesystem, but the abstraction a producer or consumer programs against is the log, not the file3.

Three design choices fall out of treating the broker as a dumb log rather than a smart queue, and the rest of this module is organized around them:

  • The broker does not track consumer progress. A partition retains every message for a configured window regardless of whether anyone has read it, and each consumer (or consumer group) remembers its own position independently1. Section 3 works through what this buys and what it costs relative to a broker-tracked-acknowledgment queue.
  • The broker does not need an application-level cache. Because the log is just a file, the operating system's own page cache does the caching, and the data path from disk to network socket can skip user space entirely via sendfile(). Section 4 derives the copy-count arithmetic behind that claim.
  • "Exactly once" has to be built, not assumed. A log that is only appended to and only pulled from still sits on top of an ordinary, lossy, retry-prone network. Sections 5 and 6 build up from at-least-once delivery to an idempotent producer protocol, and then generalize the same three-state retry-safety machinery — absent, in-flight, complete — into the idempotency-key pattern that Stripe uses for payments and that Module 18 and Module 23 cite directly rather than re-deriving.

That third point is worth being precise about before the rest of the module leans on it. Nothing about a request/response network connection tells a client, after a timeout or a reset connection, whether the server executed the request and the acknowledgment was lost, or whether the request never arrived at all. A client that wants to guarantee the operation eventually happens has exactly one option — retry — and a client that retries without further protection has exactly one risk: the first attempt actually succeeded, and the retry duplicates it. This is true whether the "server" is a Kafka broker receiving a produce request, a payments API receiving a charge request, or any other RPC boundary; it is a property of asynchronous networks, not of any one product. Kafka's idempotent producer (Section 5) and the idempotency-key pattern (Section 6) are the same fix applied at two different layers of the same problem.

What this module does not cover: the compute frameworks — Spark Streaming, Flink, Kafka Streams — that actually read these logs and turn them into derived state or downstream side effects. Module 9 picks up exactly where a consumer's poll() call returns a batch of records and asks how a framework turns a stream of such batches into fault-tolerant, stateful computation.

2. Anatomy of a partition: segments, offsets, retention

Physically, a Kafka topic-partition is a directory on disk containing a sequence of segment files, each holding a contiguous run of the partition's records3. The original design paper sizes a segment at roughly 1 GB1, and the same size shows up in current Apache Kafka documentation as the configurable roll-over point: "This file is rolled over to a fresh file when it reaches a configurable size (say 1GB)"3. Each log file is named after the offset of the first record it holds, so the first segment in a partition is 00000000000000000000.log, and the next segment starts at whatever offset the roll-over happened to land on3. A record inside a log file is a 4-byte length prefix followed by that many bytes of record data; there is no separate index structure mapping a message id to a location, because there is no message id — a record's identity is its 64-bit logical offset, the byte position it would occupy in the stream of every record ever written to that partition3.

A partition as an ordered sequence of ~1GB segment files, addressed by offset Partition 0 on disk: three segment files 00000000000000000000.log offsets 0 – 9,412,006 ~1 GB, closed sealed (eligible for deletion once its newest record ages out) 00000000009412007.log offsets 9,412,007 – 19,004,881 ~1 GB, closed sealed 00000000019004882.log offset 19,004,882 → growing, still writable the tail 0 now older than retention.ms → whole segment deleted Deletion works at segment granularity, not per record: a segment is dropped only once its newest record has aged past the retention window (default 168 h = 7 days)
Figure 2. A partition's segments are named by their first offset, rolled at a configurable size (documented default illustration of ~1 GB), and addressed purely by logical offset rather than a separate id-to-location index13. The log manager deletes whole sealed segments once the largest timestamp in the segment ages past the retention window — 168 hours by default4 — never individual records3.

2.1 Why an offset instead of an id

Kafka's designers considered giving every message a GUID and maintaining a GUID-to-offset index, the way a database maintains a secondary index. They rejected it: a consumer already has to track a position per partition, so global uniqueness bought nothing, and a random-id index would have needed a genuine persistent random-access structure kept in sync with the log on every write3. Once you have a per-partition monotonic counter for lookup purposes anyway, using that counter directly as the externally visible offset is free — the identity of a message and its physical location in the log become the same 64-bit integer, and no index has to be maintained or kept crash-consistent with the data3. This is the same "constant time suffices" argument Kafka's design document makes about persistence generally: a B-tree gives you O(log N) random access, which sounds like constant time until you remember that on spinning disks a random seek costs roughly 10 ms and only one seek can be in flight per disk at a time, so a data structure with disk seeks embedded in its complexity gets slower, not the same speed, as data grows2. An append-only log with offset lookup needs no seeks at write time and only a handful at read time to locate a segment and an in-segment position, so its performance is decoupled from how much data has accumulated2.

2.2 Retention is a segment-level decision

Because a segment is a plain file, Kafka's log manager can decide what to delete by looking only at whole segments, never by scanning inside one: it applies a time policy (drop a sealed segment once the largest record timestamp inside it is older than the retention window) and, if enabled, a size policy (keep deleting the oldest segment until total partition size is back under budget), using a copy-on-write segment list so that reads never block on a concurrent delete3. The default time-based retention is log.retention.hours = 168, i.e. 7 days4 — matching the "typically 7 days" figure in the original 2011 design paper1, unchanged as the default fourteen years later.

Note

Whole-segment deletion is also why Kafka's storage cost model is closer to "log throughput × retention window" than to "count of messages currently unread." A slow consumer does not grow storage; the retention clock does not care whether anyone has read a byte.

Caveat

Retention-based deletion and consumer-paced reading are independent clocks, and that independence has a failure mode: if a consumer group's checkpointed offset falls behind the retention window, the segment holding that offset gets deleted out from under it. Kafka's own configuration reference names this case directly — auto.offset.reset governs "what to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted)"5. The consumer's choices are then to jump to earliest (reprocess whatever is still retained, silently skipping everything already deleted), jump to latest (skip straight to the tail, silently abandoning everything unread), or throw and require a human decision5. There is no option that recovers the deleted messages — retention is not a queue's backlog, it is a hard expiry, and a consumer that stops for longer than the retention window has permanently lost data, not merely fallen behind.

Worked derivation: segments per partition and disk footprint at a given throughput

Take a partition sustaining a produce rate of \(R\) bytes/second, with a segment size of \(S = 1\text{ GB} = 2^{30}\) bytes3 and a retention window of \(T = 168\text{ h} = 604{,}800\text{ s}\)4. A new segment rolls every \(S/R\) seconds, so the number of sealed segments retained at steady state is

$$ N_{\text{segments}} \approx \frac{T}{S/R} = \frac{T \cdot R}{S} $$

and the steady-state disk footprint for that one partition is simply \(T \cdot R\) (the segment granularity does not change the total bytes retained, only how the deletion sweep is chunked). At \(R = 10\text{ MB/s}\) — a single, moderately busy partition — a segment rolls every \(2^{30}/10^7 \approx 107\text{ s}\), giving \(N_{\text{segments}} \approx 604{,}800 / 107 \approx 5{,}650\) segments and a footprint of \(10\text{ MB/s} \times 604{,}800\text{ s} \approx 5.76\text{ TB}\) per partition. Doubling the segment size to 2 GB halves the segment count to roughly 2,825 without changing the 5.76 TB footprint at all — segment size trades off "how many file-system deletes happen per day" against "how much of a partially-stale segment you retain past when its oldest record would otherwise have expired," not total retained bytes. This is why segment size is a knob tuned for filesystem and recovery-scan overhead, while retention (in bytes or in time) is the knob that actually controls disk spend.

2.3 Why ordering is a per-key guarantee, not a per-topic one

A single log gives you a total order for free — record 4 precedes record 9 inside one partition, unambiguously, because the offset sequence is the order. That guarantee stops at the partition boundary by construction: a topic with multiple partitions has no total order across them, only per-partition order, and which partition a given record lands in is a producer-side decision, not something the broker imposes. Kafka's documentation describes the producer talking directly "to the broker that is the leader for the partition without any intervening routing tier," using cluster metadata to find that leader2, and then choosing a partition either at random — "a kind of random load balancing" — or by hashing a caller-supplied key, so that, in the documentation's example, "if the key chosen was a user id then all data for a given user would be sent to the same partition"2. That single design choice is what turns "ordered log" into "ordered log per key": pick a key with the granularity you need ordering at (one user, one device, one account), and every record for that key lands in the same partition in send order, giving you exactly the ordering guarantee that workload needs and no more, at the cost of that key's traffic being confined to one partition's throughput ceiling. This is also the mechanism that makes Table 1's "per-message priority" gap unavoidable rather than an oversight: a partition's contract is FIFO-by-offset within a key, and there is no broker-side hook to reorder records that are already sitting in a sealed or growing segment ahead of where a "higher priority" record would need to be.

2.4 Log compaction: retention keyed by identity, not by age

Time- and size-based retention (Section 2.2) both delete a whole segment once it ages out, regardless of what is in it. Kafka offers a second, orthogonal retention policy for the case where what a consumer actually needs is not "everything from the last 7 days" but "the current value for every key" — a changelog rather than an event log. Under log compaction, a partition keeps a dense, sequential head exactly like an ordinary Kafka log, but its tail is periodically recompacted so that only the most recent record for each key survives2. Deletion is expressed inside the log itself: "a message with a key and a null payload will be treated as a delete from the log," called a tombstone, and a tombstone removes any earlier record with that key (and any future one) until the tombstone itself is cleaned out after its own retention window2. Offsets are never renumbered to fill the gaps a compaction leaves — reading from an offset that got compacted away simply returns whatever record now sits at the next offset that does exist, so offset semantics from Section 2.1 hold even once records have been removed from the middle of the log2. Compaction itself runs as a background process that recopies segments without blocking reads2, which keeps it consistent with this module's broader theme: even a second retention policy is implemented as whole-segment file manipulation, not per-message broker bookkeeping.

3. Pull vs push, and who remembers what was read

Kafka's designers state the alternative they rejected explicitly: "brokers should push data to the consumer" is the design most logging-centric systems of the era (they name Scribe and Apache Flume) had chosen2. A push broker controls the rate of transfer, which sounds convenient until the consumer's processing rate falls behind the producer's — at which point a push system either buffers unboundedly or overwhelms the consumer, "a denial of service attack, in essence," in the documentation's own words2. A pull-based consumer instead falls behind and catches up on its own schedule; the failure mode degrades gracefully to "consumer is behind," not "broker is out of memory." Pulling also composes better with batching: a push broker must decide, per message, whether to send immediately (wasting round trips) or wait and risk added latency, without knowing whether the downstream consumer even wants a large batch, whereas a pull consumer's request itself specifies how much it wants, so the broker can hand back everything available up to that size in one round trip2. The one weakness a naive pull design has — a consumer polling in a tight loop when there is nothing new — is closed with a "long poll": the fetch request can block server-side until either data arrives or a timeout expires2.

Pull-based consumption permits rewind; push-based delivery does not Pull (Kafka) Push (traditional queue) Broker log Consumer fetch(offset=k, max=S) consumer chooses k rewind allowed Broker queue Consumer push next message, broker marks it sent ack/nack only — no re-read of a settled msg Consumer sets its own rate and offset Broker sets delivery rate; fast consumer overload risk
Figure 3. Under pull, the consumer's request names the offset it wants, so it can deliberately rewind — "this violates the common contract of a queue, but proves to be an essential feature," e.g. re-processing after fixing a consumer bug12. Under push, the broker decides delivery rate and a message that has already been marked delivered/acknowledged is gone; a push consumer that fell behind has no equivalent of "fetch an old offset again."

3.1 Where the position lives

The deeper design fork is not pull vs. push mechanics but where the state "what has this consumer already consumed" is kept. Kafka's own design documentation frames this as the central performance question of any messaging system2. The traditional answer — the broker records consumption, either immediately when a message is handed out, or after waiting for an acknowledgment — creates a hard trade: record consumption immediately and a crashed consumer silently loses whatever it hadn't finished processing (at-most-once); wait for an acknowledgment and a consumer that crashes after processing but before acking causes redelivery and double-processing, and now the broker must track two states per message — "sent-but-unacked" and "permanently consumed" — so it can be given out exactly once pending ack and removed exactly once acked2.

Kafka sidesteps the two-state bookkeeping by dividing every topic into partitions and letting exactly one consumer, within a subscribing consumer group, own each partition at a time; the consumer's position in that partition then collapses to a single integer — the offset of the next record to consume — which can be checkpointed cheaply and periodically, making the equivalent of an acknowledgment nearly free2. That checkpoint lives with the consumer (conventionally in an internal Kafka topic, __consumer_offsets, itself just another partitioned log), not inside the broker's per-message bookkeeping.

Table 1. Kafka's stateless-log broker vs. a traditional broker-tracked queue, by mechanism126.
DimensionKafka (log)Traditional queue (JMS/AMQP-lineage)
Who tracks read positionConsumer (group), as one offset integer per partition2Broker, per message, via delivery/ack state2
Delivery directionConsumer pulls2Broker pushes2
Re-reading a past messageDeliberate rewind to any retained offset1Not supported once acknowledged2
Deletion triggerRetention time/size on whole segments, independent of reads3Acknowledgment from consumer6
Routing to consumersNone inside the broker — partition assignment only2Exchanges/bindings: direct, fanout, topic, headers routing6
Per-message priorityNot supported — partition order onlySupported by design (priority queues)6
Failure mode of a slow consumerFalls behind; no effect on other consumer groups2Backlog grows on the broker; can require flow control2
Multiple independent readers of the same dataNative — every consumer group has its own offset1Requires fanout exchange/topic exchange to duplicate the message6

4. The broker doesn't cache — the OS does

A stateless log broker still has to serve reads fast, and Kafka's answer is to not build a cache at all. The design documentation is direct about this: "we rely on the underlying file system page cache," which "avoids double buffering" — no JVM heap holding a second copy of data the kernel is already holding in RAM — and "has the additional benefit of retaining warm cache even when a broker process is restarted"1. A process restart drops an application-level cache immediately; the OS page cache survives the broker process dying, because it belongs to the kernel, not to Kafka. Betting on the page cache only works because Kafka also committed to sequential, append/scan access patterns: the design document's "constant time suffices" argument notes that cheap 1+ TB SATA/JBOD drives have "poor seek performance" but "acceptable performance for large reads and writes," at a third of the price and three times the capacity of the alternative2 — exactly matched to a log that is written and read sequentially.

Serving a read still has to move bytes from a page-cache-backed file to a TCP socket, and this is where the second half of the "let the OS do it" strategy shows up: sendfile(). The Linux system call "copies data between one file descriptor and another. Because this copying is done within the kernel, sendfile() is more efficient than the combination of read() and write()," which would otherwise require "transferring data to and from user space"7. Kafka's own documentation spells out exactly what that traditional path costs and what sendfile() removes.

Traditional read-then-write path (four copies, two system calls) vs sendfile() zero-copy path (one copy) Traditional path: read() + write() sendfile() zero-copy path kernel space user space disk pagecache 1 app buffer 2 (read()) socket buf 3 (write()) NIC 4 4 copies, 2 syscalls, 2 user/kernel crossings disk pagecache 1 NIC 2 (sendfile) 1 copy total — data never enters user space not used when TLS is enabled (SSL runs in user space)
Figure 4. Kafka's own documentation counts the naive path exactly: "there are four copies and two system calls" — disk→pagecache, pagecache→user buffer (read()), user buffer→socket buffer (write()), socket buffer→NIC. With sendfile(), "only the final copy to the NIC buffer is needed"2; with multiple consumers reading the same recent data, that one pagecache copy is reused across all of them2. Kafka does not use this path when TLS/SSL is enabled, because SSL libraries operate in user space2.

The multi-consumer point is easy to underweight: because the copy from disk into pagecache happens once regardless of how many consumer groups are reading near the tail, a partition with ten independent, caught-up consumer groups costs barely more in disk I/O than one with a single consumer — the fan-out is absorbed by the kernel reusing its own cached pages, and each sendfile() call for each consumer adds only the one unavoidable copy out to that consumer's NIC buffer2. This is also why the documentation states that on a healthy, caught-up Kafka cluster you should expect to see "no read activity on the disks whatsoever," because reads are being served entirely out of pagecache2 — the same sequential-write, sequential-read locality that made the log format viable for persistence in the first place is what makes it viable for the read fan-out too.

The other half of Kafka's efficiency story is orthogonal to the disk path but shares its motivation: when the bottleneck is network bandwidth rather than CPU or disk — the documented case is a pipeline moving messages between data centers over a wide-area link — compressing messages individually captures little of the redundancy that actually exists, because "much of the redundancy is due to repetition between messages of the same type (e.g. field names in JSON or user agents in web logs or common string values)"2. Kafka's producer instead batches many records together, compresses the batch as one unit, and writes it to the log and transmits it to consumers still in compressed form; the broker only decompresses a batch to validate record counts against the batch header, and a consumer decompresses on its own end2. This keeps compression consistent with everything else in this section: the broker still isn't doing per-message work, the compressed batch is still what sits in pagecache, and sendfile() still moves the same bytes it would have moved uncompressed — batching and compression are a producer-side and consumer-side decision that the broker's storage and transfer path is indifferent to.

5. Exactly once as a built thing, not a network guarantee

Kafka's documentation enumerates the three delivery guarantees any messaging system can offer and is blunt about the third one: "at most once," "at least once," and "exactly once — each message is processed once and only once," adding that "many systems claim to provide 'exactly-once' delivery semantics, but it is important to read the fine print, because sometimes these claims are misleading"2. No transport layer hands you exactly-once for free; TCP retransmits, clients time out and retry, and brokers crash mid-acknowledgment. What actually exists underneath is at-least-once (the network may deliver a retry of something that already succeeded) or at-most-once (a timeout might mean the write never happened, or might mean the acknowledgment was lost) — and "exactly once" is a property you build on top of one of those two by making duplicates and gaps unable to change the outcome.

On the producer side, Kafka's own solution splits into two independent problems, both closed since the 0.11.0.0 release: idempotent production, and multi-partition transactions2. Before 0.11.0.0, a producer that got no acknowledgment "had little choice but to resend the message," giving plain at-least-once semantics with real duplicate risk on retry2. The idempotent producer closes that gap the same way sequence numbers close it in any reliable-transport protocol: "the broker assigns each producer an ID and deduplicates messages using a sequence number that is sent by the producer along with every message"2. A resend after a lost acknowledgment arrives at the broker carrying the same sequence number as the original, and the broker — which has already recorded that sequence number as committed — discards the duplicate instead of appending it a second time. This is the same idempotency idea Section 6 generalizes to an entire API surface, just implemented as a two-integer (producer ID, sequence number) pair scoped to one partition instead of a client-generated string scoped to an endpoint.

Transactions extend the same idea across partitions: "the producer supports the ability to send messages atomically to multiple topic partitions using transactions, so that either all messages are successfully written or none of them are"2. Without this, a producer writing a derived event to partition A and an audit record to partition B could crash between the two writes, leaving A updated and B not — visible to any consumer, and not something an idempotent-producer sequence number alone can fix, because sequence numbers only protect against the producer's own retries, not against the process dying between two separate commits.

Kafka's transaction design keeps the asymmetry Section 3 already established between producer and consumer: "in Kafka, the consumer and producer are separate, and it is only the producer which is transactional. It is however able to make transactional updates to the consumer's position (confusingly called the 'committed offset'), and it is this which gives the overall exactly-once behavior"2. Concretely, a transactional write bundles the output records and the input offset-commit into one atomic unit; the consumer side of the pipeline is expected to run with isolation.level=read_committed so that it never observes records from an aborted or still-open transaction, and with enable.auto.commit=false so its own offset advancement stays under the transaction's control rather than happening independently on a timer2. This is also why the earlier per-partition, single-owner-per-consumer-group assignment from Section 3.1 is a prerequisite for transactional exactly-once, not an incidental fact: the guarantee only holds because exactly one consumer instance is advancing exactly one offset per partition, so there is a single, well-defined "committed offset" for the transaction to update atomically2.

The durability side of "committed" is itself a designed quantity, not a binary: a message counts as committed only once every in-sync replica for that partition has applied it, and once committed, it survives as long as at least one replica that holds the partition stays alive2. A producer can trade this durability against latency explicitly — waiting for full commit costs "on the order of 10 ms," while a producer that only needs the leader (not the full in-sync set) to have the message, or that fires sends fully asynchronously, gets lower latency at a correspondingly weaker durability guarantee2. Replication mechanics themselves — in-sync replica sets, unclean leader election, what "alive" means for a partition — are the general leader/quorum replication machinery Module 3 already derives; this module only needs the "committed" definition that machinery produces.

The consumer side has its own three-way split, and it is a direct consequence of Section 3's finding that the consumer, not the broker, owns its offset. Reading a batch, saving the new offset, and then processing it gives at-most-once: a crash between the offset save and the processing means those records are silently skipped by whoever takes over the partition2. Processing first and saving the offset after gives at-least-once: a crash after processing but before the offset save means the same records get reprocessed by the next owner. Kafka's contribution to exactly-once consumption is not a magic guarantee but a coordination primitive: when the write that results from processing a record and the offset commit for that record can be made atomic — writing both into the same downstream transactional store, or using Kafka's own transactional API to write output records and commit the input offset in one transaction — the two outcomes ("processed" and "offset advanced") cannot come apart, and the two-way split above collapses to effectively-exactly-once for that pipeline2. It is this same at-most-once / at-least-once fork, resolved by making two independent writes atomic, that Section 6 generalizes outside the boundaries of one Kafka cluster.

6. The idempotency-key pattern, generalized

Kafka's producer-side fix — a per-producer ID plus a per-message sequence number the broker deduplicates against — only works inside one Kafka cluster, because only Kafka's brokers hold that dedup table. The same problem shows up at every network boundary in a distributed system: a client sends a request, the connection drops before the response arrives, and the client cannot tell whether the operation happened or not. If the operation is "charge a card" or "deduct usage credits," blindly retrying is unsafe (double charge) and refusing to retry is also unsafe (the operation silently never completes). Stripe's 2017 engineering post on this problem generalizes Kafka's producer-dedup trick into an HTTP-level pattern now copied across the industry: "a client generates a unique ID to identify just that operation and sends it up to the server along with the normal payload," and on any retry with that same ID, "the server simply replies with a cached result of the successful operation" instead of executing it again8.

Stripe's current API documentation is more precise about the mechanics than the original blog post: the server "sav[es] the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails," so that "subsequent requests with the same key return the same result, including 500 errors"9 — caching a failure is deliberate, because a genuinely failed operation should keep failing the same way on retry, not silently start succeeding due to some unrelated later state change. Keys are pruned "after they're at least 24 hours old," are recommended to be V4 UUIDs or an equivalently high-entropy random string, and are capped at 255 characters9. If a client reuses a key with different request parameters, the server treats that as a client bug and errors rather than either re-executing with the new parameters or silently returning the old result9.

6.1 The state a two-state model cannot represent

The mechanism only works because the server tracks a third state that a naive read of "has this key been used" misses. Collapse a key's status to two values — "unseen" and "seen, with a cached result" — and a request that arrives while the first attempt for that key is still executing has no correct place to go: reading "unseen" (because the first attempt hasn't committed its result yet) lets the retry execute concurrently and duplicate the side effect; reading "seen" too early returns a result that doesn't exist yet. Stripe's own reasoning about this is captured precisely in its idempotency framing: an in-flight key is neither absent nor complete, and treating it as either turns a legitimate retry into either a duplicate write or a hang8. The current API documentation shows the resolution directly: "if... the request conflicts with another request that's executing concurrently, we don't save the idempotent result because no API endpoint initiates the execution. You can retry these requests"9 — a concurrent hit on an in-flight key is explicitly rejected as retryable, rather than either duplicated or served a fabricated result.

Idempotency-key state machine: absent, in-flight, complete, and the unsafe two-state collapse ABSENT key never seen IN-FLIGHT executing, no result yet COMPLETE result cached first request execution finishes retry hits in-flight → rejected, retryable retry hits complete → cached result replayed Unsafe two-state collapse (absent / complete only): in-flight read as absent → concurrent retry executes twice in-flight read as complete → retry served a result that doesn't exist
Figure 5. An idempotency key must resolve to one of three states, not two8. Stripe's production behavior: a retry against a completed key replays the cached status code and body, including a cached error9; a retry against an in-flight key is rejected as retryable rather than executed concurrently or served a fabricated result9.
Table 2. What each idempotency-key state means and what breaks if it is misclassified89.
StateMeansCorrect response to a retryIf misclassified as absentIf misclassified as complete
AbsentKey never submittedExecute the operation, record the key as in-flight—Server returns a fabricated result for work never done
In-flightExecution started, no result recorded yetReject as retryable; do not execute a second time9Duplicate execution — e.g. a second charge8Client is told a result exists before it does; client may act on a result that later changes
CompleteResult (status + body) cached, including cached failures9Replay the cached result verbatimDuplicate execution of an operation that already finished—

6.2 Why this vocabulary travels beyond payments

Nothing about the absent/in-flight/complete state machine is specific to charging a card. Any operation that (a) has an externally visible side effect, (b) is invoked over a network that can silently drop a response, and (c) must not double-apply if retried, needs exactly this mechanism: a client-supplied key, a server-side record of that key's state, cached replay on a completed key, and rejection (not execution) on an in-flight key89. Automatic retry only becomes safe to build into a client library once this exists — Stripe's own client libraries retry failed requests "automatically with an idempotency key using increasing backoff times and jitter"8, which would silently create duplicate side effects on every dropped response if the server had no idempotency layer underneath the retry logic.

This is the general vocabulary Module 18 and Module 23 assume rather than re-derive: Module 18 examines how Stripe's ledger design makes the "record the key's state, execute once, cache the result" mechanism safe for money movement specifically, and Module 23 applies the identical three-state machine to deducting metered token usage against a running balance on an AI inference platform, where a client-visible retry after a dropped streaming response must not double-deduct credits for tokens the model already generated. Both modules can point back to this section for the state machine itself and spend their own word budget on what is specific to payments or to token metering.

Where experts disagree

This module has treated Kafka's log model as the default, but it is a deliberate bet that trades away features earlier messaging systems considered non-negotiable. The clearest place to see the trade is a direct comparison with the AMQP 0-9-1 model that RabbitMQ implements — a lineage descended from the same broker-tracked, routing-capable design Kafka's creators explicitly rejected.

The stateless log wins: let the broker be dumb and fast

Kafka's creators1 and the project's current design documentation2

Move the bookkeeping — consumption position, redelivery tracking, per-message state — off the broker and onto the consumer, and the broker degenerates into an append-and-scan machine that the OS page cache and sendfile() can serve close to network-line-rate2. A single log can be read independently, at independent rates, by an unbounded number of consumer groups without the broker doing any per-group routing work, and retention becomes a segment-deletion sweep instead of a per-message garbage-collection problem3. This is the right trade for high-volume event streams with multiple independent readers and a genuine need to replay.

  • Multi-consumer fan-out is nearly free — one page-cache copy serves every caught-up reader2
  • Deliberate replay/rewind is native, not a special case1
  • Broker-side state is O(partitions), not O(messages) or O(consumers × messages)
  • No broker-side routing: a consumer that wants only a subset of a partition's messages must filter after reading everything2
  • No per-message priority — a partition is strictly ordered2
  • A slow consumer group never causes backpressure on the broker, which can mask a downstream problem for a long time before anyone notices

The stateful broker wins: routing and per-message control matter more than fan-out cost

RabbitMQ / AMQP 0-9-1 model6

A broker that tracks delivery and acknowledgment per message can do useful things a dumb log cannot: route one published message to zero, one, or many queues based on content or pattern via exchanges and bindings6, redeliver automatically to a different consumer if the one holding a message dies without acking6, support negative acknowledgments that requeue a specific message for retry without touching anything else6, and drop or bounce a message that has no eligible destination rather than silently discarding it into a partition nobody reads6. For task-queue-shaped workloads — a job that must land on exactly one of many workers, with priority and complex routing rules — this is a materially richer feature set than "append and let every reader filter for itself."

  • Content-based and pattern-based routing at the broker, not reimplemented by every consumer6
  • Per-message redelivery and negative-ack semantics built in6
  • Natural fit for RPC/request-reply and priority-ordered task queues
  • Broker must track delivery/ack state per message, which is exactly the two-state bookkeeping problem Kafka's documentation identifies as the hard part to get right2
  • A message gone once consumed and acked cannot be replayed to a new, independent reader without a separate fanout mechanism6
  • Throughput at Kafka-scale event-stream volumes is a much harder engineering problem for a broker doing per-message state tracking
Assessment

Neither model is strictly better; they optimize for different consumption shapes. If the workload is "many independent readers over the same high-volume stream, replay is a first-class requirement, and ordering within a partition key is sufficient" — activity logging, change-data-capture, metrics pipelines, the training-data and observability pipelines later modules in this course build on — the log model's near-zero marginal fan-out cost and native replay dominate, and this is why Kafka displaced traditional brokers for exactly that workload class. If the workload is "discrete tasks that must be routed to the right worker, retried individually on failure, and sometimes prioritized" — background job processing, RPC-style request/response, work queues with SLA-sensitive ordering — a broker that tracks per-message state and offers routing primitives removes real engineering work a log-based system would otherwise push onto the consumer. In practice, large platforms run both: a log-based backbone for event/data movement and a task-queue-style broker for job dispatch, rather than treating the choice as system-wide. What would settle further disagreement is workload-level throughput/latency benchmarks under equivalent redundancy and durability settings — most public comparisons vary broker configuration in ways that make the two models look further apart than the underlying trade-off actually is.

What is changing

The local-disk assumption underlying Section 2 is loosening. Apache Kafka's tiered storage feature (KIP-405, disabled by default as of the 4.3.x documentation) splits a partition's segments into a local tier — the current, single-disk-per-broker design this module has described — and a remote tier that offloads completed (sealed) segments to an external store such as HDFS or S310. The rationale given is that most reads are "tail reads" served straight from page cache, while older data is read "infrequently," for backfill or failure recovery, so paying object-storage latency only on the cold path costs little10. This connects directly back to Module 7's object-storage material: a tiered-storage Kafka cluster's cold segments end up sitting on the same class of disaggregated storage fabric — S3-style or Tectonic-style — that module derives from first principles, rather than on brokers' own attached disks. It also changes the retention-cost arithmetic in Section 2's derivation: local-disk footprint becomes bounded by a much shorter local retention window, while total retained history can grow far past what any single broker's disks could hold, at object-storage economics rather than local-SSD economics.

References

  1. paperKreps, J., Narkhede, N., Rao, J. Kafka: a Distributed Messaging System for Log Processing. LinkedIn, NetDB '11 workshop, 2011. https://notes.stephenholiday.com/Kafka.pdf. Accessed 2026-09-17. Used for: ~1GB segment files (§3.1), page-cache reliance and warm-restart benefit (§3.1), pull model and deliberate rewind (§3.1), stateless broker and typically-7-day retention (§3.1), offset addressing (§3.1), sendfile() (§3.1).
  2. officialApache Kafka. Design. AK 4.3.x documentation. https://kafka.apache.org/43/design/design/. Accessed 2026-09-17. Used for: "database log, not traditional messaging system" framing, push-vs-pull rationale and long-poll, consumer-position/acknowledgment tradeoff and per-partition single-integer offset, message delivery semantics (idempotent producer/sequence numbers, transactions, committed-message definition, durability/latency tradeoff), consumer-side at-most/at-least-once split and atomic offset-commit resolution, "four copies and two system calls" sendfile quote, B-tree/disk-seek "constant time suffices" argument, TLS/sendfile incompatibility, end-to-end batch compression, producer partition/key load balancing, transactional producer/consumer split and read_committed isolation, log compaction and tombstones.
  3. officialApache Kafka. Implementation: Log. AK 4.3.x documentation. https://kafka.apache.org/43/implementation/log/. Accessed 2026-09-17. Used for: on-disk log format and segment file naming by first offset, 1GB configurable roll-over size, offset-as-message-id rationale (rejected GUID index), segment-level (not per-record) time/size deletion via copy-on-write segment list.
  4. officialApache Kafka. Broker Configuration. AK 4.3.x documentation. https://kafka.apache.org/43/configuration/broker-configs/. Accessed 2026-09-17. Used for: log.retention.hours default value of 168 (7 days).
  5. officialApache Kafka. Consumer Configuration. AK 4.3.x documentation. https://kafka.apache.org/43/configuration/consumer-configs/. Accessed 2026-09-17. Used for: auto.offset.reset and its handling of an offset the broker has already deleted (earliest/latest/none behavior).
  6. officialRabbitMQ. AMQP 0-9-1 Model Explained. Live documentation. https://www.rabbitmq.com/tutorials/amqp-concepts. Accessed 2026-09-17. Used for: exchange/binding routing model, message acknowledgement and negative-acknowledgement/requeue semantics, broker removes a message only on acknowledgement, unroutable-message drop/return behavior.
  7. standardLinux man-pages project. sendfile(2). https://man7.org/linux/man-pages/man2/sendfile.2.html. Accessed 2026-09-17. Used for: sendfile() syscall semantics vs. read()+write(), kernel-space-only copying.
  8. blogStripe (Brandur Leach et al.). Designing robust and predictable APIs with idempotency. Stripe Engineering Blog, 2017. https://stripe.com/blog/idempotency. Accessed 2026-09-17. Used for: Idempotency-Key header mechanic, cached-result replay on retry, automatic client-library retry with backoff and jitter, in-flight as a distinct third state.
  9. officialStripe. Idempotent requests. API reference, live documentation. https://docs.stripe.com/api/idempotent_requests. Accessed 2026-09-17. Used for: caching status code and body including 500 errors, 24-hour key pruning, V4 UUID recommendation and 255-character key limit, parameter-mismatch error on key reuse, concurrent-in-flight-request rejection as retryable.
  10. officialApache Kafka. Tiered Storage. AK 4.3.x Operations documentation. https://kafka.apache.org/43/operations/tiered-storage/. Accessed 2026-09-17. Used for: KIP-405 local/remote tier split, tail-read vs. infrequent-backfill-read rationale, HDFS/S3 as remote tier targets, disabled-by-default status.