System Design · Module 20

Airbnb: Search Ranking, Payment Idempotency, and the Monolith-to-SOA Migration

Airbnb has published three separate, detailed engineering accounts of the same two-sided marketplace: how its search ranker went from a hand-tuned scoring function through five model generations to a two-tower deep network, how its payments system stops a guest's card from being charged twice when a network call is retried inside a service-oriented architecture, and why it split a single Rails monolith into many independently-deployed services in the first place. After this module you can read any of the three source papers or posts directly and place its numbers in context, and you have the idempotency vocabulary Module 18 (Stripe) and Module 23 (AI-platform billing) build on next.

Course 40 min read 10 sources As of 2026-09-17 Prerequisite: Module 10
Three engineering problems inside one Airbnb marketplace Two-sided marketplace guests search, book, and pay hosts 1. Search ranking GBDT to two-tower DNN 2018-2020 KDD 2019 / KDD 2020 2. Payments consistency Orpheus idempotency library 2019-2022 five nines consistency 3. Service architecture Rails monolith to Thrift SOA 2017 3-week service creation 3 made independent deployment of 1 and 2 possible
Figure 1. Three problems inside one marketplace, each with its own primary source and its own numbers: search ranking's evolution to a two-tower network24, the Orpheus idempotency library that keeps payments from double-executing57, and the 2017 service-oriented-architecture migration that let those two teams ship independently1.

1. Search ranking: from a hand-tuned function to a two-tower deep network

Airbnb's search ranker decides which handful of listings, out of the thousands available for a guest's dates and location, gets shown and in what order. Airbnb calls this "one of the biggest machine learning success stories" at the company2, and it is one of the few large-scale ranking systems where the operating company has published the full sequence of models it shipped, including the ones that failed, with the reasoning behind each step. That makes it a useful case study for anyone who has only read the polished final architecture of a production ranker: you get to see the two-year path, not just the destination.

1.1 The GBDT plateau and five model generations

The very first Airbnb search ranker was a manually crafted scoring function2 — no learned model at all, just a formula an engineer wrote down. Replacing it with a gradient-boosted decision tree (GBDT) model produced, in Airbnb's words, "one of the largest step improvements in homes bookings in Airbnb's history," with many successful GBDT iterations following it2. Over time those GBDT gains saturated: "long spells of neutral experiments" is how the paper describes the plateau that made the company willing to try something as disruptive as replacing the model family entirely2.

The transition to deep learning was not one jump from GBDT to a deep network. It was five generations, each motivated by a specific failure of the previous one2:

  1. Simple NN. A single hidden layer of 32 fully-connected ReLU units, fed the exact same features as the GBDT model, trained with L2 regression loss (booked listings assigned utility 1.0, unbooked 0.0). It was deliberately unambitious — Airbnb frames the decision as following Andrej Karpathy's advice not to be a hero on your first production neural network. Result: booking-neutral against GBDT. Its only value was proving the serving pipeline could carry live traffic end to end2.
  2. LambdaRank NN. The same network, but with two changes to the training objective: (a) a pairwise formulation over {booked, not-booked} listing pairs from the same search, instead of independent regression targets, and (b) each pair's loss weighted by the change in NDCG that would result from swapping the two listings' rank positions — so moving a booked listing from rank 2 to rank 1 matters far more than moving one from rank 10 to rank 92. This was the first architecture Airbnb calls a "breakthrough."
  3. GBDT + FM + NN. A hybrid that fed the NN two additional signals: the leaf-node index activated per tree from the GBDT model (as a categorical feature) and the booking-probability prediction from a separately-trained 32-dimensional factorization machine2. It matched the simpler models on test-set metrics but up-ranked a noticeably different set of listings — evidence that the three model families were each finding different signal in the same data.
  4. Deep NN (2 layers). Rather than adding more architectural complexity on top of generation 3, Airbnb deprecated all of it and instead scaled the training data 10x, landing on a plain two-hidden-layer network: 195 input features (after expanding categorical features to embeddings) feeding a 127-unit ReLU layer, then an 83-unit ReLU layer2. Training on 1.7 billion listing pairs closed the gap between training and test NDCG that had appeared in earlier, smaller-data models2.
  5. Two-tower architecture (2020). Covered in 1.4 below — a query/user tower and a listing tower scored by Euclidean distance, motivated by a specific pricing failure the deep NN exhibited.

The pattern across generations 1 through 4 is not "add more layers." Airbnb explicitly tested adding depth, residual connections, and batch normalization to the fully-connected network later on and got neutral offline NDCG every time4 — its own conclusion was that going deeper is "an effective technique for convolutional neural networks, but not necessarily for all DNNs," and that for a fully-connected ranking network, two hidden layers were already enough capacity4. The gains instead came from fixing the training objective (LambdaRank weighting), fixing the data scale (10x, then Protobuf/tf.data pipelines), and — as sections 1.2 through 1.4 show — fixing what the features looked like before they reached the network at all.

Timeline of Airbnb search-ranking model generations Hand-tuned scoring fn gen 0 GBDT largest single gain gen 1 Simple NN booking-neutral gen 2 LambdaRank NN first breakthrough gen 3 GBDT+FM+NN hybrid features gen 4 Deep NN 2 layers, 10x data gen 5 Each step was motivated by a specific failure of the previous step, not by adding architectural complexity (the 2020 two-tower architecture, generation 6, is covered in §1.4)
Figure 2. Airbnb's own account of its five model generations before the 2020 two-tower architecture, each launched to production traffic and tested against the previous generation in an online A/B test2. Airbnb's original figure plots relative NDCG and booking gains per generation but does not publish the exact percentages, so this timeline shows sequence and motivation rather than invented magnitudes.

1.2 What broke when GBDT features met a neural network

The first attempt to train the Simple NN on the exact same features used for the GBDT model went "very badly": training loss saturated partway through and additional steps had no effect2. The root cause was feature scale. A decision tree only cares about a feature's relative ordering — a price of \$500 versus \$5,000 only matters through which side of a threshold it lands on. A neural network's first layer computes a weighted sum of raw feature values, so an unnormalized feature with occasional huge values can produce huge gradients, which in turn can permanently zero out a ReLU unit through vanishing-gradient dynamics2.

The fix was a normalization discipline applied to essentially every input feature, so that the bulk of each feature's distribution sits inside \([-1, 1]\) with the median mapped to 02:

  • Features that look roughly normal: standard z-score, \((x - \mu)/\sigma\).
  • Features that look like a power law (most price-like and count-like features at Airbnb's scale): \(\log\!\big(\frac{1+x}{1+\text{median}}\big)\).

Airbnb also used the smoothness of a feature's distribution as a debugging tool in its own right, independent of whether the model trained successfully: a spike in an otherwise-smooth distribution was, in practice, how the team found bugs like prices logged in the wrong currency for certain geographies, or monthly prices being mistakenly logged as daily prices for stays over 28 days2. This is a case where a data-quality check and a modeling-quality check turned out to be the same plot.

The other feature-engineering pattern worth naming is what Airbnb did with location. Raw latitude/longitude has a heavily skewed distribution once you account for how zoomed-out a map view is; converting to an offset from the displayed map's center and then taking the log of that offset produced a usably smooth feature2. For within-city location preference, instead of maintaining a hand-built hierarchical pipeline tracking booking distributions by neighborhood and city, Airbnb hashed the pair (query city, listing's level-12 S2 geometry cell) into an integer and fed that integer straight into a learned embedding — the specific example the paper gives is the query "San Francisco" combined with a listing's S2 cell 539058204 hashing to categorical id 718295212. The learned embedding for that feature turned out to visibly encode neighborhood preference, including a preference for locations slightly further into the west bay over locations across traffic-heavy bridges — a pattern nobody hand-engineered2.

Table 1. Airbnb's search-ranking model generations, in the order they shipped to production2.
GenerationArchitectureTraining signalWhat changed vs. previous
GBDTGradient-boosted treesRegression on booked = 1Replaced a hand-tuned scoring function
Simple NN1 hidden layer, 32 ReLU unitsL2 regression, same GBDT featuresProved the NN serving pipeline production-ready
LambdaRank NNSame networkPairwise, ΔNDCG-weightedDirectly optimizes rank position of booked listing
GBDT+FM+NNNN + GBDT leaf index + FM score as featuresPairwiseCombines three model families' signal
Deep NN2 hidden layers (127, 83 units), 195 inputsPairwise, 1.7B pairs10x more training data instead of more layers
Two-towerSeparate query/user and listing towers, Euclidean distancePairwise, triplet-styleFixes a pricing interpretability problem (§1.4)

1.3 Listing embeddings: a skip-gram model over click sessions

In parallel with the ranker's own model evolution, Airbnb built a second representation-learning system: 32-dimensional embeddings for listings, trained not on hand-engineered features but on the sequence of listings a user clicked through during a search session3. The technique is a direct adaptation of skip-gram word embeddings from NLP — anyone who has trained word2vec will recognize the mechanism immediately, which is why it is worth drawing explicitly for an ML-systems reader rather than describing in prose alone.

A click session is defined as an uninterrupted sequence of listing clicks by one user, with a new session starting after any 30-minute gap3. Training uses negative sampling with a sliding window of length \(m = 5\): at each step, the vector of the central listing is pushed toward the vectors of listings clicked within the window before and after it (positive context), and pushed away from the vectors of randomly sampled listings (negative context)3. Two domain-specific modifications were needed on top of the base algorithm3:

  • Booked listing as global context. If a session ends in a booking, the booked listing's vector is pulled toward the central listing at every step of that session, not just when it happens to fall inside the sliding window — it is a "global" context term that never leaves the objective as the window slides.
  • Market-level negatives. Guests searching within one destination market produce positive-context listings almost entirely from that market, while random negative sampling draws from the entire global inventory. Without correction, this imbalance taught the model that "different market" and "dissimilar" were the same thing, degrading within-market similarity. The fix adds an explicit extra negative set sampled specifically from the central listing's own market.

The resulting embeddings were trained for 4.5 million active listings from more than 800 million search click sessions3 — using only 32 floating-point numbers per listing, small enough to keep every listing's vector resident in the RAM of a search-serving machine for real-time cosine-similarity computation3. New listings, which by definition have no click history to train from, get a cold-start embedding by averaging the vectors of the 3 geographically nearest listings that share the same listing type and price range3.

Skip-gram training mechanism for Airbnb listing embeddings One click session, sliding window m = 5 L1 L2 L3 (central) L4 L5 L6 positive context (within window, pushed closer) Booked listing global context: pulled toward central listing at every step, all session long Random listing negative context (random, pushed away); an extra set is drawn from the same market as the central listing to fix within-market similarity 4.5M listings · >800M click sessions · d = 32 · k = 12 nearest neighbors for Similar Listings
Figure 3. The skip-gram mechanism behind Airbnb's listing embeddings, adapted from word2vec's negative-sampling objective with two Airbnb-specific terms — booked-listing global context and market-level negatives3. An A/B test of the resulting embedding-based Similar Listings feature raised carousel click-through by 21% and increased the share of guests who discovered the listing they eventually booked through that carousel by 4.9%3.

The same embeddings feed back into the main search ranker as two real-time personalization features, computed from a rolling 2-week history maintained in Kafka for each user: EmbClickSim, the similarity between a candidate listing and the market-level centroid of listings the user recently clicked, and EmbSkipSim, the similarity to listings the user was shown but skipped in favor of a lower-ranked click3. Partial-dependence plots on the trained ranking model confirmed the features were used as intended — higher EmbClickSim pushed the model's score up, higher EmbSkipSim pushed it down — and the features launched to production search ranking in the summer of 20173.

1.4 The ABCs: architecture, bias, and cold start

Shipping the deep NN did not end the ranking team's work; it changed what the work was. Airbnb's 2020 follow-up paper frames the two years after the initial deep-learning launch as three separate problems it names architecture, bias, and cold start4.

Architecture. The team noticed that each successful ranking-model launch had been associated with a drop in the average price of search results, and hypothesized the model still hadn't converged on guests' true price preference. Removing price as a direct input and replacing it with an explicit monotonically-decreasing \(\tanh(\text{price})\) output term made the model's price behavior interpretable, but online it traded bookings for cheapness in the wrong direction: average price fell 5.7% and bookings fell 1.5%4. A "generalized monotonicity" architecture — squared weights that provably preserve a monotonic path from price to output, alongside an unconstrained subnet for everything else — produced a similar 1.6% booking drop online4. Both failures pointed to the same conclusion: forcing the network's price behavior to obey a hand-specified constraint was actively worse than whatever the unconstrained network had already learned. Individual Conditional Expectation (ICE) plots — sweeping one listing's price across a range while holding every other feature fixed, then plotting model score — showed the plain fully-connected DNN already understood "cheaper is better" on its own4; the real problem was that Airbnb's global price preference estimate was biased toward high-volume markets, and the fix was a two-tower architecture (a 100-dimensional query/user vector and a 100-dimensional listing vector, scored by Euclidean distance) that let the model learn the right price for a given query rather than a single global price preference4. Tested online against the fully-connected deep NN, the two-tower model produced +0.6% bookings, +0.7% online NDCG, and an overall +0.75% revenue increase, with average price falling 2.3% as a side effect of better relevance rather than as a direct objective4.

Bias. A model trained on logs of a previous model's rankings inherits that model's positional bias: a listing that was booked partly because it happened to be ranked first, not because it was the best match, teaches the next model that "ranked first" and "should be ranked first" are the same signal. Airbnb's fix adds position as an explicit input feature — regularized away from the relevance signal using dropout at a rate tuned to 0.15 — and fixes the position input to 0 at serving time, so relevance and positional-bias contributions to the score can be estimated and then separated4. Tested online, this produced +0.7% bookings and, unexpectedly, +1.8% revenue, plus a +1.1% booking increase specifically for boutique hotels and bed-and-breakfasts, a segment that had been quietly penalized by inherited positional bias4.

Cold start. New listings, by definition, lack the historical engagement features (past bookings, clicks, reviews) that the DNN leans on heavily — removing those features entirely dropped overall NDCG by 4.5%4, and new listings' booked-listing NDCG ran 6% below established listings' before any fix4. Rather than treating this as an explore/exploit ranking-boost problem, Airbnb reframed it as an estimation problem: predict what a new listing's engagement features will converge to, using the average of nearby listings matched on guest capacity and adjusted for seasonality. That estimator cut offline engagement-estimation error by 42% versus hand-set default values, and in an online test increased bookings of newly-created listings by 14%, increased their first-page impression share by 14%, and lifted overall bookings by 0.38%4.

Worked derivation: why the two-tower architecture cut p99 scoring latency 33%

Airbnb reports that separating the query/user tower from the listing tower reduced the computational complexity of the first hidden layer from \(O(N \cdot H \cdot (Q + L))\) to \(O(N \cdot H_l \cdot L + H_q \cdot Q)\), where \(N\) is the number of listings being scored in one search, \(Q\) and \(L\) are the number of query/user-only and listing-only input features, and \(H\), \(H_l\), \(H_q\) are hidden-layer widths4. The intuition is that in the fully-connected architecture, the query and user features — identical for every listing in the search result — get recomputed through the first hidden layer once per listing, even though their value never changes within that search; the two-tower split lets the query/user tower be evaluated exactly once per search and reused across every listing.

Airbnb does not publish the exact \(N\), \(H\), \(Q\), \(L\) values it ran in production, so the following plugs in illustrative, clearly-labeled numbers just to show the shape of the saving, not Airbnb's real configuration. Take a search result of \(N = 50\) listings, a fully-connected hidden width of \(H = 200\), and roughly balanced feature counts \(Q = 50\) (query/user) and \(L = 150\) (listing):

$$ \text{Fully connected: } N \cdot H \cdot (Q+L) = 50 \times 200 \times (50+150) = 2{,}000{,}000 \text{ multiply-adds} $$

Now split into a two-tower model with a listing-tower width \(H_l = 100\) and a query-tower width \(H_q = 100\):

$$ \text{Two-tower: } N \cdot H_l \cdot L + H_q \cdot Q = (50 \times 100 \times 150) + (100 \times 50) = 750{,}000 + 5{,}000 = 755{,}000 $$

That is roughly a 2.6x reduction in first-layer multiply-adds under these illustrative sizes — the same direction and rough order of magnitude as the -33% p99 latency figure Airbnb reports online4, without claiming to reproduce Airbnb's exact number, since the real \(H\), \(Q\), and \(L\) are not published. The mechanism generalizes past this one architecture: any feature that is constant across all \(N\) candidates being scored in one pass is a candidate for hoisting out of the per-candidate loop, which is the same optimization a retrieval-and-rerank pipeline gets from computing a query embedding once and reusing it against every candidate's precomputed embedding.

Reported online A/B-test results for the architecture, bias, and cold-start fixes 0% +15% -3% +0.6%bookings +0.7%NDCG -2.3%price +0.75%revenue A: two-tower +0.7%bookings +1.8%revenue +1.1%boutique B: position dropout +14%new-listing bookings +14%1st-page share +0.38%overall bookings C: cold start
Figure 4. Online A/B-test deltas Airbnb reports for each of the three 2020 fixes, each measured against the previous production model4. The two-tower architecture (A) trades a small price drop for relevance and revenue gains; position dropout (B) recovers bookings that inherited positional bias had suppressed, concentrated in under-ranked segments like boutique hotels; the cold-start engagement estimator (C) moves the metric it targets by far more than it moves overall bookings, because it only touches inventory without engagement history.

2. Booking and payments: making "charge the guest" happen at most once

The search-ranking case study is about getting a probability estimate right. The payments case study is about something with a much harder failure mode: getting a probability estimate wrong costs you a slightly worse-ranked listing; getting a payment operation wrong charges a real guest's card twice or fails to pay a real host at all. Airbnb's own account of this problem, and the library it built to solve it, is a direct, concrete instance of the idempotency vocabulary Module 8 introduced in general terms1 — this section is where that vocabulary meets a production payments system end to end.

2.1 Why SOA turned a function call into a distributed transaction

Module 10 covers why Airbnb split its Rails monolith into independently-deployed services and what that migration cost and bought back in general1. One specific consequence Module 10 does not chase down is what happens to a single logical operation — "charge this guest" — once it stops being one function call inside one process and becomes a chain of network calls across several services, each of which can change state and have side effects. Airbnb's payments engineers describe this plainly: such a chain "is equivalent to executing a complex distributed transaction," and without a coordination protocol such as two-phase commit, distributed transactions in this shape threaten data integrity, graceful degradation, and consistency all at once5. Requests also simply fail sometimes — connections drop, calls time out — and that failure mode gets more likely, not less, the more network hops a logical operation crosses5.

Airbnb explicitly did not reach for two-phase commit — the coordination protocol Module 4 covers, and the one Spanner runs underneath every cross-shard transaction to get synchronous, externally-consistent commits at the cost of blocking every participant until every other participant is ready9. Airbnb's payments system instead relies on write repair: every write call from a client to a server is built to be safely retriable, and a client that suspects something went wrong simply fires the same call again5. This shifts the entire correctness burden onto one property: the server-side operation has to behave identically no matter how many times the same logical request arrives. That property is idempotency, and it is the same always-writeable, retry-tolerant philosophy Dynamo built its entire replication design around a decade earlier, just applied to a single service's request handling instead of a whole key-value store's replica set10. Getting idempotency right is what the rest of this section is about. Where experts disagree returns to why Airbnb chose this path over 2PC.

2.2 Orpheus: three phases, two rules

Airbnb's requirements for an idempotency mechanism were specific enough to rule out the two obvious options5. A single custom solution per use case wasn't acceptable, because Airbnb runs many payments services on SOA and wanted one generic mechanism all of them could adopt. A separate, stand-alone idempotency service wasn't acceptable either, for a reason worth sitting with: payments needed ultra-low latency, and a dedicated idempotency service sitting in the request path would itself be a distributed dependency subject to exactly the network failures the mechanism was supposed to protect against5. The solution Airbnb shipped, called Orpheus, is therefore a library: code that runs inside each payments service's own process, reading and writing that service's own database, adding no extra network hop5.

Orpheus is built on the observation that almost any API request that talks to an external system decomposes into three phases5:

  1. Pre-RPC. Record the details of the incoming request in the local database, before anything is sent over the network.
  2. RPC. Make the live network call — to an external payment processor, an acquiring bank, or any downstream service — and receive its response.
  3. Post-RPC. Record the outcome of that call in the local database, including whether a failure response is retryable or not.

Two ground rules keep this decomposition safe5: no service call over the network happens during the Pre-RPC or Post-RPC phases, and no database interaction happens during the RPC phase itself. Network calls and database transactions are never mixed in the same phase. The Pre-RPC and Post-RPC database writes are each wrapped in their own single atomic transaction (implemented with Java lambdas so the library, not the application developer, owns transaction boundaries), so each phase can only fully succeed or fully fail — there is no partially-committed state for a crash to leave behind5. The reasoning behind separating network calls from database work this strictly is operational, not theoretical: Airbnb found that RPCs issued from inside a database-transaction context were "vulnerable" in practice, causing rapid connection-pool exhaustion and performance degradation when a downstream call was slow5 — mixing an unreliable, variable-latency operation (the network call) with a resource that has to be held open and returned quickly (a database connection inside a transaction) is a bad combination regardless of which framework you use.

Orpheus three-phase request structure Pre-RPC record request DB transaction only RPC call downstream service network only, no DB Post-RPC record response retryable? DB transaction no network calls here no DB writes here no network calls here A retry re-enters at whichever phase the previous attempt did not commit past
Figure 5. Orpheus's phase separation: every payments API request is split into Pre-RPC, RPC, and Post-RPC, with database transactions strictly confined to the two RPC-free phases so that a network failure can never leave a half-committed database write behind5.

Exception handling gets the same discipline. Airbnb classifies every error as either retryable — transient network/infrastructure failures (its own description maps these to 5XX-style statuses), where a later retry might succeed — or non-retryable — validation or state errors (4XX-style), such as trying to refund a payment that was already refunded, where every retry will fail identically5. Misclassifying in either direction has a real cost: labeling a transient error non-retryable strands the request in permanent failure, while labeling a permanent error retryable is exactly the kind of mistake that produces a double payment5.

2.3 The replica-lag double payment, and why idempotency state lives on master

The single design decision Airbnb calls out as most consequential is where idempotency state is read from. It would be natural, especially for a read-heavy table, to serve idempotency lookups from a read replica instead of the master database. Airbnb explicitly avoids this, and the failure mode it describes is worth drawing out in full5: a client submits a payment request; the request succeeds downstream and the response is written to the master database; but the client never receives that response, because the network itself failed on the way back. The client, following the idempotency contract, retries with the same key. If that retry is served by a replica that has not yet caught up to the master's write — replica lag of even a few seconds — the replica reports "no response found for this key," and the service concludes the original request never happened. It executes the payment again. The guest is charged twice, and the mechanism that was supposed to prevent exactly this is the thing that let it through, because it was reading from the wrong copy of the data.

Replica-lag double payment versus master-only reads Reading from a lagging replica Reading from master only client master DB replica Request 1 (charge) success, written replicates, lagged (response lost to client) Retry (same key) "not found" — replica not caught up yet Payment executed again — duplicate charge client master DB Request 1 (charge) success, written (response lost to client) Retry (same key) found immediately on master Cached response returned — no duplicate
Figure 6. The exact failure mode Airbnb describes: a retry served from a lagging replica cannot see a response the master already committed, so the service re-executes the payment5. Reading and writing idempotency state only on the master database eliminates this window entirely, at the cost of routing every idempotency check to the single most-loaded copy of the data.

Orpheus's tables of idempotency information are therefore always read from and written to the sharded master database, never a replica5. Because Airbnb's underlying databases are MySQL and are not configured for strong read-after-write consistency on replicas, this was the only option that closed the race outright rather than narrowing its window5. Routing every idempotency read to master obviously does not scale on one box forever, so the idempotency tables are themselves sharded — by the idempotency key, chosen specifically because idempotency keys have high cardinality and an even distribution, which is exactly what a shard key needs5.

2.4 Choosing a key, leasing a retry, and the five-nines result

Orpheus supports two different notions of what "the same request" means, and the choice is a business decision, not a technical one5:

  • Request-level idempotency. The client generates a fresh random key (a UUID is the given example) for every logically distinct request. This is the default, and it is what you want when a guest could legitimately make several different payments against the same reservation — Airbnb's own example is Pay Less Upfront, where a single booking is deliberately split into multiple separate charges over time.
  • Entity-level idempotency. The client derives a deterministic key from the entity being acted on, such as payment-1234-refund. Every refund request against payment 1234 collapses onto the same key, so even if two literally separate client calls both try to refund it, only one refund executes.

Once a key arrives, the request that owns it acquires a database row-level lock on that key, which grants a time-bounded lease5. If the server side never responds — the exact race a timeout produces — a retry using the same key is only allowed to proceed after the current lease expires; Airbnb's stated rule of thumb is to set the lease expiration longer than the RPC timeout, so a slow-but-eventually-successful call cannot be raced by its own retry5. On top of the lease, Orpheus enforces a maximum total retryable window per key, as a backstop against a client whose retry logic has gone rogue and is hammering the same key indefinitely5.

The result Airbnb reports, measured after the framework had been in production use, is five nines — 99.999% — of consistency for its payments, achieved while annual payment volume simultaneously doubled5. A 2022 follow-up post describes the same framework carried into a fully rebuilt payment-orchestration system, where every major payment workflow is now structured as a directed acyclic graph of retryable idempotent steps built on Orpheus, and the same five-nines figure is reported again years into that rebuild7. That post also extends Orpheus past synchronous request/response calls: for asynchronous communication over Airbnb's Kafka-based event bus between payments services, event processors apply the same idempotency framework to turn Kafka's at-least-once delivery guarantee (Module 8) into an effectively-exactly-once one from the payments system's point of view7.

Table 2. Two production idempotency mechanisms solving the same problem — a retried request must not execute twice — at different layers of the stack56. Module 18 covers Stripe's ledger and metering design in full; this row is a preview.
DimensionAirbnb OrpheusStripe Idempotency-Key
Enforcement pointInside each payments microservice, as an embedded libraryAt Stripe's API edge, keyed by a client-supplied HTTP header
Key granularityRequest-level (random) or entity-level (deterministic), chosen per use caseClient-generated unique ID per logical operation
In-flight stateRow-level lock with a time-bounded lease on the keyTreated as a distinct third state between absent and complete
Where state livesSharded master database, by design never a replicaStripe-operated cache of the first response, including cached error responses
Client retry policyClient-implemented; Airbnb specifies backoff/jitter as a client responsibilityStripe's own client libraries retry automatically with exponential backoff and jitter
Reported resultFive nines (99.999%) payment consistency at 2x volume growthNot separately quantified in the source used here (see Module 18)

3. The monolith-to-SOA migration, as the case study that made 1 and 2 possible

Module 10 already did the general work this topic needs: it uses this exact Airbnb source to build the vocabulary for what a monolith-to-SOA migration costs (partial failure, coordination moving from compiler to wire, operational multiplication) and what it buys back (independent deployability), and it walks through why Airbnb built a customized Thrift-based Service IDL running over HTTP instead of adopting an existing RPC framework outright1. This section does not repeat that analysis. It exists to close the loop on the two case studies above: neither the two-year search-ranking iteration in Section 1 nor the payments rebuild in Section 2 could have happened as independent, separately-shipped efforts without the service boundary this migration drew.

3.1 What's specific to this case study

Airbnb's own framing of the migration is candid about cost: moving off a monolithic Rails service while simultaneously shipping new products and features is, in the company's words, "not without its challenges"1. What Module 10 treats generically, this module can now make concrete: search ranking and payments are exactly the two domains where that cost bought back the most, because they are the two domains this module has just shown evolving on their own two-year-plus timelines, each with its own model-iteration or framework-rebuild cadence, independent of whatever the rest of the product was doing.

The payments organization's own account of its slice of the migration adds one detail Module 10's general treatment does not need: Airbnb's legacy payment data model was tightly coupled to the concept of a single-product reservation for a stay, so every new product (the example given is Airbnb Experiences) required rebuilding payment logic with its own product-specific quirks7. Payments used the SOA migration as an opportunity to redesign around a product-agnostic model instead of doing a one-to-one lift of the existing code, which is why its migration ran in four broad phases — Pricing, Payouts, Bookings, and Data Migration — rather than the smaller independent functional blocks other teams used, with the Bookings phase alone further subdivided into more than 30 separate milestones7. Because historical bookings were not moved as part of the initial cutover, Airbnb ran the legacy monolith and the new SOA payment system in parallel for most of the migration, which required a transformation layer that could serve a unified data model to client services regardless of which backend actually held a given booking's data7.

Table 3. The payments organization's four-phase SOA migration, a redesign-first path rather than a one-to-one port7.
PhaseWhat it did
PricingRemodeled per-product pricing logic into one generic pricing model usable across all Airbnb products
PayoutsRedesigned host money-movement orchestration around a product-agnostic model
BookingsRedesigned guest-side payment orchestration; subdivided into 30+ independently-testable milestones
Data MigrationMoved existing bookings/payouts from the legacy system once the new system was validated, allowing monolith deprecation
Monolith splitting into three independently-deployed service domains Rails monolith one deploy, one blast radius Service IDL (Thrift/HTTP) Search service ships model gens independently Payments service runs Orpheus, own deploy cadence N other services messaging, pricing, ... Each box independently deployable, its own release cadence -- Module 10 covers the general cost/benefit here
Figure 7. The general monolith-to-SOA topology Module 10 introduces1, specialized to this module's two case studies: search ranking and payments become services that can each run a multi-year iteration cadence (Sections 1 and 2) without coordinating a release with every other team.

3.2 Service creation in three weeks, and the bridge to Module 21

The mechanism that made splitting cheap enough to be worth doing at all — rather than a one-time investment nobody could afford to repeat — was tooling, not the IDL by itself. Airbnb's "make-me-a-service" scaffolding tool took several new Java services from inception to handling real production traffic in three weeks, an estimated savings of two to three weeks of engineering time per service compared to the prior manual process1. That is the number that turns "split the monolith" from a one-off architectural decision into a repeatable pattern a team can apply every time a new bounded problem (a new payments phase, a new ranking-model serving path) needs its own deploy cadence.

Service creation timeline using the make-me-a-service scaffolding tool Service inception IDL defined Scaffolded via make-me-a-service Production traffic 3 weeks, end to end vs. an estimated 2-3 additional weeks of engineering time under the prior process
Figure 8. Airbnb's reported timeline for taking a new Java service from inception to production traffic using its Service IDL and scaffolding tooling1.

The theme that carries forward past this module is the interface layer, not the transport underneath it. Airbnb's Service IDL made the compile-time contract a monolith gives you for free into something explicit and machine-checkable at a network boundary, without touching the HTTP stack every existing piece of infrastructure already knew how to operate1 — and Borg's job/task abstraction, already introduced in Module 10 as the general admission-control and deployment-unit precedent, is the model for what a "service" becomes as an operational unit once it is deployed this way8. Module 21 picks up a structurally similar problem one layer up the stack: an AI gateway has to define a request/response contract for a "provider call" the same way Airbnb's Service IDL defined one for a Thrift-over-HTTP call, except the callee on the other side of that boundary is a model-serving backend instead of another internal microservice. The lifecycle discipline this module's Section 2 built for payments — a request that must execute at most once, retried safely by a client that owns an idempotency key — reappears there almost unchanged, applied to a token-generating request instead of a money-moving one.

Where experts disagree

Two genuine disagreements run through this module, both visible inside Airbnb's own published record rather than imported from outside it: whether deep learning actually retires feature engineering or just moves it somewhere less visible, and whether idempotency belongs inside each service as a library or at the edge of the API as a gateway-level contract.

Deep learning retires feature engineering

The optimism Airbnb itself describes at the launch of its first deep NN2, and the general framing behind representation-learning approaches like listing embeddings3

A neural network fed with minimally-processed features and enough data will learn the interactions and transformations an engineer used to derive by hand. Airbnb's own account of the payoff is direct: "trying to do better math on the features manually has lost its luster," freeing the team to work on the optimization objective and user representation instead of endless per-feature transforms2. The listing-embedding work is the strongest version of this claim put into production: 32 learned numbers per listing implicitly encode location, price tier, and even architectural style, replacing what would otherwise be a hand-built similarity function3.

  • Removes a large, ongoing maintenance burden of hand-derived ratios and windowed aggregates.
  • Learned representations (embeddings) generalize to uses nobody hand-designed them for, such as real-time personalization features built on top of a similarity model trained for a different purpose.
  • What the network actually learned is opaque: Airbnb's own attempts at score decomposition, ablation testing, and permutation importance each produced results the team judged nonsensical or misleading before landing on a workable tool (TopBot)2.
  • Opacity is not free when a feature has business consequences: nobody could confirm from a partial-dependence plot alone whether the model understood "cheaper is better" correctly, because partial dependence itself assumes feature independence that does not hold in a DNN4.

Structure has to be engineered back in

Airbnb's own 2020 architecture, bias, and cold-start work, documented as a direct response to problems the "pure" deep NN created4

Once the deep NN was in production, Airbnb had to reintroduce exactly the kind of explicit structure the first position claims deep learning made obsolete: a price-monotonicity constraint (twice, in two different forms, both of which made things worse), an explicit position-as-control-variable term regularized with dropout to counter inherited positional bias, and a hand-designed engagement-estimation model to patch a data gap the network could not learn its way around4. The pattern across all three fixes is that the network's architecture, not just its input features, needed engineering attention driven by a concretely observed user problem, not by importing an architecture from a literature survey4.

  • Every fix Airbnb reports came with a real, cited online A/B-test number, which is a higher bar than "the offline metric improved."
  • The failures of the two stricter monotonicity architectures are informative on their own: they show that naively re-imposing hand-specified structure can be worse than the unconstrained network, which the first position's critics often assume is never true.
  • Constructing each fix required first localizing a specific user problem (skewed price preference, an under-ranked segment, missing engagement history) through ad hoc analysis tools built for the occasion — this is not a repeatable recipe so much as a research process that happened to produce three results.
  • The interpretability tools this position leans on (ICE plots, TopBot) are themselves homegrown and unvalidated against ground truth, since Airbnb states plainly that it has no objective record of which listing a guest "should" have booked2.

Idempotency as an embedded per-service library

Airbnb's Orpheus design57

Put the idempotency mechanism inside each service's own process, reading and writing that service's own database. Airbnb explicitly rejected a stand-alone idempotency service specifically because payments needed ultra-low latency and did not want to introduce a new distributed dependency subject to the very failures the mechanism exists to survive5. This keeps idempotency logic close to the data it protects and lets each service shard its own idempotency table by its own key space.

  • No added network hop or dependency in the critical path.
  • Generalizes past simple request/response: the same framework was later extended to make Kafka event processing effectively exactly-once, which a purely edge-level mechanism could not reach7.
  • Every service that wants idempotency has to adopt the library and follow its two ground rules correctly; Airbnb's own retrospective admits the phase-separation discipline can become "spaghetti" without careful engineering5.
  • Provides no protection for a caller that talks to a service which has not adopted the library — idempotency is a property of the specific service, not of the API surface as a whole.

Idempotency as an API-edge contract

Stripe's Idempotency-Key header design, covered in full in Module 186

Put the mechanism at the boundary the client actually calls: a client-generated key travels in a request header, and the API layer itself caches and replays the first response — including a cached error — for any retry carrying the same key6. Stripe's own client libraries handle the retry loop (exponential backoff with jitter) automatically, so the idempotency contract is something every caller gets by default rather than something each internal service has to separately implement correctly6.

  • One place to get the contract right; every endpoint behind the same API gateway inherits it uniformly.
  • The client-visible header makes the contract explicit and auditable from outside the company, which matters for a payments API with external integrators, unlike Orpheus's fully internal library.
  • Caching a full response at the edge assumes the edge layer can safely see and store that response, which is a harder assumption for a deep internal call chain with side effects fanning out to several downstream services than for a single external-facing API call.
  • Does not, by itself, solve the internal fan-out problem Airbnb describes — a client-facing key can guarantee the outer call executes once without guaranteeing every downstream RPC inside that call also executes exactly once, which is what pushed Airbnb to solve idempotency at each internal hop instead of only at the edge.
Assessment

Both disagreements resolve to "it depends on where the hard constraint actually binds," and Airbnb's own record is unusually explicit about where that turned out to be for a two-sided marketplace. On feature engineering: representation learning wins decisively wherever the target signal is genuinely high-dimensional and only weakly constrained by domain knowledge (listing similarity, personalization) — the embeddings needed almost no domain-specific structure and outperformed a hand-built similarity function outright3. But wherever a feature has an externally-known, one-directional relationship with the business outcome (price should not increase a ranking score, holding everything else equal) and getting that relationship visibly wrong has revenue or fairness consequences, unconstrained representation learning is the wrong default: Airbnb tried the constrained version twice, watched both attempts underperform the unconstrained network, and only succeeded once it stopped trying to force behavior and instead built an architecture (two towers) that let the network discover the query-conditional right answer on its own4. The lesson generalizes past search ranking: constrain the objective or the architecture's inductive bias, not the feature's raw value, when you need guaranteed monotonic behavior. On idempotency: an embedded per-service library is the right choice specifically when a single client-visible operation fans out into multiple independent side-effecting calls across service boundaries, because only per-hop idempotency can stop a partial failure from re-executing one of those inner calls while leaving the others alone — which is Airbnb's actual situation. An edge-level header is the right choice when the API surface is genuinely one call in, one side effect out, from the caller's point of view, which describes most of Stripe's charge/refund API and explains why Stripe's simpler mechanism is sufficient for its use case without needing Orpheus's internal phase discipline. Evidence that would settle which layer matters more for a given system: count the number of independently-failing network hops behind one client-visible API call — one hop favors the edge-level header, several hops favor the embedded library, and Airbnb's own move to wrap Kafka event processing in the same framework is direct evidence its hop count was large enough to need it7.

References

  1. blogGuo, Liang. Building Services at Airbnb, Part 1. The Airbnb Tech Blog, published 2017-12-12. https://medium.com/airbnb-engineering/building-services-at-airbnb-part-1-c4c1d8fa811b. Accessed 2026-09-17. Used for: the monolith-to-SOA migration framing, Service IDL/Thrift-over-HTTP design, and the make-me-a-service scaffolding tool's measured 3-week/2-3-week-saved figures (§3; also the primary source for Module 10).
  2. paperHaldar, Malay, Mustafa Abdool, Prashant Ramanathan, Tao Xu, Shulin Yang, Huizhong Duan, Qing Zhang, Nick Barrow-Williams, Bradley C. Turnbull, Brendan M. Collins, and Thomas Legrand. Applying Deep Learning To Airbnb Search. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19), pp. 1927-1935; arXiv:1810.09591v2. https://arxiv.org/abs/1810.09591. Accessed 2026-09-17. Used for: the five search-ranking model generations (§1.1), feature normalization and S2-cell hashing (§1.2), and the failed listing-id/multi-task attempts and interpretability tools discussed in the disagreement section.
  3. blogGrbovic, Mihajlo, Haibin Cheng, Qing Zhang, Lynn Yang, Phillippe Siclait, and Matt Jones. Listing Embeddings in Search Ranking. The Airbnb Tech Blog, published 2018-03-13 (companion post to Grbovic & Cheng, Real-time Personalization using Embeddings for Search Ranking at Airbnb, KDD 2018, pp. 311-320). https://medium.com/airbnb-engineering/listing-embeddings-for-similar-listing-recommendations-and-real-time-personalization-in-search-601172f7603e. Accessed 2026-09-17. Used for: the skip-gram listing-embedding mechanism, training scale, cold-start rule, and the 21%/4.9% Similar Listings A/B-test results (§1.3).
  4. paperHaldar, Malay, Prashant Ramanathan, Tyler Sax, Mustafa Abdool, Lanbo Zhang, Aamir Mansawala, Shulin Yang, Bradley Turnbull, and Junshuo Liao. Improving Deep Learning For Airbnb Search. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2020); arXiv:2002.05515v1. https://arxiv.org/abs/2002.05515. Accessed 2026-09-17. Used for: the architecture/bias/cold-start fixes and their reported online A/B-test deltas (§1.4), and the two-tower latency-complexity formula used in the worked derivation.
  5. blogChew, Jon, and Ninad Khisti. Avoiding double payments in a distributed payments system. The Airbnb Tech Blog, published 2019-04-16. https://medium.com/airbnb-engineering/avoiding-double-payments-in-a-distributed-payments-system-2981f6b070bb. Accessed 2026-09-17. Used for: the write-repair framing, the Orpheus three-phase design and its two ground rules, the replica-lag double-payment failure mode, request-level vs. entity-level keys, lease/retry mechanics, and the five-nines result (§2.1-§2.4).
  6. blogStripe (Brandur Leach et al.). Designing robust and predictable APIs with idempotency. Stripe, 2017. https://stripe.com/blog/idempotency. Accessed 2026-09-17. Used for: the Stripe Idempotency-Key header contrast in Table 2 and the disagreement section (§2.4; full treatment in Module 18).
  7. blogRoss, Bryon, Feifeng Yang, Sophie Behr, Theresa Johnson, Xin Lin, and Yun Jin Cho. Rebuilding Payment Orchestration at Airbnb. The Airbnb Tech Blog, published 2022-02-24. https://medium.com/airbnb-engineering/rebuilding-payment-orchestration-at-airbnb-341d194a781b. Accessed 2026-09-17. Used for: the DAG-of-idempotent-steps redesign, the Kafka-exactly-once extension, the four-phase payments migration (Pricing/Payouts/Bookings/Data Migration), and the parallel-systems/transformation-layer detail (§2.4, §3.1).
  8. paperVerma, Abhishek, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. Large-scale cluster management at Google with Borg. EuroSys '15. https://research.google.com/pubs/archive/43438.pdf. Accessed 2026-09-17. Used for: the job/task deployment-unit precedent referenced when bridging to Module 21 (§3.2).
  9. paperCorbett, James C., Jeffrey Dean, Michael Epstein, et al. Spanner: Google's Globally-Distributed Database. OSDI '12. https://research.google.com/archive/spanner-osdi2012.pdf. Accessed 2026-09-17. Used for: the two-phase-commit/distributed-transaction alternative named in §2.1.
  10. paperDeCandia, Giuseppe, Deniz Hastorun, Madan Jampani, et al. Dynamo: Amazon's Highly Available Key-value Store. SOSP '07. https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf. Accessed 2026-09-17. Used for: the always-writeable, eventually-consistent design point that Airbnb's write-repair approach resembles (§2.1).