System Design · Module 23
Billing and Metering: Ledger Design, Idempotency, Holds, and Reconciliation
Module 18 derived Stripe's idempotency-key ledger and its two metering designs for money. This module asks what changes when the thing being deducted is a token count that is still growing at the moment the deduction has to start: a credit ledger needs a durable in-flight state that checkpoints a running total, a hold that reserves the worst case before the true cost is known, and a reconciliation step that captures the actual and releases the rest — the same shape Stripe already built for a hotel deposit, applied to a still-streaming completion.
1. Deducting credit exactly once for an event that has not finished happening
Module 18 derives the mechanism this section reuses, so it is worth stating precisely what is being reused before changing anything. Stripe's mutating endpoints accept a client-generated Idempotency-Key, the server caches the status code and body of the first request made under that key, and a retry with the same key returns the cached result rather than re-executing — with a durable third state, in-flight, sitting between absent and complete so that a crash mid-request does not force a naive two-state read into either double-charging a card or serving a result that does not exist yet. A token-metered credit deduction is a mutation of exactly the same shape: it debits an account, it must survive a retried write from a crashed billing worker or a redelivered event from Module 8's at-least-once broker, and getting the in-flight window wrong has the same two failure directions payments have — read as absent, a retry double-deducts; read as complete, a caller is told the deduction succeeded before it has.
The instantiation is almost mechanical. Where Stripe keys a charge retry on a client-supplied header, a usage-metering pipeline keys a credit deduction on an identifier the platform already has for other reasons — a request id, a completion id, or the (business, id) pair Stripe's own ledger uses to tolerate out-of-order arrival1. LiteLLM's own architecture description shows this reuse happening in exactly the place Module 8 and Module 18 would predict: spend logging and rate-limit accounting run as asynchronous background tasks after the response has already been returned to the client, with no database write sitting in the request's critical path9 — the same decoupling of "the side effect happened" from "the ledger recorded it" that makes a durable in-flight state necessary in the first place, because the recording step can now fail or be delayed independently of the generation it is recording.
1.1 The three-state model, extended for a quantity that is still growing
A card charge is atomic: at the instant the card network responds, the amount is either authorized or it is not, and the in-flight window is bounded by one network round trip. A streaming completion breaks that assumption in a way Module 18 does not have to deal with. Between the moment a request is admitted and the moment it finishes, the correct answer to "how much has this request cost so far" is not absent and not complete — it is a specific, increasing number, and it changes many times a second for as long as the model keeps generating. Collapsing that window to a boolean in-flight flag, the way a payment retry can safely do, throws away the one piece of information a billing ledger most needs if the process managing the stream crashes: how many tokens had already been generated and already been billed for, versus how many the retry or the recovery path is about to generate and bill again.
Both major providers' own streaming protocols already carry exactly the running total a ledger needs, because they need it too — to tell a caller how much of a response it is paying for as the response arrives. Anthropic's Messages API streams a fixed event sequence — message_start, one or more content blocks, one or more message_delta events, then message_stop — and states outright that "the token counts shown in the usage field of the message_delta event are cumulative"2. That single word is the load-bearing detail: a consumer does not have to sum per-chunk deltas to reconstruct a running total and risk double-counting a chunk it processed twice during a retry — each message_delta already restates the total tokens generated so far, so the last value observed before a crash is, by construction, a safe checkpoint to resume billing from. OpenAI's Chat Completions streaming takes the opposite shape and is explicit about the resulting risk: with stream_options: {"include_usage": true} set, "the value for the usage field on all chunks except for the last one will be null," and only "the last chunk contains the token usage statistics for the entire request"3. Nothing is checkpointable until the very end — and the same documentation names the exact failure mode that creates: "if the stream is interrupted or cancelled, you may not receive the final usage chunk which contains the total token usage for the request"3. A ledger built against a design that only reveals usage in a final, possibly-never-delivered chunk cannot rely on the provider's own accounting at all in the crash or cancellation case; it has to count tokens itself, chunk by chunk, exactly the way Anthropic's cumulative field makes unnecessary but OpenAI's final-chunk field makes mandatory.
Each checkpoint write is itself exactly the kind of mutation Section 1 opened with: it must survive being retried without double-counting. Because Anthropic's message_delta usage field is cumulative rather than incremental2, a checkpoint write is naturally idempotent in the strongest sense — writing "running total = 190" twice, whether because a delta was redelivered by an at-least-once internal queue (Module 8's guarantee) or because a billing worker crashed and replayed its own recent history on restart, produces the same row either time, rather than a sum that would double-count on the second write. A design forced onto a final-chunk-only usage field, by contrast, has no such natural idempotency for its intermediate checkpoints — if the platform is counting deltas itself token by token to build a running total, each individual delta's contribution has to carry its own sequence number (Table 4's checkpoint sequence field) so a redelivered delta can be recognized and skipped rather than added a second time, which is Module 8's idempotency-key pattern reapplied one level down, to the stream of deltas rather than to the request as a whole.
2. Holds: reserving the worst case before the true cost is known
A checkpointed running count fixes the recovery problem — it does not, by itself, protect a balance from ever going negative. A prompt admitted at a moment when a caller has 40,000 credits remaining, with max_tokens set to a value whose worst-case cost is 60,000 credits, can be perfectly well-behaved for its first several hundred tokens and only reveal the problem once generation has already run past the point a purely reactive, checkpoint-and-bill-as-you-go design would have caught it. Stripe's payments platform solved the equivalent problem for money decades before token metering existed, and it did not solve it by billing reactively — it solved it by reserving the money before doing the thing the money pays for.
2.1 The hold: Stripe's separate authorization and capture, applied to a credit balance
Stripe calls the pattern manual capture: "you can place a hold on an eligible payment method to reserve funds that you can capture later," and its own example is a direct structural match for a still-running generation — "hotels often authorise a payment in full before a guest arrives, then capture the money when the guest checks out"4. The hotel does not know the guest's final bill (minibar, room service, a late checkout fee) at check-in any more than a token-metering platform knows a generation's final length at admission; both authorize against a worst case and settle against the actual. Stripe's documentation is explicit that this is not a vague convention but a scheduled, expiring reservation: card authorizations are typically held "for 7 days for online payments," PayPal "holds the amount for 10 days" and Stripe "automatically attempts to extend the hold for another 10 days, totalling 20 days," Klarna's authorization must be captured "by midnight of the 28th calendar day," and Affirm gives "30 days to capture the payment balance"4 — every payment method Stripe supports for this pattern has its own explicit expiry, after which "the funds are released and the payment status changes to canceled"4. A token-credit hold needs the same property for the same reason: an abandoned or stuck in-flight state must not tie up a customer's balance indefinitely, so the hold itself needs an expiry independent of whether the generation it was reserved for ever reports back.
The other detail worth carrying over exactly is what happens on settlement. "To capture the authorised funds, make a PaymentIntent capture request. This captures the total authorised amount by default. To capture less... pass the amount_to_capture option. A partial capture automatically releases the remaining amount"4. That single sentence is the entire reconciliation step Section 2.3 needs: the platform does not compute a release amount as a separate operation with its own failure mode — capturing the actual usage and releasing the remainder are the same API call, so there is no window in which the actual has been captured but the remainder has not yet been freed, or vice versa. A credit ledger built the same way — one "settle" operation that both debits the actual and frees the unused hold atomically — inherits that same absence of a window, rather than reintroducing it by treating capture and release as two separate ledger writes that could, in principle, be observed half-done.
2.2 Sizing the hold: the arithmetic, and what it costs
The hold amount is fixed by the one number a platform actually controls at admission time: the maximum number of output tokens the caller has declared it is willing to receive. Let \(p_{in}\) and \(p_{out}\) be the per-token input and output prices, \(t_{in}\) the (known, tokenizable) prompt length, and \(m\) the declared max_tokens. The worst-case cost — and therefore the hold — is:
and the amount actually owed once the stream finishes, having generated \(t_{out} \le m\) tokens, is:
$$ A = t_{in}\, p_{in} + t_{out}\, p_{out}, \qquad \text{release} = H - A = (m - t_{out})\, p_{out} $$Substituting current, official per-token prices makes the overhead concrete rather than abstract. Anthropic's published pricing for Claude Sonnet 5 is $2 per million input tokens and $10 per million output tokens5 — that is, \(p_{in} = 2\times10^{-6}\) and \(p_{out} = 10\times10^{-6}\) dollars per token. Take a request with a 3,000-token prompt and max_tokens set to 8,192 (a common ceiling for a long-form answer):
| Quantity | Value | Formula |
|---|---|---|
| Prompt cost (fixed) | $0.006000 | 3,000 × $2 / 1,000,000 |
| Hold, H (worst case, m = 8,192) | $0.087920 | $0.006 + 8,192 × $10 / 1,000,000 |
| Actual, short completion (tout = 600) | $0.012000 | $0.006 + 600 × $10 / 1,000,000 |
| Release at tout = 600 | $0.075920 | H − $0.012 (86.4% of the hold returned) |
| Actual, typical completion (tout = 2,048) | $0.026480 | $0.006 + 2,048 × $10 / 1,000,000 |
| Release at tout = 2,048 | $0.061440 | H − $0.026480 (69.9% of the hold returned) |
| Actual, full-length completion (tout = 8,192) | $0.087920 | = H exactly |
| Release at tout = 8,192 | $0.000000 | no overhead when the model uses the full budget |
The arithmetic itself is trivial; what it demonstrates is that the hold's cost is not a fixed tax, it is proportional to how far a caller's declared max_tokens overshoots what the model actually generates. At a typical, well-short-of-the-ceiling completion length, 70-85% of the reserved amount in this example is money the caller's other requests cannot touch for the seconds or minutes the generation is running, even though it was never going to be spent. This is precisely the same overhead Stripe's own hotel example carries for a card — the deposit at check-in is sized to a plausible worst case, not to the bill the guest will actually run up — and precisely why Section 4 of Module 22 treats max_tokens as a parameter with no rate-limit downside on Anthropic's platform specifically because it is not used as a reservation basis there; the hold pattern in this section is a distinct design choice with a real, quantifiable cost, not a free property of the platform.
2.3 Reconciliation: capturing the actual, releasing the rest
Tying Sections 1.2 and 2.1-2.2 together: the in-flight state's checkpointed running count is exactly the input the settlement step needs once a stream reaches its final message_stop (or, for a provider on OpenAI's design, its final usage-bearing chunk3). Settlement is the transition from in-flight to complete in Figure 2, and it is the single "capture A, release H − A" operation Section 2.1 borrows from Stripe's amount_to_capture semantics4 — not two separate writes. If the checkpointed count at settlement time disagrees with a final usage field the provider reports (possible if checkpointing under-samples relative to the true generation, or if a provider's own accounting includes a token type the platform's checkpoint loop did not track, such as thinking tokens on a model that streams them separately), the ledger settles against the provider's own authoritative final count and treats the checkpointed value as an estimate that was only ever meant to bound the in-flight risk, not to be the number actually billed.
A hold-bearing ledger row therefore needs more fields than the simple absent/in-flight/complete flag Module 18 derives for a payment retry — it needs to carry the reservation and the running checkpoint as first-class values a settlement can read and act on, not just a status string.
| Field | Purpose | What breaks without it |
|---|---|---|
| Hold amount, H | The worst-case reservation computed at admission (Section 2.2) | No basis for the release computation at settlement; the ledger cannot tell how much was ever unavailable to other requests |
| Checkpoint sequence number | Orders successive running-total updates so a delayed or duplicated delta cannot overwrite a later checkpoint with an earlier one | A retried or out-of-order delta can roll the running count backward, undercharging on crash recovery |
| Checkpointed running total | The last known-good count to resume billing from after a crash (Figure 2) | Recovery has to guess how much was already billed, reintroducing the double-charge/under-charge risk Module 8's idempotency pattern exists to remove |
| Hold expiry timestamp | Bounds how long an abandoned in-flight row can lock up credit (Section 2.4) | A crashed or orphaned generation ties up its full worst-case hold indefinitely |
| Captured amount, released amount | The two outputs of the single settlement operation (Section 2.1's atomic capture-and-release) | Splitting these into two separate writes reopens the half-done window Stripe's own atomic amount_to_capture call avoids4 |
2.4 Extending a hold for a long-running generation
Section 2.1's hold-expiry property assumes the hold outlives the generation it covers by a comfortable margin — true for a single completion that finishes in seconds, and not obviously true for an agentic tool-use loop or an extended-thinking session that can run for minutes and issue many model calls before returning a final answer. Stripe's own payment-method documentation already names this exact problem for card holds: "usually, an authorisation for an online card payment is valid for 7 days. To increase the validity period, you can place an extended hold on an online card payment"4 — a single fixed expiry does not fit every use case, so the platform exposes a way to extend it rather than forcing every integration to either accept a short window or engineer around one.
A token-credit hold has the identical failure mode at a much shorter timescale: a hold sized and timed for a single request's generation will, on a long enough agentic session, expire before the session that placed it has finished — either releasing credit the session is still actively using (letting a concurrent request spend it and starving the original session mid-loop) or, if the expiry is enforced more strictly, forcing the platform to treat an otherwise-healthy long-running session as abandoned. The two available fixes mirror Stripe's own two supported patterns exactly: extend the existing hold's expiry each time the session demonstrably makes progress (a new checkpoint arrives, a new tool call starts), the direct analogue of Stripe's extended-authorization request; or replace one long-lived hold with a fresh, shorter one placed at each step of the loop, settling the previous step's hold as that step's actual usage becomes known rather than trying to keep a single reservation alive for the whole session. Either design still ends at the same place Section 2.3 describes — capture the actual, release the rest — just applied once per step of a multi-step session instead of once per request.
3. Two designs for usage visibility, redrawn for tokens
Module 18 derives the two designs Stripe itself operates for making accumulated usage visible: Metronome's continuously-tracked running balance, and the legacy Billing Meters path, where "Stripe processes meter events asynchronously, so aggregated usage in meter event summaries and on upcoming invoices might not immediately reflect recently received meter events"6. That comparison is not re-derived here. What changes for a token-metered platform running holds is what staleness actually costs, because a hold, unlike an invoice line, is a number a second request is actively waiting on.
Under end-of-period aggregation, a hold placed against one request has no path to being released early even after that request finishes, because the aggregation job that would notice the completion and free the reservation only runs on its own schedule — in the worst case, not until the billing period closes. Every hold a caller has open simultaneously stacks against their balance for as long as the aggregation lag lasts, which is a materially different failure than Module 18's stale-dashboard-number framing: a customer running several concurrent long generations under an async-aggregation design can be locked out of a new request by holds that have already, in fact, completed and should have been released minutes ago. Under continuous tracking, the moment a settlement operation writes its capture-and-release, the balance a subsequent admission check reads reflects it immediately, because there is no separate aggregation step to wait for.
Metronome's own API surface makes this an explicit, queryable choice rather than an implicit property of the pipeline. Its balance endpoint accepts an invoice_inclusion_mode parameter with two values — FINALIZED, which excludes draft invoices, and FINALIZED_AND_DRAFT, the default, which includes pending draft-invoice deductions in the balance calculation7. That choice matters for exactly the reason this module cares about staleness: an admission check deciding whether to place a new hold wants the FINALIZED_AND_DRAFT view — the most current picture of what is already committed, including usage that has not yet made it onto a finalized invoice — while a number shown to a customer as "your last confirmed bill" wants FINALIZED. A single balance-tracking system that exposes both views as a parameter, rather than picking one, is the concrete API-level answer to the same question Module 18's Figure 5 draws as two separate visibility curves.
| Dimension | Continuous (Metronome-style) | End-of-period aggregation (legacy) |
|---|---|---|
| Hold-release latency | Immediate — visible to the next admission check as soon as settlement is written | Bounded by aggregation cadence; a finished hold can stay locked until the next run6 |
| Spend-cap enforcement lag | None by construction | Up to a full aggregation interval — a caller can exceed a cap before the system notices |
| Balance query granularity | Selectable via invoice_inclusion_mode (finalized-only vs. finalized+draft)7 | One number, only as current as the last aggregation |
| Effect of many concurrent holds | Each releases independently the instant it settles | All outstanding holds stack until the shared aggregation run clears them together |
4. Invoicing rollup: from key to workspace to org
Module 22 derives the identity hierarchy underneath an AI platform's billing in full — how OpenAI and Anthropic each incidentally reveal an organization/project or organization/workspace structure through their rate-limit documentation, and how spend limits and rate limits are two mechanisms with two different error signatures at every level of it. This section needs only the rollup direction: once a credit deduction is settled at the key level, where does it end up counted for the purpose of generating one invoice, and what constraint governs how far a lower level can spend against a higher one.
Anthropic's own rate-limit documentation states the org/workspace relationship in exactly the terms an invoicing rollup needs: "spend limits set a maximum monthly cost an organization can incur for API usage," and a workspace can be given a limit "below the org-wide limit" specifically "to protect other Workspaces from overuse," with the constraint that "unused Workspace allocation is available to other Workspaces" while "organization-wide limits always apply, even if Workspace limits add up to more"8. The arithmetic that follows from this is asymmetric in the direction that matters for billing: the sum of every workspace's allocation can legally exceed the org ceiling, because allocations are soft caps on any one workspace's share of a shared pool, not a partition of it — but the org ceiling itself is a hard, additive constraint on what actually gets spent, because it is computed from the same settled ledger rows regardless of which workspace's key triggered them.
LiteLLM's proxy architecture shows the same rollup in a system built to sit in front of several providers at once rather than a single vendor's own API: "spend for that user is tracked... team in the LiteLLM_TeamTable"9 — a virtual key's settled spend attributes up to a user_id and, if the key is attached to one, a team_id, which is the same key-to-workspace-to-org shape under different names. The practical consequence for a ledger is that a single settled row (Section 2.3's capture) is not written once; the amount is attributed at every level of the hierarchy it belongs to simultaneously, because a key-level spend limit, a workspace-level spend limit, and an org-level invoice all need to observe the same fact without re-deriving it from each other.
| Level | What is metered here | What "invoice" means at this level |
|---|---|---|
| Key | Every settled capture attributed to this key, plus any open holds9 | Usually not invoiced directly — feeds the levels above |
| Workspace / project | Sum of its keys' settled spend against its own soft allocation8 | An internal cost-center rollup, not always a separate bill |
| Organization | Sum across every workspace, checked against one hard monthly spend cap8 | The actual invoice: one bill, one cap, regardless of how workspace allocations were divided |
The org level is also where this module's hold-based design and a purely postpaid design part ways, and Anthropic's own published billing terms make a clean, contrasting example rather than a hypothetical one. Anthropic's first-party API bills a prepaid or invoiced organization directly against the same per-token prices Section 2.2 uses, but usage routed through a marketplace — Claude Platform on AWS or Claude in Microsoft Foundry — is metered differently: token usage is rated in USD at standard rates, converted to a billing unit ("100 CCU represents $1.00 USD of fees owed"), and reported "hourly" to the marketplace, with the explicit terms "arrears only (postpaid); no prepaid credits"5. A purely arrears/postpaid organization has no balance for a hold to protect in the first place — there is nothing to reserve against, because nothing is drawn down from a pre-funded pool; the marketplace simply bills for whatever was used, after the fact, once a month. Every mechanism this module builds — the hold, the checkpointed running total, the release on settlement — exists specifically because a prepaid credit balance, unlike an arrears invoice, can hit zero mid-request, and a request that would push it negative has to be refused or interrupted rather than merely appearing as a larger number on next month's bill.
5. Abuse and fraud control touchpoints
A stolen API key produces a usage pattern that looks, to a billing ledger, exactly like a card-fraud pattern looks to a payments ledger: activity that is technically authorized (the key is valid, the card number is real) but statistically unlike the account's own history — a sudden spike in request volume, generation from a geography or at an hour the account has never used before, or usage that saturates every dimension of a rate limit simultaneously rather than the one or two dimensions a legitimate workload usually stresses. Stripe's Radar product is built around exactly this framing for payments, and its documented mechanism transfers directly: Radar assigns "each card payment a fraudulent payment score between 0 and 99"10, and separately, Stripe's own health-alert system watches for two shapes of anomaly at the account level rather than the single-transaction level — a spike, "a single day increase in your fraud rate," and a sustained_attack, "an ongoing or multi-day increase"10. Neither of those two categories is specific to card numbers; they are velocity checks on an account's own behavior against its own baseline, which is exactly the signal a compromised API key produces well before anyone manually reviews the account.
What makes a usage pattern "statistically unlike the account's own history" is more legible on a token-metered platform than on a generic payments ledger, because the per-dimension rate-limit counters Module 22 derives are already a decomposition of usage into independent axes, and a stolen key tends to move all of them at once in a way a legitimate workload rarely does. A legitimate integration built against one product surface typically has a stable ratio between its request count and its token counts — a chat application's RPM and ITPM move together because each request carries roughly the same conversation length — while a key being used to, for instance, run a completely different workload than the one the account normally runs (bulk generation instead of interactive chat, or a different model tier entirely) shows up as a shift in that ratio, not just a rise in absolute volume. A pure volume-threshold alert misses this; a ratio- or shape-based check, watching whether a key's usual RPM:ITPM:OTPM proportions have moved rather than only whether any one of them crossed a fixed ceiling, catches a theft that stays under every individual dimension's raw limit but looks nothing like the account's history.
LiteLLM's virtual-key design already carries the AI-platform-native version of the softer half of this mechanism: a key can have "soft limits with alerts" in addition to "hard limits (tokens, requests, upstream-billing caps)"9. A soft-limit alert firing is functionally the same event as a Radar spike alert — a threshold crossed on an account's own usage, surfaced for review rather than acted on automatically — and a hard limit is the same idea as an outright Radar block, refusing further usage the moment a ceiling is reached rather than merely flagging it.
The hold mechanism from Section 2 does double duty here, exactly as it does for Stripe's own fraud workflow. Stripe's documentation states plainly that "cardholders can't dispute uncaptured authorisations, only fully captured payments"10, and recommends using "a separate auth and capture process together with Radar reviews" specifically to buy time to manually screen a suspicious transaction before committing to it10. A token-credit hold gives a platform the identical option: a request from a key whose usage pattern has already tripped a soft-limit alert can still be held — reserved, generation allowed to proceed — while the anomaly is reviewed, and only actually captured once the review clears, rather than either blocking a possibly-legitimate burst outright or committing the charge before anyone has looked at it.
One piece of Stripe's fraud toolkit has no clean AI-platform equivalent in anything this module can point to, and the honest statement is that the gap exists rather than inventing a mechanism to fill it: Stripe's dispute/chargeback system gives a customer a formal, card-network-adjudicated process to contest a specific charge after the fact10, with defined evidence requirements and a resolution the merchant does not unilaterally control. No source in this module's map documents an equivalent, network-adjudicated dispute primitive for a token-credit charge — the nearest thing available in practice is a manual credit memo issued against a specific invoice line at the platform's own discretion, which is a weaker, unilateral substitute for a process that, on the card-payments side, exists precisely because no single party is trusted to adjudicate its own fraud disputes.
6. Mid-stream cancellation: charging for what happened, releasing what did not
A cancelled stream reaches this module's ledger from either of two directions, and neither direction is this module's job to build: Module 26 covers the client-initiated case — a caller aborts the connection or sends an explicit stop signal partway through generation — and Module 24 covers the platform-initiated case, where admission control preempts an in-flight generation to shed load under a graceful-degradation cascade. What both directions hand this module is the same fact in the same shape: a hold was placed against a worst case, generation stopped before reaching it, and the ledger has to resolve to a charge for what was actually produced, not to the full hold and not to nothing.
What happens upstream of the platform's own ledger, at the provider the platform is calling, is documented directly. OpenRouter's own streaming-cancellation reference states the mechanism plainly: "streaming requests can be cancelled by aborting the connection. For supported providers, this immediately stops model processing and billing"12 — for providers on that list, which OpenRouter names explicitly as including OpenAI, Azure, Anthropic, Together, DeepInfra, and roughly twenty others12, cancellation is not merely a client-side abandonment of a socket; it propagates upstream and stops the generation (and the corresponding upstream cost) from continuing to run. The same reference states the failure mode for everything not on that list just as plainly: "cancellation only works for streaming requests with supported providers. For non-streaming requests or unsupported providers, the model will continue processing and you will be billed for the complete response"12 — a platform routing to a provider that does not support cancellation cannot rely on the abort signal to actually reduce upstream cost, whatever it does to the client-facing ledger.
That distinction is exactly why the hold, not the cancellation signal itself, is what makes the client-facing ledger's resolution correct regardless of which upstream case applies. If the provider does honor the abort and stops generating, the checkpointed running count from Section 1.2 is final the moment the abort is acknowledged, and settlement proceeds exactly as Section 2.3 describes for a normal completion: capture the actual, release the rest. If the provider does not honor the abort and keeps generating and billing the platform upstream regardless of what the client was told, that is a cost the platform absorbs against its own margin, not a reason to charge the customer for tokens they explicitly cancelled and, by the time the cancellation was acknowledged, had no way to have wanted.
max_tokens = 8,192 is cancelled after 3,200 tokens. The ledger captures the actual cost of those 3,200 tokens and releases the rest of the hold — the same single settlement operation Section 2.3 uses for a normal completion, triggered early. Upstream billing behavior on cancellation is per OpenRouter's documented provider-support distinction12.The one design question cancellation adds that a normal completion does not is timing: how quickly must the platform's own admission-control layer learn that a hold has been (partially) released, so that credit freed by a cancelled request becomes available to the next one without waiting on whatever metering design Section 3 uses. Envoy AI Gateway's own two-phase enforcement — "checks if processing this request would exceed the configured token limit. If the limit would be exceeded, the request is rejected... If within the limit, the request is processed and its token usage is counted towards the total"11 — counts usage only after the fact regardless of whether that request ran to completion or was cancelled early, which means a design built this way inherits exactly the staleness cost Section 3 describes for asynchronous aggregation, applied specifically to the released portion of a cancelled hold. A continuously-tracked balance has no equivalent lag: the settlement in Figure 4 is visible to the very next admission check the instant it is written, which is the same advantage Section 3 establishes for a normal completion, now shown to apply identically to an early one.
Both cancellation triggers assume the signal itself arrives — an aborted connection reaches the gateway, or admission control's preemption decision reaches the process holding the generation. Neither assumption is guaranteed: a client can vanish without a clean abort (a dropped mobile connection, a crashed process that never sends the close frame), and a preemption signal can itself be lost on a sufficiently unhealthy node. Section 2.1's hold expiry is what makes the ledger correct even then. If no cancel signal and no normal completion ever reaches the ledger, the hold's own expiry timestamp (Table 4) is the fallback resolution: once it passes, the platform settles against whatever the last checkpoint recorded, exactly as if a cancel signal had arrived at that moment, and releases the remainder. This is the same role Stripe's own authorization expiry plays for an abandoned card hold — "if the authorisation expires before you capture the funds, the funds are released"4 — applied to a stream instead of a card, and it is precisely why Section 2.1 treats the hold's expiry as mandatory rather than optional: without it, a lost signal in either direction leaves the ledger with no path out of the in-flight state at all.
Where experts disagree
Every design in this module answers the same underlying question — how does a platform protect a balance against a stream whose true cost is unknown at the moment it is admitted — and the sources in this module's map do not agree on the answer, because the three platforms they document made three different bets. The disagreement is not about whether the problem is real; all three treat it as real enough to design around. It is about whether the right response is to reserve against the worst case, meter the actual in real time and skip reservation entirely, or admit optimistically and true up afterward.
Reserve the worst case up front
This module's own adaptation of Stripe's separate-authorization-and-capture mechanism4, proven at Stripe's scale for money rather than tokens
Compute the hold from the one number known at admission — prompt length plus declared max_tokens — and refuse to admit a request whose worst case the balance cannot cover, exactly as a card issuer refuses to authorize a hold larger than available credit. The request either runs inside a reservation that, by construction, cannot be exceeded, or it never starts.
- Never caught short mid-stream: the worst case is covered before generation begins, so no in-flight check can discover a shortfall the admission check should have caught
- Works even if the platform has no way to cut a generation off supraliminally once it starts — the safety property does not depend on real-time enforcement during the stream
- Section 2.2's own arithmetic shows the overhead is not small: a typical completion well under its declared ceiling can leave 70-85% of the reservation locked up and unusable by any other request for the duration of the stream
- Can reject a request outright at admission because of a large declared
max_tokens, even though the model would very likely have produced a much shorter, easily affordable completion
Meter actual usage in real time; ignore the declared worst case entirely
Anthropic's own OTPM rate-limit design8
Anthropic's documentation states the mechanism without qualification: OTPM "is evaluated in real time as output tokens are produced, counting only the actual tokens generated," and explicitly, "the max_tokens parameter does not factor into OTPM rate limit calculations, so there is no rate limit downside to setting a higher max_tokens value"8. Applied to billing rather than rate limiting, the same principle says: never reserve against a number the caller merely declared, only ever count what was in fact produced, checkpoint by checkpoint.
- Zero reservation overhead, ever — no request is rejected at admission for a worst case it was never going to reach, and no credit sits locked up unusable while a generation that will finish well short of its ceiling runs
- Directly compatible with Section 1.2's checkpointed running count, which is already real-time by construction on a provider whose streaming protocol reports cumulative usage per delta2
- Real-time metering of the actual is not, by itself, a real-time cutoff: knowing exactly how much a generation has cost so far does not stop it from continuing to generate past the point a balance goes negative unless a separate enforcement action (killing the generation) is wired to fire on that condition
- A platform with no mechanism to interrupt an in-progress generation gets none of Position A's guarantee that a request cannot end up costing more than the balance available when it started
Admit optimistically against last-known usage; reconcile after the fact
Envoy AI Gateway's two-phase enforcement11
Check the request against the configured limit using whatever usage has already been counted from prior requests, let it run if that check passes, and only count this request's own usage toward the total once it has actually happened: "if the limit would be exceeded, the request is rejected... If within the limit, the request is processed and its token usage is counted towards the total"11.
- Simplest of the three to implement — no per-request reservation bookkeeping, no hold-and-release lifecycle, just a running total updated after the fact
- No overhead at all on the common case where a request's actual cost is small relative to the limit window
- Envoy AI Gateway's own documentation source names the resulting exposure directly: this is "an optimistic-admission design" that "admits a request before knowing its true token cost, then reconciles after the fact"11 — a single abnormally long generation, admitted while the running total still had headroom, can push cumulative usage past the configured limit within that one request, discovered only once it is already over
- The overshoot is bounded only by how large one request's worst case can be relative to the limit window, which is exactly the quantity Position A's hold is designed to bound and this design leaves unbounded
The deciding variable is not which platform is "right" in the abstract, it is how frequently a platform can checkpoint a running generation and whether it can act on a checkpoint by cutting the generation off. Position B (real-time metering, no reservation) dominates whenever both conditions hold: the streaming protocol reports a cumulative, checkpointable total on every delta — which Anthropic's own design provides and OpenAI's final-chunk-only design does not23 — and the platform has wired a hard cutoff to fire the instant a balance would go negative. Under those conditions a hold buys nothing but locked-up credit, because the real-time check is already tighter than any worst-case reservation could be. Position A (reserve the worst case) is the correct conservative default the moment either condition fails: coarse checkpointing, no cutoff mechanism, or a provider whose usage accounting only appears in a final chunk that a cancelled or crashed stream may never deliver at all. Position C (optimistic admission, reconcile after) is the wrong default specifically when a single request's worst case is large relative to the limit or balance it is checked against — exactly the condition Envoy AI Gateway's own documentation flags as the design's known gap — and is defensible only when every individual request's worst case is small enough that no single admission decision can meaningfully overshoot. What would settle this further is public, incident-level evidence rather than architectural description alone: a disclosed case of a specific balance or rate-limit overshoot traceable to one of these three designs, in the same way Module 18's Airbnb replica-lag example is concrete evidence for the idempotency-scope question — no comparably specific public account exists yet for the hold-sizing trade-off this module works through analytically in Section 2.2.
References
- blogStripe (Brandur Leach et al.). Designing robust and predictable APIs with idempotency. Stripe Engineering Blog, 2017. https://stripe.com/blog/idempotency. Accessed 2026-09-18. Used for: recap of the client-generated idempotency-key mechanism and the in-flight third state, reused unchanged from Module 18 for keying a credit-deduction event.
- officialAnthropic. Streaming Messages. Live documentation. https://platform.claude.com/docs/en/build-with-claude/streaming. Accessed 2026-09-18. Used for: message_start/content_block_delta/message_delta/message_stop event flow, and the explicit statement that usage in the message_delta event is cumulative.
- officialOpenAI. Chat Completions streaming events. API reference, live documentation. https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events. Accessed 2026-09-18. Used for: stream_options.include_usage behavior, usage field null on all but the last chunk, and the caveat that an interrupted or cancelled stream may not deliver the final usage chunk.
- officialStripe. Place a hold on a payment method. Live documentation. https://docs.stripe.com/payments/place-a-hold-on-a-payment-method. Accessed 2026-09-18. Used for: separate authorization and capture (manual capture) mechanism, per-payment-method authorization validity windows (cards ~7 days, PayPal 10+10=20 days, Klarna 28 days, Affirm 30 days), automatic release on expiry, and partial capture automatically releasing the remaining amount.
- officialAnthropic. Pricing. Live documentation. https://platform.claude.com/docs/en/about-claude/pricing. Accessed 2026-09-18. Used for: Claude Sonnet 5 official per-token pricing ($2/MTok input, $10/MTok output) underlying the worked hold-sizing derivation in Table 1.
- officialStripe. Usage-based billing. Live documentation. https://docs.stripe.com/billing/usage-based. Accessed 2026-09-18. Used for: the asynchronous meter-event aggregation caveat underlying Module 18's real-time-vs-legacy comparison, restated here for its specific hold-release-latency consequence.
- officialMetronome. Get the net balance of a customer. API reference. https://docs.metronome.com/api-reference/credits-and-commits/get-the-net-balance-of-a-customer. Accessed 2026-09-18. Used for: the invoice_inclusion_mode parameter (FINALIZED vs. FINALIZED_AND_DRAFT) controlling whether pending draft-invoice deductions are included in a real-time balance calculation.
- officialAnthropic. Rate limits (Claude API). Live documentation. https://platform.claude.com/docs/en/api/rate-limits. Accessed 2026-09-18. Used for: spend limits vs. rate limits as distinct mechanisms; org-wide spend cap vs. per-workspace allocation below it; OTPM evaluated in real time on actual output tokens with max_tokens excluded from the calculation.
- officialBerriAI / LiteLLM contributors. Life of a Request (LiteLLM Proxy architecture). Live documentation. https://docs.litellm.ai/docs/proxy/architecture. Accessed 2026-09-18. Used for: asynchronous, off-critical-path spend logging and rate-limit accounting; virtual-key hard limits and soft limits with alerts; spend rollup from key to user_id and team_id.
- officialStripe. Best practices for preventing fraud. Live documentation. https://docs.stripe.com/disputes/prevention/best-practices. Accessed 2026-09-18. Used for: Radar fraud-score model, spike/sustained_attack health-alert categories, auth-and-capture as a fraud-review buffer, and the statement that cardholders cannot dispute uncaptured authorizations.
- officialEnvoy AI Gateway project. Usage-based Rate Limiting. v0.1 documentation. https://aigateway.envoyproxy.io/docs/0.1/capabilities/usage-based-ratelimiting/. Accessed 2026-09-18. Used for: the two-phase optimistic-admission-then-reconcile enforcement design, contrasted with hold-based and real-time-metering designs in the disagreement section.
- officialOpenRouter. API Streaming (stream cancellation reference). Live documentation. https://openrouter.ai/docs/api-reference/streaming. Accessed 2026-09-18. Used for: stream cancellation stopping model processing and billing on supported providers, the provider support list, and the continued-billing behavior for non-streaming or unsupported-provider requests.