System Design · Module 26
Streaming, Cancellation, and Resumption: SSE, WebSockets, and Long-Running Requests
A chat completion is not a request/response pair; it is a long-lived connection that emits a typed sequence of events while a GPU somewhere is still generating tokens. This module builds that event protocol from the wire format up, shows exactly where it breaks a naive moderation gate, and derives the sequence-number cursor that lets a client resume the same response after its connection drops instead of paying to regenerate it.
response.created, response.in_progress, a run of response.output_text.delta events (one per emitted chunk), then response.completed — or, if generation fails partway through, a response.failed event instead of further deltas1. Section 2 adds the per-item sub-events this diagram omits for space.1. The SSE wire format, and why LLM APIs build their own protocol on top of it
A normal HTTP exchange is one request, one response, connection closed (or returned to a pool). Nothing about that model fits a chat completion: the client wants the first tokens as soon as they exist, not after the GPU backend behind Module 25's interface has finished a generation that might run for tens of seconds. The mechanism both major providers use to fix this is not bespoke — it is server-sent events (SSE), a living-standard part of the HTML specification, reused as a generic solution rather than invented per API7.
The wire format is deliberately small. An SSE response sets its content type to text/event-stream, and the body is a sequence of plain-text fields: event names the event type, data appends a line to the current event's payload buffer, id sets a last-event identifier the client will echo back on reconnect, and retry tells the client how many milliseconds to wait before reconnecting7. A blank line terminates one event and dispatches it. Nothing here is LLM-specific — it is the same four fields a stock ticker or a notification feed would use.
The browser-native way to consume this is the EventSource interface, and it has one property that matters more than any other for this module: EventSource issues a plain GET request with no body, and on a dropped connection it reconnects automatically, replaying the last-seen id value in a Last-Event-ID request header so a cooperating server can resume the feed exactly where it left off7. That reconnection machinery is free, built into every browser, and exactly what you would want for a chat completion that drops mid-stream — except that a chat completion is not a GET. Both OpenAI's and Anthropic's streaming endpoints take their prompt, tools, and generation parameters as a JSON request body on a POST, which EventSource cannot issue at all12. Every serious client SDK for these APIs therefore does not use EventSource; it issues the POST itself over fetch or an equivalent streaming HTTP client and parses the same event:/data: wire format by hand. This is also why neither provider's resumption story (Section 5) is the free Last-Event-ID mechanism the standard already provides — that mechanism is wired to a transport (GET, no body) that a completion request cannot use, so both providers had to design their own cursor on top of the same wire format instead of underneath it.
1.1 Why WebSockets exist too, and why token streaming doesn't need them
The module title mentions WebSockets for a reason: they are the other transport a frontier lab actually ships, just not for this problem. OpenAI's Realtime API — built for speech-to-speech voice agents — connects over WebRTC in a browser or a WebSocket on a server, explicitly because a voice agent needs full-duplex behavior a one-way event stream cannot express: the documentation frames the requirement as supporting "barge-in," "low first-audio latency," and "natural turn taking," meaning the user can start talking while the model is still speaking and the transport has to carry both directions concurrently6. SSE cannot do this by construction — the EventSource interface offers no method to send data back to the server over the same connection at all7, and even the hand-rolled SSE parsing OpenAI's and Anthropic's text SDKs use is still riding a response body that only flows server-to-client; any new input (a follow-up message, a stop instruction) has to go over a separate request.
Text completion streaming does not need that second direction. The client's only real-time input during generation is "keep going" or "stop" (Section 4), and both fit comfortably as a side channel — an HTTP cancel call, or simply closing the connection — layered on top of a strictly one-way event feed. A voice turn, by contrast, needs the server to hear new audio while it is still speaking, which is a genuinely bidirectional requirement SSE was never designed to carry. The choice of transport per API is therefore not an accident of implementation history; it tracks a real difference in what has to travel in which direction, and a platform gateway (Module 21) that fronts both a text-completion endpoint and a voice endpoint is fronting two different transport contracts, not one contract used two ways.
2. The typed event lifecycle of a streamed response
Figure 1 showed the skeleton. The full event set is larger, because a response is not just text: it can contain function calls, code-interpreter invocations, file search, and — for Anthropic — extended-thinking blocks, and each of those needs its own sub-events nested inside the same stream. OpenAI's Responses API documents its event stream as a union of typed events including ResponseCreatedEvent, ResponseInProgressEvent, ResponseFailedEvent, ResponseCompletedEvent, ResponseOutputItemAdded/Done (an output item — a message, a tool call — starting or finishing), ResponseContentPartAdded/Done, ResponseOutputTextDelta (the per-chunk text you render as it arrives), ResponseOutputTextAnnotationAdded, ResponseRefusalDelta/Done, ResponseFunctionCallArgumentsDelta/Done (a tool call's arguments streamed as partial JSON), a triplet of file-search events (InProgress/Searching/Completed), and a five-event code-interpreter group (InProgress, CallCodeDelta, CallCodeDone, Interpreting, CallCompleted)1.
Anthropic's Messages API organizes the same kind of information differently: rather than one flat union, it nests events inside content blocks. Every stream opens with message_start (a Message object with empty content), then for each content block in turn a content_block_start, one or more content_block_delta events, and a content_block_stop — each block carries an index matching its position in the final message's content array — followed by one or more message_delta events carrying top-level changes (including the stop reason), and finally message_stop2. The delta events themselves are typed by content-block kind: a text_delta for ordinary text, an input_json_delta carrying a partial-JSON fragment of a tool call's arguments (accumulated client-side and parsed once complete, exactly like OpenAI's function-call-arguments delta), a thinking_delta for extended-thinking content, and a signature_delta that arrives just before a thinking block's content_block_stop to let the client verify that block's integrity2. Both providers also emit two event kinds that are not part of the content lifecycle at all: a keepalive (Anthropic sends explicit ping events dispersed through the stream2) and an error event that can arrive mid-stream — Anthropic's example is an overloaded_error, which "would normally correspond to an HTTP 529 in a non-streaming context," delivered as event: error instead of a response-level HTTP status because the HTTP status line was already committed when the stream opened2.
| Lifecycle stage | OpenAI Responses API | Anthropic Messages API |
|---|---|---|
| Stream opens | response.created | message_start |
| Generation running | response.in_progress | (implicit — no top-level equivalent) |
| Item/block starts | response.output_item.added, response.content_part.added | content_block_start |
| Text chunk | response.output_text.delta | content_block_delta (type: text_delta) |
| Tool-call arguments chunk | response.function_call_arguments.delta | content_block_delta (type: input_json_delta) |
| Reasoning chunk | not exposed as a delta type in this source | content_block_delta (type: thinking_delta, plus signature_delta) |
| Item/block ends | response.output_item.done, response.content_part.done | content_block_stop |
| Top-level running totals | carried inside completion event | message_delta (one or more; cumulative usage) |
| Keepalive | not named in this source | ping |
| Mid-stream error | error | error (e.g. overloaded_error) |
| Stream ends, success | response.completed | message_stop |
| Stream ends, failure | response.failed | surfaced via the error event, not a distinct terminal type |
The two designs in Table 1 encode the same lifecycle with different granularity, and the difference is not cosmetic. OpenAI's flat union means a client dispatches on one event-type string and, for anything positional, reads an explicit item/part identifier out of the payload. Anthropic's nested design means a client must track which index is currently open before it can interpret a content_block_delta correctly — the delta itself does not repeat the block's type, only its index, so a client that lost track of a content_block_start cannot tell a text_delta from an input_json_delta without cross-referencing the index it opened against. Both designs give a client watching only for terminal events — response.completed/response.failed, or message_stop/an error event — a reliable way to know the stream is done without parsing every intermediate delta, which matters for anything downstream that only cares about the final artifact (the ledger in Module 23, for instance, which reads the cumulative usage carried in the terminal events rather than re-summing every delta itself).
The granularity of a tool-call's argument deltas is itself a tunable, not a fixed property of the protocol. Anthropic's default behavior buffers a tool's input internally and only emits a key/value pair once it is complete — the documentation notes that "current models only support emitting one complete key and value property from input at a time," so a client waiting on fine-grained partial JSON will see bursts separated by pauses while the model finishes the next field2. A per-tool opt-in, eager_input_streaming, changes that trade: it streams a tool's parameter values as they are generated rather than waiting for a field to close, at the cost of the client having to handle a stream of genuinely partial JSON fragments instead of whole key/value pairs2. This is the same tension Section 5 will make explicit for whole-response resumption — granularity and latency trade against buffering and complexity — appearing here one level down, inside a single content block's own delta stream.
3. Why streaming structurally weakens moderation
A non-streaming completion gives a moderation layer one clean opportunity: the model finishes, the full text exists, a classifier scores it, and the caller decides whether to show it. Streaming removes that opportunity by construction, and OpenAI's own documentation states the consequence plainly rather than leaving it implicit: "streaming the model's output in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate"1. The reason is not that streaming makes the classifier worse at its job on any given span of text — it is that streaming changes when the classifier's input becomes available relative to when the output the classifier is supposed to gate has already reached the user.
The same source is explicit about the timing, not just the difficulty: "if you request moderation scores with a generation request, the scores arrive after the full generated output is available. They aren't included with partial output deltas"1. Read that against Figure 1's event sequence: the moderation score is not a field on response.output_text.delta, and it is not available at response.completed either in the sense of being ready in time to act on the content that already streamed — it is a property of the finished output, computed after the fact, delivered on a timeline that has no relationship to the delta timeline a client has already rendered.
Three consequences follow directly from Figure 2, and none of them require speculation beyond what the source states. First, a moderation score cannot be a streaming gate in the literal sense of blocking delta k from reaching the client based on a score computed over deltas 1..k — the score is defined over the full output, not a prefix, so it does not exist yet at delta k for any k < N. Second, whatever a moderation score is used for once it does arrive, it is necessarily a decision about the next action (log it, flag the account, refuse to serve a follow-up turn, alert a reviewer) rather than a decision about the content the user already saw, because that content already left the server. Third, and this is the point where scope hands off cleanly to Module 29: a design that wants a real gate before anything reaches the user has to run a separate, faster classifier on a much smaller window than "the full output" — either per-delta or per some coarser interval — accepting a weaker signal in exchange for actually gating in time, and that trade (a fast partial-window classifier as policy signal, not a full-output score as a block) is exactly the four-stage safety pipeline Module 29 covers as its own subject rather than a side effect of this module's transport.
None of this is specific to a bug in either provider's implementation. It is a direct consequence of the ordering constraint streaming imposes: a classifier that needs the whole output as input cannot finish before the whole output exists, and by the time the whole output exists in a streaming response, the client already has all of it. A non-streaming response does not have this problem only because it withholds the entire output from the client until generation finishes anyway — moderation timing is free there because the client was already going to wait for completion.
It is worth distinguishing this from a superficially similar pattern this course has already covered: Module 21's async, off-critical-path spend logging also computes something after a request's events have already gone out, and it is tempting to read the moderation score the same way — "just another thing that happens after the fact and gets reconciled later." The two are not the same kind of lateness. Spend logging arrives late by design choice, to keep an accounting write off the hot path of a request that could otherwise return its result synchronously with the correct number already known; nothing prevents computing the spend figure immediately except a latency-versus-freshness trade the platform chose to make. A moderation score on a streamed response arrives late because its input — the complete output — does not exist until the stream itself has finished emitting it; there is no faster code path that would let it arrive sooner, because the quantity it depends on had not been produced yet. One lateness is an engineering choice; the other is a consequence of what streaming is.
4. Cancellation: what a client can and cannot assume about work already done server-side
A client closing its side of the connection is not the same event as the server stopping work, and the gap between those two things is exactly what a client cannot assume without checking. For a background response specifically, OpenAI exposes an explicit cancellation call rather than relying on the client simply dropping the socket: POST /v1/responses/{response_id}/cancel3. Calling it twice is documented as safe — "cancelling twice is idempotent — subsequent calls simply return the final Response object"3 — which matters operationally: a client that is unsure whether its first cancel call actually landed (a common state after a flaky network) can retry the call without a second, distinct side effect, the same idempotency shape Module 8 and Module 18 already established for retried mutations generally.
What idempotent cancellation does not tell you is whether the upstream generation actually stopped, and this is where provider behavior forks in a way a platform gateway has to design around rather than assume away. OpenRouter's own streaming-cancellation reference states the mechanism and its limit in the same paragraph: "streaming requests can be cancelled by aborting the connection. For supported providers, this immediately stops model processing and billing"4 — and it names the providers on that list explicitly, including OpenAI, Azure, Anthropic, Together, and DeepInfra among roughly twenty others4. The other branch is stated just as plainly: "cancellation only works for streaming requests with supported providers. For non-streaming requests or unsupported providers, the model will continue processing and you will be billed for the complete response"4.
The reason this fork exists at all is not carelessness on any provider's part; it is a consequence of where the compute actually happens. Cancelling a stream is a message that has to travel from the client, through however many hops sit between it and the process holding the KV cache and running the decode loop (the gateway of Module 21, possibly a router making a fallback decision per Module 24, then the backend across Module 25's interface), before that process can act on it. A platform that fronts multiple upstream providers or backends through one API — the way a gateway aggregates several models behind one interface — inherits whichever behavior its slowest-to-honor-cancellation upstream has, unless it tracks per-provider support explicitly and designs its own client-facing contract around the weakest link rather than assuming every backend behaves like the best one. What a client can assume, regardless of provider, is exactly what Figure 3 shows and nothing more: the connection closes, a second cancel call is safe to send, and whether the GPU actually stopped multiplying matrices is a fact about the specific provider being called, not about the cancel request itself. What a client cannot assume is that cancelling stops cost, stops the ledger hold from Module 23 growing, or recovers any of the tokens already generated — those are the exact questions Module 23's mid-stream-cancellation treatment resolves, building directly on the provider-support fork this section establishes.
4.1 Why a gateway has to track cancellation support as explicit per-backend state
The consequence for a gateway design (Module 21) is concrete rather than abstract. A gateway that fronts several upstream models or providers behind one client-facing API is, by construction, aggregating backends that sit on different sides of the fork in Figure 3 — some honor an abort immediately, some do not4. If the gateway exposes one uniform cancel endpoint to its own clients without tracking which upstream it is calling for a given request, it has two bad options: promise a client-facing guarantee ("cancelling always stops generation") that it cannot actually keep for every backend, or promise nothing and let every client independently discover, by observing continued billing after a cancel, which backends are trustworthy. Neither is a real contract.
The alternative is to make provider-level cancellation support a first-class piece of routing metadata, sitting alongside the KV-cache-utilization and queue-depth signals Module 25's interface already exposes for routing decisions: a boolean (or a per-backend policy) the gateway consults both when choosing a backend for a cancellable-by-design workload and when reporting cancellation status back to the client. A gateway that knows a given backend does not honor aborts can, at minimum, tell the client the truth immediately — "cancel request accepted, but this backend will run to completion and you will be billed for it" — rather than a client discovering that fact experimentally after the response finishes. This is the same discipline Module 24 already applies to fallback design: a fallback is only safe if the platform knows, in advance, what guarantee it is and is not making, not because it assumed every backend behaves identically.
5. Resumption: background mode and the sequence-number cursor
Cancellation (Section 4) is a client choosing to stop. Resumption is the opposite problem: the client did not choose to disconnect — a mobile network dropped, a laptop slept, a load balancer idle-timed the connection — and it still wants the response it was already receiving, not a new one. OpenAI's answer is to make this possible only for a response deliberately opted into it: create the response with both background: true and stream: true, and the server will let the client reconnect to the same in-progress or completed response later3.
The mechanism is a cursor, not a replay-everything restart. Every streamed event — recall Figure 1's response.created, response.in_progress, each delta, and the terminal event — carries a sequence_number field, and the documentation's own instruction to implementers is direct: "you will want to keep track of a 'cursor' corresponding to the sequence_number you receive in each streaming event"3. When the connection drops, the client reconnects with GET /v1/responses/{response_id}?stream=true&starting_after={sequence_number}, and the server resumes emitting events strictly after the one the client last saw — it does not re-send anything the client already has3.
sequence_number it processed; the server's starting_after parameter resumes exactly after that point3. This is a cursor over one response's own event log, not a general-purpose SSE reconnect — it exists only because OpenAI built it, not because SSE itself provides it (Section 1).background and stream are two independent flags, and it is worth separating what each buys before treating Figure 4 as "the" background-mode story. The documentation frames the ~10-minute disk-retention window as existing "to enable asynchronous execution and polling"3 — that is, a client can create a background: true response without ever setting stream: true, hold no open connection at all, and simply poll GET /v1/responses/{response_id} periodically until its status changes to complete. That path needs no sequence_number cursor and no reconnect logic, because there was never a live connection to lose in the first place — polling degrades gracefully to whatever interval the client chooses, including one poll after a long wait. Setting stream: true on top of background: true is what adds Figure 4's live event feed and, with it, the specific problem of what to do when that feed's connection drops. Resumption via starting_after is therefore not background mode's only purpose; it is the answer to a problem that only exists because streaming was also requested — a client that only ever intended to poll was never exposed to it.
5.1 The storage-window trade-off and resumption's precondition
Two constraints bound how useful Figure 4's protocol actually is. First, the precondition is absolute: "resumption is only possible if the background response was originally created with stream=true"3 — a client that did not opt in at request time cannot retroactively decide, mid-disconnect, that it wants this option; the decision has to be made at admission, which is precisely the kind of request-class decision Module 24's routing layer would need to make on the client's behalf if it wants to offer resumability as a platform feature rather than leaving it to whichever caller happened to set the flag. Second, the option has a clock on it: response data for a background request "is temporarily stored to disk for roughly 10 minutes to enable asynchronous execution and polling"3; after that window, retention depends on the store parameter and the platform's own abuse-monitoring settings, not on the resumption feature itself. A client that reconnects at minute 12 of an outage has no cursor to resume from — the mechanism did not fail, it expired.
Worked example: what a cursor-based resume replays versus what a naive full retry would redo
The numbers here are illustrative — a made-up scenario chosen to make the arithmetic concrete, not a benchmark reported by either provider. Suppose a response is generating 2,000 output tokens, streamed roughly one response.output_text.delta event per token, and the connection drops at sequence_number = 140 (roughly 7% of the way through, after accounting for the handful of non-delta events — created, in_progress — that also consume sequence numbers ahead of the first delta).
With the cursor-based protocol of Figure 4, reconnecting with starting_after=140 costs the client exactly one extra HTTP round trip, and the server resumes at event 141 — it has kept generating the remaining roughly 1,860 tokens the entire time the client was disconnected (the server-side generation is not paused by a client-side network drop; only the client's view of it was interrupted), so no compute is redone and no token is billed twice.
Without any resumption cursor — the situation for any request that was not created with background and stream both set to true, or for a provider with no equivalent feature at all — the client's only option on reconnect is to issue a brand-new request. That new request pays for generating all 2,000 output tokens again from the same prompt, discarding the 140 delta events' worth of output it already received and rendered; the first 140 events were not wasted from the user's perspective (they were already read), but they are entirely wasted from the compute-and-latency perspective, since the new request has no way to pick up after token 140 — it starts the decode loop over from the prompt with no memory of how far the previous attempt got. The cursor's entire value is collapsing that second case to the first one: the cost of a resume becomes one HTTP round trip instead of one full re-generation, precisely because the server, not the client, is the party that retained progress.
The size of that win moves with where in the response the disconnect happens, and it is worth walking both ends of the range on the same 2,000-token response. A disconnect at sequence_number = 5 — essentially at the start, before more than a handful of deltas have arrived — makes the cursor's advantage small in absolute terms: a naive full retry would only be redoing a few tokens' worth of generation, so the round-trip saved is real but the compute saved is negligible. A disconnect at sequence_number = 1,900 — 95% of the way through a long response — is the opposite case: a naive full retry throws away nearly the entire generation and starts over from zero, while the cursor resumes with perhaps 100 tokens left to stream, at essentially the same one-round-trip cost as the early-disconnect case. The cursor's fixed cost (one reconnect, one starting_after lookup) does not scale with how far into the response the drop occurred, but the naive alternative's cost scales directly with it — which is exactly why Section 5.1 tied resumability's value to how long a generation tends to run: the longer the response and the later a disconnect tends to land relative to its length, the more a fixed-cost cursor is worth having over a cost that grows with exactly the quantity you cannot control (when the network happens to fail).
This is a mechanical claim about wasted generation, deliberately stated without a dollar figure — what a retried-from-scratch generation actually costs the caller, and how a mid-stream disconnect nets against an existing hold, is Module 23's ledger-reconciliation problem, not this module's. What this section establishes is the precondition and the shape of the win: resumability is an opt-in feature with a short, fixed expiry, and it is valuable in direct proportion to how far into a long generation a disconnect tends to happen — which is also why it matters most for exactly the kind of long-running, tool-using agent session Module 27 builds on top of these primitives, where a single turn's generation can run long enough that a mid-turn network blip is a real operational cost if it forces a full restart.
6. A structurally different streaming stack: Twitch's live video versus SSE token streaming
Module 16 covers Twitch's ingest routing (Intelligest, IRS, Capacitor, The Well) and chat architecture in depth and is not repeated here. What is useful to pull forward is narrower: Twitch's live-video pipeline is also, in a loose sense, "streaming," and putting the two side by side sharpens exactly what is specific to token streaming rather than generic to any low-latency delivery problem.
Twitch ingests live video over RTMP or WebRTC at nearly a hundred points of presence, terminating each stream at a per-PoP media proxy (Intelligest) that queries a stateful routing service (IRS) to place the stream onto Twitch's backbone, after translating the ingest protocol into an internal canonical form for that transport5. The entire design in Module 16's account is aimed at one problem SSE token streaming does not have at all: routing a continuous, many-hours-long, high-bitrate media flow through a global network topology so it reaches every viewer with acceptable latency, where "acceptable" is measured by Twitch itself as glass-to-glass delay from broadcaster to viewer, not as a single request's response time.
| Axis | Twitch live video | SSE token streaming |
|---|---|---|
| Ingest transport | RTMP or WebRTC5 | HTTP POST, text/event-stream response body7 |
| Producers : consumers | one broadcaster : many viewers | one generation : one client |
| Routing concern | global backbone placement, PoP selection5 | none beyond the gateway's own request routing (Module 21, 24) |
| Primary latency metric | glass-to-glass delay, broadcaster to viewer | time-to-first-delta, then inter-delta gap |
| State on disconnect | none retained per viewer — rejoin at live edge | up to ~10 min of server-retained state, if opted in3 |
| Resumption unit | not applicable — there is no "the same missed segment" to fetch | sequence_number cursor over one response's own event log3 |
| What a reconnect costs | zero — client just resumes watching live | one round trip (cursor) or a full re-generation (no cursor) |
The contrast in Table 2's last two rows is the one worth carrying forward. Twitch's viewers do not need resumption in this module's sense because there is nothing to resume to — the broadcast is a shared, ongoing present, and a viewer who reconnects simply rejoins wherever the broadcast currently is, the same way changing a television channel back does not replay what aired while you were on a different channel. An LLM completion has no equivalent "live edge" to rejoin, because it is not a shared broadcast; it is one client's own private generation, which is precisely why it is a coherent idea to let that one client fetch exactly the part of it they missed. The two systems are not different tiers of the same problem — one is not a "harder" or "easier" version of the other — they are solving genuinely different problems that both happen to use the word streaming, and a platform gateway that fronts both a text-completion API and a live-media path (as some vendors increasingly do for multimodal products) cannot treat them as the same subsystem with different config.
Where experts disagree
Neither provider frames stream recovery as a debate — each simply documents the mechanism it built — but the two mechanisms embody genuinely different philosophies about who is responsible for surviving a broken connection, and a third baseline exists by default wherever neither investment has been made. Laid side by side, these are three real, currently-shipping answers to the same question: when an SSE connection drops mid-response, what happens to the tokens the server already generated?
Server-side resumption cursor: the server remembers, the client just asks for the rest
OpenAI's Responses API, background mode3
Opt a response into background: true and stream: true at creation time, and the server takes on the obligation of retaining the response's event log (on disk, for roughly ten minutes) so any later reconnect can resume from an exact sequence_number cursor via starting_after3. Generation itself is not paused by the client's disconnect — the server keeps producing tokens the whole time — so a successful resume recovers the full response at the cost of one extra round trip, with zero regenerated tokens.
- No wasted GPU compute on a resume: the exact same generation is recovered, not a new one started.
- Works uniformly regardless of what the output contains — text, tool calls, or anything else the event log records.
- The client's only required state is a single integer (the last
sequence_numberseen).
- Only available if the client opted in at request creation; cannot be retrofitted onto a request already in flight without that flag.
- The recovery window is short and fixed (~10 minutes) and is an operational property of the provider's storage policy, not something the client controls.
- Requires the provider to build and secure genuine server-side state — a resumable-response store — which is nontrivial infrastructure to have opted the whole platform into for a feature only some callers use.
Client-side capture-and-continue: the client remembers, and pays for a fresh generation to finish the job
Anthropic's documented error-recovery strategy for the Messages API2
Anthropic's own guidance for recovering "a streaming request that was interrupted because of network issues, timeouts, or other errors" is explicit and has no cursor at all: capture whatever content the client already received, then construct an entirely new request that either places that partial content as the start of a new assistant message (Claude 4.5 and earlier) or, for newer models, wraps it in a fresh user message instructing the model to "continue from where it left off," and resume streaming from there2. The same guidance states a real limitation of this approach directly: "tool use and extended thinking blocks cannot be partially recovered. You can resume streaming from the most recent text block"2 — a disconnect during a tool call or a reasoning block loses that block outright, recoverable only as prose the model is asked to reconstruct, not as the original structured content.
- Needs no special provider-side feature, no opt-in flag, and no server-side storage commitment beyond whatever the provider already does — it works on any streaming request.
- Simple mental model for the client: treat recovery as "make a new, better-informed request," which is the same shape of logic a client already needs for ordinary retries.
- Re-runs a real generation step — the continuation instruction consumes new input-token processing and produces new output, on top of whatever billing the interrupted attempt already incurred (Module 23's problem to reconcile, not this module's, but a real cost this design imposes that the cursor design does not).
- Structured content that was mid-stream when the disconnect happened — a tool call's arguments, a thinking block — is lost, not resumed; the model has to redo that reasoning from a prose instruction.
- Puts the correctness burden on the client: it must correctly capture, store, and re-submit the partial content, rather than handing the server a single integer.
No resumption at all: treat a broken stream as terminal, by default
The baseline absent either investment, and the SSE standard's own default behavior for a server that keeps no per-client history7
The EventSource interface's automatic reconnection and Last-Event-ID replay are only useful if the server on the other end is willing to interpret that header and replay history from it7 — nothing in the standard obligates a server to keep any event history at all, and a server that keeps none simply starts a fresh subscription on reconnect with no way to deliver what was missed. Most streaming integrations that have not deliberately built one of the two mechanisms above land here by default: a dropped connection ends the response as far as the client is concerned, and recovering the rest means the caller decides, as an application-level choice, whether to show a truncated answer or start over — structurally the same position a Twitch viewer is in on reconnect (Section 6), rejoining wherever things currently stand rather than recovering anything from the gap, except that an LLM response has no "current live position" to rejoin at all, so recovering nothing really does mean recovering nothing.
- Zero engineering investment: this is what happens if you build nothing, and for a low-stakes, short-generation use case that may be entirely adequate.
- No server-side storage obligation and no client-side capture-and-resubmit logic to get wrong.
- Wastes everything already generated on any disconnect during a long response — full stop, no partial credit of any kind.
- Pushes the entire recovery decision onto the end user ("try again?"), which is a worse experience than either alternative for exactly the long-running, high-value generations where a disconnect is most costly.
These are not equally good answers to the same-cost problem; they are answers calibrated to different generation lengths and different amounts of engineering investment a platform is willing to make. For a short completion — a few hundred tokens, sub-second to a few seconds of generation — the baseline (Position C) is close to strictly fine: the cost of restarting is small, and building either recovery mechanism is very likely more engineering effort than the failure mode it prevents ever burns in wasted compute. For a long, expensive generation with no tool calls or reasoning blocks mid-stream, the client-side capture-and-continue approach (Position B) is a reasonable middle ground precisely because Anthropic's own stated limitation — losing tool-use and thinking blocks — does not bite when there aren't any; the client pays for a second, shorter generation (a continuation, not a full redo) rather than losing everything. For a long-running, tool-heavy, or agentic generation — exactly Module 27's territory, where a session can run for minutes and structured tool-call state is common mid-stream — the server-side cursor (Position A) is the only one of the three that recovers that structured state intact rather than asking the model to reconstruct it from a prose instruction, and it is the only one where the recovery cost is a round trip rather than a partial re-generation. The evidence that would settle which a given platform should build by default is a distribution, not a single number: measure what fraction of a workload's generations are long enough and disconnect-prone enough (mobile clients, unreliable networks, multi-minute agent turns) that Position C's full-restart cost, multiplied by that disconnect rate, exceeds the fixed engineering and storage cost of building Position A. Below that threshold, building nothing is the right call; above it, the sequence-number cursor is worth having, and a platform serving both short chat completions and long agent sessions through one gateway will reasonably want both — cheap default behavior for the former, opt-in resumability for the latter — rather than one policy for every request class.
References
- officialOpenAI. Streaming API responses (Responses API guide). Live documentation. https://developers.openai.com/api/docs/guides/streaming-responses. Accessed 2026-09-18. Used for: the full typed SSE event union (created/in_progress/failed/completed, output-item, content-part, text-delta, refusal, function-call-arguments, file-search, code-interpreter sub-events), and the exact statement that moderation scores arrive only after the full output and are not included with partial deltas.
- officialAnthropic. Streaming Messages. Live documentation. https://platform.claude.com/docs/en/build-with-claude/streaming. Accessed 2026-09-18. Used for: the message_start/content_block_start/content_block_delta/content_block_stop/message_delta/message_stop event flow, text_delta/input_json_delta/thinking_delta/signature_delta content-block-delta types, ping and error events, and the documented client-side capture-and-continue error-recovery strategy including the tool-use/thinking-block recovery limitation.
- officialOpenAI. Background mode (OpenAI API). Live documentation. https://developers.openai.com/api/docs/guides/background. Accessed 2026-09-18. Used for: the background+stream precondition for resumability, the sequence_number cursor and starting_after resumption parameter, the ~10-minute disk storage window, and the POST .../cancel endpoint's idempotent-on-repeat behavior.
- officialOpenRouter. API Streaming (stream cancellation reference). Live documentation. https://openrouter.ai/docs/api-reference/streaming. Accessed 2026-09-18. Used for: the provider-support fork in cancellation behavior — immediate stop of processing and billing on supported providers versus continued processing and full billing on unsupported providers or non-streaming requests.
- blogTwitch. Ingesting Live Video Streams at Global Scale. Twitch Blog, published 2022-04-26. https://blog.twitch.tv/en/2022/04/26/ingesting-live-video-streams-at-global-scale/. Accessed 2026-09-17. Used for: RTMP/WebRTC ingest, the Intelligest/IRS routing architecture, and the nearly-100-PoP scale figure underlying the Twitch-versus-SSE contrast in Section 6.
- officialOpenAI. Realtime API. Live documentation. https://developers.openai.com/api/docs/guides/realtime. Accessed 2026-09-18. Used for: the WebSocket/WebRTC bidirectional transport choice for speech-to-speech voice agents, and the barge-in/turn-taking requirement motivating that choice over a one-way event stream.
- standardWHATWG. HTML Living Standard — §9.2 Server-sent events. Living standard. https://html.spec.whatwg.org/multipage/server-sent-events.html. Accessed 2026-09-18. Used for: the text/event-stream MIME type, the event/data/id/retry wire-format fields, the EventSource interface's GET-only, no-body request and its native Last-Event-ID reconnection mechanism, and SSE's fundamental unidirectionality.