System Design · Module 30

Data and Observability: Conversation Storage, Retention, Privacy, and Per-Token Tracing

Every request into an LLM platform leaves two trails: a stored conversation that someone eventually has to govern, and a live execution that someone eventually has to explain. This module covers both — Anthropic's Clio pipeline for extracting aggregate insight from stored conversations without a human reading any of them, the retention and consent boundaries that decide what is even eligible to look at, and the per-token metrics and cross-system trace that make a single streamed request's latency and cost legible across the gateway, the router, and the GPU backend that served it.

Course 30 min read 14 sources As of 2026-09-18 Prerequisite: Modules 11, 23
Two planes sharing one request: the data plane governs what happened, the observability plane explains what is happening Data plane — governs what already happened Conversation stored at rest Retention clock + consent gate Clio: aggregate privacy-preserving analysis Human analyst sees clusters only Section 1 (retention/consent) and Section 2 (Clio) — data already at rest Observability plane — explains what is happening right now Client request Gateway span 1 Router / scheduler — span 2 GPU backend span 3: TTFT, ITL one trace ID, one bill, cost attributed per span Section 3 (per-token metrics) and Section 4 (cross-system trace + cost) — one live request
Figure 1. This module covers two structurally different problems that both fall under "data and observability" for an LLM platform: governing conversation data that is already stored (top row — what may be retained, for how long, under what consent, and how Anthropic's Clio extracts aggregate insight from it without a human reading a raw conversation1), and explaining a single live request's latency and cost as it crosses the gateway, router, and GPU backend (bottom row, extending Module 11's metrics/logs/traces triad across a boundary it did not have to cross).

Before anything can be analyzed in aggregate or replayed for debugging, it has to have been stored in the first place — and "stored" is not a single state. A conversation sits behind a boundary defined by three independent variables: how long it is kept, whether it is eligible to influence a future model's training, and what exception (a legal hold, a trust-and-safety flag) can override whatever policy would otherwise apply. Getting this boundary wrong in either direction is expensive: keep everything indefinitely and every future privacy incident includes years of exposure; delete too aggressively and you lose the raw material Section 2's aggregate-analysis pipeline needs to run at all.

1.1 Two products, two defaults

Anthropic runs two different default policies depending on which product produced the conversation, and the difference is instructive because it shows retention and consent are a per-surface design decision, not a single company-wide constant. On the consumer side (Claude Free, Pro, and Max), a conversation a user deletes is "deleted from our back-end storage systems within 30 days"2. Separately, a user can opt in to letting Anthropic use chats or coding sessions to improve Claude; if they do, "we may retain your data in a de-identified format for up to 5 years in our model training pipelines"2 — a retention window two orders of magnitude longer than the deletion default, gated behind an explicit, reversible toggle rather than applied automatically. A user who leaves that setting off gets the short retention window; a user who turns it on trades retention length for the possibility their (de-identified) conversations shape a future model.

On the commercial API side, the default runs the opposite direction. Anthropic's own commitment is that "retained data is never used for model training without your express permission," and — the more consequential clause for a systems designer — "conversation content (your prompts and Claude's outputs) is not retained by default"3. Where the consumer product's default is a short, bounded retention window with an opt-in path to a long one, the API's default is closer to no retention at all for message content, with an explicit zero data retention (ZDR) arrangement available on request that goes further still: "Anthropic does not store customer prompts or responses at rest after the API response is returned"3. The one documented exception that pulls in the other direction is Anthropic's "Covered Models" designation — certain models require 30-day retention as a condition of access, and ZDR is not available for them "unless expressly authorized by Anthropic"3, which means a platform routing a request to one of those models has just accepted a retention obligation it would not have accepted for every other model in its fleet purely by choosing which model answered the request.

1.2 The exceptions that survive any policy

Every policy above has the same escape hatch, and it is worth stating plainly because it is easy to design around the default case and forget the exception exists. Under ZDR or HIPAA readiness, Anthropic "may retain data where required by law or where it has been flagged by Anthropic's automated trust and safety systems," and specifically "if a chat or session is flagged, Anthropic may retain inputs and outputs for up to 2 years"3. The consumer-side policy states the same shape of exception with a longer tail on one component: inputs and outputs for a flagged chat are retained "for up to 2 years," and the trust-and-safety classification scores derived from it for "up to 7 years"32. A platform that promises "zero retention" or "30-day retention" as a blanket customer-facing claim is describing the common case, not a hard ceiling — the actual ceiling is set by whichever exception fires, and a flagged conversation can outlive a dozen normal retention cycles of the same platform.

Retention windows are not unique to conversation content, either. Module 27 covers the mechanics of an agent execution sandbox in depth; this module's angle on the same subsystem is purely the retention number attached to it: a code-execution container's data — "execution artifacts, uploaded files, and outputs" — is "retained for up to 30 days," and files pushed through the Files API "persist until explicitly deleted"4. That is a third retention clock, running independently of both the conversation-content clock in Section 1.1 and the flagged-content clock above, attached to a completely different kind of state (a running sandbox, not a message). A platform that thinks about retention as one number is missing that a single agentic session can be governed by three or four different clocks simultaneously — conversation content, tool-execution artifacts, trust-and-safety flags — each with its own default and its own override.

Table 1. Retention windows are per-surface and per-exception, not a single company-wide number. Figures are Anthropic's own documented policy as of the access date below234.
Surface / stateDefault retentionTraining-eligible?Override available
Consumer chat, deleted by user30 daysNo—
Consumer chat, model-improvement opted inUp to 5 yearsYes, de-identifiedToggle off at any time
API message content, standardNot retained by defaultNo, without express permissionCovered Models require 30-day retention
API message content, ZDR arrangementNo storage at restNoNot available for Covered Models
Flagged chat/session (either surface)Up to 2 years (inputs/outputs)NoTrust-and-safety scores: up to 7 years (consumer)
Agent code-execution container stateUp to 30 daysNoFiles API: until explicitly deleted
A conversation's lifecycle: creation, a consent gate, a retention clock per branch, and the flagged-content exception that overrides all of them Conversation created Consumer: opt in to training? API: retained by default? no (default) yes, opted in Short window 30 days (consumer) / not retained (API) De-identified training pipeline, up to 5 yr (consumer opt-in only) Flagged by trust & safety (any surface) overrides everything above: inputs/outputs up to 2 yr, scores up to 7 yr
Figure 2. A conversation's retention window branches on a consent gate that differs by product surface (consumer opt-in vs. API default-no-retention), and a separate trust-and-safety flag can override either branch regardless of which one applied — the dashed path is not a normal state transition, it is an exception that pre-empts the whole diagram23.

Steady-state size of the flagged-content store

Flagged conversations retained at steady state
Flagged conversations added per day

Steady-state store size = daily volume × flag rate × retention days, once the platform has run longer than the retention window (a queue in equilibrium: inflow equals outflow). Anthropic's own documented flagged-content retention windows are up to 2 years (730 days) for inputs/outputs and up to 7 years (2,555 days) for trust-and-safety scores23; daily volume and flag rate are platform-specific inputs this module does not have a published figure for, so they are left as knobs rather than assumed.

2. Learning from a million conversations without reading one: the Clio pipeline

Section 1 established what may be kept and for how long. This section is about what a platform is allowed to learn from what it kept, without turning "we retain conversations" into "an employee reads your conversations." Anthropic built Clio to answer exactly that question for claude.ai: a system that analyzed 1 million real conversations across the Free and Pro tiers and produced aggregate, topic-level insight — the top category was web and mobile application development at over 10% of all conversations, followed by education at more than 7% and business strategy at nearly 6%1 — while the pipeline's own design goal was that no human analyst ever reads a raw conversation to get there.

2.1 Four stages: facets, clustering, private-info-excluded description, hierarchy

Clio's own description of the pipeline names four stages in sequence. First, extracting facets: metadata about a conversation such as its topic or language, computed per-conversation. Second, semantic clustering: grouping conversations by theme using those facets, so the unit of analysis becomes a cluster of many conversations rather than any single one. Third, cluster description: producing a summary of each cluster that explicitly excludes private information — the pipeline's own framing is "summaries excluding private information"1, which makes privacy exclusion a step in the pipeline, not a property that falls out of aggregation by accident. Fourth, building hierarchies: organizing the resulting clusters into a browsable structure so an analyst can explore from a broad category down into narrower sub-topics1.

The ordering matters for the same reason a data pipeline's stage ordering always matters: each stage's output is the next stage's only input, so anything a later stage needs (a privacy guarantee, a minimum sample size) has to have already been enforced by an earlier one. A cluster description generated after aggregation, from a cluster that already satisfies a minimum-size floor, is a fundamentally different privacy posture than generating per-conversation summaries and aggregating those afterward — the former never produces an artifact smaller than the floor; the latter can produce a temporary artifact (a per-conversation summary) that was never protected by the floor at all.

1M raw conversations Facet extraction Semantic clustering Minimum-cluster-size gate Cluster description private info excluded Hierarchy: browsable clusters, human-analyst-visible
Figure 3. Clio's four-stage pipeline. The minimum-cluster-size gate sits between clustering and description — a cluster that does not clear it never reaches the description or hierarchy stages, so no summary is ever generated for a topic too small to aggregate safely1.
  1. Facet extraction. Per-conversation metadata — topic, language — computed once, before any cross-conversation comparison happens.
  2. Semantic clustering. Conversations are grouped by theme using their facets; a minimum threshold on the number of unique users or conversations in a cluster gates everything downstream, "so that low-frequency topics... aren't inadvertently exposed"1.
  3. Cluster description. A summary is generated for each cluster that survived the gate, with private information excluded by construction; Claude itself then verifies the summary "doesn't contain any overly specific or identifying information" before it goes any further1.
  4. Building hierarchies. Verified cluster descriptions are organized into a browsable structure; only these higher-level clusters — never a raw conversation — are visible to a human analyst1.

2.2 The minimum-cluster-size floor

The gate in Figure 3 is doing specific, load-bearing work: Clio enforces "a minimum threshold for the number of unique users or conversations" a cluster must contain, precisely "so that low-frequency topics, even if they are not related to sensitive material, aren't inadvertently exposed"1. The mechanism is the same intuition behind k-anonymity in the general privacy literature — a fact is only reported in aggregate if enough distinct individuals contributed to it — applied at the cluster level rather than the row level. Its cost is symmetric with its benefit: the floor that keeps a three-person cluster from ever surfacing as "three people asked about X" is the same floor that makes a genuinely rare but non-sensitive topic invisible to the analysis, because Clio cannot distinguish "rare and identifying" from "rare and merely uncommon" without more information than the threshold check has access to. Section 4 of the disagreement below returns to exactly this trade-off from a different angle.

2.3 A model verifying its own output before a human sees it

The description stage does not rely solely on the aggregation floor. Anthropic's own account states that "Claude verifies that cluster summaries don't contain any overly specific or identifying information before they're displayed"1 — a second, content-level check layered on top of the count-level check in Section 2.2. This is worth noting as a specific design choice rather than an obvious one: the floor in Section 2.2 protects against a cluster being too small in principle; the model-verification step protects against a cluster that cleared the size floor but whose generated description nonetheless leaked something specific (a name, a verbatim quote, a detail unique enough to re-identify a contributor even inside a large cluster). Neither check subsumes the other, and the published pipeline runs both.

2.4 Validating against an independent signal

Anthropic checked Clio's own output against a system built for an entirely different purpose: the platform's Trust and Safety classification system. "The overall correlation between the Trust and Safety and the Clio classifications was r = 0.71"1 — a moderately strong agreement between two independently built systems looking at the same underlying conversations through different lenses (Clio's topic clusters versus Trust and Safety's policy-violation classifications), used operationally to surface disagreements as candidates for false positives or false negatives in either system. The same cross-check surfaced a genuine finding rather than just a sanity check: Clio "identified a network of automated accounts using similar prompt structures to generate spam for search engine optimization"1 — a coordinated-abuse pattern visible only because conversations were being compared to each other in aggregate, which is the entire reason to build a privacy-preserving aggregate-analysis system instead of relying on per-conversation review alone. Module 29 covers the real-time classifier side of trust and safety; Clio's role here is the aggregate, retrospective side, correlated against it rather than replacing it.

Table 2. The three largest conversation categories Clio surfaced from 1 million claude.ai conversations (Free and Pro tiers)1.
CategoryShare of conversations
Web and mobile application development> 10%
Education> 7%
Business strategy≈ 6%

3. Per-token metrics: the LLM-specific observability primitive

Module 11 built the general vocabulary: an SLI is "a carefully defined quantitative measure of some aspect of the level of service that is provided"5, and an SLO is a target range for that measure. That vocabulary does not specialize itself — it has to be handed a measure. For a request/response API, "request latency" is usually an adequate SLI: the client waits, then gets one answer, and one number describes the wait. A streamed LLM completion breaks that assumption structurally, not just by degree. The client does not wait once; it receives a sequence of tokens over time, and two requests with identical total latency can feel completely different to a caller — one that arrives instantly and then dribbles out one word every 400 milliseconds, and one that pauses for two seconds and then streams the rest at a smooth clip. "Request latency" as a single SLI cannot distinguish these, which is exactly why LLM serving needs its own SLI vocabulary rather than reusing Module 11's directly.

3.1 Time to first token: definition and what it bundles

Time to first token (TTFT) is defined, in NVIDIA's own benchmarking-tool documentation, as the elapsed time from request dispatch to the arrival of the first response chunk6. The same documentation is explicit about what that interval bundles together: it "includes network latency, queuing time, prompt processing, and generation of the first token"6 — four separable causes collapsed into one number. This is a deliberate simplification, not an oversight, and Section 4 of this module is precisely about un-bundling it again once the request is traced across the systems that contribute each piece: the network and queuing components live at the gateway and router hops, prompt processing is the GPU backend's prefill phase, and only the last piece is decode. A caller sees one TTFT number; a platform operator diagnosing why it moved needs the trace that Section 4 builds.

The OpenTelemetry project's semantic conventions for generative AI systems standardize this as a named, typed metric rather than an ad hoc measurement: gen_ai.server.time_to_first_token, a histogram in seconds, described as "time to generate first token for successful responses"7. Standardizing the metric name and its histogram bucket boundaries — the convention specifies buckets at 1 ms, 5 ms, 10 ms, 20 ms, 40 ms, 60 ms, 80 ms, 100 ms, 250 ms, 500 ms, 750 ms, 1 s, 2.5 s, 5 s, 7.5 s, and 10 s7 — is what lets a metric emitted by a vLLM backend, an SGLang backend, and a hosted-API gateway all land in the same dashboard without a translation layer, the same engine-agnostic-schema argument Module 25 makes for the model-server protocol's own metric names, applied here to the observability signal instead of the routing signal.

3.2 Inter-token latency: definition, formula, and a wall-clock subtlety

Once the first token has arrived, the question changes from "how long did the caller wait" to "how smoothly does the rest arrive." Inter-token latency (ITL), also called time per output token (TPOT), is the average time between successive output tokens after the first one. NVIDIA's tooling documentation gives the formula directly:

$$ \mathrm{ITL} = \frac{\text{request\_latency} - \mathrm{TTFT}}{\text{output\_sequence\_length} - 1} $$

which is exactly the total decode-phase wall-clock time divided by the number of token-to-token gaps it contains (one fewer than the token count, since the first token has no predecessor to measure a gap against)6. The same documentation adds a precision point worth keeping explicit, because it is easy to misread ITL as a per-token cost rather than a normalized rate: "ITL is generated-token-normalized decode duration. It is not the total wall-clock decode duration"6. A request that generates 500 tokens in 10 seconds and a request that generates 50 tokens in 1 second report the identical ITL (20 ms) despite an order-of-magnitude difference in how long the caller actually waited for decode to finish — ITL characterizes the backend's steady-state generation rate, not any individual request's total experience, which is why it is reported as its own metric rather than derived on demand from total latency alone.

OpenTelemetry's convention mirrors this exactly as gen_ai.server.time_per_output_token, a histogram in seconds with narrower buckets than TTFT's — 10 ms up through 2.5 s7 — reflecting that a healthy decode step is expected to be one to two orders of magnitude faster than a healthy prefill, which is itself the direct consequence of the arithmetic-intensity asymmetry between a compute-bound prefill pass over the whole prompt and a memory-bandwidth-bound decode step generating one token at a time — vocabulary this course's multi-node-llm-inference track derives from the GPU side and this module simply measures from the outside.

3.3 Throughput: per-request vs. aggregate, and why batching decouples them

Throughput answers a third, different question: not how long any one caller waited, but how many tokens the system is producing per unit time. NVIDIA's tooling defines aggregate output token throughput as total output tokens generated divided by the benchmark's wall-clock duration across every concurrent request6 — a system-level rate, not a per-request one. The distinction matters because the two move in opposite directions under exactly the mechanism Aman already knows from batched decode: continuous batching lets a GPU backend serve more concurrent streams by packing more sequences into the same decode step, which raises aggregate tokens/sec (more work retired per step) while it can simultaneously raise per-stream ITL (each individual stream's next token depends on a step that is now doing more total work, so the step itself takes longer). A dashboard that reports only the aggregate number can show "throughput is up" in the same window a per-request percentile dashboard shows "ITL is up" — both are true, and neither is more correct; they are the same batching decision viewed from the operator's side and the caller's side.

A single streamed response's timeline: queueing and prefill before the first token, then a sequence of decode steps each separated by the inter-token latency wall-clock time since request sent queue + network prefill (prompt processing) TTFT tok 1 tok 2 tok 3 tok 4 ITL ITL ... remaining tokens, same ITL cadence ... tok N (last) request_latency (end to end)
Figure 4. One streamed response's timeline. TTFT covers queueing, network, and prefill, ending at the first token; every gap after that is one sample of the inter-token latency; the whole span is request_latency, from which ITL is derived as \((\text{request\_latency} - \mathrm{TTFT}) / (N-1)\)6. Aggregate tokens/sec is a separate, system-wide measure taken across every concurrent stream, not read off this one timeline6.

These numbers are not abstract. Google Cloud's own account of adopting load- and cache-aware routing (the mechanism Module 24 covers as backend-signal-aware routing) reports concrete, measured TTFT deltas from changing only the routing decision, not the model or the hardware: "over 35% for Qwen3-Coder" in average TTFT, and for Deepseek V3.1 under bursty chat traffic, P95 TTFT improved "2x (52%)"8. The same change doubled the platform's prefix-cache hit rate, from 35% to 70%8 — a routing-quality improvement that shows up directly in the TTFT metric this section defines, because a cache-hit prefill does less compute than a cache-miss one. A platform that only tracks aggregate request latency would see this as "things got faster" without being able to say whether the improvement came from routing, from caching, or from the backend itself; a platform tracking TTFT and ITL separately, per Section 4's traced breakdown, can attribute it to the right hop.

Table 3. OpenTelemetry's semantic-convention names for the LLM-specific metrics this section defines, with their instrument type and histogram bucket range7.
Metric nameInstrumentUnitBucket range
gen_ai.server.time_to_first_tokenHistogramseconds1 ms – 10 s
gen_ai.server.time_per_output_tokenHistogramseconds10 ms – 2.5 s
gen_ai.server.request.durationHistogramseconds10 ms – 81.92 s
gen_ai.client.token.usageHistogramtokens1 – 67,108,864

Effective tokens/sec for one stream, from TTFT and ITL

Total request_latency
Effective tokens/sec (whole request)
Decode-only tokens/sec (excludes TTFT)

Inverting Section 3.2's formula: request_latency = TTFT + ITL × (N − 1). "Effective tokens/sec" divides the whole request by that total, so a large TTFT dilutes it on short completions; "decode-only tokens/sec" is 1/ITL and is unaffected by TTFT, matching NVIDIA's per-request throughput definition6.

4. Cross-system tracing and cost attribution: one span, three hops

Module 11 introduced the trace as the tool for "which of the eleven services this request touched is where the extra 400 ms actually went" — causality across service boundaries, using a propagated identifier so a span at one hop can be attributed to the same request that entered elsewhere. Every general-purpose microservice fleet has this problem; an inference platform has a specific, narrower version of it, because Module 21, Module 24, and Module 25 already fixed exactly which hops a request crosses: a gateway that authenticates and rate-limits, a router/scheduler that picks a backend using the KV-cache-utilization and queue-depth signals Module 24 and Module 25 establish, and a GPU backend that actually runs prefill and decode. This section's job is narrow: show what a trace looks like once it is built to cross those three specific hops, and what it takes to attribute not just latency but cost to each one.

4.1 Extending a trace across a boundary Module 11 did not have to cross

The mechanism that lets a span at hop three be attributed to the same request that entered at hop one is a propagated header, and the W3C's Trace Context specification standardizes its shape: a traceparent header carrying a version, a trace-id, a parent-id, and trace-flags, where the trace-id "is the ID of the whole trace forest and is used to uniquely identify a distributed trace through a system," and the parent-id is "the ID of this request as known by the caller (in some tracing systems, this is known as the span-id...)"9. Every hop that receives a request reads the incoming parent-id, records it as its own span's parent, generates a new span-id for the work it is about to do, and passes that new id forward as the next hop's parent-id. Nothing about this mechanism is inference-specific — it is the same propagation Module 11's general trace concept assumes — but the GPU backend hop is where an ordinary microservice trace usually stops and an LLM-serving trace has to keep going, because that hop is where TTFT and ITL, Section 3's metrics, are actually generated.

OpenTelemetry's semantic conventions for generative AI give that terminal hop its own span shape rather than treating it as an opaque leaf: a GenAI operation span's name follows the pattern {gen_ai.operation.name} {gen_ai.request.model}, it carries gen_ai.provider.name and, where available, gen_ai.request.model as required or conditional attributes, and it records gen_ai.usage.input_tokens and gen_ai.usage.output_tokens as the token counts the operation consumed and produced8. Those two token-count attributes are the piece Section 4.3 needs: a span that already carries "how many input and output tokens this specific hop's work consumed" is a span that already carries everything a pricing table needs to compute what that hop cost, without a separate accounting pass having to re-derive token counts from a log line after the fact.

4.2 What each hop actually contributes

Mapped onto the three-hop shape this module has been assuming since Figure 1, each span's dominant contribution is different, and each is already something a prior module in this course established rather than something new to this one. The gateway span (Module 21) is dominated by auth and budget checks against a cache, plus the parallel rate-limiter check — work that is supposed to be fast and synchronous, with the platform's own spend logging pushed off this critical path entirely, run instead as one of the asynchronous background tasks LiteLLM's own architecture documentation describes11. The router/scheduler span (Module 24) is dominated by whatever queueing the admission-control layer imposes plus the routing decision itself, scored against the KV-cache-utilization and queue-depth signals Module 25's model-server protocol exposes. The GPU backend span is where Section 3's TTFT and ITL actually happen — prefill inside TTFT, decode spread across every inter-token gap — and it is the only one of the three hops whose duration is a direct function of the request's own prompt and output length rather than of platform load.

Table 4. What each hop in the trace contributes to latency, and what it needs to attribute cost, cross-referenced to the module that already established each signal.
HopDominant latency sourceCost basisSignal established in
GatewayAuth/budget cache check, rate-limiter checkFixed per-request overhead (not token-priced)Module 21
Router / schedulerAdmission queueing, routing-decision computeFixed per-request overhead (not token-priced)Module 24, Module 25
GPU backendPrefill (inside TTFT) + decode (ITL × N−1)input_tokens × input price + output_tokens × output priceSection 3, this module

4.3 Attributing cost, not just latency, to each hop

Latency attribution and cost attribution use the same trace but answer different questions, and Table 4's third column is the reason: the gateway and router hops have a real infrastructure cost (compute, memory, the cache lookups Module 21 describes), but it is not priced per-token and does not vary with how long the completion turned out to be — the same auth check runs whether the model generates 10 tokens or 10,000. The GPU backend hop is the opposite: its cost is a direct, computable function of the two numbers its own span already carries, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens8, multiplied by that model's published per-token price. Using Claude Sonnet 5's current rate of $2 per million input tokens and $10 per million output tokens10, a request that entered with 3,000 input tokens and generated 500 output tokens attributes its cost entirely to the GPU-backend span:

$$ \text{cost} = 3{,}000 \times \frac{\$2}{1{,}000{,}000} + 500 \times \frac{\$10}{1{,}000{,}000} = \$0.006 + \$0.005 = \$0.011 $$

Every dollar of that $0.011 is attributable to one span in the trace, the GPU-backend hop, because that is the only hop whose work is priced by what it processed rather than by the fact that it ran at all. This is the same asymmetry Module 23's credit-ledger design already exploits from the billing side — a hold sized to a worst-case token count, settled against an actual token count once generation finishes — read here from the tracing side instead: the trace does not need a separate cost-computation pass, because the cost is already a deterministic function of two attributes the GenAI span convention requires it to carry.

The reason this module bothers to attribute cost per hop, rather than leaving cost as a single request-level total the way Module 23's ledger already computes it, is what happens to that number next. A request-level total tells a billing system what to charge one customer. A trace with cost attributed at the hop that actually incurred it tells a capacity-planning system something different: which of the platform's own subsystems is consuming budget, at what rate, under what load — the input Module 31 needs to decide whether a given tenant's traffic pattern justifies a dedicated GPU pool or fits comfortably inside a shared one. Module 31 treats this module's cost-attribution signal as exactly that: a feedback input into a capacity decision, not a restatement of Module 23's billing ledger.

One request's trace as three spans, each carrying its own duration and its own attributed cost, linked by a shared trace-id under a W3C Trace Context propagation chain trace-id: 4bf92f... (shared by every span below) Gateway span parent-id: (client) span-id: a1 traceparent: ...a1 Router/scheduler span parent-id: a1 span-id: b2 traceparent: ...b2 GPU backend span parent-id: b2, span-id: c3 input=3000, output=500 time gw: 4ms $0 token cost router: 12ms $0 token cost GPU backend: prefill (TTFT) + decode (ITL × 499) $0.011 attributed here Total span (request_latency) — one trace-id, cost summed across spans, only one span priced by tokens
Figure 5. A W3C Trace Context traceparent chain links three spans under one trace-id9; the GenAI span convention's token-usage attributes8 let the GPU-backend hop carry its own attributed cost, computed here at Claude Sonnet 5's published per-token rate for 3,000 input and 500 output tokens10. Gateway and router latencies are illustrative order-of-magnitude figures, not measured production numbers.

Where experts disagree

Section 2.2 flagged a real tension inside Clio's own design: a minimum-cluster-size floor protects against small clusters being identifying, at the cost of making genuinely rare-but-harmless topics invisible. That tension is one instance of a broader, older disagreement in the privacy-engineering literature about what "privacy-preserving" is even allowed to mean — a heuristic property, verified after the fact, or a mathematical guarantee, bounded in advance. The three positions below are not all about Clio specifically; they are about which standard an aggregate-analysis pipeline over user conversations should be held to, and Clio, differential privacy, and plain threshold-based aggregation are three different answers already deployed at scale.

Engineered anonymization, verified after the fact, is sufficient

Anthropic's production design for Clio1

Aggregate by theme, enforce a minimum user/conversation count before a cluster is allowed to produce a summary, generate that summary with private information excluded by construction, and have a model check the result for anything "overly specific or identifying" before a human ever sees it. No step promises a mathematical bound on worst-case leakage; the claim is that the combination of aggregation, a count floor, and content-level verification is good enough in practice, and the r = 0.71 correlation against an independently built Trust and Safety system is offered as evidence the pipeline's classifications track something real rather than noise.

  • Supports open-ended, exploratory topic discovery over free-text conversations — a statistic does not have to be defined in advance, which a formal-privacy mechanism generally requires
  • Cheap enough to run over a million conversations and re-run as the underlying model and its usage change
  • The verification step is done by the same kind of model that will consume the output, so it can catch semantically identifying content a purely statistical threshold cannot see
  • No provable bound on how much information about any individual conversation could, in principle, be recovered from a cluster description — the guarantee is "we checked," not "we proved"
  • The minimum-count floor is itself a heuristic; nothing stops a cluster from clearing the count threshold while its generated description still leaks something specific about a minority of contributors

Only a formal, bounded guarantee is defensible

Differential privacy, as Apple has implemented and disclosed it in production12

Add calibrated statistical noise before data ever leaves the device, governed by a privacy budget quantified by the parameter epsilon, so that "it is difficult to determine whether a certain user contributed to the computation of an aggregate"12 — a guarantee that holds regardless of what auxiliary information an adversary later obtains, because the noise is added before aggregation, not inferred as a property of the aggregate afterward. Apple's own deployment sets epsilon per feature and per contribution — epsilon of 4 for Safari-crash/energy domains and for emoji suggestions, epsilon of 8 for QuickType, epsilon of 2 for Health-type usage — and caps both the number of daily contributions per user and the total retention window (three months) for the raw donations12.

  • The privacy loss is bounded by a single number (epsilon), composable across queries, and independent of what an adversary already knows or later learns
  • The guarantee does not rest on trusting a downstream verification step to catch what aggregation missed
  • Requires defining a fixed statistic — a count, a histogram bucket, a most-common-value sketch — before collection; it does not naturally support Clio's open-ended "discover what topics exist" task, where the space of possible clusters is not known in advance
  • The noise that makes the guarantee hold is exactly what damages the small, rare-but-real signal both this position and Clio's own minimum-count floor are trying to protect — a low-frequency true value can be swamped by injected noise before it is ever visible, so the utility cost concentrates in the same region Section 2.2 already identified as the hardest case

Aggregation-and-threshold alone, without semantic abstraction, is not privacy

The de-anonymization literature, exemplified by Narayanan and Shmatikov's attack on the Netflix Prize dataset13

A dataset that has only had identifiers stripped and been restricted to records above some count threshold is not thereby safe: the paper "present[s] a new class of statistical de-anonymization attacks against high-dimensional micro-data" and shows that "an adversary who knows only a little bit about an individual subscriber can easily identify this subscriber's record in the dataset"13, using nothing more than a second, independently public dataset (IMDb) as auxiliary knowledge. The lesson generalizes past Netflix ratings to any sparse, high-dimensional record of individual behavior, which a raw conversation log is.

  • Empirically demonstrated, not hypothetical — a real dataset, a real re-identification, using only public auxiliary information
  • Directly explains why Clio's designers added a content-level verification step (Section 2.3) rather than trusting the count floor alone: the floor addresses this attack's precondition (a record too rare to blend in) but not a cluster description that inadvertently repeats specific, identifying detail from within an otherwise-large cluster
  • The attack targets released micro-data (per-record entries an analyst can query directly); it does not, by itself, show Clio's cluster-level summaries — which never expose a per-conversation record to the analyst at all — are similarly breakable, only that the naive version of the defense (strip identifiers, keep a count floor, stop there) is not
Assessment

For open-ended, exploratory analysis of what a platform's users are actually doing — Clio's actual task, where the set of interesting clusters is not known before the pipeline runs — the engineered-and-verified approach is the only one of the three that fits the problem shape at all; differential privacy's requirement to fix a statistic in advance is a mismatch, not a missing feature waiting to be added. For a narrower, recurring, adversarially-exposed query — a public dashboard of daily active users by feature, say, that gets queried the same way every day by parties who may be motivated to extract individual information — the formal guarantee is worth its utility cost, because the same query being run repeatedly is exactly the composability regime differential privacy's bound is designed for. What the third position settles, regardless of which of the first two a platform picks, is that a bare minimum-count threshold with no content-level check is not a defensible stopping point on its own; Clio's own inclusion of a model-verification step past the count floor is implicit agreement with that. What would move this from "reasonable design choice" to "verified" is the kind of adversarial audit Apple and Google have both published against their own differential-privacy deployments — nobody has yet published an equivalent membership-inference audit attempting to recover which individual conversations contributed to a specific Clio cluster description, and until someone does, position one's central claim rests on Anthropic's own account of its verification step, not on an independent test of it.

References

  1. blogAnthropic. Clio: Privacy-preserving insights into real-world AI use. anthropic.com/research, 2024. https://www.anthropic.com/research/clio. Accessed 2026-09-17. Used for: the four-stage pipeline (facet extraction, semantic clustering, private-info-excluded description, hierarchy); the minimum-cluster-size floor; model self-verification of cluster summaries; the r = 0.71 correlation with Trust and Safety and the spam-network finding; the top three conversation-category shares from 1M analyzed conversations.
  2. officialAnthropic. How long do you store my data? Privacy Center. https://privacy.claude.com/en/articles/10023548-how-long-do-you-store-my-data. Accessed 2026-09-18. Used for: consumer-product 30-day deletion default; up-to-5-year retention for opted-in model-improvement data; up-to-2-year/7-year flagged-content and trust-and-safety-score retention.
  3. officialAnthropic. API and data retention. platform.claude.com docs. https://platform.claude.com/docs/en/manage-claude/api-and-data-retention. Accessed 2026-09-18. Used for: API default of no message-content retention without express permission; zero data retention (ZDR) arrangement; Covered Models' 30-day retention requirement; the flagged-content/legal-hold exception that survives ZDR and HIPAA readiness.
  4. officialAnthropic. Code execution tool (Claude API). platform.claude.com docs, tool versions code_execution_20250825/20260120/20260521. https://platform.claude.com/docs/en/agents-and-tools/tool-use/code-execution-tool. Accessed 2026-09-17. Used for: the code-execution sandbox's own 30-day container-data retention window, as a third, independent retention clock distinct from conversation content.
  5. textbookJones, C., Wilkes, J., Murphy, N., Smith, C. (ed. Beyer, B.) Site Reliability Engineering, Chapter 4: Service Level Objectives. Google / O'Reilly, 2016. https://sre.google/sre-book/service-level-objectives/. Accessed 2026-09-17. Used for: the SLI definition this module specializes into per-token metrics.
  6. officialNVIDIA. Metrics Reference, AIPerf documentation. https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference. Accessed 2026-09-18. Used for: the time-to-first-token definition and its bundled components (network, queuing, prompt processing, first-token generation); aggregate output-token-throughput definition; the ITL = (request_latency − TTFT) / (output_sequence_length − 1) formula and the “generated-token-normalized decode duration, not total wall-clock decode duration” precision note.
  7. standardOpenTelemetry. Semantic Conventions for Generative AI Metrics. semantic-conventions-genai repository, docs/gen-ai/gen-ai-metrics.md. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md. Accessed 2026-09-18. Used for: the gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token, gen_ai.server.request.duration, and gen_ai.client.token.usage metric names, instrument types, units, and histogram bucket boundaries.
  8. standardOpenTelemetry. Semantic Conventions for Generative AI Systems (spans). semantic-conventions-genai repository, docs/gen-ai/gen-ai-spans.md. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md. Accessed 2026-09-18. Used for: the GenAI operation span-naming pattern, required/conditional attributes (gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model), and the gen_ai.usage.input_tokens/output_tokens attributes used for cost attribution.
  9. standardW3C. Trace Context. W3C Recommendation. https://www.w3.org/TR/trace-context/. Accessed 2026-09-18. Used for: the traceparent header's version/trace-id/parent-id/trace-flags fields and the parent-id-to-span-id propagation mechanism across hops.
  10. officialAnthropic. Pricing. platform.claude.com docs. https://platform.claude.com/docs/en/about-claude/pricing. Accessed 2026-09-18. Used for: Claude Sonnet 5's current per-token pricing ($2/MTok input, $10/MTok output) used in the cost-attribution worked example.
  11. officialBerriAI / LiteLLM contributors. Life of a Request (LiteLLM Proxy architecture). Live documentation. https://docs.litellm.ai/docs/proxy/architecture. Accessed 2026-09-17. Used for: the gateway hop's fixed, non-token-priced overhead (auth/budget cache check, rate-limiter check) referenced in Table 4.
  12. officialApple. Differential Privacy Overview. Apple privacy technical whitepaper. https://www.apple.com/privacy/docs/Differential_Privacy_Overview.pdf. Accessed 2026-09-18. Used for: the local differential privacy mechanism, the privacy-budget/epsilon concept, per-feature epsilon values (2, 4, 8), and the 3-month raw-donation retention window, in the "Where experts disagree" section.
  13. paperNarayanan, A., Shmatikov, V. Robust De-anonymization of Large Sparse Datasets (How to Break Anonymity of the Netflix Prize Dataset). IEEE Symposium on Security and Privacy, 2008. https://arxiv.org/abs/cs/0610105. Accessed 2026-09-18. Used for: the de-anonymization attack on the Netflix Prize dataset and its implication for aggregation-and-threshold-only anonymization, in the "Where experts disagree" section.