System design · Part 3
AI Inference Platform Design: Client to GPU
The complete system between a client and a GPU backend -- gateway, identity, billing, routing, streaming, agents, caching, safety, data, tenancy -- given full treatment as the part Aman builds against most directly.
21Edge and API Gateway for AI Inference Platforms35 min22Identity, Rate Limits, and Quotas: API Keys, Orgs, and Token-Bucket Enforcement40 min23Billing and Metering: Ledger Design, Idempotency, Holds, and Reconciliation35 min24Request Routing, Admission Control, and SLO Tiers: Fallbacks and Graceful Degradation35 min25The GPU Backend Handoff: Designing the Interface to Multi-Node Serving30 min26Streaming, Cancellation, and Resumption: SSE, WebSockets, and Long-Running Requests35 min27Tool Use, Agent Harnesses, and Sandboxes for Code Execution40 min28Platform Prompt/Prefix Caching: Tiers, TTLs, and Billing Implications30 min29Safety and Moderation Pipelines: Pre/Post Filters, Classifiers, and Policy Routing35 min30Data and Observability: Conversation Storage, Retention, Privacy, and Per-Token Tracing30 min31Multi-Tenancy, Isolation, and Capacity/Cost Management35 min