AI Inference Hardware · Page 6

Choosing Inference Hardware: Model-Size Fit, Cloud Pricing, and TCO Across Vendors

Pages 1-5 established what each chip family can do and how to read a benchmark chart correctly. This page turns that into a decision: which memory footprint forces which parallelism strategy, what a chip actually costs by the hour across AWS, GCP, and Azure, what that implies for cost per million tokens, and where the four still-open disagreements in this track — TPU-vs-GPU cost, AMD-vs-NVIDIA "who wins," Rubin-vs-MI455X on paper, and the specific $0.39/chip-hour TPU claim — currently stand. No new chip specs or benchmark methodology appear here; every number below was established on pages 1-5 or comes from an official cloud pricing source.

Deep dive 55 min read 24 sources As of 2026-09-17 Builds on: Pages 1-5
Three inputs — model size, traffic pattern, and budget — narrow the hardware search to a short list Three questions narrow every hardware choice in this track Model size 7B → 670B+ MoE ↓ memory fit single-chip / multi-chip / rack (§1) Traffic pattern interactive vs. offline batch ↓ $/M tokens at your tok/s/user target (§2) Budget shape on-demand vs. 1yr/3yr commit ↓ $/hour by cloud and SKU (§3) 2-3 defensible hardware choices with a sourced $/hour or $/M-token number for each (§7)
Figure 1. This page's structure follows three questions in order: what memory footprint does the model need (§1), what does that traffic pattern cost per million tokens (§2), and what does the chosen hardware cost by the hour from an official cloud price list (§3) — before interconnect, software lock-in, and next-gen roadmap context (§4-6), worked per-model-size walkthroughs (§7), and the four open disagreements (TPU-vs-GPU cost, AMD-vs-NVIDIA, Rubin-vs-MI455X, the $0.39/chip-hour claim) close the page.

1. Model-size-to-hardware fit: where the memory line falls

Every chip family in this track publishes a peak-theoretical HBM capacity per accelerator, and pages 1-4 derived each one from first principles. This section does not re-derive those numbers; it lines them up against the weight footprint of four reference model sizes — 7B, 70B, 405B, and 670B+ mixture-of-experts — using the same 1-byte-per-parameter FP8 convention pages 2 and 5 already used for their own single-GPU and roofline arithmetic. The arithmetic is deliberately simple: weight bytes only, no KV cache, no activation memory, no optimizer state. Real deployments need headroom beyond the weight footprint for exactly those reasons, so treat every "fits" verdict below as an upper bound on what a single chip could hold, not a serving-ready configuration.

1.1 Single-chip capacity ceilings

Table 1 puts the inference-relevant single-chip HBM capacities from pages 1-4 on one grid, alongside each part's peak memory bandwidth — the second number that matters once a model does fit, because bandwidth (not capacity) sets the decode-throughput ceiling once weights are resident (page 5, §4).

Table 1. Single-accelerator HBM capacity and bandwidth, inference-relevant SKUs from pages 1-4. NVIDIA figures from the H100 and H200 product pages12; AMD figures from the MI300X, MI325X, and MI350X-platform data sheets345; Google figures from the TPU v6e and TPU7x (Ironwood) system-architecture docs67; AWS figures from the Trn2 architecture docs8.
PartHBM capacityHBM bandwidthPrecision basis
H100 SXM80 GB3.35 TB/sHBM3
H200 SXM141 GB4.8 TB/sHBM3e
MI300X192 GB5.3 TB/sHBM3
MI325X256 GB6.0 TB/sHBM3e
MI350X/MI355X288 GB8.0 TB/sHBM3e
TPU v6e (Trillium)32 GB1,638 GBpsHBM
TPU7x (Ironwood), per chip192 GiB7,380 GBpsHBM
Trainium2, per chip96 GiB2.9 TB/sHBM

Two things fall out of Table 1 directly. First, capacity spans a factor of 9 across current-generation single chips (32 GB on TPU v6e to 288 GB on MI350X/MI355X) — wider than the compute spread across the same set, which is why capacity, not FLOPS, is usually the binding constraint on which chip a given model can even start on. Second, AMD's HBM3/HBM3e-generation parts hold the capacity lead at every single-chip tier: MI300X's 192 GB already exceeds H200's 141 GB, and MI350X/MI355X's 288 GB is the largest capacity any currently-shipping single accelerator in this track carries345 — the memory-capacity framing page 2 built around MI300X's "trillion-parameter" marketing claim.

A third pattern is worth reading off the bandwidth column specifically, separate from capacity: bandwidth and capacity do not move together across this table. TPU v6e has the smallest capacity in Table 1 (32 GB) but a mid-table bandwidth (1,638 GBps) — a higher bytes-per-GB ratio than H100, whose 80 GB moves at a lower 3.35 TB/s in absolute terms but represents a chip built around a different capacity/bandwidth trade entirely. Ironwood is the outlier at the top of both columns simultaneously (192 GiB and 7,380 GBps, the largest figure in each), which is consistent with page 3's own framing of Ironwood as the first TPU generation explicitly codesigned for both halves of the inference workload rather than inheriting a training-first chip's capacity/bandwidth balance. The practical reading: a chip's capacity tells you whether a model fits at all, but its bandwidth-to-capacity ratio tells you how much of that capacity you can actually exploit with a large batch before decode throughput (page 5's roofline argument, its §4.3) becomes the bottleneck instead.

1.2 Where a model crosses the single-chip line

A 7B model at FP8 needs roughly 7 GB of weights — comfortably under every capacity in Table 1, TPU v6e's 32 GB included, with room left for a meaningful KV cache and batch on any of them. A 70B model at FP8 needs roughly 70 GB: that fits on H200 (141 GB), MI300X (192 GB), MI325X (256 GB), MI350X/MI355X (288 GB), and Trainium2's single-chip 96 GiB, each with tens of gigabytes free for KV cache and activations, but it does not comfortably fit on H100's 80 GB (10 GB of headroom is thin for any real batch or long context) and it categorically does not fit on TPU v6e's 32 GB without sharding across chips. This is the exact arithmetic gap InferenceMAX measures directly: on LLaMA 3.3 70B at low interactivity, MI300X's larger per-chip capacity lets it run tensor-parallelism degree 1 (one chip, no cross-chip communication at all) at a batch size H100 and H200 cannot match without splitting the model across two chips first10 — the capacity gap in Table 1 is not an abstract number, it is the mechanism behind one of this page's own disagreements ("Where experts disagree," below, D2).

A 405B model at FP8 needs roughly 405 GB — more than any single chip in Table 1 holds, MI350X/MI355X's 288 GB included. This model size cannot run on one accelerator on any current-generation part in this track; it needs at least two-way tensor parallelism on the largest-memory GPUs (two MI350X/MI355X at 288 GB each gives 576 GB of pooled capacity over the Infinity Fabric mesh, page 2) or a wider TP degree on smaller-memory parts. This is exactly the model size MLPerf Inference v5.0 introduced as its own new benchmark, and it is the model page 1's own decomposition of the "30x" GB200-vs-H100 headline (its §4) works through in full — that decomposition is this exact model size's own real-world serving arithmetic, not a separate case.

A 670B-parameter mixture-of-experts model — DeepSeek R1/V3-class, the model InferenceMAX itself tests directly10 — needs roughly 670 GB of weights at FP8. That crosses every single-chip and even most simple two- or four-chip tensor-parallel configurations in Table 1; it is the model size that forces §1.3's rack-scale domains rather than a larger TP degree on the same node, both because MoE inference typically wants expert parallelism (routing different experts to different chips, not replicating the same weights) and because 670 GB alone, before KV cache, already exceeds four MI350X/MI355X GPUs' combined 1.15 TB only with room to spare, and exceeds two by a wide margin.

One distinction is worth making explicit here, because it is easy to get backward: the 670 GB figure is a memory-capacity number, not a compute-cost number, and mixture-of-experts routing makes those two diverge sharply. A MoE model like DeepSeek activates only a small subset of its experts for any given token, so the FLOPs per token look much closer to a dense model with a far smaller parameter count — but every deployment still has to keep all experts resident in memory, because routing decides which experts to use per token, per layer, dynamically, and a chip cannot know in advance which subset it will need. That is precisely why the memory-capacity thresholds this section works through, not the FLOPS thresholds pages 1-4 also publish, are the correct lens for sizing an MoE deployment: a 670B-parameter MoE model is cheap to compute relative to its size and expensive to store relative to its size, the inverse of what a 670B dense model would be, and Table 1's capacity column is doing the load-bearing work in this section for exactly that reason.

1.3 Rack-scale domains as their own size category

Above roughly the 405B-670B threshold, the unit of comparison stops being a chip at all and becomes a coherent memory domain. GB200 NVL72 pools 13.4 TB of HBM3e across 72 Blackwell GPUs inside one 130 TB/s NVLink5 fabric9 — roughly 186 GB per GPU, but the domain-level property that matters for a 670B-class MoE model is that any of the 72 GPUs can address any other GPU's memory directly, so a model that would need explicit expert-parallel routing logic across separate nodes on a smaller fabric can instead treat the rack as one address space (page 1, §3). AWS's Trn2 UltraServer is the equivalent AWS-side domain: 64 Trainium2 chips pooling 6,144 GiB (roughly 6.4 TB) of device memory at 185.6 TB/s of aggregate bandwidth8, large enough to hold several replicas of a 670B-class model's weights with room for KV cache, though at a coherent-domain size roughly half of GB200 NVL72's.

Google's TPU side is a different shape of the same idea, with one caveat worth stating plainly: TPU7x's documented pod maximum is 9,216 chips7, which at 192 GiB per chip implies well over a petabyte of pooled HBM in principle — but the largest single-VM topology Google's own TPU7x documentation states is 8x16x16, 2,048 chips7, not the full 9,216-chip pod. The pod ceiling is a training-scale number; what a single inference deployment can actually address as one coherent serving unit is the smaller, documented VM topology, the same distinction page 3 draws between peak pod size and what a serving slice can book (its §4). Treat "9,216 chips" as Ironwood's maximum training-scale fabric and "2,048 chips" as the more relevant ceiling for how large a single TPU-served model deployment can get before it needs to be split into independently-routed replicas rather than one coherent domain.

Even 2,048 chips at 192 GiB each is, on paper, an enormous coherent pool — well over 350 TB — but two caveats keep that number from being a straightforward "TPU wins at rack scale" conclusion. First, no cost-per-token data exists for Ironwood anywhere in this page's source set (§2's InferenceMAX and Artificial Analysis figures stop at v6e); every dollar comparison this page can make for Google's hardware is against the smaller, older TPU generation, not the one with the largest domain. Second, Ironwood's own architecture (page 3) adds SparseCore units specifically for the irregular, gather-scatter-heavy access pattern of embedding lookups and, by extension, of MoE expert routing — a genuine architectural fit for a 670B-class MoE model's routing pattern, but one this track has no independent benchmark to confirm delivers a real throughput or cost advantage on an actual MoE inference workload, as opposed to the recommendation-model workloads SparseCore was originally built for.

Model size vs. hardware category: 7B fits any single chip, 70B needs the larger-memory single chips, 405B needs multi-chip tensor parallelism, 670B+ MoE needs a rack-scale domain model size (log scale) → 7B ~7 GB weights (FP8) any single chip in Table 1 70B ~70 GB weights (FP8) H200, MI300X, MI325X, MI350X/355X, Trainium2 (single chip) H100 tight; TPU v6e, Inf2 need TP 405B ~405 GB weights (FP8) 2-way+ TP largest-memory GPUs, or GB200 NVL72 fraction 670B+ MoE ~670 GB weights (FP8) Rack-scale domain GB200 NVL72 (13.4 TB) Trn2 UltraServer (6.4 TB) expert-parallel routing
Figure 2. Weight-only FP8 footprint (1 byte/parameter) against the single-chip capacities in Table 1 and the rack-scale domains in §1.3. Real deployments need additional headroom for KV cache and activations beyond the weight footprint shown here; this figure states which hardware category becomes possible, not a serving-ready configuration. Data: pages 1-4, summarized in Table 112345678.

2. Cost per million tokens: what the same benchmark data cost when you price it

Page 5 established how tokens/sec and tokens/sec/user are measured and why the ranking between chips depends on which point on the interactivity curve you read (its §3, §6). This section prices that curve. The two reference data points below come from the same underlying Artificial Analysis dataset, cited independently by two sources that reach the same numbers by different routes1112 — itself worth noting, since a cost figure that only one analyst has ever published is harder to trust than one two independent write-ups converge on.

2.1 Interactive serving vs. offline batch: the ranking flips

At a 30 tokens/sec/user reference speed — a tight, chat-appropriate interactivity target — serving Llama 3.3 70B costs $1.06 per million input-and-output tokens on H100, $2.24 on MI300X, and $5.13 on TPU v6e11. NVIDIA is cheapest by more than 2x over AMD and by roughly 4.8x over TPU v6e at this specific operating point. Move the same three chips to their peak-throughput, offline/batch operating point — no per-user latency constraint, maximum aggregate tokens/sec — and the ranking does not just narrow, it partially inverts: TPU v6e costs $0.62 per million tokens against H100's $0.67-0.69, making TPU v6e the cheapest of the three in this regime, not the most expensive12. The absolute gap between the two regimes for TPU v6e specifically is roughly 8.3x ($5.13 down to $0.62) — a far larger swing than either NVIDIA or AMD shows between the same two operating points, which is exactly the signature of a chip whose peak-throughput hardware is strong but whose small-batch, latency-constrained throughput is comparatively weak (page 3 covers why: v6e's four-port 2D torus and 32 GB per-chip HBM were built for a cost-optimized, scale-out serving/training split, not a low-latency single-request target).

Mechanically, the tight interactivity target is what removes TPU v6e's usual advantage. A 30 tokens/sec/user bound forces a per-step latency budget small enough to cap the batch size any chip can run before an individual request's token blows past that bound — the same batch-size-vs-latency tradeoff page 5 derives in general (its §3.1) and page 3 covers architecturally for v6e specifically (its four ICI ports and 256-chip pod ceiling, sized for cost-efficient scale-out rather than a single request's tail latency). At a small, latency-constrained batch, v6e's narrower per-chip bandwidth-to-compute balance stops mattering less and starts mattering more, because there are fewer concurrent sequences to amortize the fixed cost of streaming weights from HBM across. Remove the latency bound entirely (the offline case) and that same architectural trait becomes irrelevant: a large, unconstrained batch amortizes the fixed weight-streaming cost across enough sequences that v6e's peak compute-per-dollar, not its small-batch bandwidth profile, decides the outcome — and on that axis, per-chip cost and peak-throughput hardware favor v6e.

Cost per million tokens for Llama 3.3 70B, H100 vs MI300X vs TPU v6e, at 30 tok/s/user interactive serving vs. offline/peak-throughput batch $ per million tokens 0 1 2 3 4-5 Interactive, 30 tok/s/user $1.06H100 $2.24MI300X $5.13TPU v6e Offline / peak throughput $0.67-0.69H100 $0.62TPU v6e
Figure 3. Cost per million tokens for Llama 3.3 70B on vLLM. Interactive-serving figures (30 tok/s/user reference speed) for H100, MI300X, and TPU v6e are Artificial Analysis' own measured dataset11; offline/peak-throughput figures for H100 and TPU v6e are the same dataset re-derived by Junyi Hou12 (MI300X's offline figure is not stated in either source and is omitted rather than estimated). No B200 or MI325X data appears in this dataset.

2.2 Beyond Llama 3.3 70B: the same pattern on other model families

Figure 3's precise dollar figures are specific to one model and one dataset, but InferenceMAX's own TCO-per-million-token results on two other model families show the same shape of finding — cost ranking depends on workload, not just chip generation — recur outside Llama 3.3 70B and outside the NVIDIA-vs-Google axis entirely. On GPT-OSS 120B FP4 summarization, MI355X running the open-source vLLM framework already achieves lower total cost of ownership per million tokens than B200 running vLLM, and MI355X still beats B200 running its own vendor-tuned TensorRT-LLM stack below roughly 225 tokens/sec/user of interactivity10 — an AMD-favors-batch, NVIDIA-favors-latency split on a different model, a different precision (FP4 rather than FP8), and a different vendor pairing (MI355X/B200, not MI300X/H100) than Figure 3 shows, reached independently. On DeepSeek 670B MoE FP8, the same dataset shows the opposite generational winner from GPT-OSS: B200 running SGLang delivers 1.5x faster interactivity than MI355X running SGLang at matched TCO10 — NVIDIA's advantage concentrates specifically on this largest, most MoE-heavy workload in InferenceMAX's suite, the model size §7.4 works through directly. Reading these three model families together, no vendor wins uniformly across model size and precision; the crossover threshold and even which vendor is favored shifts model-by-model, which is the strongest evidence in this track's source set that "who's cheaper" is not a property of the silicon alone.

2.3 What this data does and does not cover

Two scope limits matter before using Figure 3 as a procurement input. First, currency: Artificial Analysis' published article dates to June 8, 2025, and covers only DeepSeek R1 and Llama 4 Maverick directly, with the $1.06/$2.24/$5.13 figures specifically for Llama 3.3 70B11 — it predates B200 and MI325X entirely, the same currency gap page 5 already flags (its §6). Second, hardware coverage: this is a three-way NVIDIA-vs-AMD-vs-Google comparison with no AWS Trainium/Inferentia data point at all, consistent with page 4's documented finding that AWS silicon has zero third-party benchmark datapoints of any kind (its §9) — there is no independently measured $/M-token figure for Trainium2 or Inferentia2 anywhere in this track's source set, and none should be assumed.

Reading Figure 3 correctly means applying the same lesson page 5 draws from every crossover chart (its §6): the number that generalizes is not either bar, it is the shape — TPU v6e's cost advantage in batch and disadvantage in interactive serving both trace back to the same architectural fact (page 3's 2D torus, four ICI ports, 32 GB HBM, sized for cost-optimized scale-out rather than single-request latency), so the direction of the flip is the reusable finding, while the exact dollar figures are specific to this one model, this one framework (vLLM), and this one snapshot in time. "Where experts disagree" (D1 and D6, below) returns to this exact dataset to adjudicate two separate but related claims about it.

3. Official cloud pricing: the $/hour dimension

Cost per million tokens (§2) answers "what does this workload cost," but it bundles together a hardware price and an achieved-throughput assumption that are not always disclosed together. The $/hour a cloud actually charges for an instance is a separate, independently checkable number, fetched directly from each vendor's own pricing page or API rather than derived from a benchmark13141516. Table 2 normalizes every instance to a per-GPU or per-chip hourly rate so unlike-sized instances are comparable.

Table 2. Official cloud on-demand pricing, normalized to $/GPU-hour or $/chip-hour, as of 2026-09-17. GCP TPU tiers from Google Cloud's TPU pricing page13; GCP GPU instances from Google Cloud's accelerator pricing page14; AWS Inf2 from its own product page15; AWS p5/p5e/trn1 from Vantage.sh17; AWS p5e from Techzine, citing a documented January 2026 AWS price increase19; AWS trn2 from Spare Cores18; Azure MI300X from the Azure Retail Prices API16.
CloudInstance / SKUGPUs/chipsOn-demand ($/hr, whole instance)$/GPU-or-chip-hrCheapest listed commit
GCPTPU Ironwood (v7)1$12.00$12.003yr: $5.40/chip-hr
GCPTPU v6e (Trillium)1$2.70$2.703yr: $1.22/chip-hr
GCPTPU v5p1$4.20$4.203yr: $1.89/chip-hr
GCPTPU v5e1$1.20$1.203yr: $0.54/chip-hr
GCPa3-highgpu-8g (H100)8$88.49$11.06— (not listed)
GCPa3-ultragpu-8g (H200)8$84.81$10.60— (not listed)
GCPa4-highgpu-8g (B200)8$64.44†$8.063yr CUD: $7.09/GPU-hr
AWSinf2.xlarge (Inferentia2)1$0.76$0.763yr: $0.30/chip-hr
AWSinf2.48xlarge (Inferentia2)12$12.98$1.083yr: $0.43/chip-hr
AWSp5.48xlarge (H100)8$55.04$6.881yr reserved: $2.97/GPU-hr
AWSp5e.48xlarge (H200)8$39.80$4.98— (not published)
AWStrn1.32xlarge (Trainium1)16$21.50$1.34— (not shown here)
AWStrn2.48xlarge (Trainium2)16$13.97-$17.27§$0.87-$1.08— (not published)
AzureND96isr_MI300X_v5 (MI300X)8$48.00-$96.00$6.00-$12.00Low Priority: as low as $1.44/GPU-hr

† a4-highgpu-8g's standard on-demand price is not listed at all; the figure shown is Google's Dynamic Workload Scheduler (DWS) Flex-start rate, the closest published non-committed price14. § trn2.48xlarge's on-demand rate is not server-rendered on any AWS page; the range shown spans three sampled AWS zones from a third-party tracker18. Azure's MI300X price is region-dependent: $48.00/hr in eastus2/westus3 up to $96.00/hr in brazilsouth for the identical SKU16.

3.1 What Table 2 leaves out, and why

AWS's P5, P5e, P5en, Trn1, and Trn2 product pages do not embed a server-rendered pricing table the way the Inf2 page does15; their pricing loads through a JavaScript widget a plain fetch cannot execute. Table 2's AWS GPU/Trainium rows therefore come from third-party trackers rather than directly from aws.amazon.com, cross-checked against each other where possible. That cross-check caught a real inconsistency worth flagging explicitly: the same tracker that supplies this page's p5.48xlarge and trn1.32xlarge figures separately reports p5e.48xlarge at $1.843/hr and trn2.48xlarge at $0.125/hr on-demand17 — both implausibly low (trn2.48xlarge's own spot price on a different tracker runs $1.19-$3.28/hr, higher than that on-demand figure would allow18) and inconsistent with a separately reported, dated AWS price change showing p5e.48xlarge rising from $34.61 to $39.80/hr in January 202619. Table 2 uses the $39.80 figure for p5e and the Spare Cores range for trn2, and excludes the two implausible data points rather than either using them or silently rounding them into a table that looks more precise than the underlying data supports.

No on-demand MI325X or MI355X pricing was found on any of the three clouds' own pricing pages as of this research — Azure lists only the MI300X SKU (ND96isr_MI300X_v5)16, and neither AWS nor GCP offers a managed AMD Instinct instance type in the sources checked here. That is a real gap in the pricing data, not an oversight in this table: as of this writing, Azure is the only one of the three major clouds with a directly-fetchable, official AMD Instinct on-demand price at all. A team choosing between MI300X/MI355X on-demand rental and an NVIDIA equivalent has fewer clouds to price-shop across on the AMD side than on the NVIDIA side, which is itself a cost consideration distinct from anything in §2's per-token figures — single-cloud pricing (as Azure's MI300X currently is) removes the AWS-vs-GCP arbitrage §3.2's worked derivation showed is worth roughly 1.6x on H100, because there is no second official on-demand AMD price to check it against.

3.2 How $/hour relates to $/million-tokens — and why it is not the same number

A $/hour list price and a $/million-tokens figure are connected by exactly one variable: achieved aggregate throughput.

$$ \text{cost per million tokens} = \frac{\text{price per GPU-hour}}{3{,}600 \times \text{aggregate tokens/sec per GPU}} \times 10^{6} $$

Run that formula backward using §2's $1.06-per-million-token H100 figure11 against Table 2's two different H100 on-demand rates, and it implies two very different aggregate-throughput numbers depending purely on which cloud's price you plug in:

$$ \text{AWS: } \frac{6.88}{3{,}600 \times 1.06 \times 10^{-6}} \approx 1{,}803\ \text{tokens/sec/GPU} $$ $$ \text{GCP: } \frac{11.06}{3{,}600 \times 1.06 \times 10^{-6}} \approx 2{,}898\ \text{tokens/sec/GPU} $$

Neither Artificial Analysis nor Junyi Hou states which specific cloud or instance underlies the $1.06 figure1112, so this is not a claim about what they actually assumed — it is a demonstration of the general relationship. A 1.6x spread between AWS's and GCP's own on-demand H100 rates ($6.88 vs. $11.06 per GPU-hour, Table 2) is, by itself, enough to move the implied throughput bar behind an identical cost-per-token figure by 1.6x, with zero difference in the hardware or the software stack. The practical consequence: a $/million-tokens number is underspecified unless it also states (or lets you infer) the $/hour it was priced against, and the same achieved throughput on the same H100 costs an operator a different amount purely as a function of which cloud's list price they are paying — a lever entirely separate from anything about NVIDIA, AMD, or Google's silicon.

3.3 Spot, low-priority, and committed-use: the discount tiers Table 2 only partly shows

On-demand is the ceiling, not the floor. Every cloud in this page's source set publishes at least one discounted tier below its on-demand rate, and the discount depth varies enormously by cloud and by how much notice the operator gives. AWS's spot market prices p5.48xlarge (8×H100) at $20.79/hour against an on-demand rate of $55.04/hour17 — a 62% discount, with the standard spot caveat that AWS can reclaim the instance with two minutes' notice when it needs the capacity elsewhere. Azure's Low Priority tier on the MI300X SKU runs as low as $11.52/hour in one sampled region against an on-demand rate of $57.60/hour in the same region16 — an 80% discount, the deepest of any tier in this page's data, though "Low Priority" carries the same eviction risk as AWS spot under a different name. GCP's B200 instance shows the same pattern at a shallower depth: spot pricing on a4-highgpu-8g runs $39.63/hour against a Dynamic Workload Scheduler flex-start rate of $64.44/hour14, roughly a 39% discount.

These tiers matter most for exactly the workloads §2.1 identifies as TPU v6e's strongest case: offline batch jobs with no per-request latency SLA can tolerate an interrupted, requeued, or migrated job in a way an interactive chat service cannot, so the spot/low-priority discount and the interactive-vs-offline cost-per-token flip point in the same direction — toward routing latency-tolerant traffic to whichever combination of cheap hardware and cheap pricing tier the workload can absorb the risk of. On the committed-use side, the discount is smaller per step but available without any eviction risk at all: GCP's TPU tiers move from on-demand to a 1-year to a 3-year commitment in two discrete steps (for example, TPU v6e's $2.70 on-demand rate drops to $1.89 at 1 year and $1.22 at 3 years13, a cumulative 55% reduction with no interruption risk), and AWS's Inf2 pricing shows the same three-step shape (inf2.48xlarge: $12.98 on-demand, dropping to $5.19/hour at 3 years15, a 60% reduction). A 3-year commitment is a real bet on a specific chip generation not becoming obsolete mid-term — worth weighing against how quickly this track's own next-generation preview (§6) shows the field moving.

3.4 Power efficiency: a real signal, a smaller lever than it looks

Page 5 already measured tokens/sec per provisioned megawatt across two full generations: H100 at 900,000 tokens/sec/MW versus B200 at 2.8 million (roughly 3x), and MI300X at 750,000 versus MI355X at 2.55 million (also roughly 3x) — with B200 also measuring roughly 20% more power-efficient than same-generation MI355X10. It is tempting to read a 20% or 3x power-efficiency gap directly into a $/hour or $/million-token comparison, but page 5's own accounting explains why that temptation overstates the effect: electricity typically makes up under 20% of total cost of ownership, so a 20% tokens/sec-per-MW gap between same-generation NVIDIA and AMD parts translates to under a 4% total-cost delta, with the majority of TCO coming from each vendor's own hardware gross margin rather than the power bill10. None of Table 2's $/hour figures separately break out a power-cost component, and none of §2's $/million-token figures do either — both already bake whatever power cost the underlying instance or chip-hour price reflects into one number, so there is no separate power line item to add on top of this page's own cost data. The efficiency gap is real and worth knowing when comparing two generations at a fixed workload, but it is not the primary lever behind the cost differences Figure 3 and Table 2 already show; those differences trace mostly to list price and achieved throughput, not to the electricity meter.

3.5 $/GB-hour: pricing Table 1's capacity directly

Combining Table 1's capacity column with Table 2's on-demand rate gives a fourth lens distinct from §3.2's throughput-implied $/token: the price of one gigabyte of HBM for one hour, independent of any throughput assumption at all. Dividing each part's cheapest available on-demand $/GPU-or-chip-hour by its Table 1 capacity: H100 runs roughly $0.086/GB-hour ($6.88 ÷ 80 GB17), TPU v6e roughly $0.084/GB-hour ($2.70 ÷ 32 GB13), TPU Ironwood roughly $0.0625/GB-hour ($12.00 ÷ 192 GiB13), MI300X roughly $0.031/GB-hour ($6.00 ÷ 192 GB, Azure's cheapest listed region16), Inferentia2 roughly $0.034/GB-hour ($1.08 ÷ 32 GiB15), and Trainium2 roughly $0.009-0.011/GB-hour ($0.87-$1.08 ÷ 96 GiB18) — the cheapest capacity-per-dollar in this entire page's pricing data, by nearly an order of magnitude over H100 and TPU v6e.

Read this cautiously: $/GB-hour says nothing about compute, interconnect, or achieved throughput, and Trainium2's cheap capacity has to be read alongside page 4's own documented finding that no independent benchmark of any kind exists to confirm what that capacity actually delivers in practice (page 4, §9). It is nonetheless a genuinely useful number for exactly one question this page's other metrics do not directly answer: for a workload that is purely capacity-bound — holding a large model's weights and a generous KV cache, with compute demand modest relative to that capacity — which chip is the cheapest place to park that memory. That is a narrower question than "which chip is cheapest to serve this workload," but it is the question §7.3 and §7.4's rack-scale walk-throughs are implicitly asking when they check capacity headroom, and this metric answers it directly rather than through the $/token detour §3.2 otherwise requires.

4. Cross-vendor scale-up interconnect: current generation and next

Table 1's capacity numbers only tell part of the rack-scale-domain story (§1.3); the fabric that turns separate chips into one coherent domain is the other part, and the four vendors in this track built four structurally different fabrics, not four versions of the same one. NVLink5 and its coming NVLink6 successor are switched fabrics: an NVSwitch chip gives every GPU in the domain a uniform path to every other GPU, which is what lets NVIDIA state one flat aggregate-bandwidth number for the whole rack920. AMD's Infinity Fabric mesh (page 2) is unswitched: every GPU gets a dedicated point-to-point link to every other GPU on the same 8-GPU baseboard, which is why it has no single "domain bandwidth" figure the way a switched fabric does — there is no central switch to sum bandwidth through. Google's ICI (page 3) and AWS's NeuronLink-v3 (page 4) are torus fabrics, wraparound grids rather than either a full mesh or a central switch.

Table 3. Scale-up interconnect by vendor, current generation plus the next-generation row where a vendor has published one. NVLink5/NVLink6 from NVIDIA's GB200 NVL72 and Rubin newsroom pages920; Infinity Fabric and Helios scale-up from AMD's MI350X platform and MI400-series pages521; ICI from Google's TPU7x docs7; NeuronLink-v3 from AWS's Trn2 architecture docs8.
FabricChip familyPer-GPU/chip BWFabric shapeMax coherent domainDomain aggregate BW
NVLink5Blackwell (GB200 NVL72)1.8 TB/sSwitched (NVSwitch)72 GPUs130 TB/s
NVLink6Rubin (Vera Rubin NVL72, next-gen)3.6 TB/sSwitched (NVSwitch)72 GPUs260 TB/s
Infinity FabricCDNA4 (MI350X/MI355X)7 × 153.6 GB/sFull mesh, unswitched8 GPUsnot stated*
Helios scale-upCDNA5 (MI455X, next-gen)3.6 TB/sSwitched, "UALink over Ethernet" beyond one rack72 GPUs260 TB/s†
ICITPU7x (Ironwood)1,200 GBps3D torus9,216 chips (pod); 2,048 (max VM)not stated*
NeuronLink-v3Trainium2 (Trn2 UltraServer)1,024 GB/s intra-instanceTorus (16-chip) + ring (inter-instance)64 chipsnot stated*

* These three fabrics are unswitched (full-mesh or torus/ring), so the source documentation does not publish a single flat "domain aggregate bandwidth" figure the way NVIDIA's switched NVLink fabric does; each gives only the per-link or per-chip rate. † Helios's 260 TB/s rack-scale-up figure is AMD's own stated number21 and also arithmetically consistent with 72 × 3.6 TB/s = 259.2 TB/s, the per-GPU figure on the same page multiplied by rack GPU count.

Two things are worth reading off this table directly. First, NVLink6's per-GPU rate is exactly double NVLink5's (3.6 vs. 1.8 TB/s) and Helios scale-up's per-GPU rate is identical to NVLink6's (3.6 TB/s both) — the two vendors' next-generation racks converge on the same per-GPU scale-up bandwidth number even though, as §6 covers, their compute and memory figures do not reconcile cleanly. Second, every vendor in this track still tops out its tightest-coupled domain at either 64 or 72 chips/GPUs (NVLink5/6's 72, Helios's 72, NeuronLink-v3's 64) — Ironwood's 9,216-chip pod is the outlier, but as §1.3 already flagged, that is a training-scale fabric size, not the ceiling for one coherently-served inference deployment. At the tier that actually matters for a single MoE-class model deployment, the four vendors' domain sizes sit within a factor of roughly 1.1x of each other, not the orders-of-magnitude spread the raw pod-size numbers might suggest.

The fabric shape has a second consequence beyond raw bandwidth: how uniform the latency is across the domain. A switched fabric (NVLink5/6, and Helios's scale-up domain per AMD's own description) gives every GPU-pair roughly the same hop count and latency, which matters for MoE-style expert routing where a token can be sent to any expert on any chip in the domain with no structural reason to prefer nearby chips. A torus (ICI, NeuronLink-v3's intra-instance fabric) gives nearby chips lower latency than chips on the far side of the wraparound, which page 3 already covers as the reason twisted-torus wiring exists at all for asymmetric slice shapes (its §3.3) — a detail that matters more for training-style all-reduce collectives than for inference's more point-to-point KV-cache and expert-routing traffic, but is still a real topology difference this page's bandwidth-only table does not capture. Infinity Fabric's full mesh sits at the opposite extreme from a torus: every pair of the 8 GPUs on one baseboard is exactly one hop apart, the most latency-uniform topology of the four, at the cost of not scaling coherently past 8 GPUs at all without a separate scale-out layer.

5. Software-ecosystem lock-in: what it costs to switch

Every $/hour and $/million-tokens number in this page assumes the serving stack behind it already exists, tuned, on the target hardware. That assumption is not equally cheap to satisfy across the four vendors, and the gap is a real cost even though it does not show up in any price list.

CUDA plus TensorRT-LLM is the default the rest of the industry is implicitly measured against: page 5's own comparison of MLPerf's Closed-division numbers against InferenceMAX's open-source-framework numbers is, among other things, a measurement of how much of NVIDIA's advantage is silicon versus a decade of kernel-level tuning that a team can access either through NVIDIA's own optimized stack or by accepting the smaller open-source-framework gains (page 5, §6). ROCm's gap to that stack is real, measured, and workload-dependent, not a fixed percentage: page 2 documents concrete, dated single-node wins (up to 18x interactivity gains within 30 days on one reasoning model) sitting alongside concrete, current blockers (Kubernetes-orchestrated multi-node inference over AMD's own NIC fabric at "0% parity" with NVIDIA's tooling as of the same assessment), and a newest-hardware bring-up gap that restarts with every CDNA generation rather than compounding — MI455X's ISA is different enough from MI355X's that the software-maturity clock effectively resets (page 2, §6). JAX/XLA on TPU is a different kind of lock-in: it is not a maturity gap in the ROCm sense, it is a structurally different execution model (a compiler that statically schedules onto a systolic array, page 3, §1) that makes a straight CUDA-kernel port meaningless rather than merely slower, and it is why TPU adoption tends to follow whoever already committed to JAX or TensorFlow rather than whoever is comparing raw FLOPS. AWS's Neuron SDK sits closest to CUDA/ROCm's model — a real, general-purpose kernel language (NKI) and compiler stack, not a fixed-function toolchain (page 4, §7) — but it is also the newest and narrowest of the four ecosystems, with a software-managed on-chip memory hierarchy (no hardware cache) that raises the floor on what "get a kernel correct" costs relative to a GPU, and zero independent benchmark of any kind to check any performance claim against (page 4, §9).

The practical question when this axis matters is not "which stack is best" but "what does a switch cost for this specific workload." Switching from CUDA to ROCm for a mainstream model on vLLM or SGLang, at single-node scale, is a due-diligence item today, not an automatic blocker (page 2's own verdict). Switching to TPU/JAX is a rewrite, not a port, and is cheapest when a team is starting a workload from scratch rather than migrating an existing CUDA-based one. Switching to Neuron/NKI carries the highest floor of the four, because there is no large, pre-existing community kernel library to fall back on the way CUDA and (increasingly) ROCm both have. None of this shows up in Table 2's $/hour numbers, and all of it belongs in the same decision alongside them.

A quieter form of the same lock-in sits one level below the serving framework, in the numeric format a model's weights are quantized to. NVIDIA's NVFP4 (page 1) and the OCP Microscaling MXFP4 format AMD's chart cites for MI455X (page 2 and, again, §6's Table 4) are both 4-bit floating-point formats, but they use different block-scaling conventions — NVFP4's default recipe groups 16 elements per scale factor, where the OCP MX standard groups 32 — so a model quantized and calibrated for one is not simply a bit-for-bit drop-in on the other; it needs to be re-quantized and re-validated for accuracy against the new block size. That is a smaller, more mechanical cost than a full serving-stack port, but it is a real one, and it means a model committed to NVFP4-calibrated weights for a Rubin deployment carries a genuine, if modest, re-quantization tax if the same model later needs to move to an MXFP4-native AMD part, or vice versa.

A rough way to hold the two costs in the same frame: a switching decision is worth making when the $/hour or $/million-token gap Table 2 or Figure 3 shows is large enough, and sustained for long enough, to amortize the re-validation work a new stack requires — re-checking every parallelism strategy (TP/EP/DP), quantization format, and speculative-decoding configuration the target workload uses, on the target hardware's specific driver/ISA version, before trusting it in production. That re-validation cost is roughly fixed per model-and-stack combination regardless of deployment size, which is why it favors large, stable deployments over small or short-lived ones: a team running one model at a steady, predictable volume for a year or more can amortize a ROCm or Neuron migration's fixed cost over a large base of $/hour savings, while a team iterating through several model architectures a quarter pays that fixed cost repeatedly and rarely recovers it before the next model swap makes the prior validation work obsolete. None of this track's sources put a dollar figure on the re-validation cost itself — it is a real, load-bearing input to the decision that simply is not published anywhere, unlike every other number on this page.

6. Next-generation preview: Rubin vs. MI455X on paper

Both vendors' next flagships are announced, specified in some detail, and unshipped. AMD states MI455X/Helios volume deployment for the second half of 202621; NVIDIA states the same second-half-2026 window for the Rubin platform, while a separate NVIDIA page describes Vera Rubin as already "in full production"22 — the two NVIDIA statements are not reconciled on NVIDIA's own pages, a detail page 1 already flags (its §6.3). Whatever the exact ship date, neither part has been independently benchmarked, and every figure in this section is a vendor's own peak-theoretical projection.

AMD's own MI400-series product page carries a direct head-to-head table against "NVIDIA Vera Rubin," the only place either vendor publishes this specific comparison in table form21. Table 4 reproduces it, with one addition Table 2 of page 2 did not have reason to include: a column showing what NVIDIA's own materials state for the same part.

Table 4. AMD's own MI455X-vs-Rubin comparison21, alongside NVIDIA's own separately published Rubin figures20. The two "Rubin" columns are not the same number on FP4 throughput, and NVIDIA states no memory-bandwidth figure directly comparable to AMD's cited 22.0 TB/s at all.
MetricAMD MI455X (AMD's own)NVIDIA Rubin, per AMD's chartNVIDIA Rubin, per NVIDIA's own materials
FP4 / OCP MXFP4 (peak PFLOPS)403550*
FP8 | FP6 (peak PFLOPS)2018not stated
FP16 | BF16 (peak PFLOPS)54not stated
Memory capacity432 GB288 GBnot stated
Memory bandwidth23.3 TB/s22.0 TB/snot stated
NVLink/scale-up per GPU3.6 TB/snot in AMD's table3.6 TB/s

* NVIDIA's own newsroom release states Rubin GPU delivers "50 petaflops of NVFP4 compute for AI inference"20 — higher than the 35 PFLOPS AMD's own chart attributes to the same part, and NVIDIA does not state whether its 50-PFLOPS figure is dense or sparse, or what token-length/latency configuration "for AI inference" refers to.

FP4 peak PFLOPS: AMD MI455X's own figure, NVIDIA Rubin per AMD's comparison chart, and NVIDIA Rubin per NVIDIA's own newsroom release — three numbers, two of them for the same NVIDIA part and neither vendor stating a common basis peak FP4 PFLOPS 0 10 20 30 40-50 40 MI455X (AMD's own figure) 35 Rubin, per AMD (AMD's comparison chart) 50 Rubin, per NVIDIA (NVIDIA's own release) Same NVIDIA part, two unreconciled figures
Figure 4. Three peak-FP4-PFLOPS figures, only two of them describing NVIDIA's own part. AMD's own chart states Rubin at 35 PFLOPS21; NVIDIA's own newsroom release states 50 PFLOPS for the same GPU20. Both AMD's 40 PFLOPS MI455X figure and NVIDIA's 50 PFLOPS Rubin figure are the vendor's own unaudited, pre-shipment projection.

Two things are independently checkable even though the headline numbers are not reconciled. First, both parts are pre-shipment on both sides of the comparison: MI455X/Helios and Rubin/Vera Rubin NVL72 share the same "second half of 2026" volume-deployment language2120, so Table 4 compares vendor projection against vendor projection, not measured hardware against measured hardware, on either side. Second, AMD's own capacity and bandwidth claims for its own part are internally consistent with the rest of this track: 432 GB and 23.3 TB/s match MI455X's own product-page spec exactly21, and the increases over MI355X (1.5x capacity, 2.9x bandwidth, page 2 §7.1) are AMD's own generation-over-generation numbers, not part of the disputed cross-vendor row. What is unresolved is specifically the "NVIDIA Vera Rubin" column of AMD's own chart: neither AMD's footnotes nor NVIDIA's own materials state whether the 35-vs-50-PFLOPS gap reflects a dense-vs-sparse difference, a training-vs-inference-peak difference, or two different accounting conventions entirely — and the same is true of memory capacity and bandwidth, where AMD's chart states numbers for Rubin that do not appear anywhere in NVIDIA's own fetched materials at all. "Where experts disagree" (D5, below) treats this formally as what it is: an open disagreement this page cannot close, not a discrepancy to average away.

For a buyer making a decision today, neither Rubin nor MI455X is purchasable, so the practical question is not "which one" but "how much weight should either vendor's pre-shipment claim carry in a roadmap decision." The answer this track's own pattern suggests: not much, on its own. Every vendor-projected multiplier this track has been able to check against independent measurement after shipment — Blackwell's own "30x" claim (page 1, §4), AMD's peak-theoretical-vs-measured GEMM gap (D2, above) — has turned out smaller or more conditional than the launch-time framing implied. There is no reason specific to Rubin or MI455X to expect either vendor's current claims to behave differently once independent benchmarks exist; the safer planning assumption is that both parts will be meaningfully faster than their current-generation predecessors (both vendors' own generation-over-generation numbers, not the disputed cross-vendor row, are internally consistent and corroborated by this track's own pattern of real if smaller-than-claimed gains), without committing to either vendor's specific cross-vendor multiplier until independent data exists.

7. Worked decision walk-throughs by model-size band

Combining §1's memory-fit thresholds, §2's cost-per-token data, and §3's cloud pricing produces a short list, not a single answer, at every model size in this track's range. The four walk-throughs below name the sourced number behind each choice; none of them introduces a fact not already established on pages 1-5 or in §1-§6 above.

7.1 7B: single chip, optimize purely for $/hour

At roughly 7 GB of FP8 weights, a 7B model fits with room to spare on every chip in Table 1, TPU v6e's 32 GB included (§1.2). Once every candidate chip clears the capacity bar comfortably, the deciding variable collapses to Table 2's $/hour column: AWS's inf2.xlarge (1 Inferentia2 chip) at $0.76/hour on-demand or $0.30/hour on a 3-year commit15, and GCP's TPU v5e at $1.20/hour on-demand or $0.54/hour on a 3-year commit13, are the two cheapest listed rates in this page's entire pricing table for any accelerator. Neither chip's raw compute matters much here — a 7B model rarely saturates even a modest accelerator's FLOPS at realistic batch sizes — so the defensible choice is whichever of the two has the lower operational friction for the team's existing stack (Neuron SDK vs. JAX/XLA, §5), not a performance argument.

7.2 70B: interactive chat vs. offline batch route to different hardware

A 70B model fits single-chip on H200, MI300X, MI325X, MI350X/MI355X, and Trainium2, is tight on H100, and needs sharding on TPU v6e (§1.2). Say the traffic is a chat product targeting the same 30 tokens/sec/user interactivity Artificial Analysis measured against, at 50 billion tokens/month of combined input and output — a mid-size production volume. Applying §2's interactive-serving figures directly:

$$ 50{,}000 \times \$1.06 = \$53{,}000\text{/month (H100)} \qquad 50{,}000 \times \$2.24 = \$112{,}000\text{/month (MI300X)} \qquad 50{,}000 \times \$5.13 = \$256{,}500\text{/month (TPU v6e)} $$

(multiplying each $/million-tokens figure by 50,000 million tokens11). At this specific interactivity target, H100 is the defensible default and TPU v6e is nearly 5x more expensive. Now add a second, separate workload on the same model — a nightly document-summarization batch job, no per-user latency constraint, 20 billion tokens/month:

$$ 20{,}000 \times \$0.62 = \$12{,}400\text{/month (TPU v6e, offline)} \qquad 20{,}000 \times 0.68 \approx \$13{,}600\text{/month (H100, offline)} $$

(using the midpoint of H100's stated $0.67-$0.69 offline range12). Here TPU v6e is marginally cheaper, not 5x more expensive — the ranking inverted entirely because the workload's latency requirement changed, not because the hardware changed. The defensible two-part answer for this scenario is H100 for the interactive chat traffic and TPU v6e for the nightly batch job, not a single hardware choice for the whole 70B deployment — exactly the workload-splitting logic §2.3 and page 5's own crossover-chart lesson (its §6) predict.

Translate the $53,000/month H100 figure into a GPU count using §3.2's $/hour-to-$/token relationship, run in the direction that answers "how much hardware do I actually provision." AWS's p5.48xlarge (8×H100) costs $55.04/hour17, or, over an average 30.44-day month:

$$ \$55.04 \times 24 \times 30.44 \approx \$40{,}200\text{/month for one 8-GPU node} $$

Against the $53,000/month interactive-serving bill implied by §2.1's cost-per-token figure, that is $53{,}000 / \$40{,}200 \approx 1.3$ nodes' worth of continuously-provisioned H100 capacity, or roughly 10-11 H100 GPUs running around the clock at the 30 tokens/sec/user target — a single p5.48xlarge node plus a few spare GPUs on a second, partially-loaded node, not a large cluster. This is a sanity check, not an independent measurement: it confirms the §2.1 cost figure and the Table 2 AWS on-demand rate are consistent with each other in order of magnitude, which is the kind of cross-check worth running before committing to either number as a planning input on its own.

7.3 405B: multi-GPU is mandatory, and the choice is TP degree vs. rack fraction

At roughly 405 GB of FP8 weights, no single chip in Table 1 holds this model (§1.2) — the smallest workable configuration on current-generation hardware is two-way tensor parallelism on the largest-memory GPUs, MI350X/MI355X, giving 576 GB of pooled Infinity Fabric-mesh capacity (page 2, §4) for 405 GB of weights plus KV cache headroom. On the NVIDIA side, an 8-GPU H100 node — AWS's p5.48xlarge, 640 GB of pooled HBM3 at $55.04/hour on-demand17 — comfortably holds the same model with TP8 and leaves the most capacity margin of any single-node option in this page's pricing table. The premium option is a fraction of a GB200 NVL72 rack, matching the exact benchmark configuration NVIDIA's own MLPerf Inference v5.0 submission used for this model size, where GB200 NVL72 measured a 3.4x per-GPU throughput gain over an 8-GPU H200 system (page 1, §4.2) — worth the added cost specifically when traffic volume and latency requirements justify dedicating rack-scale NVLink bandwidth to this one model, not as a default.

Capacity headroom differs meaningfully between the two current-generation options, worth checking explicitly rather than assuming either is comfortable. Two MI350X/MI355X GPUs' 576 GB pool minus 405 GB of weights leaves 171 GB for KV cache and activations; eight H100 GPUs' 640 GB pool minus the same 405 GB leaves 235 GB. Both are workable, but the TP2 MI355X configuration runs closer to its ceiling — worth confirming against the target context length and concurrency before committing, since page 5's own roofline arithmetic (its §4.3) shows KV cache bytes scale directly with batch size and can dominate total memory traffic well before weight bytes stop mattering.

7.4 670B+ MoE: rack-scale domains, and the AMD/NVIDIA split from §2 recurs

A DeepSeek-scale, roughly 670B-parameter mixture-of-experts model needs a rack-scale coherent domain (§1.3), not a wider TP degree on the same node. InferenceMAX's own DeepSeek 670B MoE FP8 results give the workload-dependent split directly: B200 running SGLang delivers 1.5x faster interactivity than MI355X running SGLang at matched total cost of ownership, and GB200 NVL72 decisively outperforms every single-node 8-GPU alternative below roughly 90 tokens/sec/user interactivity, while a single-node B200 system running TensorRT-LLM overtakes it above that threshold10. The defensible short list at this model size is therefore interactivity-dependent in the same way §7.2's 70B example was cost-dependent: GB200 NVL72 (or a comparable rack-scale NVLink domain) for latency-sensitive, low-concurrency serving of this model; a single dense B200 node for higher-interactivity traffic that does not need the full rack's pooled batch capacity; and an MI355X SGLang cluster specifically where the TCO advantage InferenceMAX measured outweighs the multi-node ROCm-maturity caveats §5 already flags for exactly this kind of distributed deployment.

Capacity headroom at this size looks generous on paper and needs the same explicit check as §7.3's. GB200 NVL72's 13.4 TB pool could in principle hold roughly 20 replicas of a 670 GB model's weights (13.4 TB ÷ 670 GB) with zero KV cache headroom — not a real target, but useful for calibrating how much of the rack's capacity a production deployment (a handful of replicas, plus a substantial KV-cache and activation budget per replica for long-context MoE traffic) actually consumes relative to the ceiling. The Trn2 UltraServer's smaller 6,144 GiB (roughly 6.4 TB) pool supports proportionally fewer replicas at the same headroom ratio, consistent with §1.3's point that the two domains differ in size by roughly 2x, not in kind.

Decision flowchart: model size, traffic pattern, and budget narrow to a hardware shortlist Model size 7B/70B/405B/670B+ Table 1: which chips hold it single-chip? (§1) TP degree or rack fraction if not (§1.2-1.3) Traffic pattern interactive / batch Table/Fig 3: $/M tokens at your tok/s/user (§2) Table 2: $/hour by cloud and commit (§3) 2-3 hardware choices (§7)
Figure 5. The same three inputs from Figure 1, traced through the specific tables and sections that answer each one, ending at a short list of defensible hardware choices rather than one "best" chip — every walk-through in §7 follows this same path.

7.5 Sizes between the four bands

A real model rarely lands exactly on 7B, 70B, 405B, or 670B. A 13B or 34B model behaves like §7.1's 7B case — every chip in Table 1 holds it single-chip, so the deciding variable is still $/hour, not memory fit. A model in the 100-200B range (a large dense model, or a small-to-mid MoE with a comparable total weight footprint) sits inside §7.2's band: single-chip on the larger-memory parts (MI325X, MI350X/MI355X), tight-to-impossible on H100 and TPU v6e, and the interactive-vs-offline cost split still applies with the same mechanism, even though the exact dollar figures in Figure 3 were measured on 70B specifically and do not transfer numerically to a different parameter count. A model between 200B and 400B falls into §7.3's multi-GPU territory earlier than a strict 405B threshold would suggest, because KV cache and activation headroom (§7.3's own capacity check) start mattering well before the weight footprint alone crosses a single chip's ceiling. The general rule this page's four bands illustrate, not replace: compute the weight footprint at your actual target precision, compare it against Table 1's capacities with real headroom for KV cache and batch (not zero, as this section's illustrative arithmetic assumes), and only then apply §2's cost data and §3's pricing at whatever traffic pattern the deployment actually expects — the four worked bands are checkpoints on that continuum, not the only sizes worth analyzing this way.

Where experts disagree

This page owns four of this track's six sourced disagreements — more than any other page in it. Two are close cousins built from the same underlying source (Introl's infrastructure-decision blog) but adjudicate different claims within it: D1 asks whether the blog's general "TPU is broadly cheaper" framing holds up, and D6 asks specifically whether its one stated dollar figure ($0.39/chip-hour) is real — a general claim can be wrong in its framing while containing a specific number that is separately, independently checkable, which is exactly what these two disagreements found, and treating them as one would have blurred a genuinely useful distinction: "TPU cost claims need a workload qualifier" (D1's finding) is a different, more defensible statement than "this specific number is unsourced and contradicted by the vendor's own pricing page" (D6's finding). The remaining two disagreements are unrelated to each other and to the TPU pair: one is a genuine architecture-fit split that holds up under three independent measurement methods (D2); the other is a paper-spec mismatch neither vendor has reconciled and that this page cannot force to a single number either (D5).

Is Google TPU meaningfully cheaper than NVIDIA GPUs for inference?

TPU delivers a broad, workload-general price-performance advantage

Introl's infrastructure decision-framework blog23

Introl states TPU v6e delivers roughly 4x better price-performance than H100 for LLM training, recommendation systems, and batch inference, framed as a general rule for "inference-dominated workloads" rather than a narrow, qualified case23.

  • Correctly identifies that TPU economics do favor Google's hardware in at least one real regime (offline/batch, §2.1 confirms this).
  • Matches Google's own broader positioning of v6e as the cost-optimized generation (page 3).
  • States the advantage as general rather than tying it to a specific interactivity regime, which §2.1's own data shows it is not.
  • The specific multiplier and price point underlying the claim do not hold up independently (D6, below).

The advantage is real but strictly workload-dependent, and inverts at interactive latency

Junyi Hou's independent fact-check, re-deriving Artificial Analysis' own dataset1211

Using the same underlying cost-per-token dataset §2.1 already presents in full: at peak-throughput/offline serving, TPU v6e costs $0.62 per million tokens against H100's $0.67-0.69, roughly competitive; at 30 tokens/sec/user interactive serving, TPU v6e costs $5.13 against H100's $1.06, a roughly 5x gap favoring NVIDIA12.

  • Directly re-derived from a named, checkable dataset rather than an unstated source.
  • Explains the mechanism (page 3's torus/HBM-capacity design intent) rather than asserting a bare multiplier.
  • The dataset itself is a June 2025 snapshot covering only two models directly, predating B200 and MI325X (§2.3).
  • Does not cover AWS Trainium at all, so it cannot speak to a three-way TPU/GPU/Trainium comparison.
Assessment

The workload-split position is the better-supported one: it is reconstructed from a named dataset two independent write-ups cite consistently1112, and it explains the direction of the effect from architecture (page 3) rather than asserting a bare ratio. Read any "TPU is N times cheaper" claim as incomplete unless it states whether N was measured at peak throughput or at a specific tok/s/user target — the true ratio runs from roughly parity to roughly 5x across that one axis alone on the one dataset available. What would settle this further is a current-generation (B200/MI325X-era) repeat of the same measurement, which does not yet exist in this track's source set. In the meantime, the practical rule this page's own data supports is narrower than either position's headline: ask which of your workload's operating points — a fixed tok/s/user SLA, or an unconstrained batch job — the number you were quoted was measured at, and treat any TPU-vs-GPU cost claim that does not answer that question as incomplete rather than wrong.

Does AMD MI300X/MI325X beat NVIDIA H100/H200 for LLM inference?

AMD wins on large models, high concurrency, and absolute throughput

SemiAnalysis' GEMM/serving benchmark and InferenceMAX2410, corroborated by Artificial Analysis' rented-instance testing11

MI325X consistently outperforms H200 and H100 on Llama 3 405B FP8 at realistic batch sizes, and MI300X reaches higher peak aggregate throughput than NVIDIA parts at high concurrency on both DeepSeek R1 (~4,100 vs. ~3,250 tokens/sec at peak) and Llama 4 Maverick (~8,500 vs. ~6,500 tokens/sec)2411. On LLaMA 3.3 70B, InferenceMAX finds MI300X ahead specifically below roughly 20-30 tokens/sec/user, tracing directly to its larger single-chip HBM capacity (§1.2)10.

  • Converges across three independent measurement methods — a GEMM microbenchmark, an open-source-framework serving benchmark, and rented-cloud-instance testing.
  • Grounded in a checkable architectural cause (HBM capacity, Table 1) rather than an unexplained result.
  • "Wins" here means aggregate throughput or perf/$ at high concurrency, not per-request latency, where the next position holds.
  • Newest data (InferenceMAX) covers MI355X/B200; the GEMM and Artificial Analysis figures are from earlier-generation MI300X/H100/H200 comparisons.

NVIDIA wins on latency-sensitive serving and on raw measured throughput per GPU

The same SemiAnalysis GEMM benchmark and Artificial Analysis testing, read at their other operating point2411

Independently measured GEMM throughput runs roughly 14% (BF16) to 22% (FP8) higher on H100/H200 than on MI300X24 — a chip-level, framework-independent gap that exists before any serving stack enters the picture. At fixed moderate concurrency (16 concurrent queries), Artificial Analysis measures H200 ahead of MI300X on both per-query and system throughput on DeepSeek R1 (~45 vs. ~35 tokens/sec/query) and Llama 4 Maverick (H100 ahead, ~90 vs. ~70 tokens/sec/query)11.

  • A GEMM-level gap is architecture-and-driver-level evidence, not dependent on any one serving framework's tuning.
  • Consistent with NVIDIA's more mature TRT-LLM/vLLM kernel optimization on Hopper (page 5, §6).
  • The 14-22% GEMM gap does not, by itself, explain the larger throughput gaps at low concurrency — serving-stack and batching effects add to it.
  • Does not cover B200/MI355X-generation hardware directly.
Assessment

Both positions are correct, describing different operating points on the same tokens/sec-vs-tokens/sec/user curve page 5 already establishes as the reusable object (its §3). NVIDIA wins low-concurrency, latency-sensitive serving and any workload dominated by raw measured GEMM throughput; AMD wins large-model, high-concurrency, throughput-optimized serving where its larger HBM capacity lets it run bigger batches per chip. There is no single "who wins" answer independent of where your traffic sits on that curve — which is exactly §7.4's DeepSeek 670B MoE walk-through's own conclusion, not a separate finding. What would sharpen this further is an InferenceMAX-methodology run at current-generation parts (B200 vs. MI355X) specifically at the 16-32-concurrency band Artificial Analysis tested, which does not yet exist in the public record. For procurement purposes, the concurrency band a deployment actually expects to run at — not the vendor's own headline benchmark configuration — is the only input that makes this disagreement resolvable for a specific workload; both this page's §7.2 and §7.4 walk-throughs apply exactly that filter rather than picking a side in the abstract.

AMD vs NVIDIA crossover by concurrency: NVIDIA ahead at low concurrency on GEMM and moderate-concurrency serving, AMD ahead at high concurrency on aggregate throughput concurrency / batch size → measured advantage parity NVIDIA ahead GEMM 14-22%, 16-concurrency serving (H200/H100) AMD ahead peak-concurrency aggregate throughput (MI300X) 16 concurrent 750-1,000 concurrent
Figure 6. Schematic of the AMD-vs-NVIDIA crossover pattern D2 adjudicates: NVIDIA measures ahead at low concurrency (14-22% GEMM gap, and ahead on both per-query and system throughput at 16 concurrent queries), MI300X measures ahead in absolute aggregate throughput at high concurrency (roughly 4,100 vs. 3,250 tokens/sec at 750 concurrent queries on DeepSeek R1)2411. Curve shape is illustrative of the reported endpoints; the exact crossover concurrency was not published.

Whose next-gen flagship wins on paper: NVIDIA Rubin or AMD MI455X?

AMD's own chart shows MI455X ahead on every listed metric

AMD's MI400-series product page21

Per Table 4, MI455X beats the "NVIDIA Vera Rubin" figures on AMD's own chart on FP4 (+15%), FP8/FP6 (+11%), FP16/BF16 (+25%), memory capacity (+50%), and memory bandwidth (+6%)21, with AMD's own footnotes stating the percentage advantages explicitly.

  • A direct, disclosed, side-by-side comparison — the only one either vendor publishes.
  • AMD's own MI455X figures are internally consistent with the rest of MI455X's product-page spec (§6).
  • The "NVIDIA Vera Rubin" column is AMD's characterization of a competitor's unreleased part, not an NVIDIA-published number.
  • AMD does not state the basis (dense/sparse, training/inference peak) for its Rubin figures at all.

NVIDIA's own materials state a higher figure for the same part than AMD's chart shows

NVIDIA's Rubin newsroom release and Vera Rubin platform page2022

NVIDIA states Rubin GPU delivers 50 petaflops of NVFP4 compute for AI inference20 — 43% higher than the 35 PFLOPS AMD's chart attributes to the same part — and NVIDIA's own fetched pages state no HBM4 bandwidth figure at all, making AMD's cited 22.0 TB/s uncorroborated by NVIDIA's own materials2022.

  • Comes directly from the vendor whose part is being described, not a competitor's characterization.
  • The 50-PFLOPS figure is stated cleanly, without AMD's chart's ambiguity about which metric it represents.
  • NVIDIA does not state whether 50 PFLOPS is dense or sparse, or the precise workload "for AI inference" refers to — exactly the same disclosure gap AMD's chart has.
  • NVIDIA's own pages are internally inconsistent about Rubin's production status (page 1, §6.3), which does not build confidence in the precision of its other Rubin claims.
Assessment

This one does not resolve, and forcing it to resolve would misrepresent the evidence. The 35-vs-50-PFLOPS gap could be a dense-vs-sparse difference, a training-vs-inference-peak difference, or two vendors using genuinely different accounting conventions — neither AMD's chart nor NVIDIA's own materials states which, and no independent source has measured either part, because neither has shipped. The honest position is that AMD's chart shows MI455X ahead of AMD's own characterization of Rubin, and NVIDIA's own characterization of Rubin is itself higher than what AMD attributes to it — two different, non-comparable "Rubin" numbers exist in this track's source set, and treating either as settled would be a worse error than leaving the question open. What would settle it: both parts shipping and an independent InferenceMAX- or MLPerf-style benchmark run at a matched precision and workload — not available for either part as of this research. Until then, the only planning-safe use of Table 4 is directional and qualitative, not numeric: both vendors are converging on a similar per-GPU scale-up bandwidth (§4's 3.6 TB/s finding on both NVLink6 and Helios), both are targeting the same 2H 2026 shipping window, and both are claiming a meaningful generational jump over their own current flagship — three real, load-bearing facts that do not depend on resolving the 35-vs-50-PFLOPS mismatch at all.

Is Google TPU actually cheap at the $0.39/chip-hour Introl claimed?

TPU v6e committed-use pricing runs roughly $0.39/chip-hour

Introl's infrastructure decision-framework blog23

Introl's own numeric-figures table states TPU v6e pricing at $0.39/chip-hour, underpinning its separately stated "4x better price-performance than H100" headline23.

  • Presented as a specific, checkable dollar figure rather than a vague claim.
  • No cited source for the $0.39 figure itself within the blog post.
  • Directly contradicted by Google's own live pricing page (below) — no listed tier is within 3x of this number.

Google's own pricing page shows no rate anywhere near $0.39/chip-hour

Google Cloud's own TPU pricing page13, corroborating Junyi Hou's independent conclusion12

Trillium's (v6e) cheapest listed rate anywhere on Google's own pricing table — a 3-year commitment — is $1.22/chip-hour, more than 3x Introl's figure; the 1-year rate is $1.89/chip-hour and on-demand is $2.70/chip-hour13 (Table 2). No tier at any commitment length, on any TPU generation this page tracks, comes close to $0.39/chip-hour.

  • Fetched directly from the vendor's own, live, official pricing page — the highest-authority source available for this question.
  • Independently corroborates a separate analyst's conclusion that the $0.39 figure is unsourced12, reached by a different method (re-deriving cost-per-token from measured throughput data, not checking a price list).
  • Google's pricing page shows list price, which real customers can sometimes negotiate below at very large committed volume — though a discount of more than 3x off the lowest published tier would be unusual and is not evidenced anywhere in this track's sources.
TPU v6e (Trillium) listed pricing tiers versus Introl's claimed $0.39/chip-hour committed-use rate $ per chip-hour 0 1 2 3 $2.70 On-demand $1.89 1-year commit $1.22 3-year commit $0.39 Introl's claim (unsourced)
Figure 7. Every listed TPU v6e (Trillium) rate on Google's own pricing page, from on-demand down to the deepest published 3-year commitment13, against the $0.39/chip-hour figure Introl's blog states without a cited source23. The claimed rate sits more than 3x below Google's own cheapest published tier.
Assessment

This is the one disagreement on this page with a clean resolution: the $0.39/chip-hour figure does not match any rate on Google's own pricing page at any commitment length, on any generation13, and a second, independently-derived analysis reaches the same conclusion by re-deriving cost-per-token from measured throughput rather than checking a price list12 — two different methods landing on the same verdict. Treat the $0.39 figure as unsupported. This does not mean TPU v6e is never the cheaper option (§2.1's offline-batch data shows it genuinely can be, at $0.62 per million tokens against H100's $0.67-0.69) — it means the specific number used to justify that conclusion in Introl's post is wrong, which is a narrower and more useful finding than a blanket "TPU is/isn't cheap" verdict.

What is changing

Three things in this page are near-certain to be stale within a year, in ways worth naming rather than leaving implicit. First, every price in Table 2: AWS's own documented 15% January 2026 increase on p5e/p5en pricing19 is evidence that cloud accelerator pricing moves on the order of months, not years, and every $/hour figure on this page should be re-checked against each vendor's live pricing page before it anchors a procurement decision rather than trusted as a fixed reference point. Second, both next-generation flagships in §6: Rubin/Vera Rubin NVL72 and MI455X/Helios both target second-half-2026 volume deployment2021, which means the entire "next-gen preview" section and the D5 disagreement it feeds are likely to be resolved by an actual independent benchmark within roughly a year of this page's as-of date — at which point Table 4's vendor-vs-vendor projection column should be replaced with measured InferenceMAX- or MLPerf-style data rather than read as current.

Third, and most directly relevant to this track's own coverage gaps: page 4 documents that InferenceMAX (now InferenceX) promised Google TPU and AWS Trainium backend support "within two months" of its October 2025 launch, and neither had landed as of this research, nearly a year later, with TPU listed as "coming soon" on the project's own roadmap and Trainium absent from that roadmap entirely10. If and when that coverage arrives, it would directly upgrade §2's cost-per-token picture beyond the single June-2025 Artificial Analysis snapshot this page currently relies on for all three of its NVIDIA/AMD/Google comparisons, and would, for the first time, give AWS Trainium/Inferentia an independently-measured throughput or cost figure to sit alongside the vendor-only numbers this entire track has had to use for AWS silicon throughout pages 4 and 6.

References

  1. officialNVIDIA. H100 GPU (product page). 2026. https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-09-17. Used for: H100 SXM 80GB HBM3 capacity and 3.35TB/s bandwidth (Table 1, §1).
  2. officialNVIDIA. H200 GPU (product page). 2026. https://www.nvidia.com/en-us/data-center/h200/. Accessed 2026-09-17. Used for: H200 SXM 141GB HBM3e capacity and 4.8TB/s bandwidth (Table 1, §1).
  3. officialAMD. AMD Instinct MI300X Accelerator — Data Sheet. GD-83, 2025. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf. Accessed 2026-09-17. Used for: MI300X 192GB HBM3 capacity and 5.3TB/s bandwidth (Table 1, §1).
  4. officialAMD. AMD Instinct MI325X Accelerator — Data Sheet. GD-83, 2025. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/instinct-mi325x-datasheet.pdf. Accessed 2026-09-17. Used for: MI325X 256GB HBM3e capacity and 6.0TB/s bandwidth (Table 1, §1).
  5. officialAMD. AMD Instinct MI350X Platform — Data Sheet. GD-83, PID#253461432A, 2025. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/amd-instinct-mi350x-platform-brochure.pdf. Accessed 2026-09-17. Used for: MI350X/MI355X 288GB HBM3e capacity and 8.0TB/s bandwidth (Table 1, §1), 153.6GB/s per-link Infinity Fabric bandwidth (Table 3, §4).
  6. officialGoogle Cloud. TPU v6e (Trillium). Live documentation. https://docs.cloud.google.com/tpu/docs/v6e. Accessed 2026-09-17. Used for: TPU v6e 32GB HBM capacity and 1,638GBps bandwidth (Table 1, §1).
  7. officialGoogle Cloud. TPU7x (Ironwood). Live documentation. https://docs.cloud.google.com/tpu/docs/tpu7x. Accessed 2026-09-17. Used for: Ironwood 192GiB HBM capacity and 7,380GBps bandwidth (Table 1, §1), 1,200GBps ICI and 9,216-chip pod vs. 2,048-chip max VM topology (§1.3, Table 3).
  8. officialAWS. Amazon EC2 Trn2 Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-hardware/trn2-arch.html. Accessed 2026-09-17. Used for: Trainium2 single-chip 96GiB/2.9TB/s (Table 1, §1), Trn2 UltraServer 6,144GiB/185.6TB/s (§1.3), NeuronLink-v3 1,024GB/s/chip intra-instance and 64-chip domain (Table 3, §4).
  9. officialNVIDIA. GB200 NVL72 (product page). 2026. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-09-17. Used for: GB200 NVL72 rack 13.4TB HBM3e/72 GPUs (§1.3), NVLink5 130TB/s domain and 1.8TB/s per-GPU (Table 3, §4).
  10. analysisSemiAnalysis. InferenceMAX: Open Source Inference Benchmarking. October 2025. https://newsletter.semianalysis.com/p/inferencemax-open-source-inference. Accessed 2026-09-17. Used for: MI300X capacity-driven tensor-parallelism-degree-1 advantage on LLaMA 3.3 70B (§1.2), DeepSeek 670B MoE FP8 as reference model size (§1.2), B200-vs-MI355X SGLang TCO and GB200-NVL72-vs-B200 90tok/s/user crossover (§7.4), MI300X low-interactivity advantage (D2).
  11. analysisArtificial Analysis. Independent Performance Analysis of Leading GPUs. 8 June 2025. https://artificialanalysis.ai/articles/independent-analysis-of-leading-gpus-amd-nvidia. Accessed 2026-09-17. Used for: H100 $1.06/MI300X $2.24/TPU v6e $5.13 per million tokens at 30 tok/s/user (Figure 3, §2.1, §3.2, §7.2), 16-concurrency and peak-concurrency DeepSeek R1/Llama 4 Maverick throughput figures (D2).
  12. analysisJunyi Hou. Debunking introl and ainewshub: "TPU is 4x Cheaper than GPU" is an AI Hallucination. 2026. https://www.junyi.dev/en/posts/tpu-tco/. Accessed 2026-09-17. Used for: TPU v6e $0.62 vs. H100 $0.67-0.69 offline/batch cost per million tokens (Figure 3, §2.1, §7.2), the workload-dependent-not-blanket conclusion (D1, D6).
  13. officialGoogle Cloud. Cloud TPU pricing. Live pricing page. https://cloud.google.com/tpu/pricing. Accessed 2026-09-17. Used for: on-demand/1yr/3yr per-chip-hour pricing for Ironwood, Trillium (v6e), v5p, v5e (Table 2, §3), the $1.22/chip-hour 3-year Trillium rate that resolves D6.
  14. officialGoogle Cloud. GPU pricing / Accelerator-optimized VM Pricing. Live pricing page. https://cloud.google.com/products/compute/gpus-pricing. Accessed 2026-09-17. Used for: a3-highgpu-8g/a3-ultragpu-8g/a4-highgpu-8g on-demand, DWS, and CUD pricing (Table 2, §3), the $/GPU-hour derivation in §3.2.
  15. officialAWS. Amazon EC2 Inf2 Instances (product page, pricing table). 2026. https://aws.amazon.com/ec2/instance-types/inf2/. Accessed 2026-09-17. Used for: inf2.xlarge and inf2.48xlarge on-demand/1yr/3yr pricing (Table 2, §3, §7.1), the note that Inf2 is the only AWS accelerator family with server-rendered pricing.
  16. officialMicrosoft Azure. Azure Retail Prices API — Standard_ND96isr_MI300X_v5. Live API. https://prices.azure.com/api/retail/prices?$filter=armSkuName eq 'Standard_ND96isr_MI300X_v5'. Accessed 2026-09-17. Used for: ND96isr_MI300X_v5 regional on-demand pricing, $48.00-$96.00/hr for 8xMI300X, and Low Priority rate (Table 2, §3).
  17. analysisVantage. instances.vantage.sh — AWS EC2 instance pricing pages. Live pages. https://instances.vantage.sh/aws/ec2/p5.48xlarge. Accessed 2026-09-17. Used for: p5.48xlarge and trn1.32xlarge on-demand pricing (Table 2, §3, §7.3), and the internally-inconsistent p5e/trn2 figures flagged and excluded in §3.1.
  18. analysisSpare Cores. trn2.48xlarge live pricing snapshot. sparecores.com. https://sparecores.com/server/aws/trn2.48xlarge. Accessed 2026-09-17. Used for: trn2.48xlarge on-demand pricing range across sampled zones (Table 2, §3), used to cross-check and exclude Vantage's inconsistent trn2 figure.
  19. blogTechzine (Sebastian Moss, citing DataCenterDynamics). AWS increases EC2 Capacity Block prices by 15 percent. 2026-01-06. https://www.techzine.eu/news/infrastructure/137670/aws-increases-ec2-capacity-block-prices-by-15-percent/. Accessed 2026-09-17. Used for: p5e.48xlarge/p5en.48xlarge dated before/after pricing ($34.61→$39.80, $36.18→$41.61) used in Table 2 and §3.1's reconciliation of conflicting AWS pricing sources.
  20. officialNVIDIA Newsroom. NVIDIA Kicks Off the Next Generation of AI With Rubin — Six New Chips, One Incredible AI Supercomputer. 2026-01-05. https://nvidianews.nvidia.com/news/rubin-platform-ai-supercomputer. Accessed 2026-09-17. Used for: NVLink6 3.6TB/s/GPU and 260TB/s/rack (Table 3, §4), Rubin GPU's own stated 50 PFLOPS NVFP4 figure and 2H2026 availability (Table 4, §6, D5).
  21. officialAMD. AMD Instinct MI400 Series GPUs (product page). 2026. https://www.amd.com/en/products/accelerators/instinct/mi400.html. Accessed 2026-09-17. Used for: MI455X 432GB HBM4/23.3TB/s and 3.6TB/s scale-up bandwidth (Table 3, §4), AMD's own MI455X-vs-Vera-Rubin comparison table reproduced as Table 4 (§6, D5), Helios/MI455X 2H2026 deployment window.
  22. officialNVIDIA. NVIDIA Vera Rubin Platform (technology page). 2026. https://www.nvidia.com/en-us/data-center/technologies/rubin/. Accessed 2026-09-17. Used for: the "now in full production" status statement contrasted with the 2H2026 availability window (§6, D5).
  23. blogBlake Crosley / Introl. Google TPU vs NVIDIA GPU: Infrastructure Decision Framework 2025. 2026. https://introl.com/blog/google-tpu-vs-nvidia-gpu-infrastructure-decision-framework-2025. Accessed 2026-09-17. Used for: the disputed "4x better price-performance" and "$0.39/chip-hour" claims adjudicated in D1 and D6.
  24. analysisSemiAnalysis. AMD vs NVIDIA Inference Benchmark: Who Wins? Performance & Cost Per Million Tokens. 2025. https://newsletter.semianalysis.com/p/amd-vs-nvidia-inference-benchmark-who-wins-performance-cost-per-million-tokens. Accessed 2026-09-17. Used for: the independently measured 14-22% GEMM achieved-vs-peak gap and the Llama3 405B FP8 batch/concurrency crossover (D2).