AI Inference Hardware · Page 5

Inference and Training Benchmarking: MLPerf and InferenceMAX Methodology, Batch-Scaling Mechanics

A tokens/sec number is meaningless without the scenario, division, framework, and batch size that produced it. This module works through MLPerf Inference's Offline/Server/Interactive scenarios and Closed/Open divisions, SemiAnalysis's independent InferenceMAX methodology, the arithmetic that links batch size to the tokens/sec-vs-tokens/sec/user tradeoff, and the roofline reasoning that makes prefill compute-bound and decode memory-bandwidth-bound. It closes with MLPerf Training's time-to-train methodology and NVIDIA/AMD/Google's v5.0/v5.1 results, read as a corroborating signal for software-stack maturity, not a serving decision input. After reading, you can read any MLPerf Inference, MLPerf Training, or InferenceMAX chart correctly and state exactly which vendors have — and have not — submitted to which suite.

Deep dive 50 min read 14 sources As of 2026-09-17 Builds on: Pages 1-4
Three benchmark paradigms and the question each one answers Three ways to measure the same hardware MLPerf Inference Closed division vendor-tuned stack (TRT-LLM, etc.) Q: best achievable under audited rules? InferenceMAX open-source frameworks (vLLM, SGLang, TRT-LLM) continuously run Q: what does an off-the-shelf deploy get? MLPerf Training time-to-target-accuracy fixed reference model + hyperparameters Q: how fast does the stack train to target? None of the three answers "which chip should I buy" alone MLPerf = audited ceiling under disclosed config · InferenceMAX = commodity-stack floor MLPerf Training = corroborating software-maturity signal, not a serving decision input
Figure 1. MLPerf Inference's Closed division reports vendor-optimized, audited results2; InferenceMAX reports what open-source serving frameworks achieve on the same hardware, continuously4; MLPerf Training reports wall-clock time to a fixed accuracy target rather than throughput at all9. Reading a chart correctly starts with knowing which of the three produced it.

Pages 1-4 established what each chip family can do architecturally — HBM capacity and bandwidth, NVLink/Infinity Fabric/ICI/NeuronLink domain sizes, peak FLOPS at each precision. None of those numbers is a serving-throughput number, and none of them says anything about tokens/sec or tokens/sec/user under a real request pattern on its own. Getting from architecture to a defensible throughput or latency claim requires a benchmark methodology, and as of this research three matter for this track: MLPerf Inference (the standards-body, audited, vendor-submitted suite), InferenceMAX (SemiAnalysis's independent, continuously-run, open-source-framework alternative), and MLPerf Training (a different scoring paradigm entirely, used here only as a secondary signal). Reading a chart from any of the three without knowing which one produced it, under what scenario, division, and framework, is the single most common way a real spec sheet turns into a misleading procurement decision.

1. MLPerf Inference: the standards-body methodology

Every number on a spec sheet in pages 1-4 of this track is a peak: the maximum a chip's tensor cores or matrix units can produce with an unbounded batch, an unconstrained latency budget, and a benchmark-friendly workload. MLPerf Inference exists to replace that peak with a number that a load generator actually measured against a stated latency and quality bar, on a system a third party can audit. MLCommons — the industry consortium that runs it — defines a fixed vocabulary before any result can be reported: a sample is one unit of inference (an image, a sequence); a query is a set of samples issued together; a system under test (SUT) is the hardware-plus-software combination being measured; and a run is a complete execution under LoadGen's control that must meet both a latency constraint and a quality constraint for the scenario it targets1. The quality constraint matters as much as the latency one and is easy to miss on a results chart: every benchmark specifies a target accuracy relative to a reference (for example, Llama 2 70B's question-answering benchmark requires 99.9% of the FP32 reference's ROUGE score1), so a submission that quantizes aggressively enough to lose accuracy is not a faster result, it is a disqualified one.

1.1 The four scenarios: what each load pattern measures

MLPerf Inference defines four scenarios, each a different load-generation pattern paired with a different performance metric1. Single-Stream sends the next query only after the SUT finishes the previous one and reports the 90th-percentile latency — a proxy for on-device, one-request-at-a-time serving and used only in the Edge category. Multi-Stream is the same one-at-a-time issuance pattern but with 8 samples per query and a 99th-percentile latency target, also Edge-only. Offline sends every sample in the run to the SUT as a single query at time zero, with no latency constraint at all, and reports raw measured throughput — this is the batch-serving proxy: maximum tokens/sec with nobody waiting on an individual response. Server (renamed Server/Interactive as of v5.0) is the interactive-serving proxy: LoadGen issues queries according to a Poisson arrival process, and the reported metric is the maximum Poisson arrival rate the SUT can sustain while still meeting a benchmark-specific latency constraint at the 99th percentile1. Datacenter-category LLM benchmarks require the (Server | Interactive) and Offline pair; Edge-category benchmarks use Single-Stream, Multi-Stream, and Offline instead1 — the two categories do not share a scenario set, which is one reason a datacenter chip's MLPerf entry and an edge chip's MLPerf entry are never directly comparable rows in the same table.

Query arrival patterns for Single-Stream, Server/Interactive, and Offline scenarios MLPerf scenario query-arrival patterns Single-Stream next query only after previous completes — reports 90%-ile latency Server / Interactive Poisson-distributed arrivals — reports max sustained rate at 99%-ile latency bound Offline all samples, one query, t=0 no latency constraint — reports raw measured throughput time →
Figure 2. Single-Stream issues one query at a time and measures tail latency; Server/Interactive issues queries as a Poisson process and measures the maximum sustainable arrival rate under a latency bound; Offline removes the latency constraint entirely and measures raw throughput on a single batch presented at the start1. Datacenter LLM benchmarks report (Server | Interactive) and Offline; Edge benchmarks use Single-Stream/Multi-Stream and Offline instead.

Interactive is not a fifth scenario with its own load-generation logic — it shares the Poisson-arrival mechanism with Server and differs only in how tight the latency bound is. MLPerf Inference v5.0 (April 2025) introduced it for Llama 2 70B specifically because LLM serving in production had shifted toward chatbots, reasoning models, and agents that need materially faster responses than the original Server bound assumed: the Llama 2 70B Interactive category requires a 99th-percentile time-to-first-token (TTFT) of 450 ms and a 99th-percentile time-per-output-token (TPOT) of 40 ms (25 output tokens/sec), versus the original Server/Conversational category's 2,000 ms TTFT and 200 ms TPOT1. Every LLM benchmark in the current suite defines both a looser Server/Conversational bound and a tighter Interactive bound side by side — Llama 3.1 405B, for instance, runs Server at 6,000 ms/175 ms TTFT/TPOT and Interactive at 4,500 ms/80 ms1 — so a submitter chooses which latency regime to target, and the two produce different maximum-throughput numbers on identical hardware because a tighter TPOT bound caps the batch size a system can run before an individual token's service time blows the budget.

MLPerf's scenarios also specify how many queries a run needs before a percentile latency or throughput claim is statistically trustworthy, not just what pattern to issue them in. An early-stopping criterion lets a run report a valid result from fewer processed queries than a full 600-second duration would otherwise produce, at the cost of a stated penalty: the effective computed percentile is nudged slightly higher than the nominal target to counteract the added variance a smaller sample carries1. MLCommons' own suggested minimum-query table makes the tradeoff explicit — hitting a 90th-percentile tail-latency claim at 99% confidence and a 0.50% margin of error needs roughly 24,576 queries, while a 99th-percentile claim at the same 99% confidence and a tighter 0.05% margin needs roughly 270,336, an 11x larger sample for the stricter percentile1. This is why Offline's minimum sample count in Table 1 below is set at 24,576 rather than some smaller or arbitrary figure: it is calibrated to the loosest percentile claim any scenario in the suite makes, not chosen independently of the statistics behind it.

Table 1. MLPerf Inference's four scenarios: query-generation pattern, duration, and reported metric1.
ScenarioQuery generationDurationLatency constraintReported metric
Single-StreamNext query issued after prior completes600 sNone (measured)90th-percentile latency
Multi-StreamNext query (8 samples) after prior completes600 sNone (measured)99th-percentile latency
Server / InteractivePoisson-distributed arrivals600 sBenchmark-specific TTFT/TPOTMax sustained Poisson rate
OfflineAll samples, single query at start≥ 24,576 samplesNoneMeasured throughput

1.2 Closed, Network, and Open divisions: what "MLPerf" is allowed to mean

The scenario controls the load pattern; the division controls how much the submitter is allowed to change about the model itself. There are three: Closed, Network, and Open1. The Closed division is the one that produces the comparable, apples-to-apples numbers this track has cited from NVIDIA, AMD, and Google throughout — it requires "using pre-processing, post-processing, and model that is equivalent to the reference or alternative implementation," allows calibration for quantization, and explicitly "does not allow any retraining"1. A submitter can choose a different framework, a different kernel implementation, a different numeric format down to INT4/FP4 if the accuracy target is still met — but the model architecture and its weights (modulo quantization) must match MLCommons' reference. This is why a Closed-division Llama 2 70B result from NVIDIA on TensorRT-LLM and one from AMD on a different stack are still comparable: both ran the same model to the same accuracy bar, and the difference in tokens/sec is attributable to hardware and serving-stack engineering, not model substitution. The unqualified name "MLPerf" may only be used for a Closed-division result1 — a chart labeled simply "MLPerf" without a division qualifier is a Closed-division number by convention.

The Open division inverts this: it "allows using arbitrary pre- or post-processing and model, including retraining"1, and any result from it must be labeled "MLPerf Open," never plain "MLPerf." Open-division submissions exist to showcase what a different model architecture, a distilled model, or a differently trained model can do on the same hardware — useful for the submitter's own narrative, not for hardware-vs-hardware comparison, because two Open submissions might not even be running the same model. The Network division sits between the two procedurally: it inherits every Closed-division constraint and adds network-fabric requirements (the SUT must be reachable only over a permitted fabric interconnect — Ethernet, InfiniBand, IEEE 802.11, or 3GPP — with bus interconnects like PCIe, NVLink, or ICI explicitly forbidden between the load generator and the system under test)1. It exists to measure inference served over a real network boundary rather than within a single tightly-coupled box, and most datacenter accelerator submissions in this track (NVIDIA, AMD, Google, and — were it to submit — AWS) report Closed-division, non-Network results, because the systems being measured are single racks or single nodes, not disaggregated services.

Table 2. MLPerf Inference's three divisions: what each permits and what label a result must carry1.
DivisionModel/preprocessing constraintRetrainingRequired label
ClosedEquivalent to MLCommons reference implementation; quantization calibration allowedNot allowed"MLPerf" (unqualified)
NetworkInherits Closed constraints; SUT reachable only over a permitted fabric interconnectNot allowed"MLPerf" (Network category)
OpenArbitrary model, preprocessing, and postprocessingAllowed"MLPerf Open"

The audit MLCommons' own naming convention leans on is a real, if partial, check. Up to two submissions per round are selected for a compliance audit — one chosen at random across all eligible submissions and, optionally, one more selected by the review committee — and only "Available" Closed-division submissions are eligible at all, with an audit expected to complete within 90 days1. A system judged equivalent to one already audited in a prior round (same CPU, accelerator, accelerator count, and configuration) can be exempted, provided its aggregate and per-accelerator performance sit within 10% of the previously audited result1. That is a spot-check, not a full re-run of every submission — the great majority of results in any given round are never independently re-verified by MLCommons itself, only checked for internal consistency against the submitted logs. The audited-vs-InferenceMAX distinction this module returns to repeatedly (section 6, and the disagreement below) is therefore not "audited" versus "unaudited" in an absolute sense; it is "subject to a low-probability, structured spot-check under a fixed methodology" versus "measured continuously by an independent party with no submission gate at all."

1.3 v5.0 to v5.1: submission growth and generational deltas

MLPerf Inference v5.0, published April 2, 2025, produced 17,457 performance results from 23 submitting organizations2. Submissions to the existing Llama 2 70B benchmark grew 2.5x year over year, the median submitted score doubled, and the best v5.0 score ran 3.3x faster than the best v4.0 score2 — a reminder that generation-over-generation MLPerf gains come from both new silicon and a full year of serving-stack optimization on existing silicon, and the two are not separable from the headline multiplier alone. The round introduced four new benchmarks: Llama 3.1 405B (supporting input/output lengths up to 128,000 tokens), the Llama 2 70B Interactive category described above, RGAT (a graph neural network benchmark on the Illinois Graph Benchmark Heterogeneous dataset, 547 million nodes and 5.8 billion edges), and Automotive PointPainting2. New accelerators appearing for the first time in this round included AMD's Instinct MI325X, Google's TPU Trillium (v6e), and NVIDIA's B200 and GB2002 — the same three vendors' current-generation parts covered in pages 1-3 of this track.

MLPerf Inference v5.1, released September 9, 2025 — the most recent round as of this writing — set a new participation record at 27 submitting organizations, with 24 of them submitting on Llama 2 70B alone3. The best-performing systems in this round improved by as much as 50% over the best v5.0 system just six months earlier3, a faster half-year improvement rate than the year-over-year 3.3x figure from the prior round implies on its own, driven by both AMD's newly-added MI355X and further TensorRT-LLM tuning on already-shipping NVIDIA hardware. Three new benchmarks arrived: DeepSeek-R1 (the suite's first reasoning-model benchmark, with prompts drawn from five datasets), Llama 3.1 8B (replacing the older GPT-J benchmark, with a 128,000-token context length versus GPT-J's 2,048), and Whisper Large V3 (a transformer encoder-decoder speech-recognition benchmark)3. Five accelerators submitted for the first time: AMD's Instinct MI355X, Intel's Arc Pro B60, and three NVIDIA parts — GB300, RTX 4000 Ada-PCIe, and RTX Pro 6000 Blackwell Server Edition3. Because v5.0 did not include DeepSeek-R1 or MI355X, the two rounds are not directly comparable on those specific workload/hardware pairs3 — any chart spanning both rounds on a benchmark introduced in v5.1 is showing only one data point, not a trend.

MLPerf Inference submission growth from v5.0 to v5.1 0 15 27 23 orgs v5.0 (Apr 2025) 17,457 results 27 orgs (record) v5.1 (Sep 2025) best systems +50% vs v5.0 Submitting organizations, v5.0 → v5.1
Figure 3. Submitting organizations grew from 23 to a record 27 between the April 2025 and September 2025 rounds, while the best-performing systems in v5.1 improved as much as 50% over the best v5.0 system six months earlier23.

2. InferenceMAX: the independent, continuously-run alternative

InferenceMAX, launched by SemiAnalysis in October 2025, answers a different question than MLPerf's Closed division does. Where MLPerf reports the best a vendor's own engineering team can extract from a fixed reference model under audited rules, InferenceMAX reports what an off-the-shelf, open-source serving stack actually achieves on the same silicon, run continuously rather than once per submission round4. It is not affiliated with MLCommons, carries no accuracy-equivalence requirement against a reference implementation, and is explicitly framed by SemiAnalysis as the best available non-vendor-submitted cross-hardware dataset for throughput, latency, cost, and power4.

2.1 Methodology: models, precision, frameworks, sequence patterns, concurrency

InferenceMAX benchmarks three model families — LLaMA 3.3 70B, DeepSeek 670B MoE, and GPT-OSS 120B — at the precision each vendor's hardware natively supports for that model (FP8 or MXFP4/FP4), across three serving frameworks: vLLM, SGLang, and TensorRT-LLM4. It tests hardware from both NVIDIA (H100, H200, B200, GB200 NVL72) and AMD (MI300X, MI325X, MI355X), giving it broader current-generation coverage across vendors than any single MLPerf round has, since MLPerf submissions depend on which vendors and partners choose to submit in a given round. Three fixed input/output token-length patterns approximate real workload shapes: 1,024/1,024 tokens for chat, 1,024/8,192 for reasoning (a short prompt, a long generated chain of thought), and 8,192/1,024 for summarization (a long prompt, a short output)4. Input sequences are randomized between 80% and 100% of the specified length specifically to defeat prefix caching — a real optimization production serving stacks use, but one that would let a benchmark's own request pattern inflate its measured throughput in a way unrepresentative of traffic that does not share prompt prefixes4.

The load-generation design is the methodological choice that most directly produces the tokens/sec-vs-tokens/sec/user curve in section 3: InferenceMAX drives each configuration with an infinite request rate — as many requests as the system can absorb — while capping the maximum number of concurrent requests at a controlled value, and sweeps that concurrency cap across a range4. At low concurrency caps, each individual request gets more of the GPU's attention and a higher per-user token rate but the system's aggregate throughput is low; at high concurrency caps, aggregate throughput rises but each request's token rate falls as it competes for the same fixed compute and memory bandwidth. Sweeping the cap and plotting the resulting (aggregate tok/s, tok/s/user) pairs at each cap value is exactly how the tradeoff frontier in Figure 4 gets drawn — it is not a separate metric, it is the same underlying batch-concurrency knob observed from two different angles.

2.2 Headline cross-vendor results

The results are workload-dependent enough that no single hardware family wins everywhere, which is itself the headline finding. On LLaMA 3.3 70B FP8 reasoning, AMD's MI300X has a throughput advantage specifically at interactivity below roughly 20-30 tokens/sec/user, a regime where its larger HBM capacity lets it hold bigger batches at tensor-parallelism degree 1 than NVIDIA's smaller-memory parts can4. On GPT-OSS 120B FP4 summarization, MI355X running vLLM achieves lower total-cost-of-ownership per million tokens than B200 running vLLM, and can even beat B200's TensorRT-LLM numbers below 225 tokens/sec/user interactivity4 — a framework-dependent result: the comparison flips if B200 is allowed to run its own vendor-tuned stack instead of the same open-source framework AMD is compared against. On DeepSeek 670B MoE FP8, B200 running SGLang delivers 1.5x faster interactivity than MI355X running SGLang at matched TCO, and H200 running SGLang runs roughly 40% lower latency than MI325X at comparable throughput4 — NVIDIA's advantage concentrates specifically on this largest, most MoE-heavy workload. On DeepSeek R1 FP4 summarization, the rack-scale GB200 NVL72 decisively outperforms every single-node 8-GPU server below 90 tokens/sec/user interactivity, while a single-node B200 system running TensorRT-LLM overtakes GB200 NVL72 above that threshold4 — the crossover chart section 6 walks through in detail.

One additional InferenceMAX result matters directly to this track's scope: as of its methodology writeup, TPU and Trainium backends were not yet included, with SemiAnalysis stating the plan to add them "over the next two months"4. Module 4 covers in full how that timeline has played out for AWS specifically (it has not happened, nearly a year past the original estimate) — the point relevant here is that InferenceMAX's cross-vendor coverage, however methodologically strong, is currently an NVIDIA-vs-AMD dataset only; nothing in this section's results generalizes to Google TPU or AWS Trainium.

The two suites differ on every axis that determines what a number from each one is fit to answer, not just on who runs them. MLPerf's governance is a multi-vendor standards body with a fixed submission cadence and a spot-check audit process; InferenceMAX is one analysis firm's own continuously-run harness, with no submission gate and no formal audit at all, but also no six-month lag before a new optimization shows up in the data. Table 3 lays the two out side by side.

Table 3. MLPerf Inference (Closed division) vs. InferenceMAX: governance, cadence, and coverage1234.
DimensionMLPerf Inference (Closed)InferenceMAX
Governed byMLCommons (multi-vendor consortium)SemiAnalysis (single analysis firm)
Cadence~2 rounds/year (v5.0 Apr 2025, v5.1 Sep 2025)Continuous
Serving frameworkSubmitter's choice; commonly vendor-tuned (e.g. TensorRT-LLM)vLLM, SGLang, and TensorRT-LLM, same models on all
Hardware coverage (this research)Whichever vendors/partners choose to submit that roundNVIDIA and AMD only; TPU/Trainium not yet added
Accuracy constraintFixed target vs. reference (e.g. 99.9% of FP32 ROUGE)None disclosed
VerificationUp to 2 submissions/round spot-checked within 90 daysNone; continuously re-run by SemiAnalysis itself

3. The tokens/sec vs. tokens/sec/user tradeoff curve

A single "tokens/sec" number for a chip or a rack is always the aggregate across every concurrent request the system is serving, and a single "tokens/sec/user" number is always the rate one individual request experiences. The two move in opposite directions as a function of the same knob — how many requests the serving stack lets run concurrently — and any hardware comparison that quotes one without the other is comparing at an unstated, possibly mismatched point on that curve.

3.1 Batch size as the knob that draws the frontier

Continuous-batching serving stacks (vLLM, SGLang, TensorRT-LLM) execute decode steps for many in-flight requests together in one forward pass, because doing so reuses each layer's weights — already streamed from HBM for that step — across every sequence in the batch rather than paying the full weight-load cost once per request. Raising the batch size (equivalently, InferenceMAX's concurrency cap) increases aggregate throughput because more sequences advance per forward pass, but it also increases each sequence's own step latency, because a bigger batch means more FLOPs and more KV-cache bytes to move per step, and every sequence in the batch waits for the whole step to finish before its own next token is available. That per-step latency, inverted, is tokens/sec/user. Section 4 derives why this tradeoff exists mechanically; here, the result is that sweeping batch size from 1 upward traces out a curve — high tok/s/user and low aggregate tok/s at small batch, the reverse at large batch — and that curve, not a single point on it, is the actual object a hardware comparison should report.

InferenceMAX's DeepSeek R1 FP4 summarization results make the shape concrete: GB200 NVL72's 130 TB/s NVLink domain (established in module 1) lets it keep a very large batch in a single coherent memory space, so it dominates every single-node 8-GPU alternative at any interactivity below 90 tokens/sec/user — the regime where aggregate throughput matters most and per-user latency tolerance is loose. Above that 90 tok/s/user threshold, a single-node B200 system running TensorRT-LLM overtakes it4, because meeting a tight per-user latency target forces a small batch size on both systems, at which point GB200 NVL72's advantage — extra aggregate capacity from pooling 72 GPUs — is not being used, and TensorRT-LLM's per-request kernel tuning on a smaller, more tightly-coupled 8-GPU system wins instead. The crossover point is not a bug in either measurement; it is the frontier itself, reported at one specific interactivity value.

A second InferenceMAX crossover, on a different workload and a different vendor pairing, shows the same shape holds generally rather than being specific to DeepSeek R1 or to NVIDIA's rack-scale advantage. On GPT-OSS 120B FP4 summarization, MI355X running the open-source vLLM framework already achieves lower total cost of ownership per million tokens than B200 running vLLM — a same-framework comparison, isolating hardware and precision from the framework-tuning question section 6 covers separately. Give B200 back its own vendor-tuned TensorRT-LLM stack instead, and MI355X still comes out ahead below roughly 225 tokens/sec/user of interactivity, only ceding the lead above that threshold4. The threshold sits far higher here (225 tok/s/user) than on the GB200-vs-B200 DeepSeek R1 crossover (90 tok/s/user), because the two crossovers answer the same structural question — where does a small-batch, per-request-tuned system start beating a large-batch, pooled-capacity system — on two different model sizes, precisions, and hardware generations, and the threshold value itself is not portable from one workload pair to another.

Tokens per second vs tokens per second per user tradeoff frontier, with the GB200 NVL72 vs B200 crossover near 90 tok/s/user tokens/sec/user (interactivity) → aggregate tokens/sec → ~90 tok/s/user GB200 NVL72 B200 (TRT-LLM) GB200 NVL72 wins B200 wins
Figure 4. Aggregate tokens/sec falls as tokens/sec/user (interactivity) rises, for both systems, because both are moving along the same batch-size axis. Below roughly 90 tok/s/user, GB200 NVL72's pooled 72-GPU batch capacity dominates single-node B200 on DeepSeek R1 FP4 summarization; above it, a small-batch, tightly-coupled 8-GPU B200 system running TensorRT-LLM overtakes it4. Curve shapes are illustrative of the reported crossover; exact intermediate values were not published.

3.2 Tokens/sec per provisioned megawatt: the power-normalized variant

Datacenter power, not chip count, is frequently the binding constraint on how much inference capacity an operator can deploy, which motivates a third metric: tokens/sec per megawatt of provisioned power, measured across the entire deployed system rather than per chip. InferenceMAX reports MI300X at 750,000 tokens/sec per MW versus MI355X at 2,550,000 tokens/sec per MW — roughly a 3x efficiency gain from CDNA3 to CDNA4 — and H100 at 900,000 tokens/sec per MW versus B200 at 2.8 million tokens/sec per MW, also roughly 3x from Hopper to Blackwell4. At a specific operating point of 180 tokens/sec/user, B200 shows what SemiAnalysis characterizes as an "eye-popping 7x" efficiency gain over the prior generation4 — a larger multiple than the roughly-3x figure that holds at unconstrained interactivity, because tighter per-user latency targets expose the newer generation's larger memory bandwidth and NVLink domain size more directly. At rack scale, GB200 NVL72 delivers roughly 8x the tokens/sec per provisioned megawatt of a single H200 at 30 tokens/sec/user on DeepSeek R14 — combining the per-chip architecture gain with the rack-pooling effect that module 1's "30x" decomposition (section 4 there) separates into its own two factors.

Same-generation cross-vendor efficiency also has a documented gap: B200 measures roughly 20% more energy-efficient than CDNA4's MI355X4. InferenceMAX's own framing of why this gap matters less than it sounds is worth carrying forward to page 6's cost analysis: electricity typically accounts for under 20% of total cost of ownership, so a 20% tokens/sec-per-MW gap translates to under a 4% TCO delta, and the majority of TCO comes from each hardware vendor's gross margin, not from the electricity bill4. A tokens/sec/MW chart is a real efficiency signal, but reading it as a proxy for total deployment cost overstates its weight by roughly a factor of five.

Tokens per second per provisioned megawatt across GPU generations tok/s per MW (millions) 1.0 2.0 0.90M H100 2.8M B200 0.75M MI300X 2.55M MI355X ~3x efficiency gain, both vendors, one generation
Figure 5. H100→B200 and MI300X→MI355X both show roughly a 3x gain in tokens/sec per provisioned megawatt across one generation; at a fixed 180 tok/s/user interactivity target B200's gain over H100 widens to roughly 7x4.

4. Prefill/decode roofline analysis

Pages 1-3 each derive a chip's ridge point — the arithmetic intensity, in FLOPs per byte, at which an operator crosses from memory-bandwidth-bound to compute-bound, computed as peak compute divided by peak bandwidth4. That is the right tool applied to a chip; this section applies the same tool to a workload — specifically, to the two phases every autoregressive LLM inference request passes through — using the analytical framework Google DeepMind's Pope et al. developed for exactly this purpose7.

4.1 The 2N FLOPs-per-token result

An N-parameter decoder-only transformer requires approximately \(2N\) matmul FLOPs to process one token through a forward pass, because each of the model's matrix multiplications performs one multiply and one add per pair of (input activation, weight) values, and there are roughly N such weight values touched per token7. Attention's own FLOPs scale quadratically with sequence length and can matter at long context, but for a first-order model they are a smaller term than the \(2N\) dense-matmul cost and are often excluded from this approximation7. This result is what lets the two inference phases be compared on the same footing: prefill processes \(L_{\text{input}}\) tokens per sequence, across a batch of \(B\) sequences, all in one forward pass, so it performs roughly \(2N \times B \times L_{\text{input}}\) FLOPs total; decode processes exactly one new token per sequence per step, so a single decode step performs roughly \(2N \times B\) FLOPs7.

4.2 Prefill: why it is compute-bound

Prefill's memory traffic and its compute both scale with the same quantity — the number of tokens in the batch, \(B \times L_{\text{input}}\) — because every one of those tokens is multiplied against the same weight matrices in the same forward pass, and the weights only need to be streamed from HBM once per layer regardless of how many tokens are being processed against them that pass7. Symbolically, memory traffic per prefill pass is close to a fixed cost — roughly \(N \times s\) bytes, where \(s\) is bytes per parameter, essentially independent of \(L_{\text{input}}\) once the batch is large enough to amortize the weight load — while compute scales linearly with \(B \times L_{\text{input}}\). Arithmetic intensity for prefill, compute divided by bytes moved, is therefore approximately \(\frac{2N \cdot B \cdot L_{\text{input}}}{N \cdot s} = \frac{2 \cdot B \cdot L_{\text{input}}}{s}\) — it grows directly with the number of tokens being processed together, and for any workload with more than a handful of tokens per forward pass, it comfortably exceeds the ridge point of a modern accelerator. This is exactly why prefill is the compute-bound phase: the more tokens in the prompt (or the larger the prefill batch), the more the fixed cost of loading weights from HBM gets amortized across compute, and the closer the achieved throughput gets to the chip's peak FLOPS. Pope et al.'s own measurement on a 540B-parameter model confirms the scale of this: large-batch prefill achieved 76% model FLOPS utilization (MFU — the ratio of observed throughput to theoretical peak FLOPS)7, a utilization level only a compute-bound regime can reach, since a memory-bound one is capped by bandwidth long before it approaches peak FLOPS.

Pope et al.'s own offline deployment of the same 540B-parameter model makes this concrete on realistic traffic rather than a synthetic benchmark: processing 1,984 input tokens and generating 64 output tokens per example, across huge numbers of examples, their implementation reaches an overall FLOPS efficiency of 73%7 — close to, though below, the 76% large-batch-prefill figure above, because this end-to-end number blends in some decode cost for the 64 generated tokens per example rather than isolating pure prefill.

4.3 Decode: why it is memory-bandwidth-bound, and how batch size moves it

Decode's memory traffic does not shrink the way prefill's does, because two of its three components do not amortize the same way across a batch. Per step, the system must stream: the model's weights (roughly \(N \times s\) bytes, shared and reusable across every sequence in the batch, same as prefill), plus each sequence's own KV cache (bytes that scale with \(B\), because every sequence in the batch has accumulated its own distinct key/value tensors and none of that data is shared across sequences)7. Compute per step is \(2N \times B\) FLOPs — linear in batch, same as prefill's per-token cost. At small batch size, the KV-cache term is negligible next to the weight-load term, so memory traffic is close to the fixed \(N \times s\) bytes and arithmetic intensity is approximately \(\frac{2NB}{Ns} = \frac{2B}{s}\) — it grows linearly with batch, which is the mechanism behind continuous batching's entire value proposition: doubling the decode batch roughly doubles arithmetic intensity, moving the operator toward the compute-bound side of the roofline without adding a single extra byte of weight traffic. But that growth does not continue indefinitely. Pope et al. give the scale at which it stops: for a 540B-parameter model at batch size 512 and context length 2,048, the KV cache alone totals 3 TB — three times the size of the model's own parameters7. Past the point where KV-cache bytes dominate total memory traffic, arithmetic intensity flattens against further batch increases, because both the numerator (compute) and the KV-cache term in the denominator now scale with \(B\) together, and the model-weight term that was shrinking relative to the KV cache stops mattering. Decode's low-batch regime is memory-bound on weights; its high-batch, long-context regime is memory-bound on the KV cache instead — never on compute, for any batch size a real deployment runs. Pope et al.'s own generation-latency number for the same 540B model — 29 milliseconds per token at low batch size, using INT8 weights7 — is the direct cost of this: decode's per-step latency is set by how fast HBM can be read, not by how many FLOPs the chip can issue per second.

The same paper's interactive-chatbot deployment shows what sits at the opposite end of the roofline in absolute terms. Running the identical 540B model with INT8 weights on 64 TPU v4 chips, their implementation processes 64 tokens of new user input, consults a cached 1,920-token conversation history, and generates a 64-token response in a total of 1.9 seconds7 — small-batch, decode-dominated, latency-bound traffic, an order of magnitude further from peak FLOPS than the 73-76% offline/prefill figures above, on the identical model and chip family. The two deployments are the same roofline plot's two ends, not two different results to reconcile.

Roofline plot: prefill sits far to the right of the ridge point, decode sits to its left and moves rightward with batch size until the KV cache dominates arithmetic intensity, FLOP/byte (log) → achievable FLOP/s (log) → ridge point Prefill compute-bound, 76% MFU Decode, B=1 B=64 B=512, KV dominates batching moves decode rightward — until KV-cache bytes dominate
Figure 6. Prefill's arithmetic intensity (\(2BL_{\text{input}}/s\)) sits well past the ridge point at any realistic batch size, achieving 76% MFU in Pope et al.'s own measurement7. Decode starts far to the left at batch size 1 and moves right as batch grows (weight cost amortizes across the batch), but flattens once KV-cache bytes — which scale with batch and are never shared across sequences — dominate total memory traffic7.

Prefill vs. decode arithmetic intensity

Prefill AI, 2BL/s
Decode AI, low-batch, 2B/s

From section 4.2-4.3: \( \text{AI}_{\text{prefill}} \approx 2BL/s \), \( \text{AI}_{\text{decode, low-batch}} \approx 2B/s \), where \(s\) is bytes per parameter7. The decode formula holds only while KV-cache bytes stay small relative to weight bytes; at large batch and long context, KV-cache traffic (not modeled here) becomes the dominant term, and this approximation overstates achievable intensity (section 4.3).

4.4 What this implies for hardware choice

The two phases want different things from a chip, and every architectural choice covered in pages 1-4 of this track reads differently once framed this way. A chip built to serve decode-heavy, low-batch, latency-sensitive traffic wants the highest achievable bandwidth-per-dollar and enough capacity to hold a large KV cache — H200's HBM3e upgrade over H100 raises bandwidth 1.4x with unchanged compute, and NVIDIA's own figures show that gain landing almost entirely on decode-bound LLM inference (up to 1.9x on Llama 2 70B) while HPC workloads, which are more compute-bound, gain far less4 — that pattern is exactly this section's roofline prediction, not a separate observation. A chip built to serve prefill-heavy or offline-batch traffic instead wants raw FLOPs and enough capacity to hold long-context activations, without needing to pay for HBM's bandwidth-per-dollar premium — which is the stated economic logic behind NVIDIA's Rubin CPX, a prefill-dedicated GPU shipping with cheaper GDDR7 instead of HBM4, covered in full in module 1. Disaggregating prefill and decode onto physically separate hardware, rather than running both phases on the same GPU generation, is a direct consequence of this section's arithmetic: a single chip provisioned to satisfy both phases' roofline positions simultaneously is over-provisioned for whichever phase it happens to be running at any given moment.

The same lens explains why memory capacity, not raw FLOPS, is frequently the binding constraint on how large a batch a decode-serving deployment can run at all: batching more sequences to raise decode's arithmetic intensity (section 4.3) requires holding more concurrent KV caches in HBM simultaneously, and running out of that capacity — not running out of compute — is what caps the achievable batch size in practice. This is also the mechanistic reason InferenceMAX finds MI300X ahead of NVIDIA parts specifically at low interactivity on LLaMA 3.3 70B4: MI300X's larger HBM capacity (established in module 2) lets it run a larger batch at tensor-parallelism degree 1 than a smaller-memory NVIDIA part can, pushing decode's arithmetic intensity further right along the same curve Figure 6 draws, before either chip's compute ceiling becomes the constraint.

5. Multi-token prediction as a throughput-interactivity lever

Section 4.3 established that decode's arithmetic intensity at a given batch size is fixed by the ratio of compute to memory traffic per step, and that raising the batch is the primary way to move it. Multi-token prediction (MTP) — generating more than one candidate output token per decode step, then verifying and accepting some of them, in an EAGLE-style speculative-decoding scheme — is a second, independent lever: it changes the amount of useful compute done per unit of memory traffic without changing the batch size at all. Each decode step still streams the same weights and KV cache from HBM, but if the step produces two or three accepted tokens instead of one, the effective tokens/sec for the same memory traffic rises directly, which is precisely a rightward move along the same roofline axis section 4 draws, achieved architecturally rather than through batching.

InferenceMAX measures this gain directly on DeepSeek R1 670B: MTP delivers up to 2-3x the throughput at iso-interactivity — the same tokens/sec/user target — in the 70-140 tokens/sec/user band4. "Iso-interactivity" is the operative qualifier: MTP is not simply "faster decode" in the abstract, it is a way to hold the user-facing token rate constant while serving more aggregate throughput at that fixed rate, which is exactly the axis operators care about when a product has a fixed latency SLA and the question is how much traffic one rack can absorb without violating it. The effect is largest in the mid-interactivity band because that is where a marginal FLOP/byte gain from token acceptance has the most room to move the operating point before either the compute ceiling (very high interactivity, small batch) or the KV-cache-dominated plateau (very low interactivity, large batch) caps further gains.

MLPerf Inference's own rules, as of this research, permit speculative decoding — and MTP specifically — for a defined subset of benchmark/scenario pairs rather than universally, which is worth knowing when comparing two Closed-division submissions on the same model: a submitter is allowed to use EAGLE-style speculative decoding with a model-specific MTP head for DeepSeek-R1 and GPT-OSS-120B, but only in their Interactive scenario configurations, with the exact draft-step count and acceptance-topk parameters specified by the rule1. A Closed-division submission that uses MTP on an eligible benchmark and one that does not are both valid, both auditable, and not directly comparable on tokens/sec alone without knowing which path was taken — the same caution section 1.2's audit discussion raises about scheduler and framework choices applies identically to this specific optimization.

6. Reading a crossover chart, and why vendor-tuned MLPerf numbers diverge from independent numbers

Section 3.1's GB200-NVL72-vs-B200 crossover at roughly 90 tokens/sec/user is the general pattern every crossover chart in this space follows: two systems trade the lead at a specific interactivity threshold, and the threshold — not either endpoint — is the reusable fact. Below the threshold, whichever system has more pooled batch capacity (more NVLink-domain GPUs, more HBM) wins because the workload tolerates enough batching to use that capacity; above it, whichever system's per-request kernel path is more tightly tuned wins because the workload forces a small batch where pooled capacity goes unused. Reading a crossover chart correctly means asking, for your own traffic, where your actual interactivity requirement falls relative to the stated threshold — not asking which system "wins" as if that were a fixed property of the hardware. AMD-vs-NVIDIA crossovers follow the identical shape: MI325X pulls ahead of H200/MI300X/H100 specifically at higher batch size and concurrency on Llama3 405B FP8 — consistently enough that H100 barely reaches 400 tokens/sec under a 150-second latency constraint where MI325X keeps climbing5 — while H100 and H200 outperform both AMD parts under low-latency, low-concurrency conditions on the same model family45 — page 6 adjudicates this AMD-vs-NVIDIA comparison in full; the mechanism producing the crossover is the one this page owns.

Identify systems& workload tested Find your owntok/s/user target Check which sideof the threshold Check framework& precision used
  1. Identify the two systems and the workload. A crossover threshold measured on DeepSeek R1 FP4 summarization does not transfer to LLaMA 3.3 70B FP8 chat or GPT-OSS 120B FP4 summarization — model, precision, and sequence-length pattern all move the threshold (section 3.1's two worked examples land at 90 and 225 tok/s/user respectively).
  2. Find your own tokens/sec/user target. A chatbot product with a tight per-token SLA and an offline batch-summarization job sit at opposite ends of the same curve; the "right" hardware depends entirely on where your traffic falls, not on either system's peak number.
  3. Check which side of the threshold you are on. Below it, the system with more pooled batch capacity wins; above it, the system with tighter per-request kernel tuning wins — section 4's roofline argument is why.
  4. Check the framework and precision the numbers were measured under. A crossover measured with both systems on the same open-source framework answers a different question than one where the incumbent runs its own vendor-tuned stack — the next paragraph covers why that gap exists.

The gap between MLPerf's Closed-division numbers and InferenceMAX's open-source-framework numbers has a specific, checkable mechanism behind it, not just a vague "vendors optimize for their own benchmark" explanation. Three concrete places TensorRT-LLM tuning diverges from a stock vLLM or SGLang deployment: kernel selection (TensorRT-LLM ships fused, hand-tuned CUDA kernels for specific shape/precision/batch combinations that a general-purpose open-source framework's kernel selector may not reach for the same configuration), batching-scheduler heuristics (MLPerf's Closed division measures whatever scheduling policy the submitter configured for that specific benchmark's token-length and latency profile, which need not match the scheduler defaults an operator would run in production), and precision-format aggressiveness within the accuracy bound (a submission can push quantization as far as the 99% or 99.9%-of-FP32 accuracy target allows, extracting throughput a more conservative production deployment — which typically keeps more accuracy margin — would not extract). None of these three is against MLPerf's rules; the Closed division explicitly permits calibration for quantization and does not constrain which serving framework or scheduler a submitter uses, only which model1. The result is a number that is real, audited, and reproducible — but represents the ceiling of what a team with full access to the vendor's own engineering can extract from that specific hardware and benchmark configuration, not what a team deploying an open-source stack off the shelf will see.

Independently measured GEMM throughput gives a second, chip-level version of the same achieved-vs-peak gap, independent of any benchmark methodology at all: on raw matrix-multiply throughput, H100/H200 measure roughly 720 TFLOP/s at BF16 against MI300X's roughly 620 TFLOP/s (about 14% slower), and at FP8, roughly 1,280 TFLOP/s against MI300X's roughly 990 TFLOP/s (about 22% slower)5 — gaps between vendor-stated peak-theoretical FLOPS and independently measured achieved FLOPS that exist before any serving framework, scenario, or scheduler enters the picture at all. Every multiplier anywhere in this track — MLPerf-vs-InferenceMAX, vendor spec sheet vs. measured GEMM, one generation vs. the next — carries this same caution: a number's precision (14%, 22%, 3.4x, 30x) is real for the exact configuration it was measured under, and generalizes to a different configuration only as far as that configuration's dominant bottleneck (roofline regime, framework, scheduler) matches the one the number was measured in.

A third, independent lab reaches the same conclusion via a different methodology entirely — real, rented cloud instances rather than owned benchmark hardware. Artificial Analysis's own testing at 16 concurrent queries finds NVIDIA H200 achieving roughly 45 output tokens/sec per query and roughly 600 tokens/sec of system throughput on DeepSeek R1, against MI300X's roughly 35 tokens/sec per query and roughly 500 tokens/sec system throughput6 — NVIDIA ahead on both axes at this fixed, moderate concurrency point. But at peak concurrency the ranking flips on raw aggregate throughput: MI300X reaches roughly 4,100 output tokens/sec at 750 concurrent queries on DeepSeek R1, versus H200's roughly 3,250 tokens/sec peak, and roughly 8,500 tokens/sec at 1,000 concurrent queries on Llama 4 Maverick versus H100's roughly 6,500 tokens/sec peak6. Artificial Analysis's own stated conclusion — NVIDIA offers "advantages for latency-sensitive workloads," while MI300X "offers advantages for use-cases which seek to optimize for efficiency"6 — matches SemiAnalysis's GEMM-based and InferenceMAX-based findings exactly. Three independent measurement approaches (a GEMM microbenchmark, an end-to-end open-source-framework serving benchmark, and rented-instance concurrency testing) converging on the same low-concurrency/high-concurrency split is as close to a triangulated result as this space currently offers. One caveat on currency: Artificial Analysis's own testing here covers only DeepSeek R1 and Llama 4 Maverick, published June 8, 2025, and predates B200 and MI325X entirely6 — a full generation behind InferenceMAX's broader, continuously-updated hardware coverage, so treat it as corroborating the direction of the split rather than as a current-generation data point in its own right.

7. MLPerf Training as a supporting data point

Everything so far in this module measures inference. MLPerf Training, run by the same MLCommons standards body, measures a different thing entirely — not throughput at all, but wall-clock speed to a fixed destination. It matters here as a corroborating signal for cross-vendor software-stack maturity and multi-node scaling behavior, not as an input to a serving-hardware decision: a chip that trains fast is not thereby a chip that serves inference cheaply, and this section does not conflate the two. Its main relevance to this track is what it confirms about which vendors have (and have not) built out a working, audited large-scale training software stack at all.

7.1 Methodology: time-to-train-to-target-quality

MLPerf Training's scoring unit is fundamentally different from every metric in sections 1-6. A benchmark is defined as "an abstract problem that can be solved using ML by training a model based on a specific dataset or simulation environment to a target quality level"8; a run is "a complete execution of an implementation on a system, training a model from initialization to the quality target"8; and a run result is simply "the wallclock time required for a run"8. There is no scenario, no latency bound, no arrival process — the entire measurement is: start from initialization, train until a fixed accuracy or loss target is reached on the reference dataset, and report the clock time. A benchmark result is the mean of several run results with the highest and lowest dropped, normalized against MLCommons' own reference-implementation result so that a larger number always means faster8.

The Closed division carries the same spirit as its Inference counterpart but a narrower scope: it "requires using the same preprocessing, model, training method, and quality target as the reference implementation"8 — a submitter can change hardware, framework, and low-level optimization, but not the model architecture, the optimizer, or the accuracy bar it has to hit. One rule specific to training and worth flagging because it affects how "time to train" numbers should be read: by default, the time to evaluate quality counts against the wall-clock score, but a submitter may instead generate timestamped checkpoints, evaluate them after the clock has stopped, and then set the reported stop time to the timestamp of the first checkpoint that actually passed the quality bar8 — meaning the reported time-to-train number is training time to the first passing checkpoint, not training time plus evaluation overhead, whenever a submitter uses that (permitted, common) checkpoint-based scoring path. This module does not go further into training mechanics — no MFU, optimizer, or checkpointing-system internals — consistent with using these results only as a supporting data point, as module 4 already frames AWS's absence from this same suite.

MLPerf Training's general rules add a second layer of trust beneath the Closed-division constraints already described: submissions must "strive to be fair," the same system and framework must be used consistently across every run in a submission result set, the framework and system are not permitted to detect that a benchmark is running and behave differently, and — outside a few explicitly-defined exceptions — a submission is not permitted to pre-train on information about the benchmark's own dataset or a successful model's end state before the timed run begins8. That last rule matters specifically for how to read a fast time-to-train number: a submission cannot simply start from a checkpoint that already encodes what the target quality metric will look like and call the remaining wall-clock time "training."

7.2 v5.0/v5.1 cross-vendor results: NVIDIA, AMD, Google

MLPerf Training v5.0, published June 4, 2025, drew 201 performance results from 20 submitting organizations, including AMD, Google Cloud, and NVIDIA among named participants9. The round introduced a new Llama 3.1 405B pretraining benchmark, replacing the prior GPT-3-based one9, and reported round-over-round speedups on prior benchmarks: Stable Diffusion (8-processor) improved 2.28x versus v4.1, Llama 2 70B LoRA fine-tuning (8-processor) improved 2.10x versus v4.1, and — at multi-node scale — RetinaNet (64-processor) improved 1.43x and Stable Diffusion (64-processor) improved 3.68x, both measured against the earlier v3.1 round9. Newly available processors that round included AMD's Instinct MI300X (192 GB HBM3) and MI325X (256 GB HBM3e), NVIDIA's Blackwell GB200 and B200-SXM-180GB, and Google's TPU Trillium9 — the same current-generation chips covered architecturally in pages 1-3.

NVIDIA's own MLPerf Training v5.0 submission gives the clean Hopper-vs-Blackwell time-to-train comparison: on Llama 3.1 405B pretraining at 512 GPUs, Hopper completed in 269.12 minutes against Blackwell's 121.09 minutes, a 2.2x gain; on Llama 2 70B LoRA fine-tuning at 8 GPUs, Hopper took 27.93 minutes against Blackwell's 11.14 minutes, a 2.51x gain; on Stable Diffusion v2 at 8 GPUs, 33.97 versus 12.86 minutes, a 2.64x gain; and on R-GAT at 8 GPUs, 11.18 versus 4.97 minutes, a 2.25x gain11 — NVIDIA states Blackwell delivers up to 2.6x higher per-GPU training performance than Hopper across all seven MLPerf Training v5.0 benchmarks11. AMD's own blog on the same round frames its result plainly: v5.0 marked "the debut of AMD's MLPerf Training submission," including the first-ever multi-node training submission on Instinct hardware13. On its own submission, AMD's Instinct MI325X platform outperformed NVIDIA's H200 platform by up to 8% fine-tuning Llama 2-70B-LoRA, and MI300X delivered what AMD describes as "competitive performance" against NVIDIA H100 on the same workload13. Partner submissions on the same benchmark filled out a wider spread: Supermicro's liquid-cooled MI325X system (the first liquid-cooled MLPerf Training submission) reached 21.75 minutes; MangoBoost's single-node MI300X system reached 29.6 minutes, its 2-node/16-GPU configuration (AMD's first-ever multi-node submission) reached 16.32 minutes, and its 4-node/32-GPU configuration reached 10.92 minutes13. Dell, Oracle, Gigabyte, and QCT rounded out the same benchmark's 8-GPU partner field, with times spanning 22.1 to 30.42 minutes across MI300X and MI325X platforms13 — a wider set of independent partner submissions on a single AMD-hardware benchmark than any prior MLPerf Training round had carried.

MLPerf Training v5.1, released November 12, 2025 — the most recent round — drew 65 unique systems across 12 different hardware accelerators from 20 organizations, with nearly half the submissions multi-node, an 86% increase in multi-node submissions over v4.110. Two new benchmarks replaced older ones: Llama 3.1 8B pretraining replaced BERT, and Flux.1 (a transformer-based text-to-image benchmark) replaced Stable Diffusion v210. Submissions to existing benchmarks grew too — 24% more submissions on Llama 2 70B LoRA and 15% more on the newly-introduced Llama 3.1 8B benchmark than the round before, with MLCommons characterizing the round's generative-AI performance gains as "outpacing the rate of improvement predicted by Moore's Law"10. NVIDIA's v5.1 submission includes the suite's first Blackwell Ultra (GB300-class) results: Llama 3.1 405B pretraining completed in 10 minutes at 5,120 Blackwell GPUs, a result 2.7x faster than the fastest Blackwell submission the prior round and 3x faster than a same-GPU-count Hopper submission, with 85% scaling efficiency going from 512 to 5,120 GPUs12; Blackwell Ultra reached Llama 3.1 8B pretraining in 5.2 minutes at 512 GPUs and Llama 2 70B LoRA fine-tuning in 0.40 minutes at 512 GPUs12. The same submission's results on the round's other benchmarks — FLUX.1 at 12.5 minutes on 1,152 GPUs, DLRM-DCNv2 at 0.71 minutes on 64 GPUs, R-GAT at 0.84 minutes on 256 GPUs, and RetinaNet at 1.4 minutes on 512 GPUs12 — round out the full seven-benchmark spread rather than leaving Llama 3.1 405B as the only v5.1 data point.

Google's most recent official Trillium-vs-v5p training comparison is not from the v5.0 or v5.1 rounds at all — it is a Google Cloud blog from the v4.1 round, published November 14, 2024, roughly seven months before v5.014. No equivalent Google training blog for v5.0 or v5.1 was found in this research. That v4.1 comparison reported Trillium delivering up to 1.8x better performance-per-dollar than TPU v5p on the GPT3-175B benchmark, with 99% weak-scaling efficiency within a single ICI domain (versus v5p's 94%) and 97.6% scaling efficiency measured against its smallest submitted configuration14, alongside a stated cost-to-train reduction of up to 1.8x (45% lower) versus v5p on the same benchmark14. Because this data point is one full round behind NVIDIA's and AMD's most recent submissions, it should be read as directionally consistent with — not as freshly confirming — Google's current-generation training competitiveness; module 3's Ironwood coverage does not include a v5.0/v5.1-era training comparison for the same reason.

Table 4. MLPerf Training v5.0/v5.1 cross-vendor time-to-train data points on shared benchmarks, with AWS Trainium's row empty by documented fact, not by omission91011121314.
VendorRoundBenchmarkResult
NVIDIAv5.0Llama 3.1 405B pretrain, 512 GPUBlackwell 121.09 min vs. Hopper 269.12 min (2.2x)
NVIDIAv5.1Llama 3.1 405B pretrain, 5,120 GPUBlackwell 10 min (2.7x vs. best v5.0 Blackwell)
AMDv5.0Llama 2 70B LoRA, 8 GPUMI325X up to 8% ahead of H200 (AMD's own submission)
AMDv5.0Llama 2 70B LoRA, 32 GPU (4-node)MI300X 10.92 min (partner: MangoBoost, first AMD multi-node)
Googlev4.1 (most recent official)GPT3-175B pretrainTrillium 1.8x perf/$ vs. v5p, 99% scaling efficiency
AWSevery round to date—No submission
Time to train, Hopper vs Blackwell vs Blackwell Ultra, per MLPerf Training benchmark minutes (log-ish scale) 269.1 121.1 Llama 3.1 405B (512 GPU) 27.9 11.1 Llama 2 70B LoRA (8 GPU) 34.0 12.9 Stable Diffusion v2 (8 GPU) 5.2 Llama 3.1 8B Ultra, 512 GPU Hopper Blackwell
Figure 7. Hopper-to-Blackwell time-to-train gains land in a tight 2.2-2.64x band across NVIDIA's own MLPerf Training v5.0 submissions11; Blackwell Ultra's v5.1 Llama 3.1 8B result (5.2 minutes, 512 GPU) is shown at the amber bar for scale, though it is a different benchmark/GPU-count pair and not a direct multiple of the other bars12.

7.3 AWS Trainium's documented absence

AWS does not appear among the named participants in either the v5.0 or the v5.1 MLPerf Training results announcement910, even though AMD, Google, and NVIDIA all appear in both rounds. Module 4 (section 9) documents this across all three benchmark suites in full, naming every submitting organization by round and showing the same pattern holds for MLPerf Inference v5.0/v5.1 and for InferenceMAX; this section adds only the training-specific confirmation, from the same two MLCommons announcements this section already cites for NVIDIA's, AMD's, and Google's results. The practical consequence carries directly from module 4: AWS's own published Trainium/Trainium2 training-performance claims — peak FLOPS, generation-over-generation multipliers — have no independent or standardized cross-vendor benchmark of any kind to check them against, in training any more than in inference. Given that the vendor-vs-independent gaps documented elsewhere in this module run from roughly 14% (measured GEMM vs. peak-theoretical FLOPS on a mature ROCm stack) to multiples of 2x or more (MLPerf-tuned vs. open-source-framework inference throughput), there is no basis for assuming AWS's own unverified training claims sit closer to their independently-checked equivalent than any other vendor's claims have, and every reason — a narrower, newer compiler and kernel stack than either CUDA or ROCm — to suspect the gap could be larger, not smaller.

Where experts disagree

Section 6 already showed the mechanism behind why MLPerf's Closed-division numbers and InferenceMAX's open-source-framework numbers diverge — kernel tuning, scheduler configuration, and quantization aggressiveness within the accuracy bound. What is genuinely contested is not the mechanism but how to weight the two resulting kinds of number against each other when making a hardware decision.

MLPerf's Closed-division numbers are the legitimate reference point

MLCommons' own submission process, and vendors reporting their audited results23

Every Closed-division MLPerf result runs a model equivalent to MLCommons' own reference implementation, to a disclosed accuracy bar, under a scenario with a fixed, published latency constraint, and is subject to a random post-submission audit process1. That is a materially higher evidentiary bar than any single independent analyst's benchmark run: the methodology is fixed in advance by a multi-vendor working group rather than by one analysis firm's choices, the configuration is fully disclosed, and a result can be challenged and re-checked. On this view, the fact that a vendor's own engineers tuned the submission is not a flaw — it demonstrates what the hardware plus a competent software stack can do, which is a real, purchasable combination (the vendor's own optimized inference server is commercially available), not a hypothetical.

  • Audited, reproducible, and run under a methodology no single vendor controls unilaterally.
  • Discloses the full configuration (token counts, latency bound, quality target) needed to reconstruct the number.
  • Represents the ceiling extractable with the submitter's own engineering effort, not what a team deploying an off-the-shelf open-source stack will see.
  • A submitter chooses which scenario, framework, and precision to submit — a vendor can simply not submit a configuration that looks unfavorable.

Independent, continuously-run benchmarks on commodity stacks are the more decision-relevant number

SemiAnalysis's InferenceMAX methodology and results4

InferenceMAX tests the same hardware families with off-the-shelf vLLM, SGLang, and TensorRT-LLM, continuously rather than once per submission round, and explicitly documents cases where the gap between vendor-tuned and open-source-framework results is large enough to change which generation looks ahead — "open source vLLM for Hopper is faster than mostly open source TRT-LLM" in some configurations4. Most production deployments run an open-source serving stack, not a vendor's own bespoke tuning pass, so a benchmark run on that same class of stack is a closer proxy for what an operator will actually get, and running continuously (rather than once per six-month MLPerf round) captures serving-stack improvements MLPerf's submission cadence misses entirely.

  • Measures the software stack most deployments actually run, not a bespoke vendor-tuned one.
  • Not gated by vendor submission choices — SemiAnalysis controls what gets tested and when.
  • Not run by a neutral multi-vendor standards body; methodology choices (which models, which sequence-length patterns) are SemiAnalysis's own.
  • No formal audit process equivalent to MLPerf's; results are not independently re-verifiable by a third party in the same structured way.
Assessment

These measure genuinely different things, and the productive question is not which one is "right" but which one answers the question you actually have. If the decision is "what is the best this hardware plus a fully-tuned, vendor-supported stack can do" — relevant when you can buy or license that stack, as with NVIDIA's own TensorRT-LLM deployment support — MLPerf's Closed-division number, at the exact scenario and latency bound matching your workload, is the right one, and module 1's decomposition of the "30x" claim (its section 4) shows how to unpack it into its real components rather than taking the headline multiplier at face value. If the decision is "what will my team, running an open-source stack we maintain ourselves, actually get," InferenceMAX is closer to ground truth, and its continuous, multi-framework coverage is currently the best publicly available answer to that question for NVIDIA and AMD hardware — though not yet for Google TPU or AWS Trainium (section 2.2). What would most directly settle the size of the remaining gap for any specific configuration is a vendor MLPerf submission using vLLM or SGLang instead of TensorRT-LLM — nothing in MLCommons' rules prohibits it1 — but as of this research no NVIDIA MLPerf submission has used an open-source framework in the Closed division.

References

  1. repoMLCommons. MLPerf Inference Rules (inference_policies repository, master branch). 2026. https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc. Accessed 2026-09-17. Used for: the four-scenario definitions and metrics table (§ Scenarios), Closed/Network/Open division constraints and naming rules (§ Divisions), Llama 2 70B/Llama 3.1 405B Server-vs-Interactive TTFT/TPOT latency bounds, SUT/run/run-result/sample/query definitions.
  2. standardMLCommons. MLCommons Releases New MLPerf Inference v5.0 Benchmark Results. 2 April 2025. https://mlcommons.org/2025/04/mlperf-inference-v5-0-results/. Accessed 2026-09-17. Used for: v5.0 submission counts (23 orgs, 17,457 results), Llama 2 70B year-over-year growth, four new benchmarks, new accelerators (MI325X, TPU Trillium, B200, GB200).
  3. standardMLCommons. MLCommons Releases New MLPerf Inference v5.1 Benchmark Results. 9 September 2025. https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/. Accessed 2026-09-17. Used for: v5.1 record submission count (27 orgs), best-system 50% improvement over v5.0, three new benchmarks (DeepSeek-R1, Llama 3.1 8B, Whisper), five new accelerators.
  4. analysisSemiAnalysis. InferenceMAX: Open Source Inference Benchmarking. October 2025. https://newsletter.semianalysis.com/p/inferencemax-open-source-inference. Accessed 2026-09-17. Used for: InferenceMAX methodology (models, precisions, frameworks, sequence-length patterns, infinite-request-rate-with-concurrency-cap design), tokens/sec/user crossover results, tokens/sec-per-MW efficiency figures, MTP throughput gains, TPU/Trainium coverage-gap statement, MLPerf-vs-InferenceMAX conflict note.
  5. analysisSemiAnalysis. AMD vs NVIDIA Inference Benchmark: Who Wins? Performance & Cost Per Million Tokens. 2025. https://newsletter.semianalysis.com/p/amd-vs-nvidia-inference-benchmark-who-wins-performance-cost-per-million-tokens. Accessed 2026-09-17. Used for: Llama3 405B FP8 throughput crossover by batch/concurrency, low-latency-vs-high-concurrency AMD/NVIDIA split, and independently measured GEMM achieved-vs-peak-theoretical gap (H100/H200 vs. MI300X, BF16 and FP8).
  6. analysisArtificial Analysis. Independent Performance Analysis of Leading GPUs. 8 June 2025. https://artificialanalysis.ai/articles/independent-analysis-of-leading-gpus-amd-nvidia. Accessed 2026-09-17. Used for: DeepSeek R1 and Llama 4 Maverick concurrency-throughput results on rented cloud instances (H200/H100 vs. MI300X), and the explicit latency-sensitive-vs-efficiency-optimized conclusion.
  7. paperPope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., Dean, J. Efficiently Scaling Transformer Inference. arXiv:2211.05102, 9 November 2022. https://arxiv.org/abs/2211.05102. Accessed 2026-09-17. Used for: the 2N FLOPs-per-token forward-pass result (§2.2, citing Kaplan et al. 2020), prefill-vs-decode separation rationale, model FLOPS utilization (MFU) definition, memory-time-vs-compute-time framing, the 3TB KV-cache-vs-model-size numeric example (batch 512, context 2048), and the 76% large-batch-prefill MFU / 29ms-per-token low-batch-decode measured results on a 540B-parameter model.
  8. repoMLCommons. MLPerf Training Rules (training_policies repository, master branch). 2026. https://github.com/mlcommons/training_policies/blob/master/training_rules.adoc. Accessed 2026-09-17. Used for: benchmark/run/run-result/benchmark-result definitions (§ Overview), Closed division constraints (§ Divisions), the on-the-clock-vs-checkpoint-based quality-evaluation rule (§ Quality measure).
  9. standardMLCommons. New MLCommons MLPerf Training v5.0 Benchmark Results Reflect Rapid Growth and Evolution of the Field of AI. 4 June 2025. https://mlcommons.org/2025/06/mlperf-training-v5-0-results/. Accessed 2026-09-17. Used for: v5.0 submission counts (201 results, 20 orgs) and named participant list confirming AWS's absence, new Llama 3.1 405B pretraining benchmark, round-over-round speedups, newly available processors (MI300X, MI325X, GB200, B200-SXM, TPU Trillium).
  10. standardMLCommons. MLCommons Releases MLPerf Training v5.1 Results. 12 November 2025. https://mlcommons.org/2025/11/training-v5-1-results/. Accessed 2026-09-17. Used for: v5.1 submission counts (65 systems, 12 accelerators, 20 orgs) and named participant list confirming AWS's continued absence, multi-node submission growth, new Llama 3.1 8B and Flux.1 benchmarks.
  11. blogNVIDIA. NVIDIA Blackwell Delivers up to 2.6x Higher Performance in MLPerf Training v5.0. NVIDIA Developer Blog, 2025. https://developer.nvidia.com/blog/nvidia-blackwell-delivers-up-to-2-6x-higher-performance-in-mlperf-training-v5-0/. Accessed 2026-09-17. Used for: per-benchmark Hopper-vs-Blackwell time-to-train figures (Llama 3.1 405B, Llama 2 70B LoRA, Stable Diffusion v2, R-GAT) and the "up to 2.6x across all seven benchmarks" summary claim.
  12. blogNVIDIA. NVIDIA Blackwell Architecture Sweeps MLPerf Training v5.1 Benchmarks. NVIDIA Developer Blog, 2025. https://developer.nvidia.com/blog/nvidia-blackwell-architecture-sweeps-mlperf-training-v5-1-benchmarks/. Accessed 2026-09-17. Used for: v5.1 Blackwell Ultra time-to-train results (Llama 3.1 405B at 5,120 GPUs, Llama 3.1 8B and Llama 2 70B LoRA at 512 GPUs), scaling-efficiency figure.
  13. blogAMD. AMD Expands AI Momentum with First MLPerf Training Submission. AMD Blog, 4 June 2025. https://www.amd.com/en/blogs/2025/amd-drives-ai-gains-with-mlperf-training-results.html. Accessed 2026-09-17. Used for: AMD's first-ever MLPerf Training submission, MI325X-vs-H200 Llama 2 70B LoRA result, first multi-node Instinct training submission, partner (Supermicro/MangoBoost) time-to-train results.
  14. blogGoogle Cloud. Trillium MLPerf 4.1 training benchmarks. Google Cloud Blog, 14 November 2024. https://cloud.google.com/blog/products/compute/trillium-mlperf-41-training-benchmarks/. Accessed 2026-09-17. Used for: Trillium-vs-v5p performance-per-dollar and scaling-efficiency figures (most recent official Google Trillium training comparison found, one round behind v5.0/v5.1).