AI Inference Hardware · Page 4
AWS Trainium and Inferentia: NeuronCore Architecture and Instance Topology
AWS builds its own inference and training silicon around one core design, NeuronCore, now in its third generation, packaged into four chips (Inferentia1, Inferentia2, Trainium1, Trainium2) and scaled into instances that top out at a 64-chip NeuronLink-v3 domain. This page covers the chip-level compute/memory deltas across NeuronCore v1-v3, how Inf1/Inf2/Trn1/Trn2 instances scale chip count into throughput and memory, the Trn2 UltraServer topology, and the Neuron SDK's programmability model — plus the plain fact that none of this hardware has ever appeared in an independent benchmark.
1. NeuronCore evolution: v1 to v3
AWS has shipped three generations of its NeuronCore compute engine. Each generation is packaged into one or two chips, and each chip is the building block of an instance family: NeuronCore-v1 exists only inside Inferentia1 (Inf1); NeuronCore-v2 is shared between two separate chips, Inferentia2 (Inf2) and Trainium1 (Trn1); NeuronCore-v3 exists only inside Trainium2 (Trn2). The core count per chip, and the precision formats each core supports, change with every generation — this is where the useful comparison lives, not in clock speed or process node, neither of which AWS publishes.
1.1 NeuronCore-v1 (Inferentia1)
Inferentia1 is AWS's first custom inference chip. Each Inferentia1 chip integrates four NeuronCore-v1 cores, delivering 128 INT8 TOPS and 64 FP16/BF16 TFLOPS in aggregate, backed by 8 GiB of device DRAM at 50 GiB/sec of bandwidth1. That's conventional DDR4 DRAM, not HBM — the single biggest architectural difference between NeuronCore-v1 and everything that follows. AWS's own architecture page for Inferentia1 lists only three chip components: Compute, Device Memory, and NeuronLink. There is no "Programmability" entry1 — dynamic shapes, control flow, and the GPSIMD custom-operator engine all arrive starting with NeuronCore-v2, which is a real capability gap, not an omission from the documentation (§7 covers what this means in practice).
1.2 NeuronCore-v2 (Inferentia2 and Trainium1)
NeuronCore-v2 is the first Neuron core to ship in two different chips built for two different jobs. An Inferentia2 chip packs two NeuronCore-v2 cores delivering 380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, and 47.5 FP32 TFLOPS, with 32 GiB of HBM at 820 GiB/sec2. A Trainium1 chip uses the identical pair of NeuronCore-v2 cores and reports the identical compute and memory numbers — 380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, 47.5 FP32 TFLOPS, 32 GiB HBM at 820 GiB/sec, plus 1 TB/sec of DMA bandwidth3. "cFP8" here means configurable FP8 running through the same 190-TFLOPS datapath as FP16/BF16 — NeuronCore-v2 does not have a separate, higher-throughput native FP8 mode the way NeuronCore-v3 does. Both chips add the Programmability block absent from Inferentia1: dynamic shapes and control flow via NeuronCore-v2 ISA extensions, user-programmable rounding (round-nearest-even or stochastic rounding), and custom operators through embedded GPSIMD engines23.
Where Inferentia2 and Trainium1 diverge is chip-to-chip interconnect, not compute. Inferentia2's NeuronLink-v2 runs at 192 GiB/sec/chip on the multi-chip Inf2 instance sizes7, while AWS's own Trainium2 architecture page states Trainium1's NeuronLink-v2 at 384 GB/sec/chip4 — exactly double, in a different chip package built for collective communication during distributed training rather than tensor-parallel inference sharding. Same NeuronCore, same TFLOPS, different scale-up fabric, because the two chips serve different jobs.
1.3 NeuronCore-v3 (Trainium2)
Trainium2 is the current flagship, and the first chip in the family to fan a single core count out to eight. Each Trainium2 chip contains eight NeuronCore-v3 cores that collectively deliver 1,299 FP8 TFLOPS, 667 BF16/FP16/TF32 TFLOPS, 2,563 FP8/FP16/BF16/TF32 sparse TFLOPS, and 181 FP32 TFLOPS, with 96 GiB of device memory at 2.9 TB/sec of bandwidth and 3.5 TB/sec of DMA bandwidth4. This is the first generation where FP8 has a genuinely higher peak rate than BF16 (1,299 vs. 667 TFLOPS, roughly 1.95x) rather than sharing a datapath with it. NeuronLink-v3 chip-to-chip interconnect runs at 1.28 TB/sec/chip, and 16 dedicated CC-Cores orchestrate collective communication within and across instances4. Trainium2 also introduces Logical NeuronCore Configuration (LNC), which lets software combine the compute and memory resources of multiple physical NeuronCores into one logical NeuronCore — a capability none of the earlier chips expose4.
| Chip | NeuronCore | Cores/chip | INT8 (TOPS) | FP16/BF16 (TFLOPS) | FP8 (TFLOPS) | FP32 (TFLOPS) | Device memory | Memory BW |
|---|---|---|---|---|---|---|---|---|
| Inferentia1 | v1 | 4 | 128 | 64 | — | — | 8 GiB DDR4 | 50 GiB/s |
| Inferentia2 | v2 | 2 | 380 | 190 | 190 (cFP8) | 47.5 | 32 GiB HBM | 820 GiB/s |
| Trainium1 | v2 | 2 | 380 | 190–191 | 190–191 (cFP8) | 47.5–48 | 32 GiB HBM | 820 GiB/s |
| Trainium2 | v3 | 8 | — | 667 | 1,299 | 181 | 96 GiB HBM | 2.9 TB/s |
Trainium1's FP16/BF16/FP32 figures appear as 190/47.5 on the Inferentia2 and Trainium chip pages and as 191/48 on the Trainium2 architecture page's own head-to-head comparison table — the same silicon, rounded slightly differently across two AWS documents34. Trainium2 does not publish a separate INT8 TOPS figure; AWS states its integer/low-precision throughput only in the sparse FP8/FP16/BF16/TF32 line (2,563 TFLOPS)4.
2. Inf1 to Inf2: the architecture jump
AWS's own comparison of Inf1 to Inf2 claims up to 4x higher throughput and up to 10x lower latency, published 13 June 20235. The chip-level compute numbers alone do not explain a 4x throughput jump: INT8 TOPS goes from 128 to 380 (2.97x) and FP16/BF16 TFLOPS from 64 to 190 (2.97x) per chip12 — the raw peak-compute multiplier is under 3x, not 4x. The rest of the throughput gain, and essentially all of the 10x latency reduction, comes from the memory subsystem, which is where NeuronCore-v2 actually breaks from NeuronCore-v1.
Inferentia1 uses 8 GiB of DDR4 DRAM at 50 GiB/sec. Inferentia2 replaces that with 32 GiB of HBM at 820 GiB/sec5 — a 4x capacity increase and a 16.4x bandwidth increase in the same generational step. For autoregressive decode, where each generated token requires re-reading the full set of weights and the growing KV cache from device memory, bandwidth (not peak TFLOPS) sets the throughput ceiling. A 16.4x bandwidth increase against a 3x compute increase means Inferentia2 moved the ceiling for memory-bound decode far more than it moved the ceiling for compute-bound prefill, which is consistent with a throughput claim (4x) that exceeds the raw compute multiplier (2.97x) and a latency claim (10x) that a compute-only jump could not produce on its own. In roofline terms, each chip's ridge point — the arithmetic intensity (FLOP/byte) where an operator crosses from memory-bound to compute-bound — is peak compute divided by peak bandwidth. Inferentia1's ridge point is 64 FP16 TFLOPS over 50 GiB/sec ≈ 1,280 FLOP/byte; Inferentia2's is 190 TFLOPS over 820 GiB/sec ≈ 232 FLOP/byte, more than 5x lower. Any decode operator whose arithmetic intensity falls between those two values is memory-bound on Inferentia1 but compute-bound on Inferentia2 — Inferentia2 didn't just get more bandwidth in absolute terms, it moved the memory-bound/compute-bound boundary far enough down the intensity axis that a wider band of real operators land on the compute-bound side, which is the mechanism behind a throughput claim that outruns the raw TFLOPS multiplier (page 5 covers the prefill/decode roofline split in full generality).
The capacity increase has a second-order effect AWS calls out explicitly: Inf2 can deploy a 175-billion-parameter model split across multiple accelerators via NeuronLink5. At FP16 (2 bytes/parameter), 175 billion parameters occupy
$$ 175 \times 10^{9} \times 2\ \text{bytes} = 350\ \text{GB} \approx 326\ \text{GiB} $$which does not fit on a single Inferentia2 chip (32 GiB) or even six chips (192 GiB), but does fit — with roughly 58 GiB of headroom for activations and KV cache — on the 12-chip inf2.48xlarge instance's 384 GiB of aggregate device memory (§3, Table 3). Inferentia1's 8 GiB-per-chip DDR4 ceiling made this kind of single-model, multi-chip sharding impractical at any instance size AWS shipped for Inf1; Inferentia2's HBM capacity and NeuronLink-v2 tensor-parallel interconnect are what make the 175B framing a real deployment pattern rather than a marketing number.
3. Instance-family scaling: Inf1, Inf2, and Trn1
Both Inf1 and Inf2 are sold in four instance sizes, and both scale by adding whole chips rather than by changing the chip itself — the per-chip numbers from Table 1 are the unit, and instance-level compute/memory is (very close to) that unit times chip count.
| Instance | Chips | vCPUs | Host mem (GiB) | FP16/BF16 (TFLOPS) | INT8 (TOPS) | Device mem (GiB) | Device mem BW (GiB/s) | NeuronLink-v1 (GiB/s/chip) |
|---|---|---|---|---|---|---|---|---|
| inf1.xlarge | 1 | 4 | 8 | 64 | 128 | 8 | 50 | N/A |
| inf1.2xlarge | 1 | 8 | 16 | 64 | 128 | 8 | 50 | N/A |
| inf1.6xlarge | 4 | 24 | 48 | 256 | 512 | 32 | 200 | 32 |
| inf1.24xlarge | 16 | 96 | 192 | 1,024 | 2,048 | 128 | 800 | 32 |
| Instance | Chips | vCPUs | Host mem (GiB) | FP8/FP16/BF16/TF32 (TFLOPS) | FP32 (TFLOPS) | Device mem (GiB) | Instance mem BW (GiB/s) | NeuronLink-v2 (GiB/s/chip) |
|---|---|---|---|---|---|---|---|---|
| inf2.xlarge | 1 | 4 | 16 | 190 | 47.5 | 32 | 820 | N/A |
| inf2.8xlarge | 1 | 32 | 128 | 190 | 47.5 | 32 | 820 | N/A |
| inf2.24xlarge | 6 | 96 | 384 | 1,140 | 285 | 192 | 4,920 | 192 |
| inf2.48xlarge | 12 | 192 | 768 | 2,280 | 570 | 384 | 9,840 | 192 |
Scaling is close to linear in both families but not exact: inf1.xlarge and inf1.2xlarge both carry exactly one chip and identical compute/memory, differing only in host vCPU/RAM allocation, and the same is true of inf2.xlarge vs. inf2.8xlarge. Chip count only changes at the larger sizes — inf1 jumps 1→4→16 chips, inf2 jumps 1→6→12 chips — and NeuronLink only turns on once an instance has more than one chip to talk to. Inf2's NeuronLink-v2 bandwidth is constant at 192 GiB/sec/chip whether the instance has 6 or 12 chips7, so tensor-parallel collective cost per chip does not change as you go from inf2.24xlarge to inf2.48xlarge — you're adding more chips to the same fabric bandwidth per chip, not diluting it. Inf1 shows the identical pattern a generation earlier: NeuronLink-v1 holds at 32 GiB/sec/chip on both inf1.6xlarge (4 chips) and inf1.24xlarge (16 chips)6 — AWS has kept per-chip interconnect bandwidth flat across an instance family's multi-chip sizes in every generation this page covers, rather than growing or shrinking per-chip link bandwidth with chip count.
The practical decision inside each family is less about which size has the most chips and more about where the single-chip-vs-multi-chip boundary falls. inf1.xlarge/2xlarge and inf2.xlarge/8xlarge are the same single-chip hardware with different host-side vCPU and RAM allocations — pick the larger host size when preprocessing, tokenization, or a CPU-side reranker needs the extra vCPUs and host RAM, not because it changes anything about the accelerator. The real step change is at inf1.6xlarge/inf2.24xlarge and above, where NeuronLink turns on and a model too large for one chip's device memory (8 GiB on Inf1, 32 GiB on Inf2) can be sharded via tensor parallelism across chips (§2's 175B-parameter example is exactly this case on inf2.48xlarge). EFA bandwidth follows the same step pattern on Inf1 — 25 Gbps on the single-chip sizes, 25 Gbps still on inf1.6xlarge, jumping to 100 Gbps only at inf1.24xlarge6 — which matters once a deployment spans more than one instance and needs to fan out over the network rather than NeuronLink.
The programmability gap from §1.1 has a concrete instance-level consequence on Inf1: because NeuronCore-v1 has no documented support for dynamic shapes or control flow1, an Inf1 deployment has to compile against fixed input shapes ahead of time — variable-length sequences get bucketed into a small number of pre-compiled shapes rather than compiled once and dispatched dynamically. Every Inf2 instance size in Table 3, and every Trn1/Trn2 instance size (§5-§6), inherits NeuronCore-v2/v3's dynamic-shape support (§7), so this is specifically an Inf1 planning constraint, not a property of the "Inferentia" name in general.
Trn1 follows a different scaling logic built for training rather than serving. Trn1.2xlarge gives a single Trainium1 chip for small-model training and experimentation; trn1.32xlarge and trn1n.32xlarge carry the full 16 chips, connected in a 2D torus via NeuronLink-v28 — the same topology shape Trn2 uses at 4x4 (§5), just built from Trainium1's 384 GB/sec/chip links instead of Trainium2's 1,024 GB/sec/chip links. Beyond a single instance, Trn1/Trn1n instances scale out through an EC2 UltraCluster to over 100,000 Trainium chips using AWS's non-blocking, petabit-scale EFA networking8 — a loosely-coupled, Ethernet-scale fan-out, structurally different from the tightly-coupled NeuronLink domain Trn2's UltraServer builds at much smaller scale (§6).
4. Trainium2: the chip inside Trn2
A Trainium2 device is eight NeuronCore-v3 cores plus four HBM stacks (96 GiB total, 2.9 TB/sec of bandwidth4, described elsewhere in AWS's own NKI documentation as roughly 3 TB/s9), 128 DMA engines for moving data within and across devices, and either 164 or 209 CC-Cores for collective communication, depending on which AWS document you read — the two counts disagree and AWS has not reconciled them publicly as of this writing. (For what it's worth, AWS's Trainium3 architecture page separately states 16 CC-Cores for the next generation13, which is weak evidence the dedicated chip-architecture pages' "16" is the intended figure and the NKI guide's "20" is the outlier — but this is inference from a pattern, not a correction AWS has published.) Every NeuronCore has its own Tensor, Vector, Scalar, and GpSimd compute engines wired to two on-chip SRAMs, SBUF and PSUM9.
The four compute engines are specialized, not interchangeable. AWS's NKI architecture guide gives each one a data-path width and a clock frequency: the Tensor Engine, optimized for GEMM/convolution/transpose, runs at 2.4 GHz with a 4x128-element-wide path for dense FP8 input, a 2x128-wide path for dense BF16/FP16 input, and a 5x128-wide path for sparse input (1x128 wide on output); the Vector Engine runs at 0.96 GHz with a 512-wide path for BF16/FP16 and a 256-wide path for other dtypes; the Scalar Engine runs at 1.2 GHz with a 128-wide path; and the GpSimd engine — the one exposed for custom operators (§7) — also runs at 1.2 GHz10. The Tensor Engine's FP8 path being twice as wide as its BF16/FP16 path at the same clock is the concrete mechanism behind Trainium2's roughly 2x FP8-over-BF16 TFLOPS ratio (Table 1) — it is a wider datapath, not a higher clock, that gets FP8 matmuls to double throughput.
Unlike a GPU, none of this on-chip memory is hardware-cached. AWS states this explicitly: "all of the memories available to a NeuronCore are software-managed... NeuronCore does not have a hardware cache system that performs data movement across memories in a way that is opaque to the program"9. Every tensor movement between HBM and SBUF, and between SBUF and PSUM, is an explicit instruction — issued by a hand-written NKI kernel or generated by the Neuron Graph Compiler. The 128 DMA engines per chip are the mechanism that carries out those explicit moves10: with no hardware cache to issue background fills or evictions on its own, HBM↔SBUF traffic has to be spread across enough independent DMA queues to keep the roughly-20x bandwidth gap between SBUF and HBM from becoming a serialization bottleneck, which is why the DMA engine count is specified at all — on a cache-based architecture it would be an implementation detail, not a documented, chip-level number. SBUF (State Buffer) is the working scratchpad, organized as 128 partitions and holding both kernel inputs staged from HBM and intermediate results, at roughly 20x the bandwidth of HBM9. PSUM (Partial Sum Buffer) is a much smaller, dedicated accumulator that the Tensor Engine reads and writes for multi-tile matmul accumulation, also reachable by the Vector and Scalar engines to evict finished results back to SBUF9. NeuronCore-v3's SBUF is 128 partitions of 224 KiB each — 28 MiB per core, up from 24 MiB per NeuronCore-v2 core — while PSUM capacity is unchanged at 2 MiB9. Multiply 28 MiB by eight cores and you land on the 224 MiB chip-level SBUF figure AWS advertises as a 4.7x jump over NeuronCore-v2's chip-level 48 MiB (Table 4)4.
NeuronCore-v3's roughly 2x FP8-over-BF16 compute ratio (1,299 vs. 667 TFLOPS, Table 1) is not a clock-speed effect — it's a genuinely different matmul mode. The Tensor Engine supports FP8 input matrices (both FP8_E4 and FP8_E5, and mixed pairs of the two) at double the throughput of a BF16/FP16 matmul, while still accumulating in FP3210. NeuronCore-v2's "cFP8" mode, by contrast, shared the same datapath as FP16/BF16 and ran at the same rate (§1.2) — the double-throughput FP8 path is new to NeuronCore-v3.
| Metric | Trainium1 | Trainium2 | Improvement |
|---|---|---|---|
| FP8 (TFLOPS) | 191 | 1,299 | 6.7x |
| BF16/FP16/TF32 (TFLOPS) | 191 | 667 | 3.4x |
| FP32 (TFLOPS) | 48 | 181 | 3.7x |
| FP8/FP16/BF16/TF32 sparse (TFLOPS) | n/a | 2,563 | n/a |
| HBM capacity (GiB) | 32 | 96 | 3x |
| HBM bandwidth (TB/sec) | 0.8 | 2.9 | 3.6x |
| SBUF capacity (MiB) | 48 | 224 | 4.7x |
| Memory pool size (chips) | up to 16 | up to 64 | 4x |
| Inter-chip interconnect (GB/sec/chip) | 384 | 1,280 | 3.3x |
| CC-Cores | 6 | 16 | 3.3x |
"Memory pool size" is the maximum chip count a workload can address as pooled memory via NeuronLink — up to 16 chips on Trainium1 (a single trn1.32xlarge), up to 64 on Trainium2 (a full UltraServer, §6)4.
5. trn2.48xlarge: the 16-chip, 4x4 2D torus
AWS's own architecture documentation describes Trn2 as "purpose built for high-performance deep learning training and inference"11 — unlike Inf1/Inf2, which are inference-only instance families, Trn2 is explicitly positioned for both jobs on the same chip. There is no Inferentia3: NeuronCore-v3's native double-rate FP8 matmul (§4) and 96 GiB of HBM per chip cover the memory-bandwidth-bound decode workload Inferentia2 was built for, while the same chip's BF16/FP32 paths and NeuronLink-v3 collective fabric cover training — Trainium2 absorbs the role a dedicated third-generation Inferentia chip would otherwise have filled.
trn2.48xlarge (and its UltraServer-eligible twin, trn2u.48xlarge) is the base Trn2 instance size: 16 Trainium2 chips connected by NeuronLink-v3 in a 4x4 2D torus11. In a torus, every chip connects to its four grid neighbors and the grid wraps at the edges, so a chip in the corner of the logical 4x4 layout is exactly as many hops from any other chip as one in the "middle" — there is no true middle or edge from a routing standpoint. The instance carries 192 vCPUs, 2,048 GiB of host memory, 1,536 GiB of device memory at 46.4 TB/sec of aggregate bandwidth, and 3,200 Gbps of EFAv3 networking11. Every chip's NeuronLink-v3 link runs at 1,024 GB/sec/chip within the instance, per the Trn2 instance-specification table11 — slightly below the 1,280 GB/sec/chip ("1.28 TB/sec") NeuronLink-v3 figure the Trainium2 chip architecture page states as the chip's total interconnect bandwidth4. The two numbers are answering different questions: 1,280 GB/sec/chip is the chip's total NeuronLink-v3 capacity, while 1,024 GB/sec/chip is what the 4x4 torus's four-neighbor wiring pattern actually delivers per chip within a single trn2.48xlarge instance — some headroom is presumably reserved for the inter-instance ring a chip joins once it's part of an UltraServer (§6).
At the instance level, compute is 20.8 FP8 PFLOPS, 10.7 FP16/BF16/TF32 PFLOPS, 41 sparse PFLOPS, and 2.9 FP32 PFLOPS11 — exactly 16x the per-chip numbers from Table 1 (1,299 x 16 = 20,784 TFLOPS ≈ 20.8 PFLOPS; 667 x 16 = 10,672 TFLOPS ≈ 10.7 PFLOPS; 2,563 x 16 = 41,008 TFLOPS ≈ 41 PFLOPS; 181 x 16 = 2,896 TFLOPS ≈ 2.9 PFLOPS), confirming that trn2.48xlarge scaling is linear multiplication of the chip spec, not a discounted or boosted aggregate.
6. Trn2 UltraServer: 64 chips, one NeuronLink-v3 domain
A Trn2 UltraServer is four trn2u.48xlarge instances wired together via NeuronLink-v3, for a total of 64 Trainium2 chips11. AWS's connection rule is specific: "Trainium2 chips with the same coordinates in each Trn2 instance are connected in a ring topology"11 — chip 0 in instance A links to chip 0 in instances B, C, and D in a ring, chip 1 in instance A links to chip 1 in B/C/D, and so on for all 16 coordinates. The result is 16 independent four-node rings layered on top of four independent 4x4 tori — intra-instance traffic stays on the 4x4 torus at 1,024 GB/sec/chip, and inter-instance traffic uses a separate 256 GB/sec/chip allocation reserved for exactly that ring11.
| Metric | trn2.48xlarge / trn2u.48xlarge | Trn2 UltraServer |
|---|---|---|
| # Trainium2 chips | 16 | 64 |
| vCPUs | 192 | 768 |
| Host memory (GiB) | 2,048 | 8,192 |
| FP8 (PFLOPS) | 20.8 | 83.2 |
| FP16/BF16/TF32 (PFLOPS) | 10.7 | 42.8 |
| FP8/FP16/BF16/TF32 sparse (PFLOPS) | 41 | 164 |
| FP32 (PFLOPS) | 2.9 | 11.6 |
| Device memory (GiB) | 1,536 | 6,144 |
| Device memory bandwidth (TB/sec) | 46.4 | 185.6 |
| Intra-instance NeuronLink-v3 (GB/sec/chip) | 1,024 | 1,024 |
| Inter-instance NeuronLink-v3 (GB/sec/chip) | n/a | 256 |
| EFAv3 (Gbps, per instance) | 3,200 | 3,200 |
Every UltraServer number is 4x the single-instance number: 83.2 FP8 PFLOPS = 20.8 x 4, 42.8 BF16 PFLOPS = 10.7 x 4, 164 sparse PFLOPS = 41 x 4, 11.6 FP32 PFLOPS = 2.9 x 4, 185.6 TB/sec = 46.4 x 411 — clean linear scaling from chip to instance to UltraServer, with the one asymmetry being networking: EFAv3 is quoted at 3,200 Gbps "per instance" for both rows in AWS's own instance-specification table11, while AWS's Trn2 product page states the UltraServer's aggregate EFAv3 networking as 12.8 Tbps12 — exactly 4 x 3.2 Tbps, confirming the per-instance table entry is per-constituent-instance, not an UltraServer aggregate. AWS's Trn2 product page also states 30-40% better price-performance than GPU-based P5e/P5en instances and 3x better energy efficiency than Trn112 — both are AWS's own comparative claims, not independently measured (§9).
6,144 GiB of pooled device memory is enough to put a scale number on what an UltraServer can hold. At FP8 (1 byte/parameter), the UltraServer's full 6,144 GiB ≈ 6.6 TB of device memory could in principle hold roughly 6.6 trillion parameters as weights alone, with zero headroom left for KV cache, activations, or optimizer state11 — not a workable production configuration, but a useful upper bound on the scale of a single-domain MoE model this hardware topology is built to address; a real deployment would reserve a meaningful fraction of that 6,144 GiB for KV cache and leave a model in the hundreds-of-billions-to-low-single-digit-trillions range, the same regime page 6 covers for rack-scale GPU and TPU domains.
The ring-of-tori structure in Figure 4 also sets the shape of any collective that has to cross instance boundaries. A collective confined to one 4x4 torus (16 chips) stays on the 1,024 GB/sec/chip intra-instance fabric; the moment it needs to touch all 64 chips, it has to route through the 256 GB/sec/chip inter-instance rings, one per coordinate. That is a real bandwidth cliff — 4x less bandwidth per chip the instant a collective crosses from "within one trn2u.48xlarge" to "across the UltraServer" — which is exactly the kind of topology detail that determines whether a given tensor-parallel or expert-parallel sharding plan should stay inside one 16-chip instance or is worth spreading across all 64.
7. Neuron SDK programmability
Four capabilities separate NeuronCore-v2 and NeuronCore-v3 from a fixed-function inference accelerator, and none of them are present on NeuronCore-v1 (§1.1).
Dynamic shapes and control flow. Inferentia2, Trainium1, and Trainium2 all support dynamic shapes and control flow through ISA extensions on their respective NeuronCore generations234. This matters for anything with data-dependent branching or variable sequence length — a model doesn't need to be recompiled per input shape or padded to a fixed bucket the way a purely static-graph accelerator would require.
User-programmable rounding. Both Trainium1 and Trainium2 let software choose between round-nearest-even and stochastic rounding34. Stochastic rounding matters specifically for training in reduced precision: round-nearest-even is deterministic and biases accumulated gradients in a fixed direction over many steps, while stochastic rounding is unbiased in expectation, which is why it shows up as a rounding-mode choice on the training-capable Trainium chips rather than as a fixed default.
GPSIMD custom operators. Every NeuronCore from v2 onward has a GpSimd engine alongside its Tensor, Vector, and Scalar engines (§4)9, and AWS states explicitly that custom operators run "via deeply embedded GPSIMD engines"234. This is the escape hatch for an op the Neuron Graph Compiler doesn't already lower efficiently — write it directly against the GPSIMD engine (or, at a lower level, against the Neuron Kernel Interface, which exposes SBUF/PSUM placement and the four engines directly) instead of waiting on a compiler update.
Logical NeuronCore Configuration (LNC). Starting with Trainium2, the Neuron SDK can combine the compute and memory resources of multiple physical NeuronCores into a single logical NeuronCore4 — a software-level re-partitioning of the eight physical NeuronCore-v3 cores on a chip, useful for matching a logical core's compute/memory footprint to a specific layer or kernel's actual working-set size instead of hard-partitioning at the physical-core boundary.
The Neuron Kernel Interface (NKI). The GPSIMD escape hatch above is the coarse-grained version; NKI is the fine-grained one. AWS's own NKI documentation exposes it as a Python-embedded kernel language with explicit tile-level primitives — matmul, transpose, elementwise arithmetic, reductions, indexing, and RNG operations at the nki.language level, plus lower-level instruction-set access (matmul variants, DMA copy/transpose, tensor-tensor and tensor-scalar ops, collectives such as all-reduce/all-gather/reduce-scatter/all-to-all) at the nki.isa level9. A kernel written against NKI addresses SBUF, PSUM, and HBM tensors directly, in the same role that a Triton or CUDA kernel addresses shared memory and global memory explicitly rather than relying on a compiler to infer placement9 — the structural comparison to write is not "Neuron vs. a fixed-function NPU toolchain" but "Neuron's NKI vs. Triton/CUDA," a peer-level kernel language rather than a configuration surface bolted onto a black-box compiler.
Together with the software-managed SBUF/PSUM hierarchy described in §4 — where every byte movement is an explicit instruction rather than a hardware-cache decision — these four features put NeuronCore v2/v3 closer to a fully programmable accelerator (the same category as a GPU's SM plus its CUDA/ROCm compiler stack) than to a fixed-function NPU with a small, closed operator set. The tradeoff is the one every custom-ISA accelerator makes: correctness and performance for anything outside the well-trodden path depend on the maturity of the Neuron Graph Compiler and the willingness of a team to hand-write NKI kernels, not on a large, general-purpose kernel ecosystem the way CUDA benefits from a decade of community kernels.
8. Trainium3: roadmap context
Trainium3 is the fourth-generation chip and moves to a new core, NeuronCore-v4 — eight cores per chip, the same count as Trainium213. AWS's Neuron architecture documentation states the chip-level compute as 2,517 MXFP8/MXFP4 TFLOPS, 671 BF16/FP16/TF32 TFLOPS, 2,517 FP16/BF16/TF32 sparse TFLOPS, and 183 FP32 TFLOPS, backed by 144 GiB of device memory at 4.9 TB/sec, 4.9 TB/sec of DMA bandwidth, and a new NeuronLink-v4 interconnect at 2.56 TB/sec per device13. LNC, dynamic shapes/control flow, and rounding-mode selection all carry over from Trainium213; the architecture page states 16 CC-Cores for Trainium313, matching Trainium2's official architecture-page figure rather than the higher count the separate NKI guide gives for Trainium2 (§4).
AWS's own Trainium2-to-Trainium3 comparison table is worth reading in full, because it does not move uniformly — and it undercuts a literal reading of the system-level "4.4x" headline (§9 covers why headline multipliers need this kind of decomposition in general).
| Metric | Trainium2 | Trainium3 | Improvement |
|---|---|---|---|
| FP8 (TFLOPS) | 1,299 | 2,517 | 2x |
| MXFP4 (TFLOPS) | n/a | 2,517 | n/a |
| BF16/FP16/TF32 (TFLOPS) | 667 | 671 | 1.0x |
| FP32 (TFLOPS) | 181 | 183 | 1.0x |
| HBM capacity (GiB) | 96 | 144 | 1.5x |
| HBM bandwidth (TB/sec) | 2.9 | 4.9 | 1.7x |
| SBUF capacity (MiB) | 224 | 256 | 1.14x |
| Inter-chip interconnect (GB/sec/chip) | 1,280 | 2,560 | 2x |
| DMA bandwidth (TB/sec) | 3.5 | 4.9 | 1.4x |
Dense BF16/FP16/TF32 and FP32 throughput are essentially flat generation over generation (1.0x, within rounding) — all of the compute headline comes from FP8/MXFP4 doubling and the new MXFP4 mode entirely, while HBM capacity, bandwidth, and interconnect all grow at their own, independent rates (1.14x-2x). A workload that stays in BF16 gets almost nothing from the Trainium2-to-Trainium3 jump at the chip level; a workload that can move to FP8 or MXFP4 gets 2x, and a workload that is memory-capacity- or bandwidth-bound gets 1.5x-1.7x. AWS's system-level "4.4x higher performance" claim (§9) is a blend of whichever of these per-metric factors the underlying benchmark actually stresses, multiplied by whatever chip-count or topology change the Trn3 UltraServer makes over the Trn2 UltraServer — not a uniform per-chip speedup. SBUF capacity's 1.14x improvement is the smallest factor in the table, notably smaller than the 4.7x jump SBUF got from NeuronCore-v2 to NeuronCore-v3 (§4, Table 4) — consistent with SBUF sizing having already moved much closer to whatever working-set target AWS is designing toward, with HBM bandwidth (1.7x) and interconnect (2x) left as the generation's larger, unfinished scaling problems.
At the instance level, AWS announced Trainium3 and the Trn3 UltraServer on 2 December 2025, describing the chip as AWS's first on a 3 nm process14. A Trn3 UltraServer scales to up to 144 Trainium3 chips, 362 FP8 PFLOPs total, 20.7 TB of HBM3e, and 706 TB/sec of aggregate memory bandwidth, using a new NeuronSwitch-v1 all-to-all fabric AWS states doubles inter-chip interconnect bandwidth versus NeuronLink-v314. Note the topology shift implied by "up to 144 chips": Trn3 UltraServer is not simply "more of the same 4x4-torus-plus-ring" pattern Trn2 uses (§5-§6) — 144 is not a clean multiple of 16 or 64 the way Trn2's 16→64 scaling is, and neither AWS document describes the Trn3 interconnect topology in the same geometric detail given for Trn2's torus and ring. Everything in this section is a vendor architecture specification and roadmap announcement, not a shipped, independently characterized product; it is included here only as forward context for where NeuronCore-v4 sits relative to v3, consistent with this page's scope.
Laid end to end, the four generations trace a single, consistent chip-to-chip interconnect arc: NeuronLink-v1 on Inferentia1 at 32 GiB/sec/chip (§3, Table 2), NeuronLink-v2 at 192 GiB/sec/chip on Inferentia2 (§3, Table 3) but 384 GB/sec/chip on the training-oriented Trainium1 package using the same core (§1.2), NeuronLink-v3 at 1,024-1,280 GB/sec/chip on Trainium2 depending on whether the figure is instance-level or chip-level (§4-§5), and NeuronLink-v4 at 2.56 TB/sec/chip on Trainium313 — roughly an 80x increase in per-chip interconnect bandwidth from the first generation to the fourth. Device memory bandwidth tells a similar story: 50 GiB/sec (DDR4, Inferentia1) to 820 GiB/sec (HBM, Inferentia2/Trainium1) to 2.9 TB/sec (Trainium2) to 4.9 TB/sec (Trainium3), roughly 100x over the same four generations. Compute grew more slowly in comparison — a few hundred TFLOPS to a few thousand — which is the same pattern §2 identified between Inferentia1 and Inferentia2 specifically: across this entire chip family, AWS has consistently pushed memory bandwidth and interconnect harder than raw compute, generation over generation.
9. The verification gap: no independent benchmark exists
State this plainly rather than as a hedge: as of this writing, AWS Trainium and Inferentia have zero third-party benchmark datapoints anywhere. Not in MLPerf Inference. Not in InferenceMAX. Not in MLPerf Training. Every performance number in §1-§8 above is AWS's own, published on AWS's own documentation and product pages, with no independent reproduction.
MLPerf Inference v5.0 lists its 23 submitting organizations by name: AMD, ASUSTeK, Broadcom, Cisco, CoreWeave, CTuning, Dell, FlexAI, Fujitsu, GATEOverflow, Giga Computing, Google, HPE, Intel, Krai, Lambda, Lenovo, MangoBoost, NVIDIA, Oracle, Quanta Cloud Technology, Supermicro, and Sustainable Metal Cloud15. Amazon and AWS appear nowhere on that list. MLPerf Inference v5.1, the most recent round as of this research, sets a record for participation at 27 organizations — AMD, ASUSTek, Azure, Broadcom, Cisco, Coreweave, Dell, GATEOverflow, GigaComputing, Google, HPE, Intel, KRAI, Lambda, Lenovo, MangoBoost, MiTac, Nebius, NVIDIA, Oracle, Quanta Cloud Technology, Red Hat, Supermicro, TheStage AI, University of Florida, and Vultr, plus one individual submitter16 — again, no Amazon, no AWS. Azure appears as a submitting organization in v5.1; AWS does not appear in either round.
InferenceMAX, SemiAnalysis's continuously-run, open-source-framework inference benchmark, launched in October 2025 covering NVIDIA (H100, B200, GB200 NVL72) and AMD (MI300X, MI355X) hardware on vLLM/SGLang/TRT-LLM. Its own methodology writeup states plainly: "over the next two months, they're expanding InferenceMAX to include Google TPU and AWS Trainium backends"17. Two months from an October 2025 launch is roughly December 2025. As of this writing — nearly a year later — that has not happened for Trainium. The project has since been renamed InferenceX; its public GitHub repository's hardware-support roadmap lists GB300/GB200 NVL72, MI355X, B300, B200, MI325X, MI300X, H200, and H100 as currently supported, and names TPUv7x Ironwood, MI455X UALoE72, Vera Rubin NVL72, and Rubin NVL8 as "coming soon"18. AWS Trainium is not on that coming-soon list at all — it appears only in the repository's aspirational tagline text ("TPUv6e/v7/Trainium2/3"), with no scheduled date, no private-beta status, and no roadmap entry of its own18. Google TPU is closer to inclusion than AWS Trainium is, not further — the opposite of what the original two-vendor "within two months" announcement implied would happen simultaneously. Neither SemiAnalysis nor AWS has published a stated reason for the delay specific to Trainium; it is not possible from public sources to say whether the gap is infrastructure access on AWS's side, engineering prioritization on SemiAnalysis's side, or something else.
MLPerf Training shows the same pattern across two full rounds. MLPerf Training v5.0 (June 2025) names its 20 submitting organizations: AMD, ASUSTeK, Cisco, CoreWeave, Dell, GigaComputing, Google Cloud, HPE, IBM, Krai, Lambda, Lenovo, MangoBoost, Nebius, NVIDIA, Oracle, Quanta Cloud Technology, SCITIX, Supermicro, and TinyCorp19. MLPerf Training v5.1 (November 2025) names 20 organizations again: AMD, ASUSTeK, Cisco, Dell, Giga Computing, HPE, Krai, Lambda, Lenovo, MangoBoost, MiTAC, Nebius, NVIDIA, Oracle, Quanta Cloud Technology, Supermicro, University of Florida, Verda, and Wiwynn20. AWS is absent from both named lists, even though AMD, Google, and NVIDIA all appear on both1920 — AWS is the one major custom-silicon vendor in this track that has shipped three generations of training-capable chips (Trainium1, Trainium2, and the announced Trainium3) without ever submitting a single MLPerf Training result.
Read the numbers in §1-§8 accordingly. Chip-level and instance-level compute/memory/bandwidth figures (TFLOPS, GiB, GB/sec) come directly off AWS's own architecture documentation and are internally self-consistent — Table 4 and the worked chip-to-instance-to-UltraServer derivations in §5-§6 both check out arithmetically, and that consistency is a reasonable basis for capacity and topology planning. AWS's comparative and generational performance claims — "4x higher throughput" (Inf1→Inf2, §2), "30-40% better price-performance than P5e/P5en" (Trn2, §6), "4.4x higher performance" (Trainium2→Trainium3, §8) — are a different category of claim: they describe achieved, workload-dependent throughput or cost, the exact category of number that has repeatedly diverged from vendor peak specs on every chip in this track that does have independent data (page 5 covers the mechanics of why prefill and decode diverge from peak FLOPS; the "Where experts disagree" section below applies that lesson specifically to AWS's own claims).
Where experts disagree
This track's other three hardware pages can lean on independent measurement to check a vendor's own claims. AWS silicon offers no such check (§9), which raises a narrower but genuinely contested question specific to this chapter: how much weight should AWS's own peak-FLOPS and comparative performance claims carry in a sizing or procurement decision, given that they are the only numbers available?
Architecture-level specs are usable even unverified
Implicit in how this page (and the rest of this track) treats vendor spec sheets21
Memory capacity, memory bandwidth, chip count, and interconnect bandwidth are physical and electrical properties of the hardware — a chip either has 96 GiB of HBM wired to it or it doesn't, and a link either runs at 1,024 GB/sec or it doesn't. These numbers are far harder for a vendor to misrepresent than an end-to-end throughput or cost claim, because they're checkable against the physical part (pin counts, HBM stack density, die photos) independent of any software stack. Even SemiAnalysis's own independent benchmarking methodology implicitly treats this category as reliable: it re-measures achieved GEMM throughput against vendor peak-TFLOPS claims, but does not re-measure or dispute vendor-stated HBM capacity or bandwidth figures21. On this reasoning, the worked derivations in §2, §3, and §6 of this page — 175B-parameter footprint against device memory capacity, chip-level PFLOPS multiplied out to instance and UltraServer level — are sound planning inputs regardless of AWS's benchmark absence, because they rest on capacity and bandwidth, not on achieved throughput.
- Physical specs are the only numbers available for AWS silicon at all; refusing to use them leaves no basis for capacity planning.
- The same category of number (capacity, bandwidth) has not been the source of documented vendor/independent divergence elsewhere in this track — the divergence has consistently been in achieved throughput, not spec-sheet memory figures.
- "Physically checkable" is not the same as "independently checked" — nobody has actually torn down a Trainium2 die to confirm the 96 GiB/2.9 TB/s figures the way independent labs have benchmarked GPU GEMM throughput.
- Interconnect bandwidth figures in particular (NeuronLink-v2/v3 GB/sec/chip) are harder to verify externally than HBM capacity, since they depend on protocol overhead and achieved vs. quoted link rate.
Comparative and generational multipliers need independent reproduction
The pattern documented across this track's independent benchmarks (SemiAnalysis, MLCommons)1721
Every vendor-stated comparative multiplier this track has been able to check against independent measurement has proven optimistic relative to the vendor's own framing. Independently measured GEMM throughput on AMD MI300X runs 14-22% below AMD's own peak-theoretical FLOPS21. NVIDIA's own MLPerf submissions, tuned on TensorRT-LLM, show materially larger Blackwell-over-Hopper multiples than InferenceMAX's open-source-framework measurements on the same hardware17. AWS's Neuron SDK is a narrower, newer compiler and kernel stack than either CUDA or ROCm, has zero public independent benchmark of any kind, and its own documentation contains at least one internal inconsistency (the CC-Core count disagreement in §4) — there is no basis to assume the achieved-vs-peak gap on Trainium is smaller than the 14-22% gap documented on a more mature AMD stack, and every reason to suspect it could be larger. On this view, "4x higher throughput" (Inf1→Inf2), "30-40% better price-performance than P5e/P5en," and "4.4x higher performance" (Trainium2→Trainium3) should be treated as directional claims pending verification, not as inputs to a procurement decision.
- Grounded in an actual, measured pattern (14-22% GEMM gap, TRT-LLM-vs-open-source MLPerf gap) rather than pure speculation.
- Consistent with how this page treats AWS's own claims throughout §2, §6, and §8 — flagging them explicitly as AWS's characterization, not corroborated fact.
- Extrapolating a GPU software-stack gap to a structurally different, purpose-built accelerator with a compiler-driven (not hand-tuned-kernel-driven) execution model is itself an unverified assumption — Neuron's gap could just as easily be smaller.
- Without any data point, "could be larger" is exactly as unfalsifiable as AWS's own claims are — the position can state a plausible direction but not a magnitude.
Use the physical/electrical numbers — chip count, HBM capacity and bandwidth, interconnect bandwidth per chip — for exactly what this page uses them for: topology reasoning and capacity sizing (does a 175B-parameter model fit on inf2.48xlarge; does the chip-to-instance-to-UltraServer math check out). Do not use AWS's comparative percentage or multiplier claims (price-performance vs. GPU instances, generation-over-generation speedup factors) as a decision input on their own; treat every one of them as provisional. The evidence that would settle the second question is specific and currently missing: an InferenceX Trainium backend (Google TPU support is already further along on the same roadmap, so the infrastructure and precedent exist) or an AWS MLPerf Training/Inference submission. Until one of those appears, the honest position is that AWS's own numbers are the only numbers, not that they are necessarily wrong — the gap documented in §9 is an absence of evidence, and this page has been explicit throughout about where that absence changes how much weight a number should carry.
References
- officialAWS. Inferentia Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inferentia.html. Accessed 2026-09-17. Used for: Inferentia1/NeuronCore-v1 chip compute (128 INT8 TOPS, 64 FP16/BF16 TFLOPS), 8 GiB DDR4 device memory at 50 GiB/sec, and the absence of a documented Programmability block for NeuronCore-v1.
- officialAWS. Inferentia2 Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inferentia2.html. Accessed 2026-09-17. Used for: Inferentia2/NeuronCore-v2 chip compute (380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, 47.5 FP32 TFLOPS), 32 GiB HBM at 820 GiB/sec, and NeuronCore-v2's Programmability block (dynamic shapes/control flow, GPSIMD custom operators).
- officialAWS. Trainium Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium.html. Accessed 2026-09-17. Used for: Trainium1/NeuronCore-v2 chip compute and memory (identical to Inferentia2's core numbers), 1 TB/sec DMA bandwidth, NeuronLink-v2, and rounding-mode/GPSIMD programmability.
- officialAWS. Trainium2 Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html. Accessed 2026-09-17. Used for: Trainium2/NeuronCore-v3 chip compute and memory (1,299 FP8/667 BF16/2,563 sparse/181 FP32 TFLOPS, 96 GiB HBM at 2.9 TB/sec), NeuronLink-v3 (1.28 TB/sec/chip), 16 CC-Cores, SBUF 224 MiB, Logical NeuronCore Configuration, and the full Trainium-to-Trainium2 "performance improvements" table (Table 4).
- blogAWS Machine Learning Blog. AWS Inferentia2 builds on AWS Inferentia1 by delivering 4x higher throughput and 10x lower latency. Amazon Web Services, 13 June 2023. https://aws.amazon.com/blogs/machine-learning/aws-inferentia2-builds-on-aws-inferentia1-by-delivering-4x-higher-throughput-and-10x-lower-latency/. Accessed 2026-09-17. Used for: the 4x throughput/10x latency/16.4x memory-bandwidth claims, DDR4-to-HBM framing, and the 175B-parameter multi-chip deployment example.
- officialAWS. Amazon EC2 Inf1 Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-hardware/inf1-arch.html. Accessed 2026-09-17. Used for: the Inf1 instance-size table (xlarge/2xlarge/6xlarge/24xlarge chip count, vCPUs, memory, NeuronLink-v1, EFA bandwidth), Table 2.
- officialAWS. Amazon EC2 Inf2 Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-hardware/inf2-arch.html. Accessed 2026-09-17. Used for: the Inf2 instance-size table (xlarge/8xlarge/24xlarge/48xlarge chip count, vCPUs, memory, NeuronLink-v2), Table 3.
- officialAWS. Amazon EC2 Trn1/Trn1n Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-hardware/trn1-arch.html. Accessed 2026-09-17. Used for: Trn1.2xlarge (1 chip) to trn1.32xlarge/trn1n.32xlarge (16 chips, 2D torus) scaling and EC2 UltraCluster scale-out to over 100,000 Trainium chips via EFA.
- officialAWS. Memory Hierarchy (NKI concepts guide: the Trainium memory hierarchy and NeuronCore on-chip memory). AWS Neuron Documentation (NKI), 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/get-started/about/memory-hierarchy-overview.html. Accessed 2026-09-17. Used for: the software-managed (no hardware cache) memory model, SBUF/PSUM roles and ~20x SBUF-vs-HBM bandwidth, the four NeuronCore compute engines (Tensor/Vector/Scalar/GpSimd), and per-generation SBUF/PSUM capacities (24 MiB/2 MiB for NeuronCore-v2, 28 MiB/2 MiB for NeuronCore-v3).
- officialAWS. Trainium2 Architecture Guide for NKI. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/architecture/trainium2_arch.html. Accessed 2026-09-17. Used for: Trainium2 device composition (8 NeuronCore-v3, 4 HBM stacks, 128 DMA engines, 20 CC-Cores per this page), and the double-throughput FP8 matmul mode (FP8_E4/FP8_E5 inputs at 2x BF16/FP16 rate, FP32 accumulation) that explains Trainium2's roughly 2x FP8-to-BF16 compute ratio.
- officialAWS. Amazon EC2 Trn2 Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-hardware/trn2-arch.html. Accessed 2026-09-17. Used for: trn2.48xlarge's 16-chip 4x4 2D torus (Figure 3), the Trn2 UltraServer's 64-chip ring-of-tori topology (Figure 4), and the full trn2.48xlarge/UltraServer instance-specification table (Table 5).
- officialAWS. Amazon EC2 Trn2 Instances (product page). Amazon Web Services, 2026. https://aws.amazon.com/ec2/instance-types/trn2/. Accessed 2026-09-17. Used for: the UltraServer's 12.8 Tbps aggregate EFAv3 figure (confirming the per-instance table entry) and AWS's 30-40% price-performance and 3x energy-efficiency comparative claims.
- officialAWS. Trainium3 Architecture. AWS Neuron Documentation, 2026. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium3.html. Accessed 2026-09-17. Used for: Trainium3/NeuronCore-v4 chip compute and memory (2,517 MXFP8/MXFP4, 671 BF16, 183 FP32 TFLOPS; 144 GiB HBM at 4.9 TB/sec), NeuronLink-v4 (2.56 TB/sec/device), 16 CC-Cores, and the full Trainium2-to-Trainium3 performance-improvement table (Table 6).
- officialAWS. Announcing Amazon EC2 Trn3 UltraServers for faster, lower-cost generative AI training. AWS What's New, 2 December 2025. https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-ec2-trn3-ultraservers/. Accessed 2026-09-17. Used for: Trainium3 chip specs (2.52 PFLOPs FP8, 144 GB HBM3e, 4.9 TB/sec, 3 nm process), Trn3 UltraServer specs, NeuronSwitch-v1, and AWS's comparative claims vs. Trainium2.
- standardMLCommons. MLCommons Releases New MLPerf Inference v5.0 Benchmark Results. 2 April 2025. https://mlcommons.org/2025/04/mlperf-inference-v5-0-results/. Accessed 2026-09-17. Used for: the full named list of 23 MLPerf Inference v5.0 submitting organizations, confirming AWS's absence.
- standardMLCommons. MLCommons Releases New MLPerf Inference v5.1 Benchmark Results. 9 September 2025. https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/. Accessed 2026-09-17. Used for: the full named list of 27 MLPerf Inference v5.1 submitting organizations (a record), confirming AWS's absence while Azure appears.
- analysisSemiAnalysis. InferenceMAX: Open Source Inference Benchmarking. October 2025. https://newsletter.semianalysis.com/p/inferencemax-open-source-inference. Accessed 2026-09-17. Used for: InferenceMAX's launch hardware coverage, the "expanding... to include Google TPU and AWS Trainium backends" within-two-months statement, and (in the disagree section) the note that TRT-LLM-tuned MLPerf submissions diverge from open-source-framework measurements.
- repoSemiAnalysisAI. InferenceX: Open Source Continuous Inference Benchmark Research Platform (GitHub repository, formerly InferenceMAX). 2026. https://github.com/SemiAnalysisAI/InferenceX. Accessed 2026-09-17. Used for: the current (2026) hardware-support roadmap — GB300/GB200 NVL72, MI355X, B300, B200, MI325X, MI300X, H200, H100 supported; TPUv7x Ironwood, MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8 listed "coming soon"; AWS Trainium absent from the structured roadmap.
- standardMLCommons. New MLCommons MLPerf Training v5.0 Benchmark Results Reflect Rapid Growth and Evolution of the Field of AI. 4 June 2025. https://mlcommons.org/2025/06/mlperf-training-v5-0-results/. Accessed 2026-09-17. Used for: the full named list of 20 MLPerf Training v5.0 submitting organizations, confirming AWS's absence.
- standardMLCommons. MLCommons Releases MLPerf Training v5.1 Results. 12 November 2025. https://mlcommons.org/2025/11/training-v5-1-results/. Accessed 2026-09-17. Used for: the full named list of 20 MLPerf Training v5.1 submitting organizations, confirming AWS's absence in the most recent round.
- analysisSemiAnalysis. AMD vs NVIDIA Inference Benchmark: Who Wins? Performance & Cost Per Million Tokens. 2025. https://newsletter.semianalysis.com/p/amd-vs-nvidia-inference-benchmark-who-wins-performance-cost-per-million-tokens. Accessed 2026-09-17. Used for: the independently measured 14-22% achieved-vs-peak GEMM throughput gap on AMD MI300X vs. H100/H200, cited in the disagree section as the basis for discounting unverified vendor comparative claims.