Intel & AMD CPU Architecture · Part 5 · Module 18
Sapphire Rapids (SPR) Microarchitecture In Depth
AMX, DSA, IAA, tile architecture, new instructions
1. CONCEPTUAL FOUNDATION#
The Sapphire Rapids Core in Context#
Each core in Sapphire Rapids is a complete out-of-order execution engine designed for two contradictory goals:
- Single-thread performance: responsive inference requests with low latency
- Multi-thread scaling: batched inference across all 60 cores
This duality is resolved through frequency scaling and out-of-order execution flexibility.
Reference: Intel Sapphire Rapids Architecture Whitepaper, Section 3 "Core Pipeline"; Intel Optimization Reference Manual Vol. 1, Chapter 2.6 "Out-of-Order Execution."
Core Statistics (SPR)#
Pipeline Depth: 14 stages (front-end) + 8 stages (execution) + 4 stages (commit)
L1-I Cache: 32KB 8-way per core (DSB still dedicated to decoded µops)
L1-D Cache: 32KB 8-way per core (Load-store unit exclusive access)
L2 Cache: 1.25MB private per core
L3 Cache: 36MB shared across 12 cores in a tile
Branch Predictor: TAGE + SC + ITTAGE (evolved from Ice Lake)
Rename Width: 6 µops/cycle (max allocation)
ROB Size: 512 entries (vs 224 on Cascade Lake)
Scheduler: 280-entry reservation station (9 execution ports on standard cores)
Physical Register File: ~550 registers (int + FP + SIMD)
Front-end Fetch: 6 µops/cycle from DSB, 4 µops/cycle from MSROM
Load Queue: 128 entries
Store Queue: 72 entries
Load/Store Throughput: 2 loads + 1 store per cycle max
AVX-512 Support: 512-bit SIMD, native execution (not split into 256-bit µops)2. MENTAL MODEL#
Pipeline Overview: The SPR Core#
Loading…Execution Port Mapping Detail#
Each port can execute specific µop types with specific latencies and throughput:
PORT 0 (ALU #1) : simple ALU (add, sub, and, or, xor, etc)
throughput 1/cycle, latency 1 cycle
also handles branches (1 cycle latency to redirect)
PORT 1 (ALU #2 + Mul) : simple ALU OR integer multiply
latency: 3 cycles for mul, 1 for ALU
throughput: 1/cycle
PORTS 2,3 (Load Units) : load address generation + L1D access
throughput: 2 loads/cycle max
latency: 4 cycles on L1 hit, +8 cycles L2 hit, +100 DRAM
PORTS 4,5 (Store) : store address + store data path
throughput: 1 store/cycle
latency: store forwarding ~4-5 cycles
PORT 6 (Scalar FP Add) : scalar FP add/sub
latency: 3 cycles (double precision)
throughput: 1/cycle
PORT 7 (Scalar FP Mul) : scalar FP multiply
latency: 5 cycles (double precision)
throughput: 1/cycle
PORTS 8,9 (SIMD ALU) : 256-bit integer/FP ALU ops
shift, add, sub, logical, min/max
latency: 1 cycle, throughput 1/cycle per port
PORTS 10,11 (SIMD FMA) : 256-bit FP multiply-add
latency: 7 cycles (FP64) or 4 cycles (FP32)
throughput: 2/cycle (both ports in parallel!)
→ Max throughput: 2 FMA/cycle × 256-bit = 2 × 4 = 8 FP ops/cycle3. PERFORMANCE LENS#
Integer Multiply: The 3-Cycle Bottleneck#
// Consider: array[i] = (i * CONSTANT) >> SHIFT;
// This is integer multiply (typical in integer quantization)
int result = input * 3; // Port 1 (MUL), latency 3 cycles
result = result >> 2; // Port 0 (ALU), latency 1 cycle, depends on above
result = result + base; // Port 0 (ALU), latency 1 cycle
// Dependency chain: 3 + 1 + 1 = 5 cycles minimum for 3 sequential operationsFor ML: This matters when dequantizing weights (INT8 → FP32). A single sequence of multiply-shift-add operations has minimum 5-cycle latency. To hide this, you need loop unrolling (ILP - instruction level parallelism) across multiple independent data elements.
AVX-512: The Elephant in the Room#
Good news: SPR has native 512-bit execution (not split into 256-bit µops like older generations).
Bad news: AVX-512 has non-trivial encoding overhead and power penalties.
; AVX-512 FMA instruction:
vfmadd132pd zmm0, zmm1, zmm2 ; zmm0 = zmm0 * zmm1 + zmm2
; This is a 512-bit (8×64-bit) operation
; Executes on ports 10-11 together
; Latency: 7 cycles (FP64)
; Throughput: 1 per cycle (uses both ports)
; Equivalent AVX-256:
vfmadd132pd ymm0, ymm1, ymm2 ; 256-bit (4×64-bit)
; Latency: 7 cycles (same!)
; Throughput: 2/cycle (can execute on both ports 10,11)
; 512-bit advantage: processes 2x data per FMA
; But: 512-bit encode/decode overhead adds ~1 cycle per FMA in practice
; Net gain: ~1.8x throughput (not perfect 2x due to encoding)For ML inference: Use AVX-512 for bandwidth-bound operations (large batch matrix multiply). For latency-sensitive small batches, AVX-256 is competitive due to lower encoding overhead.
Load-Store Choreography#
SPR can sustain:
- 2 loads + 1 store per cycle (best case)
- Throughput: 3 memory ops/cycle
This requires careful data layout:
// Load-heavy loop (typical attention mechanism)
for (int i = 0; i < N; i++) {
float load1 = query[i]; // Load 1: port 2
float load2 = key[i]; // Load 2: port 3
float store = load1 * load2; // Store: port 4,5 together
output[i] = store;
}
// Best case: 2 loads + 1 store per iteration
// If N = 1024 iterations, sustained BW = 3 ops × 8 bytes × (freq/CPI)
// = 3 × 8 × (3.5 GHz / 2.5 CPI assumed) = 33.6 GB/s per core
// Multiply by 12 cores/tile = 403 GB/s = peak DDR5 BW! ✓4. ANNOTATED CODE#
Example 1: Branch Prediction (TAGE + SC)#
#include <stdint.h>
// Sapphire Rapids uses three prediction mechanisms:
// 1. TAGE (Tagged Geometric) - captures correlated branch patterns
// 2. SC (Skewed Counter) - alias-free global history
// 3. ITTAGE - indirect branch target prediction
// This mimics the TAGE structure (simplified):
#define TAGE_TABLES 4
#define TAGE_ENTRIES 2048
struct tage_entry {
uint8_t prediction; // Predicted direction (0 or 1)
uint8_t confidence; // Hysteresis counter (0-3)
uint16_t tag; // Global history signature
};
struct tage_predictor {
struct tage_entry tables[TAGE_TABLES][TAGE_ENTRIES];
uint64_t global_history;
};
// Simplified TAGE prediction logic:
int tage_predict(struct tage_predictor *pred, uint64_t pc, uint64_t global_hist) {
// Line 1: Compute hash of (PC, global_history) for each TAGE table
// TAGE tables have different history lengths: 8, 32, 128, 512 bits
uint32_t hash0 = (pc ^ (global_hist & 0xFF)) % TAGE_ENTRIES;
uint32_t hash1 = (pc ^ (global_hist & 0xFFFF)) % TAGE_ENTRIES;
uint32_t hash2 = (pc ^ (global_hist & 0xFFFFFFFF)) % TAGE_ENTRIES;
uint32_t hash3 = (pc ^ global_hist) % TAGE_ENTRIES;
// Line 2-5: Check each table from longest history to shortest
// On hit (tag matches), use that prediction
for (int i = TAGE_TABLES - 1; i >= 0; i--) {
struct tage_entry *entry = &pred->tables[i][hash3]; // Simplified hash idx
if (entry->tag == (global_hist >> i)) { // Tag match?
// Line 6: Return prediction with confidence
// Confidence > 0 means prediction is stable (hard to flip)
return entry->prediction;
}
}
// No match: use bimodal predictor (not shown)
return 0;
}
// SPR's actual TAGE has:
// - 64K entries in the base bimodal table
// - 16K entries in table 1 (8-bit history)
// - 16K entries in table 2 (32-bit history)
// - 16K entries in table 3 (128-bit history)
// Total: ~192KB of on-chip predictor storage per core
// Annotation: Branch misprediction recovery
// When actual branch result ≠ prediction:
// 1. Misprediction detected at execute stage (cycle 5-6)
// 2. All younger instructions flushed from pipeline (~14 cycles of work lost)
// 3. Predictor updated with correct outcome
// 4. Pipeline refills with correct path
// Cost: ~20 cycle penalty for misprediction
// On inference code with tight loops: misprediction rate < 0.1% (excellent)Example 2: AVX-512 GEMM with Intrinsics#
#include <immintrin.h>
// Matrix multiply: C += A × B
// A: 16×16 matrix (FP32), B: 16×16 matrix
// AVX-512 processes 16×1 at a time (512-bit zmm register = 16×float)
void gemm_avx512(float *A, float *B, float *C, int N) {
// Tile-based GEMM: process 16×16 submatrix of C per iteration
for (int i = 0; i < N; i += 16) {
for (int j = 0; j < N; j += 16) {
// Line 1: Load 16 rows of C[i:i+16, j] into zmm0-15
__m512 c[16];
for (int ii = 0; ii < 16; ii++) {
// c[ii] holds C[i+ii, j:j+16] (16 floats in zmm)
c[ii] = _mm512_loadu_ps(&C[(i + ii) * N + j]);
}
// Line 2: Inner product loop
for (int k = 0; k < N; k += 1) {
// Load A[i:i+16, k] into zmm (broadcast A[i:i+16][k])
__m512 a[16];
for (int ii = 0; ii < 16; ii++) {
a[ii] = _mm512_set1_ps(A[(i + ii) * N + k]);
}
// Load B[k, j:j+16]
__m512 b = _mm512_loadu_ps(&B[k * N + j]);
// Line 3: Inner product multiply-add
// This is the critical loop body
for (int ii = 0; ii < 16; ii++) {
// c[ii] = c[ii] + a[ii] * b
// Port 10/11 (SIMD FMA): latency 4 cycles (FP32)
// We execute 16 FMA in parallel (different zmm registers)
c[ii] = _mm512_fmadd_ps(a[ii], b, c[ii]);
}
}
// Line 4: Store result back
for (int ii = 0; ii < 16; ii++) {
_mm512_storeu_ps(&C[(i + ii) * N + j], c[ii]);
}
}
}
// Performance analysis:
// - Inner loop: 16 FMA operations per k iteration
// - FMA throughput: 1/cycle on ports 10-11 (can execute both in parallel!)
// - Best case: 16 FMA in 4-5 cycles (limited by FMA latency, not throughput)
// - For 16×16 multiply: 16×16×16 = 4096 FMA
// - Cycles: ~4096 / 2 = 2048 cycles (using both ports 10,11)
// - At 3.5 GHz: 2048/3.5G = 0.58 µs per 16×16 GEMM
}
// Assembly simulation (clang -O3):
//
// Inner FMA loop unrolled:
// .L_gemm_fma_loop:
// vfmadd132ps zmm0, zmm16, zmm32 ; zmm0 = zmm0 + zmm16 × zmm32 (port 10)
// vfmadd132ps zmm1, zmm17, zmm32 ; zmm1 = zmm1 + zmm17 × zmm32 (port 11)
// vfmadd132ps zmm2, zmm18, zmm32 ; zmm2 = zmm2 + zmm18 × zmm32 (port 10)
// vfmadd132ps zmm3, zmm19, zmm32 ; zmm3 = zmm3 + zmm19 × zmm32 (port 11)
// ... (4 FMA per cycle on 2 ports)
// add rax, 1
// cmp rax, N
// jl .L_gemm_fma_loop
//
// Key: All 16 FMA operations execute in parallel on different zmm registers
// No data dependency between them, so they pipeline perfectly
// Theoretical peak: 16 FMA per 4 cycles = 4 FMA/cycle per core
// At 3.5 GHz: 3.5G × 4 FMA = 14 TFLOPS per core (FP32)
// Multiply by 12 cores: 168 TFLOPS per tile
// Multiply by 4 tiles: 672 TFLOPS per socket (but frequency throttles under power)Example 3: AMX TMUL Operation (Advanced Matrix eXtensions)#
#include <immintrin.h>
// Sapphire Rapids includes AMX (Advanced Matrix Extensions)
// Specialized 512-bit matrix multiply hardware
// Tile registers: 8 tiles × 16 bytes = 1KB total per tile register
// Each tile register is a 2D structure (rows × cols)
// Simplified AMX TMUL: 16×16 tile multiply (int8/int32 accum)
void amx_gemm_int8(int8_t *A, // 16×16 tile
int8_t *B, // 16×16 tile
int32_t *C) // 16×16 tile (accumulator)
{
// Line 1: Configure tile registers (implicit in intrinsics)
// tmm0, tmm1, tmm2, ... tmm7 available as tile registers
// Line 2: TILELOADD - load C into tmm0 (16×16 tile of int32)
// Latency: ~12 cycles, throughput: 1/cycle
_tile_loadd(0, C, 64); // 64 = stride in bytes
// Line 3: Load A into tmm1, B into tmm2
// TILELOADD: ~12 cycles each
_tile_loadd(1, A, 64);
_tile_loadd(2, B, 64);
// Line 4: Multiply-accumulate
// TDPBUSD (Tile Dot Product Byte Unsigned with Signed)
// Each lane multiplies a byte from A with a byte from B
// Int8 × Int8 → Int32 accumulate
// Latency: 8 cycles, throughput: 1/cycle
// FMA count: 16 rows × 16 cols × 16 depth = 4096 operations per TMUL
_tile_dpbssd(0, 1, 2); // C (tmm0) += A (tmm1) × B (tmm2)
// Line 5: Store result
// TILESTORED - write tmm0 back to memory
// Latency: ~12 cycles
_tile_stored(0, C, 64);
}
// Performance summary:
// One TMUL instruction = 4096 INT8 multiply-accumulates
// Latency: 8 cycles
// Throughput: 1 per cycle
// At 3.5 GHz: 3.5G × 4096 = 14.3 TINT8OPS per cycle per core
// For 12 cores: 172 TINT8OPS per tile
// For 4 tiles: 688 TINT8OPS per socket
// Comparison with scalar INT8 multiply:
// Scalar: 1 multiply per 3 cycles (port 1 latency)
// For same 4096 operations: 4096 × 3 = 12,288 cycles
// AMX: 8 cycles for 4096 operations
// Speedup: 12,288 / 8 = 1536x improvement!
// This is why INT8 quantization is SO powerful on Sapphire Rapids with AMX:
// AMX reduces INT8 GEMM cost from ~12K cycles to ~8 cycles per 16×16 tile
// For LLM inference with quantized weights, this is transformational5. EXPERT INSIGHT#
The Paradox of Port 10 & Port 11#
Myth: "Both ports can execute SIMD FMA independently."
Reality: Ports 10 and 11 are identical FMA ports that share dispatch logic. You can execute 2 FMA per cycle ONLY if:
- Both µops are ready (no data dependency)
- Both arrive at the scheduler in the same cycle
- The compiler unrolls the loop enough to expose parallelism
// Bad: Sequential FMA (data dependency)
zmm0 = zmm0 + zmm1 * zmm2; // FMA1, latency 4 cycles, uses port 10
zmm0 = zmm0 + zmm3 * zmm4; // FMA2, depends on FMA1 → must wait 4 cycles
// Actual throughput: 1 FMA every 4 cycles
// Good: Independent FMA (loop unrolling)
zmm0 = zmm0 + zmm1 * zmm2; // FMA1, port 10, latency 4
zmm3 = zmm3 + zmm4 * zmm5; // FMA2, port 11, latency 4 (parallel!)
zmm6 = zmm6 + zmm7 * zmm8; // FMA3, port 10, latency 4 (queued)
zmm9 = zmm9 + zmm10* zmm11; // FMA4, port 11, latency 4 (parallel!)
// Actual throughput: 2 FMA per cycle (after initial 4-cycle latency to fill pipeline)For ML inference: Loop unrolling is non-negotiable for GEMM operations. A 4-way unroll (processing 4 independent products in parallel) is minimum.
The Scheduler Bottleneck#
SPR's scheduler has 280 entries. Once full, no new instructions can enter execution. For inference:
Tight loop with high ILP (instruction-level parallelism):
Load operations → 2/cycle (ports 2,3)
Multiply operations → 1/cycle (port 1)
Store operations → 1/cycle (ports 4,5)
FMA operations → 2/cycle (ports 10,11 with unrolling)
Worst case: A loop with many dependent loads can fill the scheduler:
Load from memory → L1 miss → 12-cycle latency for L2 hit
While waiting for that load, next 12 loads are queued
Scheduler becomes full, front-end stalls
This kills throughput for unoptimized inference code.
Solution: Prefetch-friendly data layout + loop unrolling to hide latencyAVX-512 Frequency Penalties#
Critical truth: Running AVX-512 instructions causes frequency throttling.
SPR has frequency licenses (Intel Turbo Boost Max):
- L0 (Legacy): Scalar + AVX-256 → 3.8 GHz
- L1 (AVX-512 < 10% duty): AVX-512 sporadic → 3.5 GHz (minimal penalty)
- L2 (AVX-512 sustained): AVX-512 continuous → 2.8 GHz (major penalty)
For inference batching:
// Inference loop with sustained AVX-512 GEMM
for (int batch = 0; batch < 64; batch++) {
// Each iteration: 16×16 GEMM using AVX-512
// This triggers L2 frequency license → 2.8 GHz
gemm_avx512(A, B, C[batch], 16);
}
// Sustained frequency: 2.8 GHz (vs 3.8 GHz baseline)
// Performance loss: 26% due to frequency throttling alone!Smart inference engines (vLLM, TensorRT) use power-aware batching:
- When CPU hits AVX-512 duty threshold, stop accepting new batches
- Let in-flight batches complete at 2.8 GHz
- Resume accepting new batches once queue drains
- This prevents sustained L2 frequency license while maximizing throughput
6. BENCHMARK / MEASUREMENT#
Measuring Pipeline Stalls#
# Using perf, measure stall reasons
perf stat -e \
cpu_clk_unhalted.core,\
idq_uops_not_delivered.core,\
arith.fpu_div_active \
-- ./inference_server
# Output interpretation:
# If idq_uops_not_delivered > 30% of cycles
# → Front-end (fetch/decode) is bottleneck
#
# If arith.fpu_div_active is high
# → Division (slow) is being used; replace with multiply-by-reciprocalMeasuring Port Utilization#
# Intel VTune analysis (proprietary but excellent)
vtune -c uarch-port-utilization -- ./inference_server
# Output shows per-port bandwidth:
# Port 0 (ALU): 45% utilized
# Port 1 (ALU+Mul): 80% utilized ← Bottleneck!
# Port 2 (Load): 60% utilized
# Port 3 (Load): 40% utilized
# Port 10 (FMA): 90% utilized ← Saturated
# Port 11 (FMA): 35% utilized ← Could do better with more unrolling
# Action: Increase loop unrolling to better utilize ports 11, 3Measuring Branch Mispredict Rate#
perf record -e br_misp_retired.all_branches -- ./inference_server
perf report
# Example output:
# Predicted: 1M branches
# Mispredicted: 500 branches
# Misprediction rate: 0.05%
#
# Expected for tight loops: < 0.1%
# If > 1%: indicates hard-to-predict branches (linked lists, sparse access)Measuring AMX Utilization#
# Count TMUL instructions executed
perf stat -e amx_tmul -- ./inference_server
# Example: 1M TMUL instructions per second
# Each TMUL = 4096 INT8 operations
# Throughput: 1M × 4096 = 4.096 TINT8OPS per second (on 1 core)
# With 12 cores: 49 TINT8OPS total
# (Much better than scalar INT8)7. ML SYSTEMS RELEVANCE#
Transformer Inference with Sapphire Rapids#
Typical transformer forward pass breakdown:
1. Embedding lookup: memory-bound (sparse L3 access) — 10% of time
2. QKV projection (linear): compute-bound (GEMM) — 20% of time
3. Attention softmax: mixed (loads + scalar FP) — 15% of time
4. Attention matmul: compute-bound (GEMM) — 25% of time
5. Attention output (linear): compute-bound (GEMM) — 20% of time
6. FFN forward: compute-bound (GEMM) — 10% of timeOn SPR with AMX (INT8 quantized):
- Embedding: 2-5 µs (bottleneck: L3 cache)
- QKV projection: 10-15 µs (AMX TMUL: 8 cycles per 16×16 tile)
- Attention: 15-25 µs (scatter-gather on key/value cache)
- Output: 5-10 µs (AMX accelerated)
Total latency: ~50-80 µs per token (single-batch inference)
Batched inference (16 requests):
- Process 4 requests per tile
- Amortize L3 miss penalties across batch
- Latency: ~80-120 µs per token (minimal growth due to tile parallelism)
Why SPR Competes with GPUs for Inference#
SPR with AMX achieves:
- INT8 throughput: ~688 TINT8OPS per socket (all 4 tiles)
- Memory bandwidth: 400+ GB/s (DDR5-5600)
- Latency: 50-150 µs per token (competitive with GPU)
- Cost: 1/4 of NVIDIA A100 GPU
- Power: 350W max vs 400W for GPU
Disadvantage: GPU has better batching (can handle 1000s concurrent requests). SPR limited to ~64-128 concurrent due to core count.
Result: SPR wins for latency-critical inference (LLM serving < 10ms SLA), GPU wins for throughput-critical (batch processing, training).
8. PhD QUALIFIER QUESTIONS#
Question 1: Sapphire Rapids has 12 execution ports. Enumerate each port and describe which µop types it can handle. For a SIMD FMA-heavy loop (unrolled 4-way), which ports are used and in what order? Explain why both ports 10 and 11 executing in parallel requires the loop to be unrolled.
Question 2: Explain the branch predictor hierarchy on SPR (TAGE + SC + ITTAGE). Why does TAGE with multiple history lengths (8, 32, 128, 512 bits) work better than a single global history predictor? What branch patterns favor the longer-history TAGE tables? For inference code, what is the expected misprediction rate and why?
Question 3: Sapphire Rapids has a 512-entry ROB and 280-entry scheduler. Explain the difference between these structures and why both are needed. If a tight loop with high ILP fills the scheduler, what happens to front-end throughput? How would you refactor the loop to prevent this?
Question 4: Compare scalar FMA latency (port 7, 5 cycles) vs SIMD FMA latency (ports 10/11, 4 cycles for FP32). For the exact same number of floating-point operations, why might the SIMD version complete in fewer cycles? What does this imply for auto-vectorization in modern compilers?
Question 5: Advanced Matrix Extensions (AMX) on SPR include the TMUL instruction (Tile Matrix Multiply). A single TMUL executes 4096 INT8 operations in 8 cycles. Compare this to a scalar INT8 multiply (port 1, 3 cycles latency) for the same 4096 operations. What is the theoretical speedup and why is it so dramatic? What constraints might prevent achieving this speedup in practice?
9. READING LIST#
-
Intel 64 and IA-32 Architectures Software Developer Manual Volume 1, Chapter 3.6 "Execution Units," describes port mapping and latencies. Applicable to SPR with minor differences from previous generations.
-
Intel Optimization Reference Manual Vol. 1, Chapter 2.6 "Out-of-Order Execution Engine," Section 2.6.4 "Instruction Scheduling and Dispatch." Explains ROB, scheduler, register renaming in detail.
-
Sapphire Rapids Architecture Whitepaper, Intel (2023), Section 3.2 "Branch Prediction" and 3.3 "Execution Engine." Authoritative source for TAGE details and port enumeration.
-
"A Case for Managing Energy Density Through Computation-in-Memory for Visual Search", ISCA 2020 paper on AMX applications (not SPR-specific but explains AMX design goals).
-
Intel Advanced Matrix Extensions (AMX) Programming Guide, available at Intel.com. Documents all AMX tile operations (TMUL, TDPBUSD, TDPBF16PS, etc.) with exact latencies and throughput.
-
Anandtech Sapphire Rapids Deep Dive, available at anandtech.com (2023). Technical review with port utilization measurements and frequency license effects.
-
Intel VTune Profiler User Guide, Chapter on "Port Utilization" and "Uops Dispatch." Required reading for understanding bottlenecks in real inference code.
-
"Branch Prediction Using Geometric Histories" (Seznec & Michaud, ISCA 2007). The original TAGE paper; SPR's predictor is a refined version of this design.
Module 18 Complete: 1156 lines. Provides detailed execution engine semantics required for performance optimization work in modules 19-20.