Intel & AMD CPU Architecture · Part 4 · Module 14

Performance Engineering Framework

Top-down analysis, bottleneck identification, roofline methodology

25 min read

1. CONCEPTUAL FOUNDATION#

Modern CPU performance analysis has moved from single-metric thinking to multi-dimensional characterization. The field has crystallized around three orthogonal frameworks that together provide complete insight into why a kernel executes at a particular speed.

The Roofline Model#

The Roofline Model (Williams et al., CACM 2009) defines achievable performance as the minimum of two physical ceilings:

Text
Achievable GFLOPs = min(Peak_Compute, Peak_Bandwidth × Arithmetic_Intensity)

Where:

  • Peak_Compute = theoretical peak FP operations per second (e.g., Sapphire Rapids: 1.5 TFLOPs for AVX-512 FP64)
  • Peak_Bandwidth = memory bandwidth to L1/L2/L3/DRAM (e.g., Sapphire Rapids: 12.8 GB/s per core to L1)
  • Arithmetic_Intensity (AI) = FLOPs per byte of data moved from a reference level (usually DRAM)

For a SpMM kernel loading a sparse matrix once and executing 2M FLOPs per M bytes:

Text
AI = 2M FLOPs / M bytes = 2 FLOPs/byte

At 2 FLOPs/byte on a 12.8 GB/s DDR5 link:

Text
Achievable = min(1500 GFLOPs, 12.8 GB/s × 2) = min(1500, 25.6) GFLOPs = 25.6 GFLOPs

This kernel is memory-bandwidth-bound, not compute-bound. No amount of vectorization helps; only algorithm changes (like dataflow tiling to reuse data) will improve performance.

Reference: Williams et al., Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures, CACM 52(4), April 2009. See also Dendibakh, Performance Analysis and Tuning on Modern CPUs, Ch. 6.

Top-Down Microarchitecture Analysis (TMA)#

Intel's Top-Down Microarchitecture Analysis (Yasin et al., ISPASS 2014) decomposes CPU stalls into a four-level hierarchy:

Diagram
Loading…

The critical insight: a cycle not executing useful instructions is wasted. TMA tells you where those cycles go.

On Sapphire Rapids (and 12th-gen Intel), TMA 4.0 refines Backend_Bound further:

Diagram
Loading…

Reference: Yasin et al., Top-Down Microarchitecture Analysis Methodology, ISPASS 2014. Intel's official TMA documentation: https://www.intel.com/content/dam/develop/external/us/en/documents/tma-methodology.pdf

Working-Set Analysis and Cache Residency#

A critical but often-ignored analysis: does your data fit in cache?

Define:

  • Working Set Size (WSS) = unique memory locations touched per unit time
  • Cache Residency = fraction of accesses hitting a particular cache level

For an ML inference engine loading weight matrices:

  • Batch size 1, attention heads with 64×64 key/query matrices
  • If WSS × element_size < 32 MB (Sapphire Rapids L3), the entire attention head graph fits in L3
  • Every access hits L3 in ~40 cycles, not DRAM in ~300 cycles

Practical calculation:

C
// For transformer inference: estimate per-token working set
working_set_bytes = (hidden_dim * num_layers * 2) // weights + activations
                  + (seq_len * hidden_dim * 4)    // KV cache + attention
                  + (batch_size * seq_len * hidden_dim * 8) // activations

bool fits_in_l3 = (working_set_bytes < 51_200_000); // Sapphire Rapids L3

If true, your code should achieve ~3-4 GB/s per core (L3 bandwidth), not 12.8 GB/s (theoretical).

Little's Law Applied to Pipelines#

Little's Law: Average_In_System = Arrival_Rate × Response_Time

Applied to a CPU pipeline:

Text
Occupancy = (Throughput_per_cycle) × (Latency_in_cycles)

Example: An AVX-512 FP64 add has 3-cycle latency. If you dispatch adds to the same register every cycle (throughput = 1 add/cycle):

Text
Occupancy = 1 add/cycle × 3 cycles = 3 independent adds in flight

The pipeline is "half-empty"—you need 6 independent adds to keep it full (throughput = 2 adds/cycle on Sapphire Rapids' dual FP pipes).

Implication for ML inference: if your operation graph is a long chain of sequential operations (like autoregressive decoding), you're limited by latency, not throughput. Parallelization strategies (batching, speculative decoding) combat this.

Reference: Dendibakh, Ch. 2.

Latency-Bound vs Throughput-Bound Transformation#

Latency-bound code: Limited by the latency of the longest dependency chain.

  • Example: Autoregressive decoding where token t+1 depends on outputs of token t
  • Instruction-level parallelism (ILP) is the solution

Throughput-bound code: Limited by execution port saturation or cache bandwidth.

  • Example: Dense matrix multiply where many independent operations exist
  • Data-level parallelism (vectorization) and thread-level parallelism (multithreading) solve it

Transformation strategy:

  1. Measure: is your code latency-bound or throughput-bound? (TMA → Core_Bound → Ports_Utilization)
  2. If latency-bound: unroll loops, expose ILP, use prefetching to hide memory latency
  3. If throughput-bound: vectorize, multithread, increase data reuse (Roofline AI)

For ML systems: batch-size 1 inference is often latency-bound (long decoding chains). Batch-size 8+ becomes throughput-bound (matrix multiply bottleneck).


2. MENTAL MODEL#

Diagram
Loading…

3. PERFORMANCE LENS#

Understanding these frameworks directly impacts engineering decisions:

Roofline for Algorithm Selection:

  • Your quantized SpMM kernel computes 4 weights per 0.5 bytes (INT4 packed)
  • AI = 8 FLOPs/byte on a 25.6 GB/s memory roof → achievable 200 GFLOPs
  • Dense GEMM with AI > 8 is compute-bound at 1500 GFLOPs
  • Decision: Use quantized SpMM for sparse models (>80% sparsity), dense GEMM for dense models

TMA for Bottleneck Prioritization:

  • If TMA shows 45% Backend_Bound, 30% Memory_Bound (DRAM):
    • First, optimize memory access patterns (blocking, prefetch, data layout)
    • Only then optimize compute (vectorization, instruction-level parallelism)
  • If TMA shows 35% Frontend_Bound:
    • Reduce code complexity, improve branch prediction, tune for i-cache

Working-Set Analysis for Cache Tuning:

  • Transformer layer weights: 768 × 768 × 2 (fp16) = 1.15 MB per layer
  • Batch 1, 12 layers → 14 MB of weights + activations < 51.2 MB L3
  • Implication: all weights stay resident in L3, data reuse is dominated by on-chip bandwidth
  • Batch 16 → 224 MB, exceeds L3 → must optimize DRAM prefetch

Little's Law for Batching Decisions:

  • Single-token autoregressive: latency-bound, batch size 1 = 1 token in flight
  • Add speculative decoding: 5 tokens in flight × 5 ns latency = higher throughput
  • Add batching: 64 tokens × different sequences = 64× throughput improvement

4. ANNOTATED CODE#

Example 1: Measuring Roofline AI for a Kernel#

C
// File: roofline_measurement.c
// Compute peak theoretical roofline for a tensor operation
// Measure arithmetic intensity and compare to roofline ceiling

#include <immintrin.h>
#include <stdint.h>
#include <stdio.h>

// Sapphire Rapids: peak 1.5 TFLOPs FP64, 12.8 GB/s DRAM bandwidth
#define PEAK_COMPUTE_GFLOPs 1500.0  // GFLOPs
#define PEAK_BANDWIDTH_GBs 12.8     // GB/s

// Measure memory traffic using RAPL energy counters
typedef struct {
    uint64_t cycles;
    uint64_t instructions;
    uint64_t llc_loads;      // L3 cache line loads
    uint64_t llc_stores;     // L3 cache line stores
    uint64_t bytes_dram_rd;  // DRAM bytes read
    uint64_t bytes_dram_wr;  // DRAM bytes written
} perf_counters_t;

// Dense GEMM: C[M×N] += A[M×K] × B[K×N]
// Arithmetic intensity = 2MNK / (M*K + K*N + M*N) bytes, assuming all from DRAM
void gemm_f64(double *C, const double *A, const double *B,
              int M, int N, int K) {
    // Assuming optimal blocking where A, B fit in L3
    // But we count all loads from memory hierarchy

    for (int i = 0; i < M; i++) {
        for (int j = 0; j < N; j++) {
            double sum = C[i*N + j];  // L1 hit (reused 64 times)

            // Vectorized FMA: 4 FLOPs per iteration (AVX-512 would be 8)
            for (int k = 0; k < K; k += 4) {
                __m256d a = _mm256_loadu_pd(&A[i*K + k]);    // L1/L2 load
                __m256d b0 = _mm256_loadu_pd(&B[(k+0)*N + j]); // stride-N load
                __m256d b1 = _mm256_loadu_pd(&B[(k+1)*N + j]);
                __m256d b2 = _mm256_loadu_pd(&B[(k+2)*N + j]);
                __m256d b3 = _mm256_loadu_pd(&B[(k+3)*N + j]);

                sum = _mm256_fmadd_pd(a, b0, sum);  // 4 FMA = 8 FLOPs
                // ... (reuse A[i*K+k] across all j, high arithmetic intensity)
            }
            C[i*N + j] = sum;
        }
    }
}

// Analyze roofline for this GEMM
void analyze_roofline(int M, int N, int K, perf_counters_t *counters) {
    // FLOPs: 2*M*N*K (assuming fused multiply-add)
    double flops = 2.0 * M * N * K;

    // Bytes: count first-load from DRAM
    // Optimal case: A[M×K] loaded once, B[K×N] loaded once, C[M×N] loaded/stored
    // = (M*K + K*N + M*N) * 8 bytes (FP64)
    double bytes_estimate = (M*K + K*N + M*N) * 8.0;

    // Arithmetic intensity
    double ai = flops / bytes_estimate;

    // Achievable GFLOPs from roofline
    double gflops_memory_limited = PEAK_BANDWIDTH_GBs * ai;
    double achievable_gflops = (gflops_memory_limited < PEAK_COMPUTE_GFLOPs)
                               ? gflops_memory_limited
                               : PEAK_COMPUTE_GFLOPs;

    // Measure actual performance from counters
    double actual_gflops = flops / (counters->cycles / 3.0e9);  // assuming 3 GHz

    printf("Roofline Analysis:\n");
    printf("  FLOPs: %.2e\n", flops);
    printf("  Bytes (est.): %.2e\n", bytes_estimate);
    printf("  AI: %.2f FLOPs/byte\n", ai);
    printf("  Memory ceiling: %.0f GFLOPs (AI × 12.8 GB/s)\n", gflops_memory_limited);
    printf("  Compute ceiling: %.0f GFLOPs\n", PEAK_COMPUTE_GFLOPs);
    printf("  Roofline ceiling: %.0f GFLOPs\n", achievable_gflops);
    printf("  Actual measured: %.0f GFLOPs\n", actual_gflops);
    printf("  Efficiency: %.1f%% of roofline\n", 100.0 * actual_gflops / achievable_gflops);
}

Example 2: TMA-Guided Optimization Loop#

Diagram
Loading…

Example 3: Cache Residency Analysis for Transformer Layers#

C
// File: cache_residency_transformer.c
// Compute working set size and verify L3 residency

#include <stdint.h>
#include <stdio.h>
#include <math.h>

typedef struct {
    int hidden_dim;          // 768, 1024, etc.
    int num_heads;           // 12, 16, etc.
    int num_layers;          // 12, 24, etc.
    int batch_size;          // token batch
    int seq_len;             // context length
} transformer_config_t;

// Compute memory footprint for forward pass
typedef struct {
    uint64_t weights_bytes;   // All W_q, W_k, W_v, W_o, FFN weights
    uint64_t kv_cache_bytes;  // KV cache for all layers
    uint64_t activation_bytes; // Layer outputs
    uint64_t total_bytes;
} memory_footprint_t;

void compute_footprint(transformer_config_t *cfg, memory_footprint_t *footprint) {
    // Weights per layer (fp16 = 2 bytes)
    // W_q, W_k, W_v, W_o: 4 × (hidden_dim × hidden_dim)
    // W_ff1, W_ff2: (hidden_dim × 4*hidden_dim) + (4*hidden_dim × hidden_dim)
    uint64_t weights_per_layer =
        (4 * cfg->hidden_dim * cfg->hidden_dim +  // attention matrices
         cfg->hidden_dim * 4 * cfg->hidden_dim +   // FFN first
         4 * cfg->hidden_dim * cfg->hidden_dim) * 2; // FFN second (fp16)

    footprint->weights_bytes = weights_per_layer * cfg->num_layers;

    // KV cache: per layer, per head: (seq_len × head_dim)
    // head_dim = hidden_dim / num_heads
    int head_dim = cfg->hidden_dim / cfg->num_heads;
    uint64_t kv_per_layer = cfg->num_heads * seq_len * head_dim * 2 * 2; // K and V
    footprint->kv_cache_bytes = kv_per_layer * cfg->num_layers;

    // Activations: hidden states + attention scores + intermediate states
    uint64_t activation_per_layer =
        (cfg->batch_size * cfg->seq_len * cfg->hidden_dim +  // hidden state
         cfg->batch_size * cfg->num_heads * cfg->seq_len * cfg->seq_len) * 2; // attention
    footprint->activation_bytes = activation_per_layer * cfg->num_layers;

    footprint->total_bytes = footprint->weights_bytes +
                            footprint->kv_cache_bytes +
                            footprint->activation_bytes;
}

#define L3_CACHE_BYTES (51 * 1024 * 1024)  // Sapphire Rapids, 51 MB

void analyze_residency(transformer_config_t *cfg) {
    memory_footprint_t footprint;
    compute_footprint(cfg, &footprint);

    printf("Transformer Memory Analysis:\n");
    printf("  Config: hidden=%d, heads=%d, layers=%d, batch=%d, seq_len=%d\n",
           cfg->hidden_dim, cfg->num_heads, cfg->num_layers,
           cfg->batch_size, cfg->seq_len);
    printf("  Weights: %.1f MB\n", footprint.weights_bytes / 1e6);
    printf("  KV Cache: %.1f MB\n", footprint.kv_cache_bytes / 1e6);
    printf("  Activations: %.1f MB\n", footprint.activation_bytes / 1e6);
    printf("  Total: %.1f MB\n", footprint.total_bytes / 1e6);
    printf("  L3 Cache: %.1f MB\n", L3_CACHE_BYTES / 1e6);

    if (footprint.total_bytes < L3_CACHE_BYTES) {
        printf("  ✓ FITS IN L3: expect L3 latency (~40ns)\n");
        printf("    Effective bandwidth: 120 GB/s (L3 bandwidth)\n");
    } else if (footprint.weights_bytes < L3_CACHE_BYTES) {
        printf("  ✗ Weights fit in L3, but activations spill\n");
        printf("    → Reuse: only weights are cached\n");
        printf("    → Bottleneck: KV cache DRAM bandwidth\n");
    } else {
        printf("  ✗ DOES NOT FIT IN L3: expect DRAM latency (~300ns)\n");
        printf("    Effective bandwidth: ~1 GB/s per core (DRAM)\n");
    }

    // Estimate memory bandwidth utilization
    // Assuming compute is fast (not memory-bound), bytes per useful FLOP
    double weights_reuse = 1.0;  // each weight loaded once per token
    double bytes_per_token = footprint.weights_bytes / cfg->batch_size +
                            footprint.activation_bytes / cfg->batch_size;
    double flops_per_token = 2.0 * cfg->hidden_dim * cfg->hidden_dim * cfg->num_layers;
    double arithmetic_intensity = flops_per_token / bytes_per_token;

    printf("  Arithmetic Intensity: %.2f FLOPs/byte\n", arithmetic_intensity);
    printf("  (Higher = more compute-bound, lower = more memory-bound)\n");
}

int main() {
    // Test case: BERT-base (768-dim, 12 layers, 12 heads)
    transformer_config_t bert_base = {
        .hidden_dim = 768,
        .num_heads = 12,
        .num_layers = 12,
        .batch_size = 1,
        .seq_len = 512,
    };

    analyze_residency(&bert_base);
    printf("\n");

    // Test case: GPT-3 style (1024-dim inference)
    transformer_config_t gpt3_small = {
        .hidden_dim = 1024,
        .num_heads = 16,
        .num_layers = 24,
        .batch_size = 8,   // batched inference
        .seq_len = 2048,
    };

    analyze_residency(&gpt3_small);

    return 0;
}

5. EXPERT INSIGHT#

Non-Obvious Truth #1: Roofline Ceilings Are Soft#

Many junior engineers plot roofline and assume "I cannot do better than this line." False. Roofline assumes:

  • All memory bandwidth is equally accessible
  • Compute and memory overlap perfectly

In reality:

  • L3 bandwidth > DRAM bandwidth: your code might be "DRAM-bound" on roofline but achieve 200 GB/s (L3 roofline) through prefetch tuning
  • Overlap incomplete: if memory stalls block the pipeline before compute pipes fill, you hit compute ceiling before roofline predicts
  • Vectorization penalty: moving from scalar (4-cycle latency) to AVX-512 (same latency) doesn't improve roofline, but does reduce achievable throughput if dependency chains are long

Lesson: Use roofline as a lower bound ("we cannot exceed this"), not an upper bound. Expert optimization often exceeds the original roofline by changing memory layout, blocking, or vectorization strategy.

Non-Obvious Truth #2: TMA 4.0 Conceals High-Level Stalls#

TMA tells you which hardware unit is stalled, but not why you stalled it.

Example: TMA says "30% DRAM_Bound". Possible causes:

  1. Suboptimal prefetch: you're issuing loads sequentially (128-byte L2 prefetch latency)
  2. Too many pending requests: hardware prefetch is overloaded, loads queue in port buffers
  3. LLC miss pattern: your access pattern defeats the LLC's coherence predictor
  4. NUMA locality: in multi-socket systems, remote NUMA accesses cost 2-3× more

Fix: Measure finer granularity: perf stat -e cache-references,cache-misses,LLC-loads,LLC-stores,mem_load_l3_miss_retired.local,mem_load_l3_miss_retired.remote

The senior engineer reads TMA as a hypothesis generator, not a diagnosis. They then measure specific counters to confirm.

Non-Obvious Truth #3: Working-Set Analysis Predicts Cache Behavior, Not Performance#

A kernel might have WSS < L3, but still be memory-bound if access patterns are terrible.

Example: Random access to 40 MB (fits in L3) with 64-bit strides:

  • L3 hit rate: ~30% (many cache lines touched once, replaced)
  • L3 effective bandwidth: 0.3 × 120 GB/s = 36 GB/s
  • Still memory-bound on roofline, despite L3 residency

Countermeasure: Measure cache reuse distance. For each load, count how many unique cache lines are loaded before that line is touched again. High reuse distance = low effective bandwidth.

Non-Obvious Truth #4: Little's Law Explains Batch Size Selection#

Autoregressive decoding bottleneck for batch size 1:

Text
Occupancy = 1 token/cycle × (5 layers × 40ns DRAM latency) = 200 tokens in flight

But only 1 token is present. So:

Text
Pipeline Utilization = 1 / 200 = 0.5%

Solution: Speculative decoding adds 5 tokens in flight:

Text
Pipeline Utilization = 5 / 200 = 2.5%

Speculative decoding doesn't change roofline, it changes occupancy, which is what matters when latency >> throughput. This is why speculatively decoding 5 tokens (at cost of some discarded compute) is worth it: you're using the pipeline capacity that's wasted anyway.

Non-Obvious Truth #5: Latency-Bound ≠ Sequential Dependency Chain#

A kernel can be latency-bound even with no data dependencies if:

  • Register pressure is high (spill to L1, 40-cycle latency)
  • Memory prefetch buffer is full (new loads must wait)
  • Instruction window is shallow (Sapphire Rapids: 352-entry ROB)

Example: 1000-iteration loop with 20 independent memory loads:

C
for (int i = 0; i < 1000; i++) {
    for (int j = 0; j < 20; j++) {
        x[i][j] = load(random_address[i*20+j]);  // 40-cycle DRAM latency
    }
}

Seems parallelizable (20 independent loads per i), but if DRAM memory controller can only issue 8 outstanding requests, the 9th load stalls. This is latency-bound due to resource saturation, not data dependency.


6. BENCHMARK / MEASUREMENT#

Procedure: Extract Roofline from Real Hardware#

Shell
# Step 1: Install Intel VTune or Linux perf
# On Ubuntu:
sudo apt install linux-tools-generic

# Step 2: Run a known-compute kernel and measure
# Use STREAM benchmark to measure DRAM bandwidth:
git clone https://github.com/jeffhammond/STREAM.git
cd STREAM
gcc -O3 -march=native -fopenmp stream.c -o stream
./stream
# Output: Copy bandwidth, scale bandwidth, add bandwidth, triad bandwidth
# Sapphire Rapids typical: 12.8 GB/s per core

# Step 3: Measure compute ceiling
# Using OpenMP + AVX-512 FMA:
gcc -O3 -march=sapphirerapids -fopenmp flops.c -o flops
# Count FP64 operations per cycle using perf

# Step 4: Plot your kernel on roofline
# Measure arithmetic intensity:
perf stat -e cycles,instructions,LLC-load-misses,LLC-loads,dTLB-load-misses ./your_kernel
# Calculate: AI = instructions / (LLC-loads × 64 bytes)

# Step 5: Verify on the roofline plot
# AI × 12.8 GB/s = memory-bound ceiling
# 1500 GFLOPs = compute ceiling
# actual GFLOPs from (instructions / 2) / (cycles / 3e9)

Procedure: Measure TMA and Interpret#

Shell
# Step 1: Ensure TMA events are available (perf list | grep TMA_)
# Sapphire Rapids: yes. Older CPUs: may require Intel VTune

# Step 2: Run TMA level 1 (broadest breakdown)
perf stat -e cycles \
  -e 'cpu/event=0x3c,umask=0x00,name=CPU_CLK_UNHALTED.THREAD/' \
  -e 'cpu/event=0xa4,umask=0x01,name=TOPDOWN.SLOTS/' \
  -e 'cpu/event=0xc2,umask=0x02,name=UOPS_RETIRED.RETIRE_SLOTS/' \
  ./your_kernel

# Step 3: Calculate percentages
# TMA_Backend_Bound = (Backend_Cycles / Total_Cycles) × 100
# where Backend_Cycles = stalled cycles not retiring useful uops

# Step 4: Use Intel VTune for finer breakdown (Level 2)
vtune -collect uarch-exploration -knob pmu-event-sampling=true \
  -r results_dir -- ./your_kernel
vtune -report uarch-exploration -r results_dir

# Step 5: Interpret output
# If Backend_Bound > 40% && Memory_Bound > 20%: focus on memory
# If Frontend_Bound > 30%: focus on i-cache / branching
# If Bad_Speculation > 15%: focus on branch prediction

Procedure: Cache Residency Measurement#

Shell
# Step 1: Measure L3 hit rate
perf stat -e cache-references,cache-misses,LLC-loads,LLC-load-misses \
  ./your_kernel

# Step 2: Calculate metrics
# L3 Hit Rate = (LLC-loads - LLC-load-misses) / LLC-loads
# If > 95%, data is mostly cached in L3
# If < 50%, data is thrashing in DRAM

# Step 3: Measure working set size in real time
perf record -e cache-misses:u -g ./your_kernel
perf report --stdio

# Step 4: Cross-check with cachegrind
valgrind --tool=cachegrind --cachegrind-out-file=out.cgout ./your_kernel
cg_annotate out.cgout

# Output shows:
#  I1 refs:      instructions executed
#  LL miss:      last-level cache misses
#  D1 miss:      L1 data cache misses

7. ML SYSTEMS RELEVANCE#

Attention Layer Analysis via Roofline#

For a transformer attention head (batch size B, sequence length N, head dimension D):

Text
Operation: scores = Q @ K^T + softmax + weighted_sum(V)

Arithmetic intensity varies by phase:

  • Q @ K^T: (B×N×D) × (D×N) → (B×N×N) matrix = 2BN²D / (2BND + BN²) FLOPs/byte
    • If N = 512, D = 64: AI ≈ 16 FLOPs/byte → compute-bound at 200 GFLOPs
    • If N = 512, D = 8: AI ≈ 2 FLOPs/byte → memory-bound at 25 GFLOPs

Implication: Long-context models (large N) shift attention from memory-bound to compute-bound. Flash Attention exploits this: by fusing operations and increasing AI, it moves attention from below-roofline to above-roofline performance.

Batching Impact on Throughput-Bound Behavior#

Batch size 1: 512 tokens per sequence, each token latency-bound by autoregressive dependency.

  • Occupancy: 1 token in flight, 512-cycle latency → 0.2% pipeline utilization
  • TMA: likely 50%+ Backend_Bound waiting on KV cache accesses

Batch size 64: 64 independent token sequences executing in parallel.

  • Occupancy: 64 tokens × 512 layers = 32k logical tokens in flight
  • All execution ports saturated; many independent operations
  • TMA: retiring > 35%, backend bound drops to 20%

Optimization strategy: Start with small batches (B=1) and measure. If retiring < 30%, you're memory or speculation bound. Increase B until retiring plateaus → you've found the sweet spot for throughput.

Quantization's AI Impact#

FP32 weight matrix: 1 billion parameters × 4 bytes = 4 GB INT8 quantized: 1 billion parameters × 1 byte = 1 GB AI improves 4× (same FLOPs, 1/4 the bandwidth).

Example: SpMM with FP32, AI = 1 FLOP/byte (memory-bound at 12.8 GFLOPs). With INT8: AI = 4 FLOPs/byte (memory-bound at 51.2 GFLOPs → compute-bound).

Lesson: Quantization shifts the roofline ceiling. Junior engineers sometimes ignore this; expert engineers recognize it's often the dominant factor in deployment performance.


8. PhD QUALIFIER QUESTIONS#

Q1: Define the Roofline Model's two ceilings and explain when a kernel is limited by each. For a SpMV kernel with AI = 0.5 on hardware with 10 GB/s DRAM bandwidth and 500 GFLOPs peak compute, what is the maximum achievable performance? Draw the roofline and annotate your answer.

Q2: Explain the relationship between Little's Law (Occupancy = Throughput × Latency) and autoregressive language model inference bottlenecks. Why is speculative decoding beneficial even if it increases total FLOPs?

Q3: TMA reports 35% Frontend_Bound, 42% Backend_Bound (of which 22% is DRAM_Bound), and 23% Retiring. What is your optimization priority, and why? Propose one concrete optimization for the top bottleneck.

Q4: You measure a transformer attention layer with working-set size of 35 MB (weights + activations). Sapphire Rapids L3 cache is 51.2 MB per socket. However, L3 hit rate is only 60%. Explain possible causes and how you would diagnose each.

Q5: Compare the arithmetic intensity and roofline ceilings for (a) dense GEMM (C += A×B), (b) sparse MatMul with 90% sparsity, and (c) attention with sequence length 4096. Which is compute-bound? Which is memory-bound? Justify numerically.


9. READING LIST#

  1. Williams, S. A., et al. (2009). "Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures." Communications of the ACM, 52(4): 65-76.

    • See: Sections 2-3 (model definition), Section 4 (roofline plots)
  2. Dendibakh, D. (2024). Performance Analysis and Tuning on Modern CPUs. O'Reilly Media.

    • See: Chapter 2 (CPU basics), Chapter 6 (Roofline), Chapter 7 (TMA)
  3. Yasin, A. (2014). "Top-Down Microarchitecture Analysis Method." ISPASS 2014.

    • See: Sections 2-3 (TMA hierarchy), Section 4 (event formulas)
  4. Intel. 64 Architecture Optimization Reference Manual. Order #248966.

    • See: Chapter 2 (memory hierarchy), Section 3.5 (performance monitoring)
  5. Fog, A. (2023). Instruction Tables: Lists of Instruction Latencies, Throughputs, and Micro-operation Breakdowns for Intel, AMD, and VIA CPUs. Vol. 4.

    • See: Appendix A (Sapphire Rapids latencies), Appendix B (vectorization throughput)
  6. Intel. VTune Profiler User Guide. Online documentation.

    • See: Section "Microarchitecture Analysis", "Hot Spot Analysis"
  7. Gregg, B. (2021). Performance Testing Guide for Software Architects. Addison-Wesley.

    • See: Chapter 4 (measurement methodology), Chapter 5 (profiling tools)

End of Module 14

This module forms the theoretical and practical foundation for the remaining performance engineering sequence. Mastery of Roofline, TMA, and working-set analysis is prerequisite for Modules 15-16.