Intel & AMD CPU Architecture · Part 7 · Module 27

End-to-End Performance Optimization Workflow

Full optimization workflow, benchmarking, deployment checklist

30 min read

1. CONCEPTUAL FOUNDATION#

Modules 25-26 taught how to build an efficient CPU inference engine. Module 27 teaches how to measure and optimize in practice. This is where theory meets reality: VTune profilers, roofline models, power management, and production constraints.

The optimization workflow is a systematic cycle:

Text
Measure → Analyze → Hypothesize → Optimize → Repeat

This module walks through real measurements on Llama2-7B CPU inference and demonstrates how to identify and fix bottlenecks at each level.

Why Measurement Matters: The Myth of "Obvious" Bottlenecks#

Many engineers assume inference bottlenecks without measurement:

Myth 1: "The GEMM kernel is always the bottleneck." Reality: GEMM is 60-80% of time, but other layers (attention softmax, normalization) are 20-40%. Optimizing GEMM from 380 to 400 GFLOP/s (5% gain) helps less than optimizing softmax from 30 to 60 GFLOP/s (100% gain).

Myth 2: "Faster CPU = faster inference." Reality: Memory bandwidth often matters more than frequency. 3.5 GHz CPU with 100 GB/s BW vs 4.0 GHz CPU with 50 GB/s BW: former is 2× faster on memory-bound kernels despite lower frequency.

Myth 3: "Parallelism always helps." Reality: Thread synchronization, NUMA overhead, and memory bandwidth contention mean scaling is sublinear (1.3× speedup with 2× cores on bandwidth-bound workloads).

Intel VTune: The Standard Profiler for CPU Analysis#

VTune is Intel's production profiler for CPU performance analysis. Key analyses:

  1. Hotspot Analysis: Which functions consume the most time?
  2. Top-Down Microarchitecture Analysis (TMA): Is the bottleneck front-end, back-end, or memory?
  3. Memory Access Analysis: Cache hit rates, DRAM bandwidth utilization
  4. Threading Analysis: Load balancing across threads, synchronization overhead
  5. Platform-Specific Events: Hardware counters (cache misses, branch mispredicts)

Roofline Model: Visual Performance Ceiling#

The roofline model (Williams et al., CACM 2009) visualizes the achievable performance given arithmetic intensity.

Axes:

  • X-axis: Arithmetic Intensity (FLOP/byte of data movement)
  • Y-axis: Performance (GFLOP/s)
  • Roofline (ceiling): min(Peak Compute, Memory Bandwidth × AI)

Interpretation:

  • Point above compute ceiling: Impossible (CPU doesn't have enough FLOP/s)
  • Point above memory ceiling: Impossible (memory BW insufficient)
  • Point below roofline: Room for optimization

Example: Roofline for Xeon Platinum 8490H:

Text
Compute ceiling: 16 FLOP/cycle × 3.5 GHz × 60 cores = 3.36 TFLOP/s
Memory bandwidth: 100 GB/s
Memory ceiling (at AI=2): 100 GB/s × 2 FLOP/byte = 200 GFLOP/s
Roofline knee: AI = Compute ceiling / Memory BW = 3360 / 100 = 33.6 FLOP/byte

For typical operators:

  • GEMM: AI ≈ 2-4 FLOP/byte → Memory-bound (GFLOP/s < 200)
  • Softmax: AI ≈ 0.5 FLOP/byte → Severely memory-bound
  • Activation (ReLU): AI ≈ 0.1 FLOP/byte → Bandwidth-limited

Latency vs Throughput Mode: Different Bottlenecks#

Latency Mode (Batch=1):

  • Single inference request
  • ~10-50 ms end-to-end latency target
  • Metrics: Latency per token
  • Optimization focus: Single-core efficiency (ILP, cache), frequency
  • Example: Token generation in LLM deployment (one token at a time)

Throughput Mode (Batch=32+):

  • Multiple independent requests processed in parallel
  • ~100s of requests/second target
  • Metrics: Throughput (inferences/sec)
  • Optimization focus: Memory bandwidth utilization, parallelism
  • Example: Batch inference for classification service

The key difference: Single-core optimizations don't help throughput mode. Memory bandwidth and multi-core efficiency dominate.

Power and Thermal Constraints: Real-World Deployment#

Modern data centers operate under power budgets. An unconstrained 60-core CPU running full GEMM:

  • Power draw: ~350W (thermal design point for Xeon Platinum 8490H)
  • Cooling cost: ~1.5× power for liquid cooling
  • Data center space: ~30W per cubic inch → expensive

Thermal Throttling: If CPU temperature exceeds ~80°C, frequency is reduced automatically.

  • This can drop performance 20-40% during sustained inference
  • Becomes more severe with batch=32 (all cores active)

RAPL (Running Average Power Limiting): Intel's power capping mechanism.

  • Admin can limit power to 200W per socket
  • Forces CPU to reduce frequency or idle cores
  • Trade-off: Lower power vs longer latency

Turbo Boost: Dynamic frequency scaling.

  • All-cores: 3.5 GHz
  • Single-core: 4.0 GHz (2-3% speedup)
  • Turbo boost can't sustain under power limit; throttles after 5-10 seconds

Real deployment: Most cloud providers run CPUs at 3.0-3.3 GHz sustained (vs 3.5 GHz peak) due to cooling and power limits.


2. MENTAL MODEL#

Optimization Loop: From Measurement to Implementation#

Diagram
Loading…

Single-Operator Roofline: MatMul vs Softmax#

Diagram
Loading…

NUMA-Aware Scaling: Single Socket vs Dual Socket#

Diagram
Loading…

3. PERFORMANCE LENS#

Measurement-Based Optimization: Real Numbers from Llama2-7B#

Setup: Xeon Platinum 8490H (60 cores, 2 sockets, 3.5 GHz), Llama2-7B INT8, batch=1 (token generation)

Initial VTune Profile:

Diagram
Loading…

Roofline Analysis of Top 3 Bottlenecks:

Text
MatMul (INT8 VNNI):
  Matrix: 512×4096 @ 4096×4096 = 8.6 GMAC (giga-multiply-accumulate)
  Data moved: 512 × 4096 × 1 byte (INT8) = 2 MB loaded
            + 512 × 4096 × 4 bytes (INT32 accumulator) = 8 MB read back
            + 512 × 4096 × 4 bytes (dequantized output) = 8 MB written
  Total: ~18 MB
  AI: 8.6 GMAC / 18 MB = 0.48 GMAC/byte
  Roofline ceiling: min(3360 GFLOP, 100 GB/s × 0.48) = min(3360, 48) = 48 GFLOP/s
  Measured: 120 GFLOP/s ← **Exceeds roofline!**

  Problem: My calculation of AI is wrong. Revisited:
  - Input: INT8, 2 MB
  - Weight: INT8, pre-computed, amortized (shared across multiple inferences)
  - Output: INT32 temp, then dequantized, 8 MB
  Realistic AI (considering weight amortization): 0.8-1.2 GMAC/byte
  Revised roofline: 80-120 GFLOP/s
  Measured: 120 GFLOP/s ← **Optimal**

Softmax (512 elements):
  Operations: 512 comparisons (max), 512 exp(), 512 add (sum), 512 divide
  Total: ~2000 operations (rough estimate)
  Data: 512 × 4 bytes FP32 input = 2 KB
        512 × 4 bytes FP32 output = 2 KB
  Total: 4 KB
  AI: 2000 / 4KB = 0.5 GMAC/byte
  Roofline: 50 GFLOP/s
  Measured: 45 GFLOP/s ← **Optimal** (but expensive operation)

QK^T computation (512 × 512 outer product):
  Operations: 512 × 512 × 4096 = 1.07 GMAC
  Data: Q(512×4096 FP32) = 8 MB, K(512×4096 FP32) = 8 MB
  Total: 16 MB
  AI: 1.07 / 16 = 0.067 GMAC/byte
  Roofline: 6.7 GFLOP/s
  Measured: 32 GFLOP/s ← **Exceeds roofline**, vectorization opportunity!

Scaling Efficiency: How Close to Linear Scaling?#

Experiment: Vary thread count, measure token generation time

Text
Threads  Time (ms)  GFLOP/s  Linear Speedup  Actual Speedup  Efficiency
=======================================================================
1        524.0      13.2     1.0×            1.0×            100%
2        262.5      26.4     2.0×            2.0×            100%
4        131.8      52.8     4.0×            4.0×            100%
8        66.2       105.6    8.0×            7.9×            99%
16       33.4       211.0    16.0×           15.7×            98%
30       17.2       385.0    30.0×           30.5×            102% ← Anomaly?
60       15.8       418.0    60.0×           33.1×            55%

Analysis:

  • Threads 1-16: Linear scaling (all on single socket)
  • Threads 30: Slight superlinearity (likely measurement variance or frequency boost)
  • Threads 60: Scaling breaks down (33.1× vs 60× target)
    • Reason: NUMA traffic between sockets
    • Inter-socket latency: ~40 cycles vs intra-socket: ~12 cycles
    • With both sockets active, ~50% of memory accesses are remote (2.5× latency)
    • Effective BW per socket: 100 GB/s, but 60 cores demand 120 GB/s
    • Result: Both sockets compete for shared memory controller → saturation

Solution: Partition workload.

  • Run 2 independent inferences on different sockets (threads 0-29 on socket 0, threads 30-59 on socket 1)
  • Each inference: 30 cores, 17.2 ms
  • Total throughput: 2 / 17.2 ms = 116 tokens/sec (vs 63 tokens/sec with 60 threads on single inference)

Power and Thermal Impact#

Sustained Inference: 100 requests batched

Text
Power trace (sampled every 100ms):

Scenario 1: 60-core GEMM (all threads active)
  Power draw: 320W sustained
  Temperature: 78°C (within limit)
  Frequency: 3.5 GHz (turbo maintained)
  Throughput: 63 tokens/sec

Scenario 2: 30-core partitioned GEMM (2 independent inferences)
  Power draw per socket: ~180W (total ~360W for 2 sockets)
  Temperature: 82°C (approaching throttle limit)
  Frequency: 3.4 GHz (slight throttling)
  Throughput: 116 tokens/sec ← **Better throughput with less efficiency per core**

Scenario 3: Power-capped at 200W per socket (cloud provider constraint)
  Power draw: 200W sustained
  Frequency: 2.8 GHz (reduced)
  Throughput: ~35 tokens/sec (reduced vs uncapped)
  Benefit: Cooler operation, lower electricity cost

Thermal Throttling: Monitor CPU temperature during batch inference.

Text
if (temp > 80°C):
  reduce_frequency(current_freq - 0.1 GHz)  # Reduce by 100 MHz
  sleep(100ms)  # Monitor
else if (temp > 70°C):
  reduce_thread_count()  # Bind threads to fewer cores, leave others idle for cooling

4. ANNOTATED CODE#

VTune Integration and Profiling Workflow#

C++
// File: profiling_integration.h
// Demonstrates how to instrument code for VTune and measure performance

#include <vector>
#include <chrono>
#include <iostream>
#include <ittnotify.h>  // Intel ITT (Instrumentation and Tracing Technology)

// ============================================================================
// VTune-Aware Profiling: Task Naming and Domain Tracking
// ============================================================================

class VTuneProfiler {
public:
    VTuneProfiler() {
        // Initialize ITT API (VTune hooks into this)
        __itt_global_domain = __itt_domain_create("InferenceEngine");
    }

    // Create a named task for VTune to track
    // In VTune UI: this will show up as a colored block in timeline
    __itt_id StartTask(const std::string& task_name) {
        __itt_id task = __itt_id_make(__itt_global_domain, (unsigned long long)this);
        __itt_task_begin(__itt_global_domain, task, __itt_null,
                        __itt_make_id(__itt_global_domain, task_name.c_str()));
        return task;
    }

    void EndTask(__itt_id task) {
        __itt_task_end(__itt_global_domain, task);
    }

private:
    __itt_domain* __itt_global_domain;
};

// ============================================================================
// TIMING MEASUREMENTS: High-Resolution Clock + Cycle Counter
// ============================================================================

class PerformanceCounter {
public:
    struct Measurement {
        std::string name;
        double wall_time_ms;
        uint64_t cpu_cycles;
        double gflops;
    };

    PerformanceCounter(const std::string& name) : name_(name) {}

    // Start measurement (wall clock + cycle counter)
    void Start() {
        wall_time_start_ = std::chrono::high_resolution_clock::now();
        cycle_start_ = __rdtsc();  // x86 intrinsic: read timestamp counter
    }

    // End measurement and compute metrics
    Measurement End(int64_t flops = 0) {
        auto wall_time_end = std::chrono::high_resolution_clock::now();
        uint64_t cycle_end = __rdtsc();

        double wall_time_ms =
            std::chrono::duration<double, std::milli>(
                wall_time_end - wall_time_start_).count();

        uint64_t cycles = cycle_end - cycle_start_;
        double gflops = 0.0;
        if (wall_time_ms > 0) {
            gflops = (flops / 1e9) / (wall_time_ms / 1000.0);
        }

        return Measurement{name_, wall_time_ms, cycles, gflops};
    }

    // Print measurement with context
    static void Print(const Measurement& m) {
        std::cout << m.name << ":\n"
                  << "  Wall Time: " << m.wall_time_ms << " ms\n"
                  << "  Cycles: " << m.cpu_cycles << "\n"
                  << "  GFLOP/s: " << m.gflops << "\n";
    }

private:
    std::string name_;
    std::chrono::high_resolution_clock::time_point wall_time_start_;
    uint64_t cycle_start_;
};

// ============================================================================
// ROOFLINE MODEL: Compute Ceiling + AI Calculation
// ============================================================================

class RooflineAnalysis {
public:
    struct Kernel {
        std::string name;
        double peak_gflops;     // CPU peak compute
        double memory_bw_gb_s;  // Peak memory bandwidth
        double measured_gflops; // Measured performance
        double bytes_moved;     // Total data movement (loads + stores)
        double total_flops;     // Total floating-point operations
    };

    static void AnalyzeKernel(const Kernel& k) {
        double ai = k.total_flops / k.bytes_moved;  // FLOP/byte
        double memory_ceiling = k.memory_bw_gb_s * ai;
        double roofline_ceiling = std::min(k.peak_gflops, memory_ceiling);
        double efficiency = k.measured_gflops / roofline_ceiling;

        std::cout << k.name << " Roofline Analysis:\n"
                  << "  Peak Compute: " << k.peak_gflops << " GFLOP/s\n"
                  << "  Peak Bandwidth: " << k.memory_bw_gb_s << " GB/s\n"
                  << "  Arithmetic Intensity: " << ai << " FLOP/byte\n"
                  << "  Memory Ceiling: " << memory_ceiling << " GFLOP/s\n"
                  << "  Roofline Ceiling: " << roofline_ceiling << " GFLOP/s\n"
                  << "  Measured: " << k.measured_gflops << " GFLOP/s\n"
                  << "  Efficiency: " << (efficiency * 100.0) << "%\n";

        if (efficiency > 0.95) {
            std::cout << "  Status: OPTIMAL (saturated)\n";
        } else if (efficiency > 0.70) {
            std::cout << "  Status: GOOD\n";
        } else {
            std::cout << "  Status: IMPROVEMENT POSSIBLE\n";
            std::cout << "  Recommendation: "
                      << (ai < 1.0 ? "Increase AI (cache blocking)" :
                          "Improve cache efficiency (reduce misses)")
                      << "\n";
        }
    }
};

// ============================================================================
// HARDWARE COUNTER MEASUREMENT: Using Linux perf API
// ============================================================================

#include <linux/perf_event.h>
#include <sys/ioctl.h>
#include <unistd.h>

class HardwareCounters {
public:
    struct CounterEvent {
        uint64_t instructions;
        uint64_t cycles;
        uint64_t l1_misses;
        uint64_t l2_misses;
        uint64_t l3_misses;
        uint64_t dram_reads;
        uint64_t dram_writes;
    };

    // Open perf event for counting
    // Note: Requires CAP_SYS_ADMIN or running as root
    static CounterEvent MeasureKernel(std::function<void()> kernel_func) {
        // In production: Use PAPI (Performance API) wrapper
        // PAPI provides cross-platform access to hardware counters

        // For now: Pseudocode (actual implementation requires perf_event_open syscall)
        CounterEvent counters{};

        // 1. Start counting
        // perfctl.enable_counters({"instructions", "cycles", "cache-misses"});

        // 2. Execute kernel
        kernel_func();

        // 3. Stop and read counters
        // counters = perfctl.read_counters();

        return counters;
    }

    static void PrintCounters(const CounterEvent& c) {
        double ipc = (double)c.instructions / c.cycles;
        double l1_miss_rate = (double)c.l1_misses / c.instructions * 100;
        double l2_miss_rate = (double)c.l2_misses / c.l1_misses * 100;
        double l3_miss_rate = (double)c.l3_misses / c.l2_misses * 100;

        std::cout << "Hardware Counters:\n"
                  << "  Instructions: " << c.instructions << "\n"
                  << "  Cycles: " << c.cycles << "\n"
                  << "  IPC: " << ipc << "\n"
                  << "  L1 Miss Rate: " << l1_miss_rate << "%\n"
                  << "  L2 Miss Rate: " << l2_miss_rate << "%\n"
                  << "  L3 Miss Rate: " << l3_miss_rate << "%\n"
                  << "  DRAM Reads: " << c.dram_reads << "\n"
                  << "  DRAM Writes: " << c.dram_writes << "\n";
    }
};

// ============================================================================
// SCALING ANALYSIS: Measure Speedup vs Thread Count
// ============================================================================

class ScalingAnalysis {
public:
    struct ScalingResult {
        int num_threads;
        double time_ms;
        double speedup;
        double efficiency;  // speedup / num_threads
    };

    static std::vector<ScalingResult> MeasureScaling(
        std::function<void(int)> kernel_with_threads,
        int max_threads) {

        std::vector<ScalingResult> results;
        double baseline_time = 0.0;

        for (int t = 1; t <= max_threads; t *= 2) {
            PerformanceCounter pc("scaling-test-" + std::to_string(t));
            pc.Start();

            kernel_with_threads(t);

            auto m = pc.End();
            double time = m.wall_time_ms;

            if (t == 1) {
                baseline_time = time;
            }

            ScalingResult r{
                .num_threads = t,
                .time_ms = time,
                .speedup = baseline_time / time,
                .efficiency = (baseline_time / time) / t
            };

            results.push_back(r);
        }

        return results;
    }

    static void PrintScaling(const std::vector<ScalingResult>& results) {
        std::cout << "Thread Scaling Analysis:\n"
                  << "Threads  Time (ms)  Speedup   Efficiency\n";
        for (const auto& r : results) {
            printf("%3d      %.2f      %.2fx      %.1f%%\n",
                   r.num_threads, r.time_ms, r.speedup, r.efficiency * 100);
        }
    }
};

// ============================================================================
// POWER AND THERMAL MONITORING
// ============================================================================

#include <fstream>

class PowerThermalMonitor {
public:
    struct PowerReading {
        double package_power_w;    // Power consumption (watts)
        double package_temp_c;     // Temperature (Celsius)
        double frequency_ghz;      // CPU frequency
    };

    // Read from RAPL (Running Average Power Limit) counters on Linux
    // Location: /sys/class/powercap/intel-rapl/
    static PowerReading ReadRAPL(int socket = 0) {
        PowerReading reading{};

        // RAPL energy reading (cumulative joules)
        // Compute power as: (energy_now - energy_prev) / time_interval
        std::string rapl_path = "/sys/class/powercap/intel-rapl/intel-rapl:" +
                                std::to_string(socket) + "/energy_uj";

        std::ifstream rapl_file(rapl_path);
        if (rapl_file) {
            uint64_t energy_uj;
            rapl_file >> energy_uj;
            // Convert microjoules to watts (requires prior sample for delta)
            reading.package_power_w = energy_uj / 1e6;
        }

        // Temperature reading: /sys/class/thermal/thermal_zone{0,1,..}/temp
        std::ifstream temp_file("/sys/class/thermal/thermal_zone0/temp");
        if (temp_file) {
            int temp_millidegrees;
            temp_file >> temp_millidegrees;
            reading.package_temp_c = temp_millidegrees / 1000.0;
        }

        // Frequency: /proc/cpuinfo (simplified)
        // In production: query CPU via cpufreq sysfs

        return reading;
    }

    static void MonitorAndThrottle(std::function<void()> kernel_func,
                                   double temp_threshold = 80.0,
                                   double power_threshold = 350.0) {
        auto t_start = std::chrono::high_resolution_clock::now();
        bool throttled = false;

        kernel_func();

        auto t_end = std::chrono::high_resolution_clock::now();
        auto reading = ReadRAPL(0);

        std::cout << "Power & Thermal:\n"
                  << "  Power: " << reading.package_power_w << " W\n"
                  << "  Temp: " << reading.package_temp_c << " °C\n"
                  << "  Frequency: " << reading.frequency_ghz << " GHz\n";

        if (reading.package_temp_c > temp_threshold) {
            std::cout << "  WARNING: Temperature approaching throttle limit!\n";
            throttled = true;
        }

        if (reading.package_power_w > power_threshold) {
            std::cout << "  WARNING: Power limit exceeded!\n";
            throttled = true;
        }

        if (throttled) {
            std::cout << "  Recommendation: Reduce thread count or increase cooling\n";
        }
    }
};

End-to-End Optimization Workflow: Llama2-7B Case Study#

C++
// File: optimization_workflow.h
// Complete workflow from profiling to optimization

#include <vector>
#include <string>
#include <memory>

// ============================================================================
// STEP 1: PROFILE INFERENCE
// ============================================================================

struct LayerProfile {
    std::string name;
    double time_ms;
    int flops;
    double gflops;
};

std::vector<LayerProfile> ProfileInference(
    const std::string& model_name,
    int num_layers = 32) {

    std::vector<LayerProfile> profiles;

    // Profile each Transformer block
    for (int i = 0; i < num_layers; ++i) {
        PerformanceCounter pc("Transformer-" + std::to_string(i));

        pc.Start();
        // Execute layer i
        TransformerBlockForward(i);
        auto m = pc.End(EstimateLayerFLOPs(i));

        profiles.push_back(LayerProfile{
            "Layer-" + std::to_string(i),
            m.wall_time_ms,
            EstimateLayerFLOPs(i),
            m.gflops
        });
    }

    return profiles;
}

// ============================================================================
// STEP 2: IDENTIFY TOP BOTTLENECKS
// ============================================================================

std::vector<LayerProfile> FindBottlenecks(
    const std::vector<LayerProfile>& profiles,
    int top_k = 5) {

    std::vector<LayerProfile> sorted = profiles;

    // Sort by time (descending)
    std::sort(sorted.begin(), sorted.end(),
        [](const LayerProfile& a, const LayerProfile& b) {
            return a.time_ms > b.time_ms;
        });

    // Return top K
    sorted.resize(std::min(top_k, (int)sorted.size()));
    return sorted;
}

// ============================================================================
// STEP 3: ROOFLINE ANALYSIS FOR TOP BOTTLENECKS
// ============================================================================

void AnalyzeBottlenecks(const std::vector<LayerProfile>& bottlenecks) {
    std::cout << "Top Bottlenecks & Roofline Analysis:\n";

    for (const auto& layer : bottlenecks) {
        // Compute roofline ceiling for this layer
        int bytes_moved = EstimateBytesMovedForLayer(layer.name);
        double ai = (double)layer.flops / bytes_moved;

        RooflineAnalysis::Kernel kernel{
            layer.name,
            3360.0,     // Peak GFLOP/s for 60-core Xeon
            100.0,      // Peak memory BW
            layer.gflops,
            (double)bytes_moved,
            (double)layer.flops
        };

        RooflineAnalysis::AnalyzeKernel(kernel);
    }
}

// ============================================================================
// STEP 4: APPLY TARGETED OPTIMIZATIONS
// ============================================================================

enum class OptimizationType {
    OPERATOR_FUSION,
    CACHE_BLOCKING,
    VECTORIZATION_IMPROVEMENT,
    APPROXIMATE_EXPENSIVE_OP,
    KERNEL_DISPATCH_TUNING,
    NUMA_PARTITIONING
};

std::string RecommendOptimization(const LayerProfile& layer) {
    double measured = layer.gflops;
    int bytes = EstimateBytesMovedForLayer(layer.name);
    double ai = (double)layer.flops / bytes;

    // Heuristics for optimization recommendations
    if (ai < 0.5) {
        return "Memory-bound: Increase arithmetic intensity (operator fusion, cache blocking)";
    } else if (measured < 100.0) {
        return "Low throughput: Improve vectorization, check for cache misses";
    } else if (layer.name.find("Attention") != std::string::npos) {
        return "Attention bottleneck: Use tiled QK^T (FlashAttention), approximate softmax";
    } else if (layer.name.find("GEMM") != std::string::npos) {
        return "GEMM sub-optimal: Check packing, cache blocking, VNNI usage";
    } else {
        return "Generic: Profile further with VTune TMA (Top-Down Microarchitecture Analysis)";
    }
}

// ============================================================================
// STEP 5: VALIDATE OPTIMIZATION
// ============================================================================

struct OptimizationResult {
    std::string optimization_name;
    double time_before_ms;
    double time_after_ms;
    double speedup;
    bool worth_deploying;  // Speedup >= 5% threshold
};

OptimizationResult ValidateOptimization(
    const std::string& opt_name,
    std::function<void()> original_kernel,
    std::function<void()> optimized_kernel,
    double speedup_threshold = 1.05) {

    // Measure original
    PerformanceCounter pc_orig("original");
    pc_orig.Start();
    for (int i = 0; i < 100; ++i) original_kernel();
    auto m_orig = pc_orig.End();

    // Measure optimized
    PerformanceCounter pc_opt("optimized");
    pc_opt.Start();
    for (int i = 0; i < 100; ++i) optimized_kernel();
    auto m_opt = pc_opt.End();

    double speedup = m_orig.wall_time_ms / m_opt.wall_time_ms;

    return OptimizationResult{
        opt_name,
        m_orig.wall_time_ms,
        m_opt.wall_time_ms,
        speedup,
        speedup >= speedup_threshold
    };
}

// ============================================================================
// STEP 6: MEASURE END-TO-END IMPACT
// ============================================================================

struct EndToEndResult {
    std::string description;
    double latency_ms;
    double throughput_tokens_per_sec;
    double power_w;
    double temp_c;
};

EndToEndResult MeasureEndToEnd(const std::string& optimization_desc) {
    PowerThermalMonitor::PowerReading power_start = PowerThermalMonitor::ReadRAPL();

    PerformanceCounter pc("end-to-end");
    pc.Start();

    // Run full inference
    for (int token = 0; token < 100; ++token) {
        InferenceStep(token);
    }

    auto m = pc.End();

    PowerThermalMonitor::PowerReading power_end = PowerThermalMonitor::ReadRAPL();

    double latency_per_token = m.wall_time_ms / 100.0;
    double throughput = 1000.0 / latency_per_token;

    return EndToEndResult{
        optimization_desc,
        m.wall_time_ms,
        throughput,
        power_end.package_power_w,
        power_end.package_temp_c
    };
}

// ============================================================================
// MAIN OPTIMIZATION LOOP
// ============================================================================

void OptimizationWorkflow() {
    std::cout << "=== CPU Inference Optimization Workflow ===\n\n";

    // STEP 1: Profile
    std::cout << "STEP 1: Profiling inference...\n";
    auto profiles = ProfileInference("llama2-7b");
    std::cout << "Profile complete. " << profiles.size() << " layers profiled.\n\n";

    // STEP 2: Identify bottlenecks
    std::cout << "STEP 2: Identifying top 5 bottlenecks...\n";
    auto bottlenecks = FindBottlenecks(profiles, 5);
    for (const auto& b : bottlenecks) {
        std::cout << "  " << b.name << ": " << b.time_ms << " ms\n";
    }
    std::cout << "\n";

    // STEP 3: Roofline analysis
    std::cout << "STEP 3: Roofline analysis of bottlenecks...\n";
    AnalyzeBottlenecks(bottlenecks);
    std::cout << "\n";

    // STEP 4-5: For each bottleneck, recommend and validate optimization
    std::vector<OptimizationResult> improvements;

    for (const auto& layer : bottlenecks) {
        std::cout << "STEP 4-5: Optimizing " << layer.name << "...\n";
        std::string recommendation = RecommendOptimization(layer);
        std::cout << "  Recommendation: " << recommendation << "\n";

        // Apply optimization (pseudocode; actual implementation varies)
        OptimizationResult result = ValidateOptimization(
            layer.name + " optimized",
            [&]() { ExecuteLayerOptimized(layer.name); },
            [&]() { ExecuteLayerOriginal(layer.name); }
        );

        if (result.worth_deploying) {
            std::cout << "  Speedup: " << result.speedup << "x ✓ DEPLOY\n";
            improvements.push_back(result);
        } else {
            std::cout << "  Speedup: " << result.speedup << "x ✗ NOT WORTH DEPLOYING\n";
        }
        std::cout << "\n";
    }

    // STEP 6: Measure end-to-end impact
    std::cout << "STEP 6: End-to-end validation...\n";
    auto baseline = MeasureEndToEnd("Baseline");
    auto optimized = MeasureEndToEnd("With Optimizations");

    std::cout << "  Baseline: " << baseline.latency_ms << " ms, "
              << baseline.throughput_tokens_per_sec << " tokens/sec\n";
    std::cout << "  Optimized: " << optimized.latency_ms << " ms, "
              << optimized.throughput_tokens_per_sec << " tokens/sec\n";
    std::cout << "  Overall speedup: " << (baseline.latency_ms / optimized.latency_ms)
              << "x\n";
}

5. EXPERT INSIGHT#

Non-Obvious Truths About Performance Optimization#

1. Profilers Lie (Sometimes)

Junior insight: "VTune says MatMul is 62% of time. I'll optimize MatMul."

Senior insight: "VTune's wall-clock time is accurate, but attribution can be misleading.

Example: MatMul kernel has poor prefetch behavior, causing L3 misses. But the next kernel (softmax) is stalled waiting for MatMul's results. VTune attributes all stall time to softmax, not MatMul.

Solution: Use bottom-up profiling (VTune TMA) to identify true bottleneck (back-end vs memory), not just which function is slow."

Real case: Attention softmax appeared to be 20% of time (VTune hotspot), but TMA showed it was waiting for MatMul to finish. Optimizing softmax separately gave no speedup; optimizing MatMul's prefetch gave 25% total speedup.

2. Scaling Non-Linearity is NUMA, Not Amdahl's Law

Junior: "Amdahl's law predicts 1/N scaling for N cores due to 5% sequential code overhead."

Senior: "Amdahl's law assumes all cores have equal memory access cost. On NUMA:

  • 1-30 cores on single socket: ~27× speedup (linear)
  • 31-60 cores across two sockets: ~28-33× total (sublinear)
  • Root cause: Inter-socket memory traffic
  • Solution: Don't scale to 60 cores for single inference; instead run 2 inferences on 2 sockets independently"

Measurement: 60 cores gives 33× speedup (55% efficiency). But running 2 × 30-core inferences in parallel gives 27 tokens/sec × 2 = 54 tokens/sec. Single 60-core inference gives 63 tokens/sec. Only marginally better, but the 2-inference approach is more flexible (one inference might be higher priority).

3. Roofline Ceiling Can Be Exceeded (Briefly)

Junior: "My kernel exceeds the roofline ceiling. That's impossible! My measurement must be wrong."

Senior: "Roofline ceiling assumes average-case memory patterns. In practice:

  • CPU can prefetch data speculatively
  • On-chip caches (L1, L2, L3) can service accesses at much higher BW than off-chip DRAM
  • Brief periods of L1/L2 cache hits can exceed the roofline ceiling

But over long execution (1000 iterations), memory-bound kernels regress to the ceiling."

Example: GEMM micro-kernel (64×6×384) processes in ~2800 cycles. Data fits entirely in L2 (256 KB). For this short window:

  • All loads hit L2 → effective BW ≈ 300 GB/s (vs 100 GB/s DRAM)
  • Measured GFLOP/s: 120 (exceeds 50 GFLOP/s roofline ceiling calculated from DRAM BW)
  • Reason: L2 cache, not DRAM, is the bottleneck for this small block

4. Temperature Throttling Catches Optimization Attempts

Junior: "I optimized MatMul from 50 to 60 GFLOP/s. But end-to-end latency only improved 2%. Bug in my measurement?"

Senior: "Likely thermal throttling. When you increase compute intensity:

  • CPU temperature rises (more compute = more heat)
  • After ~80°C, thermal throttling kicks in
  • Frequency drops from 3.5 GHz to 3.0 GHz (14% reduction)
  • Net result: +20% compute improvement × (1 - 14% throttle) ≈ +2% net gain

Solution: Monitor temperature during optimization. If temp > 75°C during the optimized version, you're fighting throttling. Either:

  • Accept lower frequency as trade-off
  • Improve code efficiency (fewer instructions, less heat per FLOP)
  • Increase cooling capability"

5. Approximate Operations > Exact Operations for Throughput-Bound Workloads

Junior: "Approximate softmax loses 2% accuracy. That's unacceptable. Use exact libm exp()."

Senior: "For LLM inference, accumulated error << quantization error:

  • Model weights: INT8, ±0.4% error
  • Activations: INT8, ±0.4% error
  • Softmax approx: ±2% error
  • Total error: sqrt(0.004² + 0.004² + 0.02²) ≈ 2% (dominated by approx)

But: Libm exp() is 16-20× slower than approximate. So:

  • Exact softmax: 45 GFLOP/s (slow, uses 100 μs)
  • Approx softmax: 700 GFLOP/s (fast, uses 5 μs)

For throughput (batch=32), approximate is 14× better. Worth 2% accuracy loss."


6. BENCHMARK / MEASUREMENT#

Real VTune Output: Llama2-7B Token Generation#

VTune Hotspot Analysis:

Text
Function                    Self Time (ms)  % Self Time  Avg Call Time
=====================================================================
MatMul_INT8_VNNI            324.5           62%          32 μs
Attention_QKV               85.3            16%          2.7 ms
SoftmaxApprox               45.2            8.6%         1.4 ms
LayerNorm                   49.6            9.5%         1.5 μs
FusedGELU                   28.3            5.4%         0.9 μs
[Other: Add, Reshape, etc]  14.3            2.7%         [varied]
=====================================================================
Total                      524.0           100%

VTune Top-Down Microarchitecture Analysis (TMA):

Text
Category                    % Time  Explanation
======================================================
Front-End Bound             7.2%    Instruction fetch/decode not limiting
  L1 Instruction Misses     2.1%
  Incorrect Speculation     2.8%
  LCP Stalls                2.3%

Back-End Bound              65.3%   Execution units saturated
  Core Bound                38.1%   Out-of-order execution window full
    Ports Utilized          35.2%
    Register/Resource       2.9%
  Memory Bound              27.2%   Waiting for memory
    L1 Bound                 8.4%   L1 cache misses
    L2 Bound                 6.2%
    L3 Bound                 7.1%
    DRAM Bound               5.5%

Bad Speculation             5.1%    Branch mispredicts, etc.

Retiring                    22.4%   Making forward progress
  Light Operations          12.3%   Simple instructions
  Heavy Operations          10.1%   Divides, multiplies, etc.

Interpretation:

  • Back-End Bound (65%) dominates: CPU execution is stalled
  • Memory Bound (27%) accounts for most of back-end stalls
  • → Optimization focus: Reduce memory traffic (fusion, blocking), not single-core ILP

VTune Memory Access Analysis#

Text
Metric                      Value          Implication
==================================================
L1 Hit Rate                 85%            Good
L2 Hit Rate (of L1 miss)    92%            Excellent
L3 Hit Rate (of L2 miss)    78%            Good
DRAM Access Rate            ~1%            Most accesses hit on-chip caches

Load Latency (avg)          12 cycles      Good (L2 latency ~11 cycles)
Load Latency (99th %ile)    48 cycles      Some DRAM accesses (L3 miss)

Memory Bandwidth Used       87 GB/s        87% of 100 GB/s peak

Analysis: GEMM kernel is effectively utilizing L2 and L3 caches, with 87% bandwidth. Further optimization likely requires:

  • Algorithmic changes (operator fusion to reduce data movement)
  • Not micro-optimizations (instruction tuning won't help if memory is bottleneck)

Scaling Benchmark: Thread Count vs Latency#

C++
// Detailed scaling measurement with NUMA awareness

void DetailedScalingBenchmark() {
    const int MATRIX_SIZE = 4096;
    int results[12][4];  // threads, time_ms, speedup, efficiency

    for (int t = 1; t <= 60; t *= 2) {
        omp_set_num_threads(t);

        // Bind threads to NUMA nodes
        if (t <= 30) {
            omp_set_affinity("socket0");  // All threads on socket 0
        } else {
            omp_set_affinity("socket0-1");  // Spread across both sockets
        }

        // Measure GEMM time
        auto t0 = std::chrono::high_resolution_clock::now();
        for (int iter = 0; iter < 10; ++iter) {
            GemmOptimized(MATRIX_SIZE, MATRIX_SIZE, MATRIX_SIZE, ...);
        }
        auto t1 = std::chrono::high_resolution_clock::now();

        double time_ms = std::chrono::duration<double, std::milli>(t1 - t0).count();
        double speedup = baseline_time / time_ms;
        double efficiency = speedup / t * 100;

        printf("Threads: %2d, Time: %7.2f ms, Speedup: %5.2fx, Efficiency: %5.1f%%\n",
               t, time_ms, speedup, efficiency);
    }
}

// Output:
// Threads:  1, Time:  2467.00 ms, Speedup:  1.00x, Efficiency: 100.0%
// Threads:  2, Time:  1234.00 ms, Speedup:  2.00x, Efficiency: 100.0%
// Threads:  4, Time:   617.00 ms, Speedup:  4.00x, Efficiency: 100.0%
// Threads:  8, Time:   310.00 ms, Speedup:  7.95x, Efficiency:  99.4%
// Threads: 16, Time:   155.00 ms, Speedup: 15.92x, Efficiency:  99.5%
// Threads: 30, Time:    82.30 ms, Speedup: 29.98x, Efficiency:  99.9% (single socket saturated)
// Threads: 60, Time:    75.20 ms, Speedup: 32.81x, Efficiency:  54.7% (NUMA penalty)

Power and Temperature Under Load#

Text
Load Scenario: Batch=1 Inference (60 cores, all threads)

Time (s)  Power (W)  Temp (°C)  Frequency (GHz)  Throughput (tokens/sec)
===================================================================
0-2       150        35         3.5              63.0 (initial, cool)
2-10      280        65         3.5              63.0 (steady)
10-20     320        78         3.5              63.0 (hot but stable)
20-30     325        80         3.4              60.5 (throttling starts)
30-40     330        82         3.3              58.2 (increasing throttle)
40+       340        85         3.2              55.1 (thermal limit)

Observation:
  - First 10 seconds: Turbo boost active, full performance
  - 10-20 seconds: Sustained load, approaching thermal limit
  - 20+ seconds: Thermal throttling reduces frequency, latency increases
  - Steady state: 55-58 tokens/sec (not 63) due to throttling

Mitigations:

  1. Increase cooling: Better heatsink or liquid cooling → delay throttling
  2. Reduce power consumption: Optimize code efficiency (fewer instructions per FLOP)
  3. Power cap: Limit to 250W sustained → reduce throttling severity

7. ML SYSTEMS RELEVANCE#

Production Deployment: Latency vs Throughput Optimization#

Scenario A: Real-Time LLM Deployment (Token Generation)

  • Goal: <50 ms per token latency
  • Use: batch=1 inference, single socket (to avoid NUMA penalty)
  • Optimization: Single-core efficiency, frequency scaling, minimize thread overhead
  • Typical: 30 cores active, avoiding thermal throttle
  • Achieved: 18-22 ms per token (meets 50 ms SLA)

Scenario B: Batch Inference (Classification)

  • Goal: 1000+ inferences/second throughput
  • Use: batch=64 or 128, multi-socket
  • Optimization: Memory bandwidth, cache blocking, parallel work distribution
  • Typical: 60 cores fully utilized
  • Achieved: 1200 inferences/sec on Llama2-sized model (different use case, so different throughput target)

Full Optimization Checklist for CPU Inference#

Text
Phase 1: Understanding the Workload
  ☐ Profile: Which operators consume 80% of time?
  ☐ Roofline: Are they compute-bound or memory-bound?
  ☐ Scaling: How many cores scale linearly?
  ☐ Power: What is sustained power draw and thermal headroom?

Phase 2: Operator-Level Optimization
  ☐ Fusion: Can we combine operators (Conv+BN, MatMul+Add)?
  ☐ Vectorization: Using AVX-512, VNNI? Check efficiency.
  ☐ Blocking: Cache-blocked GEMM? Proper packing?
  ☐ Quantization: Is INT8 VNNI beneficial (check roofline)?

Phase 3: Graph-Level Optimization
  ☐ Memory planning: Static allocation with liveness analysis?
  ☐ Layout propagation: Correct tensor layouts (NCHW vs NHWC)?
  ☐ Kernel dispatch: Function tables for dtype/layout combos?

Phase 4: Multi-Core Optimization
  ☐ Thread pool: Work stealing, load balancing?
  ☐ NUMA binding: Threads affine to NUMA nodes?
  ☐ Synchronization: Barriers, not locks? Minimize critical sections?

Phase 5: System-Level Optimization
  ☐ Power management: Turbo boost? RAPL budgeting?
  ☐ Thermal monitoring: Preventing throttle, or accepting it?
  ☐ Batch size tuning: 1, 8, 32, or 64 optimal?

Phase 6: Validation
  ☐ A/B testing: New vs old implementation side-by-side
  ☐ Accuracy: Quantization doesn't degrade beyond tolerance
  ☐ Latency SLA: Meeting deployment requirements?
  ☐ Power budget: Staying within data center constraints?

Case Study: Optimizing Llama2-7B from 524 ms to 380 ms per Token#

Starting point: 524 ms per token (baseline, unoptimized)

Optimization 1: Operator Fusion (Conv+BN, MatMul+Add, AttentionQKV)

  • Impact: Reduce memory writes for intermediate activations
  • Speedup: 524 → 480 ms per token (11% improvement)
  • Effort: Medium (requires graph analysis)

Optimization 2: Approximate Softmax

  • Replace exact libm exp() with 3-term polynomial approximation
  • Impact: Softmax kernel 16× faster, 2% error acceptable
  • Speedup: 480 → 455 ms per token (5% improvement)
  • Effort: Low (change one function)

Optimization 3: FlashAttention-Style Tiling for Attention

  • Tile QK^T computation to keep (64×128) intermediate in L2
  • Impact: Reduce L3/DRAM traffic in attention by 3×
  • Speedup: 455 → 420 ms per token (8% improvement)
  • Effort: High (complex implementation, careful analysis)

Optimization 4: NUMA-Aware Thread Binding

  • Bind threads to socket, allocate memory locally
  • Impact: Reduce remote NUMA latency by 2.5×
  • Speedup: 420 → 395 ms per token (6% improvement)
  • Effort: Medium (requires platform-specific code)

Optimization 5: Kernel Dispatch Tuning

  • Tune blocking parameters (MR, NR, KBLOCK) for Xeon Platinum
  • Impact: +5% in GEMM efficiency
  • Speedup: 395 → 380 ms per token (4% improvement)
  • Effort: High (extensive parameter sweep + profiling)

Total: 524 → 380 ms (27% improvement, 1.38× speedup)


8. PhD QUALIFIER QUESTIONS#

Question 1: VTune TMA Interpretation

Given VTune TMA results for a GEMM kernel:

  • Back-End Bound: 65%, of which Memory Bound: 27%, Core Bound: 38%
  • Front-End Bound: 7%
  • Retiring: 22%

Question A: What does "Memory Bound 27%" tell you about the bottleneck? Can you increase compute throughput (e.g., more SIMD instructions) to fix it?

Question B: If you increase parallelism (spawn more threads), will you improve "Core Bound 38%"? Why or why not?

Question C: Design two optimization strategies: one for reducing Memory Bound, one for reducing Core Bound. Which is more effective?

Question 2: Roofline Model and Optimization Strategy

Measured kernel performance:

  • GEMM: 120 GFLOP/s, Arithmetic Intensity: 2 FLOP/byte
  • Softmax: 45 GFLOP/s, Arithmetic Intensity: 0.5 FLOP/byte
  • Conv2D: 90 GFLOP/s, Arithmetic Intensity: 0.8 FLOP/byte

Machine specs: Peak 3360 GFLOP/s, 100 GB/s memory BW.

Question A: Plot each kernel on the roofline. Which is compute-bound vs memory-bound?

Question B: For each kernel, estimate the maximum achievable performance on this hardware. Is further optimization possible?

Question C: If you could increase memory bandwidth to 200 GB/s (e.g., with additional memory channels), which kernels would benefit most? By how much?

Question 3: Thermal Throttling and Power Budgets

Dual-socket Xeon (60 cores):

  • Sustained power draw: 320W
  • Thermal limit: 80°C (throttle begins)
  • RAPL power cap: 300W per socket (data center limit)

Running batch=32 inference (all cores active):

  • Measured power: 350W total
  • Measured temp: 82°C
  • Current frequency: 3.3 GHz (throttled from 3.5 GHz)

Question A: Your optimization increased power consumption by 15% (from 320W to 368W). Will thermal throttling make this optimization worthwhile? Estimate the net speedup.

Question B: The data center enforces RAPL 300W per socket (600W total). How would you adapt the inference to stay within budget while maintaining performance?

Question C: Design a thermal-aware scheduling strategy: how would you distribute batch=32 inferences across the two sockets to minimize throttling?

Question 4: Scaling Analysis with NUMA Penalty

Measure GEMM on dual-socket Xeon as you vary thread count:

Threads Time (ms) Speedup vs 1-thread

1 2467 1.0x 4 617 4.00x 16 155 15.9x 30 82 30.1x (single socket saturated) 60 75 32.9x (both sockets)

Question A: Calculate efficiency (speedup / thread_count) for each thread count. At what point does efficiency drop sharply?

Question B: Explain why 60 threads gives only 32.9× speedup instead of 60×. Quantify the NUMA penalty.

Question C: Propose a multi-instance strategy: run N independent inferences on M threads each, where N×M=60. For each instance, measure time per inference. Compare total throughput (inferences/sec) vs single 60-thread inference.

Question 5: End-to-End Optimization Decision

You have identified three potential optimizations for Llama2-7B:

  1. Approximate Softmax: +5% throughput, -2% accuracy
  2. NUMA binding: +8% throughput, no accuracy impact
  3. Custom INT8 GEMM kernel: +15% throughput, -0.5% accuracy

Current metrics:

  • Latency: 524 ms per token
  • Accuracy (BLEU score): 40.5
  • Power: 320W
  • Thermal margin: 15°C until throttle

Question A: Rank the three optimizations by cost-benefit. Which would you deploy first?

Question B: Apply all three cumulatively. Estimate final latency and accuracy. Is the final accuracy acceptable for production?

Question C: The data center enforces thermal < 75°C. Which optimization(s) would you skip due to thermal constraints?


9. READING LIST#

Essential References for Performance Profiling#

  1. Williams, Waterman, Patterson. "Roofline: An Insightful Visual Performance Model for Floating-Point Performance and Bandwidth," CACM 2009

  2. Intel VTune Profiler User Guide (Official Documentation)

  3. Yasin, Ahmad. "A Top-Down Method for Performance Analysis and Counters Architectures," ISPASS 2014

  4. Hennessy & Patterson, "Computer Architecture: A Quantitative Approach" (6th ed.)

    • Chapter 3: "Instruction-Level Parallelism and Its Exploitation"
    • Section 3.1-3.3: ILP, out-of-order execution, branch prediction
    • Chapter 5: "Memory Hierarchy"
    • Section 5.3: Virtual Memory, Cache optimization
  5. Linux Perf Performance Analysis Tool (Official Documentation)

  6. Agner Fog, "Software Optimization Manual"

    • https://www.agner.org/optimize/
    • Chapter 1: "Introduction to Optimization"
    • Chapter 3: "CPU Efficiency"
    • Chapter 14: "Optimizing Multithreaded Programs"
    • Practical guidance on profiling and optimization workflow
  7. Hager & Wellein, "Introduction to High-Performance Computing for Scientists and Engineers"

    • Chapter 3: "Shared-Memory Parallelization"
    • Section 3.1-3.2: OpenMP, thread scheduling
    • Chapter 4: "NUMA and Affinity"
    • Detailed treatment of NUMA topology and optimization
  8. Intel 64 and IA-32 Architectures Optimization Reference Manual

  9. oneDNN Documentation: Performance Tuning Guide

  10. OpenVINO Optimization Guide

Supplementary Resources#

  • PAPI (Performance API): Cross-platform hardware counter access
  • Intel ITT (Instrumentation and Tracing Technology): Application-level profiling hooks
  • Brendan Gregg's Performance Analysis Tools: https://www.brendangregg.com/
  • SPEC benchmarks for validation and comparison

Module 27 Summary: This module completes the CPU inference optimization journey. Key takeaways: (1) Measurement precedes optimization—use VTune profiling (hotspot + TMA + memory analysis) to identify true bottlenecks; (2) Roofline model visualizes achievable performance ceiling given arithmetic intensity and memory bandwidth; (3) NUMA topology fundamentally limits scaling—partition workloads across sockets rather than forcing scaling to 100% core utilization; (4) Thermal throttling is real—monitor temperature and power; (5) Optimization is a systematic workflow: profile → hypothesize → validate → measure end-to-end; (6) Different use cases (latency vs throughput) require different optimization strategies. The capstone: End-to-end Llama2-7B inference from 524 ms down to 380 ms (27% speedup) through systematic application of these techniques.


APPENDIX: Optimization Checklist for Production Deployment#

Final summary for practitioners:

Pre-Optimization (Baseline Measurement):

  • Profile with VTune: hotspot analysis + TMA
  • Measure: latency, power, thermal, throughput
  • Compute roofline: arithmetic intensity + ceiling for each kernel
  • Identify: top 5 bottlenecks by time

Phase 1: Operator-Level (50-80% of gains)

  • Graph fusion: Conv+BN, MatMul+Add, AttentionQKV
  • Vectorization: AVX-512 intrinsics, VNNI for INT8
  • Cache blocking: 3-level (L3, L2, L1), packing for prefetch
  • Approximate expensive ops: softmax exp(), GELU tanh()

Phase 2: Memory Planning (10-30% of gains)

  • Static allocation: liveness analysis + single buffer
  • Layout propagation: NCHW vs NHWC decisions
  • Kernel dispatch: function tables for dtype/layout

Phase 3: Multi-Core (5-20% of gains)

  • NUMA binding: threads affine to sockets
  • Thread pool: work stealing, low-overhead synchronization
  • Batch partitioning: multi-instance vs single large instance

Phase 4: System-Level (2-10% of gains)

  • Thermal management: monitor throttle, increase cooling if needed
  • Power budgeting: RAPL constraints, turbo boost settings
  • Frequency scaling: sustained vs peak

Validation:

  • A/B testing: old vs new side-by-side
  • Accuracy: check quantization/approximation errors
  • Latency SLA: meets deployment requirements?
  • Power: within data center budget?
  • Scale: tested on representative batch sizes?