MLsys · Appendix E

Hardware Comparison Matrix

1. Introduction#

Hardware choice fundamentally determines inference latency, cost, and power consumption. This appendix provides detailed, real-world comparisons of the dominant inference platforms: Intel Xeon Scalable (Platinum SPR), AMD EPYC Genoa, Apple M4 Max, Qualcomm Snapdragon 8 Elite, Raspberry Pi 5, and NVIDIA H100/H200. Each row is measurable on real hardware today.


2. Complete Hardware Comparison Matrix#

Diagram
Loading…

3. Detailed Platform Analysis#

3.1 Intel Xeon Platinum SPR (4th Gen Sapphire Rapids)#

Text
Specifications:
- Cores: 32, 40, 60 (different SKUs)
- Architecture: P-core (Performance), E-core (Efficiency) hybrid
- Clock: 2.0-3.8 GHz base/turbo
- Instruction set: AVX-512 (crucial for inference optimization)
- Memory: 12 channels DDR5

Inference Strengths:
+ Mature ecosystem (TVM, ONNX Runtime heavily optimized)
+ AVX-512 enables 4-8x speedup for int8 matmul
+ Per-core turbo allows single-threaded latency optimization
+ Cost-effective for batch serving (good FLOPS/dollar)

Weaknesses:
- High power (350-500W) limits edge deployment
- Memory bandwidth per-core lower than specialized hardware
- Requires tuning (thread pool, affinity, vectorization)

Realistic ResNet-50 Performance (FP16, B=128):
- Throughput: 108 samples/sec per socket
- Total throughput (2 sockets): 216 samples/sec
- Power: 450W TDP ÷ 216 = 2.1 W/sample

Use case: Enterprise batch inference (image tagging, recommendation prefetch)
Typical deployment: 2-socket systems in data centers

3.2 AMD EPYC Genoa (9004 Series)#

Text
Specifications:
- Cores: 12-128 (extreme range of SKUs)
- Architecture: Zen 5 with 3D V-Cache (stacked cache)
- Clock: 2.45-3.7 GHz
- Memory: 12 channels DDR5, up to 768GB per CPU

Inference Strengths:
+ More cores than Intel (128 vs 60) enables higher throughput
+ 3D V-Cache increases cache hit rate (~3x more cache)
+ Better power efficiency (lower power per core)
+ Competitive cost

Weaknesses:
- Newer ecosystem (fewer optimizations than Intel)
- AVX-512 support limited vs Intel
- Some frameworks have better Intel tuning

Realistic Performance (vs Intel):
- Same FP32 latency: ~12.3ms (slightly better than Intel)
- Better aggregate throughput: 164 samples/sec (vs 108)
- Better power efficiency: 41 mW/TFLOP (vs 57)

Comparison: AMD is 50% faster for batch throughput, slightly more efficient
Viable alternative where cost is critical

3.3 Apple M4 Max (Mobile Processor)#

Text
Specifications:
- Cores: 10 (8 Performance + 2 Efficiency)
- Architecture: ARM-based, custom Apple design
- Clock: 3.5-4.4 GHz
- GPU: 10-core integrated GPU

Inference Strengths:
+ Exceptional efficiency (4.3 mW/TFLOP, best in class)
+ Low power enables passive cooling (no fan needed)
+ Integrated GPU & ML accelerators (Neural Engine)
+ macOS/iOS ecosystem

Weaknesses:
- Limited to Apple devices
- Smaller batch sizes (not optimized for 256-batch inference)
- Less mature ML ecosystem vs x86

Realistic Performance:
- ResNet-50: 6.2ms (FP32), 4.1ms (FP16) single image
- BERT-base: 16ms (FP32) for single sample
- LLaMA-7B: 850ms prefill, 140ms/token decode (slow on CPU)

Use case: On-device ML (macOS apps, iPad ML)
Not suitable for: Data center inference

3.4 Qualcomm Snapdragon 8 Elite (Mobile SoC)#

Text
Specifications:
- Cores: 8 (3 Performance + 5 Efficiency)
- ISA: ARM v9.2 with custom accelerators
- Hexagon DSP: Custom DSP for NN acceleration
- Memory: LPDDR5X (102 GB/s)

Inference Strengths:
+ Ultra-low power (5-8W total SoC)
+ Hardware NN accelerator (Hexagon DSP)
+ Qualcomm NPU (Neural Processing Unit)
+ 1 billion+ phones deployed

Weaknesses:
- Power efficiency only for <1-second inference
- Limited for long-running batch jobs
- Framework support less mature than mobile

Realistic Performance:
- ResNet-50: 8.5ms (CPU only), 2ms (with Hexagon acceleration)
- BERT-base: Not practical (too slow on mobile)

Use case: Real-time mobile AI (camera, voice, gesture)
Best for: Single-request latency <100ms

3.5 Raspberry Pi 5#

Text
Specifications:
- CPU: Broadcom BCM2712 (4-core ARM)
- Clock: 2.4 GHz
- Memory: 4-8GB LPDDR5
- I/O: PCI-e for GPU acceleration

Inference Strengths:
+ Extremely low cost ($60-100)
+ Low power (5W)
+ Educational/hobbyist use

Weaknesses:
- Very slow for modern models (125ms per ResNet-50 image!)
- Only viable for tiny models (<50M params)
- Not practical for production deployment

Realistic Performance:
- ResNet-50: 125ms (FP32), 65ms (FP16), 32ms (INT8)
- Throughput: 8 images/sec (vs 100+ for servers)
- MobileNet-v2: 20ms, viable for real-time

Use case: Educational projects, IoT edge inference
Not suitable for: Production ML services

3.6 NVIDIA H100 GPU#

Text
Specifications:
- CUDA Cores: 21,888
- Architecture: Hopper (4nm process)
- Memory: 80GB HBM3 (3,360 GB/s bandwidth)
- NVLink: 7 × 400 GB/s (connect 8 GPUs)
- Tensor Cores: FP8, BF16, TF32, FP32, FP64

Inference Strengths:
+ Extreme throughput: 1000x faster than CPUs for matrix ops
+ Unified memory model (all GPU types)
+ Mature ecosystem (CUDA, TensorRT, etc.)
+ Production battle-tested

Weaknesses:
- High cost ($40K per GPU)
- High power (700W) requires data center infrastructure
- Latency for single-request (<50ms perfect, not optimized for <5ms)
- Not suitable for <1W applications

Realistic Performance (Batch-focused):
- ResNet-50: 0.65ms (FP32), 0.35ms (FP16) with B=128
- Throughput: 196,923 samples/sec (vs 108 on CPU!)
- 1,818x faster than CPU for batch inference

Comparison: CPU vs H100 for 1M images
- CPU: 1M ÷ 108 = 9,260 seconds = 2.6 hours
- H100: 1M ÷ 196,923 = 5 seconds (1,852x speedup!)
- Cost: H100 $40K, but cost amortized over 3-5 year lifetime

Use case: Cloud AI services (OpenAI, Anthropic, etc.)
Production deployment: 8x H100 with NVLink per service

4. Cost-Performance Analysis#

4.1 Cost Per Inference (Amortized)#

Diagram
Loading…

4.2 "Bang for Buck" (Throughput per Dollar)#

Diagram
Loading…

5. Deployment Decision Matrix#

Diagram
Loading…

6. Summary & Key Takeaways#

For batch inference (>100 req/sec):

  • Best performance: NVIDIA H100 (1000x CPU throughput)
  • Best cost-performance: Intel Xeon SPR or AMD EPYC
  • Power budget: H100 requires 700W, SPR 350-500W

For real-time inference (<50ms latency):

  • Best: Apple M4 Max (6ms for ResNet, 4ms FP16)
  • Good: Snapdragon 8E (with NPU acceleration)
  • Worst: Raspberry Pi (125ms)

For production deployment:

  • H100: Necessary for 10K+ req/sec (cloud APIs)
  • SPR/Genoa: Sweet spot for 1-5K req/sec with cost sensitivity
  • M4/Snapdragon: Only option for mobile/edge

Hardware trends (2024-2025):

  • Specialized inference chips becoming competitive (AWS Neuron, Apple Neural Engine)
  • Mobile chips approaching laptop performance
  • CPU inference closing gap through vectorization (AVX-512, SVE)
  • Quantization essential on all platforms (Int8 ubiquitous)

Next steps:

  1. Measure your actual workload (batch size, latency SLO)
  2. Benchmark on target hardware (don't trust specs)
  3. Account for 30% overhead (framework, queuing, etc.)
  4. Add 20% headroom for spikes
  5. Evaluate total cost of ownership (not just hardware)