MLsys · Appendix E
Hardware Comparison Matrix
1. Introduction#
Hardware choice fundamentally determines inference latency, cost, and power consumption. This appendix provides detailed, real-world comparisons of the dominant inference platforms: Intel Xeon Scalable (Platinum SPR), AMD EPYC Genoa, Apple M4 Max, Qualcomm Snapdragon 8 Elite, Raspberry Pi 5, and NVIDIA H100/H200. Each row is measurable on real hardware today.
2. Complete Hardware Comparison Matrix#
Diagram
Loading…3. Detailed Platform Analysis#
3.1 Intel Xeon Platinum SPR (4th Gen Sapphire Rapids)#
Text
Specifications:
- Cores: 32, 40, 60 (different SKUs)
- Architecture: P-core (Performance), E-core (Efficiency) hybrid
- Clock: 2.0-3.8 GHz base/turbo
- Instruction set: AVX-512 (crucial for inference optimization)
- Memory: 12 channels DDR5
Inference Strengths:
+ Mature ecosystem (TVM, ONNX Runtime heavily optimized)
+ AVX-512 enables 4-8x speedup for int8 matmul
+ Per-core turbo allows single-threaded latency optimization
+ Cost-effective for batch serving (good FLOPS/dollar)
Weaknesses:
- High power (350-500W) limits edge deployment
- Memory bandwidth per-core lower than specialized hardware
- Requires tuning (thread pool, affinity, vectorization)
Realistic ResNet-50 Performance (FP16, B=128):
- Throughput: 108 samples/sec per socket
- Total throughput (2 sockets): 216 samples/sec
- Power: 450W TDP ÷ 216 = 2.1 W/sample
Use case: Enterprise batch inference (image tagging, recommendation prefetch)
Typical deployment: 2-socket systems in data centers3.2 AMD EPYC Genoa (9004 Series)#
Text
Specifications:
- Cores: 12-128 (extreme range of SKUs)
- Architecture: Zen 5 with 3D V-Cache (stacked cache)
- Clock: 2.45-3.7 GHz
- Memory: 12 channels DDR5, up to 768GB per CPU
Inference Strengths:
+ More cores than Intel (128 vs 60) enables higher throughput
+ 3D V-Cache increases cache hit rate (~3x more cache)
+ Better power efficiency (lower power per core)
+ Competitive cost
Weaknesses:
- Newer ecosystem (fewer optimizations than Intel)
- AVX-512 support limited vs Intel
- Some frameworks have better Intel tuning
Realistic Performance (vs Intel):
- Same FP32 latency: ~12.3ms (slightly better than Intel)
- Better aggregate throughput: 164 samples/sec (vs 108)
- Better power efficiency: 41 mW/TFLOP (vs 57)
Comparison: AMD is 50% faster for batch throughput, slightly more efficient
Viable alternative where cost is critical3.3 Apple M4 Max (Mobile Processor)#
Text
Specifications:
- Cores: 10 (8 Performance + 2 Efficiency)
- Architecture: ARM-based, custom Apple design
- Clock: 3.5-4.4 GHz
- GPU: 10-core integrated GPU
Inference Strengths:
+ Exceptional efficiency (4.3 mW/TFLOP, best in class)
+ Low power enables passive cooling (no fan needed)
+ Integrated GPU & ML accelerators (Neural Engine)
+ macOS/iOS ecosystem
Weaknesses:
- Limited to Apple devices
- Smaller batch sizes (not optimized for 256-batch inference)
- Less mature ML ecosystem vs x86
Realistic Performance:
- ResNet-50: 6.2ms (FP32), 4.1ms (FP16) single image
- BERT-base: 16ms (FP32) for single sample
- LLaMA-7B: 850ms prefill, 140ms/token decode (slow on CPU)
Use case: On-device ML (macOS apps, iPad ML)
Not suitable for: Data center inference3.4 Qualcomm Snapdragon 8 Elite (Mobile SoC)#
Text
Specifications:
- Cores: 8 (3 Performance + 5 Efficiency)
- ISA: ARM v9.2 with custom accelerators
- Hexagon DSP: Custom DSP for NN acceleration
- Memory: LPDDR5X (102 GB/s)
Inference Strengths:
+ Ultra-low power (5-8W total SoC)
+ Hardware NN accelerator (Hexagon DSP)
+ Qualcomm NPU (Neural Processing Unit)
+ 1 billion+ phones deployed
Weaknesses:
- Power efficiency only for <1-second inference
- Limited for long-running batch jobs
- Framework support less mature than mobile
Realistic Performance:
- ResNet-50: 8.5ms (CPU only), 2ms (with Hexagon acceleration)
- BERT-base: Not practical (too slow on mobile)
Use case: Real-time mobile AI (camera, voice, gesture)
Best for: Single-request latency <100ms3.5 Raspberry Pi 5#
Text
Specifications:
- CPU: Broadcom BCM2712 (4-core ARM)
- Clock: 2.4 GHz
- Memory: 4-8GB LPDDR5
- I/O: PCI-e for GPU acceleration
Inference Strengths:
+ Extremely low cost ($60-100)
+ Low power (5W)
+ Educational/hobbyist use
Weaknesses:
- Very slow for modern models (125ms per ResNet-50 image!)
- Only viable for tiny models (<50M params)
- Not practical for production deployment
Realistic Performance:
- ResNet-50: 125ms (FP32), 65ms (FP16), 32ms (INT8)
- Throughput: 8 images/sec (vs 100+ for servers)
- MobileNet-v2: 20ms, viable for real-time
Use case: Educational projects, IoT edge inference
Not suitable for: Production ML services3.6 NVIDIA H100 GPU#
Text
Specifications:
- CUDA Cores: 21,888
- Architecture: Hopper (4nm process)
- Memory: 80GB HBM3 (3,360 GB/s bandwidth)
- NVLink: 7 × 400 GB/s (connect 8 GPUs)
- Tensor Cores: FP8, BF16, TF32, FP32, FP64
Inference Strengths:
+ Extreme throughput: 1000x faster than CPUs for matrix ops
+ Unified memory model (all GPU types)
+ Mature ecosystem (CUDA, TensorRT, etc.)
+ Production battle-tested
Weaknesses:
- High cost ($40K per GPU)
- High power (700W) requires data center infrastructure
- Latency for single-request (<50ms perfect, not optimized for <5ms)
- Not suitable for <1W applications
Realistic Performance (Batch-focused):
- ResNet-50: 0.65ms (FP32), 0.35ms (FP16) with B=128
- Throughput: 196,923 samples/sec (vs 108 on CPU!)
- 1,818x faster than CPU for batch inference
Comparison: CPU vs H100 for 1M images
- CPU: 1M ÷ 108 = 9,260 seconds = 2.6 hours
- H100: 1M ÷ 196,923 = 5 seconds (1,852x speedup!)
- Cost: H100 $40K, but cost amortized over 3-5 year lifetime
Use case: Cloud AI services (OpenAI, Anthropic, etc.)
Production deployment: 8x H100 with NVLink per service4. Cost-Performance Analysis#
4.1 Cost Per Inference (Amortized)#
Diagram
Loading…4.2 "Bang for Buck" (Throughput per Dollar)#
Diagram
Loading…5. Deployment Decision Matrix#
Diagram
Loading…6. Summary & Key Takeaways#
For batch inference (>100 req/sec):
- Best performance: NVIDIA H100 (1000x CPU throughput)
- Best cost-performance: Intel Xeon SPR or AMD EPYC
- Power budget: H100 requires 700W, SPR 350-500W
For real-time inference (<50ms latency):
- Best: Apple M4 Max (6ms for ResNet, 4ms FP16)
- Good: Snapdragon 8E (with NPU acceleration)
- Worst: Raspberry Pi (125ms)
For production deployment:
- H100: Necessary for 10K+ req/sec (cloud APIs)
- SPR/Genoa: Sweet spot for 1-5K req/sec with cost sensitivity
- M4/Snapdragon: Only option for mobile/edge
Hardware trends (2024-2025):
- Specialized inference chips becoming competitive (AWS Neuron, Apple Neural Engine)
- Mobile chips approaching laptop performance
- CPU inference closing gap through vectorization (AVX-512, SVE)
- Quantization essential on all platforms (Int8 ubiquitous)
Next steps:
- Measure your actual workload (batch size, latency SLO)
- Benchmark on target hardware (don't trust specs)
- Account for 30% overhead (framework, queuing, etc.)
- Add 20% headroom for spikes
- Evaluate total cost of ownership (not just hardware)