MLsys · Appendix B
Model Memory Sizing Calculator
1. Introduction#
Predicting model memory requirements is critical for deployment planning: OOM (out-of-memory) errors cause cascading failures, while over-provisioning wastes hardware costs. This appendix provides exact formulas and a complete Python calculator for predicting memory requirements across all major model families: vision transformers, language models, diffusion models, and multimodal systems.
The key insight: total memory = parameter memory + activation memory + KV-cache memory + overhead. Each component scales differently with batch size, sequence length, and precision.
2. Exact Memory Formulas#
2.1 Parameter Memory#
Parameters are learned weights, stored once in memory (before inference).
Parameter Memory (GB) = (num_parameters × bytes_per_parameter) / 10^9
Precision to bytes per parameter:
- FP32: 4 bytes
- FP16: 2 bytes
- INT8: 1 byte
- INT4: 0.5 bytes
Example: BERT-large
num_parameters = 340M
FP32: 340M × 4 bytes = 1.36 GB
FP16: 340M × 2 bytes = 0.68 GB
INT8: 340M × 1 byte = 0.34 GB2.2 Activation Memory#
Activations are intermediate results during forward pass, recomputed each batch.
Activation memory depends on:
- Batch size (B)
- Sequence length (S)
- Hidden dimension (H)
- Number of layers (L)
For Transformer:
Activation memory ≈ B × S × H × L × 4 bytes × (1 + recomputation_factor)
Recomputation factor:
- No recomputation: 1.0 (keep all activations in memory)
- Full recomputation: 0.1 (recompute, minimal storage)
- Selective: 0.3-0.5 (trade compute for memory)
Example: BERT-base with B=32, S=384, H=768, L=12
Activation memory = 32 × 384 × 768 × 12 × 4 / 10^9 ≈ 1.19 GB
With full recomputation: 1.19 × 0.1 ≈ 0.12 GB2.3 KV-Cache Memory (Autoregressive Models)#
During autoregressive generation, KV (key-value) cache grows with sequence length.
KV-Cache per token (bytes) = 2 × num_heads × head_dim × num_layers × batch_size × 2
For LLaMA-7B (num_heads=32, head_dim=128, layers=32):
Per token per batch = 2 × 32 × 128 × 32 × batch_size × 2 bytes
= 524,288 × batch_size bytes
≈ 0.5 MB per token per batch element
For 2048 token sequence:
KV-cache = 0.5 MB/token × 2048 tokens ≈ 1.0 GB per batch element
Critical: KV-cache can exceed parameter memory for large sequences!
Optimization: Use key-value cache quantization (INT8) to reduce by 4x2.4 Total Memory During Inference#
Total memory = Parameter memory + Activation memory + KV-cache + Overhead
Overhead includes:
- Optimizer states (Adam: 8 bytes per param for momentum + variance)
- Gradient buffers: 4 bytes per param (FP32 gradients)
- Framework overhead: 10-20% of above
Example: Fine-tuning LLaMA-7B
- Parameter memory (FP32): 28 GB
- Activations (B=4, S=2048): 2 GB
- KV-cache (B=4, S=2048): 4 GB
- Optimizer states: 28 × 8 = 224 GB (!!)
- Total for training: ~258 GB (impossible on single GPU!)
Solution: Use LORA (8-bit quantization + parameter-efficient fine-tuning)
- Parameter memory: 28 GB (quantized to 7 GB)
- LORA adapters: 0.1-0.2 GB
- KV-cache: 4 GB
- Total: ~11 GB (fits on single A100 80GB)3. Complete Python Calculator#
Loading…4. Model Family Memory Reference#
4.1 Language Models#
Loading…4.2 Large Language Models (Decoder-only)#
Loading…4.3 Vision Models#
Loading…5. Hardware Requirements by Model#
Loading…6. Optimization Strategies#
6.1 Reducing Memory Without Quality Loss#
Loading…6.2 KV-Cache Optimization#
KV-cache is often the bottleneck for long sequences. Optimizations:
1. Quantize KV-cache: 4x reduction
- FP32 → FP16: 2x
- FP16 → INT8: 2x more
- Combined: 4x (minimal quality loss)
2. Sparse attention: 10-100x reduction
- Only attend to recent tokens
- Trade: Requires architecture change
3. Paged attention: Flexible memory
- Allocate in pages, not contiguous
- Enables larger batch sizes without OOM
4. Multi-query attention: 4-8x reduction
- Single head for K, V (vs per-head in MQA)
- Minimal quality loss7. Summary & Key Takeaways#
Memory prediction formula:
Total Memory (GB) = (Parameters × bytes/param +
Activation +
KV-cache) / 10^9Quick estimates:
- Parameters dominate for small sequences
- KV-cache dominates for long sequences (>1K tokens)
- Quantization essential: 4x reduction without accuracy loss
- For 70B+ models: INT8 or INT4 quantization mandatory
Hardware matching:
- <10GB model: RTX 4090 / RTX 3090
- 10-40GB model: A100 (40GB) / A10
- 40-80GB model: A100 (80GB) / H100 (80GB)
- 80GB+ model: Multi-GPU or CPU offloading
Next steps:
- Calculate exact memory requirements for your model
- Choose appropriate precision (FP32 vs FP16 vs INT8)
- Test batch size limits
- Implement KV-cache optimization if needed
- Measure actual memory usage (may differ from formula)