MLsys · Appendix B

Model Memory Sizing Calculator

1. Introduction#

Predicting model memory requirements is critical for deployment planning: OOM (out-of-memory) errors cause cascading failures, while over-provisioning wastes hardware costs. This appendix provides exact formulas and a complete Python calculator for predicting memory requirements across all major model families: vision transformers, language models, diffusion models, and multimodal systems.

The key insight: total memory = parameter memory + activation memory + KV-cache memory + overhead. Each component scales differently with batch size, sequence length, and precision.


2. Exact Memory Formulas#

2.1 Parameter Memory#

Parameters are learned weights, stored once in memory (before inference).

Text
Parameter Memory (GB) = (num_parameters × bytes_per_parameter) / 10^9

Precision to bytes per parameter:
- FP32: 4 bytes
- FP16: 2 bytes
- INT8: 1 byte
- INT4: 0.5 bytes

Example: BERT-large
num_parameters = 340M
FP32: 340M × 4 bytes = 1.36 GB
FP16: 340M × 2 bytes = 0.68 GB
INT8: 340M × 1 byte = 0.34 GB

2.2 Activation Memory#

Activations are intermediate results during forward pass, recomputed each batch.

Text
Activation memory depends on:
- Batch size (B)
- Sequence length (S)
- Hidden dimension (H)
- Number of layers (L)

For Transformer:
Activation memory ≈ B × S × H × L × 4 bytes × (1 + recomputation_factor)

Recomputation factor:
- No recomputation: 1.0 (keep all activations in memory)
- Full recomputation: 0.1 (recompute, minimal storage)
- Selective: 0.3-0.5 (trade compute for memory)

Example: BERT-base with B=32, S=384, H=768, L=12
Activation memory = 32 × 384 × 768 × 12 × 4 / 10^9 ≈ 1.19 GB
With full recomputation: 1.19 × 0.1 ≈ 0.12 GB

2.3 KV-Cache Memory (Autoregressive Models)#

During autoregressive generation, KV (key-value) cache grows with sequence length.

Text
KV-Cache per token (bytes) = 2 × num_heads × head_dim × num_layers × batch_size × 2

For LLaMA-7B (num_heads=32, head_dim=128, layers=32):
Per token per batch = 2 × 32 × 128 × 32 × batch_size × 2 bytes
                    = 524,288 × batch_size bytes
                    ≈ 0.5 MB per token per batch element

For 2048 token sequence:
KV-cache = 0.5 MB/token × 2048 tokens ≈ 1.0 GB per batch element

Critical: KV-cache can exceed parameter memory for large sequences!
Optimization: Use key-value cache quantization (INT8) to reduce by 4x

2.4 Total Memory During Inference#

Text
Total memory = Parameter memory + Activation memory + KV-cache + Overhead

Overhead includes:
- Optimizer states (Adam: 8 bytes per param for momentum + variance)
- Gradient buffers: 4 bytes per param (FP32 gradients)
- Framework overhead: 10-20% of above

Example: Fine-tuning LLaMA-7B
- Parameter memory (FP32): 28 GB
- Activations (B=4, S=2048): 2 GB
- KV-cache (B=4, S=2048): 4 GB
- Optimizer states: 28 × 8 = 224 GB (!!)
- Total for training: ~258 GB (impossible on single GPU!)

Solution: Use LORA (8-bit quantization + parameter-efficient fine-tuning)
- Parameter memory: 28 GB (quantized to 7 GB)
- LORA adapters: 0.1-0.2 GB
- KV-cache: 4 GB
- Total: ~11 GB (fits on single A100 80GB)

3. Complete Python Calculator#

Diagram
Loading…

4. Model Family Memory Reference#

4.1 Language Models#

Diagram
Loading…

4.2 Large Language Models (Decoder-only)#

Diagram
Loading…

4.3 Vision Models#

Diagram
Loading…

5. Hardware Requirements by Model#

Diagram
Loading…

6. Optimization Strategies#

6.1 Reducing Memory Without Quality Loss#

Diagram
Loading…

6.2 KV-Cache Optimization#

Text
KV-cache is often the bottleneck for long sequences. Optimizations:

1. Quantize KV-cache: 4x reduction
   - FP32 → FP16: 2x
   - FP16 → INT8: 2x more
   - Combined: 4x (minimal quality loss)

2. Sparse attention: 10-100x reduction
   - Only attend to recent tokens
   - Trade: Requires architecture change

3. Paged attention: Flexible memory
   - Allocate in pages, not contiguous
   - Enables larger batch sizes without OOM

4. Multi-query attention: 4-8x reduction
   - Single head for K, V (vs per-head in MQA)
   - Minimal quality loss

7. Summary & Key Takeaways#

Memory prediction formula:

Text
Total Memory (GB) = (Parameters × bytes/param +
                     Activation +
                     KV-cache) / 10^9

Quick estimates:

  • Parameters dominate for small sequences
  • KV-cache dominates for long sequences (>1K tokens)
  • Quantization essential: 4x reduction without accuracy loss
  • For 70B+ models: INT8 or INT4 quantization mandatory

Hardware matching:

  • <10GB model: RTX 4090 / RTX 3090
  • 10-40GB model: A100 (40GB) / A10
  • 40-80GB model: A100 (80GB) / H100 (80GB)
  • 80GB+ model: Multi-GPU or CPU offloading

Next steps:

  1. Calculate exact memory requirements for your model
  2. Choose appropriate precision (FP32 vs FP16 vs INT8)
  3. Test batch size limits
  4. Implement KV-cache optimization if needed
  5. Measure actual memory usage (may differ from formula)