ML Systems Engineering · Part 1 · Module 4
Knowledge Distillation — Systems Perspective
Teacher-student frameworks, distillation losses, production pipelines
A Comprehensive Treatment for ML Systems Engineers#
Author's Note: This module synthesizes recent advances in neural network compression through the lens of systems thinking. Distillation has evolved from a post-hoc compression technique to a foundational design pattern for efficient inference. We examine both classic offline distillation and modern deployment-time approaches like speculative decoding, with rigorous mathematical foundations and production-ready implementations.
1. SYSTEMS OVERVIEW: Distillation as Offline Compression#
1.1 Problem Formulation#
Knowledge distillation (KD) occupies a critical position in the ML systems pipeline. Unlike post-training quantization (PTQ), which applies static compression after training completes, distillation performs online compression during training, allowing the model to adapt its feature representations to preserve task-relevant information while reducing model capacity.
The fundamental value proposition of distillation emerges from a systems design perspective:
Cost-Benefit Analysis:
- Training Cost: Requires additional training iterations with a teacher model on full data
- Inference Benefit: Reduced model size (parameters), reduced memory bandwidth, reduced latency
- Decision Point: When
For a model deployed across inference instances for duration :
Quantitative Framework:
Given:
- Teacher model: parameters , latency per token
- Student model: parameters (target )
- Distillation training time: (where is the training overhead multiplier, typically )
- Expected inference instances:
- Deployment duration: (months/years)
- Inference cost per unit time: proportional to (speedup)
The break-even analysis determines viability:
where and are teacher and student latencies respectively.
1.2 Decision Matrix: When to Use Distillation#
| Scenario | PTQ Sufficient? | Distillation Justified? | Reasoning |
|---|---|---|---|
| High-volume inference (>1M req/day), latency-sensitive | No | Yes | Cumulative speedup benefits > training cost |
| Single-instance edge deployment | Maybe | Maybe | Depends on device computational power |
| Research/exploratory phase | Yes | No | Frequent model changes amortize distillation poorly |
| Production LLM serving (billions of requests) | No | Yes | Multi-month ROI from microsecond improvements |
| Embedded/IoT with storage constraints | Yes (if size) | Sometimes | PTQ often sufficient; distillation if speed critical |
1.3 The Distillation Spectrum#
Distillation exists on a spectrum of training-time vs deployment-time approaches:
Training-Time Deployment-Time
| |
Standard KD Task-Specific KD Adapter KD Speculative Decoding
Static Student Dynamic Draft Model
Fully Trained Generated On-DemandKey distinctions:
- Standard KD: One fixed student model trained once
- Task-Specific KD: Different students for different downstream tasks
- Adapter-based KD: Shared representations with task-specific heads
- Speculative Decoding: Draft model selected at runtime; different teacher-student pairs for different input distributions
- Self-Distillation: Model distills from intermediate/ensemble checkpoints of itself
2. THEORETICAL FOUNDATION#
2.1 Core Mathematics: KL Divergence and Temperature Scaling#
2.1.1 Standard Distillation Loss
The distillation objective combines task loss with knowledge transfer loss:
where:
- are student and teacher logits respectively
- is the ground truth label
- balances task learning and knowledge transfer
- is the knowledge transfer loss, typically KL divergence
The temperature-scaled softmax introduces a crucial hyperparameter:
where is the temperature parameter.
Derivation of temperature scaling:
The KL divergence between teacher and student softmax distributions is:
where superscripts denote teacher (T) and student (S) distributions.
Substituting temperature-scaled softmax:
Critical insight: As , the softmax approaches uniform distribution:
This creates a smoothing effect that:
- Reveals "dark knowledge" (information in soft targets)
- Reduces probability mass concentration on correct label
- Creates richer gradient signal for student learning
Gradient analysis:
For the distillation loss with respect to student logits:
The factor scales the gradient. Key observation:
- High (): Smoother probability distributions, gentler gradients (good for knowledge transfer)
- Low (): Sharper probability distributions, steeper gradients (good for task fitting)
Optimal temperature selection remains dataset-dependent, but empirical values often range:
- Vision tasks:
- NLP tasks:
- Large LLMs:
2.1.2 Weighted KD Loss with Distillation Weight Scheduling
Advanced implementations employ curriculum-based weight scheduling:
where increases from initial (emphasizing task learning) to final (emphasizing knowledge transfer).
This addresses a fundamental tension:
- Early training: Student needs strong signal on true labels
- Late training: Student should internalize teacher's fine-grained distinctions
Complete KD loss with scheduling:
2.2 Task-Specific vs General-Purpose Distillation#
2.2.1 General-Purpose Distillation
Objective: Train a single student that preserves teacher's general representations across diverse tasks.
Use case: Foundation models with multiple downstream applications (BERT → sentiment analysis, NER, QA, etc.)
Architecture: Typically uses layer-wise knowledge transfer:
where are hidden representations at layer .
Advantage: Single model works across tasks Disadvantage: Must compromise on task-specific optimization
2.2.2 Task-Specific Distillation
Objective: Optimize student for particular downstream task.
Architecture: Teacher trained on target task; student distilled to match teacher's task-specific representations.
Loss formulation:
where are teacher and student probability distributions on the specific task.
Empirical observation: Task-specific distillation achieves 15-30% better compression ratios than general-purpose, at the cost of training multiple students.
2.3 DistilBERT: Case Study in Effective Distillation#
Reference: Sanh et al. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter"
DistilBERT represents a landmark in production distillation:
- Teacher: BERT-base (110M parameters)
- Student: 2x layer reduction → 66M parameters (40% reduction)
- Training: Standard MLM task on full BERT corpus
- Results: 97% accuracy retention with 2x speedup, 40% model size reduction
Key technical insights:
-
Aggressive layer reduction: Removing every other layer (12→6) while maintaining width proved highly effective
-
Triple loss combination: where is cosine similarity loss between pooled representations
-
Temperature and weighting:
- (relatively low, preserving gradient sharpness)
- (heavy emphasis on KD loss)
- Cosine loss weight:
-
Why it works:
- BERT encodes substantial task-agnostic linguistic knowledge in intermediate layers
- Layer reduction forces meaningful dimensionality selection
- Cosine loss provides geometric constraint on learned representations
Deployment characteristics:
- Inference latency: ~2x speedup on GPU, ~3-4x on CPU (layer count reduction is compute-bound on CPU)
- Memory bandwidth: 40% reduction (proportional to parameter count)
- Quantization-friendly: Distilled models quantize to INT8 with better accuracy than quantized teachers
2.4 TinyBERT: Pushing Compression Further#
Reference: Jiao et al. (2020). "TinyBERT: Distilling BERT for Natural Language Understanding"
TinyBERT achieves 15-20x compression while maintaining >95% of teacher accuracy on GLUE:
Key innovations:
-
Progressive layer reduction:
- Stage 1: General-purpose distillation on MLM
- Stage 2: Task-specific adaptation (further tuning on downstream tasks)
- Final model: 4 layers, 312 hidden dimensions (vs BERT-base: 12 layers, 768 dims)
-
Multi-granularity knowledge distillation:
where each component transfers different aspects of knowledge:
- Embedding layer: Token embedding matrix alignment
- Attention: Layer-wise attention map matching via MSE
- Hidden states: Transformation + cosine similarity
- Output: Standard KD on logits
-
Attention transfer mechanism: For attention matrix (teacher) and (student):
where is number of attention heads. This forces student to match teacher's attention patterns.
-
Layer-wise mutual learning: Intermediate layers of teacher provide supervision to all student layers:
Practical results:
- MNLI: 15.2x compression with 95.3% of teacher accuracy
- SST-2: 18.4x compression with 94.8% of teacher accuracy
- Inference speedup: 9-15x on GPU (layer count + dimension reduction), 25-35x on CPU
Why CPU speedup is dramatic: CPU compute is memory-bandwidth-bound for transformer inference. Reducing from 312→768 dims and 4→12 layers provides disproportionate speedup on CPU vs GPU.
2.5 MobileBERT: Mobile-First Distillation#
Reference: Sun et al. (2020). "MobileBERT: Task-Agnostic Distillation of BERT for Efficient Mobile Solutions"
MobileBERT targets mobile inference with explicit optimization for mobile hardware constraints.
Architecture design principles:
-
Bottleneck structure:
TextInput → Wide Layer → Bottleneck → Wide Layer → OutputEmulates mobile neural networks (MobileNet) design pattern where:
- Wide layers (768 dims) enable expressiveness
- Bottlenecks (256 dims) reduce computation
- Pattern repeats, reducing total FLOPs
-
Knowledge adaptation module: Adds learnable projection layer between teacher and student:
Improves knowledge transfer when teacher and student dimensions differ significantly.
-
Distillation loss:
where matches intermediate feature representations through adapters.
Mobile hardware considerations:
- Memory constraints: 2-4GB RAM typical on mobile; MobileBERT fits entirely in cache
- Computation: 4-8 CPU cores; benefits from layer reduction more than width reduction
- Power: Every ML operation draws from battery; total MACs and memory access both critical
Results:
- Model size: 25MB (vs BERT-base 400MB+)
- Inference latency: 170ms per sequence on Pixel 4 (vs 3.5s for BERT)
- Battery draw: 80% reduction due to computation reduction
- Accuracy: 92% of BERT on GLUE average
2.6 Distillation for Large Language Models#
LLM distillation presents novel challenges compared to encoder-only models.
2.6.1 Sequence-Level Knowledge Distillation
Challenge: LLMs generate variable-length sequences; match cross-entropy minimization of multi-token sequences.
Approach: Direct logit matching at each token position:
where are teacher/student distributions over vocabulary at position .
Key distinction from encoder KD: Student must generate text at inference time; knowledge must be embedded in autoregressive generation dynamics.
2.6.2 Token-Level Knowledge Distillation
Refinement: Weight tokens by importance:
where weights increase for:
- High-entropy positions (model uncertainty)
- Positions with large teacher-student divergence
- Task-critical tokens (e.g., final answer token)
Adaptive weighting scheme:
where (emphasize difficult positions).
2.7 Speculative Decoding: Deployment-Time Distillation#
Speculative decoding represents a paradigm shift: rather than training a separate student model offline, generate candidate tokens with a fast draft model at inference time, then verify with teacher model.
2.7.1 Core Algorithm and Motivation
Problem statement: Teacher LLM bottleneck is autoregressive token generation (1 token per forward pass). Can we generate multiple candidate tokens per teacher forward pass?
Solution: Use draft model (smaller/faster) to speculatively generate tokens, then verify with teacher in parallel using KV-cache.
Algorithmic flow:
Input: prompt x, teacher model p_T, draft model p_d, target length T_target
1. Encode input and initialize KV-cache for teacher
2. For t = 1 to T_target:
a. Draft model generates γ tokens: x_{t}, x_{t+1}, ..., x_{t+γ-1}
b. Append draft tokens to sequence
c. Teacher processes all γ tokens in parallel using KV-cache
d. For each draft token i in 1..γ:
- If p_T(x_{t+i} | x_{t+i-1}) > p_d(x_{t+i} | x_{t+i-1}):
Accept token, continue
- Else:
Reject remaining draft tokens
Sample replacement from p_T using rejection sampling
e. Update sequence and KV-cache2.7.2 Acceptance-Rejection Scheme: Full Derivation
This is the mathematical core of speculative decoding.
Objective: Maintain exact distribution matching despite using draft model.
Key lemma: Given draft distribution and teacher distribution , construct acceptance probability such that:
Construction: Acceptance probability for token :
Proof of correctness:
Let be the distribution after acceptance:
We construct:
where .
For the acceptance case:
For tokens where :
- Acceptance probability = 1
- Contributes to final distribution
For tokens where :
- Acceptance probability =
- Contributes to final distribution
Rejection-sampling correction:
For rejected tokens, sample from correction distribution:
Complete proof:
Consider partition of vocabulary into three sets:
- (teacher prefers)
- (draft prefers)
Total probability from :
Wait, this requires careful analysis. Let me restate correctly:
For : Always accept draft sample, contributing For : Reject with probability
When rejection occurs, sample from distribution designed to add exactly to final probability.
Total probability mass:
This proves the algorithm maintains teacher distribution exactly.
2.7.3 Acceptance Rate Analysis
Critical metric: Acceptance rate .
Expected acceptance rate for draft token at position :
This is the Hellinger affinity between distributions (normalized).
Relationship to KL divergence:
By Pinsker's inequality:
Practical implications:
If , then:
For : (77.6% tokens accepted) For : (48.4% tokens accepted) For : (19.2% tokens accepted)
Speedup formula:
where is draft budget (tokens drafted per teacher forward pass).
With and :
2.7.4 KV-Cache Rollback Mechanism
Challenge: Draft model generates tokens that may be rejected; KV-cache must be rolled back.
Mechanism:
-
Save branching point: Store KV-cache before draft forward passes
-
Draft generation: Generate tokens, computing intermediate KV-caches
-
Teacher verification: Process all draft tokens with teacher in parallel:
-
Acceptance checking:
- If position (first rejection): Rollback to
- Sample replacement token from rejection distribution
- Continue from that point
- If all positions accepted: Continue from
Memory implications:
- Store additional KV values
- With , seq_len = 2048, model = 7B: ~5GB additional memory
- Acceptable on modern GPUs (80GB H100)
2.8 Self-Speculative Decoding: Early Exit Layers#
Insight: Train model with intermediate exit points; early layer can serve as draft model.
Architecture:
Input
↓
Layer 1 → [Early Exit Head] → Can draft from here
↓
Layer 2 → [Early Exit Head]
↓
...
↓
Layer 12 → [Final Output Head]Key advantage: Single model, no separate draft model needed.
Training objective:
where decreases with depth (emphasis on early exit accuracy).
Inference algorithm:
- Pass input through early layers (layer 6, for example)
- Generate draft tokens using early-exit logits
- Continue through remaining layers (layers 7-12) to verify
Speedup analysis:
If early layer provides draft:
- Draft computation:
- Verification:
For (halfway):
Practical tradeoff: Simpler engineering (single model) but 20-40% less speedup compared to optimized draft model.
2.9 Medusa: Multi-Head Speculative Decoding#
Reference: Cai et al. (2024). "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads"
Medusa addresses a fundamental challenge: speculative decoding assumes token acceptance is independent, but this fails when draft model consistently diverges from teacher.
2.9.1 Tree-Based Speculation
Rather than linear speculation ( tokens in sequence), generate branching tree of candidate sequences:
Loading…Advantages:
- Hedges against draft model uncertainty
- Increases probability that teacher agrees with at least one branch
- Parallel verification of multiple branches
2.9.2 Multi-Head Architecture
Medusa adds multiple prediction heads at different layers:
Input → Layer 1 → Head 1: predicts next token
Layer 2 → Head 2: predicts 2nd token
Layer 3 → Head 3: predicts 3rd token
...
Layer 32 → Final head (teacher)Head structure:
where is hidden state at layer .
Key insight: Each head operates at different layers of the transformer, capturing different levels of semantic understanding.
2.9.3 Tree Attention Implementation
Challenge: Verify multiple token sequences in parallel without exponential expansion.
Solution: Batch verification with tree attention structure.
Attention mask construction:
For tree with branching factor and depth :
Sequence: [s_1, s_2, s_3, ...]
+ [h_1, h_2, h_3] (first layer of heads)
+ [h_11, h_12, h_21, h_22, h_31, h_32] (second layer)
Attention mask:
- All positions attend to all previous positions in main sequence
- Each head attends to parent node in tree
- Head nodes attend to token position that generated themComputational benefit:
Suppose candidate pool = sequences, verification depth = :
- Naive: separate forward passes = teacher layers
- Tree attention: Single forward pass with tree mask = teacher layers
Speedup:
2.10 EAGLE: Feature-Level Autoregressive Prediction#
Reference: Li et al. (2024). "EAGLE: Speculative Decoding with Autoregressive Language Model Heads"
EAGLE differs fundamentally from Medusa: rather than predicting tokens directly, predicts hidden states at teacher's layer, then projects to vocabulary.
2.10.1 Architecture
Loading…Key difference: Predicts features, not logits.
2.10.2 Feature Prediction Advantage
Claim: Predicting features is easier than predicting tokens directly.
Intuition:
- Hidden states are continuous, more predictable than discrete tokens
- Feature space smoother than vocabulary space
- Model learns to predict natural feature trajectories
Quantitative analysis:
Let be feature space, be vocabulary.
Token prediction:
Feature prediction:
The KL divergence depends on learned projection ; with good features, even imperfect yields high token-level agreement.
2.10.3 Training EAGLE Heads
Supervised learning objective:
where is the prediction function (MLP + attention).
Key training detail: Use teacher representations as targets, not random initialization.
Multi-step prediction:
For draft depth :
Each trained separately to minimize feature prediction error at that depth.
2.10.4 Empirical Results
Typical configuration (LLaMA 7B):
- Draft depth:
- Acceptance rate: 0.75-0.85 (higher than token-level speculation due to feature smoothness)
- Speedup: 2.0-2.5x end-to-end
- Model size overhead: +5-8% parameters for EAGLE heads
3. HARDWARE MAPPING: Why Speculative Decoding Excels on CPU and Apple Silicon#
3.1 Compute Utilization: GPU vs CPU vs Apple Silicon#
The dramatic effectiveness of speculative decoding on CPU stems from fundamental hardware differences.
3.1.1 GPU (NVIDIA H100)
Computation profile:
- Peak FP16 FLOPS: 1.98 TFLOPS
- Memory bandwidth: 3.35 TB/s
- Compute-to-memory ratio: FLOP per byte
Transformer inference bottleneck (1 token generation):
Attention computation: where
FLOPs: FLOPs Memory access: Reading KV-cache
Arithmetic intensity: FLOPs/byte
Interpretation: GPU is memory-bound; adding tokens increases memory access faster than computation.
With speculative decoding:
- tokens in parallel
- Total FLOPs: FLOPs
- Memory access: (nearly same per-token cost due to KV-cache parallel access)
- Speedup: ~3-4x (memory bandwidth better utilized with larger batches)
3.1.2 CPU (Intel Xeon or AMD EPYC)
Computation profile:
- Peak FP32 FLOPS: ~400 GFLOPS (per socket)
- Memory bandwidth: ~100 GB/s
- Compute-to-memory ratio: FLOPS per byte
Critical difference: CPU has much lower memory bandwidth relative to compute. Single-token generation is severely bandwidth-limited.
Single token inference bottleneck:
- Weights to load: 7B × 2 bytes (FP16) = 14 GB
- Time to load:
- Computation time:
- Total: dominated by 140 ms memory load
With speculative decoding ():
- Draft model (smaller, loaded once): → 20 ms
- Generate 5 draft tokens: (draft model FLOPs)
- Verify 5 tokens with teacher: (naive, without tricks)
- But with KV-cache reuse: Once weights loaded, subsequent tokens use cached KVs
- Effective per-token: ~30 ms (amortized weight loading)
- Speedup: ~3-4x despite "naive" analysis
3.1.3 Apple Silicon (M1/M2/M3)
Computation profile:
- Peak FP16 FLOPS: ~1.1 TFLOPS (8-core GPU, M1)
- Memory bandwidth: 100 GB/s (unified memory architecture)
- Compute-to-memory ratio: FLOPS per byte
Revolutionary aspect: Unified Memory Architecture (UMA)
Traditional GPUs: PCIe bus separates CPU and GPU memory
CPU DRAM (8-16GB) ←→ PCIe (16 GB/s) ←→ GPU DRAM (12-80GB)Apple Silicon: Single address space with zero-copy
Unified Memory: CPU and GPU access same physical RAM at full bandwidth (100 GB/s)Speculative decoding advantage on Apple Silicon:
- Draft model cache-residency: 1-7B draft model fits in shared L3 cache + main RAM
- Zero-copy KV-cache updates: Draft model generates KV vectors; teacher immediately reads them without copying
- Memory bandwidth efficiency: Both draft and teacher access KV-cache at full 100 GB/s
Quantitative speedup:
Scenario: LLaMA 7B teacher + LLaMA 1B draft on M1 Max (32GB unified memory)
Single token (teacher only):
- Weight loading: 14 GB / 100 GB/s = 140 ms
- Computation: 7B × 2 / (1.1 TFLOPS) = 12.7 ms
- Total: ~150 ms
Speculative with 5 draft tokens:
- Draft weight loading: 2 GB / 100 GB/s = 20 ms (once)
- Draft generation (5 tokens): 5 × (2B × 2) / (1.1 TFLOPS) ≈ 18 ms
- Teacher verification (5 tokens parallel): 14 GB + 5×(output computation) ≈ 145 ms
- Total: ~180 ms for 5 tokens = 36 ms per token
- Speedup: 150/36 ≈ 4.2x
3.2 Quantitative Hardware Comparison Table#
| Metric | H100 GPU | CPU (Xeon) | Apple M1 Max |
|---|---|---|---|
| Peak FLOPS | 1.98 TFLOPS | 400 GFLOPS | 1.1 TFLOPS |
| Memory BW | 3.35 TB/s | 100 GB/s | 100 GB/s |
| FLOPS/byte | 0.59 | 4.0 | 11.0 |
| 7B model latency | 20-30ms | 140-200ms | 150-180ms |
| Spec. decode speedup | 2.5-3.5x | 3.5-4.5x | 4.0-5.0x |
| Memory model | Discrete | Discrete | Unified |
Key insight: Speculative decoding is most effective on CPU-like and UMA architectures, where memory bandwidth is precious and draft model can remain cache-resident.
4. IMPLEMENTATION DEEP DIVE#
4.1 Speculative Decoding with Acceptance-Rejection#
Full production-quality implementation:
import torch
import torch.nn.functional as F
from typing import Tuple, Optional
import numpy as np
class SpeculativeDecoder:
"""
Speculative decoding with exact distribution matching.
Reference: Leviathan et al. "Speculative Decoding with Autoregressive Auxiliary Models"
ICML 2023
"""
def __init__(self, teacher_model, draft_model, max_draft_len=5, temperature=1.0):
"""
Args:
teacher_model: Full LLM (e.g., LLaMA 7B)
draft_model: Smaller LLM (e.g., LLaMA 1B)
max_draft_len: Maximum draft tokens to speculate (γ)
temperature: Softmax temperature for sampling
"""
self.teacher = teacher_model
self.draft = draft_model
self.max_draft_len = max_draft_len
self.temperature = temperature
# Statistics tracking
self.stats = {
'total_draft_tokens': 0,
'total_accepted': 0,
'rejections': [], # Position where rejection occurs
'acceptance_rate': []
}
def draft_speculate(self, input_ids: torch.Tensor, cache_teacher,
cache_draft) -> Tuple[torch.Tensor, dict]:
"""
Generate draft tokens using smaller model.
Args:
input_ids: Current sequence [batch_size, seq_len]
cache_teacher: Teacher KV-cache (saved at branching point)
cache_draft: Draft model KV-cache
Returns:
draft_tokens: Generated candidate tokens [batch_size, max_draft_len]
draft_logits: Probabilities from draft model
"""
batch_size = input_ids.shape[0]
draft_tokens = []
draft_logits_list = []
current_input = input_ids
draft_cache = cache_draft
for i in range(self.max_draft_len):
# Draft model forward pass
with torch.no_grad():
outputs = self.draft(
input_ids=current_input[:, -1:], # Only last token needed
past_key_values=draft_cache
)
logits = outputs.logits[:, -1, :] # [batch_size, vocab_size]
draft_cache = outputs.past_key_values
# Sample from draft distribution
probs = F.softmax(logits / self.temperature, dim=-1)
sampled_token = torch.multinomial(probs, num_samples=1) # [batch_size, 1]
draft_tokens.append(sampled_token)
draft_logits_list.append(probs)
# Append to sequence for next iteration
current_input = torch.cat([current_input, sampled_token], dim=1)
draft_tokens = torch.cat(draft_tokens, dim=1) # [batch_size, max_draft_len]
return draft_tokens, draft_logits_list, draft_cache
def verify_and_accept(self, input_ids: torch.Tensor, draft_tokens: torch.Tensor,
draft_logits: list, cache_teacher) -> Tuple[torch.Tensor, int]:
"""
Teacher model verifies draft tokens. Exact distribution matching via
acceptance-rejection sampling.
Args:
input_ids: Original sequence [batch_size, seq_len]
draft_tokens: Candidate tokens from draft [batch_size, max_draft_len]
draft_logits: Draft model logit distributions
cache_teacher: Teacher KV-cache from branching point
Returns:
accepted_tokens: Final accepted tokens
num_accepted: Number of consecutive accepted tokens
"""
batch_size = input_ids.shape[0]
# Create augmented sequence with draft tokens
seq_with_draft = torch.cat([input_ids, draft_tokens], dim=1)
# Teacher model processes all draft tokens in parallel
# (This is the key speedup: parallel verification)
with torch.no_grad():
# Forward pass through teacher using tree attention
# (Full sequence includes original + draft tokens)
outputs = self.teacher(
input_ids=seq_with_draft[:, input_ids.shape[1]:],
past_key_values=cache_teacher,
use_cache=True
)
teacher_logits = outputs.logits # [batch_size, max_draft_len, vocab_size]
teacher_cache = outputs.past_key_values
# Acceptance-rejection scheme
accepted_tokens = []
num_accepted = 0
for i in range(self.max_draft_len):
# Get distributions
draft_prob = draft_logits[i] # [batch_size, vocab_size]
teacher_prob = F.softmax(teacher_logits[:, i, :] / self.temperature, dim=-1)
draft_token = draft_tokens[:, i] # [batch_size]
# Acceptance probability for each example in batch
# α = min(1, p_T(x) / p_d(x))
draft_token_prob = draft_prob.gather(1, draft_token.unsqueeze(1)).squeeze(1)
teacher_token_prob = teacher_prob.gather(1, draft_token.unsqueeze(1)).squeeze(1)
# Avoid division by zero
alpha = torch.minimum(
torch.ones_like(teacher_token_prob),
teacher_token_prob / (draft_token_prob + 1e-10)
)
# Accept/reject for each batch element
accept_mask = torch.rand_like(alpha) < alpha
# For accepted items, use draft token
# For rejected items, resample from teacher
final_tokens = draft_token.clone()
# Rejection-sampling correction for rejected items
rejection_mask = ~accept_mask
if rejection_mask.any():
# Compute rejection distribution:
# p_reject(x) = max(0, p_T(x) - p_d(x)) / (1 - sum_y min(p_T(y), p_d(y)))
min_probs = torch.minimum(draft_prob, teacher_prob)
prob_accept_total = min_probs.sum(dim=1, keepdim=True)
# Correction distribution (normalized)
correction_dist = (teacher_prob - draft_prob).clamp(min=0)
correction_dist = correction_dist / (correction_dist.sum(dim=1, keepdim=True) + 1e-10)
# Sample from teacher using rejection distribution for rejected items
rejected_samples = torch.multinomial(
correction_dist[rejection_mask],
num_samples=1
).squeeze(1)
final_tokens[rejection_mask] = rejected_samples
accepted_tokens.append(final_tokens)
# Check if all batch items accept: if not, stop speculation
if not accept_mask.all():
num_accepted = i
break
else:
num_accepted = i + 1
# Stack accepted tokens
if num_accepted > 0:
accepted_tokens = torch.stack(accepted_tokens[:num_accepted], dim=1)
else:
# All rejected, return empty or resample
accepted_tokens = torch.tensor([], dtype=torch.long)
return accepted_tokens, num_accepted, teacher_cache
def generate(self, input_ids: torch.Tensor, max_new_tokens: int,
temperature: float = 1.0) -> torch.Tensor:
"""
Generate tokens using speculative decoding.
Args:
input_ids: Prompt [batch_size, seq_len]
max_new_tokens: Number of tokens to generate
temperature: Sampling temperature
Returns:
complete_sequence: [batch_size, seq_len + max_new_tokens]
"""
batch_size = input_ids.shape[0]
current_seq = input_ids.clone()
# Initialize KV-caches
with torch.no_grad():
teacher_init = self.teacher(input_ids, use_cache=True)
draft_init = self.draft(input_ids, use_cache=True)
cache_teacher = teacher_init.past_key_values
cache_draft = draft_init.past_key_values
tokens_generated = 0
while tokens_generated < max_new_tokens:
remaining_tokens = max_new_tokens - tokens_generated
draft_budget = min(self.max_draft_len, remaining_tokens)
# Step 1: Speculation
draft_tokens, draft_logits, cache_draft = self.draft_speculate(
current_seq, cache_teacher, cache_draft
)
# Step 2: Verification & Acceptance
accepted_tokens, num_accepted, cache_teacher = self.verify_and_accept(
current_seq, draft_tokens, draft_logits, cache_teacher
)
# Update statistics
self.stats['total_draft_tokens'] += draft_budget
if num_accepted > 0:
self.stats['total_accepted'] += num_accepted
# Step 3: Append accepted tokens
if num_accepted > 0:
current_seq = torch.cat([current_seq, accepted_tokens], dim=1)
tokens_generated += num_accepted
else:
# All rejected: sample from teacher at current position
with torch.no_grad():
teacher_out = self.teacher(
current_seq[:, -1:],
past_key_values=cache_teacher,
use_cache=True
)
logits = teacher_out.logits[:, -1, :]
probs = F.softmax(logits / temperature, dim=-1)
sampled = torch.multinomial(probs, num_samples=1)
current_seq = torch.cat([current_seq, sampled], dim=1)
cache_teacher = teacher_out.past_key_values
tokens_generated += 1
# Compute acceptance rate
if self.stats['total_draft_tokens'] > 0:
acc_rate = self.stats['total_accepted'] / self.stats['total_draft_tokens']
self.stats['acceptance_rate'] = acc_rate
return current_seq
# Usage example:
if __name__ == "__main__":
# Pseudo-code for usage
# decoder = SpeculativeDecoder(teacher_model, draft_model, max_draft_len=5)
# generated = decoder.generate(prompt_ids, max_new_tokens=256)
passKey implementation details:
- Parallel verification: Teacher processes all draft tokens at once (tree attention)
- Distribution matching: Acceptance probability ensures final distribution equals teacher's
- KV-cache management: Reuse cached KVs from branching point
- Batch efficiency: Process multiple sequences in parallel; individual sequences may have different acceptance rates
4.2 Medusa Multi-Head Architecture#
Loading…4.3 EAGLE Feature-Level Prediction#
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional, Tuple
class EAGLEHead(nn.Module):
"""
Feature-level prediction head for EAGLE.
Predicts the next hidden state rather than tokens directly.
"""
def __init__(self, hidden_size: int, intermediate_size: int = 2048):
"""
Args:
hidden_size: Model hidden dimension
intermediate_size: MLP intermediate dimension
"""
super().__init__()
self.hidden_size = hidden_size
# Two-layer MLP for feature prediction
self.fc1 = nn.Linear(hidden_size, intermediate_size)
self.fc2 = nn.Linear(intermediate_size, hidden_size)
self.ln = nn.LayerNorm(hidden_size)
def forward(self, hidden_state: torch.Tensor) -> torch.Tensor:
"""
Args:
hidden_state: [batch_size, hidden_size] or [batch_size, seq_len, hidden_size]
Returns:
predicted_state: Same shape as input, predicting next token's features
"""
x = F.gelu(self.fc1(hidden_state))
x = self.fc2(x)
x = self.ln(x)
return x
class EAGLEModel(nn.Module):
"""
Base model augmented with EAGLE heads for feature prediction.
"""
def __init__(self, base_model, hidden_size: int, num_prediction_heads: int = 5):
"""
Args:
base_model: Pretrained LLM
hidden_size: Hidden dimension
num_prediction_heads: Number of consecutive predictions (draft depth)
"""
super().__init__()
self.base_model = base_model
self.hidden_size = hidden_size
self.num_prediction_heads = num_prediction_heads
# Create cascading prediction heads
self.eagle_heads = nn.ModuleList([
EAGLEHead(hidden_size)
for _ in range(num_prediction_heads)
])
def forward(self, input_ids: torch.Tensor, past_key_values=None,
output_hidden_states: bool = False):
"""
Args:
input_ids: [batch_size, seq_len]
past_key_values: Cached KV states
output_hidden_states: Whether to return all hidden states
Returns:
outputs: Base model outputs
eagle_predictions: List of predicted hidden states
"""
# Get base model output
outputs = self.base_model(
input_ids,
past_key_values=past_key_values,
output_hidden_states=output_hidden_states,
use_cache=True
)
# Feature-level prediction
hidden_states = outputs.hidden_states[-1] if output_hidden_states else \
outputs.last_hidden_state
# Take last token hidden state
last_hidden = hidden_states[:, -1:, :] # [batch_size, 1, hidden_size]
eagle_predictions = []
current_hidden = last_hidden
# Cascade predictions: each head predicts next token's features
for eagle_head in self.eagle_heads:
predicted_hidden = eagle_head(current_hidden)
eagle_predictions.append(predicted_hidden)
current_hidden = predicted_hidden # Use prediction as input to next head
return outputs, eagle_predictions
def get_draft_logits(self, eagle_predictions: list,
lm_head: nn.Module) -> torch.Tensor:
"""
Convert predicted hidden states to logits for verification.
Args:
eagle_predictions: List of predicted hidden states
lm_head: Language model head (same as base model's)
Returns:
draft_logits: [batch_size, num_heads, vocab_size]
"""
draft_logits = []
for pred_hidden in eagle_predictions:
logits = lm_head(pred_hidden) # [batch_size, 1, vocab_size]
draft_logits.append(logits.squeeze(1))
return torch.stack(draft_logits, dim=1) # [batch_size, num_heads, vocab_size]
class EAGLETrainer:
"""
Training utility for EAGLE heads.
"""
def __init__(self, model: EAGLEModel, learning_rate: float = 1e-4):
"""
Args:
model: EAGLEModel instance
learning_rate: Learning rate for training
"""
self.model = model
self.optimizer = torch.optim.AdamW(model.eagle_heads.parameters(),
lr=learning_rate)
self.loss_fn = nn.MSELoss()
def train_step(self, input_ids: torch.Tensor) -> float:
"""
Training step: minimize feature prediction error.
Args:
input_ids: [batch_size, seq_len] training batch
Returns:
loss: Scalar loss value
"""
self.optimizer.zero_grad()
# Forward pass with hidden states
outputs, eagle_predictions = self.model(
input_ids,
output_hidden_states=True
)
# Get teacher hidden states (from base model)
teacher_hiddens = outputs.hidden_states[-1] # [batch_size, seq_len, hidden_size]
# Compute loss: EAGLE predicts shifted positions
total_loss = 0.0
for head_idx, predicted_hidden in enumerate(eagle_predictions):
# Target: hidden state at position t + head_idx + 1
target_start = head_idx + 1
target_end = target_start + 1
if target_end <= teacher_hiddens.shape[1]:
target_hidden = teacher_hiddens[:, target_start:target_end, :]
# MSE loss on features
loss = self.loss_fn(predicted_hidden, target_hidden)
total_loss += loss
# Backward pass
total_loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.eagle_heads.parameters(), 1.0)
self.optimizer.step()
return total_loss.item()
def eagle_generate(model: EAGLEModel, input_ids: torch.Tensor,
max_new_tokens: int, temperature: float = 1.0,
lm_head = None) -> torch.Tensor:
"""
Generation using EAGLE feature-level prediction.
Args:
model: EAGLEModel instance
input_ids: [batch_size, seq_len] prompt
max_new_tokens: Tokens to generate
temperature: Sampling temperature
lm_head: Language model head (default: use model.base_model.lm_head)
Returns:
Generated sequence
"""
batch_size = input_ids.shape[0]
current_seq = input_ids.clone()
if lm_head is None:
lm_head = model.base_model.lm_head
with torch.no_grad():
outputs, _ = model(input_ids, output_hidden_states=True)
cache = outputs.past_key_values
for step in range(max_new_tokens):
# Get base model output and EAGLE predictions
with torch.no_grad():
outputs, eagle_predictions = model(
current_seq[:, -1:],
past_key_values=cache,
output_hidden_states=False
)
cache = outputs.past_key_values
# Get draft logits from EAGLE predictions
draft_logits = model.get_draft_logits(eagle_predictions, lm_head)
# Main token from base model
main_logits = outputs.logits[:, -1, :] # [batch_size, vocab_size]
main_probs = torch.softmax(main_logits / temperature, dim=-1)
# Draft tokens from EAGLE
draft_probs = torch.softmax(draft_logits / temperature, dim=-1)
# Acceptance-rejection scheme
draft_tokens = torch.argmax(draft_logits, dim=-1) # [batch_size, num_heads]
# Accept main token
accepted_tokens = torch.argmax(main_logits, dim=-1).unsqueeze(1) # [batch_size, 1]
# Check if any draft tokens accepted
num_drafted = draft_logits.shape[1]
for draft_idx in range(num_drafted):
draft_token = draft_tokens[:, draft_idx]
draft_prob = draft_probs[:, draft_idx, :].gather(1, draft_token.unsqueeze(1))
main_prob = main_probs.gather(1, draft_token.unsqueeze(1))
alpha = torch.minimum(
torch.ones_like(main_prob),
main_prob / (draft_prob + 1e-10)
)
accept_mask = torch.rand_like(alpha) < alpha
if accept_mask.any():
accepted_tokens = torch.cat([
accepted_tokens,
draft_token.unsqueeze(1)
], dim=1)
else:
break
# Append tokens
current_seq = torch.cat([current_seq, accepted_tokens], dim=1)
return current_seq5. KEY PAPERS AND REFERENCES#
5.1 Foundational Speculative Decoding#
Leviathan et al. (2023). "Speculative Decoding with Autoregressive Auxiliary Models"
- ICML 2023 Oral Presentation
- First formal treatment of speculative decoding
- Proves exact distribution matching with acceptance-rejection
- Demonstrates 2-3x speedup on LLaMA
- Key contribution: Rigorous mathematical framework for draft model selection
5.2 Modern Variants#
Liu et al. (2024). "Online Speculative Decoding"
- ICML 2024
- Adapts draft model online to match teacher at inference time
- Improves acceptance rate from fixed draft model
- ~1.5x improvement over standard speculative decoding
- Key insight: Draft model should adapt to input distribution
Li et al. (2024). "EAGLE: Speculative Decoding by Feature Prediction"
- ICML 2024
- Feature-level rather than token-level prediction
- 2.5-3x speedup on various LLMs
- Better generalization across domains
- Key contribution: Hidden state prediction is more learnable than token prediction
Cai et al. (2024). "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads"
- ArXiv 2024
- Tree-based speculation with multiple heads
- 2.2x speedup with single model
- No separate draft model required
- Key advantage: Simpler to deploy than speculative decoding
5.3 Classical Distillation#
Sanh et al. (2019). "DistilBERT, a distilled version of BERT"
- ICLR 2020
- 40% reduction in model size, 2x speedup, 97% accuracy retention
- Standard KD + attention matching + layer removal
- Landmark work demonstrating practical distillation value
- Key insight: Layer count reduction is more impactful than parameter reduction for latency
Jiao et al. (2020). "TinyBERT: Distilling BERT for Natural Language Understanding"
- ICLR 2021
- 15-20x compression with 95% accuracy
- Multi-granularity knowledge transfer
- Task-specific vs general-purpose distillation
- Key contribution: Progressive distillation methodology
Sun et al. (2020). "MobileBERT: Task-Agnostic Distillation of BERT for Efficient Mobile Solutions"
- ACL 2020
- Mobile-specific architecture design
- Bottleneck structures matching MobileNet patterns
- 25MB model size, 170ms latency on Pixel 4
- Key insight: Hardware-aware distillation design
6. SYSTEMS TRADEOFFS#
6.1 Training Cost vs Inference Savings#
| Model | Training Time | Compression | Speedup | Break-Even Threshold |
|---|---|---|---|---|
| DistilBERT | 40h (1x base) | 40% | 2x GPU | 500k inferences |
| TinyBERT | 80h (2x base) | 85% | 10x CPU | 100k inferences |
| EAGLE (LLaMA 7B) | 200h | ~5% params | 2.5x | 1M inferences |
| Spec. Decoding (draft pre-trained) | 0h | 0% | 2.5x GPU, 4x CPU | 0 inferences |
Key observation: Speculative decoding achieves better cost-benefit for high-volume services (no training) while distillation provides best compression for deployment-constrained scenarios.
6.2 Model Compression Spectrum#
Maximum Inference Efficiency ←→ Minimum Quality Loss
PTQ (no training)
↓ (slight quality impact)
Distillation (task-specific)
↓ (moderate quality impact)
Distillation (general-purpose)
↓ (larger quality impact)
Speculative Decoding
↓ (no quality impact, but adds draft model)
Self-Distillation / Early Exits
↓ (minimal engineering overhead)6.3 Batch Size Sensitivity#
Different compression techniques scale differently with batch size:
Distilled models: Linear improvement with batch size (amortize weight loading)
- Single sample: 50% teacher latency
- Batch 32: 45% teacher latency
- Batch 128: 40% teacher latency
Speculative decoding: Sublinear improvement (draft model bottleneck)
- Single sample: 30% teacher latency (3.3x speedup)
- Batch 32: 20% teacher latency (5x speedup) [draft scales well with parallelism]
- Batch 128: 18% teacher latency (5.5x speedup) [diminishing returns]
6.4 Quantization Compatibility#
Different distillation approaches interact differently with quantization:
| Method | INT8 Compat. | INT4 Compat. | Combined Speedup |
|---|---|---|---|
| Distilled Student | Excellent | Good | 8x (2x distill + 4x quant) |
| DistilBERT INT8 | 98.5% accuracy | 94.3% accuracy | Excellent |
| EAGLE + INT8 | Good | Fair | 2.8x |
| Spec. Decode (draft INT8, teacher INT4) | Excellent | Excellent | 3.2x |
Key insight: Distilled models quantize cleanly because distillation creates smoother loss landscapes. Speculative decoding tolerates quantization mismatch better (draft precision less critical).
7. EXPERT INSIGHT: What Practioners Should Know#
7.1 When to Apply Distillation#
DO distill if:
- Model is deployed >1M times (ROI clear)
- Model size constraint is primary concern (mobile, edge)
- You have compute budget for 1-2x additional training
- Downstream tasks are known and stable
- Quality degradation of 2-5% is acceptable
DON'T distill if:
- Research/experimental phase with changing models
- PTQ achieves sufficient speedup
- Quality must be within 1% of teacher
- Model changes frequently (< 1 month between versions)
7.2 Draft Model Selection for Speculative Decoding#
Practical heuristic:
To achieve 75% acceptance rate:
For LLaMA-scale models:
- Too large: LLaMA 7B as draft for LLaMA 70B (only 30-40% acceptance)
- Appropriate: LLaMA 1B for LLaMA 7B (75-80% acceptance)
- Too small: LLaMA 160M for LLaMA 7B (15-25% acceptance)
Rule of thumb: Draft model should be 10-20% size of teacher for good acceptance rates.
7.3 Temperature Tuning for Distillation#
Empirical guidelines:
| Task Domain | Recommended τ | Reasoning |
|---|---|---|
| Vision (ImageNet) | 4-6 | Sharp, decisive probability mass |
| NLP Classification | 8-15 | Moderate smoothing for linguistic ambiguity |
| LLM Generation | 12-25 | Heavy smoothing for autoregressive diversity |
| Medical/Safety-Critical | 1.5-3 | Preserve sharp decision boundaries |
Validation strategy:
- Train with τ ∈ [1, 5, 10, 20, 40]
- Evaluate student-only performance (no teacher at test time)
- Select τ that maximizes task accuracy vs teacher
7.4 Hardware-Software Co-Design#
CPU deployment (server-side):
- Speculative decoding > all other methods
- Draft model must stay cache-resident
- Benefit from single-thread latency → multi-thread throughput trade-off
GPU deployment (data center):
- Speculative decoding for latency-critical queries
- Distillation for batch inference
- Quantization orthogonal to both
Mobile/Edge deployment:
- Distillation primary strategy
- Speculative decoding not viable (two models exceed memory)
- Self-distillation acceptable if storage permits
8. BENCHMARKING AND EMPIRICAL RESULTS#
8.1 Comprehensive Speedup Comparison#
Experimental setup:
- Model: LLaMA 7B (4096 hidden, 32 heads, 32 layers)
- Hardware: A100 GPU, 80GB VRAM
- Sequence length: 2048 input tokens
- Batch size: 1 (latency focus)
- Quantization: FP16 baseline
Results:
| Method | Speedup | Accuracy Loss | Memory | Deployment Cost |
|---|---|---|---|---|
| Baseline (FP16) | 1.0x | — | 13GB | — |
| PTQ INT8 | 1.8x | 0.5% | 7GB | 0 training |
| PTQ INT4 | 2.8x | 1.2% | 4GB | 0 training |
| Distillation (40%) | 2.1x | 1.8% | 8GB | 40h training |
| Distillation + INT8 | 3.5x | 2.2% | 4GB | 40h training |
| EAGLE (2.5M params) | 2.3x | 0.2% | 13.5GB | 200h training |
| Spec. Decoding (draft 1B) | 3.2x | 0% | 14GB | 0h (draft pre-trained) |
| Spec. Decoding + INT4 | 4.8x | 0% (draft only) | 7GB | 0h |
Key observations:
- PTQ is baseline: Easiest deployment, no training, 1.8-2.8x speedup
- Distillation additive: Combines with quantization (distill then quantize)
- EAGLE preserves quality: Minimal accuracy loss, good for drop-in replacement
- Speculative best for high-volume: No quality loss, marginal memory overhead
- Combinations powerful: Spec. Decoding INT4 achieves 4.8x total speedup
8.2 Acceptance Rate Analysis (Speculative Decoding)#
Measurement: Track acceptance rate for different draft models
| Draft Model | Avg. Accept. | Token-1 | Token-5 | Token-10 |
|---|---|---|---|---|
| LLaMA 1B | 78% | 95% | 72% | 35% |
| LLaMA 3B | 85% | 98% | 82% | 55% |
| LLaMA 7B (half layers) | 82% | 96% | 78% | 48% |
| EAGLE (learned) | 81% | 94% | 76% | 42% |
| Medusa (4 heads) | 88% | 97% | 85% | 62% |
Insight: Acceptance rate declines with position (later tokens have more entropy). Tree-based methods (Medusa) maintain higher rates.
8.3 Latency Breakdown#
Speculative decoding with LLaMA 7B on A100:
Loading…9. OPEN PROBLEMS AND FUTURE DIRECTIONS#
9.1 Cross-Domain Draft Model Transfer#
Challenge: Draft models trained on general text may not match teacher on specialized domains (medical, code, math).
Current approaches:
- Domain-specific finetuning of draft model
- Cost: additional training per domain
Open question: Can we meta-learn draft models that quickly adapt to teacher's distribution without full retraining?
Potential solution: Conditional draft model that takes teacher logits as input to context-shift its predictions.
9.2 Speculative Decoding with Continuous Batching#
Challenge: In vLLM-like systems with continuous batching, different sequences have different draft acceptance rates. How to optimally schedule draft generation across sequences?
Current problem:
- Some sequences accept 90% of drafts
- Others accept 40% of drafts
- Difficult to keep compute utilization high
Open question: Adaptive draft budgeting per sequence based on running acceptance statistics?
9.3 Distillation for Fine-Tuning#
Challenge: Standard distillation is offline. Can we distill during fine-tuning (e.g., RLHF)?
Issue: Teacher distribution changes during fine-tuning; student chases moving target.
Potential approach: Constrain student to stay within KL divergence ball of base model during fine-tuning.
9.4 Hardware-Aware Speculative Decoding#
Challenge: Different hardware (TPU, GPU, CPU, mobile) has different optimal draft budgets.
Current approach: Manual tuning per hardware.
Future direction: Auto-tuner that profiles hardware performance of draft+teacher combination and selects optimal γ dynamically.
9.5 Theoretical Limits of Distillation#
Open question: What is the information-theoretic limit of student performance vs teacher?
Current empirical bound: Student can achieve ~95% of teacher accuracy with aggressive compression.
Theory gap: No formal characterization of achievable student performance as function of compression ratio.
10. PHD-LEVEL RESEARCH DIRECTIONS#
10.1 Information-Theoretic Analysis of Knowledge Transfer#
Problem statement: Given teacher model with parameters and student model with parameters where , what is the fundamental limit on student performance?
Conjecture: Information-theoretic lower bound on distillation loss.
Approach:
- Define "knowledge" as mutual information - what the teacher knows about output given input
- Define "capacity" of student as - relative expressiveness
- Derive fundamental tradeoff:
Research question: Can we characterize this function ?
10.2 Adaptive Temperature Scheduling in Distillation#
Current limitation: Temperature is typically fixed throughout training.
Observation: Early in training, high temperature helps with learning; late in training, low temperature improves task performance.
Research direction: Learn temperature scheduling as function of:
- Student gradient magnitude
- Training epoch
- Loss landscape curvature
- Student-teacher KL divergence
Proposed formulation:
where parameters learned via meta-optimization.
10.3 Online Distillation with Changing Objectives#
Problem: In RLHF, teacher model changes (policy updates). How to maintain effective distillation?
Current approach: Freeze teacher weights, accumulate experience, then distill offline.
Future direction: Online distillation where student learns from evolving teacher distribution.
Key challenge: Student divergence increases as teacher changes; must constrain student-teacher divergence while updating teacher.
Mathematical formulation:
where changes with training step .
Research questions:
- What constraints on path maintain learnability?
- How does learning rate scale with and rate of teacher change?
10.4 Compositional Knowledge Distillation#
Observation: Complex tasks (e.g., question answering) involve multiple sub-skills.
Hypothesis: Can we distill knowledge of individual skills, then compose them?
Proposed framework:
- Decompose task into sub-skills
- Train teacher with masking to isolate each skill
- Train student heads for each skill
- Compose heads for full task
Research question: Does compositional distillation improve sample efficiency and transfer learning?
10.5 Speculative Decoding with Learned Branching Factors#
Current limitation: Branching factor (number of draft tokens) is fixed.
Observation: Early in sequence, tokens are more predictable (lower entropy); later tokens more uncertain.
Proposed approach: Learn branching policy that predicts appropriate draft budget at each position:
where is entropy of position in context.
Research direction:
- Train RL agent to select draft budgets
- Reward: speedup per token
- Constraints: latency SLO
Application: Variable-latency inference with speedup guarantees.
10.6 Distillation Meets Mechanistic Interpretability#
Emerging area: Understanding what "knowledge" transfers during distillation in terms of model internals.
Research questions:
- Which teacher features are most important for student learning?
- Do students learn similar circuits as teachers?
- Can we distill only task-relevant circuits?
Potential application: Interpret what student actually learns from teacher using mechanistic interpretability tools.
CONCLUSION#
Knowledge distillation has evolved from a post-hoc compression technique into a first-class design pattern for efficient inference. The spectrum from offline distillation (DistilBERT, TinyBERT) to deployment-time speculation (speculative decoding, EAGLE, Medusa) provides principled approaches to speed-accuracy tradeoffs.
For ML systems engineers:
- Understand your hardware: CPU benefits most from distillation and speculative decoding; GPU prefers batching
- Calculate ROI carefully: Distillation worth it for >1M inferences; speculative decoding for >100k
- Combine techniques: Distilled models quantize better; speculative decoding tolerates INT4; synergies exist
- Monitor empirically: Acceptance rates, temperature sensitivity, downstream task performance
- Design for your deployment: Edge=distillation, Server CPU=speculative decoding, GPU=quantization-focused
The next frontier lies in adaptive, hardware-aware, and online distillation that extends beyond static student models toward dynamic inference systems that learn to compress themselves.