ML Systems Engineering · Part 6 · Module 22
TinyML — Inference on Microcontrollers
TFLite Micro, memory-constrained inference, quantization for MCUs
Abstract#
Inference on resource-constrained microcontroller units (MCUs) represents the extreme frontier of edge AI deployment. With 256 KB to 2 MB of SRAM, no operating system, no virtual memory, and no dynamic heap, MCU-based inference demands radically different system architecture than mobile or server inference. This module examines TensorFlow Lite Micro (TFLite Micro), the reference framework for MCU inference, alongside emerging approaches like MCUNet and MCUNetV2 that co-design neural architecture with memory-aware compilation. We explore interpreter-based execution, memory arena management, operator fusion, INT8/INT4 quantization strategies, and novel techniques like patch-based inference for overcoming SRAM constraints. Special attention to CMSIS-NN (ARM's DSP library for MCU), on-device training capabilities, and emerging RISC-V alternatives rounds out the landscape. This module synthesizes low-level hardware knowledge (Module 21) into practical deployment pipelines.
1. Introduction: The MCU Inference Challenge#
Microcontroller units occupy a unique niche in AI deployment:
Defining constraints:
- Memory: 256 KB – 2 MB SRAM (vs. 8-12 GB phone)
- Compute: 100-500 MMACS INT8 (vs. 5-10 TOPS phone)
- Execution model: Bare metal, no OS, no heap allocator
- Typical models: Speech recognition (1-5 MB), anomaly detection (100 KB)
- Latency target: 10-1000 ms (not real-time like mobile)
Traditional approach: Store model weights in external flash, stream activations through SRAM.
New challenge (2020+): Deploy 1-10 MB models (RNNs, small CNNs) on 256-512 KB MCU.
1.1 Why MCU Inference Matters#
- Scale: 20+ billion MCUs deployed globally (automotive, industrial, IoT)
- Cost: 5 per unit (phones are 1500)
- Power: <1 mW in sleep, 10-50 mW active (months/years on battery)
- Latency tolerance: 100 ms acceptable (voice wakeword detection, motion sensing)
1.2 Execution Model Fundamentals#
MCU program layout:
Loading…Key insight: Weights static in flash, activations dynamic in SRAM (reused across layers).
2. TensorFlow Lite Micro Architecture#
2.1 Design Philosophy#
TFLite Micro (released 2019, led by Google Brain) is purpose-built for MCUs:
Core principles:
- Statically allocated memory: No dynamic malloc (real-time guarantee)
- Operator registry: Selective inclusion of ops (only conv, fc, relu needed)
- Flat buffer format: Model metadata embedded in linear buffer (no parsing overhead)
- Minimal dependencies: C++11, no STL, no exceptions
Code footprint:
- Core interpreter: ~50 KB (stripped)
- CMSIS-NN ops: ~20 KB
- Total executable: <100 KB on typical MCU
2.2 Flat Buffer Model Format#
TFLite uses flatbuffer encoding (Google standard for serialization):
Structure:
Loading…Advantages for MCU:
- Zero-copy loading: Model pointer into flash, no parsing required
- Alignment safety: Flatbuffer aligns weights to cache line (32 bytes)
- Metadata compact: Shapes/types in-line, minimal overhead
2.3 Interpreter Execution Model#
TFLite Micro uses a simple tree-walk interpreter (not graph optimization, not XLA):
// Pseudo-code: TFLite Micro interpreter loop
struct Interpreter {
Tensor* tensors; // All tensor metadata
Operator* operators; // All ops in order
uint8_t* arena; // Activation buffer
const Model* model; // Flatbuffer model (in flash)
};
void Interpreter::Invoke() {
for (Operator* op : operators) {
// 1. Validate inputs exist (check tensor buffer_idx)
for (int input_idx : op->inputs) {
Tensor* input = &tensors[input_idx];
ASSERT(input->buffer != NULL, "uninitialized input");
}
// 2. Allocate output buffer (in arena)
for (int output_idx : op->outputs) {
Tensor* output = &tensors[output_idx];
output->buffer = arena.allocate(output->bytes);
}
// 3. Execute kernel
BuiltinOperator op_type = op->builtin_code;
switch (op_type) {
case kConv2D:
conv2d_kernel(op, tensors);
break;
case kFullyConnected:
fc_kernel(op, tensors);
break;
// ... more ops
}
// 4. Mark output tensor as ready
tensors[output_idx].buffer_valid = true;
}
}Characteristics:
- Single-threaded: Interpreter runs sequentially
- No optimization: Operators executed in model-defined order (no fusion)
- Arena allocation: Outputs allocated/deallocated in LIFO stack order
2.4 Memory Arena Management#
TFLite Micro uses a linear arena allocator (essentially a stack):
struct Arena {
uint8_t* base; // Start of SRAM buffer
size_t size; // Total size
size_t position; // Current allocation pointer
};
uint8_t* Arena::allocate(size_t num_bytes) {
if (position + num_bytes > size) {
return NULL; // Out of memory
}
uint8_t* ptr = base + position;
position += num_bytes;
return ptr;
}
void Arena::reset() {
position = 0; // Reset for next inference
}Tensor allocation sequence:
Loading…2.5 Operator Registration#
TFLite Micro requires explicit registration of each operator kernel:
namespace {
const TFLMRegistration* const micro_builtin_data[] = {
&tflite::micro::Register_ABS(),
&tflite::micro::Register_ADD(),
&tflite::micro::Register_CONV_2D(),
&tflite::micro::Register_DEPTHWISE_CONV_2D(),
&tflite::micro::Register_FULLY_CONNECTED(),
&tflite::micro::Register_RELU(),
// Only registered ops can execute
};
}Benefit: Linker removes unused ops → binary ~20% smaller.
2.6 MCU Porting Guide#
Porting TFLite Micro to new MCU platform:
// 1. Implement platform-specific I/O
namespace tflite {
namespace micro {
void LogString(const char* s) {
// Write to UART, log buffer, etc.
}
}
}
// 2. Implement memory allocation (if no stdlib)
void* malloc(size_t size) {
static uint8_t heap[256 * 1024];
static size_t used = 0;
if (used + size > sizeof(heap)) return NULL;
void* ptr = heap + used;
used += size;
return ptr;
}
// 3. Link against CMSIS-NN (if ARM MCU)
// ...
// 4. Main inference loop
int main() {
// Load model from flash
const unsigned char* model_data = model_tflite;
tflite::Model* model = tflite::GetModel(model_data);
// Create interpreter
static uint8_t arena[256 * 1024];
tflite::MicroInterpreter interpreter(model, resolver, arena, 256*1024);
// Allocate tensors
interpreter.AllocateTensors();
// Fill input
float* input = interpreter.typed_input_tensor<float>(0);
memcpy(input, sensor_data, input_size);
// Run inference
interpreter.Invoke();
// Read output
float* output = interpreter.typed_output_tensor<float>(0);
for (int i = 0; i < output_size; i++) {
printf("Output[%d] = %f\n", i, output[i]);
}
return 0;
}3. MCUNet: Neural Architecture Co-Design for Tiny Devices#
3.1 The Co-Design Problem (Lin et al., NeurIPS 2020)#
Standard models (MobileNetV2, EfficientNet) designed for mobile (8 GB DRAM, 5W power) are oversized for MCU (256 KB SRAM, 50 mW power).
MCUNet's insight: Joint optimization of model architecture and runtime execution for extreme memory constraint.
3.2 TinyNAS: Automated Architecture Search#
Search space targets MCU-friendly architectures:
Loading…Search algorithm: DNAS (Differentiable NAS) + hardware-aware loss:
Loss = accuracy_loss + λ₁ * peak_memory + λ₂ * latency3.3 TinyEngine: Memory-Aware Execution#
Core innovation: In-place operations and layer fusion to minimize peak memory.
Traditional execution (layer-by-layer):
Input (50 KB) → Conv1 (100 KB output) → Relu1 (100 KB) → Conv2 (80 KB output) → ...
Peak memory = 50 + 100 + 80 = 230 KBTinyEngine execution (fused, in-place):
Loading…Graph coloring for buffer reuse:
// MCUNet's buffer allocation strategy
struct TensorAllocation {
Tensor* tensor;
int first_use; // Operator index
int last_use; // Operator index
int memory_offset; // Position in arena
int size;
};
// Build lifetime intervals
for (Operator op : model) {
for (Tensor output : op.outputs) {
allocation[output].last_use = op.index;
}
}
// Greedy allocation (offline)
std::vector<TensorAllocation> arena_plan = GreedyColoringAllocate(allocations);
// At runtime, follow plan (no dynamic allocation)3.4 Example: MCUNet Results#
Target: ImageNet classification on STM32H745 (Cortex-M7, 1.3 MB SRAM)
| Model | Peak SRAM | Model Size | Latency (M7) | Accuracy (ImageNet) |
|---|---|---|---|---|
| MobileNetV2 (0.5×) | 1.2 MB | 2.2 MB | 500 ms | 70.6% |
| EfficientNet-Lite0 | 2.5 MB | 4.8 MB | 800 ms | 75.1% |
| MCUNet-5 | 256 KB | 350 KB | 800 ms | 60.3% |
| MCUNet-160 | 1.0 MB | 1.5 MB | 400 ms | 74.4% |
Key result: MCUNet-160 achieves 74.4% accuracy (near EfficientNet-Lite0 at 75.1%) while fitting in 1 MB model + 1 MB SRAM (vs. EfficientNet 4.8 MB model alone).
4. MCUNetV2: Patch-Based Inference#
4.1 Motivation: Overcoming SRAM Bottleneck#
MCUNet-160's 1 MB SRAM requirement is still 4× typical MCU capacity (256 KB). MCUNetV2 (Lin et al., NeurIPS 2021) introduces patch-based inference to eliminate this barrier.
Key idea: Process images patch by patch, not all-at-once.
4.2 Patch-Based Processing#
Traditional convolution (full image):
Input: 224×224×3 → Conv5×5 → Output: 220×220×64
Memory: 224×224×3 (150 KB) + 220×220×64 (3 MB) = huge!Patch-based (16×16 patches):
Loading…4.3 Architecture Constraints for Patch Processing#
Design requirement: Model must support arbitrary input resolutions (no global pooling in middle of network).
Typical MCUNetV2 architecture:
Input: (H, W, 3) [variable H, W for different patches]
↓
Conv layers: stride=1, padding maintains spatial dims
↓
No global pooling (instead: adaptive spatial pooling later)
↓
Dense layers: operate on flattened spatial feature mapsExample architecture string:
block_1: [depthwise_conv 3×3 stride 1, channels 32]
block_2: [conv 1×1 stride 1, channels 16, then expand to 64]
block_3: [depthwise_conv 5×5 stride 1, channels 64]
...
final_conv: [conv 1×1 channels C] ← per-patch output
global_pool: [adaptive 2D pool] ← patch outputs pooled
dense: [FC to num_classes]4.4 Patch Tiling Implementation#
// MCUNetV2 inference loop
int patch_size = 16;
int overlap = 8; // Receptive field padding
for (int y = 0; y < 224; y += patch_size - overlap) {
for (int x = 0; x < 224; x += patch_size - overlap) {
// 1. Extract patch from input image
uint8_t patch[16*16*3];
ExtractPatch(input_image, 224, 224, x, y, patch_size, patch);
// 2. Set interpreter input
uint8_t* model_input = interpreter.input(0)->data.uint8;
memcpy(model_input, patch, 16*16*3);
// 3. Run inference
interpreter.Invoke();
// 4. Collect output patch
float* model_output = interpreter.output(0)->data.f;
int out_size = patch_size - overlap; // Receptive field crop
AccumulateOutputPatch(output_feature_map, model_output, out_size);
}
}
// 5. Final global pooling
FinalPrediction = GlobalAveragePool(output_feature_map);4.5 MCUNetV2 Results#
Target: ImageNet on STM32L476 (Cortex-M4, 96 KB SRAM)
| Model | Peak SRAM | Model Size | Latency | Accuracy |
|---|---|---|---|---|
| MCUNetV2-10 | 96 KB | 150 KB | 2.5 s | 50.3% |
| MCUNetV2-20 | 96 KB | 350 KB | 8 s | 63.1% |
| MCUNetV2-40 | 192 KB | 700 KB | 30 s | 69.2% |
Breakthrough: Runs ImageNet classification on 96 KB MCU (256× smaller SRAM than MCUNet-160)!
5. Quantization for MCU: INT8 and INT4#
5.1 INT8 Quantization Strategies#
MCU quantization goals (different from mobile):
- Reduce model size (weights in flash)
- Enable fixed-point arithmetic (no floating-point hardware on many MCUs)
- Improve energy per operation
Symmetric INT8 quantization (common for MCU):
x_quantized = clamp(round(x_float / scale), -128, 127)
x_float ≈ scale * x_quantized
scale = max(|x|) / 128Example: Weights for fully-connected layer:
import numpy as np
weights_fp32 = np.random.randn(1024, 2048) # FC layer
scale = np.max(np.abs(weights_fp32)) / 128.0
weights_int8 = np.round(weights_fp32 / scale).astype(np.int8)
# Verify: max error
max_error = np.max(np.abs(weights_fp32 - weights_int8 * scale))
print(f"Quantization scale: {scale:.6f}")
print(f"Max error: {max_error:.6f}")5.2 INT4 Quantization: Further Compression#
For extreme memory constraints, INT4 packing reduces model size by 2×:
Loading…Cost: Unpacking adds ~10-20% computational overhead (bit manipulation).
Trade-off: 2× model size reduction vs. ~15% latency increase.
5.3 CMSIS-NN: ARM DSP Library#
CMSIS-NN (Cortex Microcontroller Software Interface Standard) provides hand-optimized INT8 kernels:
Architecture:
- Intrinsics for ARM Cortex-M7/M55: SMLAD, SMLAWB (4-lane INT8 MAC in single instruction)
- Fully-connected:
arm_fully_connected_q7()(INT8 input/output, INT32 accumulator) - Convolution:
arm_convolve_HWC_q7_fast()(uses CMSIS-NN im2col + GEMM)
CMSIS-NN INT8 FC kernel (conceptual):
void arm_fully_connected_q7(
const q7_t *pV, // Input vector (Q7: INT8)
const q7_t *pM, // Weight matrix (Q7)
uint16_t dim_vec, // Input vector length
uint16_t num_of_rows, // Number of output neurons
uint16_t bias_shift, // Bias scaling
uint16_t out_shift, // Output scaling
const q7_t *bias_c, // Bias vector
q7_t *pOut, // Output vector
q15_t *vec_buffer // Temporary buffer
) {
// 1. Convert Q7 to Q15 (sign-extend)
for (int i = 0; i < dim_vec; i++) {
vec_buffer[i] = (q15_t)pV[i] << 7; // Sign-extend INT8 → INT16
}
// 2. Matrix-vector multiply (GEMV)
for (int i = 0; i < num_of_rows; i++) {
int sum = 0;
const q7_t *pW = &pM[i * dim_vec];
// Unroll 2: SMLAD instruction (4-lane INT16 MAC per cycle)
for (int j = 0; j < dim_vec; j += 2) {
q31_t v_input = *(q31_t*)&vec_buffer[j]; // 2 INT16 values
q31_t v_weight = arm_nn_read_q7x4(&pW[j]); // 4 INT8 values (sign-ext)
sum = __smlad(v_input, v_weight, sum); // 4-lane multiply-add
}
// 3. Bias and scaling
sum += (int)bias_c[i] << bias_shift;
sum = (sum + (1 << (out_shift - 1))) >> out_shift;
// 4. Clamp to INT8 range
pOut[i] = (q7_t)__SSAT(sum, 8);
}
}Performance (STM32H745 Cortex-M7):
- Scalar INT8 MAC: ~1 cycle per MAC (theoretical 200 MMACS at 200 MHz)
- CMSIS-NN INT8 GEMV: ~0.6 cycles per MAC (due to SMLAD 4-lane operations)
- Speedup: ~1.7× vs. scalar
6. Memory Planning and Graph Coloring#
6.1 Tensor Lifetime Analysis#
Goal: Compute minimum SRAM needed by analyzing when each tensor is live.
struct TensorInfo {
int tensor_idx;
int first_op; // First operator that reads this tensor
int last_op; // Last operator that produces this tensor
size_t size_bytes;
bool is_weight; // True if static (in flash)
bool is_input; // True if network input
};
// Build lifetime intervals
std::vector<TensorInfo> GetTensorLifetimes(const Model& model) {
std::vector<TensorInfo> lifetimes;
for (size_t op_idx = 0; op_idx < model.operators.size(); op_idx++) {
const Operator& op = model.operators[op_idx];
// Mark input tensors' first use
for (int tensor_idx : op.inputs) {
if (lifetimes[tensor_idx].first_op == -1) {
lifetimes[tensor_idx].first_op = op_idx;
}
}
// Mark output tensors' last use
for (int tensor_idx : op.outputs) {
lifetimes[tensor_idx].last_op = op_idx;
}
}
return lifetimes;
}6.2 Graph Coloring Algorithm#
Interval graph coloring assigns SRAM addresses to tensors such that non-overlapping lifetimes reuse same memory:
def graph_coloring_allocate(tensor_lifetimes, arena_size):
"""
Assign SRAM offsets to tensors to minimize peak memory.
"""
# Sort tensors by first use (earliest first)
tensor_lifetimes.sort(key=lambda t: t.first_op)
allocations = {}
active_intervals = []
for tensor in tensor_lifetimes:
if tensor.is_weight:
continue # Skip weights (in flash)
# Remove expired intervals (tensors no longer needed)
active_intervals = [t for t in active_intervals
if t.last_op >= tensor.first_op]
# Find earliest available offset that doesn't conflict
offset = 0
for active_tensor in sorted(active_intervals,
key=lambda t: allocations[t.tensor_idx]):
active_offset = allocations[active_tensor.tensor_idx]
if offset == active_offset:
# Conflict, try next offset
offset = active_offset + active_tensor.size_bytes
offset = (offset + 31) & ~31 # Align to 32-byte cache line
else:
break
allocations[tensor.tensor_idx] = offset
active_intervals.append(tensor)
# Compute peak memory (max allocation extent)
peak_memory = max(allocations[t] + t.size_bytes
for t in tensor_lifetimes
if not t.is_weight)
return allocations, peak_memory6.3 Example: Tensor Lifetime Analysis#
Simple 3-layer model:
Loading…7. On-Device Training on MCU#
7.1 Motivation and Feasibility#
On-device training (not just inference) on MCU enables:
- Personalization: Fine-tune model on user data without cloud
- Federated learning: Train across devices, aggregate gradients
- Low power: Local training uses less bandwidth than cloud round-trip
Feasibility: Extremely limited (shallow networks, INT8 precision, single pass).
7.2 Quantization-Aware Fine-Tuning#
Approach: Pre-trained model (via normal QAT on desktop), then fine-tune on MCU.
// MCU-based fine-tuning (pseudo-code)
struct MicroTrainer {
Model* model;
Optimizer optimizer; // SGD (simplest), no Adam (requires 2 buffers per param)
};
void FinetuneOneStep(MicroTrainer& trainer, float* input_data) {
// 1. Forward pass (inference)
float* logits = ForwardPass(trainer.model, input_data);
// 2. Compute loss (cross-entropy simplified)
float loss = 0.0;
for (int i = 0; i < num_classes; i++) {
float p_i = softmax(logits[i]);
loss -= label[i] * log(p_i); // Cross-entropy
}
// 3. Backward pass (backprop through layers)
float* grad = BackwardPass(trainer.model, logits, label);
// 4. Update weights (SGD: w ← w - lr * grad)
for (int i = 0; i < num_params; i++) {
int8_t* w = &trainer.model->weights[i];
float w_float = dequantize(w);
w_float -= learning_rate * grad[i];
w[i] = quantize(w_float);
}
}7.3 Limitations and Current State#
Practical constraints:
- Forward-backward: ~2× memory of inference (gradients must be stored)
- Precision loss: INT8 gradients lose significant information
- No optimizer state: SGD only (Adam needs 2 buffers per parameter)
- Convergence slow: Very small learning rates required for INT8
Current practice: Companies like Qualcomm, ARM prototype on-device training but rely on off-device pre-training + on-device fine-tuning (not training from scratch).
8. RISC-V for Edge AI#
8.1 RISC-V Opportunity in MCU Space#
RISC-V advantages:
- Open ISA (no ARM licensing cost)
- Modular: base I + extensions (M, F, D, V for ML)
- Growing ecosystem (SiFive, Alibaba, Western Digital)
Market position (2024): ~5-10% MCU share, growing.
8.2 RVV (RISC-V Vector Extension)#
RVV design:
- Scalable vector length (like ARM SVE)
- Predicated operations
- Typical mobile RISC-V: 128-256 bit (same as ARM NEON)
RVV vector multiply (pseudo-assembly):
vle8.v v0, (x0) # Load 16 INT8 values (128-bit RVV)
vle8.v v1, (x1) # Load 16 INT8 weights
vmul.vv v2, v0, v1 # Element-wise multiply (16× int8)
vadd.vv v3, v3, v2 # Accumulate
add x0, x0, x2 # Next input
add x1, x1, x2 # Next weight
blt x3, x4, loop # Loop if more dataPerformance parity: RVV achieves similar throughput to ARM NEON for same vector length.
8.3 RISC-V ML Ecosystem#
TFLite Micro support: Partial (interpreter works, but CMSIS-NN equivalent missing).
Emerging solutions:
- Alibaba TVM + RISC-V: LLVM backend generates RVV instructions
- SiFive specific optimizations: Hand-tuned kernels for SiFive HiFive1 Rev B
9. Example: Complete MCU Inference Pipeline#
9.1 From Keras Model to Deployed MCU Binary#
Step 1: Train and quantize on desktop (TensorFlow):
# Train model
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(train_images, train_labels, epochs=10)
# Quantize to INT8
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
def representative_dataset():
for image in test_images[:100]:
yield [image[np.newaxis, ...].astype(np.float32)]
converter.representative_dataset = representative_dataset
quantized_model = converter.convert()
with open('model.tflite', 'wb') as f:
f.write(quantized_model)Step 2: Convert to C array (embedding model in firmware):
xxd -i model.tflite > model_data.hStep 3: MCU firmware (STM32H745):
#include "tensorflow/lite/micro/all_ops_resolver.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "model_data.h"
// Global tensors and arena
static uint8_t tensor_arena[256 * 1024];
static tflite::MicroErrorReporter micro_error_reporter;
static tflite::MicroInterpreter* interpreter = nullptr;
void setup_inference() {
// Get model from flash
const tflite::Model* model = tflite::GetModel(model_tflite);
// Create resolver with only needed ops
static tflite::MicroMutableOpResolver<5> resolver;
resolver.AddBuiltin(tflite::BuiltinOperator_CONV_2D,
tflite::ops::micro::Register_CONV_2D());
resolver.AddBuiltin(tflite::BuiltinOperator_MAX_POOL_2D,
tflite::ops::micro::Register_MAX_POOL_2D());
resolver.AddBuiltin(tflite::BuiltinOperator_FULLY_CONNECTED,
tflite::ops::micro::Register_FULLY_CONNECTED());
resolver.AddBuiltin(tflite::BuiltinOperator_SOFTMAX,
tflite::ops::micro::Register_SOFTMAX());
static tflite::MicroInterpreter static_interpreter(
model, resolver, tensor_arena, 256 * 1024, µ_error_reporter);
interpreter = &static_interpreter;
interpreter->AllocateTensors();
}
void run_inference(uint8_t* image_data) {
// Fill input
uint8_t* input = interpreter->typed_input_tensor<uint8_t>(0);
memcpy(input, image_data, 28 * 28);
// Invoke
interpreter->Invoke();
// Read output
float* output = interpreter->typed_output_tensor<float>(0);
int predicted_class = argmax(output, 10);
printf("Predicted class: %d\n", predicted_class);
}
int main() {
setup_inference();
// Capture image from camera
uint8_t image_buffer[28 * 28];
capture_image_from_sensor(image_buffer);
// Run inference
run_inference(image_buffer);
return 0;
}10. Conclusion and Future Directions#
10.1 State-of-the-Art (2024)#
Achievable on STM32H745 (1.3 MB SRAM):
- Small CNNs: MNIST-scale (60% ImageNet-equivalent)
- Speech: Keyword spotting (wake word detection)
- Anomaly detection: Sensor-based (accelerometer, temperature)
- Latency: 100-500 ms typical
Not yet feasible:
- Large language models (even 1B parameter models)
- Real-time video processing
- Multi-model ensembles
10.2 Emerging Techniques#
- Sparse computation: Exploiting model sparsity for 2-3× speedup
- Heterogeneous execution: Offload convolution to DSP, other ops to CPU
- Neural architecture search: AutoML specifically targeting MCU constraints
- Post-training compression: Further quantization + pruning after QAT
10.3 Key Insights#
- Memory footprint dominates, not compute (opposite of datacenter)
- Arena-based allocation enables predictable, bounded memory usage
- Quantization to INT8 is mandatory for any non-trivial model
- Architecture co-design (MCUNet) essential for extreme constraints
- RISC-V opportunity: Open ISA could disrupt ARM monopoly in MCU ML
10.4 Design Principles for MCU Inference#
- Profile first: Measure memory, latency on target before optimizing
- Quantize early: INT8 from start, INT4 if desperate
- Reuse buffers: Offline compute tensor lifetimes, allocate greedily
- Fuse operations: Conv+ReLU, ReLU+Add eliminate intermediate writes
- Avoid allocators: Static memory planning only
- Benchmark relentlessly: Cycle counts ≠ wall-clock time on MCU (cache effects)
Further Reading#
- Lin et al.: "MCUNet: Tiny Deep Learning on IoT Devices" (NeurIPS 2020)
- Lin et al.: "MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning" (NeurIPS 2021)
- Google TensorFlow Lite Micro: https://www.tensorflow.org/lite/microcontrollers
- ARM CMSIS-NN: https://github.com/ARM-software/CMSIS-NN
- Qualcomm Hexagon DSP ML Benchmarks (2023)