Cloud-sized models don't fit on edge devices. Optimization techniques shrink models while preserving accuracy.

Quantization

Reduce numerical precision to shrink models and speed inference:

• Post-Training Quantization (PTQ) — convert trained model from FP32 to INT8 with minimal accuracy loss
• Quantization-Aware Training (QAT) — simulate quantization during training for better accuracy
• Mixed Precision — use different precision for different layers based on sensitivity
• Binary/Ternary Networks — extreme quantization to 1-2 bits for maximum compression

Quantization typically provides 2-4x speedup with < 1% accuracy loss.

Pruning

Remove unnecessary connections from neural networks:

1. Weight Pruning — remove individual weights below a threshold
2. Structured Pruning — remove entire neurons, channels, or layers
3. Lottery Ticket Hypothesis — find small subnetworks that match full model performance
4. Dynamic Pruning — adjust which parts of the model run based on input complexity

Knowledge Distillation

Train a small "student" model to mimic a large "teacher" model:

• Response Distillation — student learns from teacher's soft output probabilities
• Feature Distillation — student learns to match teacher's internal representations
• Online Distillation — student and teacher train simultaneously
• Self-Distillation — model distills knowledge from its deeper layers to shallower ones

Neural Architecture Search (NAS)

Automatically design efficient architectures:

• EfficientNet — balanced scaling of depth, width, and resolution
• MobileNet — depthwise separable convolutions for mobile deployment
• Once-for-All — train one network that can be deployed at different sizes
• Hardware-Aware NAS — optimize architecture for specific hardware constraints

Optimization Frameworks

• TensorFlow Lite — Google's framework for mobile and embedded deployment
• ONNX Runtime — cross-platform inference with hardware acceleration
• TensorRT — NVIDIA's inference optimizer for GPU deployment
• Core ML — Apple's framework for iOS/macOS deployment
• Apache TVM — compiler for diverse hardware targets

Best Practice: Apply quantization before pruning. Pruning after quantization can further degrade accuracy in an unpredictable way because the interaction between weight sparsity and low-precision arithmetic is not well-characterized. Establish your quantization baseline first, then prune on top of it.

Common Pitfall: Post-training quantization to INT4 often produces significant accuracy degradation for NLP models, even though INT8 is usually acceptable. Always benchmark accuracy on your specific task at each quantization level rather than assuming published benchmarks transfer to your use case.

# TensorFlow Lite quantization example
import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model('model/')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# INT8 post-training quantization
converter.target_spec.supported_types = [tf.int8]
tflite_model = converter.convert()

with open('model_quantized.tflite', 'wb') as f:
    f.write(tflite_model)
# Result: typically 4x smaller, 2-3x faster on edge hardware