Efficient & Edge Vision
What you’ll learn. How to make a vision model fast and small enough for its deployment target: where the time actually goes (compute-bound versus memory-bound), post-training quantization and quantization-aware training with the underlying arithmetic, knowledge distillation, pruning and its limits, efficient architectures, export and runtimes (ONNX, TensorRT, OpenVINO, Core ML, TFLite/LiteRT, ExecuTorch), and a latency-measurement methodology that avoids the common mistakes. Architectures referenced are introduced in CNN Architectures and Vision Transformers.
What “efficient” means
Section titled “What “efficient” means”There are several budgets, and they conflict:
| Budget | Metric | Typical constraint |
|---|---|---|
| Latency | ms per image (p50/p95/p99) | real-time video (33 ms at 30 fps), interactive UI |
| Throughput | images/s at batch size | offline processing, servers |
| Memory | weights + activations + KV/feature caches | phone RAM, GPU VRAM |
| Energy | joules per inference | battery, thermal limits |
| Model size | MB on disk | app-store limits, OTA updates |
| Accuracy | task metric | non-negotiable floor |
Proxy metrics such as parameter count and FLOPs are not latency. Latency depends on memory traffic, operator support, kernel quality, batch size and hardware. Use the roofline intuition: an operator’s attainable speed is , where arithmetic intensity is FLOPs per byte moved. Large dense convolutions and matmuls are compute-bound; depthwise convolutions, normalization, elementwise ops and attention softmax are memory-bound. That is why a MobileNet with 10x fewer FLOPs than ResNet-50 is often not 10x faster on a GPU.
Measuring latency correctly
Section titled “Measuring latency correctly”Most reported “speedups” are measurement errors. Rules:
- Warm up (10-50 iterations): the first runs include kernel autotuning, lazy initialization, caching.
- Synchronize on GPUs: CUDA calls are asynchronous; use events or
torch.cuda.synchronize()before reading the clock. - Report percentiles (p50, p95, p99), not the mean, for latency-sensitive systems; check thermal throttling on phones and Jetsons with sustained runs.
- Fix batch size, input resolution, precision and power mode and report them. Measure on the actual target device with the actual runtime.
- Include pre/post-processing (resize, normalization, NMS, decoding): on edge devices these can take as long as the model (Object Detection).
- Measure end-to-end including host-device transfers when the pipeline requires them.
import time, torch
@torch.no_grad()def benchmark(model, x, warmup=20, iters=200): model.eval() for _ in range(warmup): model(x) if x.is_cuda: torch.cuda.synchronize() times = [] for _ in range(iters): t0 = time.perf_counter() model(x) if x.is_cuda: torch.cuda.synchronize() times.append((time.perf_counter() - t0) * 1e3) times.sort() return {"p50_ms": times[len(times)//2], "p95_ms": times[int(len(times)*0.95)], "throughput_img_s": x.size(0) * 1e3 / (sum(times) / len(times))}Quantization
Section titled “Quantization”Replace float32/float16 weights and activations with low-bit integers (int8 typical; int4 for weights; fp8 on recent GPUs). Benefits: 4x smaller than fp32 (2x vs fp16), less memory traffic, and int8 tensor-core/NPU kernels are much faster than fp32.
Affine (uniform) quantization
Section titled “Affine (uniform) quantization”Map real value to an integer with scale and zero-point :
For int8 over a range : , (asymmetric). Symmetric quantization sets with range (common for weights; simpler kernels). Rounding error per element is at most , so outliers that stretch the range harm everyone: one big activation spike can waste most of the 256 levels.
import torch
def quantize_sym(x, bits=8): qmax = 2 ** (bits - 1) - 1 s = x.abs().max() / qmax q = torch.clamp(torch.round(x / s), -qmax - 1, qmax).to(torch.int8) return q, s
w = torch.randn(256, 256)q, s = quantize_sym(w)print("max abs error:", (w - q.float() * s).abs().max().item(), " bound s/2 =", (s / 2).item())Granularity: per-tensor (one scale) versus per-channel for weights (one scale per output channel; much more accurate for convolutions, the standard) versus per-group (blocks of 32-128 weights, common for 4-bit weights in LLM/VLM work).
PTQ versus QAT
Section titled “PTQ versus QAT”- Post-training quantization (PTQ): take a trained fp model; run a small calibration set (100-1000 representative images) to record activation ranges (min/max, percentile, or entropy/KL-based, MSE-optimal); set scales. Minutes, no training. Usually within 1% of fp accuracy on classification CNNs, but can hurt detection/segmentation heads, tiny networks (MobileNets with depthwise convs), and transformers (see outliers below).
- Quantization-aware training (QAT): insert “fake quantization” ops that quantize-dequantize in the forward pass, while the backward pass uses the straight-through estimator (treat as identity, with gradient zeroed outside clipping range). The network learns to be robust to rounding. Fine-tune for a few epochs at low learning rate; recovers most of the PTQ loss.
Practical points:
- Calibrate with data from your deployment distribution, not random noise or the training augmentations.
- Keep sensitive layers (first conv, last classifier, box-regression heads, softmax/attention scores, normalization) in higher precision (“mixed precision”); find them with layer-wise sensitivity analysis (quantize one layer at a time and watch metrics).
- BatchNorm folding: fold BN into preceding conv weights before quantizing: .
- ViTs and outliers: transformer activations (especially after LayerNorm and in FFN outputs) have a few huge channels; naive int8 activation quantization breaks accuracy. Remedies: per-channel/per-token activation scales, SmoothQuant-style migration of difficulty from activations to weights, keeping softmax/LayerNorm in fp16, or QAT. For VLM language backbones weight-only int4/int8 (GPTQ, AWQ) is popular since decoding is memory-bound.
- fp16/bf16 is the “free lunch” first step on GPUs (
torch.autocast); fp8 is increasingly available on newer accelerators with scaled formats (E4M3/E5M2).
# PyTorch FX/eager-mode details change across versions; torchao and ExecuTorch are the current direction.# A common portable route: export to ONNX and quantize with ONNX Runtime.from onnxruntime.quantization import quantize_static, CalibrationDataReader, QuantType
class Reader(CalibrationDataReader): def __init__(self, batches): self.it = iter(batches) # list of {"input": np.ndarray} def get_next(self): return next(self.it, None)
# quantize_static("model.onnx", "model.int8.onnx", Reader(calib_batches),# weight_type=QuantType.QInt8, per_channel=True)Knowledge distillation
Section titled “Knowledge distillation”Train a small student to mimic a large teacher. Classic soft-target loss (Hinton et al., 2015):
where softened teacher probabilities carry “dark knowledge” about class similarity. Vision variants:
- Feature distillation / hint layers: match intermediate feature maps (with a projection to align shapes), important for detection and segmentation where outputs are dense.
- Attention/relation distillation, and DETR/YOLO-specific distillation of box and class logits.
- Data-free or unlabeled-data distillation: the teacher can label large unlabeled sets (pseudo-labeling), which is how many foundation models ship small versions: DINOv2 distills the giant model into ViT-S/B/L; Depth Anything and SAM-style efficient variants (MobileSAM, EfficientSAM) follow the same approach (Self-Supervised & Foundation Vision Models, Segmentation).
- DeiT’s distillation token showed a CNN teacher improving a ViT student (Vision Transformers).
Distillation is often the highest-leverage compression: a small student trained with a strong teacher on lots of unlabeled in-domain data frequently beats the same student trained on labels alone by several points.
Pruning and sparsity
Section titled “Pruning and sparsity”- Unstructured pruning zeros individual weights by magnitude. Can reach 80-90% sparsity with small accuracy loss using iterative prune-and-fine-tune, but ordinary dense hardware gets no speedup; it only helps model size after compression, or on sparse-aware hardware/kernels.
- Structured pruning removes whole channels, filters, attention heads or layers. Gives real speedups on any hardware because the result is a smaller dense network. Criteria: L1 norm of filters, BN scale () magnitudes (“network slimming” with L1 penalty), or sensitivity/Taylor importance. Dependency graphs (Torch-Pruning) keep layer shapes consistent across residual connections.
- N:M semi-structured sparsity (e.g., 2:4 on NVIDIA Ampere+) gives up to ~2x matmul speedup in supported kernels with modest accuracy loss after fine-tuning.
- Token pruning / merging for ViTs (ToMe, DynamicViT, EViT) reduces the number of tokens per layer, effectively a dynamic compute saver.
- The lottery ticket hypothesis is an intriguing result but rarely a practical recipe.
Honest summary: for deployment, structured pruning + distillation, or simply choosing a smaller architecture and training it well, usually beats clever unstructured sparsity.
Efficient architectures
Section titled “Efficient architectures”- CNNs: MobileNetV2/V3/V4 (inverted residuals, SE, NAS), EfficientNet/V2, ShuffleNet, GhostNet, RegNet, RepVGG and structural re-parameterization (train with multi-branch blocks, merge into a single conv at inference; used in YOLOv6/v7, RepViT, FastViT, MobileOne), and ConvNeXt variants.
- Hybrid/efficient ViTs: MobileViT, EfficientFormer, EdgeNeXt, FastViT, EfficientViT (linear attention), RepViT, and TinyViT (distilled). Window attention, linear attention and token reduction cut cost; FlashAttention helps for long sequences on GPUs.
- Detectors/segmenters: YOLO “n/s” variants, RT-DETR-small, PP-YOLOE/PicoDet, NanoDet; MobileSAM; lightweight segmentation (PIDNet, SegFormer-B0).
- The practical rule: pick a mature architecture family with good export support on your runtime before optimizing; an op that the NPU doesn’t support falls back to CPU and can dominate latency.
Export and runtimes
Section titled “Export and runtimes”PyTorch/TF model --> export (ONNX / TorchScript / ExecuTorch / SavedModel) --> graph optimization (fusion, constant folding, layout) --> backend: TensorRT | ONNX Runtime (CUDA/TensorRT/OpenVINO/CoreML/NNAPI EPs) | OpenVINO | Core ML | TFLite/LiteRT | ExecuTorch | TVM --> device kernels (GPU, NPU, DSP, CPU)- ONNX is the interchange format; ONNX Runtime executes with pluggable execution providers (CPU, CUDA, TensorRT, OpenVINO, DirectML, CoreML, NNAPI/QNN).
- TensorRT (NVIDIA GPUs, Jetson): fuses layers (conv+BN+ReLU), picks tactics per layer on the specific GPU, supports fp16/int8/fp8 with calibration, dynamic shapes via optimization profiles. Engines are hardware- and version-specific; build them on the target. Typical speedup over eager PyTorch fp32: 2-10x depending on model and precision.
- OpenVINO for Intel CPUs/iGPUs/NPUs; Core ML for Apple devices (Neural Engine; convert via coremltools); TFLite/LiteRT and ExecuTorch for Android/embedded; NNAPI/QNN for Qualcomm; TVM/IREE compilers.
torch.compilegives graph-level fusions inside PyTorch with little effort; try it before leaving PyTorch for server inference.
import torch, torchvisionm = torchvision.models.resnet50(weights="DEFAULT").eval()x = torch.randn(1, 3, 224, 224)torch.onnx.export(m, x, "resnet50.onnx", input_names=["input"], output_names=["logits"], dynamic_axes={"input": {0: "batch"}, "logits": {0: "batch"}}, opset_version=17)# build an fp16 TensorRT engine on the target machine (flags vary by version)trtexec --onnx=resnet50.onnx --fp16 --saveEngine=resnet50_fp16.plan \ --minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 --maxShapes=input:32x3x224x224Export pitfalls: unsupported/custom ops (deformable attention, NMS variants, roi-align) need plugins or rewriting; dynamic shapes slow compilation; numerical differences versus PyTorch must be validated (compare outputs on a test set, not one image); keep pre/post-processing identical between training and deployment (resize interpolation, color order, normalization; see Image Fundamentals).
Systems-level tricks
Section titled “Systems-level tricks”- Batching (dynamic batching in Triton/TorchServe) for throughput; resolution scaling (accuracy falls gracefully, latency roughly quadratically) and frame skipping/tracker interleaving (run the detector every frames, track in between; see Video, 3D & Spatial Vision).
- Cascades: cheap model first, expensive model only for uncertain cases.
- Region-of-interest cropping to avoid processing the full frame.
- Hardware-aware choices: channels-last (NHWC) memory format and tensor-core-friendly channel counts (multiples of 8 or 16); fp16/bf16 on GPUs; int8 on NPUs and DSPs.
- Pipeline overlap: decode, preprocess on GPU (DALI, NVJPEG,
torchvision.io.decode_jpeg(device="cuda")), inference, postprocess in separate streams/threads.
A decision procedure
Section titled “A decision procedure”- Define target device, latency/throughput budget, accuracy floor.
- Establish baseline: eager fp32 on target; profile (PyTorch profiler, Nsight, trtexec).
- Cheap wins: fp16/bf16,
torch.compile, channels-last, batch size, ONNX Runtime/TensorRT. - Choose a smaller architecture or resolution; distill from the large model.
- Quantize (PTQ first; QAT if accuracy drops); validate on your data and on edge cases.
- Re-measure end to end on the device; keep a regression test comparing outputs and accuracy.
Key takeaways
Section titled “Key takeaways”- FLOPs and parameters are poor latency proxies; roofline reasoning (compute-bound vs memory-bound) explains why. Always benchmark on target hardware with warmup, synchronization and percentiles.
- int8 quantization uses scale/zero-point affine maps; use per-channel weights, calibrate on representative data, fall back to higher precision for sensitive layers, and use QAT (straight-through estimator) when PTQ loses accuracy. Transformers need outlier-aware handling.
- Distillation, especially with unlabeled data, is often the best accuracy-per-latency lever; structured pruning yields real speedups, unstructured usually does not.
- Export through ONNX to TensorRT/OpenVINO/Core ML/LiteRT; engines are hardware-specific, unsupported ops cause fallbacks, and numerical parity must be verified.
- Pre- and post-processing and system design (tracker interleaving, cascades, ROI) often matter more than model tweaks.
Exercises
Section titled “Exercises”- Conceptual. A convolution with 256 input and 256 output channels on a feature map: compute FLOPs (multiply-adds) and the arithmetic intensity in fp16 (ignore activation reuse subtleties). Compare with a depthwise on 256 channels.
- Conceptual. Why does one activation outlier hurt per-tensor int8 quantization, and name two mitigations.
- Hands-on. Benchmark
resnet50andmobilenet_v3_largeat batch 1 and 64 on CPU and (if available) GPU using the benchmark function above. Does the FLOP ranking hold? - Hands-on. Quantize a weight tensor with per-tensor versus per-channel symmetric int8 and compare reconstruction error when channel magnitudes vary by 100x.
- Conceptual. Why does unstructured 90% sparsity typically not speed up a GPU convolution?
Solutions (sketch)
- Standard conv: MACs; weights B MB, activations B MB each for in/out, so intensity is roughly 2 FLOPs / ~4.4 MB, about 840 FLOPs/B, compute-bound. Depthwise: MACs for the same ~3.2 MB of activation traffic, about 4-5 FLOPs/B, strongly memory-bound.
- Scale , so most values collapse into a few levels. Mitigations: percentile/clipping calibration, per-channel scales, SmoothQuant migration, mixed precision for the layer, QAT.
- Typically MobileNet wins on CPU at batch 1; at batch 64 on GPU the gap shrinks or reverses relative to FLOP ratios.
- Per-channel error is far lower because small channels get small scales.
- Dense kernels still process the zero entries; speedups require sparse-aware kernels or structured patterns such as 2:4.
Further reading
Section titled “Further reading”- Jacob et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference” (2018).
- Nagel et al., “A White Paper on Neural Network Quantization” (Qualcomm, 2021).
- Xiao et al., “SmoothQuant” (2022); Frantar et al., “GPTQ” (2022); Lin et al., “AWQ” (2023).
- Hinton, Vinyals, Dean, “Distilling the Knowledge in a Neural Network” (2015).
- Han et al., “Deep Compression” (2015); Liu et al., “Learning Efficient Convolutional Networks through Network Slimming” (2017).
- Ding et al., “RepVGG” (2021).
- Williams, Waterman, Patterson, “Roofline: An Insightful Visual Performance Model” (2009).
- Documentation: ONNX Runtime, NVIDIA TensorRT, OpenVINO, PyTorch (torch.ao/torchao, ExecuTorch), LiteRT.