Building CV Systems in Practice
What you’ll learn. A repeatable project workflow from problem definition to monitored deployment; which library to reach for (torchvision, timm, HuggingFace Transformers/Datasets, Ultralytics, Detectron2/MMDetection, Albumentations, FiftyOne); concrete code for classification fine-tuning, detection fine-tuning and zero-shot/VLM baselines; a fine-tuning recipe with sensible defaults; deployment patterns; and a systematic failure-analysis loop. This lecture ties together all earlier ones; links point to the details.
The workflow
Section titled “The workflow”1 Define task, output, metric tied to business cost, latency/hardware budget2 Look inspect 100+ raw images, collect baselines (heuristic, zero-shot VLM)3 Data collect, label with guidelines, split by independent unit4 Baseline strong pretrained model + simple training; get end-to-end pipeline working EARLY5 Iterate error analysis -> fix data/labels/model, one change at a time6 Optimize distill/quantize/export for target hardware7 Deploy versioned artifact, monitoring, feedback loop for new dataPrinciples:
- Build the dumbest end-to-end pipeline first (data in, metric out, a deployable artifact). Most projects fail on integration and data issues, not model choice.
- Fix the evaluation before the model. A trusted test set, split by independent unit, with slices and confidence intervals (Data, Augmentation & Evaluation).
- Change one thing at a time and log it (Weights & Biases, MLflow, TensorBoard; config files under version control; fixed seeds).
- Prefer pretrained, then fine-tune. From-scratch training is rarely justified.
- Decide with the cost of errors, not with a leaderboard metric.
Step 0: is it even a learning problem?
Section titled “Step 0: is it even a learning problem?”Before training, ask:
- Would a classical method do? Fixed camera, fixed lighting, rigid part: thresholding, template matching, or geometric measurement (Image Fundamentals).
- Does a zero-shot VLM or open-vocabulary detector already solve it well enough? Try 50 images with CLIP/SigLIP zero-shot, a VLM prompt, or Grounding DINO (Vision-Language Models, Object Detection). Even if too slow or inaccurate for production, it gives a baseline and an auto-labeler.
- Do you have a sensor solution (better lighting, a depth camera, a barcode) that makes the model’s job easy? Better input often beats better models.
Choosing a model and library
Section titled “Choosing a model and library”| Task | Practical starting point | Notes |
|---|---|---|
| Classification | timm (ConvNeXt, EfficientNet, ViT/DeiT, DINOv2/SigLIP backbones) or HF Transformers; or frozen foundation features + logistic regression |
Start with linear probe on DINOv2/SigLIP features (lecture 4); fine-tune if needed |
| Detection | Ultralytics YOLO (fast iteration, export), RT-DETR, torchvision Faster R-CNN, MMDetection/Detectron2 for research | Check the license (Ultralytics code is AGPL-3.0 with an enterprise license; verify for commercial use) |
| Segmentation | SegFormer / UperNet (HF), Mask2Former (HF/Detectron2), nnU-Net (medical), YOLO-seg; SAM 2 for prompted masks | Segmentation |
| Retrieval / similarity | DINOv2 or SigLIP embeddings + FAISS | cosine similarity, ANN index |
| OCR / documents | OCR engine pipeline or document VLM | verify numbers |
| Depth / 3D | Depth Anything-family, then geometry tools | Video, 3D & Spatial Vision |
| Generation | diffusers + LoRA |
Diffusion & Image Generation |
Library notes:
- torchvision: models,
transforms.v2(image+boxes+masks), ops (nms,roi_align), datasets, I/O. Stable, minimal. - timm (PyTorch Image Models): hundreds of pretrained backbones with consistent API, data configs, and training scripts.
- HuggingFace Transformers/Datasets/Evaluate/PEFT: models, processors that encode correct preprocessing,
Trainer, LoRA via PEFT, hub for checkpoints and datasets. - Ultralytics: one-API YOLO training, validation, tracking, export.
- Albumentations: fast augmentation with box/mask support. FiftyOne: dataset visualization and error analysis. CVAT/Label Studio: labeling.
Recipe 1: fine-tuning an image classifier
Section titled “Recipe 1: fine-tuning an image classifier”A strong, boring default.
import torch, timm, torch.nn as nnfrom torch.utils.data import DataLoaderfrom torchvision import datasetsfrom timm.data import resolve_data_config, create_transform
name = "convnext_tiny.fb_in22k_ft_in1k" # example; list options with timm.list_models(pretrained=True)model = timm.create_model(name, pretrained=True, num_classes=NUM_CLASSES, drop_path_rate=0.1)cfg = resolve_data_config({}, model=model)train_tf = create_transform(**cfg, is_training=True, auto_augment="rand-m9-mstd0.5-inc1")val_tf = create_transform(**cfg, is_training=False)
train_ds = datasets.ImageFolder("data/train", train_tf)val_ds = datasets.ImageFolder("data/val", val_tf)tl = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=8, pin_memory=True, drop_last=True)vl = DataLoader(val_ds, batch_size=128, num_workers=8)
dev = "cuda"model.to(dev)head = [p for n, p in model.named_parameters() if n.startswith(model.default_cfg["classifier"])]body = [p for n, p in model.named_parameters() if not n.startswith(model.default_cfg["classifier"])]opt = torch.optim.AdamW([{"params": body, "lr": 1e-4}, {"params": head, "lr": 1e-3}], weight_decay=0.05)EPOCHS = 15sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=[1e-4, 1e-3], total_steps=EPOCHS * len(tl), pct_start=0.1)crit = nn.CrossEntropyLoss(label_smoothing=0.1)scaler = torch.amp.GradScaler()
for ep in range(EPOCHS): model.train() for x, y in tl: x, y = x.to(dev, non_blocking=True), y.to(dev, non_blocking=True) with torch.autocast("cuda", dtype=torch.bfloat16): loss = crit(model(x), y) opt.zero_grad(set_to_none=True) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step(); sched.step() model.eval(); correct = n = 0 with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): for x, y in vl: p = model(x.to(dev)).argmax(1).cpu() correct += (p == y).sum().item(); n += len(y) print(f"epoch {ep}: val acc {correct / n:.3f}")(The GradScaler is unnecessary with bf16; keep it only for fp16.)
The fine-tuning recipe, with reasons
Section titled “The fine-tuning recipe, with reasons”- Start with a linear probe (frozen backbone, train head): fast sanity check; tells you how good the features already are.
- Then full fine-tune with a small learning rate for the backbone (1e-5 to 1e-4 for AdamW), larger for the new head (1e-3), warmup + cosine/one-cycle decay, weight decay 0.01-0.05. Too high a backbone LR destroys pretrained features.
- Layer-wise LR decay (0.65-0.9 per layer) for ViTs (Vision Transformers).
- Resolution: fine-tune at the resolution of your deployment objects; raising resolution helps small details.
- Augmentation + regularization matched to data size: RandAugment/TrivialAugment, random-resized crop, drop-path, label smoothing, Mixup/CutMix for larger sets; fewer for tiny sets.
- Class imbalance: class-weighted loss, balanced sampling, or focal loss; evaluate per class.
- Early stopping/checkpointing on a validation metric; average the last few checkpoints (EMA) for stability.
- Seeds and variance: with small data, run 3 seeds before concluding that a change helped.
- Parameter-efficient options (LoRA, adapters, BitFit) for big backbones or many tasks sharing one base model; full fine-tuning for the best single-task accuracy.
Recipe 2: detection with Ultralytics
Section titled “Recipe 2: detection with Ultralytics”path: datasets/widgetstrain: images/trainval: images/valnames: {0: scratch, 1: dent, 2: crack}from ultralytics import YOLOmodel = YOLO("yolo11s.pt") # a pretrained checkpoint (names change by release)model.train(data="data.yaml", epochs=100, imgsz=960, batch=16, patience=20, hsv_h=0.015, fliplr=0.5, mosaic=1.0, close_mosaic=10, seed=0)metrics = model.val(split="val", conf=0.001, iou=0.6) # mAP evaluation uses low confprint(metrics.box.map50, metrics.box.map)model.export(format="onnx", imgsz=960, dynamic=False, simplify=True) # then TensorRT/OpenVINOpreds = model.predict("sample.jpg", conf=0.35) # deployment threshold from the PR curvePractical: set imgsz so the smallest object is at least roughly 8-16 px; use tiled inference for very high-res images with tiny objects; choose the deployment confidence threshold from the validation PR curve for your cost tradeoff, not from the mAP setting (Object Detection); tune NMS IoU for crowding.
Recipe 3: zero-shot and VLM baselines
Section titled “Recipe 3: zero-shot and VLM baselines”# zero-shot classification with open_clip (works as a labeler and as a baseline)import torch, open_clipfrom PIL import Imagemodel, _, pre = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")tok = open_clip.get_tokenizer("ViT-B-32"); model.eval()classes = ["scratched metal surface", "clean metal surface"]with torch.no_grad(): t = model.encode_text(tok([f"a photo of a {c}" for c in classes])); t /= t.norm(dim=-1, keepdim=True) i = model.encode_image(pre(Image.open("part.jpg")).unsqueeze(0)); i /= i.norm(dim=-1, keepdim=True) print((100 * i @ t.T).softmax(-1))For structured extraction, prompt a VLM for JSON and validate with a schema (pydantic/jsonschema). Measure it like any other model on your labeled set. Common production pattern: VLM labels → human verification → train small specialist (Vision-Language Models).
Reproducibility and engineering hygiene
Section titled “Reproducibility and engineering hygiene”- Pin library versions (
requirements.txt/lockfile, Docker image), record git commit, config, dataset version and seed with every run. - Make preprocessing a single shared function/module used by training, evaluation and serving; most “model works in the notebook but not in production” bugs are preprocessing mismatches (BGR/RGB, resize interpolation, normalization, EXIF rotation; see Image Fundamentals).
- Unit-test the data pipeline (shapes, label ranges, augmentations preserve boxes) and visualize batches.
- Use data loaders with enough workers; profile whether you are GPU-bound or data-bound (decode and augment can bottleneck). Consider GPU decoding (NVJPEG/DALI) and caching resized data.
- Mixed precision (bf16) and
torch.compilefor speed; gradient accumulation for large effective batch sizes; checkpoint often.
Deployment
Section titled “Deployment”Choose the pattern by constraints:
| Pattern | When | Notes |
|---|---|---|
| Batch/offline | analytics, nightly processing | cheapest; optimize throughput |
| Online API (GPU server) | interactive apps | dynamic batching (Triton, TorchServe, BentoML, vLLM-style servers for VLMs); autoscaling; p95 latency SLOs |
| Edge device | privacy, offline, low latency | quantize and export for the device (Efficient & Edge Vision) |
| On-device mobile | consumer apps | Core ML/LiteRT/ExecuTorch; battery and thermal |
| Hybrid cascade | cost control | small model on-device, large model in the cloud for hard cases |
Production checklist:
- Versioned model artifact + config + preprocessing; rollback path.
- Input validation (size, format, corrupt files); timeouts; fallbacks when the model is unsure (confidence threshold, human review queue).
- Monitoring: input statistics (brightness, resolution, embedding drift versus training distribution), prediction distribution (class frequencies, confidence histograms), latency, errors; sampled human review for ground truth. Drift in embeddings from a foundation backbone is a cheap and effective signal.
- Feedback loop: log low-confidence and disagreement cases (with consent and privacy controls), label them, add to training and to the regression set; retrain on a schedule or trigger.
- Privacy, security, compliance: faces and plates are personal data in many jurisdictions; minimize retention, anonymize where possible, document intended use, and consider adversarial inputs when the system is security-relevant.
Failure analysis loop
Section titled “Failure analysis loop”When metrics disappoint, resist the urge to swap the architecture. Do this instead:
# 1. collect predictions with per-image loss/confidence and metadata on the validation set# 2. sort by loss, review the worst 100 by eye; tag each: label error / ambiguous / data gap / hard / bug# 3. count tags; fix the largest bucketimport numpy as np, torch, torch.nn.functional as F
@torch.no_grad()def collect(model, loader, dev="cuda"): rows = [] model.eval() for x, y, paths in loader: # loader yields file paths for inspection logits = model(x.to(dev)).float().cpu() loss = F.cross_entropy(logits, y, reduction="none") prob = logits.softmax(1); conf, pred = prob.max(1) rows += list(zip(paths, y.tolist(), pred.tolist(), conf.tolist(), loss.tolist())) return sorted(rows, key=lambda r: -r[4])Typical error buckets and fixes:
| Observation | Likely cause | Fix |
|---|---|---|
| Confident errors that look right on inspection | label errors | relabel; clean with confident learning |
| Errors concentrated in one condition (night, glare, a camera) | data gap | targeted collection or synthetic augmentation for that slice |
| Confusion between two similar classes | ambiguous definition or too-low resolution | refine guidelines, merge classes, raise resolution/crops |
| Small-object misses | resolution/stride | larger input, tiling, higher-res features (FPN level) |
| Train accuracy high, val low | overfitting/leakage-removed split | more data, stronger augmentation, smaller LR, regularization, verify splits |
| Both train and val low | underfitting, bad labels, bug | check label mapping, LR, capacity, preprocessing |
| Good offline, bad in production | distribution shift or preprocessing mismatch | compare input statistics; replay production images through the evaluation pipeline |
| Occasional wild predictions on unusual input | no OOD handling | add confidence thresholds, OOD detector, human fallback |
Sanity tests that catch many bugs: overfit a tiny batch (32 images) to ~100% (if you can’t, there is a bug); shuffle labels (accuracy should drop to chance); feed a constant image; check that augmented samples still look correctly labeled; verify class index mapping between training and serving.
Key takeaways
Section titled “Key takeaways”- Work in a loop: define the metric tied to cost, build a trustworthy evaluation set, get an end-to-end baseline early, then iterate by error analysis.
- Start from pretrained backbones (timm/HF/torchvision/Ultralytics); do a linear probe, then fine-tune with a lower backbone LR, warmup/cosine schedule, augmentation and regularization suited to data size.
- Share one preprocessing implementation across training and serving; most deployment bugs are preprocessing mismatches.
- Use zero-shot VLMs and open-vocabulary models as baselines and auto-labelers, then train a small specialist for production.
- Deploy with versioning, monitoring (input and prediction drift), human fallback and a feedback loop; analyze failures by bucket before changing models.
Exercises
Section titled “Exercises”- Conceptual. Your detector scores 0.62 mAP50 offline but a stakeholder says it “misses a lot” in production. List five hypotheses and how you would test each.
- Conceptual. Why use a lower learning rate for the backbone than the new head when fine-tuning?
- Hands-on. Pick a small public image dataset (e.g., Oxford Flowers, or 5 classes from Food-101). Compare: (a) logistic regression on frozen DINOv2 features, (b) zero-shot CLIP, (c) full fine-tune of a ConvNeXt-Tiny. Report accuracy and time for each.
- Hands-on. Train a YOLO model on a small custom set, then run the failure-analysis loop: gather the 50 worst images by loss/FN, tag causes, and fix the largest bucket. Report the metric change.
- Conceptual. Design a monitoring dashboard for a deployed defect classifier: which 5 signals and what alerts?
Solutions (sketch)
- Candidates: distribution shift (production images differ: test by comparing embedding/brightness statistics); threshold mismatch (mAP uses conf 0.001; production uses 0.5, so recall low: sweep thresholds); preprocessing mismatch (replay production frames through the eval pipeline); different object sizes/resolution (per-size AP); labels in test set not matching what stakeholders consider a miss (review misses together).
- Pretrained features are already good; large updates with random-head gradients early in training can corrupt them (“feature distortion”). The new head starts random and needs larger steps.
- Expect (a) and (b) to take minutes with surprisingly high accuracy, (c) highest accuracy with the most compute; foundation features often win at tiny data.
- Often the largest bucket is label inconsistency or missing small-object labels; relabeling gives larger gains than hyperparameters.
- Input brightness/resolution drift, embedding drift vs. training reference (e.g., MMD or mean distance), prediction class frequencies, confidence histogram/low-confidence rate, latency p95 and error rate; plus sampled human-review accuracy. Alert on sustained deviations beyond a baseline band.
Further reading
Section titled “Further reading”- Andrew Ng, “A Chat with Andrew on MLOps: From Model-centric to Data-centric AI” (2021).
- Andrej Karpathy, “A Recipe for Training Neural Networks” (blog, 2019).
- Chip Huyen, Designing Machine Learning Systems (2022).
- Sculley et al., “Hidden Technical Debt in Machine Learning Systems” (2015).
- Wightman, “PyTorch Image Models (timm)” documentation and “ResNet strikes back” (2021).
- HuggingFace documentation: Transformers (vision), Datasets, PEFT, diffusers; Ultralytics documentation; torchvision documentation.
- Kornblith et al., “Do Better ImageNet Models Transfer Better?” (2019); Kumar et al., “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution” (2022).
- Northcutt et al., “Confident Learning” (2021).