Skip to content

Building CV Systems in Practice

What you’ll learn. A repeatable project workflow from problem definition to monitored deployment; which library to reach for (torchvision, timm, HuggingFace Transformers/Datasets, Ultralytics, Detectron2/MMDetection, Albumentations, FiftyOne); concrete code for classification fine-tuning, detection fine-tuning and zero-shot/VLM baselines; a fine-tuning recipe with sensible defaults; deployment patterns; and a systematic failure-analysis loop. This lecture ties together all earlier ones; links point to the details.

1 Define task, output, metric tied to business cost, latency/hardware budget
2 Look inspect 100+ raw images, collect baselines (heuristic, zero-shot VLM)
3 Data collect, label with guidelines, split by independent unit
4 Baseline strong pretrained model + simple training; get end-to-end pipeline working EARLY
5 Iterate error analysis -> fix data/labels/model, one change at a time
6 Optimize distill/quantize/export for target hardware
7 Deploy versioned artifact, monitoring, feedback loop for new data

Principles:

  • Build the dumbest end-to-end pipeline first (data in, metric out, a deployable artifact). Most projects fail on integration and data issues, not model choice.
  • Fix the evaluation before the model. A trusted test set, split by independent unit, with slices and confidence intervals (Data, Augmentation & Evaluation).
  • Change one thing at a time and log it (Weights & Biases, MLflow, TensorBoard; config files under version control; fixed seeds).
  • Prefer pretrained, then fine-tune. From-scratch training is rarely justified.
  • Decide with the cost of errors, not with a leaderboard metric.

Before training, ask:

  • Would a classical method do? Fixed camera, fixed lighting, rigid part: thresholding, template matching, or geometric measurement (Image Fundamentals).
  • Does a zero-shot VLM or open-vocabulary detector already solve it well enough? Try 50 images with CLIP/SigLIP zero-shot, a VLM prompt, or Grounding DINO (Vision-Language Models, Object Detection). Even if too slow or inaccurate for production, it gives a baseline and an auto-labeler.
  • Do you have a sensor solution (better lighting, a depth camera, a barcode) that makes the model’s job easy? Better input often beats better models.
Task Practical starting point Notes
Classification timm (ConvNeXt, EfficientNet, ViT/DeiT, DINOv2/SigLIP backbones) or HF Transformers; or frozen foundation features + logistic regression Start with linear probe on DINOv2/SigLIP features (lecture 4); fine-tune if needed
Detection Ultralytics YOLO (fast iteration, export), RT-DETR, torchvision Faster R-CNN, MMDetection/Detectron2 for research Check the license (Ultralytics code is AGPL-3.0 with an enterprise license; verify for commercial use)
Segmentation SegFormer / UperNet (HF), Mask2Former (HF/Detectron2), nnU-Net (medical), YOLO-seg; SAM 2 for prompted masks Segmentation
Retrieval / similarity DINOv2 or SigLIP embeddings + FAISS cosine similarity, ANN index
OCR / documents OCR engine pipeline or document VLM verify numbers
Depth / 3D Depth Anything-family, then geometry tools Video, 3D & Spatial Vision
Generation diffusers + LoRA Diffusion & Image Generation

Library notes:

  • torchvision: models, transforms.v2 (image+boxes+masks), ops (nms, roi_align), datasets, I/O. Stable, minimal.
  • timm (PyTorch Image Models): hundreds of pretrained backbones with consistent API, data configs, and training scripts.
  • HuggingFace Transformers/Datasets/Evaluate/PEFT: models, processors that encode correct preprocessing, Trainer, LoRA via PEFT, hub for checkpoints and datasets.
  • Ultralytics: one-API YOLO training, validation, tracking, export.
  • Albumentations: fast augmentation with box/mask support. FiftyOne: dataset visualization and error analysis. CVAT/Label Studio: labeling.

A strong, boring default.

import torch, timm, torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets
from timm.data import resolve_data_config, create_transform
name = "convnext_tiny.fb_in22k_ft_in1k" # example; list options with timm.list_models(pretrained=True)
model = timm.create_model(name, pretrained=True, num_classes=NUM_CLASSES, drop_path_rate=0.1)
cfg = resolve_data_config({}, model=model)
train_tf = create_transform(**cfg, is_training=True, auto_augment="rand-m9-mstd0.5-inc1")
val_tf = create_transform(**cfg, is_training=False)
train_ds = datasets.ImageFolder("data/train", train_tf)
val_ds = datasets.ImageFolder("data/val", val_tf)
tl = DataLoader(train_ds, batch_size=64, shuffle=True, num_workers=8, pin_memory=True, drop_last=True)
vl = DataLoader(val_ds, batch_size=128, num_workers=8)
dev = "cuda"
model.to(dev)
head = [p for n, p in model.named_parameters() if n.startswith(model.default_cfg["classifier"])]
body = [p for n, p in model.named_parameters() if not n.startswith(model.default_cfg["classifier"])]
opt = torch.optim.AdamW([{"params": body, "lr": 1e-4}, {"params": head, "lr": 1e-3}], weight_decay=0.05)
EPOCHS = 15
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=[1e-4, 1e-3], total_steps=EPOCHS * len(tl), pct_start=0.1)
crit = nn.CrossEntropyLoss(label_smoothing=0.1)
scaler = torch.amp.GradScaler()
for ep in range(EPOCHS):
model.train()
for x, y in tl:
x, y = x.to(dev, non_blocking=True), y.to(dev, non_blocking=True)
with torch.autocast("cuda", dtype=torch.bfloat16):
loss = crit(model(x), y)
opt.zero_grad(set_to_none=True)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step(); sched.step()
model.eval(); correct = n = 0
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
for x, y in vl:
p = model(x.to(dev)).argmax(1).cpu()
correct += (p == y).sum().item(); n += len(y)
print(f"epoch {ep}: val acc {correct / n:.3f}")

(The GradScaler is unnecessary with bf16; keep it only for fp16.)

  1. Start with a linear probe (frozen backbone, train head): fast sanity check; tells you how good the features already are.
  2. Then full fine-tune with a small learning rate for the backbone (1e-5 to 1e-4 for AdamW), larger for the new head (1e-3), warmup + cosine/one-cycle decay, weight decay 0.01-0.05. Too high a backbone LR destroys pretrained features.
  3. Layer-wise LR decay (0.65-0.9 per layer) for ViTs (Vision Transformers).
  4. Resolution: fine-tune at the resolution of your deployment objects; raising resolution helps small details.
  5. Augmentation + regularization matched to data size: RandAugment/TrivialAugment, random-resized crop, drop-path, label smoothing, Mixup/CutMix for larger sets; fewer for tiny sets.
  6. Class imbalance: class-weighted loss, balanced sampling, or focal loss; evaluate per class.
  7. Early stopping/checkpointing on a validation metric; average the last few checkpoints (EMA) for stability.
  8. Seeds and variance: with small data, run 3 seeds before concluding that a change helped.
  9. Parameter-efficient options (LoRA, adapters, BitFit) for big backbones or many tasks sharing one base model; full fine-tuning for the best single-task accuracy.
data.yaml
path: datasets/widgets
train: images/train
val: images/val
names: {0: scratch, 1: dent, 2: crack}
from ultralytics import YOLO
model = YOLO("yolo11s.pt") # a pretrained checkpoint (names change by release)
model.train(data="data.yaml", epochs=100, imgsz=960, batch=16, patience=20,
hsv_h=0.015, fliplr=0.5, mosaic=1.0, close_mosaic=10, seed=0)
metrics = model.val(split="val", conf=0.001, iou=0.6) # mAP evaluation uses low conf
print(metrics.box.map50, metrics.box.map)
model.export(format="onnx", imgsz=960, dynamic=False, simplify=True) # then TensorRT/OpenVINO
preds = model.predict("sample.jpg", conf=0.35) # deployment threshold from the PR curve

Practical: set imgsz so the smallest object is at least roughly 8-16 px; use tiled inference for very high-res images with tiny objects; choose the deployment confidence threshold from the validation PR curve for your cost tradeoff, not from the mAP setting (Object Detection); tune NMS IoU for crowding.

# zero-shot classification with open_clip (works as a labeler and as a baseline)
import torch, open_clip
from PIL import Image
model, _, pre = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")
tok = open_clip.get_tokenizer("ViT-B-32"); model.eval()
classes = ["scratched metal surface", "clean metal surface"]
with torch.no_grad():
t = model.encode_text(tok([f"a photo of a {c}" for c in classes])); t /= t.norm(dim=-1, keepdim=True)
i = model.encode_image(pre(Image.open("part.jpg")).unsqueeze(0)); i /= i.norm(dim=-1, keepdim=True)
print((100 * i @ t.T).softmax(-1))

For structured extraction, prompt a VLM for JSON and validate with a schema (pydantic/jsonschema). Measure it like any other model on your labeled set. Common production pattern: VLM labels → human verification → train small specialist (Vision-Language Models).

  • Pin library versions (requirements.txt/lockfile, Docker image), record git commit, config, dataset version and seed with every run.
  • Make preprocessing a single shared function/module used by training, evaluation and serving; most “model works in the notebook but not in production” bugs are preprocessing mismatches (BGR/RGB, resize interpolation, normalization, EXIF rotation; see Image Fundamentals).
  • Unit-test the data pipeline (shapes, label ranges, augmentations preserve boxes) and visualize batches.
  • Use data loaders with enough workers; profile whether you are GPU-bound or data-bound (decode and augment can bottleneck). Consider GPU decoding (NVJPEG/DALI) and caching resized data.
  • Mixed precision (bf16) and torch.compile for speed; gradient accumulation for large effective batch sizes; checkpoint often.

Choose the pattern by constraints:

Pattern When Notes
Batch/offline analytics, nightly processing cheapest; optimize throughput
Online API (GPU server) interactive apps dynamic batching (Triton, TorchServe, BentoML, vLLM-style servers for VLMs); autoscaling; p95 latency SLOs
Edge device privacy, offline, low latency quantize and export for the device (Efficient & Edge Vision)
On-device mobile consumer apps Core ML/LiteRT/ExecuTorch; battery and thermal
Hybrid cascade cost control small model on-device, large model in the cloud for hard cases

Production checklist:

  • Versioned model artifact + config + preprocessing; rollback path.
  • Input validation (size, format, corrupt files); timeouts; fallbacks when the model is unsure (confidence threshold, human review queue).
  • Monitoring: input statistics (brightness, resolution, embedding drift versus training distribution), prediction distribution (class frequencies, confidence histograms), latency, errors; sampled human review for ground truth. Drift in embeddings from a foundation backbone is a cheap and effective signal.
  • Feedback loop: log low-confidence and disagreement cases (with consent and privacy controls), label them, add to training and to the regression set; retrain on a schedule or trigger.
  • Privacy, security, compliance: faces and plates are personal data in many jurisdictions; minimize retention, anonymize where possible, document intended use, and consider adversarial inputs when the system is security-relevant.

When metrics disappoint, resist the urge to swap the architecture. Do this instead:

# 1. collect predictions with per-image loss/confidence and metadata on the validation set
# 2. sort by loss, review the worst 100 by eye; tag each: label error / ambiguous / data gap / hard / bug
# 3. count tags; fix the largest bucket
import numpy as np, torch, torch.nn.functional as F
@torch.no_grad()
def collect(model, loader, dev="cuda"):
rows = []
model.eval()
for x, y, paths in loader: # loader yields file paths for inspection
logits = model(x.to(dev)).float().cpu()
loss = F.cross_entropy(logits, y, reduction="none")
prob = logits.softmax(1); conf, pred = prob.max(1)
rows += list(zip(paths, y.tolist(), pred.tolist(), conf.tolist(), loss.tolist()))
return sorted(rows, key=lambda r: -r[4])

Typical error buckets and fixes:

Observation Likely cause Fix
Confident errors that look right on inspection label errors relabel; clean with confident learning
Errors concentrated in one condition (night, glare, a camera) data gap targeted collection or synthetic augmentation for that slice
Confusion between two similar classes ambiguous definition or too-low resolution refine guidelines, merge classes, raise resolution/crops
Small-object misses resolution/stride larger input, tiling, higher-res features (FPN level)
Train accuracy high, val low overfitting/leakage-removed split more data, stronger augmentation, smaller LR, regularization, verify splits
Both train and val low underfitting, bad labels, bug check label mapping, LR, capacity, preprocessing
Good offline, bad in production distribution shift or preprocessing mismatch compare input statistics; replay production images through the evaluation pipeline
Occasional wild predictions on unusual input no OOD handling add confidence thresholds, OOD detector, human fallback

Sanity tests that catch many bugs: overfit a tiny batch (32 images) to ~100% (if you can’t, there is a bug); shuffle labels (accuracy should drop to chance); feed a constant image; check that augmented samples still look correctly labeled; verify class index mapping between training and serving.

  • Work in a loop: define the metric tied to cost, build a trustworthy evaluation set, get an end-to-end baseline early, then iterate by error analysis.
  • Start from pretrained backbones (timm/HF/torchvision/Ultralytics); do a linear probe, then fine-tune with a lower backbone LR, warmup/cosine schedule, augmentation and regularization suited to data size.
  • Share one preprocessing implementation across training and serving; most deployment bugs are preprocessing mismatches.
  • Use zero-shot VLMs and open-vocabulary models as baselines and auto-labelers, then train a small specialist for production.
  • Deploy with versioning, monitoring (input and prediction drift), human fallback and a feedback loop; analyze failures by bucket before changing models.
  1. Conceptual. Your detector scores 0.62 mAP50 offline but a stakeholder says it “misses a lot” in production. List five hypotheses and how you would test each.
  2. Conceptual. Why use a lower learning rate for the backbone than the new head when fine-tuning?
  3. Hands-on. Pick a small public image dataset (e.g., Oxford Flowers, or 5 classes from Food-101). Compare: (a) logistic regression on frozen DINOv2 features, (b) zero-shot CLIP, (c) full fine-tune of a ConvNeXt-Tiny. Report accuracy and time for each.
  4. Hands-on. Train a YOLO model on a small custom set, then run the failure-analysis loop: gather the 50 worst images by loss/FN, tag causes, and fix the largest bucket. Report the metric change.
  5. Conceptual. Design a monitoring dashboard for a deployed defect classifier: which 5 signals and what alerts?
Solutions (sketch)
  1. Candidates: distribution shift (production images differ: test by comparing embedding/brightness statistics); threshold mismatch (mAP uses conf 0.001; production uses 0.5, so recall low: sweep thresholds); preprocessing mismatch (replay production frames through the eval pipeline); different object sizes/resolution (per-size AP); labels in test set not matching what stakeholders consider a miss (review misses together).
  2. Pretrained features are already good; large updates with random-head gradients early in training can corrupt them (“feature distortion”). The new head starts random and needs larger steps.
  3. Expect (a) and (b) to take minutes with surprisingly high accuracy, (c) highest accuracy with the most compute; foundation features often win at tiny data.
  4. Often the largest bucket is label inconsistency or missing small-object labels; relabeling gives larger gains than hyperparameters.
  5. Input brightness/resolution drift, embedding drift vs. training reference (e.g., MMD or mean distance), prediction class frequencies, confidence histogram/low-confidence rate, latency p95 and error rate; plus sampled human-review accuracy. Alert on sustained deviations beyond a baseline band.
  • Andrew Ng, “A Chat with Andrew on MLOps: From Model-centric to Data-centric AI” (2021).
  • Andrej Karpathy, “A Recipe for Training Neural Networks” (blog, 2019).
  • Chip Huyen, Designing Machine Learning Systems (2022).
  • Sculley et al., “Hidden Technical Debt in Machine Learning Systems” (2015).
  • Wightman, “PyTorch Image Models (timm)” documentation and “ResNet strikes back” (2021).
  • HuggingFace documentation: Transformers (vision), Datasets, PEFT, diffusers; Ultralytics documentation; torchvision documentation.
  • Kornblith et al., “Do Better ImageNet Models Transfer Better?” (2019); Kumar et al., “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution” (2022).
  • Northcutt et al., “Confident Learning” (2021).