Skip to content

Object Detection

What you’ll learn. The detection problem and its geometry (boxes, IoU, anchors); the two-stage R-CNN family; the one-stage lineage (SSD, RetinaNet, YOLO); anchor-free detectors; transformer detectors (DETR, Deformable DETR, RT-DETR) with set prediction and Hungarian matching; open-vocabulary detection (OWL-ViT, Grounding DINO, YOLO-World); and how to compute IoU, NMS and mAP, including COCO conventions. Backbones are covered in CNN Architectures and Vision Transformers.

Given an image, output a set of (box,class,score)(\text{box}, \text{class}, \text{score}) triples. A box is (x1,y1,x2,y2)(x_1,y_1,x_2,y_2) or (cx,cy,w,h)(c_x,c_y,w,h). Difficulties: variable number of outputs, multiple scales, heavy class imbalance (most locations are background), and a metric that requires both correct class and accurate localization.

IoU⁡(A,B)=∣A∩B∣∣A∪B∣.\operatorname{IoU}(A,B) = \frac{|A\cap B|}{|A\cup B|}.
import torch
def box_iou(a, b): # a: (N,4), b: (M,4) in xyxy
area = lambda x: (x[:, 2]-x[:, 0]).clamp(0) * (x[:, 3]-x[:, 1]).clamp(0)
lt = torch.max(a[:, None, :2], b[None, :, :2])
rb = torch.min(a[:, None, 2:], b[None, :, 2:])
inter = (rb - lt).clamp(0).prod(-1)
return inter / (area(a)[:, None] + area(b)[None] - inter)

Regression losses evolved from L1/smooth-L1 on box coordinates to IoU-based losses (GIoU, DIoU, CIoU), which optimize the metric more directly and handle non-overlapping boxes (GIoU adds a penalty from the smallest enclosing box).

  • R-CNN (2014): ~2000 region proposals from Selective Search, each warped and run through a CNN, then SVM classification and box regression. Accurate but extremely slow (tens of seconds per image).
  • Fast R-CNN (2015): run the CNN once for the whole image; RoI pooling crops each proposal’s features from the shared map. Joint classification and regression with a multi-task loss.
  • Faster R-CNN (2015): replace Selective Search with a learned Region Proposal Network (RPN) sharing the backbone. At every feature map location, kk anchors (reference boxes at several scales/aspect ratios) are scored for objectness and refined by regression offsets.
  • Mask R-CNN (2017): adds a mask branch and replaces RoI pooling by RoIAlign (bilinear sampling, no quantization). See Segmentation.
  • Feature Pyramid Network (FPN) (2017): a top-down pathway with lateral connections yields semantically strong features at multiple scales; assign proposals to pyramid levels by size. Nearly every modern detector uses an FPN or equivalent.

Box regression encodes offsets relative to an anchor (xa,ya,wa,ha)(x_a,y_a,w_a,h_a): tx=(x−xa)/wat_x=(x-x_a)/w_a, ty=(y−ya)/hat_y=(y-y_a)/h_a, tw=log⁡(w/wa)t_w=\log(w/w_a), th=log⁡(h/ha)t_h=\log(h/h_a).

Two-stage detectors remain strong for accuracy-critical, small-object problems and are the foundation for instance segmentation, but they are more complex to deploy than one-stage detectors.

Skip proposals; predict classes and boxes densely.

  • SSD (2016): predictions from multiple feature maps with default boxes.
  • RetinaNet (2017): showed that the accuracy gap of one-stage detectors came from foreground/background imbalance (about 10510^5 background anchors versus a handful of objects) and introduced focal loss:
FL⁡(pt)=−αt(1−pt)γlog⁡pt,\operatorname{FL}(p_t) = -\alpha_t(1-p_t)^\gamma \log p_t,

where ptp_t is the probability of the true class. The (1−pt)γ(1-p_t)^\gamma factor (typically γ=2\gamma=2) down-weights easy examples so a flood of easy negatives doesn’t dominate the gradient.

  • YOLO (“You Only Look Once”, Redmon 2016): divide the image into a grid; each cell predicts boxes, objectness and class probabilities in a single forward pass. It prioritized real-time speed.

YOLO’s story is one of steady engineering integration rather than a single idea:

  • v2/v3 (Redmon): anchor boxes via k-means on training boxes, batch norm, multi-scale prediction with an FPN-style neck, Darknet backbones.
  • v4/v5 (2020; v5 by Ultralytics): CSP backbones, PAN neck, mosaic augmentation, CIoU loss, easy PyTorch tooling that made YOLOv5 the default for practitioners.
  • YOLOX, v6, v7: decoupled heads (separate classification and regression branches), anchor-free prediction, SimOTA dynamic label assignment, re-parameterized convolutions.
  • YOLOv8 / 11 (Ultralytics) and later versions: anchor-free, decoupled head, task-aligned assignment and distribution focal loss for box edges, unified detect/segment/pose/classify tasks.
  • YOLOv10 (2024): NMS-free training via dual assignments (one-to-many during training, one-to-one for inference) removing NMS from the pipeline.
  • Newer numbered releases through 2025-2026 (YOLOv12 with attention-centric blocks, YOLO26, and others) continue to appear from various groups. Versions are not canonical (the “YOLO” brand is not controlled by one group), and headline gains are often small and benchmarked with differing protocols. Evaluate on your data.

Label assignment is the quiet heart of modern one-stage detectors: deciding which predictions are positive for each ground-truth box. Evolution: IoU thresholds with anchors → ATSS (adaptive thresholds) → SimOTA/TOOD (assign by a combined cost of classification and localization quality, dynamically).

Anchor-free detectors predict directly from locations: FCOS predicts distances from a location to the four box sides plus a “centerness” score; CenterNet predicts object centers as heatmap peaks plus size. These remove anchor hyperparameters.

Dense predictors output many overlapping boxes per object. NMS: sort by score; keep the top box; discard any remaining box (of the same class) with IoU above a threshold (e.g., 0.5-0.7); repeat.

from torchvision.ops import nms, batched_nms
keep = batched_nms(boxes, scores, class_ids, iou_threshold=0.6) # per-class NMS

Problems: threshold tuning (crowded scenes suppress true neighbors), not differentiable, sequential (awkward on accelerators). Variants: Soft-NMS (decay scores instead of deleting), Weighted Box Fusion (for ensembles), and the end-to-end alternatives below that eliminate it.

DETR (Carion et al., 2020) recasts detection as set prediction. A CNN/ViT backbone yields features; a transformer encoder-decoder takes NN learned object queries (e.g., N=100N=100) and each query outputs one box+class (or “no object”). There are no anchors and no NMS.

Training needs a one-to-one assignment between predictions and ground truths. DETR uses the Hungarian algorithm to find the matching σ^\hat\sigma minimizing total matching cost:

σ^=arg⁡min⁡σ∑iLmatch(yi,y^σ(i)),Lmatch=−p^σ(i)(ci)+λ1∥bi−b^σ(i)∥1+λ2LGIoU.\hat\sigma = \arg\min_{\sigma}\sum_{i} \mathcal{L}_\text{match}(y_i, \hat y_{\sigma(i)}),\qquad \mathcal{L}_\text{match} = -\hat p_{\sigma(i)}(c_i) + \lambda_1\|b_i-\hat b_{\sigma(i)}\|_1 + \lambda_2 \mathcal{L}_\text{GIoU}.

Then the loss (classification + L1 + GIoU) is computed on matched pairs; unmatched queries are trained to predict “no object”. One-to-one matching is what lets the model learn to output exactly one box per object, hence no NMS.

import torch
from scipy.optimize import linear_sum_assignment
def hungarian_match(prob, boxes, gt_cls, gt_boxes, w_cls=1.0, w_l1=5.0):
# prob: (N,C), boxes: (N,4), gt_cls: (M,), gt_boxes: (M,4)
cost = -w_cls * prob[:, gt_cls] + w_l1 * torch.cdist(boxes, gt_boxes, p=1) # (N,M)
rows, cols = linear_sum_assignment(cost.detach().cpu().numpy())
return rows, cols # query i matched to GT j

Original DETR trained slowly (500 epochs) and was weak on small objects. Fixes:

  • Deformable DETR: attention over a small learned set of sampling points per query around a reference point, over multi-scale features; 10x fewer epochs, better small objects.
  • Conditional/DAB-DETR: queries as explicit box coordinates (anchors).
  • DN-DETR / DINO (DETR with Improved deNoising anchor boxes): add noisy ground-truth queries during training to stabilize matching; contrastive denoising, mixed query selection. DINO-DETR reached top COCO results with large backbones. (Note: unrelated to the DINO self-supervised method despite sharing a name.)
  • Group/Hybrid DETR, Co-DETR: add one-to-many auxiliary heads in training to give more positive supervision, still deploying one-to-one.
  • RT-DETR (2023) and RT-DETRv2/v3: a real-time DETR with an efficient hybrid encoder (intra-scale attention on the top level, cross-scale CNN-based fusion), IoU-aware query selection, and flexible decoder layer count for speed tuning; reported to match or beat comparably sized YOLOs in speed/accuracy while needing no NMS. Variants such as D-FINE refined box regression as distribution refinement. RT-DETR-style models are now in Ultralytics, HuggingFace Transformers and other toolkits.

Classical detectors have a fixed label set. Open-vocabulary detectors accept text queries by replacing the classification layer with similarity to text embeddings (Vision-Language Models).

  • OWL-ViT / OWLv2: a CLIP-style ViT with per-token box heads; classify each token’s embedding against text (or image-exemplar) embeddings. OWLv2 self-trains on pseudo-labels from web image-text data to scale up.
  • GLIP, Grounding DINO: fuse text and image features deeply (cross-modality attention in the encoder and decoder), treating detection as phrase grounding; Grounding DINO is widely used for “detect anything I describe” and, combined with SAM, to auto-label masks (Grounded-SAM, Segmentation).
  • YOLO-World: injects CLIP text embeddings into a YOLO via a re-parameterizable vision-language PAN, “prompt-then-detect” with offline vocabularies for real-time speed.
  • Newer large-VLM-based detectors (some VLMs output boxes as text tokens, e.g., Florence-2, Qwen-VL family, PaliGemma) blur detection into language modeling.

Open-vocabulary models are great for bootstrapping labels and rare categories, but accuracy on fine-grained, domain-specific classes (specific defects, PCB components) is typically below that of a closed-set detector trained on in-domain data. A common production pattern: use an open-vocabulary model to auto-label, human-verify, then train a small supervised detector.

For one class: sort detections by confidence. A detection is a true positive if IoU with an unmatched ground-truth box of the same class is ≥\ge threshold tt; otherwise a false positive. Sweeping the confidence threshold traces a precision-recall curve; AP is the (interpolated) area under it:

AP=∫01p(r) dr,precision=TPTP+FP, recall=TPTP+FN.\mathrm{AP} = \int_0^1 p(r)\,dr,\qquad \mathrm{precision}=\frac{TP}{TP+FP},\ \mathrm{recall}=\frac{TP}{TP+FN}.

mAP averages AP over classes. Conventions differ:

  • PASCAL VOC: AP at IoU 0.5 (older versions with 11-point interpolation).
  • COCO: AP\mathrm{AP} (also written mAP@[.5:.95]) averages over IoU thresholds 0.50, 0.55, …, 0.95 and classes; AP50\mathrm{AP}_{50}, AP75\mathrm{AP}_{75}; and APS/APM/APL\mathrm{AP}_S/\mathrm{AP}_M/\mathrm{AP}_L for small (<32^2 px area), medium, large objects; plus AR\mathrm{AR} recall metrics, with up to 100 detections per image.
# pip install torchmetrics pycocotools
from torchmetrics.detection import MeanAveragePrecision
metric = MeanAveragePrecision(iou_type="bbox")
preds = [dict(boxes=torch.tensor([[10., 10., 50., 50.]]), scores=torch.tensor([0.9]), labels=torch.tensor([1]))]
target = [dict(boxes=torch.tensor([[12., 11., 48., 52.]]), labels=torch.tensor([1]))]
metric.update(preds, target)
print(metric.compute()["map"])

Practical points: mAP does not tell you the operating point to deploy; pick a confidence threshold from the PR curve for your cost tradeoff (F1, or recall at fixed precision). Report per-class and per-size AP; check for confusions with a confusion matrix at IoU 0.5. Make sure that the evaluation uses the same NMS, score threshold (low, e.g. 0.001, for mAP) and image size across models, otherwise comparisons mislead (Data, Augmentation & Evaluation).

  • Start from a pretrained detector and fine-tune; data of 500-5,000 labeled images is often enough for a narrow domain.
  • Mosaic/MixUp/copy-paste augmentation helps small datasets; turn mosaic off for the last few epochs (YOLO practice) so the model sees realistic layouts.
  • Train/infer at an image size matched to your smallest objects; small-object recall is often a resolution problem (or use tiling/SAHI-style sliced inference).
  • Label quality: inconsistent boxes put a ceiling on mAP; audit with the model’s high-loss samples.
from ultralytics import YOLO
model = YOLO("yolo11n.pt") # a pretrained checkpoint name; versions change
model.train(data="data.yaml", epochs=100, imgsz=640, batch=16)
metrics = model.val() # prints mAP50, mAP50-95
results = model("image.jpg", conf=0.25)
  • Detection = classify + localize a variable-size set; IoU is the geometric core; AP/mAP summarize PR curves, with COCO averaging IoU 0.5 to 0.95.
  • Two-stage (Faster R-CNN + FPN) is accurate and extensible; one-stage (RetinaNet, YOLO) is faster, and focal loss plus modern label assignment closed the accuracy gap.
  • NMS removes duplicates but adds hyperparameters and latency; DETR-family models use Hungarian one-to-one matching to avoid it, and RT-DETR makes this real-time.
  • Open-vocabulary detectors (OWL, Grounding DINO, YOLO-World) answer text queries; excellent for auto-labeling, usually weaker than a trained closed-set model on niche classes.
  • Version numbers in the YOLO family are not authoritative; benchmark candidates on your data and hardware.
  1. Conceptual. Why does focal loss help dense one-stage detectors but is less necessary for Faster R-CNN’s second stage?
  2. Conceptual. Why does DETR need one-to-one matching, and what would happen to duplicates if it used one-to-many matching without NMS?
  3. Hands-on. Implement NMS in pure PyTorch and compare to torchvision.ops.nms on random boxes.
  4. Hands-on. Given a list of predictions and ground truths, compute AP at IoU 0.5 by hand (sort by score, accumulate TP/FP, integrate the PR curve), then verify with torchmetrics.
  5. Conceptual. A detector has mAP@[.5:.95] of 0.40 and AP50 of 0.70. What does the gap tell you?
Solutions (sketch)
  1. The second stage sees a pre-filtered, balanced sample of proposals (e.g., 1:3 positive:negative); one-stage sees all locations, so easy negatives overwhelm.
  2. One-to-one targets teach each object to be claimed by exactly one query; with one-to-many targets, several queries will fire on an object and duplicates must be removed by NMS.
  3. Sort by score; loop: take best, compute IoU with the rest via box_iou, drop those above the threshold.
  4. Remember to mark each GT matched at most once; use the monotone precision envelope.
  5. Detections are mostly findable but loosely localized; precision at strict IoU falls. Look at box regression quality, label noise, and resolution.
  • Girshick et al., “Rich feature hierarchies…” (R-CNN, 2014); Girshick, “Fast R-CNN” (2015); Ren et al., “Faster R-CNN” (2015).
  • Lin et al., “Feature Pyramid Networks for Object Detection” (2017); “Focal Loss for Dense Object Detection” (RetinaNet, 2017).
  • Redmon et al., “You Only Look Once” (2016) and the YOLOv2/v3 reports.
  • Tian et al., “FCOS” (2019); Zhou et al., “Objects as Points” (CenterNet, 2019).
  • Carion et al., “End-to-End Object Detection with Transformers” (DETR, 2020); Zhu et al., “Deformable DETR” (2020); Zhang et al., “DINO: DETR with Improved DeNoising Anchor Boxes” (2022).
  • Zhao et al., “DETRs Beat YOLOs on Real-time Object Detection” (RT-DETR, 2023).
  • Minderer et al., “Simple Open-Vocabulary Object Detection with Vision Transformers” (OWL-ViT, 2022); Liu et al., “Grounding DINO” (2023); Cheng et al., “YOLO-World” (2024).
  • Lin et al., “Microsoft COCO: Common Objects in Context” (2014) and the COCO evaluation documentation.