Skip to content

Segmentation

What you’ll learn. The three segmentation task definitions and their metrics (mIoU, mask AP, PQ); fully convolutional networks and encoder-decoder designs (FCN, U-Net, DeepLab); instance segmentation with Mask R-CNN; unified mask-classification architectures (MaskFormer, Mask2Former); and promptable segmentation with SAM and SAM 2, including how they are used for labeling and in pipelines. Prior lectures: Object Detection, Vision Transformers.

Semantic every pixel gets a class; instances of a class are merged "stuff + things" by class
Instance separate mask per object instance; only countable "things" each mask: class + score
Panoptic every pixel: class AND instance id (stuff gets class only) unifies the two
  • Semantic segmentation: e.g., road, sky, person. Two adjacent people are one blob.
  • Instance segmentation: detect each object and give a pixel mask; background “stuff” (sky, grass) is ignored.
  • Panoptic segmentation (Kirillov et al., 2018): each pixel belongs to exactly one segment, “things” with instance ids, “stuff” without.
  • Promptable segmentation (SAM): given a prompt (point, box, mask, text in some variants), return the mask for that object, class-agnostic.
  • IoU per class, averaged: mIoU for semantic segmentation. Compute from a confusion matrix accumulated over the whole dataset (not per image), as IoUc=TPcTPc+FPc+FNc\mathrm{IoU}_c = \frac{TP_c}{TP_c+FP_c+FN_c}.
  • Mask AP (COCO-style) for instances: like box AP but IoU is between masks.
  • Panoptic Quality: PQ=∑(p,g)∈TPIoU(p,g)∣TP∣⏟segmentation quality (SQ)×∣TP∣∣TP∣+12∣FP∣+12∣FN∣⏟recognition quality (RQ)\mathrm{PQ} = \underbrace{\frac{\sum_{(p,g)\in TP}\mathrm{IoU}(p,g)}{|TP|}}_{\text{segmentation quality (SQ)}}\times\underbrace{\frac{|TP|}{|TP|+\tfrac12|FP|+\tfrac12|FN|}}_{\text{recognition quality (RQ)}}, where a match requires IoU >0.5>0.5 (which makes matching unique).
  • Boundary-sensitive metrics (Boundary IoU, trimap accuracy) matter for matting and thin structures, since plain IoU barely changes for boundary errors on large objects.
import numpy as np
def confusion(pred, gt, n):
k = (gt >= 0) & (gt < n) # ignore index (e.g., 255) excluded
return np.bincount(n * gt[k].astype(int) + pred[k], minlength=n * n).reshape(n, n)
def miou(conf):
tp = np.diag(conf); fp = conf.sum(0) - tp; fn = conf.sum(1) - tp
return np.nanmean(tp / (tp + fp + fn + 1e-9)), tp / (tp + fp + fn + 1e-9)

Long et al. (2015) turned classification CNNs into fully convolutional networks by replacing FC layers with convolutions so the network accepts any input size and outputs a spatial map. Because pooling shrinks resolution by 32x, coarse predictions are upsampled (transposed convolution or bilinear) and fused with higher-resolution early features (skip connections) to recover detail.

U-Net (Ronneberger et al., 2015, originally for biomedical images) is a symmetric encoder-decoder with skip connections that concatenate encoder features to the decoder at each resolution:

input -> [enc1] -> [enc2] -> [enc3] -> [bottleneck]
| | | |
+--------concat-----+--- up ---> [dec3] -> [dec2] -> [dec1] -> 1x1 conv -> mask

The encoder captures context (what), the decoder restores resolution, and skips carry fine spatial detail (where). U-Net works with very small datasets (tens of images plus heavy augmentation such as elastic deformation) and is the dominant architecture in medical imaging (nnU-Net automates its configuration and is a strong baseline that is hard to beat) and, in modified form, the denoising network in diffusion models (Diffusion & Image Generation).

import torch, torch.nn as nn
def conv_block(i, o):
return nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(True),
nn.Conv2d(o, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(True))
class TinyUNet(nn.Module):
def __init__(self, cin=3, ncls=2, c=32):
super().__init__()
self.e1, self.e2, self.b = conv_block(cin, c), conv_block(c, 2*c), conv_block(2*c, 4*c)
self.pool = nn.MaxPool2d(2)
self.u2 = nn.ConvTranspose2d(4*c, 2*c, 2, 2); self.d2 = conv_block(4*c, 2*c)
self.u1 = nn.ConvTranspose2d(2*c, c, 2, 2); self.d1 = conv_block(2*c, c)
self.out = nn.Conv2d(c, ncls, 1)
def forward(self, x):
e1 = self.e1(x); e2 = self.e2(self.pool(e1)); b = self.b(self.pool(e2))
d2 = self.d2(torch.cat([self.u2(b), e2], 1))
d1 = self.d1(torch.cat([self.u1(d2), e1], 1))
return self.out(d1)
print(TinyUNet()(torch.randn(1, 3, 64, 64)).shape) # (1, 2, 64, 64)

Input sizes should be divisible by 2depth2^{\text{depth}}, otherwise concatenations misalign.

DeepLab uses atrous (dilated) convolutions to enlarge the receptive field without reducing resolution, ASPP (parallel dilated convs at several rates) for multi-scale context, and often a CRF or decoder for boundaries. PSPNet pools at multiple grid sizes (pyramid pooling). Transformer-era versions: SETR, SegFormer (hierarchical transformer encoder plus a lightweight all-MLP decoder, efficient and widely used), and UperNet with Swin/ConvNeXt backbones.

Loss functions. Cross-entropy per pixel is standard; for class imbalance (tiny lesions) add Dice loss 1−2∣P∩G∣∣P∣+∣G∣1-\frac{2|P\cap G|}{|P|+|G|} or focal loss. Combine them (CE + Dice) as in nnU-Net.

Mask R-CNN (He et al., 2017) extends Faster R-CNN with a small FCN mask head applied to each RoI, predicting a 28×2828\times28 binary mask per class, using RoIAlign to avoid quantization misalignment. Masks are predicted per-class with a sigmoid (no competition between classes), decoupled from classification. It is a reliable baseline and remains available in torchvision and Detectron2.

Alternatives: single-stage instance methods (YOLACT: prototype masks combined with predicted coefficients; SOLO: predict masks by location; CondInst), which are faster, and the unified query-based models below. Ultralytics YOLO-seg uses the prototype-mask idea.

import torchvision
from torchvision.models.detection import maskrcnn_resnet50_fpn
model = maskrcnn_resnet50_fpn(weights="DEFAULT").eval()
# out = model([img_tensor_0_1])[0]; out["masks"]: (K,1,H,W) soft masks; threshold at 0.5

Mask classification: MaskFormer and Mask2Former

Section titled “Mask classification: MaskFormer and Mask2Former”

Per-pixel classification treats semantic and instance segmentation differently. MaskFormer (Cheng et al., 2021) reframed everything as mask classification: predict a set of NN binary masks, each with a class label. This is like DETR (Object Detection) with masks; the same model and loss handle semantic, instance and panoptic tasks by changing only inference post-processing.

Mask2Former (2022) is the best-known version:

  • Backbone + pixel decoder (multi-scale deformable attention) produce multi-scale per-pixel embeddings.
  • A transformer decoder with NN queries (e.g., 100) refines queries over feature scales.
  • Masked attention: each query’s cross-attention is restricted to the foreground region of its previous mask prediction, which accelerates convergence and localizes features.
  • Each query produces a class distribution (incl. “no object”) and a mask via dot-product with per-pixel embeddings: mi=σ(qi⊤E)m_i = \sigma(q_i^\top E).
  • Training uses Hungarian matching and losses (binary CE + Dice on masks; cross-entropy on class), computed on randomly sampled points to save memory.

It set state of the art on COCO panoptic, ADE20K semantic and others with a single architecture. OneFormer adds task-conditioned queries for one model, one training run. Many 2024-2025 systems use Mask2Former-style decoders on top of strong foundation backbones (DINOv2-family and others), and benchmarks keep changing. Open-vocabulary variants (X-Decoder, OpenSeeD, FC-CLIP, Grounded-SAM pipelines) classify masks by text similarity.

Segment Anything (SAM), Kirillov et al., 2023, trained a promptable model on a huge mask dataset (SA-1B: about 11M images and over a billion masks) built via a data engine: model-assisted annotation → semi-automatic → fully automatic mask generation, with humans correcting in earlier stages.

Architecture (three parts, designed for interactivity):

image --> heavy ViT image encoder (MAE-pretrained) --> image embedding (computed once)
prompt (points / box / rough mask / text-in-some-variants) --> prompt encoder
(image embedding, prompt tokens) --> light mask decoder (2-layer transformer) --> 3 candidate masks + IoU scores

Since the decoder is lightweight (milliseconds), once the image is embedded you can click repeatedly. A prompt can be ambiguous (a point on a shirt: shirt, person or outfit?), so SAM outputs multiple masks (whole/part/subpart) with predicted quality scores, and is trained with the minimum loss across them (ambiguity-aware).

# pip install segment-anything (or the newer sam2 repo / HF transformers)
import numpy as np
from segment_anything import sam_model_registry, SamPredictor
sam = sam_model_registry["vit_b"](checkpoint="sam_vit_b.pth")
pred = SamPredictor(sam)
pred.set_image(image_rgb_uint8) # heavy step, once per image
masks, scores, _ = pred.predict(point_coords=np.array([[320, 240]]),
point_labels=np.array([1]), # 1 = foreground click
multimask_output=True)

SAM 2 (Meta, 2024) generalizes to video: a hierarchical (Hiera) image encoder, a memory attention mechanism with a memory bank of past frame embeddings and predictions, and a streaming design so you can prompt on any frame and propagate masks through the video, with interactive corrections. It was trained on a large video dataset (SA-V) and also improves image segmentation speed and quality over SAM 1. Subsequent versions and community variants (efficient SAMs such as MobileSAM/EfficientSAM, domain-adapted medical SAMs, and SAM 3-style concept-prompt models that segment by text/exemplar concepts) appeared in 2025; details here are fast-moving, so check the current releases.

  • It is class-agnostic and needs prompts; it does not know what a mask is (“a car”). Pair it with a detector/open-vocabulary grounding model (Grounding DINO + SAM) to get text → mask.
  • Excellent boundary quality on natural images, annotation acceleration (click-to-mask), auto-mask generation (grid of point prompts, then filter by predicted IoU and stability, dedupe with NMS).
  • Weaker on domain shifts: medical (low-contrast), thin structures, camouflaged objects, remote sensing, transparent objects. Fine-tuning the decoder (or adapters/LoRA, MedSAM) usually helps.
  • Cost: the ViT-H encoder is heavy (hundreds of ms on a strong GPU); ViT-B/small/efficient variants help on edge (Efficient & Edge Vision).
  1. Define the task precisely: do you need classes (semantic), counts/separation (instance), or just “cut out this object” (promptable)?
  2. Quick baseline: SegFormer/UperNet (semantic), Mask R-CNN or YOLO-seg (instance), Mask2Former (any, higher accuracy), U-Net/nnU-Net (medical/small data).
  3. Labeling: use SAM to speed up polygons/masks. A common loop: auto-label with Grounding DINO + SAM, human review, train a specialized small model.
  4. Augmentation: apply identical geometric transforms to image and mask (use nearest-neighbor interpolation for masks); albumentations handles this.
  5. Evaluation: report mIoU per class, inspect boundary errors and rare classes; ignore-index pixels must be excluded.
  6. Deployment: mask post-processing (threshold, hole filling, polygon simplification) is often a larger source of errors than the network.
  • Semantic, instance and panoptic tasks differ in whether instances are separated and whether stuff is included; metrics are mIoU, mask AP and PQ respectively.
  • U-Net’s concatenating skips combine context and detail; it remains the default for small-data and medical tasks and underlies diffusion denoisers.
  • Mask R-CNN = Faster R-CNN + RoIAlign + mask head; Mask2Former unifies all three tasks with query-based mask classification and masked attention.
  • SAM separates a heavy image encoder from a light prompt-conditioned decoder, producing multiple ambiguity-aware masks; SAM 2 adds streaming memory for video.
  • Foundation segmenters are class-agnostic and prompt-driven; combine with detectors/VLMs for semantics, and fine-tune in specialized domains.
  1. Conceptual. Why is mIoU computed from a dataset-level confusion matrix rather than averaging per-image IoUs?
  2. Conceptual. Why does SAM output three masks per prompt, and how is it trained to handle that?
  3. Hands-on. Train the TinyUNet above on a synthetic dataset of random ellipses; report Dice and IoU. Then remove skip connections and compare boundary quality.
  4. Hands-on. Use SAM’s automatic mask generator on a photo; filter masks by predicted_iou and area, and count how many remain.
  5. Conceptual. For PQ, why is a match defined as IoU >0.5>0.5?
Solutions (sketch)
  1. Per-image averaging overweights images where a class is tiny or absent and is undefined when absent in both; dataset-level aggregation weights by pixels and is the standard definition.
  2. A click is ambiguous; three outputs cover part/object/group levels. Loss uses the minimum over outputs (or backprops only the best), plus an IoU-prediction head for ranking.
  3. Without skips, boundaries become blurry and Dice drops notably, especially for small objects.
  4. Tune pred_iou_thresh and stability_score_thresh; counts depend heavily on them.
  5. With IoU > 0.5 any ground-truth segment can match at most one predicted segment (segments don’t overlap in panoptic), so matching is unique without a bipartite solve.
  • Long, Shelhamer, Darrell, “Fully Convolutional Networks for Semantic Segmentation” (2015).
  • Ronneberger, Fischer, Brox, “U-Net” (2015); Isensee et al., “nnU-Net” (2021).
  • Chen et al., “DeepLab” series (2014-2018).
  • He et al., “Mask R-CNN” (2017); Bolya et al., “YOLACT” (2019).
  • Kirillov et al., “Panoptic Segmentation” (2018).
  • Cheng et al., “Per-Pixel Classification is Not All You Need” (MaskFormer, 2021); “Masked-attention Mask Transformer” (Mask2Former, 2022).
  • Xie et al., “SegFormer” (2021).
  • Kirillov et al., “Segment Anything” (2023); Ravi et al., “SAM 2: Segment Anything in Images and Videos” (2024).