Self-Supervised & Foundation Vision Models
What you’ll learn. Why labels are the bottleneck and how self-supervised learning (SSL) sidesteps them; the main families (contrastive: SimCLR/MoCo; self-distillation: BYOL/DINO/DINOv2/DINOv3; masked reconstruction: MAE; joint-embedding predictive: I-JEPA/V-JEPA; language-supervised: CLIP/SigLIP); how to use the resulting features (linear probe, k-NN, fine-tuning, dense features); and how to think about “foundation” vision models honestly. The field is fast-moving; treat model names as snapshots.
Why self-supervision
Section titled “Why self-supervision”Supervised ImageNet pretraining needs 1.2M curated labels; the web has billions of unlabeled images. SSL defines a pretext task whose labels come from the data itself, forcing the network to learn features useful for downstream tasks. The measure of success is how good the representation is: linear-probe accuracy (train only a linear classifier on frozen features), k-NN accuracy, and transfer performance on detection, segmentation, depth and retrieval (Data, Augmentation & Evaluation).
Four families dominate:
Contrastive pull views of same image together, push others apart (SimCLR, MoCo)Self-distillation student predicts teacher's output on another view (BYOL, DINO, DINOv2/v3)Masked modeling hide patches, reconstruct pixels/features (MAE, BEiT, data2vec)Joint-embedding pred. predict *latent* of masked region from context (I-JEPA, V-JEPA)Language-supervised align image with caption text (CLIP, SigLIP)Contrastive learning: SimCLR and MoCo
Section titled “Contrastive learning: SimCLR and MoCo”Idea. Take an image, create two random augmented views (crop, flip, color jitter, blur). Their embeddings should be close; embeddings of different images should be far. The standard objective is InfoNCE / NT-Xent. For a batch of images, there are views. For a positive pair with normalized embeddings and temperature :
This is a -way classification problem: identify your positive among negatives. It is a lower bound on mutual information between views; low (e.g. 0.1-0.5) focuses on hard negatives.
import torch, torch.nn.functional as F
def nt_xent(z1, z2, tau=0.2): z = F.normalize(torch.cat([z1, z2]), dim=1) # (2N, d) sim = z @ z.t() / tau n = z1.size(0) sim.fill_diagonal_(float("-inf")) # exclude self-similarity targets = torch.cat([torch.arange(n, 2*n), torch.arange(0, n)]) return F.cross_entropy(sim, targets)SimCLR (Chen et al., 2020) found that (a) the composition of augmentations (crop + color jitter) is crucial: crops force “global vs. local” consistency and color jitter prevents the shortcut of matching color histograms; (b) a non-linear projection head (MLP) on top of the encoder helps, then is thrown away; (c) large batches (thousands) help because there are more negatives. That last point is expensive.
MoCo (He et al., 2019) decouples the number of negatives from batch size: a queue of past embeddings from a slowly updated momentum encoder () provides tens of thousands of consistent negatives. MoCo v3 adapted it to ViTs and reported training instabilities fixed by freezing the patch projection.
Non-contrastive: BYOL, SimSiam, and avoiding collapse
Section titled “Non-contrastive: BYOL, SimSiam, and avoiding collapse”Without negatives, the trivial solution is to output a constant (collapse). BYOL (2020) uses a student (online) network with a predictor head that regresses the output of a momentum teacher; stop-gradient plus the predictor asymmetry avoids collapse empirically. SimSiam simplifies further (no momentum). Another route is explicit regularization: Barlow Twins pushes the cross-correlation matrix of the two views’ embeddings toward the identity (decorrelating dimensions), and VICReg adds variance and covariance terms. Collapse-avoidance is the central design problem of this family.
DINO, DINOv2, DINOv3: self-distillation at scale
Section titled “DINO, DINOv2, DINOv3: self-distillation at scale”DINO (Caron et al., 2021) is self-distillation with no labels: a student ViT is trained to match the output distribution of a momentum-averaged teacher on different crops. The teacher sees two global crops; the student sees global and small local crops, so it learns “local-to-global” correspondence. Collapse is avoided by centering (subtract a running mean of teacher outputs) and sharpening (low teacher temperature). Remarkable emergent property: the self-attention of the final [CLS] token segments foreground objects without any segmentation supervision, and the features give strong k-NN classification.
DINOv2 (Oquab et al., 2023) scaled this recipe: a curated dataset (LVD-142M, built by retrieving images near curated sets from a large web pool), combined image-level DINO loss with a patch-level iBOT masked-token objective, plus KoLeo regularization to spread features, Sinkhorn-Knopp centering, high-resolution adaptation phase, and distillation from the giant model into smaller ViTs. The resulting frozen features work off-the-shelf for classification, retrieval, depth (linear head), segmentation and correspondence, which is why DINOv2 became a default “visual backbone” for robotics, 3D and depth work (Video, 3D & Spatial Vision). A follow-up added registers to fix high-norm artifact tokens (Vision Transformers).
DINOv3 (2025) scaled further (a model of about 7B parameters trained on the order of a billion or more curated images) and introduced techniques aimed at keeping dense feature quality from degrading during long training (a “Gram anchoring” regularizer on patch-feature similarity structure), with distilled smaller families, and variants for satellite imagery. I describe these from memory of the release; check the paper for exact numbers. The broad trend is solid: self-supervised ViTs at scale give state-of-the-art dense features without labels.
Masked autoencoders (MAE)
Section titled “Masked autoencoders (MAE)”Borrowing BERT’s masked-language-modeling idea (Attention and Transformers): hide a large fraction of patches and reconstruct them. MAE (He et al., 2021) design choices:
- Very high masking ratio (75%): images are spatially redundant, so low masking makes the task trivial; high masking forces semantic understanding.
- Asymmetric encoder/decoder: the ViT encoder processes only the visible 25% of patches (big speedup, about 3x or more), and a light decoder receives encoded visible tokens plus mask tokens and predicts pixels (normalized per patch) of the masked patches.
- Loss: MSE on masked patches only.
def random_masking(x, mask_ratio=0.75): # x: (B, N, D) patch embeddings B, N, D = x.shape keep = int(N * (1 - mask_ratio)) noise = torch.rand(B, N, device=x.device) ids_shuffle = noise.argsort(dim=1) ids_keep = ids_shuffle[:, :keep] x_vis = torch.gather(x, 1, ids_keep.unsqueeze(-1).expand(-1, -1, D)) return x_vis, ids_shuffle # decoder uses ids_shuffle to restore orderMAE features are not linearly separable out of the box (linear-probe is mediocre) but fine-tune extremely well and give strong detection backbones (ViTDet). Reconstruction targets matter: pixels favor low-level detail; predicting features of a teacher (as in BEiT, data2vec, or MaskFeat) yields more semantic representations. Extensions to video (VideoMAE with 90% masking, since video is even more redundant) are covered in Video, 3D & Spatial Vision.
JEPA: predict in latent space
Section titled “JEPA: predict in latent space”LeCun’s joint-embedding predictive architecture argues that predicting pixels wastes capacity on unpredictable detail. I-JEPA (Assran et al., 2023): a context encoder sees a block of an image; a predictor predicts the representations (from a momentum target encoder) of other masked blocks. No pixel decoder, no hand-crafted augmentations needed. V-JEPA and V-JEPA 2 extend this to video, with V-JEPA 2 reported to support robotic planning with a small amount of robot data. The appeal: learn world-model-like abstractions. The status in 2026: a research direction with competitive but not uniformly dominant results; DINO-style and CLIP-style remain more widely deployed.
Language-supervised: CLIP and SigLIP
Section titled “Language-supervised: CLIP and SigLIP”Another form of “free” supervision: 400M-billions of image-text pairs scraped from the web. CLIP (Radford et al., 2021) trains an image encoder and a text encoder jointly with a symmetric contrastive loss over a batch: the matched pairs are positives and the mismatched pairs are negatives:
with a learned temperature. This gives a shared embedding space enabling zero-shot classification: embed class names as prompts (“a photo of a {class}”), pick the closest. Zero-shot CLIP ViT-L matched supervised ResNet-50 on ImageNet without seeing its labels, and showed notable robustness to distribution shift (Data, Augmentation & Evaluation).
import torch, open_clipmodel, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")tok = open_clip.get_tokenizer("ViT-B-32")text = tok(["a photo of a cat", "a photo of a dog"])# image = preprocess(PIL_image).unsqueeze(0)with torch.no_grad(): # img_f = model.encode_image(image); txt_f = model.encode_text(text) # probs = (100.0 * F.normalize(img_f) @ F.normalize(txt_f).T).softmax(dim=-1) passSigLIP (Zhai et al., 2023) replaces the softmax over the whole batch with an independent sigmoid binary loss per pair: with for matches, otherwise. It needs no global normalization across the batch, so it scales to very large batch sizes with less communication, and works better at small batch sizes. SigLIP 2 (2025) added captioning, self-distillation and masked-prediction auxiliary objectives, multilingual data and variable-resolution (NaFlex) variants, improving dense and localization quality. Open reproductions (OpenCLIP/LAION, DataComp, MetaCLIP) show data curation matters as much as the loss.
Known CLIP limitations: poor at counting, spatial relations and fine-grained attributes (bag-of-words behavior), text rendering/OCR is weaker, and embeddings are global (not great for dense tasks). These motivated the generative/captioning-based pretraining used inside Vision-Language Models.
Using foundation features
Section titled “Using foundation features”| Method | What it needs | Typical use |
|---|---|---|
| k-NN on frozen features | nothing | retrieval, quick quality check |
| Linear probe | labels, minutes | cheap classification |
| Frozen backbone + small head | labels | depth, segmentation, detection heads |
| Full fine-tuning | labels, GPU | highest accuracy when data suffice |
| Parameter-efficient (LoRA, adapters, prompt) | labels, small | many tasks sharing one backbone |
import torchdino = torch.hub.load("facebookresearch/dinov2", "dinov2_vits14") # needs network on first calldino.eval()x = torch.randn(1, 3, 224, 224)with torch.no_grad(): out = dino.forward_features(x)patch_tokens = out["x_norm_patchtokens"] # (1, 256, 384) dense features, patch size 14cls = out["x_norm_clstoken"] # (1, 384) global featurePitfalls and honest caveats
Section titled “Pitfalls and honest caveats”- Augmentation is the supervision. In contrastive/self-distillation methods, the choice of augmentations defines what invariances are learned. Color invariance is wrong for fruit-ripeness inspection; rotation invariance is wrong for digit orientation; domain-specific SSL needs domain-specific augmentations.
- Collapse and shortcuts. Monitor feature variance/rank during training; an apparent low loss can be a collapsed solution.
- Benchmarks. Linear-probe ImageNet accuracy is a narrow metric; evaluate on dense tasks and your own data.
- Data provenance and bias. Web-scale models inherit biases and licensing issues; curation pipelines are part of the model.
- “Foundation model” is a claim, not a property. Always validate on your domain (medical, industrial, remote sensing) where natural-image pretraining can transfer poorly; domain-specific SSL on unlabeled in-domain data can win.
- Model names and “state of the art” claims here (DINOv3, SigLIP 2, V-JEPA 2, and so on) will date quickly; consult recent papers, HuggingFace model hubs, and timm/OpenCLIP before choosing.
Key takeaways
Section titled “Key takeaways”- SSL removes the label bottleneck by deriving supervision from the data: augmentation invariance (contrastive/self-distillation), reconstruction (MAE), latent prediction (JEPA) or text alignment (CLIP/SigLIP).
- InfoNCE treats each positive among many negatives as classification; MoCo uses a momentum queue, SimCLR large batches; SigLIP swaps softmax for per-pair sigmoid.
- BYOL/DINO avoid collapse by asymmetry, centering and sharpening, with no negatives.
- MAE’s high masking ratio and encoder-on-visible-only design makes it efficient and good for fine-tuning; DINOv2/v3 provide strongest frozen dense features; CLIP/SigLIP provide language-aligned features.
- Choose features by task: DINO-style for dense/geometric, CLIP/SigLIP for text-aligned, MAE-style for fine-tuning.
Exercises
Section titled “Exercises”- Conceptual. Why does SimCLR need large batches while MoCo doesn’t? What is the cost MoCo pays instead?
- Conceptual. Explain how centering and sharpening in DINO each prevent a different failure mode (uniform collapse vs. one-dimension dominance).
- Hands-on. Extract DINOv2-small features for a 10-class dataset (e.g., CIFAR-10 resized to 224) and compare a linear probe against k-NN ().
- Hands-on. Implement MAE random masking and verify the encoder compute scales with the number of visible tokens by timing a ViT at 25% versus 100% tokens.
- Conceptual. A CLIP model classifies “a photo of a dog on a skateboard” and “a photo of a skateboard on a dog” similarly. Why?
Solutions (sketch)
- More negatives per positive tighten the bound; in SimCLR negatives come from the batch. MoCo keeps a queue but its keys come from a lagging encoder, so the queue’s embeddings are slightly stale (mitigated by high momentum).
- Centering prevents one dimension from dominating (but alone pushes toward uniform output); sharpening (low teacher temperature) prevents the uniform collapse. Together they balance.
- DINOv2 features typically give linear-probe accuracy well above k-NN on small datasets; both are high on CIFAR-10-like data.
- Roughly linear in tokens for MLP, quadratic for attention, so speedup of 4x to more than 4x on the encoder.
- Contrastive training on noisy web captions rarely requires word order to separate negatives, so the model behaves bag-of-words-like.
Further reading
Section titled “Further reading”- Chen et al., “A Simple Framework for Contrastive Learning of Visual Representations” (SimCLR, 2020).
- He et al., “Momentum Contrast for Unsupervised Visual Representation Learning” (MoCo, 2019).
- Grill et al., “Bootstrap Your Own Latent” (BYOL, 2020).
- Caron et al., “Emerging Properties in Self-Supervised Vision Transformers” (DINO, 2021).
- Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision” (2023); the DINOv3 paper (2025).
- He et al., “Masked Autoencoders Are Scalable Vision Learners” (2021).
- Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture” (I-JEPA, 2023).
- Radford et al., “Learning Transferable Visual Models From Natural Language Supervision” (CLIP, 2021).
- Zhai et al., “Sigmoid Loss for Language Image Pre-Training” (SigLIP, 2023).
- Balestriero et al., “A Cookbook of Self-Supervised Learning” (2023).