Skip to content

Vision Transformers

What you’ll learn. The ViT architecture end to end (patch embedding, position embeddings, class token), why it needed huge data and how DeiT removed that requirement, why hierarchical designs (Swin, PVT, and friends) matter for dense prediction, how positional information and resolution changes are handled, and a principled comparison of ViT and CNN inductive biases. Assumes the transformer basics from Attention and Transformers.

A transformer consumes a sequence of vectors. Pixels as tokens are infeasible: a 224×224224\times224 image has 50,176 pixels, and self-attention costs O(N2)O(N^2) in sequence length NN. The ViT solution (Dosovitskiy et al., 2020, “An Image is Worth 16x16 Words”) is to cut the image into non-overlapping patches.

For patch size PP, a H×WH\times W image yields N=HW/P2N = HW/P^2 patches; with P=16P=16 and 2242224^2 that is N=196N=196. Each patch (a vector of length P2CP^2 C) is linearly projected to dimension DD. This projection is exactly a convolution with kernel size PP and stride PP:

import torch, torch.nn as nn
class PatchEmbed(nn.Module):
def __init__(self, img=224, patch=16, in_ch=3, dim=768):
super().__init__()
self.proj = nn.Conv2d(in_ch, dim, kernel_size=patch, stride=patch)
self.n = (img // patch) ** 2
def forward(self, x): # (B,3,H,W)
x = self.proj(x) # (B,D,H/P,W/P)
return x.flatten(2).transpose(1, 2) # (B,N,D)

Then:

  1. Prepend a learnable [CLS] token.
  2. Add position embeddings (learned, 1D, one per position), because attention is permutation-invariant.
  3. Apply LL standard pre-norm transformer encoder blocks (multi-head self-attention plus MLP, with residuals and LayerNorm).
  4. Classify from the final [CLS] embedding (some later models average-pool patch tokens instead; both work).
class Block(nn.Module):
def __init__(self, dim, heads, mlp_ratio=4):
super().__init__()
self.n1, self.n2 = nn.LayerNorm(dim), nn.LayerNorm(dim)
self.attn = nn.MultiheadAttention(dim, heads, batch_first=True)
self.mlp = nn.Sequential(nn.Linear(dim, dim*mlp_ratio), nn.GELU(), nn.Linear(dim*mlp_ratio, dim))
def forward(self, x):
h = self.n1(x)
x = x + self.attn(h, h, h, need_weights=False)[0]
return x + self.mlp(self.n2(x))
class ViT(nn.Module):
def __init__(self, img=224, patch=16, dim=768, depth=12, heads=12, classes=1000):
super().__init__()
self.pe = PatchEmbed(img, patch, 3, dim)
self.cls = nn.Parameter(torch.zeros(1, 1, dim))
self.pos = nn.Parameter(torch.randn(1, self.pe.n + 1, dim) * 0.02)
self.blocks = nn.Sequential(*[Block(dim, heads) for _ in range(depth)])
self.norm = nn.LayerNorm(dim); self.head = nn.Linear(dim, classes)
def forward(self, x):
x = self.pe(x)
x = torch.cat([self.cls.expand(len(x), -1, -1), x], 1) + self.pos
x = self.norm(self.blocks(x))
return self.head(x[:, 0])

Standard sizes: ViT-B (12 layers, D=768D=768, 12 heads, about 86M parameters), ViT-L (24, 1024, 16; about 304M), ViT-H, and later ViT-g and larger (billions). Names such as ViT-B/16 mean base with patch 16.

Per layer, attention costs O(N2D)O(N^2 D) and the MLP O(ND2)O(N D^2). For N=196N=196, D=768D=768 the MLP dominates; at high resolution (NN in the thousands) attention dominates, which is why windowed/linear/efficient attention and FlashAttention matter for vision. Halving patch size quadruples NN and roughly quadruples-to-16x the cost.

Trained on ImageNet-1k alone, ViT-B underperformed comparable ResNets; trained on JFT-300M (a large proprietary dataset) or ImageNet-21k, it matched or beat the best CNNs at lower pretraining compute. The explanation: ViT has weak built-in inductive bias, so it must learn locality and translation equivariance from data. With enough data this is a feature (less restrictive); with little data it is a handicap.

Touvron et al. (DeiT, 2020) showed a ViT can reach about 83% top-1 on ImageNet-1k alone using an aggressive training recipe:

  • AdamW with a cosine schedule, long training (300 epochs), warmup.
  • Strong augmentation: RandAugment, Mixup, CutMix, random erasing, repeated augmentation.
  • Regularization: stochastic depth, label smoothing.
  • Distillation token: an extra learnable token trained to match the hard label predictions of a CNN teacher, injecting the CNN’s inductive bias through the loss.

This recipe (also carried into ConvNeXt, see CNN Architectures) became the default for training any supervised vision model. Later recipes (DeiT III, “How to train your ViT”) refined it. Lesson: augmentation and regularization substitute for architectural bias.

Learned absolute position embeddings are tied to a grid size. To fine-tune at higher resolution (common: pretrain at 224, fine-tune at 384), you keep the patch size fixed, get more tokens, and interpolate the pretrained position embeddings (bicubic, on the 2D grid). Alternatives that cope better with variable resolution:

  • Relative position bias (Swin): a learned bias added to attention logits depending on offset.
  • Conditional/convolutional position encodings (CPVT, Twins): a depthwise convolution inside the network injects position implicitly.
  • 2D rotary embeddings (RoPE-2D): used in several recent ViTs (EVA-02, and many VLM encoders) to extrapolate to new resolutions.
  • Native-resolution training (NaViT, “patch n’ pack”): pack variable-size images as variable-length sequences without resizing, which matters for documents and OCR (Vision-Language Models).

Reported frequently in practice: ViTs are somewhat sensitive to resolution mismatch; test on your own input sizes.

Plain ViT has a single-resolution (stride-16) feature map and global attention, which is awkward for detection and segmentation that need multi-scale features and high-resolution inputs. Hierarchical ViTs restore the CNN pyramid:

  • PVT (Pyramid Vision Transformer): progressively merge patches and reduce key/value resolution in attention.
  • Swin Transformer (Liu et al., 2021): attention within non-overlapping local windows (e.g., 7×77\times7 tokens), cost linear in image size. To let windows communicate, alternate layers use shifted windows (shift by half a window), implemented by a cyclic shift with attention masks. Patch merging between stages (concatenate 2×22\times2 neighbors, linear reduce) gives strides 4, 8, 16, 32. Swin became the default detection/segmentation backbone for several years.
  • MViT, CvT, CoAtNet, MaxViT: mix convolutions and attention (convolutions early for local patterns and efficiency, attention later).
  • Hiera and other simplifications: a hierarchical backbone with plain operations, relying on masked pretraining rather than architectural tricks (used in SAM 2, see Segmentation).

Window attention complexity: with window size MM, O(NM2D)O(N M^2 D) instead of O(N2D)O(N^2 D).

Plain ViT: patch16 -> [global attn x L] -> one stride-16 map
Swin-style: patch4 -> [win-attn] -> merge -> [win-attn] -> merge -> ...
strides 4, 8, 16, 32 (feature pyramid for FPN / UperNet)

For plain ViT in dense tasks, ViTDet showed you can build a simple feature pyramid from the single-scale output and use windowed attention with a few global blocks, so a plain ViT pretrained with MAE can be a strong detection backbone, reducing the need for a bespoke hierarchical design.

Inductive bias is the set of assumptions an architecture bakes in.

Property CNN ViT
Locality hard-wired (small kernels) learned (attention patterns)
Translation equivariance built-in (weight sharing) approximate, via position embeddings/data
Hierarchy built-in via stride none (plain) / added (Swin)
Receptive field grows with depth global from layer 1
Data efficiency better when data small needs more data or strong regularization
Scaling with data/params saturates earlier scales well; often still improving
Shape vs texture texture-biased in standard training more shape-biased in some studies

Empirical findings worth knowing (with caveats, since they depend on recipe):

  • Early ViT layers learn a mix of local and global heads; lower layers show local attention, and learned position embeddings come to encode 2D distance.
  • ViTs are often reported to be more robust to certain corruptions and occlusion than ResNets trained in the standard way, but the gap largely closes for CNNs trained with the same modern recipe and data scale. Treat any robustness claim as recipe-dependent (Data, Augmentation & Evaluation).
  • ViTs have known artifacts: high-norm outlier tokens appear in large supervised and self-supervised ViTs, acting as “registers” for global information and corrupting attention maps. “Vision Transformers Need Registers” (Darcet et al., 2023) added a few extra learnable tokens to fix this, and register-style tokens appear in DINOv2-with-registers and later models.
  • Per-token cost and memory grow with resolution much more steeply than for CNNs; efficient attention kernels (FlashAttention, windowing, token pruning/merging as in ToMe) are key to practical deployment (Efficient & Edge Vision).

ViT encoders are now the default visual front-end for multimodal models: CLIP/SigLIP ViTs feed Vision-Language Models; DINOv2/v3 ViTs provide dense features for depth, segmentation and robotics (Self-Supervised & Foundation Vision Models); Diffusion Transformers (DiT) operate on latent patches (Diffusion & Image Generation); video models extend patches to space-time “tubelets” (Video, 3D & Spatial Vision). Specific best-in-class names at any moment (EVA, SigLIP 2, DINOv3, InternViT, and so on) change rapidly; verify against current leaderboards.

  • Fine-tune pretrained checkpoints (timm, HuggingFace) rather than training from scratch; from-scratch ViT on a small dataset usually underperforms a ResNet.
  • Use layer-wise learning-rate decay when fine-tuning (lower layers move less), a smaller LR than for CNNs, AdamW, and warmup.
  • Match preprocessing (mean/std, resize/crop) to the checkpoint’s config.
  • When changing resolution, interpolate position embeddings; check results at the target size.
  • For dense tasks, prefer a hierarchical backbone or ViTDet-style adapters.
import timm, torch
m = timm.create_model("vit_base_patch16_224", pretrained=True)
cfg = timm.data.resolve_data_config({}, model=m)
tfm = timm.data.create_transform(**cfg) # correct mean/std/resize for this checkpoint
  • ViT = patchify (strided conv) + position embeddings + [CLS] + standard transformer; attention cost is quadratic in the number of patches.
  • Weak inductive bias means ViTs need lots of data (JFT/IN-21k, self-supervision) or a strong recipe plus distillation (DeiT).
  • Hierarchical windowed designs (Swin and relatives) give linear-cost attention and multi-scale features for dense prediction; plain ViT with simple adapters (ViTDet) also works.
  • Resolution changes require position-embedding interpolation or relative/rotary encodings; native-resolution packing exists for documents.
  • At scale the CNN/ViT gap is mostly data and recipe; transformers win on ecosystem and multimodal compatibility.
  1. Conceptual. How many tokens does ViT-B/16 see at 3842384^2 input? How does the attention matrix size compare with 2242224^2?
  2. Conceptual. Why can a patch embedding be implemented as a convolution, and what would a patch size of 1 imply for cost?
  3. Hands-on. Load a pretrained vit_base_patch16_224 from timm, evaluate on 1,000 validation images at 224 and at 384 without interpolating embeddings (resize the embedding by hand vs. letting timm do it) and compare accuracy.
  4. Hands-on. Visualize [CLS]-to-patch attention in the last layer for several images. Do the maps highlight objects? Try a DINO-style checkpoint and compare.
  5. Conceptual. Swin shifts windows between layers. What would break if windows were never shifted?
Solutions (sketch)
  1. 384/16=24384/16 = 24, so N=576N=576 (+1 CLS). Attention matrix grows from 1962≈3.8×104196^2\approx 3.8\times10^4 to 5762≈3.3×105576^2\approx 3.3\times10^5 entries per head: about 8.6x.
  2. Linear projection of each flattened non-overlapping patch equals conv with kernel=stride=PP. P=1P=1 gives N=HWN=HW tokens: 50k tokens at 224 and a ∼2.5×109\sim2.5\times10^9-entry attention matrix per head.
  3. Without proper interpolation accuracy typically drops noticeably; with bicubic interpolation of the pos-embed grid the drop is small.
  4. Supervised ViTs often give noisy maps; DINO ViTs produce cleaner object-segmentation-like maps.
  5. Tokens in different windows could never exchange information within a stage; receptive field would be limited to a window until patch merging, hurting global context.
  • Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” (2020).
  • Touvron et al., “Training data-efficient image transformers & distillation through attention” (DeiT, 2020).
  • Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows” (2021).
  • Wang et al., “Pyramid Vision Transformer” (2021).
  • Li et al., “Exploring Plain Vision Transformer Backbones for Object Detection” (ViTDet, 2022).
  • Steiner et al., “How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers” (2021).
  • Darcet et al., “Vision Transformers Need Registers” (2023).
  • Dehghani et al., “Patch n’ Pack: NaViT” (2023).