Vision Transformers
What you’ll learn. The ViT architecture end to end (patch embedding, position embeddings, class token), why it needed huge data and how DeiT removed that requirement, why hierarchical designs (Swin, PVT, and friends) matter for dense prediction, how positional information and resolution changes are handled, and a principled comparison of ViT and CNN inductive biases. Assumes the transformer basics from Attention and Transformers.
From pixels to tokens
Section titled “From pixels to tokens”A transformer consumes a sequence of vectors. Pixels as tokens are infeasible: a image has 50,176 pixels, and self-attention costs in sequence length . The ViT solution (Dosovitskiy et al., 2020, “An Image is Worth 16x16 Words”) is to cut the image into non-overlapping patches.
For patch size , a image yields patches; with and that is . Each patch (a vector of length ) is linearly projected to dimension . This projection is exactly a convolution with kernel size and stride :
import torch, torch.nn as nn
class PatchEmbed(nn.Module): def __init__(self, img=224, patch=16, in_ch=3, dim=768): super().__init__() self.proj = nn.Conv2d(in_ch, dim, kernel_size=patch, stride=patch) self.n = (img // patch) ** 2 def forward(self, x): # (B,3,H,W) x = self.proj(x) # (B,D,H/P,W/P) return x.flatten(2).transpose(1, 2) # (B,N,D)Then:
- Prepend a learnable [CLS] token.
- Add position embeddings (learned, 1D, one per position), because attention is permutation-invariant.
- Apply standard pre-norm transformer encoder blocks (multi-head self-attention plus MLP, with residuals and LayerNorm).
- Classify from the final [CLS] embedding (some later models average-pool patch tokens instead; both work).
class Block(nn.Module): def __init__(self, dim, heads, mlp_ratio=4): super().__init__() self.n1, self.n2 = nn.LayerNorm(dim), nn.LayerNorm(dim) self.attn = nn.MultiheadAttention(dim, heads, batch_first=True) self.mlp = nn.Sequential(nn.Linear(dim, dim*mlp_ratio), nn.GELU(), nn.Linear(dim*mlp_ratio, dim)) def forward(self, x): h = self.n1(x) x = x + self.attn(h, h, h, need_weights=False)[0] return x + self.mlp(self.n2(x))
class ViT(nn.Module): def __init__(self, img=224, patch=16, dim=768, depth=12, heads=12, classes=1000): super().__init__() self.pe = PatchEmbed(img, patch, 3, dim) self.cls = nn.Parameter(torch.zeros(1, 1, dim)) self.pos = nn.Parameter(torch.randn(1, self.pe.n + 1, dim) * 0.02) self.blocks = nn.Sequential(*[Block(dim, heads) for _ in range(depth)]) self.norm = nn.LayerNorm(dim); self.head = nn.Linear(dim, classes) def forward(self, x): x = self.pe(x) x = torch.cat([self.cls.expand(len(x), -1, -1), x], 1) + self.pos x = self.norm(self.blocks(x)) return self.head(x[:, 0])Standard sizes: ViT-B (12 layers, , 12 heads, about 86M parameters), ViT-L (24, 1024, 16; about 304M), ViT-H, and later ViT-g and larger (billions). Names such as ViT-B/16 mean base with patch 16.
Cost model
Section titled “Cost model”Per layer, attention costs and the MLP . For , the MLP dominates; at high resolution ( in the thousands) attention dominates, which is why windowed/linear/efficient attention and FlashAttention matter for vision. Halving patch size quadruples and roughly quadruples-to-16x the cost.
What ViT showed, and what it needed
Section titled “What ViT showed, and what it needed”Trained on ImageNet-1k alone, ViT-B underperformed comparable ResNets; trained on JFT-300M (a large proprietary dataset) or ImageNet-21k, it matched or beat the best CNNs at lower pretraining compute. The explanation: ViT has weak built-in inductive bias, so it must learn locality and translation equivariance from data. With enough data this is a feature (less restrictive); with little data it is a handicap.
DeiT: making ViTs data-efficient
Section titled “DeiT: making ViTs data-efficient”Touvron et al. (DeiT, 2020) showed a ViT can reach about 83% top-1 on ImageNet-1k alone using an aggressive training recipe:
- AdamW with a cosine schedule, long training (300 epochs), warmup.
- Strong augmentation: RandAugment, Mixup, CutMix, random erasing, repeated augmentation.
- Regularization: stochastic depth, label smoothing.
- Distillation token: an extra learnable token trained to match the hard label predictions of a CNN teacher, injecting the CNN’s inductive bias through the loss.
This recipe (also carried into ConvNeXt, see CNN Architectures) became the default for training any supervised vision model. Later recipes (DeiT III, “How to train your ViT”) refined it. Lesson: augmentation and regularization substitute for architectural bias.
Positional information and resolution
Section titled “Positional information and resolution”Learned absolute position embeddings are tied to a grid size. To fine-tune at higher resolution (common: pretrain at 224, fine-tune at 384), you keep the patch size fixed, get more tokens, and interpolate the pretrained position embeddings (bicubic, on the 2D grid). Alternatives that cope better with variable resolution:
- Relative position bias (Swin): a learned bias added to attention logits depending on offset.
- Conditional/convolutional position encodings (CPVT, Twins): a depthwise convolution inside the network injects position implicitly.
- 2D rotary embeddings (RoPE-2D): used in several recent ViTs (EVA-02, and many VLM encoders) to extrapolate to new resolutions.
- Native-resolution training (NaViT, “patch n’ pack”): pack variable-size images as variable-length sequences without resizing, which matters for documents and OCR (Vision-Language Models).
Reported frequently in practice: ViTs are somewhat sensitive to resolution mismatch; test on your own input sizes.
Hierarchical vision transformers
Section titled “Hierarchical vision transformers”Plain ViT has a single-resolution (stride-16) feature map and global attention, which is awkward for detection and segmentation that need multi-scale features and high-resolution inputs. Hierarchical ViTs restore the CNN pyramid:
- PVT (Pyramid Vision Transformer): progressively merge patches and reduce key/value resolution in attention.
- Swin Transformer (Liu et al., 2021): attention within non-overlapping local windows (e.g., tokens), cost linear in image size. To let windows communicate, alternate layers use shifted windows (shift by half a window), implemented by a cyclic shift with attention masks. Patch merging between stages (concatenate neighbors, linear reduce) gives strides 4, 8, 16, 32. Swin became the default detection/segmentation backbone for several years.
- MViT, CvT, CoAtNet, MaxViT: mix convolutions and attention (convolutions early for local patterns and efficiency, attention later).
- Hiera and other simplifications: a hierarchical backbone with plain operations, relying on masked pretraining rather than architectural tricks (used in SAM 2, see Segmentation).
Window attention complexity: with window size , instead of .
Plain ViT: patch16 -> [global attn x L] -> one stride-16 mapSwin-style: patch4 -> [win-attn] -> merge -> [win-attn] -> merge -> ... strides 4, 8, 16, 32 (feature pyramid for FPN / UperNet)For plain ViT in dense tasks, ViTDet showed you can build a simple feature pyramid from the single-scale output and use windowed attention with a few global blocks, so a plain ViT pretrained with MAE can be a strong detection backbone, reducing the need for a bespoke hierarchical design.
ViT versus CNN: inductive bias
Section titled “ViT versus CNN: inductive bias”Inductive bias is the set of assumptions an architecture bakes in.
| Property | CNN | ViT |
|---|---|---|
| Locality | hard-wired (small kernels) | learned (attention patterns) |
| Translation equivariance | built-in (weight sharing) | approximate, via position embeddings/data |
| Hierarchy | built-in via stride | none (plain) / added (Swin) |
| Receptive field | grows with depth | global from layer 1 |
| Data efficiency | better when data small | needs more data or strong regularization |
| Scaling with data/params | saturates earlier | scales well; often still improving |
| Shape vs texture | texture-biased in standard training | more shape-biased in some studies |
Empirical findings worth knowing (with caveats, since they depend on recipe):
- Early ViT layers learn a mix of local and global heads; lower layers show local attention, and learned position embeddings come to encode 2D distance.
- ViTs are often reported to be more robust to certain corruptions and occlusion than ResNets trained in the standard way, but the gap largely closes for CNNs trained with the same modern recipe and data scale. Treat any robustness claim as recipe-dependent (Data, Augmentation & Evaluation).
- ViTs have known artifacts: high-norm outlier tokens appear in large supervised and self-supervised ViTs, acting as “registers” for global information and corrupting attention maps. “Vision Transformers Need Registers” (Darcet et al., 2023) added a few extra learnable tokens to fix this, and register-style tokens appear in DINOv2-with-registers and later models.
- Per-token cost and memory grow with resolution much more steeply than for CNNs; efficient attention kernels (FlashAttention, windowing, token pruning/merging as in ToMe) are key to practical deployment (Efficient & Edge Vision).
ViTs in the modern stack
Section titled “ViTs in the modern stack”ViT encoders are now the default visual front-end for multimodal models: CLIP/SigLIP ViTs feed Vision-Language Models; DINOv2/v3 ViTs provide dense features for depth, segmentation and robotics (Self-Supervised & Foundation Vision Models); Diffusion Transformers (DiT) operate on latent patches (Diffusion & Image Generation); video models extend patches to space-time “tubelets” (Video, 3D & Spatial Vision). Specific best-in-class names at any moment (EVA, SigLIP 2, DINOv3, InternViT, and so on) change rapidly; verify against current leaderboards.
Practical advice
Section titled “Practical advice”- Fine-tune pretrained checkpoints (timm, HuggingFace) rather than training from scratch; from-scratch ViT on a small dataset usually underperforms a ResNet.
- Use layer-wise learning-rate decay when fine-tuning (lower layers move less), a smaller LR than for CNNs, AdamW, and warmup.
- Match preprocessing (mean/std, resize/crop) to the checkpoint’s config.
- When changing resolution, interpolate position embeddings; check results at the target size.
- For dense tasks, prefer a hierarchical backbone or ViTDet-style adapters.
import timm, torchm = timm.create_model("vit_base_patch16_224", pretrained=True)cfg = timm.data.resolve_data_config({}, model=m)tfm = timm.data.create_transform(**cfg) # correct mean/std/resize for this checkpointKey takeaways
Section titled “Key takeaways”- ViT = patchify (strided conv) + position embeddings + [CLS] + standard transformer; attention cost is quadratic in the number of patches.
- Weak inductive bias means ViTs need lots of data (JFT/IN-21k, self-supervision) or a strong recipe plus distillation (DeiT).
- Hierarchical windowed designs (Swin and relatives) give linear-cost attention and multi-scale features for dense prediction; plain ViT with simple adapters (ViTDet) also works.
- Resolution changes require position-embedding interpolation or relative/rotary encodings; native-resolution packing exists for documents.
- At scale the CNN/ViT gap is mostly data and recipe; transformers win on ecosystem and multimodal compatibility.
Exercises
Section titled “Exercises”- Conceptual. How many tokens does ViT-B/16 see at input? How does the attention matrix size compare with ?
- Conceptual. Why can a patch embedding be implemented as a convolution, and what would a patch size of 1 imply for cost?
- Hands-on. Load a pretrained
vit_base_patch16_224from timm, evaluate on 1,000 validation images at 224 and at 384 without interpolating embeddings (resize the embedding by hand vs. letting timm do it) and compare accuracy. - Hands-on. Visualize [CLS]-to-patch attention in the last layer for several images. Do the maps highlight objects? Try a DINO-style checkpoint and compare.
- Conceptual. Swin shifts windows between layers. What would break if windows were never shifted?
Solutions (sketch)
- , so (+1 CLS). Attention matrix grows from to entries per head: about 8.6x.
- Linear projection of each flattened non-overlapping patch equals conv with kernel=stride=. gives tokens: 50k tokens at 224 and a -entry attention matrix per head.
- Without proper interpolation accuracy typically drops noticeably; with bicubic interpolation of the pos-embed grid the drop is small.
- Supervised ViTs often give noisy maps; DINO ViTs produce cleaner object-segmentation-like maps.
- Tokens in different windows could never exchange information within a stage; receptive field would be limited to a window until patch merging, hurting global context.
Further reading
Section titled “Further reading”- Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” (2020).
- Touvron et al., “Training data-efficient image transformers & distillation through attention” (DeiT, 2020).
- Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows” (2021).
- Wang et al., “Pyramid Vision Transformer” (2021).
- Li et al., “Exploring Plain Vision Transformer Backbones for Object Detection” (ViTDet, 2022).
- Steiner et al., “How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers” (2021).
- Darcet et al., “Vision Transformers Need Registers” (2023).
- Dehghani et al., “Patch n’ Pack: NaViT” (2023).