Skip to content

CNN Architectures That Mattered

What you’ll learn. The key convolutional architectures in historical order, what problem each solved, and the transferable design lessons: depth and residual connections, normalization, bottlenecks, grouped/depthwise convolutions, compound scaling, and how ConvNeXt modernized a ResNet by borrowing transformer-era training recipes. You should already know what a convolution, pooling and backprop are (CNNs and RNNs).

Most vision systems have the shape backbone → neck → head. The backbone turns an image into a hierarchy of feature maps, typically at strides 4, 8, 16, 32 relative to the input. The head does classification, detection, or segmentation. Architectures below are backbones; their feature maps are what detection and segmentation models consume (Object Detection, Segmentation).

image 224x224x3
stem -> 56x56 (stride 4)
stage1 -> 56x56, C
stage2 -> 28x28, 2C
stage3 -> 14x14, 4C
stage4 -> 7x7, 8C
global avg pool -> linear -> logits

The pattern “halve resolution, double channels” is nearly universal: compute per stage stays roughly constant while the receptive field grows.

Eight learned layers (5 conv, 3 FC), about 60M parameters, mostly in the fully connected layers. Contributions that mattered:

  • ReLU instead of tanh/sigmoid: non-saturating, trains several times faster.
  • Dropout in the FC layers and heavy data augmentation (random crops, flips, PCA color jitter) against overfitting.
  • GPU training (two GPUs, split channels, the origin of “groups”).
  • Large early filters (11×1111\times11, stride 4).

Top-5 ImageNet error was about 16% against about 26% for the runner-up. The lesson: with enough data and compute, learned features beat engineered ones (Image Fundamentals).

VGG used only 3×33\times3 convolutions, stacked. Two stacked 3×33\times3 layers have the same receptive field as one 5×55\times5 (5×55\times5 coverage) but with 2⋅9C2=18C22\cdot 9C^2 = 18C^2 weights instead of 25C225C^2, plus an extra nonlinearity. Three stacked cover 7×77\times7 with 27C227C^2 versus 49C249C^2.

VGG-16 has about 138M parameters, mostly in the FC head, and is expensive (about 15 GFLOPs at 224 px). It is conceptually clean, which is why VGG features were long used for perceptual losses and style transfer. The lesson: prefer small kernels and depth.

Inception / GoogLeNet (2014): multi-scale and bottlenecks

Section titled “Inception / GoogLeNet (2014): multi-scale and bottlenecks”

Inception modules run 1×11\times1, 3×33\times3, 5×55\times5 and pooling branches in parallel and concatenate them, letting the network choose scale. To keep cost under control, 1×11\times1 convolutions reduce channels before the expensive kernels. GoogLeNet reached strong accuracy with only about 6.8M parameters. Later versions factorized 5×55\times5 into two 3×33\times3 and n×nn\times n into 1×n1\times n followed by n×1n\times1, and added batch normalization.

Lessons: the 1×11\times1 convolution as a channel mixer/bottleneck; factorization saves compute; global average pooling can replace huge FC layers.

For activations xx over a mini-batch, BN computes per-channel

x^=x−μBσB2+ϵ,y=γx^+β.\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2+\epsilon}},\qquad y = \gamma\hat{x} + \beta.

It lets you use much larger learning rates, makes training less sensitive to initialization, and acts as a mild regularizer. At inference it uses running statistics, so it folds into the preceding convolution (a free speedup, see Efficient & Edge Vision). The standard explanation (reducing “internal covariate shift”) is disputed; later work argues BN smooths the loss landscape. Its weaknesses: depends on batch size (bad for batch of 1-2, as in detection/segmentation with big images) and behaves differently in train/eval. Alternatives: GroupNorm, LayerNorm (the transformer standard), and these matter in ConvNeXt below.

Plain networks deeper than about 20 layers got worse training error, not just test error: an optimization problem, not overfitting. He et al. proposed learning a residual:

y=F(x;W)+x.\mathbf{y} = \mathcal{F}(\mathbf{x}; W) + \mathbf{x}.

If the identity is optimal, the block just has to drive F→0\mathcal F \to 0, easier than fitting an identity with a stack of nonlinear layers. The gradient perspective is clearer: unrolling LL blocks, xL=x0+∑iFi(xi)\mathbf{x}_L = \mathbf{x}_0 + \sum_{i} \mathcal{F}_i(\mathbf{x}_i), so

∂L∂xl=∂L∂xL(1+∂∂xl∑i=lL−1Fi),\frac{\partial \mathcal{L}}{\partial \mathbf{x}_l} = \frac{\partial \mathcal{L}}{\partial \mathbf{x}_L}\left(1 + \frac{\partial}{\partial \mathbf{x}_l}\sum_{i=l}^{L-1}\mathcal{F}_i\right),

and the “1” term gives an unobstructed gradient path from the loss to every layer. ResNet-50 uses bottleneck blocks (1×11\times1 reduce, 3×33\times3, 1×11\times1 expand by 4x), about 25.6M parameters and about 4.1 GFLOPs at 224 px; ResNet-152 won ILSVRC 2015 with about 3.6% top-5 error, below the commonly quoted human estimate of about 5%.

import torch.nn as nn
class Bottleneck(nn.Module):
def __init__(self, cin, cmid, stride=1):
super().__init__()
cout = cmid * 4
self.f = nn.Sequential(
nn.Conv2d(cin, cmid, 1, bias=False), nn.BatchNorm2d(cmid), nn.ReLU(inplace=True),
nn.Conv2d(cmid, cmid, 3, stride, 1, bias=False), nn.BatchNorm2d(cmid), nn.ReLU(inplace=True),
nn.Conv2d(cmid, cout, 1, bias=False), nn.BatchNorm2d(cout),
)
self.skip = nn.Identity() if (cin == cout and stride == 1) else nn.Sequential(
nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
self.act = nn.ReLU(inplace=True)
def forward(self, x):
return self.act(self.f(x) + self.skip(x))

Skip connections are the single most influential idea in this lecture: transformers, diffusion U-Nets and almost every deep model use them. Follow-ups: ResNeXt (grouped convolutions, “cardinality”), DenseNet (concatenate all earlier features), Wide ResNet (width can beat depth), pre-activation ResNet (BN-ReLU-conv ordering gives cleaner identity paths), and Squeeze-and-Excitation (channel attention).

Efficient designs: depthwise separable convolutions

Section titled “Efficient designs: depthwise separable convolutions”

A standard k×kk\times k convolution with CinC_\text{in} inputs and CoutC_\text{out} outputs costs k2CinCoutk^2 C_\text{in} C_\text{out} multiply-adds per output pixel. A depthwise separable convolution splits it into a per-channel k×kk\times k depthwise convolution (k2Cink^2 C_\text{in}) followed by a 1×11\times1 pointwise one (CinCoutC_\text{in}C_\text{out}). The ratio is

k2Cin+CinCoutk2CinCout=1Cout+1k2,\frac{k^2 C_\text{in} + C_\text{in}C_\text{out}}{k^2 C_\text{in}C_\text{out}} = \frac{1}{C_\text{out}} + \frac{1}{k^2},

about 8-9x cheaper for k=3k=3 and wide layers. Xception and MobileNet v1 (2017) built whole networks from it. MobileNetV2 introduced the inverted residual: expand channels with 1×11\times1 (by 6x), depthwise 3×33\times3, project back down with a linear 1×11\times1 (no ReLU on the narrow layer, to avoid destroying information), with the skip connection between the narrow ends. MobileNetV3 added neural architecture search and hard-swish. See Efficient & Edge Vision for deployment concerns.

Prior work scaled one axis: depth (ResNet-152), width (Wide ResNet) or resolution. Tan and Le argued all three should scale together. With coefficient ϕ\phi:

d=αϕ,w=βϕ,r=γϕ,αβ2γ2≈2,d = \alpha^\phi,\quad w = \beta^\phi,\quad r = \gamma^\phi,\qquad \alpha\beta^2\gamma^2 \approx 2,

so FLOPs grow roughly 2ϕ2^\phi (FLOPs scale linearly with depth and quadratically with width and resolution). They found a base network (B0) by NAS using MBConv blocks (MobileNetV2-style inverted residuals plus Squeeze-and-Excitation) and swish activations, then scaled to B1-B7. EfficientNet-B7 reached the best ImageNet accuracy of its time at a fraction of the parameters of competitors (about 66M versus GPipe’s 557M). EfficientNetV2 added Fused-MBConv (regular conv in early stages, which is faster on accelerators) and progressive learning (small images with weak augmentation early, larger with stronger later).

Radosavovic et al. (RegNet, 2020) studied populations of networks rather than single designs and found a simple quantization: widths increase roughly linearly with depth. A takeaway: a well-parametrized design space with good defaults beats one-off NAS-found architectures.

After Vision Transformers and Swin appeared to outperform ResNets, Liu et al. asked how much of that gap was the architecture and how much was the training recipe. They took ResNet-50 and applied changes step by step:

  1. Training recipe: AdamW, 300 epochs, heavy augmentation (Mixup, CutMix, RandAugment, random erasing), label smoothing, stochastic depth, EMA. This alone gave a large improvement (ResNet-50 gains roughly 2-3 points, from about 76% to about 78-79% top-1).
  2. Macro design: stage ratio changed to (3,3,9,3) like Swin; “patchify” stem as a 4×44\times4 stride-4 convolution.
  3. Depthwise convolution (like attention’s per-channel spatial mixing) with larger width.
  4. Inverted bottleneck (the MLP-block shape in transformers: expand 4x then project).
  5. Large kernels: 7×77\times7 depthwise.
  6. Micro design: GELU instead of ReLU, fewer activations and norms per block, LayerNorm instead of BatchNorm, separate downsampling layers.

The result matched or beat Swin at similar FLOPs. ConvNeXt V2 added a masked-autoencoder-style pretraining and Global Response Normalization. Follow-up work pushed kernels further (RepLKNet, SLaK, up to 31×3131\times31 and beyond), suggesting large effective receptive fields matter.

# using timm: one line to get a ConvNeXt, pretrained
import timm, torch
m = timm.create_model("convnext_tiny", pretrained=True, num_classes=10)
x = torch.randn(2, 3, 224, 224)
print(m(x).shape) # torch.Size([2, 10])
  1. Residual connections make depth trainable. Everything modern has them.
  2. Normalization is not optional; its form (BN vs LN vs GN) depends on batch size and architecture family.
  3. Bottlenecks and channel mixing (1×11\times1 / MLP) carry most of the compute and parameters; spatial mixing (conv or attention) is comparatively cheap.
  4. Hierarchy (multi-scale features) is essential for dense tasks; this is why Swin and hierarchical ViTs re-introduced pyramid structure into transformers.
  5. Training recipe often matters as much as architecture. Many “new architecture” gains in 2018-2021 papers shrank when baselines were retrained with modern recipes (“ResNet strikes back”, timm’s recipes). Compare architectures only under equal training budget and recipe.
  6. Scale up data and model together. Under large-scale pretraining (billions of images, see Self-Supervised & Foundation Vision Models), the gap between CNNs and ViTs becomes small; CNN inductive biases help most in small-data regimes.
  7. Measure the real metric: latency, memory, and throughput on target hardware, not parameter counts.

For classification/backbones in production: ConvNeXt and ResNet-class networks remain common for their simplicity, well-supported export paths and efficient inference; hybrids and ViTs dominate when pretrained foundation weights are available (DINOv2/v3, SigLIP, see lecture 4 and 7). On mobile and embedded, MobileNet/EfficientNet/RepVGG-style reparameterizable networks and tiny ViT hybrids are common. Specific rankings change monthly; check the timm leaderboards and benchmark on your own task and hardware before choosing.

  • Small stacked kernels (VGG), 1×11\times1 bottlenecks (Inception/ResNet) and residual connections (ResNet) are the durable ideas of the 2012-2016 era.
  • Depthwise separable convolutions cut cost by roughly 1/Cout+1/k21/C_\text{out} + 1/k^2, but are memory-bound; FLOPs do not equal latency.
  • EfficientNet scales depth, width and resolution jointly; ConvNeXt shows that much of the ViT advantage was training recipe plus macro/micro design choices.
  • Compare architectures only under identical training recipes and measure on the deployment target.
  1. Conceptual. Show that two stacked 3×33\times3 convolutions have a 5×55\times5 receptive field. How many weights does each option use for CC channels in and out?
  2. Conceptual. Why does the identity path in a residual block help with vanishing gradients? Use the derivative expression above.
  3. Hands-on. Count parameters and FLOPs of resnet50, mobilenet_v3_large, efficientnet_b0 and convnext_tiny (use torchvision plus fvcore or ptflops), then time each on your GPU/CPU at batch sizes 1 and 32. Does ordering by FLOPs match ordering by latency?
  4. Hands-on. Implement an inverted residual block and verify that replacing a standard 3×33\times3 conv with depthwise separable reduces parameters by the predicted ratio.
  5. Conceptual. Why does MobileNetV2 omit the ReLU after the projection 1×11\times1?
Solutions (sketch)
  1. Each 3×33\times3 adds 2 to the field: 3→53 \to 5. Weights 2⋅9C2=18C22\cdot9C^2=18C^2 versus 25C225C^2 (about 28% fewer).
  2. The “1” term in the derivative passes the upstream gradient unchanged to earlier layers regardless of F\mathcal{F}.
  3. Typically no: depthwise-heavy networks have lower FLOPs but less-than-proportional latency gains on GPUs; CPUs often track FLOPs more closely.
  4. Ratio for k=3k=3, Cout=128C_\text{out}=128: 1/128+1/9≈0.1191/128+1/9\approx0.119.
  5. In a low-dimensional (narrow) space, ReLU would zero many activations and destroy information that cannot be recovered; the manifold of interest is better preserved linearly.
  • Krizhevsky et al., “ImageNet Classification with Deep CNNs” (2012).
  • Simonyan and Zisserman, “Very Deep Convolutional Networks” (VGG, 2014).
  • Szegedy et al., “Going Deeper with Convolutions” (2014).
  • Ioffe and Szegedy, “Batch Normalization” (2015).
  • He et al., “Deep Residual Learning for Image Recognition” (2015), and “Identity Mappings in Deep Residual Networks” (2016).
  • Howard et al., “MobileNets” (2017); Sandler et al., “MobileNetV2” (2018).
  • Tan and Le, “EfficientNet: Rethinking Model Scaling” (2019).
  • Radosavovic et al., “Designing Network Design Spaces” (2020).
  • Liu et al., “A ConvNet for the 2020s” (ConvNeXt, 2022).
  • Wightman et al., “ResNet strikes back” (2021).