CNNs, RNNs & Sequence Models
What you’ll learn. How convolution encodes locality and weight sharing and how to compute output shapes and parameter counts; the evolution from LeNet to ResNet and why skip connections matter; how recurrent networks process sequences with a hidden state; why vanilla RNNs forget (vanishing/exploding gradients) and how LSTM and GRU gates fix it; and where these models stand relative to transformers today.
Why not just an MLP?
Section titled “Why not just an MLP?”An MLP on a image has inputs; a single hidden layer with 1000 units needs weights and ignores image structure. Images have locality (nearby pixels correlate), translation equivariance (a cat is a cat anywhere), and hierarchical composition (edges, textures, parts, objects). CNNs build these priors into the architecture: fewer parameters, better generalization from the same data. Sequences have order and variable length; RNNs share parameters across time steps.
Convolution
Section titled “Convolution”In deep learning “convolution” is usually cross-correlation: slide a kernel over the input and take dot products. For a 2D input and kernel of size ,
with stride and padding . (True convolution flips the kernel; since kernels are learned the distinction is immaterial.)
Multiple channels: input , a layer has filters each of shape , producing feature maps. Parameters: , independent of image size (weight sharing).
Output size:
Example: , , , : . “Same” padding for odd at stride 1 uses .
import torch, torch.nn as nnconv = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3)x = torch.randn(1, 3, 224, 224)print(conv(x).shape) # (1, 64, 112, 112)print(sum(p.numel() for p in conv.parameters())) # 64*(3*49)+64 = 9472Receptive field: the input region affecting one output unit. It grows with depth: stacking two convs gives a receptive field with weights versus for one conv, plus an extra nonlinearity. This VGG-style insight (stack small kernels) persisted for years.
Special convolutions:
- : mixes channels per position; used for dimension reduction (“bottleneck”) and as the per-pixel MLP.
- Depthwise separable (MobileNet): per-channel spatial conv followed by pointwise; cost vs .
- Dilated convs expand receptive field without extra parameters or downsampling.
- Transposed convs upsample (decoders, segmentation, GANs); can create checkerboard artifacts, so resize-then-conv is a common alternative.
- Grouped convs (ResNeXt) reduce cost by splitting channels.
- 1D convs for audio/time series, 3D for video/volumetric data.
Pooling downsamples: max pooling keeps the strongest response (small translation invariance); average pooling smooths; global average pooling replaces large fully connected heads. Many modern nets use strided convs instead.
A manual implementation clarifies the mechanics:
import numpy as npdef conv2d(x, w, b=0, stride=1, pad=0): # x: (Cin, H, W); w: (Cout, Cin, k, k) x = np.pad(x, ((0, 0), (pad, pad), (pad, pad))) Cout, Cin, k, _ = w.shape H = (x.shape[1] - k) // stride + 1; W = (x.shape[2] - k) // stride + 1 out = np.zeros((Cout, H, W)) for i in range(H): for j in range(W): patch = x[:, i*stride:i*stride+k, j*stride:j*stride+k] out[:, i, j] = np.tensordot(w, patch, axes=([1, 2, 3], [0, 1, 2])) + b return outFrameworks implement this via im2col plus matrix multiplication or specialized algorithms (Winograd, FFT) in cuDNN: a convolution is a big matrix multiply in disguise. Gradients are also convolutions (the backward pass w.r.t. input is a transposed convolution with flipped kernel; w.r.t. weights is a correlation of input with output gradients).
CNN architectures
Section titled “CNN architectures”A canonical flow: [conv-norm-activation] × n → downsample → ... → global pool → linear classifier, with channels increasing as spatial size shrinks.
- LeNet-5 (1998): digit recognition, the template.
- AlexNet (2012): ReLU, dropout, GPUs, data augmentation; won ImageNet by a large margin and launched the deep learning era.
- VGG (2014): uniform stacks; simple but heavy (~138M parameters, mostly in FC layers).
- GoogLeNet/Inception (2014): multi-branch modules, bottlenecks.
- ResNet (2015): residual blocks enabling 50 to 152+ layers.
- DenseNet, EfficientNet (compound scaling of depth/width/resolution), RegNet, ConvNeXt (2022, a CNN modernized with transformer-era design choices: large kernels, LayerNorm, GELU, fewer activations).
- Vision Transformers (ViT, 2020) and hybrids now dominate large-scale pretraining, though CNNs remain strong for small data, low latency, edge devices, and dense prediction. For the vision-specific continuation see the computer vision section of this site.
Residual networks
Section titled “Residual networks”Plain very deep networks showed degradation: a 56-layer net had higher training error than a 20-layer one, an optimization problem, not overfitting. ResNet reformulates a block to learn a residual:
If identity is optimal the block just drives , which is easy. Gradient flow: , so gradient reaches early layers through the identity path even when is small. Unrolling, a ResNet behaves like an ensemble of shallower paths. When shapes change, use a strided conv on the shortcut (projection). Bottleneck blocks () keep deep ResNets (50/101/152) affordable. The same residual-stream idea is the core of every transformer; normalization and init interplay is in Training Deep Networks.
class BasicBlock(nn.Module): def __init__(self, c): super().__init__() self.f = nn.Sequential( nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c), nn.ReLU(inplace=True), nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c)) def forward(self, x): return torch.relu(x + self.f(x))Practical CNN recipe: start from an ImageNet-pretrained backbone (torchvision/timm), fine-tune with a small LR (and a larger LR for the new head), use data augmentation (random resized crop, flip, color jitter, mixup/CutMix), AdamW or SGD with momentum and cosine schedule, and standardize with ImageNet mean/std. Transfer learning beats training from scratch for almost all small and medium datasets.
Recurrent neural networks
Section titled “Recurrent neural networks”For sequence an RNN keeps a hidden state summarizing the past:
The same weights are used at every step (parameter sharing over time), so it handles any length. Output modes: many-to-one (sentiment), many-to-many aligned (tagging), sequence-to-sequence (encoder-decoder for translation), and autoregressive generation, where is fed back as the next input.
class VanillaRNN: def __init__(self, nin, nh, seed=0): rng = np.random.default_rng(seed) self.Wxh = rng.normal(0, 1/np.sqrt(nin), (nh, nin)) self.Whh = rng.normal(0, 1/np.sqrt(nh), (nh, nh)) self.b = np.zeros(nh) def forward(self, xs): h = np.zeros(self.b.shape); hs = [] for x in xs: h = np.tanh(self.Whh @ h + self.Wxh @ x + self.b); hs.append(h) return np.array(hs)Backpropagation through time and vanishing gradients
Section titled “Backpropagation through time and vanishing gradients”Training unrolls the recurrence into a deep network (depth ) with shared weights and applies backprop (BPTT). The gradient of the loss at step with respect to an early state involves
If the largest singular value of the effective Jacobian is below 1 (tanh’ also shrinks), the product decays exponentially in : vanishing gradients, so the network cannot learn dependencies spanning many steps. If it is above 1 the product explodes. Bengio, Simard, Frasconi (1994) analyzed this; Pascanu et al. (2013) formalized it. Remedies: gradient clipping for explosion; for vanishing, gated architectures, orthogonal/identity-initialized recurrent matrices, and shortcuts. Truncated BPTT limits unroll length for long sequences.
Hochreiter and Schmidhuber (1997) introduced a cell state with an additive update, the “constant error carousel,” plus gates (sigmoids in (0,1)) that control information flow:
Why it helps: (elementwise, plus indirect terms), so when the gradient passes unchanged across many steps. Initialize the forget-gate bias to about 1 so the cell remembers by default. LSTM learns what to store, forget, and expose.
Cho et al. (2014) simplified the design: two gates and no separate cell.
The update gate interpolates between keeping old state and writing new, a leaky additive path again. GRUs have fewer parameters and perform comparably to LSTMs on many tasks.
import torch.nn as nnlstm = nn.LSTM(input_size=128, hidden_size=256, num_layers=2, batch_first=True, bidirectional=True, dropout=0.2)x = torch.randn(32, 50, 128) # (batch, time, features)out, (h, c) = lstm(x) # out: (32, 50, 512)Variants: bidirectional RNNs read both directions (for tasks with full-sequence access; not for generation); stacked layers; encoder-decoder with attention (Bahdanau, 2014) removed the fixed-size bottleneck by letting the decoder look at all encoder states, the direct ancestor of self-attention.
Other sequence models
Section titled “Other sequence models”- 1D/temporal convolutions (TCN, WaveNet): causal, dilated convs give large receptive fields with parallel training; WaveNet generated raw audio.
- Transformers (2017+): replace recurrence with attention; fully parallel training across positions; direct access to any previous token; now the default for language, and increasingly for vision, audio, and time series. Cost: attention is in sequence length.
- State-space and linear-recurrent models (S4, Mamba and successors, RWKV, linear attention variants): return to recurrent computation with structured, often linear state updates that train in parallel via scans/convolutions and run inference in memory per step. They are competitive at some scales and used in hybrids with attention; whether they displace transformers at the frontier is an open and fast-moving question, and the strongest large models as of 2026 still rely primarily on attention-based architectures, often with efficiency tweaks or hybrid layers.
When to use what
Section titled “When to use what”| Data | Reasonable default |
|---|---|
| Images (any serious scale) | Pretrained ViT or modern CNN backbone; fine-tune |
| Small image datasets, edge | Efficient CNN (MobileNet/EfficientNet), pretrained |
| Text, code | Transformer (pretrained LLM) |
| Long streaming signals with strict memory | LSTM/GRU or state-space model |
| Tabular | Gradient boosting (Classical Models) |
| Short time series | Simple baselines, then TCN/transformer/boosting; strong baselines are hard to beat |
Pitfalls
Section titled “Pitfalls”- Forgetting that BatchNorm behaves differently in train/eval; frozen BN when fine-tuning with tiny batches.
- Wrong tensor layout: PyTorch conv is
(N,C,H,W); RNN default is(T,N,F)unlessbatch_first=True. - Not clipping gradients in RNNs.
- Padding variable-length sequences without masking or
pack_padded_sequence, so padding contaminates states. - Evaluating on images with different preprocessing than training.
- Upsampling artifacts from transposed convs; aliasing from strided downsampling without blur.
Key takeaways
Section titled “Key takeaways”- Convolutions share weights and exploit locality; parameter count is regardless of image size; know the output-size formula.
- Stacked small kernels grow the receptive field cheaply; convs mix channels.
- Residual connections solve the degradation problem by providing an identity gradient path.
- RNNs share weights across time; BPTT multiplies Jacobians, causing vanishing/exploding gradients.
- LSTM/GRU use additive, gated state updates so gradients can flow over long ranges.
- Attention-based models now dominate sequences, with recurrent/state-space models an active alternative for efficiency.
Exercises
Section titled “Exercises”- Computation. A layer maps to using a conv with stride 2 and padding 1. Verify the output size and count parameters and multiply-accumulates.
- Conceptual. Why does a ResNet block with at initialization help optimization at great depth?
- Hands-on. Train a small CNN on CIFAR-10 with and without residual connections at depth 20/56. Compare training loss curves.
- Hands-on. Build a character-level language model with an LSTM on a small text; sample from it with different temperatures. Compare to a vanilla RNN on a copy-task of length 50.
- Derivation. For a linear RNN , show behaves like , relating the spectral radius to vanishing/exploding behavior.
Solutions (1, 5)
- . Parameters . MACs M.
- . If then , dominated by . Magnitude decays, explodes; backward gradients involve with the same behavior.
Further reading
Section titled “Further reading”- LeCun et al., “Gradient-Based Learning Applied to Document Recognition” (1998).
- Krizhevsky, Sutskever, Hinton, “ImageNet Classification with Deep CNNs” (2012).
- He, Zhang, Ren, Sun, “Deep Residual Learning for Image Recognition” (2015).
- Dumoulin and Visin, “A guide to convolution arithmetic for deep learning” (2016).
- Hochreiter and Schmidhuber, “Long Short-Term Memory” (1997); Cho et al., “Learning Phrase Representations using RNN Encoder-Decoder” (2014).
- Pascanu, Mikolov, Bengio, “On the difficulty of training recurrent neural networks” (2013).
- Olah, “Understanding LSTM Networks” (blog, 2015).
- Liu et al., “A ConvNet for the 2020s” (2022).
- Gu and Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces” (2023).
- Zhang, Lipton, Li, Smola, Dive into Deep Learning (d2l.ai).