Skip to content

CNNs, RNNs & Sequence Models

What you’ll learn. How convolution encodes locality and weight sharing and how to compute output shapes and parameter counts; the evolution from LeNet to ResNet and why skip connections matter; how recurrent networks process sequences with a hidden state; why vanilla RNNs forget (vanishing/exploding gradients) and how LSTM and GRU gates fix it; and where these models stand relative to transformers today.

An MLP on a 224×224×3224\times224\times3 image has 150,528150{,}528 inputs; a single hidden layer with 1000 units needs ∼1.5×108\sim1.5\times10^8 weights and ignores image structure. Images have locality (nearby pixels correlate), translation equivariance (a cat is a cat anywhere), and hierarchical composition (edges, textures, parts, objects). CNNs build these priors into the architecture: fewer parameters, better generalization from the same data. Sequences have order and variable length; RNNs share parameters across time steps.

In deep learning “convolution” is usually cross-correlation: slide a kernel KK over the input and take dot products. For a 2D input XX and kernel KK of size k×kk\times k,

Y[i,j]=∑u=0k−1∑v=0k−1K[u,v]  X[i⋅s+u−p,  j⋅s+v−p]+b,Y[i,j]=\sum_{u=0}^{k-1}\sum_{v=0}^{k-1}K[u,v]\;X[i\cdot s+u-p,\;j\cdot s+v-p]+b,

with stride ss and padding pp. (True convolution flips the kernel; since kernels are learned the distinction is immaterial.)

Multiple channels: input Cin×H×WC_{in}\times H\times W, a layer has CoutC_{out} filters each of shape Cin×k×kC_{in}\times k\times k, producing CoutC_{out} feature maps. Parameters: Cout(Cink2+1)C_{out}(C_{in}k^2+1), independent of image size (weight sharing).

Output size:

Hout=⌊H+2p−ks+1⌋(dilation d: k→d(k−1)+1).H_{out}=\left\lfloor\frac{H+2p-k}{s}+1\right\rfloor\quad(\text{dilation }d:\ k\to d(k-1)+1).

Example: 224224, k=7k=7, s=2s=2, p=3p=3: ⌊(224+6−7)/2+1⌋=112\lfloor(224+6-7)/2+1\rfloor=112. “Same” padding for odd kk at stride 1 uses p=(k−1)/2p=(k-1)/2.

import torch, torch.nn as nn
conv = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3)
x = torch.randn(1, 3, 224, 224)
print(conv(x).shape) # (1, 64, 112, 112)
print(sum(p.numel() for p in conv.parameters())) # 64*(3*49)+64 = 9472

Receptive field: the input region affecting one output unit. It grows with depth: stacking two 3×33\times3 convs gives a 5×55\times5 receptive field with 2⋅9C2=18C22\cdot9C^2=18C^2 weights versus 25C225C^2 for one 5×55\times5 conv, plus an extra nonlinearity. This VGG-style insight (stack small kernels) persisted for years.

Special convolutions:

  • 1×11\times1: mixes channels per position; used for dimension reduction (“bottleneck”) and as the per-pixel MLP.
  • Depthwise separable (MobileNet): per-channel spatial conv followed by 1×11\times1 pointwise; cost ≈k2C+C2\approx k^2C+C^2 vs k2C2k^2C^2.
  • Dilated convs expand receptive field without extra parameters or downsampling.
  • Transposed convs upsample (decoders, segmentation, GANs); can create checkerboard artifacts, so resize-then-conv is a common alternative.
  • Grouped convs (ResNeXt) reduce cost by splitting channels.
  • 1D convs for audio/time series, 3D for video/volumetric data.

Pooling downsamples: max pooling keeps the strongest response (small translation invariance); average pooling smooths; global average pooling replaces large fully connected heads. Many modern nets use strided convs instead.

A manual implementation clarifies the mechanics:

import numpy as np
def conv2d(x, w, b=0, stride=1, pad=0):
# x: (Cin, H, W); w: (Cout, Cin, k, k)
x = np.pad(x, ((0, 0), (pad, pad), (pad, pad)))
Cout, Cin, k, _ = w.shape
H = (x.shape[1] - k) // stride + 1; W = (x.shape[2] - k) // stride + 1
out = np.zeros((Cout, H, W))
for i in range(H):
for j in range(W):
patch = x[:, i*stride:i*stride+k, j*stride:j*stride+k]
out[:, i, j] = np.tensordot(w, patch, axes=([1, 2, 3], [0, 1, 2])) + b
return out

Frameworks implement this via im2col plus matrix multiplication or specialized algorithms (Winograd, FFT) in cuDNN: a convolution is a big matrix multiply in disguise. Gradients are also convolutions (the backward pass w.r.t. input is a transposed convolution with flipped kernel; w.r.t. weights is a correlation of input with output gradients).

A canonical flow: [conv-norm-activation] × n → downsample → ... → global pool → linear classifier, with channels increasing as spatial size shrinks.

  • LeNet-5 (1998): digit recognition, the template.
  • AlexNet (2012): ReLU, dropout, GPUs, data augmentation; won ImageNet by a large margin and launched the deep learning era.
  • VGG (2014): uniform 3×33\times3 stacks; simple but heavy (~138M parameters, mostly in FC layers).
  • GoogLeNet/Inception (2014): multi-branch modules, 1×11\times1 bottlenecks.
  • ResNet (2015): residual blocks enabling 50 to 152+ layers.
  • DenseNet, EfficientNet (compound scaling of depth/width/resolution), RegNet, ConvNeXt (2022, a CNN modernized with transformer-era design choices: large kernels, LayerNorm, GELU, fewer activations).
  • Vision Transformers (ViT, 2020) and hybrids now dominate large-scale pretraining, though CNNs remain strong for small data, low latency, edge devices, and dense prediction. For the vision-specific continuation see the computer vision section of this site.

Plain very deep networks showed degradation: a 56-layer net had higher training error than a 20-layer one, an optimization problem, not overfitting. ResNet reformulates a block to learn a residual:

y=x+F(x),F=conv→norm→ReLU→conv→norm.y=x+F(x),\qquad F=\text{conv}\to\text{norm}\to\text{ReLU}\to\text{conv}\to\text{norm}.

If identity is optimal the block just drives F→0F\to0, which is easy. Gradient flow: ∂y/∂x=I+∂F/∂x\partial y/\partial x=I+\partial F/\partial x, so gradient reaches early layers through the identity path even when ∂F/∂x\partial F/\partial x is small. Unrolling, a ResNet behaves like an ensemble of shallower paths. When shapes change, use a 1×11\times1 strided conv on the shortcut (projection). Bottleneck blocks (1×1→3×3→1×11\times1\to3\times3\to1\times1) keep deep ResNets (50/101/152) affordable. The same residual-stream idea is the core of every transformer; normalization and init interplay is in Training Deep Networks.

class BasicBlock(nn.Module):
def __init__(self, c):
super().__init__()
self.f = nn.Sequential(
nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c), nn.ReLU(inplace=True),
nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c))
def forward(self, x):
return torch.relu(x + self.f(x))

Practical CNN recipe: start from an ImageNet-pretrained backbone (torchvision/timm), fine-tune with a small LR (and a larger LR for the new head), use data augmentation (random resized crop, flip, color jitter, mixup/CutMix), AdamW or SGD with momentum and cosine schedule, and standardize with ImageNet mean/std. Transfer learning beats training from scratch for almost all small and medium datasets.

For sequence x1,…,xTx_1,\dots,x_T an RNN keeps a hidden state hth_t summarizing the past:

ht=tanh⁡(Whhht−1+Wxhxt+b),yt=Whyht.h_t=\tanh(W_{hh}h_{t-1}+W_{xh}x_t+b),\qquad y_t=W_{hy}h_t.

The same weights are used at every step (parameter sharing over time), so it handles any length. Output modes: many-to-one (sentiment), many-to-many aligned (tagging), sequence-to-sequence (encoder-decoder for translation), and autoregressive generation, where yty_t is fed back as the next input.

class VanillaRNN:
def __init__(self, nin, nh, seed=0):
rng = np.random.default_rng(seed)
self.Wxh = rng.normal(0, 1/np.sqrt(nin), (nh, nin))
self.Whh = rng.normal(0, 1/np.sqrt(nh), (nh, nh))
self.b = np.zeros(nh)
def forward(self, xs):
h = np.zeros(self.b.shape); hs = []
for x in xs:
h = np.tanh(self.Whh @ h + self.Wxh @ x + self.b); hs.append(h)
return np.array(hs)

Backpropagation through time and vanishing gradients

Section titled “Backpropagation through time and vanishing gradients”

Training unrolls the recurrence into a deep network (depth TT) with shared weights and applies backprop (BPTT). The gradient of the loss at step TT with respect to an early state hkh_k involves

∂hT∂hk=∏t=k+1Tdiag⁡(tanh⁡′(at)) Whh.\frac{\partial h_T}{\partial h_k}=\prod_{t=k+1}^{T}\operatorname{diag}\big(\tanh'(a_t)\big)\,W_{hh}.

If the largest singular value of the effective Jacobian is below 1 (tanh’ ≤1\le1 also shrinks), the product decays exponentially in T−kT-k: vanishing gradients, so the network cannot learn dependencies spanning many steps. If it is above 1 the product explodes. Bengio, Simard, Frasconi (1994) analyzed this; Pascanu et al. (2013) formalized it. Remedies: gradient clipping for explosion; for vanishing, gated architectures, orthogonal/identity-initialized recurrent matrices, and shortcuts. Truncated BPTT limits unroll length for long sequences.

Hochreiter and Schmidhuber (1997) introduced a cell state ctc_t with an additive update, the “constant error carousel,” plus gates (sigmoids in (0,1)) that control information flow:

ft=σ(Wf[ht−1,xt]+bf)forget gateit=σ(Wi[ht−1,xt]+bi)input gatec~t=tanh⁡(Wc[ht−1,xt]+bc)candidatect=ft⊙ct−1+it⊙c~tcell updateot=σ(Wo[ht−1,xt]+bo)output gateht=ot⊙tanh⁡(ct)\begin{aligned} f_t&=\sigma(W_f[h_{t-1},x_t]+b_f)&&\text{forget gate}\\ i_t&=\sigma(W_i[h_{t-1},x_t]+b_i)&&\text{input gate}\\ \tilde c_t&=\tanh(W_c[h_{t-1},x_t]+b_c)&&\text{candidate}\\ c_t&=f_t\odot c_{t-1}+i_t\odot\tilde c_t&&\text{cell update}\\ o_t&=\sigma(W_o[h_{t-1},x_t]+b_o)&&\text{output gate}\\ h_t&=o_t\odot\tanh(c_t) \end{aligned}

Why it helps: ∂ct/∂ct−1=ft\partial c_t/\partial c_{t-1}=f_t (elementwise, plus indirect terms), so when ft≈1f_t\approx1 the gradient passes unchanged across many steps. Initialize the forget-gate bias to about 1 so the cell remembers by default. LSTM learns what to store, forget, and expose.

Cho et al. (2014) simplified the design: two gates and no separate cell.

zt=σ(Wz[ht−1,xt]),rt=σ(Wr[ht−1,xt]),h~t=tanh⁡(W[rt⊙ht−1,xt]),ht=(1−zt)⊙ht−1+zt⊙h~t.\begin{aligned} z_t&=\sigma(W_z[h_{t-1},x_t]),\quad r_t=\sigma(W_r[h_{t-1},x_t]),\\ \tilde h_t&=\tanh(W[r_t\odot h_{t-1},x_t]),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t. \end{aligned}

The update gate ztz_t interpolates between keeping old state and writing new, a leaky additive path again. GRUs have fewer parameters and perform comparably to LSTMs on many tasks.

import torch.nn as nn
lstm = nn.LSTM(input_size=128, hidden_size=256, num_layers=2, batch_first=True,
bidirectional=True, dropout=0.2)
x = torch.randn(32, 50, 128) # (batch, time, features)
out, (h, c) = lstm(x) # out: (32, 50, 512)

Variants: bidirectional RNNs read both directions (for tasks with full-sequence access; not for generation); stacked layers; encoder-decoder with attention (Bahdanau, 2014) removed the fixed-size bottleneck by letting the decoder look at all encoder states, the direct ancestor of self-attention.

  • 1D/temporal convolutions (TCN, WaveNet): causal, dilated convs give large receptive fields with parallel training; WaveNet generated raw audio.
  • Transformers (2017+): replace recurrence with attention; fully parallel training across positions; direct access to any previous token; now the default for language, and increasingly for vision, audio, and time series. Cost: attention is O(T2)O(T^2) in sequence length.
  • State-space and linear-recurrent models (S4, Mamba and successors, RWKV, linear attention variants): return to recurrent computation with structured, often linear state updates that train in parallel via scans/convolutions and run inference in O(1)O(1) memory per step. They are competitive at some scales and used in hybrids with attention; whether they displace transformers at the frontier is an open and fast-moving question, and the strongest large models as of 2026 still rely primarily on attention-based architectures, often with efficiency tweaks or hybrid layers.
Data Reasonable default
Images (any serious scale) Pretrained ViT or modern CNN backbone; fine-tune
Small image datasets, edge Efficient CNN (MobileNet/EfficientNet), pretrained
Text, code Transformer (pretrained LLM)
Long streaming signals with strict memory LSTM/GRU or state-space model
Tabular Gradient boosting (Classical Models)
Short time series Simple baselines, then TCN/transformer/boosting; strong baselines are hard to beat
  • Forgetting that BatchNorm behaves differently in train/eval; frozen BN when fine-tuning with tiny batches.
  • Wrong tensor layout: PyTorch conv is (N,C,H,W); RNN default is (T,N,F) unless batch_first=True.
  • Not clipping gradients in RNNs.
  • Padding variable-length sequences without masking or pack_padded_sequence, so padding contaminates states.
  • Evaluating on images with different preprocessing than training.
  • Upsampling artifacts from transposed convs; aliasing from strided downsampling without blur.
  • Convolutions share weights and exploit locality; parameter count is Cout(Cink2+1)C_{out}(C_{in}k^2+1) regardless of image size; know the output-size formula.
  • Stacked small kernels grow the receptive field cheaply; 1×11\times1 convs mix channels.
  • Residual connections y=x+F(x)y=x+F(x) solve the degradation problem by providing an identity gradient path.
  • RNNs share weights across time; BPTT multiplies Jacobians, causing vanishing/exploding gradients.
  • LSTM/GRU use additive, gated state updates so gradients can flow over long ranges.
  • Attention-based models now dominate sequences, with recurrent/state-space models an active alternative for efficiency.
  1. Computation. A layer maps 64×56×5664\times56\times56 to 128×28×28128\times28\times28 using a 3×33\times3 conv with stride 2 and padding 1. Verify the output size and count parameters and multiply-accumulates.
  2. Conceptual. Why does a ResNet block with F→0F\to0 at initialization help optimization at great depth?
  3. Hands-on. Train a small CNN on CIFAR-10 with and without residual connections at depth 20/56. Compare training loss curves.
  4. Hands-on. Build a character-level language model with an LSTM on a small text; sample from it with different temperatures. Compare to a vanilla RNN on a copy-task of length 50.
  5. Derivation. For a linear RNN ht=Wht−1h_t=Wh_{t-1}, show ∥hT∥\|h_T\| behaves like ρ(W)T\rho(W)^T, relating the spectral radius to vanishing/exploding behavior.
Solutions (1, 5)
  1. Hout=⌊(56+2−3)/2+1⌋=28H_{out}=\lfloor(56+2-3)/2+1\rfloor=28. Parameters =128(64⋅9+1)=73,856=128(64\cdot9+1)=73{,}856. MACs =128⋅28⋅28⋅64⋅9≈57.8=128\cdot28\cdot28\cdot64\cdot9\approx57.8M.
  2. hT=WTh0h_T=W^Th_0. If W=VΛV−1W=V\Lambda V^{-1} then WT=VΛTV−1W^T=V\Lambda^TV^{-1}, dominated by λmax⁡T\lambda_{\max}^T. Magnitude <1<1 decays, >1>1 explodes; backward gradients involve (W⊤)T−k(W^\top)^{T-k} with the same behavior.
  • LeCun et al., “Gradient-Based Learning Applied to Document Recognition” (1998).
  • Krizhevsky, Sutskever, Hinton, “ImageNet Classification with Deep CNNs” (2012).
  • He, Zhang, Ren, Sun, “Deep Residual Learning for Image Recognition” (2015).
  • Dumoulin and Visin, “A guide to convolution arithmetic for deep learning” (2016).
  • Hochreiter and Schmidhuber, “Long Short-Term Memory” (1997); Cho et al., “Learning Phrase Representations using RNN Encoder-Decoder” (2014).
  • Pascanu, Mikolov, Bengio, “On the difficulty of training recurrent neural networks” (2013).
  • Olah, “Understanding LSTM Networks” (blog, 2015).
  • Liu et al., “A ConvNet for the 2020s” (2022).
  • Gu and Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces” (2023).
  • Zhang, Lipton, Li, Smola, Dive into Deep Learning (d2l.ai).