Training Deep Networks
What you’ll learn. Why deep networks are hard to train and the techniques that made them trainable: variance-preserving initialization, normalization layers, residual connections, regularization (dropout, weight decay, augmentation), learning-rate warmup and decay, mixed-precision arithmetic, gradient clipping, and a systematic debugging recipe. Ends with scaling intuition: how loss behaves with data, parameters, and compute, and what is practical at large scale.
Why depth is hard
Section titled “Why depth is hard”The gradient to an early layer is a product of per-layer Jacobians (Backpropagation). With layers whose typical gain is , signals and gradients scale like : for and that is ; for it is . Training deep networks is largely about keeping that gain near 1, in the forward and backward direction, at initialization and throughout training.
Initialization
Section titled “Initialization”Goal: preserve activation variance forward and gradient variance backward. For a layer with inputs of variance and independent zero-mean weights of variance ,
To keep variance constant: (LeCun). Considering the backward pass as well gives the Xavier/Glorot compromise , derived for tanh/sigmoid. ReLU zeroes half of the pre-activations, halving variance, so He/Kaiming init uses .
import torch.nn as nnlayer = nn.Linear(512, 512)nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")nn.init.zeros_(layer.bias)Other practices:
- Zero-init the last layer of each residual branch (or scale it by , GPT-2 style) so each block starts near the identity.
- Small init for output embeddings/logits so the initial loss is near (for classes). Check initial loss: if it differs a lot from the init or loss is off.
- Transformers typically use for most weights, with scaled-down residual projections. PyTorch defaults are reasonable for standard layers but not always optimal for deep transformers.
- Never initialize all weights equal: symmetry is never broken.
Normalization
Section titled “Normalization”Normalization layers rescale activations to stabilize training, allow higher learning rates, and reduce sensitivity to init.
Batch normalization (Ioffe and Szegedy, 2015): for each feature, normalize across the batch, then apply learned scale and shift :
At inference, running averages of are used. Great for CNNs; problematic with small batches, sequences of variable length, and distributed settings; behaves differently in train vs eval mode (a major bug source: forgetting model.eval()). Why it works is debated: smoothing of the loss landscape is a leading explanation more than the originally claimed “internal covariate shift.”
Layer normalization normalizes across the features of each example (no batch dependence), standard in transformers. RMSNorm drops mean-centering: , cheaper and common in recent LLMs. GroupNorm suits small-batch vision.
Placement in transformers: post-LN (normalize after the residual add, original Transformer) is harder to train deep and needs warmup; pre-LN (normalize inputs to each sublayer) is more stable and now standard (Attention & Transformers).
import torchx = torch.randn(8, 16, 64) # (batch, tokens, features)ln = torch.nn.LayerNorm(64)y = ln(x)print(y.mean(-1).abs().max(), y.std(-1).mean()) # ~0, ~1 per tokenResidual connections
Section titled “Residual connections”gives gradients an identity path: . This made 100+ layer networks trainable (ResNet, 2015) and is in every transformer. See CNNs and RNNs.
Regularization
Section titled “Regularization”- Weight decay: with AdamW use decoupled decay (typically 0.01 to 0.1), usually not applied to biases and norm parameters (Calculus & Optimization).
- Dropout: during training zero each activation with probability and scale survivors by (“inverted dropout”) so the expected activation is unchanged; at inference do nothing. Acts like ensembling over sub-networks and reduces co-adaptation. Common to . Large pretrained LLMs often use little or no dropout because a single epoch over huge data barely overfits; small-data fine-tuning benefits more.
- Data augmentation: random crops, flips, color jitter, mixup/CutMix for images; noise, masking, back-translation for text/audio. Often the most effective regularizer.
- Label smoothing: replace one-hot with on the true class and elsewhere; reduces overconfidence.
- Early stopping on validation loss.
- Stochastic depth / DropPath, weight averaging (EMA of weights) for smoother final models.
import numpy as npdef dropout(x, p, training, rng=np.random.default_rng(0)): if not training or p == 0: return x mask = rng.random(x.shape) >= p return x * mask / (1 - p)Learning-rate schedules
Section titled “Learning-rate schedules”The learning rate is the most important hyperparameter. A constant LR is rarely optimal: large early steps make fast progress; small late steps settle into a minimum (SGD noise scales with LR).
- Warmup: increase LR linearly from near 0 over the first 0.5-5% of steps (for LLMs often a few hundred to a few thousand steps). Needed because early Adam second-moment estimates are unreliable and early gradients are large and unstable, especially with post-LN, large batches, and mixed precision.
- Decay: cosine decay to ~10% of peak is the common default; linear decay and step decay (drop by 10x at set epochs, classic ResNet recipe) also work.
- Warmup-stable-decay (WSD): hold LR constant, then decay at the end; popular because the stable phase lets you branch checkpoints and extend training without committing to a total step count in advance.
- One-cycle (warm up to a high LR then anneal to near zero) can give fast convergence (“super-convergence”) for short budgets.
- Cyclic/restarts (SGDR) escape and re-explore.
- ReduceLROnPlateau: drop LR when validation stalls; simple and robust for small projects.
import mathdef lr_at(step, total, peak=3e-4, warmup=500, min_ratio=0.1): if step < warmup: return peak * (step + 1) / warmup t = (step - warmup) / max(1, total - warmup) return peak * (min_ratio + (1 - min_ratio) * 0.5 * (1 + math.cos(math.pi * t)))Find a good peak with an LR range test (sweep exponentially and stop at divergence), then pick roughly 3-10x below the divergence point. Rule of thumb for AdamW on transformers: peak LR decreases as model size grows (about for ~100M parameters down to or below for multi-billion). Treat as a starting point and tune.
Batch size
Section titled “Batch size”Larger batches give less noisy gradients and better hardware utilization, but diminishing returns beyond the critical batch size (roughly where gradient noise equals signal), which grows during training. Scale the LR with the batch size (linear for SGD with warmup; sub-linear for Adam as a heuristic). Gradient accumulation simulates a large batch on small memory: sum gradients over micro-batches before stepping (divide each loss by ).
Gradient clipping
Section titled “Gradient clipping”Clip the global gradient norm to a threshold (commonly 1.0) to guard against rare huge gradients:
Essential for RNNs and transformer pretraining. Logging the pre-clip norm is a valuable health signal: sustained spikes precede loss spikes.
Mixed precision
Section titled “Mixed precision”Floating point formats: fp32 (8 exponent, 23 mantissa bits), fp16 (5, 10), bf16 (8, 7), and 8-bit formats (FP8 E4M3/E5M2) on recent GPUs. Matrix multiplies dominate compute, and lower precision multiplies faster and halves memory traffic.
- fp16 has a narrow range (max ≈65504; small gradients underflow), so it needs loss scaling: multiply the loss by a large factor before backward, unscale gradients afterward, skip steps with inf/NaN.
- bf16 has fp32’s range with less precision: no loss scaling needed; the default on modern accelerators (Ampere/Hopper GPUs, TPUs).
- Keep master weights and optimizer states in fp32; run matmuls/convs in low precision; keep reductions (softmax, layer norm, loss) in fp32.
- FP8 training is used in frontier-scale runs with per-tensor scaling; it requires careful recipes and is more fragile.
import torchmodel = model.cuda(); opt = torch.optim.AdamW(model.parameters(), lr=3e-4)for x, y in loader: x, y = x.cuda(), y.cuda() with torch.autocast(device_type="cuda", dtype=torch.bfloat16): loss = torch.nn.functional.cross_entropy(model(x), y) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step(); opt.zero_grad(set_to_none=True)Memory accounting for training with Adam in mixed precision, per parameter: bf16 weights (2) + fp32 master (4) + Adam (8) + gradients (2 to 4) bytes, so a 7B-parameter model needs about 112 GB before activations. That is why sharding (ZeRO/FSDP), activation checkpointing, and 8-bit optimizers exist.
A debugging recipe
Section titled “A debugging recipe”Karpathy’s “recipe for training neural networks” captures hard-won practice:
- Become one with the data. Look at examples, label noise, class balance, duplicates.
- Build the end-to-end skeleton with a simple model and fixed seeds; get evaluation right early.
- Verify initial loss equals the expected value ( for -way classification).
- Overfit a tiny batch (e.g. 10 examples) to ~0 loss. If you cannot, there is a bug: wrong labels alignment, missing zero_grad, frozen params, LR too low/high, wrong loss reduction.
- Check input to the model: decode a batch right before the forward pass and visualize it (augmentation bugs and label shuffles hide here).
- Compare against a baseline and a known-good reference implementation.
- Overfit, then regularize: first get a large model to fit the training set, then add data, augmentation, weight decay, dropout.
- Tune: random search over LR, weight decay, batch size beats grid search; LR matters most.
- Squeeze: ensembles, longer training, EMA.
Symptoms and likely causes:
| Symptom | Suspects |
|---|---|
| Loss is NaN/inf | LR too high, fp16 overflow, log(0), divide by zero, bad data, missing grad clipping |
| Loss flat from the start | LR too low, dead ReLUs, detached graph, bad init, frozen parameters, labels unrelated to inputs |
| Train loss falls, val loss rises | Overfitting, leakage in the wrong direction, train/eval mode mismatch (dropout/BN) |
| Train/val both high | Underfitting, too little capacity, too strong regularization, too few steps |
| Loss spikes mid-run | LR too high at that stage, bad batch, Adam / issues; clip, lower LR, skip bad batches, restart from checkpoint |
| Validation fluctuates wildly | Small val set, BN running stats, high LR |
| Works on one GPU, not on many | Effective batch size/LR mismatch, unsynced BN, seeds, data sharding duplicates |
Monitor: loss curves (train and val), gradient norm, parameter and update norms (update/weight ratio around is a classic healthy range), activation statistics per layer, learning rate, throughput (tokens/s), and a few sample predictions.
Scaling intuition
Section titled “Scaling intuition”Empirically, for language models, test loss follows smooth power laws in parameters , dataset size , and compute (Kaplan et al., 2020). The “Chinchilla” analysis (Hoffmann et al., 2022) found that for a fixed compute budget, parameters and training tokens should be scaled together: roughly 20 tokens per parameter for compute-optimal training under their setup, with FLOPs. In practice, because inference cost depends on model size, many production models are deliberately trained far past compute-optimal (hundreds to thousands of tokens per parameter) to get a smaller, cheaper model at equal quality.
Practical consequences:
- Small-scale experiments predict large-scale behavior if the hyperparameters scale correctly. Methods like maximal-update parametrization (P) aim to make optimal LR transfer across widths so you tune on a small proxy.
- Throughput is measured by model FLOPs utilization (MFU); good large-scale runs reach roughly 30-55% on modern GPUs. Data loading, communication, and memory bandwidth are common bottlenecks.
- Parallelism: data parallel (replicate model, split batch), tensor/pipeline parallel (split the model), sharded optimizer/parameters (ZeRO, FSDP), sequence/context parallel.
- Data quality often beats data quantity; deduplication and filtering are major levers.
- Reported scaling exponents and optimal ratios depend on architecture, data, and training recipe, and the field has revised them repeatedly (e.g. data-constrained and inference-aware extensions); treat specific numbers as approximate.
Key takeaways
Section titled “Key takeaways”- Keep per-layer gain near 1: He/Xavier init, residual connections, and normalization.
- BatchNorm for CNNs with decent batch sizes; LayerNorm/RMSNorm (pre-LN) for transformers.
- Use AdamW with warmup plus cosine (or WSD) decay; tune the peak LR first.
- bf16 autocast with fp32 master weights is the standard mixed-precision recipe; fp16 needs loss scaling.
- Clip gradients, log the norm, and watch for spikes.
- Debug by checking initial loss and overfitting a tiny batch before anything fancy.
- Scaling laws guide compute allocation; exact constants are recipe-dependent.
Exercises
Section titled “Exercises”- Derivation. For a ReLU layer with , show the forward variance gain is and conclude .
- Hands-on. Train a 20-layer MLP on MNIST with (a) default init, (b) He init, (c) He init plus LayerNorm. Plot per-layer activation std at step 0 and training loss.
- Hands-on. Implement the LR range test and warmup-cosine schedule; compare a constant LR with the schedule on a small transformer or CNN.
- Hands-on. Induce each symptom in the table above on purpose (e.g., forget
zero_grad, omitmodel.eval()) and observe the curves. - Computation. Estimate the training memory (weights, grads, Adam states, mixed precision) for a 1B-parameter model, and the FLOPs to train it on 20B tokens using .
Solutions (1, 5)
- and for ReLU of a symmetric zero-mean , . So . Gain 1 needs .
- About 16 bytes/param, so ~16 GB plus activations. FLOPs , which at 40% MFU on a 400 TFLOP/s (bf16 peak) GPU, i.e. 160 TFLOP/s effective, is s, about 8.7 GPU-days.
Further reading
Section titled “Further reading”- Karpathy, “A Recipe for Training Neural Networks” (blog, 2019).
- Glorot and Bengio (2010); He et al., “Delving Deep into Rectifiers” (2015).
- Ioffe and Szegedy, “Batch Normalization” (2015); Ba, Kiros, Hinton, “Layer Normalization” (2016); Zhang and Sennrich, “Root Mean Square Layer Normalization” (2019).
- Srivastava et al., “Dropout” (2014).
- Micikevicius et al., “Mixed Precision Training” (2018).
- Kaplan et al., “Scaling Laws for Neural Language Models” (2020); Hoffmann et al., “Training Compute-Optimal Large Language Models” (2022).
- Yang et al., “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer” (2022).
- Smith, “Cyclical Learning Rates for Training Neural Networks” (2015).