Skip to content

Backpropagation & Autodiff

What you’ll learn. How reverse-mode automatic differentiation computes the gradient of a scalar loss with respect to millions of parameters at about the cost of one extra forward pass; how to think in computational graphs; the local-gradient rules for common operations; the difference between forward and reverse mode; and how to build a small, working autodiff engine (in the style of Karpathy’s micrograd) and train a neural network with it.

Training needs ∇θL\nabla_\theta L for a loss LL that is a composition of thousands of operations. Three ways to get derivatives:

  1. Numerical differentiation: L(θ+ϵei)−L(θ−ϵei)2ϵ\frac{L(\theta+\epsilon e_i)-L(\theta-\epsilon e_i)}{2\epsilon}. Needs two forward passes per parameter: hopeless for 10910^9 parameters; imprecise. Use only for gradient checking.
  2. Symbolic differentiation: manipulates expressions; suffers expression swell and cannot handle control flow naturally.
  3. Automatic differentiation (AD): applies the chain rule to the actual sequence of elementary operations executed, with exact (to floating-point) results. This is what PyTorch, JAX, and TensorFlow do.

Any computation decomposes into primitive operations, forming a directed acyclic graph (DAG): nodes are values, edges are data dependencies.

Example: L=(wx+b−y)2L=(wx+b-y)^2 with x=2,w=3,b=1,y=5x=2,w=3,b=1,y=5:

w=3 ──┐
├─ mul ─ a=6 ─┐
x=2 ──┘ ├─ add ─ c=7 ─┐
b=1 ────────────────┘ ├─ sub ─ d=2 ─ square ─ L=4
y=5 ──────────────────────────────┘

Forward pass: evaluate nodes in topological order. Backward pass: starting from ∂L/∂L=1\partial L/\partial L=1, propagate gradients to inputs in reverse topological order.

For a node vv computed from inputs u1,…,uku_1,\dots,u_k with local derivatives ∂v/∂uj\partial v/\partial u_j, the gradient flowing in from downstream is vˉ=∂L/∂v\bar v=\partial L/\partial v (the adjoint). Then

uˉj+=vˉ⋅∂v∂uj.\bar u_j\mathrel{+}=\bar v\cdot\frac{\partial v}{\partial u_j}.

The += is essential: if a variable is used in several places (fan-out), gradients from all uses sum (multivariate chain rule). Forgetting accumulation is the classic bug.

Walk through the example: Lˉ=1\bar L=1. dd: L=d2L=d^2, so dˉ=2d=4\bar d=2d=4. cc: d=c−yd=c-y, cˉ=dˉ=4\bar c=\bar d=4, yˉ=−4\bar y=-4. aa: aˉ=4\bar a=4, bˉ=4\bar b=4. ww: a=wxa=wx, so wˉ=aˉ⋅x=8\bar w=\bar a\cdot x=8, xˉ=aˉ⋅w=12\bar x=\bar a\cdot w=12.

Check with L=(wx+b−y)2L=(wx+b-y)^2: ∂L/∂w=2(wx+b−y)x=2⋅2⋅2=8\partial L/\partial w=2(wx+b-y)x=2\cdot2\cdot2=8. ✓.

Operation v=…v=\dots Backward (uˉ\bar u contributions)
u1+u2u_1+u_2 uˉ1+=vˉ\bar u_1\mathrel{+}=\bar v, uˉ2+=vˉ\bar u_2\mathrel{+}=\bar v
u1⋅u2u_1\cdot u_2 uˉ1+=vˉ u2\bar u_1\mathrel{+}=\bar v\,u_2, uˉ2+=vˉ u1\bar u_2\mathrel{+}=\bar v\,u_1
upu^p uˉ+=vˉ p up−1\bar u\mathrel{+}=\bar v\,p\,u^{p-1}
eue^u uˉ+=vˉ v\bar u\mathrel{+}=\bar v\,v
ln⁡u\ln u uˉ+=vˉ/u\bar u\mathrel{+}=\bar v/u
tanh⁡u\tanh u uˉ+=vˉ(1−v2)\bar u\mathrel{+}=\bar v(1-v^2)
ReLU(uu) uˉ+=vˉ⋅1[u>0]\bar u\mathrel{+}=\bar v\cdot\mathbb 1[u>0]
V=ABV=AB (matrices) Aˉ+=VˉB⊤\bar A\mathrel{+}=\bar VB^\top, Bˉ+=A⊤Vˉ\bar B\mathrel{+}=A^\top\bar V

Every framework op is defined by a forward function and a backward function (a “vector–Jacobian product”, VJP). Adding a new op means writing these two.

Let f:Rn→Rmf:\mathbb{R}^n\to\mathbb{R}^m with Jacobian J∈Rm×nJ\in\mathbb{R}^{m\times n}.

  • Forward mode propagates a tangent x˙\dot x and computes Jx˙J\dot x (a Jacobian–vector product, JVP) per pass. One pass per input dimension gives the full Jacobian: cheap when n≪mn\ll m.
  • Reverse mode propagates adjoints and computes yˉ⊤J\bar y^\top J (VJP) per pass. One pass per output dimension: cheap when m≪nm\ll n.

Neural-network training has a scalar loss (m=1m=1) and huge nn, so reverse mode gives the whole gradient in one backward pass costing about 2-3x the forward pass (the “cheap gradient principle”). The price is memory: reverse mode must store the intermediate activations from the forward pass. This is why training memory scales with batch size and depth, and why activation (gradient) checkpointing trades recomputation for memory (Training Deep Networks). JAX exposes both (jax.jvp, jax.vjp); forward-over-reverse gives Hessian-vector products without forming the Hessian (Calculus & Optimization).

Below is a complete scalar autodiff engine. It is slow (one Python object per number) but shows the whole idea in about 60 lines.

import math
class Value:
def __init__(self, data, _children=(), _op=""):
self.data = float(data)
self.grad = 0.0
self._backward = lambda: None
self._prev = set(_children)
self._op = _op
def __repr__(self):
return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad += out.grad # d(out)/d(self) = 1
other.grad += out.grad
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def __pow__(self, p):
assert isinstance(p, (int, float))
out = Value(self.data ** p, (self,), f"**{p}")
def _backward():
self.grad += p * self.data ** (p - 1) * out.grad
out._backward = _backward
return out
def exp(self):
out = Value(math.exp(self.data), (self,), "exp")
def _backward():
self.grad += out.data * out.grad
out._backward = _backward
return out
def log(self):
out = Value(math.log(self.data), (self,), "log")
def _backward():
self.grad += out.grad / self.data
out._backward = _backward
return out
def tanh(self):
t = math.tanh(self.data)
out = Value(t, (self,), "tanh")
def _backward():
self.grad += (1 - t * t) * out.grad
out._backward = _backward
return out
def relu(self):
out = Value(max(0.0, self.data), (self,), "relu")
def _backward():
self.grad += (out.data > 0) * out.grad
out._backward = _backward
return out
def backward(self):
topo, visited = [], set()
def build(v):
if v not in visited:
visited.add(v)
for c in v._prev:
build(c)
topo.append(v) # children before parents
build(self)
self.grad = 1.0
for v in reversed(topo): # parents before children
v._backward()
# derived operations
def __neg__(self): return self * -1
def __sub__(self, o): return self + (-o if isinstance(o, Value) else Value(-o))
def __radd__(self, o): return self + o
def __rmul__(self, o): return self * o
def __rsub__(self, o): return Value(o) + (-self)
def __truediv__(self, o): return self * (o if isinstance(o, Value) else Value(o)) ** -1

Key design points:

  • Each operation stores a closure _backward that knows how to push out.grad to its inputs using local derivatives.
  • backward() first topologically sorts the graph, then runs each closure in reverse order, so a node’s gradient is complete (all downstream contributions accumulated) before it is propagated further.
  • += accumulates for fan-out.
  • Gradients live on the nodes; you must zero them before each new backward pass (this is why PyTorch has optimizer.zero_grad()).

Test against the worked example and against calculus:

w, x, b, y = Value(3), Value(2), Value(1), Value(5)
L = (w * x + b - y) ** 2
L.backward()
print(w.grad, x.grad, b.grad) # 8.0 12.0 4.0
# gradient with fan-out: f = a*a + a => df/da = 2a + 1
a = Value(3.0); f = a * a + a; f.backward(); print(a.grad) # 7.0
import random
class Neuron:
def __init__(self, nin, act=True):
self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
self.b = Value(0.0); self.act = act
def __call__(self, x):
s = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
return s.tanh() if self.act else s
def parameters(self): return self.w + [self.b]
class Layer:
def __init__(self, nin, nout, **kw): self.neurons = [Neuron(nin, **kw) for _ in range(nout)]
def __call__(self, x):
out = [n(x) for n in self.neurons]
return out[0] if len(out) == 1 else out
def parameters(self): return [p for n in self.neurons for p in n.parameters()]
class MLP:
def __init__(self, nin, nouts):
sz = [nin] + nouts
self.layers = [Layer(sz[i], sz[i+1], act=(i < len(nouts) - 1)) for i in range(len(nouts))]
def __call__(self, x):
for layer in self.layers: x = layer(x)
return x
def parameters(self): return [p for l in self.layers for p in l.parameters()]
random.seed(1)
net = MLP(3, [4, 4, 1])
xs = [[2.0, 3.0, -1.0], [3.0, -1.0, 0.5], [0.5, 1.0, 1.0], [1.0, 1.0, -1.0]]
ys = [1.0, -1.0, -1.0, 1.0]
for step in range(100):
loss = sum((net(x) - y) ** 2 for x, y in zip(xs, ys)) # forward
for p in net.parameters(): p.grad = 0.0 # zero grads
loss.backward() # backward
for p in net.parameters(): p.data -= 0.05 * p.grad # SGD update
print(loss.data) # near 0

This is the same loop as every PyTorch training script (PyTorch in Practice): forward, zero grad, backward, step.

Real frameworks make each node a tensor and each op’s backward a tensor VJP (the table’s matrix rule is the heart of Neural Networks). Important details:

  • Broadcasting: if uu was broadcast in the forward pass, its gradient must be summed over the broadcast dimensions in the backward pass.
  • Reductions: the backward of sum broadcasts the gradient back; of mean also divides by the count; of max routes gradient to the argmax.
  • Dynamic vs static graphs: PyTorch builds the graph on the fly as code runs (define-by-run), which allows ordinary Python control flow and easy debugging. JAX traces pure functions and compiles them with XLA (jit); TensorFlow 1 used static graphs. torch.compile captures PyTorch code into optimized graphs for speed.
  • Non-differentiable ops (argmax, sampling, rounding) have zero or undefined gradients; workarounds include the straight-through estimator, Gumbel-softmax, and REINFORCE-style score-function gradients (Reinforcement Learning Basics).
  • Detach / no_grad: cut the graph where you do not want gradients (target networks, evaluation).
  • Higher-order gradients: if the backward pass is itself recorded as differentiable operations, you can differentiate through it (needed for gradient penalties, meta-learning).

Vanishing and exploding gradients, seen in the graph

Section titled “Vanishing and exploding gradients, seen in the graph”

The gradient to early layers is a product of Jacobians, ∏ℓdiag⁡(σ′(zℓ))Wℓ\prod_\ell\operatorname{diag}(\sigma'(z_\ell))W_\ell. If typical singular values are below 1 the product shrinks exponentially with depth; above 1 it explodes. This is the root of the issues treated in Training Deep Networks and CNNs and RNNs. Residual connections create an additive path (y=x+f(x)y=x+f(x), Jacobian I+∂fI+\partial f) that keeps gradient flow healthy.

  1. Gradient check: compare to central differences in float64 on a tiny network (ϵ≈10−5\epsilon\approx10^{-5}, relative error <10−6<10^{-6} is good).
  2. In PyTorch: torch.autograd.gradcheck on custom Functions.
  3. Print gradient norms per layer; look for zeros (dead units, detached graph) and NaN/inf.
  4. Test invariants: the gradient of a sum of losses is the sum of gradients; the loss should decrease along −∇L-\nabla L for small steps.
  5. Overfit a tiny batch; if it cannot, suspect a bug in the graph or loss.
  • AD applies the chain rule to the executed sequence of primitive operations; it is exact and not the same as numerical or symbolic differentiation.
  • Reverse mode computes the gradient of a scalar loss in one backward sweep at roughly the cost of a small constant times the forward pass, at the price of storing activations.
  • Every op needs only its local derivative; gradients accumulate over fan-out; the backward traversal is in reverse topological order.
  • A working engine is small: a Value with data, grad, children, and a backward closure.
  • Tensor autodiff adds broadcasting, reductions, and VJPs; the backward of a matmul is two matmuls.
  • Gradient checking and gradient-norm monitoring are your debugging tools.
  1. Hands-on. Add sigmoid, sqrt, and __radd__-based sum support to Value, and gradient-check each against finite differences.
  2. Derivation. For V=ABV=AB show Aˉ=VˉB⊤\bar A=\bar VB^\top by considering ∂L/∂Aij\partial L/\partial A_{ij}.
  3. Conceptual. A node’s output is used by three downstream nodes. Write its gradient in terms of their adjoints. What breaks if you use = instead of +=?
  4. Hands-on. Train the micrograd MLP on the two-spirals data from Neural Networks with a softmax/cross-entropy or hinge loss. How slow is it relative to NumPy?
  5. Conceptual. Why is forward-mode AD better for computing the Jacobian of f:R3→R1000f:\mathbb{R}^3\to\mathbb{R}^{1000}, and reverse-mode for f:R106→Rf:\mathbb{R}^{10^6}\to\mathbb{R}?
Solutions (2, 3)
  1. Vik=∑jAijBjkV_{ik}=\sum_jA_{ij}B_{jk}, so ∂L/∂Aij=∑kVˉikBjk=(VˉB⊤)ij\partial L/\partial A_{ij}=\sum_k\bar V_{ik}B_{jk}=(\bar VB^\top)_{ij}.
  2. uˉ=∑k=13vˉk ∂vk/∂u\bar u=\sum_{k=1}^3\bar v_k\,\partial v_k/\partial u. With = only the last contribution survives, giving silently wrong gradients (a bug that still trains, badly).
  • Karpathy, “The spelled-out intro to neural networks and backpropagation: building micrograd” (video lecture, 2022) and the micrograd repository.
  • Rumelhart, Hinton, Williams, “Learning representations by back-propagating errors” (1986).
  • Baydin, Pearlmutter, Radul, Siskind, “Automatic Differentiation in Machine Learning: a Survey” (2018).
  • Griewank and Walther, Evaluating Derivatives (2nd ed., 2008).
  • Paszke et al., “Automatic differentiation in PyTorch” (2017).
  • JAX documentation, “The Autodiff Cookbook”.
  • Goodfellow, Bengio, Courville, Deep Learning (2016), chapter 6.5.