Skip to content

Neural Networks from Scratch

What you’ll learn. How a single artificial neuron works, why stacking them with nonlinearities gives expressive models, the common activation functions and their trade-offs, what the universal approximation theorem does and does not say, and how to implement a trained MLP in pure NumPy, including a manual backward pass that you can check numerically. Backpropagation then generalizes the manual derivation.

A neuron computes a=σ(w⊤x+b)a=\sigma(w^\top x+b): a linear function (weights ww, bias bb) followed by a nonlinearity σ\sigma. With a sigmoid it is logistic regression (Classical Models). A layer applies noutn_{out} neurons in parallel: h=σ(Wx+b)h=\sigma(Wx+b) with W∈Rnout×ninW\in\mathbb{R}^{n_{out}\times n_{in}}. A multilayer perceptron (MLP) composes layers:

h0=x,hℓ=σℓ(Wℓhℓ−1+bℓ),y^=hL.h_0=x,\quad h_\ell=\sigma_\ell(W_\ell h_{\ell-1}+b_\ell),\quad \hat y=h_L.
input hidden 1 hidden 2 output
x1 ---\ /-- h1 --\ /-- g1 --\
x2 ----+--+--- h2 ---+---+--- g2 ---+----> y_hat
x3 ---/ \-- h3 --/ \-- g3 --/
W1,b1 relu W2,b2 relu W3,b3

The whole network is a differentiable function of its parameters θ={Wℓ,bℓ}\theta=\{W_\ell,b_\ell\}, trained by gradient descent on a loss (Calculus & Optimization).

Rosenblatt’s perceptron (1958) is a neuron with a step activation, y^=1[w⊤x+b>0]\hat y=\mathbb 1[w^\top x+b>0], with the update rule: for each misclassified example, w←w+η (y−y^)xw\leftarrow w+\eta\,(y-\hat y)x. It converges if and only if the data is linearly separable (the perceptron convergence theorem). It cannot represent XOR: no single line separates {(0,0),(1,1)}\{(0,0),(1,1)\} from {(0,1),(1,0)}\{(0,1),(1,0)\}. Minsky and Papert’s 1969 analysis of such limits contributed to the first AI winter. A hidden layer fixes it: XOR == AND(OR, NAND), two hidden units then one output.

import numpy as np
def perceptron(X, y, lr=1.0, epochs=20):
w = np.zeros(X.shape[1]); b = 0.0
for _ in range(epochs):
for xi, yi in zip(X, y):
pred = int(w @ xi + b > 0)
w += lr * (yi - pred) * xi
b += lr * (yi - pred)
return w, b
X = np.array([[0,0],[0,1],[1,0],[1,1]]); y_and = np.array([0,0,0,1])
print(perceptron(X, y_and)) # learns AND; would never converge on XOR

Without σ\sigma, two layers collapse: W2(W1x+b1)+b2=(W2W1)x+…W_2(W_1x+b_1)+b_2=(W_2W_1)x+\dots, which is just one linear map. Depth gives power only through nonlinearities.

Name Formula Notes
Sigmoid 1/(1+e−z)1/(1+e^{-z}) Output in (0,1); saturates (gradient →0\to0 for large ∣z∣\lvert z\rvert); not zero-centered. Used for binary output probabilities and gates.
Tanh tanh⁡z\tanh z Zero-centered; still saturates. Used in RNN cells.
ReLU max⁡(0,z)\max(0,z) Cheap, no saturation for z>0z>0, sparse activations. “Dying ReLU”: units stuck at 0 get zero gradient.
Leaky ReLU / PReLU max⁡(αz,z)\max(\alpha z,z) Small slope for z<0z<0 avoids dead units.
GELU z Φ(z)z\,\Phi(z) Smooth ReLU-like; standard in BERT/GPT-style transformers.
SiLU / Swish z σ(z)z\,\sigma(z) Smooth, non-monotonic; widely used.
SwiGLU (gated) (SiLU(xW)⊙xV)(\text{SiLU}(xW)\odot xV) Gated linear unit variant; common in recent LLMs’ feed-forward blocks.
Softmax ezk/∑ezje^{z_k}/\sum e^{z_j} Output layer for multiclass probabilities.

The vanishing gradient problem is visible from derivatives: σ′(z)≤0.25\sigma'(z)\le0.25, so multiplying many such factors through depth shrinks gradients exponentially. ReLU’s derivative is exactly 1 on the active side, a main reason deep networks became trainable (Training Deep Networks). Default today: ReLU/GELU/SiLU in hidden layers; the output activation matches the loss (identity for regression, sigmoid for binary, softmax for multiclass, folded into the loss for numerical stability).

Theorem (Cybenko 1989; Hornik 1991; Leshno et al. 1993). A feedforward network with one hidden layer and a non-polynomial activation can approximate any continuous function on a compact set to arbitrary accuracy, given enough hidden units.

Intuition with ReLU in 1D: each hidden unit adds a “hinge”; a sum of hinges can form any piecewise-linear function, which approximates any continuous curve as hinges multiply. In dd dimensions each unit creates a half-space boundary and combinations carve out complex regions.

What it does not say:

  • It does not bound the width; it may be exponentially large in dd.
  • It does not say gradient descent will find the weights.
  • It does not say the network will generalize from finite data.

Depth helps in practice: there are functions that deep networks represent with polynomially many units where shallow ones need exponentially many (depth separation results), and deep compositional hierarchies match the structure of images and language. The number of linear regions of a ReLU network grows exponentially with depth. See CNNs and RNNs for architectures that bake in useful structure.

We implement a 2-hidden-layer MLP for classification with ReLU and softmax cross-entropy. Convention: batch X∈RB×dX\in\mathbb{R}^{B\times d}, weights W∈Rdin×doutW\in\mathbb{R}^{d_{in}\times d_{out}} (row vectors), so a layer is Z=HW+bZ=HW+b.

Forward:

Z1=XW1+b1, H1=ReLU(Z1), Z2=H1W2+b2, H2=ReLU(Z2), Z3=H2W3+b3, P=softmax(Z3).Z_1=XW_1+b_1,\ H_1=\text{ReLU}(Z_1),\ Z_2=H_1W_2+b_2,\ H_2=\text{ReLU}(Z_2),\ Z_3=H_2W_3+b_3,\ P=\text{softmax}(Z_3).

Backward (loss L=−1B∑log⁡Pi,yiL=-\tfrac1B\sum\log P_{i,y_i}). Using ∂L/∂Z3=(P−Y)/B\partial L/\partial Z_3=(P-Y)/B with YY one-hot (Information Theory), the chain rule for Z=HW+bZ=HW+b gives

∂L∂W=H⊤∂L∂Z,∂L∂b=∑i∂L∂Zi,∂L∂H=∂L∂ZW⊤,\frac{\partial L}{\partial W}=H^\top\frac{\partial L}{\partial Z},\quad\frac{\partial L}{\partial b}=\sum_i\frac{\partial L}{\partial Z_i},\quad\frac{\partial L}{\partial H}=\frac{\partial L}{\partial Z}W^\top,

and through ReLU ∂L/∂Z1=∂L/∂H1⊙1[Z1>0]\partial L/\partial Z_1=\partial L/\partial H_1\odot\mathbb 1[Z_1>0].

import numpy as np
class MLP:
def __init__(self, sizes, seed=0):
rng = np.random.default_rng(seed)
# He initialization: variance 2/fan_in suits ReLU
self.W = [rng.normal(0, np.sqrt(2 / m), (m, n)) for m, n in zip(sizes[:-1], sizes[1:])]
self.b = [np.zeros(n) for n in sizes[1:]]
def forward(self, X):
self.H = [X]; self.Z = []
for i, (W, b) in enumerate(zip(self.W, self.b)):
z = self.H[-1] @ W + b
self.Z.append(z)
self.H.append(np.maximum(z, 0) if i < len(self.W) - 1 else z)
z = self.H[-1] - self.H[-1].max(1, keepdims=True)
e = np.exp(z)
return e / e.sum(1, keepdims=True)
def loss_and_grads(self, X, y):
B = len(X)
P = self.forward(X)
loss = -np.log(P[np.arange(B), y] + 1e-12).mean()
dZ = P.copy(); dZ[np.arange(B), y] -= 1; dZ /= B
gW, gb = [None] * len(self.W), [None] * len(self.W)
for i in reversed(range(len(self.W))):
gW[i] = self.H[i].T @ dZ
gb[i] = dZ.sum(0)
if i > 0:
dH = dZ @ self.W[i].T
dZ = dH * (self.Z[i - 1] > 0)
return loss, gW, gb
def step(self, gW, gb, lr):
for i in range(len(self.W)):
self.W[i] -= lr * gW[i]; self.b[i] -= lr * gb[i]

Train on a nonlinear problem (two spirals)

Section titled “Train on a nonlinear problem (two spirals)”
def spirals(n=300, noise=0.2, seed=0):
rng = np.random.default_rng(seed)
t = np.sqrt(rng.random(n)) * 3 * np.pi
a = np.c_[t * np.cos(t), t * np.sin(t)] / 10
X = np.r_[a, -a] + noise * rng.normal(size=(2 * n, 2)) / 3
y = np.r_[np.zeros(n), np.ones(n)].astype(int)
return X, y
X, y = spirals()
net = MLP([2, 64, 64, 2])
for epoch in range(2000):
loss, gW, gb = net.loss_and_grads(X, y)
net.step(gW, gb, lr=0.5)
if epoch % 400 == 0:
print(epoch, round(loss, 4), (net.forward(X).argmax(1) == y).mean())

A linear classifier cannot exceed about 50-60% on this data; the MLP drives training accuracy near 100%. (Full-batch gradient descent here for simplicity; real training uses minibatches.)

Always verify manual gradients against finite differences:

def grad_check(net, X, y, eps=1e-5):
loss, gW, _ = net.loss_and_grads(X, y)
W = net.W[0]
i, j = 0, 3
W[i, j] += eps; lp, _, _ = net.loss_and_grads(X, y)
W[i, j] -= 2*eps; lm, _, _ = net.loss_and_grads(X, y)
W[i, j] += eps
print(gW[0][i, j], (lp - lm) / (2 * eps)) # should agree to ~6 digits
grad_check(MLP([2, 16, 2]), X[:20], y[:20])
  • Initialization must preserve signal variance across layers: Xavier/Glorot (Var⁡=2/(nin+nout)\operatorname{Var}=2/(n_{in}+n_{out})) for tanh/sigmoid, He (2/nin2/n_{in}) for ReLU. Zero initialization makes all neurons in a layer identical forever (symmetry), so weights need randomness (biases can be zero).
  • Width and depth: wider gives capacity and often easier optimization; deeper gives hierarchical features but needs residual connections and normalization to train (Training Deep Networks).
  • Parameter count for an MLP with sizes (784,512,512,10)(784,512,512,10): 784⋅512+512+512⋅512+512+512⋅10+10≈670784\cdot512+512+512\cdot512+512+512\cdot10+10\approx670k.
  • Input scaling: standardize inputs; unscaled features slow convergence (ill-conditioning).
  • Overfitting: MLPs memorize easily; use validation, weight decay, dropout, early stopping (ML Fundamentals).
  • Where MLPs fit today: they are rarely the whole model for images or text, but they are the feed-forward sublayer in every transformer, the heads of most models, and still strong for small tabular/regression problems and coordinate networks (NeRF-style).
  • Applying softmax and then a cross-entropy that applies softmax again.
  • Mismatched shapes silently broadcast (e.g. labels (B,1) vs (B,)).
  • Forgetting to cache pre-activations for the backward pass.
  • Learning rate too high for the init: loss becomes NaN. First test: can the net overfit 20 examples to zero loss? If not, there is a bug.
  • Initializing all weights equal.
  • A neuron is a linear map plus a nonlinearity; an MLP stacks them; without nonlinearity depth is useless.
  • The perceptron solves only linearly separable problems; hidden layers solve XOR and far more.
  • ReLU-family and GELU/SiLU activations avoid saturation; match the output activation to the loss.
  • Universal approximation guarantees existence, not learnability, efficiency, or generalization.
  • The backward pass of a layer is three matrix products; gradient checking catches mistakes.
  • Initialization scale and input scaling determine whether training starts well.
  1. Conceptual. Show that a network with only linear activations is equivalent to a single linear layer. What does this imply for depth?
  2. Construction. Hand-pick weights for a 2-2-1 ReLU network computing XOR on binary inputs.
  3. Hands-on. Extend the MLP class with minibatch SGD and momentum. Compare convergence with full-batch GD on spirals.
  4. Hands-on. Replace ReLU with sigmoid in a 10-layer MLP and plot the per-layer gradient norms. Explain the result.
  5. Conceptual. Why does the universal approximation theorem not make deep learning “solved”?
Solutions (2)
  1. h1=ReLU(x1+x2)h_1=\text{ReLU}(x_1+x_2), h2=ReLU(x1+x2−1)h_2=\text{ReLU}(x_1+x_2-1), output =h1−2h2=h_1-2h_2. Inputs: (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→2−2=02-2=0.
  • Goodfellow, Bengio, Courville, Deep Learning (2016), chapters 6 and 8.
  • Nielsen, Neural Networks and Deep Learning (free online book).
  • Zhang, Lipton, Li, Smola, Dive into Deep Learning (d2l.ai).
  • Cybenko, “Approximation by Superpositions of a Sigmoidal Function” (1989).
  • Hornik, “Approximation Capabilities of Multilayer Feedforward Networks” (1991).
  • Glorot and Bengio, “Understanding the difficulty of training deep feedforward neural networks” (2010).
  • He et al., “Delving Deep into Rectifiers” (2015).
  • Minsky and Papert, Perceptrons (1969).