Neural Networks from Scratch
What you’ll learn. How a single artificial neuron works, why stacking them with nonlinearities gives expressive models, the common activation functions and their trade-offs, what the universal approximation theorem does and does not say, and how to implement a trained MLP in pure NumPy, including a manual backward pass that you can check numerically. Backpropagation then generalizes the manual derivation.
From logistic regression to a neuron
Section titled “From logistic regression to a neuron”A neuron computes : a linear function (weights , bias ) followed by a nonlinearity . With a sigmoid it is logistic regression (Classical Models). A layer applies neurons in parallel: with . A multilayer perceptron (MLP) composes layers:
input hidden 1 hidden 2 output x1 ---\ /-- h1 --\ /-- g1 --\ x2 ----+--+--- h2 ---+---+--- g2 ---+----> y_hat x3 ---/ \-- h3 --/ \-- g3 --/ W1,b1 relu W2,b2 relu W3,b3The whole network is a differentiable function of its parameters , trained by gradient descent on a loss (Calculus & Optimization).
The perceptron
Section titled “The perceptron”Rosenblatt’s perceptron (1958) is a neuron with a step activation, , with the update rule: for each misclassified example, . It converges if and only if the data is linearly separable (the perceptron convergence theorem). It cannot represent XOR: no single line separates from . Minsky and Papert’s 1969 analysis of such limits contributed to the first AI winter. A hidden layer fixes it: XOR AND(OR, NAND), two hidden units then one output.
import numpy as np
def perceptron(X, y, lr=1.0, epochs=20): w = np.zeros(X.shape[1]); b = 0.0 for _ in range(epochs): for xi, yi in zip(X, y): pred = int(w @ xi + b > 0) w += lr * (yi - pred) * xi b += lr * (yi - pred) return w, b
X = np.array([[0,0],[0,1],[1,0],[1,1]]); y_and = np.array([0,0,0,1])print(perceptron(X, y_and)) # learns AND; would never converge on XORWhy nonlinearity is essential
Section titled “Why nonlinearity is essential”Without , two layers collapse: , which is just one linear map. Depth gives power only through nonlinearities.
Activation functions
Section titled “Activation functions”| Name | Formula | Notes |
|---|---|---|
| Sigmoid | Output in (0,1); saturates (gradient for large ); not zero-centered. Used for binary output probabilities and gates. | |
| Tanh | Zero-centered; still saturates. Used in RNN cells. | |
| ReLU | Cheap, no saturation for , sparse activations. “Dying ReLU”: units stuck at 0 get zero gradient. | |
| Leaky ReLU / PReLU | Small slope for avoids dead units. | |
| GELU | Smooth ReLU-like; standard in BERT/GPT-style transformers. | |
| SiLU / Swish | Smooth, non-monotonic; widely used. | |
| SwiGLU (gated) | Gated linear unit variant; common in recent LLMs’ feed-forward blocks. | |
| Softmax | Output layer for multiclass probabilities. |
The vanishing gradient problem is visible from derivatives: , so multiplying many such factors through depth shrinks gradients exponentially. ReLU’s derivative is exactly 1 on the active side, a main reason deep networks became trainable (Training Deep Networks). Default today: ReLU/GELU/SiLU in hidden layers; the output activation matches the loss (identity for regression, sigmoid for binary, softmax for multiclass, folded into the loss for numerical stability).
Universal approximation
Section titled “Universal approximation”Theorem (Cybenko 1989; Hornik 1991; Leshno et al. 1993). A feedforward network with one hidden layer and a non-polynomial activation can approximate any continuous function on a compact set to arbitrary accuracy, given enough hidden units.
Intuition with ReLU in 1D: each hidden unit adds a “hinge”; a sum of hinges can form any piecewise-linear function, which approximates any continuous curve as hinges multiply. In dimensions each unit creates a half-space boundary and combinations carve out complex regions.
What it does not say:
- It does not bound the width; it may be exponentially large in .
- It does not say gradient descent will find the weights.
- It does not say the network will generalize from finite data.
Depth helps in practice: there are functions that deep networks represent with polynomially many units where shallow ones need exponentially many (depth separation results), and deep compositional hierarchies match the structure of images and language. The number of linear regions of a ReLU network grows exponentially with depth. See CNNs and RNNs for architectures that bake in useful structure.
Forward and backward pass in NumPy
Section titled “Forward and backward pass in NumPy”We implement a 2-hidden-layer MLP for classification with ReLU and softmax cross-entropy. Convention: batch , weights (row vectors), so a layer is .
Forward:
Backward (loss ). Using with one-hot (Information Theory), the chain rule for gives
and through ReLU .
import numpy as np
class MLP: def __init__(self, sizes, seed=0): rng = np.random.default_rng(seed) # He initialization: variance 2/fan_in suits ReLU self.W = [rng.normal(0, np.sqrt(2 / m), (m, n)) for m, n in zip(sizes[:-1], sizes[1:])] self.b = [np.zeros(n) for n in sizes[1:]]
def forward(self, X): self.H = [X]; self.Z = [] for i, (W, b) in enumerate(zip(self.W, self.b)): z = self.H[-1] @ W + b self.Z.append(z) self.H.append(np.maximum(z, 0) if i < len(self.W) - 1 else z) z = self.H[-1] - self.H[-1].max(1, keepdims=True) e = np.exp(z) return e / e.sum(1, keepdims=True)
def loss_and_grads(self, X, y): B = len(X) P = self.forward(X) loss = -np.log(P[np.arange(B), y] + 1e-12).mean() dZ = P.copy(); dZ[np.arange(B), y] -= 1; dZ /= B gW, gb = [None] * len(self.W), [None] * len(self.W) for i in reversed(range(len(self.W))): gW[i] = self.H[i].T @ dZ gb[i] = dZ.sum(0) if i > 0: dH = dZ @ self.W[i].T dZ = dH * (self.Z[i - 1] > 0) return loss, gW, gb
def step(self, gW, gb, lr): for i in range(len(self.W)): self.W[i] -= lr * gW[i]; self.b[i] -= lr * gb[i]Train on a nonlinear problem (two spirals)
Section titled “Train on a nonlinear problem (two spirals)”def spirals(n=300, noise=0.2, seed=0): rng = np.random.default_rng(seed) t = np.sqrt(rng.random(n)) * 3 * np.pi a = np.c_[t * np.cos(t), t * np.sin(t)] / 10 X = np.r_[a, -a] + noise * rng.normal(size=(2 * n, 2)) / 3 y = np.r_[np.zeros(n), np.ones(n)].astype(int) return X, y
X, y = spirals()net = MLP([2, 64, 64, 2])for epoch in range(2000): loss, gW, gb = net.loss_and_grads(X, y) net.step(gW, gb, lr=0.5) if epoch % 400 == 0: print(epoch, round(loss, 4), (net.forward(X).argmax(1) == y).mean())A linear classifier cannot exceed about 50-60% on this data; the MLP drives training accuracy near 100%. (Full-batch gradient descent here for simplicity; real training uses minibatches.)
Gradient check
Section titled “Gradient check”Always verify manual gradients against finite differences:
def grad_check(net, X, y, eps=1e-5): loss, gW, _ = net.loss_and_grads(X, y) W = net.W[0] i, j = 0, 3 W[i, j] += eps; lp, _, _ = net.loss_and_grads(X, y) W[i, j] -= 2*eps; lm, _, _ = net.loss_and_grads(X, y) W[i, j] += eps print(gW[0][i, j], (lp - lm) / (2 * eps)) # should agree to ~6 digitsgrad_check(MLP([2, 16, 2]), X[:20], y[:20])Practical design considerations
Section titled “Practical design considerations”- Initialization must preserve signal variance across layers: Xavier/Glorot () for tanh/sigmoid, He () for ReLU. Zero initialization makes all neurons in a layer identical forever (symmetry), so weights need randomness (biases can be zero).
- Width and depth: wider gives capacity and often easier optimization; deeper gives hierarchical features but needs residual connections and normalization to train (Training Deep Networks).
- Parameter count for an MLP with sizes : k.
- Input scaling: standardize inputs; unscaled features slow convergence (ill-conditioning).
- Overfitting: MLPs memorize easily; use validation, weight decay, dropout, early stopping (ML Fundamentals).
- Where MLPs fit today: they are rarely the whole model for images or text, but they are the feed-forward sublayer in every transformer, the heads of most models, and still strong for small tabular/regression problems and coordinate networks (NeRF-style).
Common bugs
Section titled “Common bugs”- Applying softmax and then a cross-entropy that applies softmax again.
- Mismatched shapes silently broadcast (e.g. labels
(B,1)vs(B,)). - Forgetting to cache pre-activations for the backward pass.
- Learning rate too high for the init: loss becomes NaN. First test: can the net overfit 20 examples to zero loss? If not, there is a bug.
- Initializing all weights equal.
Key takeaways
Section titled “Key takeaways”- A neuron is a linear map plus a nonlinearity; an MLP stacks them; without nonlinearity depth is useless.
- The perceptron solves only linearly separable problems; hidden layers solve XOR and far more.
- ReLU-family and GELU/SiLU activations avoid saturation; match the output activation to the loss.
- Universal approximation guarantees existence, not learnability, efficiency, or generalization.
- The backward pass of a layer is three matrix products; gradient checking catches mistakes.
- Initialization scale and input scaling determine whether training starts well.
Exercises
Section titled “Exercises”- Conceptual. Show that a network with only linear activations is equivalent to a single linear layer. What does this imply for depth?
- Construction. Hand-pick weights for a 2-2-1 ReLU network computing XOR on binary inputs.
- Hands-on. Extend the
MLPclass with minibatch SGD and momentum. Compare convergence with full-batch GD on spirals. - Hands-on. Replace ReLU with sigmoid in a 10-layer MLP and plot the per-layer gradient norms. Explain the result.
- Conceptual. Why does the universal approximation theorem not make deep learning “solved”?
Solutions (2)
- , , output . Inputs: (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→.
Further reading
Section titled “Further reading”- Goodfellow, Bengio, Courville, Deep Learning (2016), chapters 6 and 8.
- Nielsen, Neural Networks and Deep Learning (free online book).
- Zhang, Lipton, Li, Smola, Dive into Deep Learning (d2l.ai).
- Cybenko, “Approximation by Superpositions of a Sigmoidal Function” (1989).
- Hornik, “Approximation Capabilities of Multilayer Feedforward Networks” (1991).
- Glorot and Bengio, “Understanding the difficulty of training deep feedforward neural networks” (2010).
- He et al., “Delving Deep into Rectifiers” (2015).
- Minsky and Papert, Perceptrons (1969).