Skip to content

Calculus & Optimization

What you’ll learn. The multivariable calculus needed for ML (gradients, Jacobians, Hessians, chain rule), what convexity buys you, why gradient descent works and when it is slow, and how SGD, momentum, RMSProp, and Adam/AdamW differ. Includes NumPy implementations you can run.

For f:R→Rf:\mathbb{R}\to\mathbb{R}, the derivative is the best linear approximation: f(x+h)≈f(x)+f′(x)hf(x+h)\approx f(x)+f'(x)h. Everything in multivariable calculus is this idea with matrices.

Gradient. For f:Rn→Rf:\mathbb{R}^n\to\mathbb{R},

f(x+h)≈f(x)+∇f(x)⊤h,∇f=(∂f∂x1,…,∂f∂xn)⊤.f(x+h)\approx f(x)+\nabla f(x)^\top h,\qquad \nabla f=\Big(\tfrac{\partial f}{\partial x_1},\dots,\tfrac{\partial f}{\partial x_n}\Big)^\top.

The gradient points in the direction of steepest ascent, and −∇f-\nabla f is steepest descent: among unit directions uu, ∇f⊤u\nabla f^\top u is maximized by u=∇f/∥∇f∥u=\nabla f/\|\nabla f\| (Cauchy–Schwarz). The gradient is orthogonal to level sets.

Jacobian. For f:Rn→Rmf:\mathbb{R}^n\to\mathbb{R}^m, the Jacobian J∈Rm×nJ\in\mathbb{R}^{m\times n} has entries Jij=∂fi/∂xjJ_{ij}=\partial f_i/\partial x_j, and f(x+h)≈f(x)+Jhf(x+h)\approx f(x)+J h. A neural-network layer’s Jacobian is how small changes in its input change its output; chaining layers multiplies Jacobians.

Chain rule. For h=g∘fh = g\circ f: Jh(x)=Jg(f(x)) Jf(x)J_h(x)=J_g(f(x))\,J_f(x). For a scalar loss LL at the end, reverse-mode differentiation computes ∇xL=Jf⊤∇yL\nabla_x L = J_f^\top \nabla_y L — a vector–Jacobian product without ever forming JJ. This is backpropagation.

Hessian. For scalar ff, Hij=∂2f/∂xi∂xjH_{ij}=\partial^2 f/\partial x_i\partial x_j, symmetric when ff is C2C^2. Second-order Taylor expansion:

f(x+h)≈f(x)+∇f⊤h+12h⊤Hh.f(x+h)\approx f(x)+\nabla f^\top h+\tfrac12 h^\top H h.

The Hessian describes curvature. At a critical point (∇f=0\nabla f=0): H≻0H\succ0 is a local minimum, H≺0H\prec0 a maximum, mixed-sign eigenvalues a saddle. In high-dimensional deep-learning loss landscapes, saddle points are far more common than local minima, and most “bad” local minima are rarely the practical obstacle.

f(x)f(x) ∇f\nabla f
a⊤xa^\top x aa
12x⊤Ax\tfrac12 x^\top A x (AA symmetric) AxAx
12∥Xw−y∥2\tfrac12\|Xw-y\|^2 X⊤(Xw−y)X^\top(Xw-y)
log⁡∑iexi\log\sum_i e^{x_i} softmax⁡(x)\operatorname{softmax}(x)
σ(z)\sigma(z) (sigmoid) σ(z)(1−σ(z))\sigma(z)(1-\sigma(z))

Always verify a hand-derived gradient numerically:

import numpy as np
def num_grad(f, x, eps=1e-6):
g = np.zeros_like(x)
for i in range(x.size):
e = np.zeros_like(x); e.flat[i] = eps
g.flat[i] = (f(x + e) - f(x - e)) / (2 * eps) # central difference
return g
A = np.random.randn(4, 4); A = A @ A.T
f = lambda x: 0.5 * x @ A @ x
x = np.random.randn(4)
print(np.allclose(num_grad(f, x), A @ x, atol=1e-5)) # True

A set is convex if the segment between any two of its points lies in it. A function is convex if

f(θx+(1−θ)y)≤θf(x)+(1−θ)f(y),θ∈[0,1],f(\theta x+(1-\theta)y)\le\theta f(x)+(1-\theta)f(y),\quad\theta\in[0,1],

equivalently f(y)≥f(x)+∇f(x)⊤(y−x)f(y)\ge f(x)+\nabla f(x)^\top(y-x) (the function lies above its tangent), or H⪰0H\succeq0 everywhere.

Why care: for convex ff, every local minimum is global, and gradient-based methods have clean convergence guarantees. Convex examples: linear/ridge/lasso regression, logistic regression, SVM, softmax cross-entropy in the logits/linear-model parameters. Non-convex: neural networks as a function of weights (permutation symmetries alone guarantee non-convexity). We nonetheless optimize them with the same methods, and empirically it works well; theory lags behind practice.

Strong convexity and smoothness. ff is LL-smooth if ∇f\nabla f is LL-Lipschitz (H⪯LIH\preceq LI), and μ\mu-strongly convex if H⪰μIH\succeq\mu I. The ratio κ=L/μ\kappa=L/\mu is the condition number and controls convergence speed.

θt+1=θt−η ∇L(θt).\theta_{t+1}=\theta_t-\eta\,\nabla L(\theta_t).

Step size. On an LL-smooth function, a step η≤1/L\eta\le 1/L guarantees descent. Derivation: by smoothness, L(θ−ηg)≤L(θ)−η∥g∥2+Lη22∥g∥2L(\theta-\eta g)\le L(\theta)-\eta\|g\|^2+\tfrac{L\eta^2}{2}\|g\|^2, which decreases when η<2/L\eta<2/L and is best at η=1/L\eta=1/L.

Rates. Convex and LL-smooth: L(θT)−L⋆=O(1/T)L(\theta_T)-L^\star=O(1/T). Strongly convex: linear (geometric) rate (1−1κ)T\big(1-\tfrac1\kappa\big)^T with η=1/L\eta=1/L, so you need about κlog⁡(1/ϵ)\kappa\log(1/\epsilon) iterations. On a quadratic 12θ⊤Aθ\tfrac12\theta^\top A\theta, GD behaves independently along each eigenvector: the error along eigendirection ii shrinks by (1−ηλi)(1-\eta\lambda_i) per step. The stable step is limited by λmax⁡\lambda_{\max} but progress along small-λ\lambda directions is slow. That is the “narrow valley zigzag.”

def gd(grad, x0, lr, steps):
x = x0.copy(); path = [x.copy()]
for _ in range(steps):
x = x - lr * grad(x); path.append(x.copy())
return np.array(path)
A = np.diag([1.0, 50.0]) # condition number 50
grad = lambda x: A @ x
path = gd(grad, np.array([5.0, 1.0]), lr=1/50, steps=200)
print(path[-1]) # first coordinate still not near 0

Preconditioning / Newton. If you rescale coordinates by curvature you remove the condition-number problem. Newton’s method does that fully: θ←θ−H−1∇L\theta\leftarrow\theta-H^{-1}\nabla L. It converges quadratically near a minimum but costs O(n2)O(n^2) memory and O(n3)O(n^3) time, impractical for n∼109n\sim10^9. Adaptive optimizers (below) are cheap diagonal approximations of preconditioning; second-order-flavored methods (K-FAC, Shampoo and its variants, Muon-style orthogonalized updates) have gained real traction for large-model training in recent years, though AdamW remains the default.

Machine-learning losses are averages: L(θ)=1N∑iℓ(θ;xi,yi)L(\theta)=\tfrac1N\sum_i \ell(\theta;x_i,y_i). Computing the full gradient costs O(N)O(N). SGD uses a random minibatch BB:

gB=1∣B∣∑i∈B∇ℓi,E[gB]=∇L,Var⁡[gB]∝1∣B∣.g_B=\tfrac1{|B|}\sum_{i\in B}\nabla\ell_i,\qquad \mathbb{E}[g_B]=\nabla L,\qquad \operatorname{Var}[g_B]\propto\tfrac1{|B|}.

The gradient estimate is unbiased but noisy. Consequences:

  • Noise prevents converging to an exact point at a constant LR; the iterate hovers in a ball whose radius scales with η\eta and the variance. Decay the LR (see Training Deep Networks) or average iterates.
  • Larger batches reduce variance, with diminishing returns once noise is below the signal. Rule of thumb: when you scale batch size by kk, scale the LR by about kk (linear scaling, for SGD) up to a point, with warmup. For Adam, a square-root scaling is a commonly cited heuristic. Treat both as starting points, not laws.
  • Noise has a useful side effect: it helps escape sharp minima and saddles and is associated with better generalization.
  • The cost per step is independent of NN, the crucial point for big data.

Plain GD zigzags across steep directions and crawls along shallow ones. Heavy-ball momentum accumulates a velocity:

vt+1=βvt+gt,θt+1=θt−ηvt+1.v_{t+1}=\beta v_t+g_t,\qquad\theta_{t+1}=\theta_t-\eta v_{t+1}.

Directions where gradients consistently agree build up speed (effective step η/(1−β)\eta/(1-\beta)); oscillating directions cancel. For quadratics, momentum improves the iteration count from O(κ)O(\kappa) to O(κ)O(\sqrt\kappa) with tuned parameters. Nesterov momentum evaluates the gradient at the look-ahead point θt−ηβvt\theta_t-\eta\beta v_t and achieves the optimal O(1/T2)O(1/T^2) rate for smooth convex problems.

def sgd_momentum(grad, x0, lr, beta, steps):
x, v = x0.copy(), np.zeros_like(x0)
for _ in range(steps):
v = beta * v + grad(x)
x = x - lr * v
return x
print(sgd_momentum(grad, np.array([5.0, 1.0]), lr=1/50, beta=0.9, steps=200))

Compare with the plain GD result above: the first coordinate gets much closer to zero.

Different parameters need different step sizes: rare features have small, infrequent gradients; some layers have gradients orders of magnitude larger than others.

AdaGrad divides by the root of accumulated squared gradients; effective LR only decreases. RMSProp replaces the sum with an exponential moving average. Adam (Kingma and Ba, 2014) combines momentum (first moment) with RMSProp (second moment):

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,m^t=mt1−β1t,v^t=vt1−β2t,θt+1=θt−η m^tv^t+ϵ.\begin{aligned} m_t&=\beta_1 m_{t-1}+(1-\beta_1)g_t,\\ v_t&=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,\\ \hat m_t&=\frac{m_t}{1-\beta_1^t},\quad \hat v_t=\frac{v_t}{1-\beta_2^t},\\ \theta_{t+1}&=\theta_t-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}. \end{aligned}

Bias correction (1−βt1-\beta^t) fixes the underestimate from zero initialization in early steps. Defaults: β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−8\epsilon=10^{-8}; LLM training often uses β2≈0.95\beta_2\approx0.95 for faster adaptation and tolerates larger ϵ\epsilon in low precision. Per-parameter updates have magnitude about η\eta regardless of gradient scale, which is why Adam is forgiving across architectures. It is closely related to sign descent.

AdamW (Loshchilov and Hutter, 2017) decouples weight decay: θ←θ−η(m^/(v^+ϵ)+λθ)\theta\leftarrow\theta-\eta(\hat m/(\sqrt{\hat v}+\epsilon)+\lambda\theta). With plain ℓ2\ell_2 regularization inside the gradient, Adam’s normalization distorts the penalty; decoupled decay behaves as intended and is the standard for transformers.

def adam(grad, x0, lr=1e-2, b1=0.9, b2=0.999, eps=1e-8, steps=500):
x = x0.copy(); m = np.zeros_like(x); v = np.zeros_like(x)
for t in range(1, steps + 1):
g = grad(x)
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g * g
mh, vh = m / (1 - b1**t), v / (1 - b2**t)
x = x - lr * mh / (np.sqrt(vh) + eps)
return x
print(adam(grad, np.array([5.0, 1.0])))

Practical comparisons. SGD with momentum often generalizes slightly better on vision tasks (ResNets) when well tuned; Adam/AdamW dominates for transformers and sparse or heterogeneous-gradient problems. Adam memory cost is 2 extra copies of the parameters (fp32 states), a major part of training memory; 8-bit and factored variants (Adafactor) cut it.

Learning rate is the most important hyperparameter

Section titled “Learning rate is the most important hyperparameter”

Too small: slow. Too large: oscillation or divergence. A quick diagnostic is the LR range test: sweep the LR exponentially over a few hundred steps and pick a value a bit below where the loss starts to rise. Schedules (warmup, cosine decay) are covered in Training Deep Networks.

To minimize f(x)f(x) subject to g(x)=0g(x)=0, form L=f+λg\mathcal L=f+\lambda g and set ∇xL=0\nabla_x\mathcal L=0. At the optimum the gradients of ff and gg are parallel. For inequality constraints, KKT conditions add λ≥0\lambda\ge0 and complementary slackness. The SVM dual and PCA derivation (Linear Algebra) both use this. Regularization min⁡L+λ∥θ∥2\min L+\lambda\|\theta\|^2 is the penalty form of a norm constraint.

  • Exploding or vanishing gradients from products of Jacobians with large or small singular values; see Neural Networks and Training Deep Networks. Gradient clipping caps ∥g∥\|g\|.
  • Non-differentiable points (ReLU at 0, ℓ1\ell_1): use subgradients; in practice it rarely matters.
  • Forgetting to zero gradients or accidentally detaching the graph in autodiff frameworks.
  • Judging an optimizer on training loss only; always watch validation.
  • Numerical overflow in exp: use log-sum-exp tricks.
  • The gradient is the linear approximation of a scalar function; the Jacobian generalizes it; the Hessian captures curvature.
  • Convex problems have no spurious local minima; deep nets are non-convex but trainable with first-order methods.
  • GD speed is governed by the condition number κ\kappa; momentum and preconditioning attack it.
  • SGD trades exact gradients for cheap, noisy ones; noise is both a nuisance and a regularizer.
  • Adam normalizes per-parameter step sizes; AdamW decouples weight decay and is the default for transformers.
  • Tune the learning rate first.
  1. Conceptual. Why does GD with step η=2/L\eta=2/L fail to converge on a quadratic with λmax⁡=L\lambda_{\max}=L?
  2. Derivation. Compute the gradient and Hessian of the logistic loss ℓ(w)=log⁡(1+e−y w⊤x)\ell(w)=\log(1+e^{-y\,w^\top x}) and show it is convex.
  3. Hands-on. Minimize the Rosenbrock function f(x,y)=(1−x)2+100(y−x2)2f(x,y)=(1-x)^2+100(y-x^2)^2 with GD, momentum, and Adam from (−1.5,2)(-1.5, 2). Plot the trajectories.
  4. Hands-on. Implement the LR range test on a small MLP and report the LR you would choose.
  5. Conceptual. Explain why Adam’s update magnitude is roughly invariant to gradient scale, and one situation where that is harmful.
Solutions (1, 2)
  1. Along the top eigendirection the error is multiplied by 1−ηλmax⁡=1−2=−11-\eta\lambda_{\max}=1-2=-1, so it oscillates forever with constant magnitude. Above 2/L2/L it grows.
  2. With p=σ(−y w⊤x)p=\sigma(-y\,w^\top x): ∇ℓ=−y p x\nabla\ell=-y\,p\,x and H=σ(z)(1−σ(z)) xx⊤H=\sigma(z)(1-\sigma(z))\,xx^\top with z=w⊤xz=w^\top x. Since σ(1−σ)>0\sigma(1-\sigma)>0 and xx⊤⪰0xx^\top\succeq0, H⪰0H\succeq0, so ℓ\ell is convex.
  • Boyd and Vandenberghe, Convex Optimization (2004), free online.
  • Nocedal and Wright, Numerical Optimization (2nd ed., 2006).
  • Goodfellow, Bengio, Courville, Deep Learning (2016), chapters 4 and 8.
  • Kingma and Ba, “Adam: A Method for Stochastic Optimization” (2014).
  • Loshchilov and Hutter, “Decoupled Weight Decay Regularization” (2017).
  • Sebastian Ruder, “An overview of gradient descent optimization algorithms” (2016).
  • Bottou, Curtis, Nocedal, “Optimization Methods for Large-Scale Machine Learning” (2018).