Calculus & Optimization
What you’ll learn. The multivariable calculus needed for ML (gradients, Jacobians, Hessians, chain rule), what convexity buys you, why gradient descent works and when it is slow, and how SGD, momentum, RMSProp, and Adam/AdamW differ. Includes NumPy implementations you can run.
Derivatives as linear approximations
Section titled “Derivatives as linear approximations”For , the derivative is the best linear approximation: . Everything in multivariable calculus is this idea with matrices.
Gradient. For ,
The gradient points in the direction of steepest ascent, and is steepest descent: among unit directions , is maximized by (Cauchy–Schwarz). The gradient is orthogonal to level sets.
Jacobian. For , the Jacobian has entries , and . A neural-network layer’s Jacobian is how small changes in its input change its output; chaining layers multiplies Jacobians.
Chain rule. For : . For a scalar loss at the end, reverse-mode differentiation computes — a vector–Jacobian product without ever forming . This is backpropagation.
Hessian. For scalar , , symmetric when is . Second-order Taylor expansion:
The Hessian describes curvature. At a critical point (): is a local minimum, a maximum, mixed-sign eigenvalues a saddle. In high-dimensional deep-learning loss landscapes, saddle points are far more common than local minima, and most “bad” local minima are rarely the practical obstacle.
Useful gradients
Section titled “Useful gradients”| ( symmetric) | |
| (sigmoid) |
Always verify a hand-derived gradient numerically:
import numpy as np
def num_grad(f, x, eps=1e-6): g = np.zeros_like(x) for i in range(x.size): e = np.zeros_like(x); e.flat[i] = eps g.flat[i] = (f(x + e) - f(x - e)) / (2 * eps) # central difference return g
A = np.random.randn(4, 4); A = A @ A.Tf = lambda x: 0.5 * x @ A @ xx = np.random.randn(4)print(np.allclose(num_grad(f, x), A @ x, atol=1e-5)) # TrueConvexity
Section titled “Convexity”A set is convex if the segment between any two of its points lies in it. A function is convex if
equivalently (the function lies above its tangent), or everywhere.
Why care: for convex , every local minimum is global, and gradient-based methods have clean convergence guarantees. Convex examples: linear/ridge/lasso regression, logistic regression, SVM, softmax cross-entropy in the logits/linear-model parameters. Non-convex: neural networks as a function of weights (permutation symmetries alone guarantee non-convexity). We nonetheless optimize them with the same methods, and empirically it works well; theory lags behind practice.
Strong convexity and smoothness. is -smooth if is -Lipschitz (), and -strongly convex if . The ratio is the condition number and controls convergence speed.
Gradient descent
Section titled “Gradient descent”Step size. On an -smooth function, a step guarantees descent. Derivation: by smoothness, , which decreases when and is best at .
Rates. Convex and -smooth: . Strongly convex: linear (geometric) rate with , so you need about iterations. On a quadratic , GD behaves independently along each eigenvector: the error along eigendirection shrinks by per step. The stable step is limited by but progress along small- directions is slow. That is the “narrow valley zigzag.”
def gd(grad, x0, lr, steps): x = x0.copy(); path = [x.copy()] for _ in range(steps): x = x - lr * grad(x); path.append(x.copy()) return np.array(path)
A = np.diag([1.0, 50.0]) # condition number 50grad = lambda x: A @ xpath = gd(grad, np.array([5.0, 1.0]), lr=1/50, steps=200)print(path[-1]) # first coordinate still not near 0Preconditioning / Newton. If you rescale coordinates by curvature you remove the condition-number problem. Newton’s method does that fully: . It converges quadratically near a minimum but costs memory and time, impractical for . Adaptive optimizers (below) are cheap diagonal approximations of preconditioning; second-order-flavored methods (K-FAC, Shampoo and its variants, Muon-style orthogonalized updates) have gained real traction for large-model training in recent years, though AdamW remains the default.
Stochastic gradient descent
Section titled “Stochastic gradient descent”Machine-learning losses are averages: . Computing the full gradient costs . SGD uses a random minibatch :
The gradient estimate is unbiased but noisy. Consequences:
- Noise prevents converging to an exact point at a constant LR; the iterate hovers in a ball whose radius scales with and the variance. Decay the LR (see Training Deep Networks) or average iterates.
- Larger batches reduce variance, with diminishing returns once noise is below the signal. Rule of thumb: when you scale batch size by , scale the LR by about (linear scaling, for SGD) up to a point, with warmup. For Adam, a square-root scaling is a commonly cited heuristic. Treat both as starting points, not laws.
- Noise has a useful side effect: it helps escape sharp minima and saddles and is associated with better generalization.
- The cost per step is independent of , the crucial point for big data.
Momentum
Section titled “Momentum”Plain GD zigzags across steep directions and crawls along shallow ones. Heavy-ball momentum accumulates a velocity:
Directions where gradients consistently agree build up speed (effective step ); oscillating directions cancel. For quadratics, momentum improves the iteration count from to with tuned parameters. Nesterov momentum evaluates the gradient at the look-ahead point and achieves the optimal rate for smooth convex problems.
def sgd_momentum(grad, x0, lr, beta, steps): x, v = x0.copy(), np.zeros_like(x0) for _ in range(steps): v = beta * v + grad(x) x = x - lr * v return x
print(sgd_momentum(grad, np.array([5.0, 1.0]), lr=1/50, beta=0.9, steps=200))Compare with the plain GD result above: the first coordinate gets much closer to zero.
Adaptive methods
Section titled “Adaptive methods”Different parameters need different step sizes: rare features have small, infrequent gradients; some layers have gradients orders of magnitude larger than others.
AdaGrad divides by the root of accumulated squared gradients; effective LR only decreases. RMSProp replaces the sum with an exponential moving average. Adam (Kingma and Ba, 2014) combines momentum (first moment) with RMSProp (second moment):
Bias correction () fixes the underestimate from zero initialization in early steps. Defaults: , , ; LLM training often uses for faster adaptation and tolerates larger in low precision. Per-parameter updates have magnitude about regardless of gradient scale, which is why Adam is forgiving across architectures. It is closely related to sign descent.
AdamW (Loshchilov and Hutter, 2017) decouples weight decay: . With plain regularization inside the gradient, Adam’s normalization distorts the penalty; decoupled decay behaves as intended and is the standard for transformers.
def adam(grad, x0, lr=1e-2, b1=0.9, b2=0.999, eps=1e-8, steps=500): x = x0.copy(); m = np.zeros_like(x); v = np.zeros_like(x) for t in range(1, steps + 1): g = grad(x) m = b1 * m + (1 - b1) * g v = b2 * v + (1 - b2) * g * g mh, vh = m / (1 - b1**t), v / (1 - b2**t) x = x - lr * mh / (np.sqrt(vh) + eps) return x
print(adam(grad, np.array([5.0, 1.0])))Practical comparisons. SGD with momentum often generalizes slightly better on vision tasks (ResNets) when well tuned; Adam/AdamW dominates for transformers and sparse or heterogeneous-gradient problems. Adam memory cost is 2 extra copies of the parameters (fp32 states), a major part of training memory; 8-bit and factored variants (Adafactor) cut it.
Learning rate is the most important hyperparameter
Section titled “Learning rate is the most important hyperparameter”Too small: slow. Too large: oscillation or divergence. A quick diagnostic is the LR range test: sweep the LR exponentially over a few hundred steps and pick a value a bit below where the loss starts to rise. Schedules (warmup, cosine decay) are covered in Training Deep Networks.
Constrained optimization in one page
Section titled “Constrained optimization in one page”To minimize subject to , form and set . At the optimum the gradients of and are parallel. For inequality constraints, KKT conditions add and complementary slackness. The SVM dual and PCA derivation (Linear Algebra) both use this. Regularization is the penalty form of a norm constraint.
Pitfalls
Section titled “Pitfalls”- Exploding or vanishing gradients from products of Jacobians with large or small singular values; see Neural Networks and Training Deep Networks. Gradient clipping caps .
- Non-differentiable points (ReLU at 0, ): use subgradients; in practice it rarely matters.
- Forgetting to zero gradients or accidentally detaching the graph in autodiff frameworks.
- Judging an optimizer on training loss only; always watch validation.
- Numerical overflow in
exp: use log-sum-exp tricks.
Key takeaways
Section titled “Key takeaways”- The gradient is the linear approximation of a scalar function; the Jacobian generalizes it; the Hessian captures curvature.
- Convex problems have no spurious local minima; deep nets are non-convex but trainable with first-order methods.
- GD speed is governed by the condition number ; momentum and preconditioning attack it.
- SGD trades exact gradients for cheap, noisy ones; noise is both a nuisance and a regularizer.
- Adam normalizes per-parameter step sizes; AdamW decouples weight decay and is the default for transformers.
- Tune the learning rate first.
Exercises
Section titled “Exercises”- Conceptual. Why does GD with step fail to converge on a quadratic with ?
- Derivation. Compute the gradient and Hessian of the logistic loss and show it is convex.
- Hands-on. Minimize the Rosenbrock function with GD, momentum, and Adam from . Plot the trajectories.
- Hands-on. Implement the LR range test on a small MLP and report the LR you would choose.
- Conceptual. Explain why Adam’s update magnitude is roughly invariant to gradient scale, and one situation where that is harmful.
Solutions (1, 2)
- Along the top eigendirection the error is multiplied by , so it oscillates forever with constant magnitude. Above it grows.
- With : and with . Since and , , so is convex.
Further reading
Section titled “Further reading”- Boyd and Vandenberghe, Convex Optimization (2004), free online.
- Nocedal and Wright, Numerical Optimization (2nd ed., 2006).
- Goodfellow, Bengio, Courville, Deep Learning (2016), chapters 4 and 8.
- Kingma and Ba, “Adam: A Method for Stochastic Optimization” (2014).
- Loshchilov and Hutter, “Decoupled Weight Decay Regularization” (2017).
- Sebastian Ruder, “An overview of gradient descent optimization algorithms” (2016).
- Bottou, Curtis, Nocedal, “Optimization Methods for Large-Scale Machine Learning” (2018).