Skip to content

Linear Algebra for ML

What you’ll learn. How to think of vectors and matrices as geometry and as linear maps, not just arrays; how norms and inner products measure size and similarity; what eigenvalues and singular values tell you; and how these tools produce PCA, low-rank approximations, and the matrix multiplications that dominate deep learning compute. Code is NumPy.

A dataset with nn examples and dd features is a matrix X∈Rn×dX \in \mathbb{R}^{n \times d}. A linear model is y^=Xw\hat y = Xw. A neural-network layer is h=σ(Wx+b)h = \sigma(Wx + b). A transformer is, in compute terms, mostly matrix multiplications. When you understand what a matrix does to space, you can reason about conditioning, rank, redundancy, and why some optimization problems are hard.

A vector x∈Rdx \in \mathbb{R}^d is an ordered list of numbers, but two geometric readings are more useful: a point in space, and an arrow with direction and length.

Key operations:

  • Addition and scaling: αx+βy\alpha x + \beta y (a linear combination).
  • Inner (dot) product: ⟨x,y⟩=x⊤y=∑ixiyi=∥x∥ ∥y∥cos⁡θ\langle x, y\rangle = x^\top y = \sum_i x_i y_i = \|x\|\,\|y\|\cos\theta.

The last identity is the reason dot products measure similarity: if two vectors point the same way, cos⁡θ=1\cos\theta = 1; orthogonal gives 0. Attention scores, embedding retrieval, and cosine similarity are all dot products.

Span, independence, basis. The span of vectors is everything reachable by linear combinations. Vectors are linearly independent if none lies in the span of the others. A basis is an independent set that spans the space; dd independent vectors form a basis of Rd\mathbb{R}^d. The same vector has different coordinates in different bases; much of linear algebra is choosing a good basis (eigenbasis, singular vectors, principal components).

A norm ∥⋅∥\|\cdot\| measures length and satisfies non-negativity, homogeneity ∥αx∥=∣α∣∥x∥\|\alpha x\| = |\alpha|\|x\|, and the triangle inequality.

Norm Definition ML use
ℓ2\ell_2 ∑xi2\sqrt{\sum x_i^2} Euclidean distance, weight decay, gradient clipping
ℓ1\ell_1 ∑∣xi∣\sum \lvert x_i\rvert Lasso, sparsity
ℓ∞\ell_\infty max⁡i∣xi∣\max_i \lvert x_i\rvert Adversarial perturbation budgets
ℓ0\ell_0 “norm” number of nonzeros Sparsity (not a true norm; non-convex)
Frobenius ∑ijAij2\sqrt{\sum_{ij} A_{ij}^2} Matrix error, weight decay on matrices
Spectral σmax⁡(A)\sigma_{\max}(A) Lipschitz bounds, spectral normalization

Why does ℓ1\ell_1 give sparsity? The unit ℓ1\ell_1 ball has corners on the axes; the minimum of a loss subject to an ℓ1\ell_1 constraint tends to touch a corner, where some coordinates are exactly zero. The ℓ2\ell_2 ball is round, so no coordinate is preferred.

A matrix A∈Rm×nA \in \mathbb{R}^{m\times n} represents a linear map x↦Axx \mapsto Ax from Rn\mathbb{R}^n to Rm\mathbb{R}^m. Linear means A(αx+βy)=αAx+βAyA(\alpha x+\beta y)=\alpha Ax+\beta Ay. Read the columns of AA as where the standard basis vectors land; AxAx is then a combination of those columns with weights xx.

Matrix product C=ABC = AB composes maps: apply BB, then AA. It is associative but not commutative. Shapes: (m×k)(k×n)=(m×n)(m\times k)(k\times n) = (m\times n). Shape-checking is the number one debugging tool in ML code.

Useful facts:

  • Transpose: (AB)⊤=B⊤A⊤(AB)^\top = B^\top A^\top.
  • Inverse exists only for square full-rank matrices; (AB)−1=B−1A−1(AB)^{-1}=B^{-1}A^{-1}. In practice you almost never form an inverse; you solve Ax=bAx=b with a factorization (np.linalg.solve).
  • Rank is the dimension of the column space (= dimension of the row space). Rank r<min⁡(m,n)r < \min(m,n) means information is lost or redundant.
  • Null space: {x:Ax=0}\{x : Ax = 0\}. Dimension is n−rn - r (rank-nullity).
  • Trace: tr⁡(A)=∑iAii\operatorname{tr}(A)=\sum_i A_{ii}; cyclic, tr⁡(ABC)=tr⁡(BCA)\operatorname{tr}(ABC)=\operatorname{tr}(BCA); equals the sum of eigenvalues.
  • Determinant: the signed volume scaling factor; zero iff singular; product of eigenvalues.

A matrix QQ is orthogonal if Q⊤Q=IQ^\top Q = I. Orthogonal maps are rotations/reflections: they preserve lengths and angles, ∥Qx∥=∥x∥\|Qx\|=\|x\|. They are perfectly conditioned, which is why orthogonal initialization and orthogonal parametrizations are sometimes used to keep signals and gradients from blowing up or vanishing in deep or recurrent nets.

Projecting bb onto the column space of AA (full column rank) gives A(A⊤A)−1A⊤bA(A^\top A)^{-1}A^\top b. The least-squares problem min⁡w∥Xw−y∥22\min_w \|Xw - y\|_2^2 is exactly this projection. Setting the gradient 2X⊤(Xw−y)2X^\top(Xw-y) to zero gives the normal equations

X⊤X w=X⊤y⇒w⋆=(X⊤X)−1X⊤y.X^\top X\, w = X^\top y \quad\Rightarrow\quad w^\star = (X^\top X)^{-1}X^\top y.

Geometrically the residual y−Xw⋆y - Xw^\star is orthogonal to every column of XX. Numerically, forming X⊤XX^\top X squares the condition number, so use QR or SVD (np.linalg.lstsq) rather than the normal equations for ill-conditioned problems.

For a square AA, a nonzero vv with Av=λvAv=\lambda v is an eigenvector with eigenvalue λ\lambda: a direction the map only stretches. If AA has nn independent eigenvectors, A=VΛV−1A = V\Lambda V^{-1}.

For a symmetric matrix (covariance matrices, Hessians, Gram matrices) the spectral theorem says: eigenvalues are real, eigenvectors can be chosen orthonormal, and

A=QΛQ⊤=∑iλiqiqi⊤.A = Q\Lambda Q^\top = \sum_i \lambda_i q_i q_i^\top.

So a symmetric matrix is “rotate into the eigenbasis, scale each axis independently, rotate back.”

Positive semi-definite (PSD): x⊤Ax≥0x^\top A x \ge 0 for all xx, equivalently all eigenvalues ≥0\ge 0. Covariance matrices and X⊤XX^\top X are always PSD. Positive definite (all >0>0) matrices are invertible and define a valid inner product and a convex quadratic 12x⊤Ax\tfrac12 x^\top A x.

Quadratic forms. The level sets of x⊤Axx^\top A x for PD AA are ellipsoids with axes along eigenvectors and semi-axis lengths 1/λi1/\sqrt{\lambda_i}. This picture is the best intuition for gradient descent on a quadratic: it zigzags when the ellipse is elongated, i.e. when λmax⁡/λmin⁡\lambda_{\max}/\lambda_{\min} is large. That ratio is the condition number κ\kappa; see Calculus & Optimization.

Power iteration finds the dominant eigenvector: repeat v←Av/∥Av∥v \leftarrow Av/\|Av\|. It converges at a rate governed by ∣λ2/λ1∣|\lambda_2/\lambda_1| and underlies PageRank and many large-scale eigen-solvers.

import numpy as np
A = np.array([[2.0, 1.0], [1.0, 3.0]])
lam, Q = np.linalg.eigh(A) # eigh for symmetric: sorted, orthonormal
print(lam) # real eigenvalues
print(np.allclose(Q @ np.diag(lam) @ Q.T, A)) # True

Eigendecomposition needs square matrices. The SVD works for every matrix:

A=UΣV⊤,U∈Rm×m, V∈Rn×n orthogonal, Σ=diag⁡(σ1≥σ2≥⋯≥0).A = U\Sigma V^\top,\quad U\in\mathbb{R}^{m\times m},\ V\in\mathbb{R}^{n\times n}\text{ orthogonal},\ \Sigma = \operatorname{diag}(\sigma_1\ge\sigma_2\ge\dots\ge 0).

Interpretation: any linear map is rotate (V⊤V^\top), stretch along axes (Σ\Sigma), rotate (UU). The σi\sigma_i are singular values; viv_i are right singular vectors, uiu_i left singular vectors. Relations to eigen-stuff:

  • A⊤A=VΣ⊤ΣV⊤A^\top A = V\Sigma^\top\Sigma V^\top: right singular vectors are eigenvectors of A⊤AA^\top A, with eigenvalues σi2\sigma_i^2.
  • rank⁡(A)\operatorname{rank}(A) = number of nonzero singular values.
  • ∥A∥2=σ1\|A\|_2=\sigma_1, ∥A∥F2=∑σi2\|A\|_F^2=\sum\sigma_i^2, κ(A)=σ1/σr\kappa(A)=\sigma_1/\sigma_r.

The “economy” or thin SVD keeps only min⁡(m,n)\min(m,n) columns and is what libraries return by default if you ask.

Writing A=∑iσiuivi⊤A=\sum_i \sigma_i u_i v_i^\top as a sum of rank-one pieces, the best rank-kk approximation in both Frobenius and spectral norm is to keep the top kk terms:

Ak=∑i=1kσiuivi⊤,∥A−Ak∥F2=∑i>kσi2.A_k = \sum_{i=1}^k \sigma_i u_i v_i^\top,\qquad \|A-A_k\|_F^2=\sum_{i>k}\sigma_i^2.

A rank-kk matrix of size m×nm\times n needs k(m+n+1)k(m+n+1) numbers instead of mnmn. This is the basis for image compression, latent semantic analysis, recommender-system matrix factorization, and LoRA fine-tuning, where a weight update ΔW\Delta W is constrained to be BABA with small inner dimension rr.

rng = np.random.default_rng(0)
A = rng.normal(size=(100, 5)) @ rng.normal(size=(5, 80)) # rank 5
A += 0.01 * rng.normal(size=A.shape) # small noise
U, s, Vt = np.linalg.svd(A, full_matrices=False)
print(np.round(s[:8], 2)) # 5 large values, then a cliff
k = 5
A_k = (U[:, :k] * s[:k]) @ Vt[:k]
print(np.linalg.norm(A - A_k) / np.linalg.norm(A)) # small relative error

A+=VΣ+U⊤A^+ = V\Sigma^+U^\top, where Σ+\Sigma^+ inverts the nonzero singular values. w=A+yw=A^+y is the minimum-norm least-squares solution. This explains why gradient descent on an underdetermined linear regression, started from zero, converges to the minimum-norm interpolating solution, a small piece of the “implicit regularization” story.

Let X∈Rn×dX\in\mathbb{R}^{n\times d} be centered (columns have zero mean). The sample covariance is C=1n−1X⊤XC=\tfrac1{n-1}X^\top X. Principal component analysis finds orthonormal directions of maximum variance. The first direction solves

max⁡∥w∥=1w⊤Cw,\max_{\|w\|=1} w^\top C w,

whose solution is the top eigenvector of CC (Rayleigh quotient), with variance λ1\lambda_1. Subsequent components are the next eigenvectors, orthogonal to earlier ones.

Via SVD of the centered data, X=UΣV⊤X=U\Sigma V^\top: the columns of VV are the principal directions, the eigenvalues of CC are σi2/(n−1)\sigma_i^2/(n-1), and the projected data (scores) are XVk=UkΣkXV_k = U_k\Sigma_k. Using SVD is numerically preferable to forming CC.

def pca(X, k):
Xc = X - X.mean(axis=0)
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
var = s**2 / (len(X) - 1)
return Xc @ Vt[:k].T, Vt[:k], var / var.sum() # scores, components, explained ratio
X = rng.normal(size=(500, 3)) @ np.array([[3, 0, 0], [1, 1, 0], [0, 0, 0.1]])
Z, comps, ratio = pca(X, 2)
print(ratio)

Two equivalent views of PCA: maximum variance in the projection, and minimum reconstruction error (Eckart–Young on the centered matrix). Practical notes: standardize features if units differ, since PCA is scale-sensitive; PCA only captures linear structure; choose kk by explained-variance ratio or a scree plot cliff. More on PCA as a model in Classical Models.

Two derivatives are worth memorizing (denominator layout, gradient as column):

  • ∇x(a⊤x)=a\nabla_x (a^\top x) = a
  • ∇x(x⊤Ax)=(A+A⊤)x\nabla_x (x^\top A x) = (A + A^\top)x, which is 2Ax2Ax for symmetric AA.
  • For f(W)=∥Wx−y∥2f(W) = \|Wx - y\|^2, ∇Wf=2(Wx−y)x⊤\nabla_W f = 2(Wx-y)x^\top.

A reliable strategy: derive for scalars/indices, then match shapes. The full treatment (Jacobians, Hessians) is in Calculus & Optimization, and its use in Backpropagation.

  • Batching is matmul. A batch of BB inputs through a layer is XW⊤XW^\top with X∈RB×dinX\in\mathbb{R}^{B\times d_{in}}. GPUs are matmul machines; FLOPs for (m×k)(k×n)(m\times k)(k\times n) are about 2mkn2mkn.
  • Broadcasting silently expands dimensions. Many subtle bugs are an unintended (n,1)+(n,)(n,1)+(n,) giving (n,n)(n,n). Assert shapes.
  • Numerical precision. In float16/bfloat16, ill-conditioned systems and large sums lose accuracy; see Training Deep Networks.
  • Never invert explicitly. Use solve, lstsq, Cholesky for PD systems.
  • Einsum is clearer for higher-rank tensors: np.einsum("bij,bjk->bik", A, B) is batched matmul.
  • A matrix is a linear map; its columns show where basis vectors go, and products compose maps.
  • Dot products measure alignment; norms measure size; the choice of norm shapes regularization behavior (ℓ1\ell_1 sparse, ℓ2\ell_2 smooth).
  • Symmetric matrices diagonalize with orthonormal eigenvectors; PSD means non-negative eigenvalues, as for covariances and Gram matrices.
  • SVD exists for every matrix and gives the optimal low-rank approximation (Eckart–Young); singular values reveal rank and conditioning.
  • PCA is the eigendecomposition of the covariance or equivalently the SVD of centered data.
  • Prefer factorizations over explicit inverses; mind conditioning.
  1. Conceptual. Why is A⊤AA^\top A always positive semi-definite? When is it positive definite?
  2. Conceptual. A 1000×10001000\times1000 matrix has singular values that decay as σi=1/i2\sigma_i = 1/i^2. Roughly what rank-kk do you need to capture 99.9% of ∥A∥F2\|A\|_F^2?
  3. Hands-on. Implement power iteration in NumPy and compare to np.linalg.eigh on a random symmetric matrix. How does convergence speed depend on λ2/λ1\lambda_2/\lambda_1?
  4. Hands-on. Take a grayscale image as a matrix, compute its SVD, and plot reconstruction error versus kk. At what kk is the result visually acceptable, and what is the compression ratio?
  5. Derivation. Show that the first principal component maximizes w⊤Cww^\top C w subject to ∥w∥=1\|w\|=1 using a Lagrange multiplier.
Solutions (1, 2, 5)
  1. x⊤A⊤Ax=∥Ax∥2≥0x^\top A^\top A x=\|Ax\|^2\ge0. It is strictly positive for all x≠0x\neq0 iff AA has full column rank (null space is {0}\{0\}).
  2. ∑σi2=∑1/i4≈1.0823\sum\sigma_i^2=\sum 1/i^4\approx1.0823. The tail beyond kk is about ∫k∞x−4dx=1/(3k3)\int_k^\infty x^{-4}dx = 1/(3k^3). Need that below 0.001×1.080.001\times1.08, so k3>308k^3 > 308, k≈7k\approx 7.
  3. Lagrangian L=w⊤Cw−λ(w⊤w−1)L=w^\top Cw-\lambda(w^\top w-1); ∇wL=2Cw−2λw=0\nabla_w L=2Cw-2\lambda w=0 gives Cw=λwCw=\lambda w. Then w⊤Cw=λw^\top Cw=\lambda, so maximizing picks the largest eigenvalue.
  • Gilbert Strang, Linear Algebra and Its Applications and MIT OCW 18.06 lectures.
  • Gilbert Strang, Linear Algebra and Learning from Data (2019).
  • Trefethen and Bau, Numerical Linear Algebra (1997).
  • Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapters 2–4 and 10.
  • Golub and Van Loan, Matrix Computations (reference).
  • Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021).