Linear Algebra for ML
What you’ll learn. How to think of vectors and matrices as geometry and as linear maps, not just arrays; how norms and inner products measure size and similarity; what eigenvalues and singular values tell you; and how these tools produce PCA, low-rank approximations, and the matrix multiplications that dominate deep learning compute. Code is NumPy.
Why linear algebra is the language of ML
Section titled “Why linear algebra is the language of ML”A dataset with examples and features is a matrix . A linear model is . A neural-network layer is . A transformer is, in compute terms, mostly matrix multiplications. When you understand what a matrix does to space, you can reason about conditioning, rank, redundancy, and why some optimization problems are hard.
Vectors
Section titled “Vectors”A vector is an ordered list of numbers, but two geometric readings are more useful: a point in space, and an arrow with direction and length.
Key operations:
- Addition and scaling: (a linear combination).
- Inner (dot) product: .
The last identity is the reason dot products measure similarity: if two vectors point the same way, ; orthogonal gives 0. Attention scores, embedding retrieval, and cosine similarity are all dot products.
Span, independence, basis. The span of vectors is everything reachable by linear combinations. Vectors are linearly independent if none lies in the span of the others. A basis is an independent set that spans the space; independent vectors form a basis of . The same vector has different coordinates in different bases; much of linear algebra is choosing a good basis (eigenbasis, singular vectors, principal components).
A norm measures length and satisfies non-negativity, homogeneity , and the triangle inequality.
| Norm | Definition | ML use |
|---|---|---|
| Euclidean distance, weight decay, gradient clipping | ||
| Lasso, sparsity | ||
| Adversarial perturbation budgets | ||
| “norm” | number of nonzeros | Sparsity (not a true norm; non-convex) |
| Frobenius | Matrix error, weight decay on matrices | |
| Spectral | Lipschitz bounds, spectral normalization |
Why does give sparsity? The unit ball has corners on the axes; the minimum of a loss subject to an constraint tends to touch a corner, where some coordinates are exactly zero. The ball is round, so no coordinate is preferred.
Matrices as linear maps
Section titled “Matrices as linear maps”A matrix represents a linear map from to . Linear means . Read the columns of as where the standard basis vectors land; is then a combination of those columns with weights .
Matrix product composes maps: apply , then . It is associative but not commutative. Shapes: . Shape-checking is the number one debugging tool in ML code.
Useful facts:
- Transpose: .
- Inverse exists only for square full-rank matrices; . In practice you almost never form an inverse; you solve with a factorization (
np.linalg.solve). - Rank is the dimension of the column space (= dimension of the row space). Rank means information is lost or redundant.
- Null space: . Dimension is (rank-nullity).
- Trace: ; cyclic, ; equals the sum of eigenvalues.
- Determinant: the signed volume scaling factor; zero iff singular; product of eigenvalues.
Orthogonality
Section titled “Orthogonality”A matrix is orthogonal if . Orthogonal maps are rotations/reflections: they preserve lengths and angles, . They are perfectly conditioned, which is why orthogonal initialization and orthogonal parametrizations are sometimes used to keep signals and gradients from blowing up or vanishing in deep or recurrent nets.
Projections and least squares
Section titled “Projections and least squares”Projecting onto the column space of (full column rank) gives . The least-squares problem is exactly this projection. Setting the gradient to zero gives the normal equations
Geometrically the residual is orthogonal to every column of . Numerically, forming squares the condition number, so use QR or SVD (np.linalg.lstsq) rather than the normal equations for ill-conditioned problems.
Eigenvalues and eigenvectors
Section titled “Eigenvalues and eigenvectors”For a square , a nonzero with is an eigenvector with eigenvalue : a direction the map only stretches. If has independent eigenvectors, .
For a symmetric matrix (covariance matrices, Hessians, Gram matrices) the spectral theorem says: eigenvalues are real, eigenvectors can be chosen orthonormal, and
So a symmetric matrix is “rotate into the eigenbasis, scale each axis independently, rotate back.”
Positive semi-definite (PSD): for all , equivalently all eigenvalues . Covariance matrices and are always PSD. Positive definite (all ) matrices are invertible and define a valid inner product and a convex quadratic .
Quadratic forms. The level sets of for PD are ellipsoids with axes along eigenvectors and semi-axis lengths . This picture is the best intuition for gradient descent on a quadratic: it zigzags when the ellipse is elongated, i.e. when is large. That ratio is the condition number ; see Calculus & Optimization.
Power iteration finds the dominant eigenvector: repeat . It converges at a rate governed by and underlies PageRank and many large-scale eigen-solvers.
import numpy as np
A = np.array([[2.0, 1.0], [1.0, 3.0]])lam, Q = np.linalg.eigh(A) # eigh for symmetric: sorted, orthonormalprint(lam) # real eigenvaluesprint(np.allclose(Q @ np.diag(lam) @ Q.T, A)) # TrueSingular value decomposition
Section titled “Singular value decomposition”Eigendecomposition needs square matrices. The SVD works for every matrix:
Interpretation: any linear map is rotate (), stretch along axes (), rotate (). The are singular values; are right singular vectors, left singular vectors. Relations to eigen-stuff:
- : right singular vectors are eigenvectors of , with eigenvalues .
- = number of nonzero singular values.
- , , .
The “economy” or thin SVD keeps only columns and is what libraries return by default if you ask.
Low-rank approximation (Eckart–Young)
Section titled “Low-rank approximation (Eckart–Young)”Writing as a sum of rank-one pieces, the best rank- approximation in both Frobenius and spectral norm is to keep the top terms:
A rank- matrix of size needs numbers instead of . This is the basis for image compression, latent semantic analysis, recommender-system matrix factorization, and LoRA fine-tuning, where a weight update is constrained to be with small inner dimension .
rng = np.random.default_rng(0)A = rng.normal(size=(100, 5)) @ rng.normal(size=(5, 80)) # rank 5A += 0.01 * rng.normal(size=A.shape) # small noise
U, s, Vt = np.linalg.svd(A, full_matrices=False)print(np.round(s[:8], 2)) # 5 large values, then a cliff
k = 5A_k = (U[:, :k] * s[:k]) @ Vt[:k]print(np.linalg.norm(A - A_k) / np.linalg.norm(A)) # small relative errorPseudoinverse
Section titled “Pseudoinverse”, where inverts the nonzero singular values. is the minimum-norm least-squares solution. This explains why gradient descent on an underdetermined linear regression, started from zero, converges to the minimum-norm interpolating solution, a small piece of the “implicit regularization” story.
PCA: the link
Section titled “PCA: the link”Let be centered (columns have zero mean). The sample covariance is . Principal component analysis finds orthonormal directions of maximum variance. The first direction solves
whose solution is the top eigenvector of (Rayleigh quotient), with variance . Subsequent components are the next eigenvectors, orthogonal to earlier ones.
Via SVD of the centered data, : the columns of are the principal directions, the eigenvalues of are , and the projected data (scores) are . Using SVD is numerically preferable to forming .
def pca(X, k): Xc = X - X.mean(axis=0) U, s, Vt = np.linalg.svd(Xc, full_matrices=False) var = s**2 / (len(X) - 1) return Xc @ Vt[:k].T, Vt[:k], var / var.sum() # scores, components, explained ratio
X = rng.normal(size=(500, 3)) @ np.array([[3, 0, 0], [1, 1, 0], [0, 0, 0.1]])Z, comps, ratio = pca(X, 2)print(ratio)Two equivalent views of PCA: maximum variance in the projection, and minimum reconstruction error (Eckart–Young on the centered matrix). Practical notes: standardize features if units differ, since PCA is scale-sensitive; PCA only captures linear structure; choose by explained-variance ratio or a scree plot cliff. More on PCA as a model in Classical Models.
Matrix calculus preview
Section titled “Matrix calculus preview”Two derivatives are worth memorizing (denominator layout, gradient as column):
- , which is for symmetric .
- For , .
A reliable strategy: derive for scalars/indices, then match shapes. The full treatment (Jacobians, Hessians) is in Calculus & Optimization, and its use in Backpropagation.
Practical notes for ML code
Section titled “Practical notes for ML code”- Batching is matmul. A batch of inputs through a layer is with . GPUs are matmul machines; FLOPs for are about .
- Broadcasting silently expands dimensions. Many subtle bugs are an unintended giving . Assert shapes.
- Numerical precision. In float16/bfloat16, ill-conditioned systems and large sums lose accuracy; see Training Deep Networks.
- Never invert explicitly. Use
solve,lstsq, Cholesky for PD systems. - Einsum is clearer for higher-rank tensors:
np.einsum("bij,bjk->bik", A, B)is batched matmul.
Key takeaways
Section titled “Key takeaways”- A matrix is a linear map; its columns show where basis vectors go, and products compose maps.
- Dot products measure alignment; norms measure size; the choice of norm shapes regularization behavior ( sparse, smooth).
- Symmetric matrices diagonalize with orthonormal eigenvectors; PSD means non-negative eigenvalues, as for covariances and Gram matrices.
- SVD exists for every matrix and gives the optimal low-rank approximation (Eckart–Young); singular values reveal rank and conditioning.
- PCA is the eigendecomposition of the covariance or equivalently the SVD of centered data.
- Prefer factorizations over explicit inverses; mind conditioning.
Exercises
Section titled “Exercises”- Conceptual. Why is always positive semi-definite? When is it positive definite?
- Conceptual. A matrix has singular values that decay as . Roughly what rank- do you need to capture 99.9% of ?
- Hands-on. Implement power iteration in NumPy and compare to
np.linalg.eighon a random symmetric matrix. How does convergence speed depend on ? - Hands-on. Take a grayscale image as a matrix, compute its SVD, and plot reconstruction error versus . At what is the result visually acceptable, and what is the compression ratio?
- Derivation. Show that the first principal component maximizes subject to using a Lagrange multiplier.
Solutions (1, 2, 5)
- . It is strictly positive for all iff has full column rank (null space is ).
- . The tail beyond is about . Need that below , so , .
- Lagrangian ; gives . Then , so maximizing picks the largest eigenvalue.
Further reading
Section titled “Further reading”- Gilbert Strang, Linear Algebra and Its Applications and MIT OCW 18.06 lectures.
- Gilbert Strang, Linear Algebra and Learning from Data (2019).
- Trefethen and Bau, Numerical Linear Algebra (1997).
- Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapters 2–4 and 10.
- Golub and Van Loan, Matrix Computations (reference).
- Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021).