Skip to content

Probability Essentials

What you’ll learn. The probability toolkit used throughout ML: sample spaces and random variables, conditional probability and Bayes’ rule, the distributions you will meet constantly, expectation and variance rules, the law of large numbers and the central limit theorem, the multivariate Gaussian, and a first definition of entropy. Simulations in NumPy.

ML models make predictions under uncertainty about data, parameters, and the future. Probability gives a consistent language for that. Many “loss functions” are negative log-likelihoods, regularizers are priors, and generative models are probability distributions you can sample. See Statistics & Estimation and Information Theory for the continuation.

A probability space has a sample space Ω\Omega (outcomes), events (subsets), and a measure PP with P(Ω)=1P(\Omega)=1, P(A)≥0P(A)\ge0, and additivity for disjoint events. From these: P(Ac)=1−P(A)P(A^c)=1-P(A) and P(A∪B)=P(A)+P(B)−P(A∩B)P(A\cup B)=P(A)+P(B)-P(A\cap B).

A random variable XX is a function from outcomes to numbers. A discrete XX has a probability mass function p(x)=P(X=x)p(x)=P(X=x); a continuous XX has a density p(x)≥0p(x)\ge0 with P(a≤X≤b)=∫abp(x)dxP(a\le X\le b)=\int_a^b p(x)dx. A density is not a probability and can exceed 1. The CDF is F(x)=P(X≤x)F(x)=P(X\le x).

Joint, marginal, conditional. For two variables:

p(x)=∑yp(x,y) (marginalization),p(x∣y)=p(x,y)p(y),p(x,y)=p(x∣y)p(y).p(x)=\sum_y p(x,y)\ \text{(marginalization)},\qquad p(x\mid y)=\frac{p(x,y)}{p(y)},\qquad p(x,y)=p(x\mid y)p(y).

The chain rule extends: p(x1,…,xn)=∏ip(xi∣x<i)p(x_1,\dots,x_n)=\prod_i p(x_i\mid x_{<i}). This factorization is exactly how autoregressive language models define a distribution over text.

Independence: p(x,y)=p(x)p(y)p(x,y)=p(x)p(y). Conditional independence X⊥Y∣ZX\perp Y\mid Z: p(x,y∣z)=p(x∣z)p(y∣z)p(x,y\mid z)=p(x\mid z)p(y\mid z). Naive Bayes assumes features are conditionally independent given the class. Independent means uncorrelated, but uncorrelated does not imply independent (take X∼N(0,1)X\sim N(0,1), Y=X2Y=X^2).

p(θ∣D)=p(D∣θ) p(θ)p(D),p(D)=∫p(D∣θ)p(θ) dθ.p(\theta\mid D)=\frac{p(D\mid\theta)\,p(\theta)}{p(D)},\qquad p(D)=\int p(D\mid\theta)p(\theta)\,d\theta.

Posterior ∝\propto likelihood ×\times prior. The evidence p(D)p(D) normalizes.

Worked example (diagnostic test). A disease has prevalence 1%. A test has sensitivity 95% (P(+∣sick)=0.95P(+\mid\text{sick})=0.95) and false-positive rate 5%. What is P(sick∣+)P(\text{sick}\mid +)?

0.95⋅0.010.95⋅0.01+0.05⋅0.99=0.00950.0095+0.0495≈0.16.\frac{0.95\cdot0.01}{0.95\cdot0.01+0.05\cdot0.99}=\frac{0.0095}{0.0095+0.0495}\approx0.16.

Only 16%, because the base rate is low. Ignoring the prior (base-rate neglect) is the classic error; it appears in ML as evaluating a rare-event classifier with accuracy alone. See ML Fundamentals.

import numpy as np
rng = np.random.default_rng(0)
N = 1_000_000
sick = rng.random(N) < 0.01
pos = np.where(sick, rng.random(N) < 0.95, rng.random(N) < 0.05)
print(sick[pos].mean()) # ~0.16

E[g(X)]=∑xg(x)p(x)\mathbb{E}[g(X)]=\sum_x g(x)p(x) or ∫g(x)p(x)dx\int g(x)p(x)dx. Rules:

  • Linearity always holds: E[aX+bY]=aEX+bEY\mathbb{E}[aX+bY]=a\mathbb{E}X+b\mathbb{E}Y, even for dependent variables.
  • Var⁡(X)=E[(X−μ)2]=E[X2]−μ2\operatorname{Var}(X)=\mathbb{E}[(X-\mu)^2]=\mathbb{E}[X^2]-\mu^2; Var⁡(aX+b)=a2Var⁡(X)\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X).
  • Cov⁡(X,Y)=E[XY]−EX EY\operatorname{Cov}(X,Y)=\mathbb{E}[XY]-\mathbb{E}X\,\mathbb{E}Y; Var⁡(X+Y)=Var⁡X+Var⁡Y+2Cov⁡(X,Y)\operatorname{Var}(X+Y)=\operatorname{Var}X+\operatorname{Var}Y+2\operatorname{Cov}(X,Y).
  • For independent variables, E[XY]=EXEY\mathbb{E}[XY]=\mathbb{E}X\mathbb{E}Y and variances add. The mean of nn i.i.d. samples has variance σ2/n\sigma^2/n; this is why larger minibatches reduce gradient noise.
  • Law of total expectation: E[X]=EY[E[X∣Y]]\mathbb{E}[X]=\mathbb{E}_Y[\mathbb{E}[X\mid Y]]. Law of total variance: Var⁡X=E[Var⁡(X∣Y)]+Var⁡(E[X∣Y])\operatorname{Var}X=\mathbb{E}[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(\mathbb{E}[X\mid Y]).

The covariance matrix Σ=E[(X−μ)(X−μ)⊤]\Sigma=\mathbb{E}[(X-\mu)(X-\mu)^\top] is symmetric PSD (Linear Algebra). Correlation is covariance normalized to [−1,1][-1,1] and captures only linear dependence.

Jensen’s inequality. For convex ff, f(EX)≤Ef(X)f(\mathbb{E}X)\le\mathbb{E}f(X). It underlies the ELBO in Generative Models and the non-negativity of KL divergence.

Monte Carlo. If you can sample xi∼px_i\sim p, then E[g(X)]≈1n∑g(xi)\mathbb{E}[g(X)]\approx\tfrac1n\sum g(x_i), with error O(1/n)O(1/\sqrt n) independent of dimension. This is the basis of stochastic training and of many estimators in RL and generative modeling.

Distribution Support Parameters Mean / Variance Where it shows up
Bernoulli(pp) {0,1} pp pp / p(1−p)p(1-p) Binary labels, dropout masks
Binomial(n,pn,p) 0..n n,pn,p npnp / np(1−p)np(1-p) Counts of successes
Categorical(π\pi) 1..K simplex π\pi – Class labels, next-token distribution
Poisson(λ\lambda) 0,1,2.. λ\lambda λ\lambda / λ\lambda Event counts
Uniform(a,ba,b) [a,b][a,b] a,ba,b a+b2\tfrac{a+b}2 / (b−a)212\tfrac{(b-a)^2}{12} Initialization, sampling
Gaussian(μ,σ2\mu,\sigma^2) R\mathbb{R} μ,σ2\mu,\sigma^2 μ\mu / σ2\sigma^2 Noise, priors, latents
Exponential(λ\lambda) [0,∞)[0,\infty) λ\lambda 1/λ1/\lambda / 1/λ21/\lambda^2 Waiting times
Beta(α,β\alpha,\beta) [0,1][0,1] α,β\alpha,\beta αα+β\tfrac{\alpha}{\alpha+\beta} Prior over a probability
Dirichlet(α\alpha) simplex α\alpha – Prior over categorical

Two relationships to remember: the Beta is conjugate to the Bernoulli/Binomial (a Beta prior plus coin-flip data gives a Beta posterior: add successes to α\alpha and failures to β\beta), and the Dirichlet is conjugate to the Categorical. Conjugacy gives closed-form Bayesian updates.

Softmax converts logits zz to a categorical distribution: πk=ezk/∑jezj\pi_k=e^{z_k}/\sum_j e^{z_j}. Temperature TT rescales z/Tz/T: T→0T\to0 approaches argmax, T→∞T\to\infty approaches uniform. Compute it stably by subtracting max⁡z\max z.

p(x)=12πσ2exp⁡(−(x−μ)22σ2).p(x)=\frac1{\sqrt{2\pi\sigma^2}}\exp\Big(-\frac{(x-\mu)^2}{2\sigma^2}\Big).

Why it is everywhere: (1) the CLT (below); (2) among all distributions with a given mean and variance it has maximum entropy; (3) it is closed under linear maps, marginalization, and conditioning; (4) log-density is quadratic, so MLE gives least squares.

Multivariate Gaussian N(μ,Σ)\mathcal N(\mu,\Sigma):

p(x)=1(2π)d/2∣Σ∣1/2exp⁡(−12(x−μ)⊤Σ−1(x−μ)).p(x)=\frac1{(2\pi)^{d/2}|\Sigma|^{1/2}}\exp\Big(-\tfrac12(x-\mu)^\top\Sigma^{-1}(x-\mu)\Big).

Level sets are ellipsoids with axes along the eigenvectors of Σ\Sigma. Facts: if x∼N(μ,Σ)x\sim\mathcal N(\mu,\Sigma) then Ax+b∼N(Aμ+b,AΣA⊤)Ax+b\sim\mathcal N(A\mu+b,A\Sigma A^\top); marginals are Gaussian; conditionals are Gaussian with

μa∣b=μa+ΣabΣbb−1(xb−μb),Σa∣b=Σaa−ΣabΣbb−1Σba.\mu_{a|b}=\mu_a+\Sigma_{ab}\Sigma_{bb}^{-1}(x_b-\mu_b),\qquad\Sigma_{a|b}=\Sigma_{aa}-\Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba}.

To sample, use x=μ+Lzx=\mu+Lz with Σ=LL⊤\Sigma=LL^\top (Cholesky) and z∼N(0,I)z\sim\mathcal N(0,I). This reparameterization is central to VAEs and diffusion models.

mu = np.array([1.0, -1.0])
Sigma = np.array([[2.0, 1.2], [1.2, 1.0]])
L = np.linalg.cholesky(Sigma)
x = mu + rng.normal(size=(100_000, 2)) @ L.T
print(x.mean(0), np.cov(x.T)) # close to mu, Sigma

In high dimensions Gaussian samples concentrate on a thin shell of radius about d\sqrt d, not near the mean: a reminder that low-dimensional geometric intuition fails.

Law of large numbers. The sample mean Xˉn→μ\bar X_n\to\mu as n→∞n\to\infty (almost surely under mild conditions). It justifies estimating expectations by averages.

Central limit theorem. For i.i.d. XiX_i with mean μ\mu and finite variance σ2\sigma^2,

n Xˉn−μσ →d N(0,1).\sqrt n\,\frac{\bar X_n-\mu}{\sigma}\ \xrightarrow{d}\ \mathcal N(0,1).

The mean of many independent contributions is approximately Gaussian regardless of the original distribution. The error of a Monte Carlo estimate or a minibatch gradient is thus approximately Gaussian with standard deviation σ/n\sigma/\sqrt n. Caveats: the CLT needs finite variance (heavy-tailed data such as Cauchy fails), convergence can be slow for skewed distributions, and it is about the center, not the tails.

n = 50
means = rng.exponential(1.0, size=(100_000, n)).mean(axis=1)
z = (means - 1.0) / (1.0 / np.sqrt(n))
print(z.mean(), z.std()) # ~0, ~1; histogram looks bell-shaped

Concentration. Beyond the CLT, bounds such as Markov, Chebyshev (P(∣X−μ∣≥kσ)≤1/k2P(|X-\mu|\ge k\sigma)\le1/k^2), and Hoeffding give finite-sample guarantees and are the basis of generalization bounds.

For a discrete distribution, entropy measures average surprise:

H(p)=−∑xp(x)log⁡p(x).H(p)=-\sum_x p(x)\log p(x).

Uniform over KK outcomes gives log⁡K\log K (maximum); a point mass gives 0. Entropy in nats uses ln⁡\ln, in bits log⁡2\log_2. A model’s softmax entropy indicates confidence. Full treatment, including cross-entropy and KL, is in Information Theory.

  • Conflating density and probability; densities change under reparameterization.
  • Assuming independence when samples are correlated (time series, duplicated data, same-user records) which breaks σ/n\sigma/\sqrt n.
  • Multiplying many probabilities underflows: work in log space.
  • “Gaussian” assumed for heavy-tailed data (financial returns, gradient noise in some regimes).
  • Confusing the posterior over parameters with the predictive distribution over new data.
  • Marginalization, conditioning, and the chain rule are the three moves; Bayes’ rule follows from them.
  • Linearity of expectation holds without independence; variance of sums needs covariances.
  • Know Bernoulli, categorical, Gaussian, Beta/Dirichlet; softmax links logits to categorical distributions.
  • The Gaussian is closed under linear maps and conditioning, and sampling is μ+Lz\mu+Lz.
  • LLN justifies averaging; CLT says the average is approximately normal with std σ/n\sigma/\sqrt n.
  • Entropy measures uncertainty; work in log space.
  1. Conceptual. A classifier with 99% accuracy is run on a dataset where 99% of examples are negative. Why might it be useless? Use Bayes’ rule to frame the answer.
  2. Derivation. Show that for X∼Bernoulli(p)X\sim\text{Bernoulli}(p) the variance is p(1−p)p(1-p), and find the pp that maximizes it.
  3. Hands-on. Verify the CLT by averaging n∈{1,2,5,30}n\in\{1,2,5,30\} draws from a skewed distribution (lognormal). Plot histograms and a Q–Q plot against the normal. At what nn does it look normal?
  4. Hands-on. You observe 7 heads in 10 flips. With a Beta(2,2) prior compute the posterior and its mean; compare with the MLE 0.7.
  5. Conceptual. Why does the standard error of a minibatch gradient fall as 1/B1/\sqrt{B}, and what does that imply about the benefit of doubling batch size?
Solutions (2, 4, 5)
  1. EX=p\mathbb{E}X=p, EX2=p\mathbb{E}X^2=p (since X2=XX^2=X), so Var⁡=p−p2\operatorname{Var}=p-p^2. Derivative 1−2p=01-2p=0 gives p=1/2p=1/2.
  2. Posterior is Beta(2+7, 2+3)=Beta(9,5), mean 9/14≈0.6439/14\approx0.643, pulled toward the prior mean 0.5 from the MLE 0.7.
  3. Averaging BB independent per-example gradients divides variance by BB, so std falls as 1/B1/\sqrt B. Doubling BB reduces noise by only ≈29%\approx29\% while doubling compute per step.
  • Bertsekas and Tsitsiklis, Introduction to Probability (2nd ed., 2008).
  • Blitzstein and Hwang, Introduction to Probability (2019), with Harvard Stat 110 lectures.
  • Murphy, Probabilistic Machine Learning: An Introduction (2022).
  • Bishop, Pattern Recognition and Machine Learning (2006), chapters 1–2.
  • Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapter 6.
  • Jaynes, Probability Theory: The Logic of Science (2003).