Probability Essentials
What you’ll learn. The probability toolkit used throughout ML: sample spaces and random variables, conditional probability and Bayes’ rule, the distributions you will meet constantly, expectation and variance rules, the law of large numbers and the central limit theorem, the multivariate Gaussian, and a first definition of entropy. Simulations in NumPy.
Why probability
Section titled “Why probability”ML models make predictions under uncertainty about data, parameters, and the future. Probability gives a consistent language for that. Many “loss functions” are negative log-likelihoods, regularizers are priors, and generative models are probability distributions you can sample. See Statistics & Estimation and Information Theory for the continuation.
Foundations
Section titled “Foundations”A probability space has a sample space (outcomes), events (subsets), and a measure with , , and additivity for disjoint events. From these: and .
A random variable is a function from outcomes to numbers. A discrete has a probability mass function ; a continuous has a density with . A density is not a probability and can exceed 1. The CDF is .
Joint, marginal, conditional. For two variables:
The chain rule extends: . This factorization is exactly how autoregressive language models define a distribution over text.
Independence: . Conditional independence : . Naive Bayes assumes features are conditionally independent given the class. Independent means uncorrelated, but uncorrelated does not imply independent (take , ).
Bayes’ rule
Section titled “Bayes’ rule”Posterior likelihood prior. The evidence normalizes.
Worked example (diagnostic test). A disease has prevalence 1%. A test has sensitivity 95% () and false-positive rate 5%. What is ?
Only 16%, because the base rate is low. Ignoring the prior (base-rate neglect) is the classic error; it appears in ML as evaluating a rare-event classifier with accuracy alone. See ML Fundamentals.
import numpy as nprng = np.random.default_rng(0)N = 1_000_000sick = rng.random(N) < 0.01pos = np.where(sick, rng.random(N) < 0.95, rng.random(N) < 0.05)print(sick[pos].mean()) # ~0.16Expectation, variance, covariance
Section titled “Expectation, variance, covariance”or . Rules:
- Linearity always holds: , even for dependent variables.
- ; .
- ; .
- For independent variables, and variances add. The mean of i.i.d. samples has variance ; this is why larger minibatches reduce gradient noise.
- Law of total expectation: . Law of total variance: .
The covariance matrix is symmetric PSD (Linear Algebra). Correlation is covariance normalized to and captures only linear dependence.
Jensen’s inequality. For convex , . It underlies the ELBO in Generative Models and the non-negativity of KL divergence.
Monte Carlo. If you can sample , then , with error independent of dimension. This is the basis of stochastic training and of many estimators in RL and generative modeling.
Distributions you must know
Section titled “Distributions you must know”| Distribution | Support | Parameters | Mean / Variance | Where it shows up |
|---|---|---|---|---|
| Bernoulli() | {0,1} | / | Binary labels, dropout masks | |
| Binomial() | 0..n | / | Counts of successes | |
| Categorical() | 1..K | simplex | – | Class labels, next-token distribution |
| Poisson() | 0,1,2.. | / | Event counts | |
| Uniform() | / | Initialization, sampling | ||
| Gaussian() | / | Noise, priors, latents | ||
| Exponential() | / | Waiting times | ||
| Beta() | Prior over a probability | |||
| Dirichlet() | simplex | – | Prior over categorical |
Two relationships to remember: the Beta is conjugate to the Bernoulli/Binomial (a Beta prior plus coin-flip data gives a Beta posterior: add successes to and failures to ), and the Dirichlet is conjugate to the Categorical. Conjugacy gives closed-form Bayesian updates.
Softmax converts logits to a categorical distribution: . Temperature rescales : approaches argmax, approaches uniform. Compute it stably by subtracting .
The Gaussian
Section titled “The Gaussian”Why it is everywhere: (1) the CLT (below); (2) among all distributions with a given mean and variance it has maximum entropy; (3) it is closed under linear maps, marginalization, and conditioning; (4) log-density is quadratic, so MLE gives least squares.
Multivariate Gaussian :
Level sets are ellipsoids with axes along the eigenvectors of . Facts: if then ; marginals are Gaussian; conditionals are Gaussian with
To sample, use with (Cholesky) and . This reparameterization is central to VAEs and diffusion models.
mu = np.array([1.0, -1.0])Sigma = np.array([[2.0, 1.2], [1.2, 1.0]])L = np.linalg.cholesky(Sigma)x = mu + rng.normal(size=(100_000, 2)) @ L.Tprint(x.mean(0), np.cov(x.T)) # close to mu, SigmaIn high dimensions Gaussian samples concentrate on a thin shell of radius about , not near the mean: a reminder that low-dimensional geometric intuition fails.
Limit theorems
Section titled “Limit theorems”Law of large numbers. The sample mean as (almost surely under mild conditions). It justifies estimating expectations by averages.
Central limit theorem. For i.i.d. with mean and finite variance ,
The mean of many independent contributions is approximately Gaussian regardless of the original distribution. The error of a Monte Carlo estimate or a minibatch gradient is thus approximately Gaussian with standard deviation . Caveats: the CLT needs finite variance (heavy-tailed data such as Cauchy fails), convergence can be slow for skewed distributions, and it is about the center, not the tails.
n = 50means = rng.exponential(1.0, size=(100_000, n)).mean(axis=1)z = (means - 1.0) / (1.0 / np.sqrt(n))print(z.mean(), z.std()) # ~0, ~1; histogram looks bell-shapedConcentration. Beyond the CLT, bounds such as Markov, Chebyshev (), and Hoeffding give finite-sample guarantees and are the basis of generalization bounds.
A first look at entropy
Section titled “A first look at entropy”For a discrete distribution, entropy measures average surprise:
Uniform over outcomes gives (maximum); a point mass gives 0. Entropy in nats uses , in bits . A model’s softmax entropy indicates confidence. Full treatment, including cross-entropy and KL, is in Information Theory.
Pitfalls
Section titled “Pitfalls”- Conflating density and probability; densities change under reparameterization.
- Assuming independence when samples are correlated (time series, duplicated data, same-user records) which breaks .
- Multiplying many probabilities underflows: work in log space.
- “Gaussian” assumed for heavy-tailed data (financial returns, gradient noise in some regimes).
- Confusing the posterior over parameters with the predictive distribution over new data.
Key takeaways
Section titled “Key takeaways”- Marginalization, conditioning, and the chain rule are the three moves; Bayes’ rule follows from them.
- Linearity of expectation holds without independence; variance of sums needs covariances.
- Know Bernoulli, categorical, Gaussian, Beta/Dirichlet; softmax links logits to categorical distributions.
- The Gaussian is closed under linear maps and conditioning, and sampling is .
- LLN justifies averaging; CLT says the average is approximately normal with std .
- Entropy measures uncertainty; work in log space.
Exercises
Section titled “Exercises”- Conceptual. A classifier with 99% accuracy is run on a dataset where 99% of examples are negative. Why might it be useless? Use Bayes’ rule to frame the answer.
- Derivation. Show that for the variance is , and find the that maximizes it.
- Hands-on. Verify the CLT by averaging draws from a skewed distribution (lognormal). Plot histograms and a Q–Q plot against the normal. At what does it look normal?
- Hands-on. You observe 7 heads in 10 flips. With a Beta(2,2) prior compute the posterior and its mean; compare with the MLE 0.7.
- Conceptual. Why does the standard error of a minibatch gradient fall as , and what does that imply about the benefit of doubling batch size?
Solutions (2, 4, 5)
- , (since ), so . Derivative gives .
- Posterior is Beta(2+7, 2+3)=Beta(9,5), mean , pulled toward the prior mean 0.5 from the MLE 0.7.
- Averaging independent per-example gradients divides variance by , so std falls as . Doubling reduces noise by only while doubling compute per step.
Further reading
Section titled “Further reading”- Bertsekas and Tsitsiklis, Introduction to Probability (2nd ed., 2008).
- Blitzstein and Hwang, Introduction to Probability (2019), with Harvard Stat 110 lectures.
- Murphy, Probabilistic Machine Learning: An Introduction (2022).
- Bishop, Pattern Recognition and Machine Learning (2006), chapters 1–2.
- Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapter 6.
- Jaynes, Probability Theory: The Logic of Science (2003).