Skip to content

Statistics & Estimation

What you’ll learn. How to turn data into parameter estimates (MLE, MAP), how to judge an estimator (bias, variance, MSE, consistency), why ML losses are negative log-likelihoods, how to quantify uncertainty (confidence intervals, bootstrap), and how to test hypotheses without fooling yourself, including comparing models.

We observe data D={x1,…,xn}D=\{x_1,\dots,x_n\} assumed drawn from a distribution p(x;θ)p(x;\theta) with unknown parameter θ\theta. An estimator θ^=T(D)\hat\theta=T(D) is a function of the data. Because DD is random, θ^\hat\theta is a random variable with its own distribution (the sampling distribution). Everything in this lecture follows from taking that seriously. Prerequisite: Probability Essentials.

The likelihood is L(θ)=p(D;θ)\mathcal L(\theta)=p(D;\theta) viewed as a function of θ\theta. For i.i.d. data it factorizes, so we maximize the log-likelihood:

θ^MLE=arg⁡max⁡θ∑i=1nlog⁡p(xi;θ)=arg⁡min⁡θ[−1n∑ilog⁡p(xi;θ)].\hat\theta_{\text{MLE}}=\arg\max_\theta\sum_{i=1}^n\log p(x_i;\theta)=\arg\min_\theta\Big[-\tfrac1n\sum_i\log p(x_i;\theta)\Big].

Logs turn products into sums (numerically stable, easy to differentiate) and do not change the argmax.

Example: Bernoulli. For kk successes in nn trials, ℓ(p)=klog⁡p+(n−k)log⁡(1−p)\ell(p)=k\log p+(n-k)\log(1-p). Setting the derivative to zero: k/p−(n−k)/(1−p)=0k/p-(n-k)/(1-p)=0, so p^=k/n\hat p=k/n.

Example: Gaussian. μ^=xˉ\hat\mu=\bar x and σ^2=1n∑(xi−xˉ)2\hat\sigma^2=\tfrac1n\sum(x_i-\bar x)^2. Note the MLE of variance divides by nn, not n−1n-1, and is biased.

Regression as MLE. Suppose y=w⊤x+εy=w^\top x+\varepsilon, ε∼N(0,σ2)\varepsilon\sim\mathcal N(0,\sigma^2). Then −log⁡p(y∣x,w)=12σ2(y−w⊤x)2+const-\log p(y\mid x,w)=\tfrac1{2\sigma^2}(y-w^\top x)^2+\text{const}. Maximizing likelihood equals minimizing squared error. Likewise, a categorical model with softmax gives cross-entropy loss, and a Bernoulli model gives logistic loss. A Laplace noise model gives absolute error (ℓ1\ell_1). Choosing a loss is choosing a noise model; see ML Fundamentals and Information Theory.

import numpy as np
from scipy.optimize import minimize
rng = np.random.default_rng(0)
x = rng.normal(3.0, 2.0, size=200)
def nll(params):
mu, log_s = params
s = np.exp(log_s) # parametrize in log space for positivity
return 0.5 * np.sum(((x - mu) / s) ** 2) + len(x) * log_s + 0.5 * len(x) * np.log(2 * np.pi)
res = minimize(nll, x0=[0.0, 0.0])
print(res.x[0], np.exp(res.x[1])) # ~ x.mean(), ~ x.std()

Under regularity conditions the MLE is:

  • Consistent: θ^→θ0\hat\theta\to\theta_0 as n→∞n\to\infty.
  • Asymptotically normal: n(θ^−θ0)→N(0, I(θ0)−1)\sqrt n(\hat\theta-\theta_0)\to\mathcal N\big(0,\,I(\theta_0)^{-1}\big), where the Fisher information is I(θ)=E[(∂θlog⁡p)2]=−E[∂θ2log⁡p]I(\theta)=\mathbb{E}\big[(\partial_\theta\log p)^2\big]=-\mathbb{E}[\partial^2_\theta\log p].
  • Asymptotically efficient: it attains the Cramér–Rao bound Var⁡(θ^)≥1/(nI(θ))\operatorname{Var}(\hat\theta)\ge 1/(nI(\theta)) for unbiased estimators.

Fisher information is curvature of the log-likelihood: sharper peak means more information and a more precise estimate. It also underlies natural gradient and the Hessian approximations used in second-order training methods (Calculus & Optimization). Caveat: these are large-sample results; with few data and many parameters (the normal regime in deep learning) the MLE overfits.

Add a prior p(θ)p(\theta). The posterior is p(θ∣D)∝p(D∣θ)p(θ)p(\theta\mid D)\propto p(D\mid\theta)p(\theta) and the maximum a posteriori estimate is

θ^MAP=arg⁡max⁡θ[log⁡p(D∣θ)+log⁡p(θ)].\hat\theta_{\text{MAP}}=\arg\max_\theta\big[\log p(D\mid\theta)+\log p(\theta)\big].

The prior acts as a regularizer:

  • Gaussian prior θ∼N(0,τ2I)\theta\sim\mathcal N(0,\tau^2I) gives log⁡p(θ)=−12τ2∥θ∥2\log p(\theta)=-\tfrac1{2\tau^2}\|\theta\|^2: ℓ2\ell_2 regularization / weight decay, λ=σ2/τ2\lambda=\sigma^2/\tau^2 for regression.
  • Laplace prior gives ℓ1\ell_1 (lasso) penalty and sparse solutions.
  • Flat prior recovers the MLE.

As nn grows, the likelihood dominates and MAP converges to MLE; with little data the prior matters more. MAP is a point estimate; a fully Bayesian approach keeps the whole posterior and averages predictions, p(y∗∣x∗,D)=∫p(y∗∣x∗,θ)p(θ∣D)dθp(y^*\mid x^*,D)=\int p(y^*\mid x^*,\theta)p(\theta\mid D)d\theta. Exact Bayesian inference is intractable for neural networks; approximations include variational inference, MC dropout, deep ensembles (often the strongest practical baseline for uncertainty), and Laplace approximations.

Beta-Bernoulli example. Prior Beta(α,β\alpha,\beta), data kk successes in nn: posterior Beta(α+k,β+n−k\alpha+k,\beta+n-k), MAP =α+k−1α+β+n−2=\frac{\alpha+k-1}{\alpha+\beta+n-2}, which for α=β=2\alpha=\beta=2 and k=nk=n never hits the degenerate value 1 that the MLE gives. This “pseudo-count” idea is Laplace smoothing in naive Bayes.

Define bias Bias⁡(θ^)=E[θ^]−θ\operatorname{Bias}(\hat\theta)=\mathbb{E}[\hat\theta]-\theta. Mean squared error decomposes:

E[(θ^−θ)2]=Bias⁡2+Var⁡(θ^).\mathbb{E}\big[(\hat\theta-\theta)^2\big]=\operatorname{Bias}^2+\operatorname{Var}(\hat\theta).

Proof sketch: add and subtract Eθ^\mathbb{E}\hat\theta inside the square; the cross term vanishes.

Example: sample variance. σ^MLE2=1n∑(xi−xˉ)2\hat\sigma^2_{MLE}=\tfrac1n\sum(x_i-\bar x)^2 has expectation n−1nσ2\tfrac{n-1}n\sigma^2: biased low, because xˉ\bar x is fitted to the same data. Bessel’s correction 1n−1\tfrac1{n-1} makes it unbiased. Interestingly the biased version can have lower MSE: an unbiased estimator is not automatically best.

Biased can be better. Ridge regression and shrinkage estimators accept bias to reduce variance, lowering MSE; the James–Stein estimator beats the MLE for the mean of 3 or more Gaussians. This is the statistical core of the bias–variance trade-off that reappears for predictors in ML Fundamentals.

Uncertainty: standard errors and confidence intervals

Section titled “Uncertainty: standard errors and confidence intervals”

The standard error of the mean is σ^/n\hat\sigma/\sqrt n. By the CLT an approximate 95% confidence interval is

xˉ±1.96 sn.\bar x\pm1.96\,\frac{s}{\sqrt n}.

For small nn with roughly normal data use the Student tt quantile with n−1n-1 degrees of freedom instead of 1.96.

What a confidence interval means. If you repeated the whole experiment many times, 95% of the intervals so constructed would contain the true θ\theta. It does not mean ”θ\theta lies in this interval with probability 95%”; that statement belongs to a Bayesian credible interval. For a single fixed interval, the true value either is or is not inside.

When no formula is available (a median, an F1 score, a model’s AUC), resample the data with replacement BB times, recompute the statistic, and use the empirical distribution:

def bootstrap_ci(data, stat, B=5000, alpha=0.05, seed=0):
rng = np.random.default_rng(seed)
n = len(data)
stats = np.array([stat(data[rng.integers(0, n, n)]) for _ in range(B)])
return np.quantile(stats, [alpha / 2, 1 - alpha / 2])
correct = rng.random(500) < 0.83 # per-example correctness of a model
print(bootstrap_ci(correct.astype(float), np.mean))

Always report an interval on benchmark scores: a 1-point difference on a 500-example test set is usually within noise (standard error of an accuracy near 0.8 with n=500n=500 is about 1.81.8 points).

Frame a question as two hypotheses: null H0H_0 (no effect) and alternative H1H_1. Compute a test statistic, and the p-value = probability, under H0H_0, of a statistic at least as extreme as observed. Reject H0H_0 if p<αp<\alpha (commonly 0.05).

Errors: Type I (false positive, probability α\alpha) and Type II (false negative, probability β\beta). Power =1−β=1-\beta depends on effect size, nn, and noise. Plan sample size before collecting data.

Common tests. One-sample/two-sample tt-test for means; paired tt-test when measurements are matched (same test examples for two models); chi-square for counts; McNemar’s test for paired classifier errors; permutation tests, which make few assumptions.

from scipy import stats
a = rng.normal(0.80, 0.05, 20) # e.g. 20 seeds of model A
b = rng.normal(0.82, 0.05, 20)
print(stats.ttest_ind(a, b, equal_var=False)) # Welch t-test
  • Not the probability that H0H_0 is true.
  • Not the effect size: with huge nn, trivial differences give tiny p-values.
  • Not reproducible evidence on its own.

Multiple comparisons. Run 20 independent tests at α=0.05\alpha=0.05 and the chance of at least one false positive is 1−0.9520≈64%1-0.95^{20}\approx64\%. Control with Bonferroni (α/m\alpha/m) or the Benjamini–Hochberg procedure for the false discovery rate. This matters when you try many hyperparameters, prompts, or features and then report the best. Tuning on the test set is the same fallacy.

  • Evaluate over multiple random seeds; report mean and standard deviation or interval. Seed variance in deep learning is often 0.5 to 2 points on small benchmarks.
  • Use the same test examples and paired analyses (paired bootstrap, McNemar) because per-example errors are correlated.
  • Distinguish statistical significance from practical significance; ask whether the difference matters for the product.
  • For online experiments (A/B tests) decide the sample size and stopping rule in advance; peeking and stopping when p<0.05p<0.05 inflates false positives.

Frequentists treat θ\theta as fixed and data as random; Bayesians treat θ\theta as random with a prior and condition on observed data. In ML practice you use both: MLE/SGD for fitting, Bayesian priors for regularization, frequentist confidence intervals for evaluation.

  • Using the biased MLE variance (np.var default ddof=0) when you need the unbiased one (ddof=1).
  • Overlapping confidence intervals do not by themselves imply no significant difference; test the difference directly.
  • Treating dependent samples as i.i.d. (augmented copies, same patient or user across train/test) which understates uncertainty. See data leakage in ML Fundamentals.
  • Optional stopping and p-hacking.
  • Reading MLE asymptotics as valid in the small-nn, large-dd regime.
  • An estimator is a random variable; bias, variance, and MSE describe its quality, and MSE == bias2+^2+ variance.
  • Most ML losses are negative log-likelihoods: squared error (Gaussian), cross-entropy (categorical), logistic (Bernoulli).
  • MAP adds a log-prior; Gaussian prior is weight decay, Laplace prior is ℓ1\ell_1.
  • MLE is consistent, asymptotically normal, and efficient, governed by Fisher information, but overfits in small-data regimes.
  • Confidence intervals are about the procedure; the bootstrap works for arbitrary statistics.
  • Guard against multiple comparisons and report uncertainty on model comparisons.
  1. Derivation. Show that the MLE for σ2\sigma^2 in a Gaussian is 1n∑(xi−xˉ)2\tfrac1n\sum(x_i-\bar x)^2 and that its expectation is n−1nσ2\tfrac{n-1}n\sigma^2.
  2. Derivation. Derive ridge regression as MAP with a Gaussian prior; express λ\lambda in terms of σ2,τ2\sigma^2,\tau^2.
  3. Hands-on. Simulate the sampling distribution of xˉ\bar x and of the median for n=30n=30 from a Cauchy distribution. Which estimator is better behaved? Why?
  4. Hands-on. Train a small classifier with 10 seeds, compute a bootstrap CI on mean test accuracy, and compare with the tt-interval.
  5. Conceptual. You tried 40 prompt variants and one beat the baseline with p=0.03p=0.03. How worried should you be, and what would you do?
Solutions (2, 5)
  1. Posterior ∝exp⁡(−12σ2∥y−Xw∥2−12τ2∥w∥2)\propto\exp(-\tfrac1{2\sigma^2}\|y-Xw\|^2-\tfrac1{2\tau^2}\|w\|^2). Negative log is 12σ2[∥y−Xw∥2+σ2τ2∥w∥2]\tfrac1{2\sigma^2}[\|y-Xw\|^2+\tfrac{\sigma^2}{\tau^2}\|w\|^2], so λ=σ2/τ2\lambda=\sigma^2/\tau^2, giving w=(X⊤X+λI)−1X⊤yw=(X^\top X+\lambda I)^{-1}X^\top y.
  2. Expected false positives at α=0.05\alpha=0.05 across 40 tests is 2, so a single p=0.03p=0.03 is unremarkable. Apply Bonferroni (0.05/40=0.001250.05/40=0.00125) or BH, and confirm the winner on fresh held-out data.
  • Casella and Berger, Statistical Inference (2nd ed., 2002).
  • Wasserman, All of Statistics (2004).
  • Efron and Hastie, Computer Age Statistical Inference (2016).
  • Gelman et al., Bayesian Data Analysis (3rd ed., 2013).
  • Murphy, Probabilistic Machine Learning: An Introduction (2022), chapters 3–4.
  • Efron and Tibshirani, An Introduction to the Bootstrap (1993).
  • Dietterich, “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms” (1998).