Statistics & Estimation
What you’ll learn. How to turn data into parameter estimates (MLE, MAP), how to judge an estimator (bias, variance, MSE, consistency), why ML losses are negative log-likelihoods, how to quantify uncertainty (confidence intervals, bootstrap), and how to test hypotheses without fooling yourself, including comparing models.
The estimation problem
Section titled “The estimation problem”We observe data assumed drawn from a distribution with unknown parameter . An estimator is a function of the data. Because is random, is a random variable with its own distribution (the sampling distribution). Everything in this lecture follows from taking that seriously. Prerequisite: Probability Essentials.
Maximum likelihood estimation
Section titled “Maximum likelihood estimation”The likelihood is viewed as a function of . For i.i.d. data it factorizes, so we maximize the log-likelihood:
Logs turn products into sums (numerically stable, easy to differentiate) and do not change the argmax.
Example: Bernoulli. For successes in trials, . Setting the derivative to zero: , so .
Example: Gaussian. and . Note the MLE of variance divides by , not , and is biased.
Regression as MLE. Suppose , . Then . Maximizing likelihood equals minimizing squared error. Likewise, a categorical model with softmax gives cross-entropy loss, and a Bernoulli model gives logistic loss. A Laplace noise model gives absolute error (). Choosing a loss is choosing a noise model; see ML Fundamentals and Information Theory.
import numpy as npfrom scipy.optimize import minimize
rng = np.random.default_rng(0)x = rng.normal(3.0, 2.0, size=200)
def nll(params): mu, log_s = params s = np.exp(log_s) # parametrize in log space for positivity return 0.5 * np.sum(((x - mu) / s) ** 2) + len(x) * log_s + 0.5 * len(x) * np.log(2 * np.pi)
res = minimize(nll, x0=[0.0, 0.0])print(res.x[0], np.exp(res.x[1])) # ~ x.mean(), ~ x.std()Properties of the MLE
Section titled “Properties of the MLE”Under regularity conditions the MLE is:
- Consistent: as .
- Asymptotically normal: , where the Fisher information is .
- Asymptotically efficient: it attains the Cramér–Rao bound for unbiased estimators.
Fisher information is curvature of the log-likelihood: sharper peak means more information and a more precise estimate. It also underlies natural gradient and the Hessian approximations used in second-order training methods (Calculus & Optimization). Caveat: these are large-sample results; with few data and many parameters (the normal regime in deep learning) the MLE overfits.
MAP and Bayesian estimation
Section titled “MAP and Bayesian estimation”Add a prior . The posterior is and the maximum a posteriori estimate is
The prior acts as a regularizer:
- Gaussian prior gives : regularization / weight decay, for regression.
- Laplace prior gives (lasso) penalty and sparse solutions.
- Flat prior recovers the MLE.
As grows, the likelihood dominates and MAP converges to MLE; with little data the prior matters more. MAP is a point estimate; a fully Bayesian approach keeps the whole posterior and averages predictions, . Exact Bayesian inference is intractable for neural networks; approximations include variational inference, MC dropout, deep ensembles (often the strongest practical baseline for uncertainty), and Laplace approximations.
Beta-Bernoulli example. Prior Beta(), data successes in : posterior Beta(), MAP , which for and never hits the degenerate value 1 that the MLE gives. This “pseudo-count” idea is Laplace smoothing in naive Bayes.
Judging estimators: bias, variance, MSE
Section titled “Judging estimators: bias, variance, MSE”Define bias . Mean squared error decomposes:
Proof sketch: add and subtract inside the square; the cross term vanishes.
Example: sample variance. has expectation : biased low, because is fitted to the same data. Bessel’s correction makes it unbiased. Interestingly the biased version can have lower MSE: an unbiased estimator is not automatically best.
Biased can be better. Ridge regression and shrinkage estimators accept bias to reduce variance, lowering MSE; the James–Stein estimator beats the MLE for the mean of 3 or more Gaussians. This is the statistical core of the bias–variance trade-off that reappears for predictors in ML Fundamentals.
Uncertainty: standard errors and confidence intervals
Section titled “Uncertainty: standard errors and confidence intervals”The standard error of the mean is . By the CLT an approximate 95% confidence interval is
For small with roughly normal data use the Student quantile with degrees of freedom instead of 1.96.
What a confidence interval means. If you repeated the whole experiment many times, 95% of the intervals so constructed would contain the true . It does not mean ” lies in this interval with probability 95%”; that statement belongs to a Bayesian credible interval. For a single fixed interval, the true value either is or is not inside.
Bootstrap
Section titled “Bootstrap”When no formula is available (a median, an F1 score, a model’s AUC), resample the data with replacement times, recompute the statistic, and use the empirical distribution:
def bootstrap_ci(data, stat, B=5000, alpha=0.05, seed=0): rng = np.random.default_rng(seed) n = len(data) stats = np.array([stat(data[rng.integers(0, n, n)]) for _ in range(B)]) return np.quantile(stats, [alpha / 2, 1 - alpha / 2])
correct = rng.random(500) < 0.83 # per-example correctness of a modelprint(bootstrap_ci(correct.astype(float), np.mean))Always report an interval on benchmark scores: a 1-point difference on a 500-example test set is usually within noise (standard error of an accuracy near 0.8 with is about points).
Hypothesis testing
Section titled “Hypothesis testing”Frame a question as two hypotheses: null (no effect) and alternative . Compute a test statistic, and the p-value = probability, under , of a statistic at least as extreme as observed. Reject if (commonly 0.05).
Errors: Type I (false positive, probability ) and Type II (false negative, probability ). Power depends on effect size, , and noise. Plan sample size before collecting data.
Common tests. One-sample/two-sample -test for means; paired -test when measurements are matched (same test examples for two models); chi-square for counts; McNemar’s test for paired classifier errors; permutation tests, which make few assumptions.
from scipy import stats
a = rng.normal(0.80, 0.05, 20) # e.g. 20 seeds of model Ab = rng.normal(0.82, 0.05, 20)print(stats.ttest_ind(a, b, equal_var=False)) # Welch t-testWhat a p-value is not
Section titled “What a p-value is not”- Not the probability that is true.
- Not the effect size: with huge , trivial differences give tiny p-values.
- Not reproducible evidence on its own.
Multiple comparisons. Run 20 independent tests at and the chance of at least one false positive is . Control with Bonferroni () or the Benjamini–Hochberg procedure for the false discovery rate. This matters when you try many hyperparameters, prompts, or features and then report the best. Tuning on the test set is the same fallacy.
Comparing models in practice
Section titled “Comparing models in practice”- Evaluate over multiple random seeds; report mean and standard deviation or interval. Seed variance in deep learning is often 0.5 to 2 points on small benchmarks.
- Use the same test examples and paired analyses (paired bootstrap, McNemar) because per-example errors are correlated.
- Distinguish statistical significance from practical significance; ask whether the difference matters for the product.
- For online experiments (A/B tests) decide the sample size and stopping rule in advance; peeking and stopping when inflates false positives.
Frequentist vs Bayesian, briefly
Section titled “Frequentist vs Bayesian, briefly”Frequentists treat as fixed and data as random; Bayesians treat as random with a prior and condition on observed data. In ML practice you use both: MLE/SGD for fitting, Bayesian priors for regularization, frequentist confidence intervals for evaluation.
Pitfalls
Section titled “Pitfalls”- Using the biased MLE variance (
np.vardefaultddof=0) when you need the unbiased one (ddof=1). - Overlapping confidence intervals do not by themselves imply no significant difference; test the difference directly.
- Treating dependent samples as i.i.d. (augmented copies, same patient or user across train/test) which understates uncertainty. See data leakage in ML Fundamentals.
- Optional stopping and p-hacking.
- Reading MLE asymptotics as valid in the small-, large- regime.
Key takeaways
Section titled “Key takeaways”- An estimator is a random variable; bias, variance, and MSE describe its quality, and MSE bias variance.
- Most ML losses are negative log-likelihoods: squared error (Gaussian), cross-entropy (categorical), logistic (Bernoulli).
- MAP adds a log-prior; Gaussian prior is weight decay, Laplace prior is .
- MLE is consistent, asymptotically normal, and efficient, governed by Fisher information, but overfits in small-data regimes.
- Confidence intervals are about the procedure; the bootstrap works for arbitrary statistics.
- Guard against multiple comparisons and report uncertainty on model comparisons.
Exercises
Section titled “Exercises”- Derivation. Show that the MLE for in a Gaussian is and that its expectation is .
- Derivation. Derive ridge regression as MAP with a Gaussian prior; express in terms of .
- Hands-on. Simulate the sampling distribution of and of the median for from a Cauchy distribution. Which estimator is better behaved? Why?
- Hands-on. Train a small classifier with 10 seeds, compute a bootstrap CI on mean test accuracy, and compare with the -interval.
- Conceptual. You tried 40 prompt variants and one beat the baseline with . How worried should you be, and what would you do?
Solutions (2, 5)
- Posterior . Negative log is , so , giving .
- Expected false positives at across 40 tests is 2, so a single is unremarkable. Apply Bonferroni () or BH, and confirm the winner on fresh held-out data.
Further reading
Section titled “Further reading”- Casella and Berger, Statistical Inference (2nd ed., 2002).
- Wasserman, All of Statistics (2004).
- Efron and Hastie, Computer Age Statistical Inference (2016).
- Gelman et al., Bayesian Data Analysis (3rd ed., 2013).
- Murphy, Probabilistic Machine Learning: An Introduction (2022), chapters 3–4.
- Efron and Tibshirani, An Introduction to the Bootstrap (1993).
- Dietterich, “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms” (1998).