Skip to content

Machine Learning Fundamentals

What you’ll learn. The formal setup of learning from data; how loss functions relate to probabilistic assumptions; why generalization (not training fit) is the goal; the bias–variance trade-off and what modern overparameterized models changed; regularization; honest evaluation with train/validation/test splits and cross-validation; the metrics that matter for classification and regression; and the data-leakage mistakes that silently inflate results.

Data are pairs (xi,yi)(x_i,y_i) drawn i.i.d. from an unknown distribution D\mathcal D. We choose a model family (hypothesis class) H\mathcal H of functions fθf_\theta and want ff that predicts well on new data from D\mathcal D. Define:

  • Loss ℓ(f(x),y)\ell(f(x),y): cost of one prediction.
  • Risk (true error) R(f)=E(x,y)∼D[ℓ(f(x),y)]R(f)=\mathbb{E}_{(x,y)\sim\mathcal D}[\ell(f(x),y)]: what we actually care about.
  • Empirical risk R^(f)=1n∑iℓ(f(xi),yi)\hat R(f)=\tfrac1n\sum_i\ell(f(x_i),y_i): what we can compute.

Empirical risk minimization (ERM) picks f^=arg⁡min⁡f∈HR^(f)\hat f=\arg\min_{f\in\mathcal H}\hat R(f). The gap R(f^)−R^(f^)R(\hat f)-\hat R(\hat f) is the generalization gap, and the entire subject is about controlling it.

  • Supervised: labels yy available. Regression (y∈Ry\in\mathbb{R}) and classification (yy categorical).
  • Unsupervised: only xx. Clustering, density estimation, dimensionality reduction, representation learning.
  • Self-supervised: labels derived from the data itself (predict the next token, a masked patch). This is how LLMs and modern vision encoders are pretrained (Attention & Transformers).
  • Semi-supervised, weakly supervised: mixtures of labeled and unlabeled data.
  • Reinforcement learning: learning from reward through interaction (Reinforcement Learning Basics).

Losses encode what “wrong” means and, via maximum likelihood, a noise model.

Task Loss Likelihood view
Regression MSE (y^−y)2(\hat y-y)^2 Gaussian noise; predicts the conditional mean
Regression, outliers MAE / Huber Laplace noise; MAE predicts the conditional median
Binary classification Logistic / BCE −ylog⁡p−(1−y)log⁡(1−p)-y\log p-(1-y)\log(1-p) Bernoulli
Multiclass Cross-entropy −log⁡py-\log p_y Categorical
Margin classifiers Hinge max⁡(0,1−yf)\max(0,1-y f) – (SVM)
Ranking / retrieval Pairwise, contrastive/InfoNCE –

The 0–1 loss (count errors) is the true goal for classification but is non-differentiable and non-convex, so we optimize a surrogate (cross-entropy, hinge). A good surrogate is calibrated: minimizing it also drives the 0–1 risk to its optimum. See Information Theory for why cross-entropy is natural.

A model with enough capacity can memorize the training set: a degree-n−1n-1 polynomial fits nn points exactly yet oscillates wildly between them. Training error always decreases with capacity; test error is U-shaped (classically).

error
| \ / test error
| \ /
| \ _____ /
| \_____/ \____/
| \___________________ training error
+--------------------------------------> model capacity
underfit sweet spot overfit
  • Underfitting: high error on both train and validation. Fix: more capacity, better features, longer training, less regularization.
  • Overfitting: low train error, high validation error. Fix: more data, regularization, simpler model, augmentation, early stopping.

For squared loss with y=f⋆(x)+εy=f^\star(x)+\varepsilon, Var⁡ε=σ2\operatorname{Var}\varepsilon=\sigma^2, and a model f^\hat f trained on a random dataset, the expected error at a point xx is

E[(y−f^(x))2]=(f⋆(x)−Ef^(x))2⏟bias2+Var⁡(f^(x))⏟variance+σ2⏟irreducible.\mathbb{E}\big[(y-\hat f(x))^2\big]=\underbrace{\big(f^\star(x)-\mathbb{E}\hat f(x)\big)^2}_{\text{bias}^2}+\underbrace{\operatorname{Var}\big(\hat f(x)\big)}_{\text{variance}}+\underbrace{\sigma^2}_{\text{irreducible}}.
  • Bias: systematic error from wrong assumptions (a line fitted to a curve).
  • Variance: sensitivity to the particular training sample (a wiggly fit changes a lot if a few points change).
  • Flexible models have low bias and high variance; rigid models the opposite.
  • Ways to cut variance: more data, regularization, bagging/ensembling (random forests). Ways to cut bias: richer models, boosting.

Modern caveat: double descent. In heavily overparameterized models (neural nets, high-degree interpolators), test error can fall again after the interpolation threshold where training error hits zero. Big networks that memorize noise can still generalize because SGD and architecture impose an implicit bias toward simple solutions. The classical U-curve is a good intuition for small models, not a law for deep learning. Also, the clean bias–variance decomposition is specific to squared loss.

Anything that reduces variance (generalization gap) at the cost of some bias.

  • ℓ2\ell_2 / weight decay / ridge: min⁡R^(θ)+λ∥θ∥22\min\hat R(\theta)+\lambda\|\theta\|_2^2; shrinks weights; equals a Gaussian prior (MAP). Closed form for linear regression: w=(X⊤X+λI)−1X⊤yw=(X^\top X+\lambda I)^{-1}X^\top y, which also fixes ill-conditioning.
  • ℓ1\ell_1 / lasso: induces sparsity (feature selection); Laplace prior.
  • Elastic net: both.
  • Early stopping: stop when validation loss stops improving; acts like ℓ2\ell_2 regularization for linear models.
  • Dropout, data augmentation, label smoothing, noise injection, ensembling: deep-learning regularizers (Training Deep Networks).
  • Simpler architecture / fewer features / smaller trees.
  • More data: the best regularizer.

The regularization strength λ\lambda is a hyperparameter, chosen by validation, not learned from the training loss (which would pick λ=0\lambda=0).

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(0)
X = rng.uniform(-3, 3, (60, 1)); y = np.sin(X[:, 0]) + 0.3 * rng.normal(size=60)
for alpha in [1e-6, 1e-2, 1, 100]:
model = make_pipeline(PolynomialFeatures(12), StandardScaler(), Ridge(alpha=alpha))
score = cross_val_score(model, X, y, cv=5, scoring="neg_mean_squared_error").mean()
print(alpha, -score) # too small: overfit; too large: underfit

Three disjoint sets:

  • Train: fit parameters.
  • Validation: choose hyperparameters, architectures, thresholds, early-stopping point.
  • Test: touch once at the end for an unbiased estimate of RR. If you iterate against the test set, it becomes a validation set and your estimate is optimistic.

Typical splits: 80/10/10 or 70/15/15; with millions of examples 98/1/1 is fine. Use stratified splits for imbalanced classes. For time series, split by time (train on past, test on future); random splits leak the future. For grouped data (patients, users, documents) split by group.

kk-fold CV partitions data into kk folds, trains on k−1k-1 and validates on the held-out fold, and averages. It gives a lower-variance estimate than a single split when data is scarce. k=5k=5 or 1010 is standard; leave-one-out has high variance and cost. Use nested CV (an inner loop for tuning, an outer loop for evaluation) when reporting performance of a tuned model; otherwise the estimate is optimistic. For deep learning with large data, a single holdout is the norm because retraining is expensive.

Choose the metric that matches the decision, not the one that is convenient.

Classification (binary; confusion matrix entries TP, FP, FN, TN):

accuracy=TP+TNN,precision=TPTP+FP,recall=TPTP+FN,F1=2PRP+R.\text{accuracy}=\tfrac{TP+TN}{N},\quad\text{precision}=\tfrac{TP}{TP+FP},\quad\text{recall}=\tfrac{TP}{TP+FN},\quad F_1=\tfrac{2PR}{P+R}.
  • Accuracy misleads under imbalance: predicting “no fraud” for everything scores 99.9% when fraud is 0.1%.
  • Precision matters when false alarms are costly; recall when misses are costly. FβF_\beta weighs recall β\beta times more.
  • ROC AUC: probability a random positive scores above a random negative; threshold-free, but optimistic with heavy imbalance. PR AUC / average precision is more informative when positives are rare.
  • Log loss and calibration: do predicted probabilities match frequencies? Check reliability diagrams; fix with temperature scaling or isotonic regression.
  • Multiclass: macro-average (each class equal) vs micro-average (each example equal); top-kk accuracy.
  • Threshold choice is a business decision: move the threshold using validation data according to costs.

Regression: MSE/RMSE (penalizes large errors), MAE (robust), R2=1−SSresSStotR^2=1-\tfrac{SS_{res}}{SS_{tot}} (fraction of variance explained; can be negative on test data), MAPE (breaks near zero).

Baselines first. Always compare against a trivial baseline (majority class, mean, last value, simple linear model). A fancy model at 92% means nothing if the majority class is 91%.

from sklearn.metrics import precision_recall_fscore_support, roc_auc_score, average_precision_score
# y_true (0/1), p (predicted probability)
# precision, recall, f1, _ = precision_recall_fscore_support(y_true, p > 0.5, average="binary")
# roc_auc_score(y_true, p); average_precision_score(y_true, p)

Leakage means information unavailable at prediction time sneaks into training or evaluation. It produces spectacular validation scores that collapse in production. Common forms:

  1. Preprocessing before splitting: fitting a scaler, imputer, PCA, or target encoder on the full dataset leaks test statistics. Fit on train only, via a Pipeline.
  2. Duplicates and near-duplicates across train/test (repeated rows, augmented copies, the same image re-cropped, web-scraped benchmarks present in LLM pretraining data: “contamination”).
  3. Target leakage: a feature derived from the label or from the future (a “discharge code” used to predict hospital admission; a field filled in after the outcome).
  4. Group leakage: the same patient/user/session on both sides of the split.
  5. Temporal leakage: random split of time-ordered data; features computed with future windows.
  6. Selection with the test set: feature selection or hyperparameter tuning using test labels, including repeated peeking at the leaderboard.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, cross_val_score
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
# scaler is re-fit inside each CV training fold; no leakage:
# cross_val_score(pipe, X, y, cv=GroupKFold(5), groups=user_ids)

Training and deployment data rarely match: covariate shift (p(x)p(x) changes), label shift (p(y)p(y) changes), concept drift (p(y∣x)p(y\mid x) changes). A test set drawn from the same distribution as training does not guarantee production performance. Monitor inputs and outputs, hold out data from the future or from different sources, and plan retraining.

  1. Define the decision, metric, and baseline.
  2. Split data correctly (time/group aware) and lock the test set.
  3. Build a simple baseline pipeline end to end.
  4. Inspect errors by slice; look at the data.
  5. Increase model capacity or features only as validation shows underfitting; add regularization as it shows overfitting.
  6. Tune on validation (or CV), evaluate once on test, report intervals (Statistics & Estimation).
  7. Monitor in production.

Concrete models to apply this to are in Classical Models and Neural Networks.

  • The goal is low true risk; ERM minimizes empirical risk, and the generalization gap is the central issue.
  • Loss functions correspond to probabilistic noise models; cross-entropy for classification, MSE for Gaussian regression.
  • Error decomposes into bias, variance, and irreducible noise; overparameterized deep models show double descent, so the U-curve is not universal.
  • Regularization trades a little bias for less variance; tune its strength on validation data.
  • Keep the test set untouched; use cross-validation for small data and nested CV for tuned-model estimates.
  • Pick metrics by cost structure; accuracy fails under imbalance; always compare to a baseline.
  • Leakage is the most common cause of results that do not reproduce.
  1. Conceptual. A model has 99% training accuracy and 70% validation accuracy. List three diagnoses and the corresponding fixes.
  2. Conceptual. Explain why fitting a StandardScaler on the whole dataset before cross-validation is leakage even though no labels are used.
  3. Computation. A test has 90% recall and 90% specificity. Prevalence is 1%. Compute precision.
  4. Hands-on. Using polynomial regression on noisy sine data, plot train and validation error versus degree (1 to 15) and versus ridge λ\lambda. Mark the sweet spot.
  5. Hands-on. Deliberately create target leakage in a toy dataset (add a noisy copy of the label as a feature), observe the metric jump, then detect it using feature importance.
Solutions (3)
  1. Per 10,000 people: 100 positives, 90 TP; 9,900 negatives with 10% false positives = 990 FP. Precision =90/(90+990)≈8.3%=90/(90+990)\approx8.3\%. Low prevalence makes precision low even for a good test.
  • Hastie, Tibshirani, Friedman, The Elements of Statistical Learning (2nd ed., 2009).
  • James, Witten, Hastie, Tibshirani, An Introduction to Statistical Learning (2nd ed., 2021).
  • Shalev-Shwartz and Ben-David, Understanding Machine Learning: From Theory to Algorithms (2014).
  • Belkin, Hsu, Ma, Mandal, “Reconciling modern machine learning practice and the classical bias-variance trade-off” (2019).
  • Kaufman, Rosset, Perlich, “Leakage in Data Mining” (2012).
  • Murphy, Probabilistic Machine Learning: An Introduction (2022).
  • scikit-learn User Guide: “Cross-validation” and “Metrics and scoring”.