Machine Learning Fundamentals
What you’ll learn. The formal setup of learning from data; how loss functions relate to probabilistic assumptions; why generalization (not training fit) is the goal; the bias–variance trade-off and what modern overparameterized models changed; regularization; honest evaluation with train/validation/test splits and cross-validation; the metrics that matter for classification and regression; and the data-leakage mistakes that silently inflate results.
The learning problem
Section titled “The learning problem”Data are pairs drawn i.i.d. from an unknown distribution . We choose a model family (hypothesis class) of functions and want that predicts well on new data from . Define:
- Loss : cost of one prediction.
- Risk (true error) : what we actually care about.
- Empirical risk : what we can compute.
Empirical risk minimization (ERM) picks . The gap is the generalization gap, and the entire subject is about controlling it.
Types of learning
Section titled “Types of learning”- Supervised: labels available. Regression () and classification ( categorical).
- Unsupervised: only . Clustering, density estimation, dimensionality reduction, representation learning.
- Self-supervised: labels derived from the data itself (predict the next token, a masked patch). This is how LLMs and modern vision encoders are pretrained (Attention & Transformers).
- Semi-supervised, weakly supervised: mixtures of labeled and unlabeled data.
- Reinforcement learning: learning from reward through interaction (Reinforcement Learning Basics).
Loss functions
Section titled “Loss functions”Losses encode what “wrong” means and, via maximum likelihood, a noise model.
| Task | Loss | Likelihood view |
|---|---|---|
| Regression | MSE | Gaussian noise; predicts the conditional mean |
| Regression, outliers | MAE / Huber | Laplace noise; MAE predicts the conditional median |
| Binary classification | Logistic / BCE | Bernoulli |
| Multiclass | Cross-entropy | Categorical |
| Margin classifiers | Hinge | – (SVM) |
| Ranking / retrieval | Pairwise, contrastive/InfoNCE | – |
The 0–1 loss (count errors) is the true goal for classification but is non-differentiable and non-convex, so we optimize a surrogate (cross-entropy, hinge). A good surrogate is calibrated: minimizing it also drives the 0–1 risk to its optimum. See Information Theory for why cross-entropy is natural.
Generalization
Section titled “Generalization”A model with enough capacity can memorize the training set: a degree- polynomial fits points exactly yet oscillates wildly between them. Training error always decreases with capacity; test error is U-shaped (classically).
error | \ / test error | \ / | \ _____ / | \_____/ \____/ | \___________________ training error +--------------------------------------> model capacity underfit sweet spot overfit- Underfitting: high error on both train and validation. Fix: more capacity, better features, longer training, less regularization.
- Overfitting: low train error, high validation error. Fix: more data, regularization, simpler model, augmentation, early stopping.
Bias–variance decomposition
Section titled “Bias–variance decomposition”For squared loss with , , and a model trained on a random dataset, the expected error at a point is
- Bias: systematic error from wrong assumptions (a line fitted to a curve).
- Variance: sensitivity to the particular training sample (a wiggly fit changes a lot if a few points change).
- Flexible models have low bias and high variance; rigid models the opposite.
- Ways to cut variance: more data, regularization, bagging/ensembling (random forests). Ways to cut bias: richer models, boosting.
Modern caveat: double descent. In heavily overparameterized models (neural nets, high-degree interpolators), test error can fall again after the interpolation threshold where training error hits zero. Big networks that memorize noise can still generalize because SGD and architecture impose an implicit bias toward simple solutions. The classical U-curve is a good intuition for small models, not a law for deep learning. Also, the clean bias–variance decomposition is specific to squared loss.
Regularization
Section titled “Regularization”Anything that reduces variance (generalization gap) at the cost of some bias.
- / weight decay / ridge: ; shrinks weights; equals a Gaussian prior (MAP). Closed form for linear regression: , which also fixes ill-conditioning.
- / lasso: induces sparsity (feature selection); Laplace prior.
- Elastic net: both.
- Early stopping: stop when validation loss stops improving; acts like regularization for linear models.
- Dropout, data augmentation, label smoothing, noise injection, ensembling: deep-learning regularizers (Training Deep Networks).
- Simpler architecture / fewer features / smaller trees.
- More data: the best regularizer.
The regularization strength is a hyperparameter, chosen by validation, not learned from the training loss (which would pick ).
import numpy as npfrom sklearn.linear_model import Ridgefrom sklearn.preprocessing import PolynomialFeatures, StandardScalerfrom sklearn.pipeline import make_pipelinefrom sklearn.model_selection import cross_val_score
rng = np.random.default_rng(0)X = rng.uniform(-3, 3, (60, 1)); y = np.sin(X[:, 0]) + 0.3 * rng.normal(size=60)
for alpha in [1e-6, 1e-2, 1, 100]: model = make_pipeline(PolynomialFeatures(12), StandardScaler(), Ridge(alpha=alpha)) score = cross_val_score(model, X, y, cv=5, scoring="neg_mean_squared_error").mean() print(alpha, -score) # too small: overfit; too large: underfitEvaluation protocol
Section titled “Evaluation protocol”Three disjoint sets:
- Train: fit parameters.
- Validation: choose hyperparameters, architectures, thresholds, early-stopping point.
- Test: touch once at the end for an unbiased estimate of . If you iterate against the test set, it becomes a validation set and your estimate is optimistic.
Typical splits: 80/10/10 or 70/15/15; with millions of examples 98/1/1 is fine. Use stratified splits for imbalanced classes. For time series, split by time (train on past, test on future); random splits leak the future. For grouped data (patients, users, documents) split by group.
Cross-validation
Section titled “Cross-validation”-fold CV partitions data into folds, trains on and validates on the held-out fold, and averages. It gives a lower-variance estimate than a single split when data is scarce. or is standard; leave-one-out has high variance and cost. Use nested CV (an inner loop for tuning, an outer loop for evaluation) when reporting performance of a tuned model; otherwise the estimate is optimistic. For deep learning with large data, a single holdout is the norm because retraining is expensive.
Metrics
Section titled “Metrics”Choose the metric that matches the decision, not the one that is convenient.
Classification (binary; confusion matrix entries TP, FP, FN, TN):
- Accuracy misleads under imbalance: predicting “no fraud” for everything scores 99.9% when fraud is 0.1%.
- Precision matters when false alarms are costly; recall when misses are costly. weighs recall times more.
- ROC AUC: probability a random positive scores above a random negative; threshold-free, but optimistic with heavy imbalance. PR AUC / average precision is more informative when positives are rare.
- Log loss and calibration: do predicted probabilities match frequencies? Check reliability diagrams; fix with temperature scaling or isotonic regression.
- Multiclass: macro-average (each class equal) vs micro-average (each example equal); top- accuracy.
- Threshold choice is a business decision: move the threshold using validation data according to costs.
Regression: MSE/RMSE (penalizes large errors), MAE (robust), (fraction of variance explained; can be negative on test data), MAPE (breaks near zero).
Baselines first. Always compare against a trivial baseline (majority class, mean, last value, simple linear model). A fancy model at 92% means nothing if the majority class is 91%.
from sklearn.metrics import precision_recall_fscore_support, roc_auc_score, average_precision_score# y_true (0/1), p (predicted probability)# precision, recall, f1, _ = precision_recall_fscore_support(y_true, p > 0.5, average="binary")# roc_auc_score(y_true, p); average_precision_score(y_true, p)Data leakage
Section titled “Data leakage”Leakage means information unavailable at prediction time sneaks into training or evaluation. It produces spectacular validation scores that collapse in production. Common forms:
- Preprocessing before splitting: fitting a scaler, imputer, PCA, or target encoder on the full dataset leaks test statistics. Fit on train only, via a
Pipeline. - Duplicates and near-duplicates across train/test (repeated rows, augmented copies, the same image re-cropped, web-scraped benchmarks present in LLM pretraining data: “contamination”).
- Target leakage: a feature derived from the label or from the future (a “discharge code” used to predict hospital admission; a field filled in after the outcome).
- Group leakage: the same patient/user/session on both sides of the split.
- Temporal leakage: random split of time-ordered data; features computed with future windows.
- Selection with the test set: feature selection or hyperparameter tuning using test labels, including repeated peeking at the leaderboard.
from sklearn.pipeline import make_pipelinefrom sklearn.preprocessing import StandardScalerfrom sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import GroupKFold, cross_val_score
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))# scaler is re-fit inside each CV training fold; no leakage:# cross_val_score(pipe, X, y, cv=GroupKFold(5), groups=user_ids)Distribution shift
Section titled “Distribution shift”Training and deployment data rarely match: covariate shift ( changes), label shift ( changes), concept drift ( changes). A test set drawn from the same distribution as training does not guarantee production performance. Monitor inputs and outputs, hold out data from the future or from different sources, and plan retraining.
A practical workflow
Section titled “A practical workflow”- Define the decision, metric, and baseline.
- Split data correctly (time/group aware) and lock the test set.
- Build a simple baseline pipeline end to end.
- Inspect errors by slice; look at the data.
- Increase model capacity or features only as validation shows underfitting; add regularization as it shows overfitting.
- Tune on validation (or CV), evaluate once on test, report intervals (Statistics & Estimation).
- Monitor in production.
Concrete models to apply this to are in Classical Models and Neural Networks.
Key takeaways
Section titled “Key takeaways”- The goal is low true risk; ERM minimizes empirical risk, and the generalization gap is the central issue.
- Loss functions correspond to probabilistic noise models; cross-entropy for classification, MSE for Gaussian regression.
- Error decomposes into bias, variance, and irreducible noise; overparameterized deep models show double descent, so the U-curve is not universal.
- Regularization trades a little bias for less variance; tune its strength on validation data.
- Keep the test set untouched; use cross-validation for small data and nested CV for tuned-model estimates.
- Pick metrics by cost structure; accuracy fails under imbalance; always compare to a baseline.
- Leakage is the most common cause of results that do not reproduce.
Exercises
Section titled “Exercises”- Conceptual. A model has 99% training accuracy and 70% validation accuracy. List three diagnoses and the corresponding fixes.
- Conceptual. Explain why fitting a
StandardScaleron the whole dataset before cross-validation is leakage even though no labels are used. - Computation. A test has 90% recall and 90% specificity. Prevalence is 1%. Compute precision.
- Hands-on. Using polynomial regression on noisy sine data, plot train and validation error versus degree (1 to 15) and versus ridge . Mark the sweet spot.
- Hands-on. Deliberately create target leakage in a toy dataset (add a noisy copy of the label as a feature), observe the metric jump, then detect it using feature importance.
Solutions (3)
- Per 10,000 people: 100 positives, 90 TP; 9,900 negatives with 10% false positives = 990 FP. Precision . Low prevalence makes precision low even for a good test.
Further reading
Section titled “Further reading”- Hastie, Tibshirani, Friedman, The Elements of Statistical Learning (2nd ed., 2009).
- James, Witten, Hastie, Tibshirani, An Introduction to Statistical Learning (2nd ed., 2021).
- Shalev-Shwartz and Ben-David, Understanding Machine Learning: From Theory to Algorithms (2014).
- Belkin, Hsu, Ma, Mandal, “Reconciling modern machine learning practice and the classical bias-variance trade-off” (2019).
- Kaufman, Rosset, Perlich, “Leakage in Data Mining” (2012).
- Murphy, Probabilistic Machine Learning: An Introduction (2022).
- scikit-learn User Guide: “Cross-validation” and “Metrics and scoring”.