Classical Models
What you’ll learn. The core non-deep models, how each works mathematically, their inductive biases, and when they win. Despite the deep learning boom, gradient-boosted trees remain the default for tabular data, and linear models remain the first baseline for nearly everything. Code uses scikit-learn plus short from-scratch snippets.
Linear regression
Section titled “Linear regression”Model . Minimize squared error . Closed form (normal equations) , or solved by QR/SVD, or by gradient descent (Linear Algebra, Calculus & Optimization).
- Ridge adds : ; stabilizes collinear features.
- Lasso adds : sparse weights, no closed form (coordinate descent).
- Feature scaling matters for regularized models and for gradient descent.
- Nonlinearity via feature engineering: polynomial terms, interactions, splines.
- Assumptions for inference (coefficient p-values): linearity, independent homoscedastic Gaussian errors; for prediction these matter much less.
import numpy as nprng = np.random.default_rng(0)X = rng.normal(size=(200, 3)); w_true = np.array([2.0, -1.0, 0.0])y = X @ w_true + 0.1 * rng.normal(size=200)w = np.linalg.lstsq(X, y, rcond=None)[0]print(np.round(w, 2)) # ~ [2, -1, 0]Logistic regression
Section titled “Logistic regression”For binary : with . The decision boundary is a hyperplane. Train by minimizing cross-entropy (negative log-likelihood):
The loss is convex with no closed-form solution; use L-BFGS or gradient methods. For classes use softmax (multinomial logistic regression). Coefficients are interpretable as log-odds changes. With separable data the unregularized MLE diverges (), which is one reason regularization (scikit-learn default , i.e. ) is on by default.
def fit_logreg(X, y, lr=0.1, steps=2000, l2=1e-3): w = np.zeros(X.shape[1]); b = 0.0 for _ in range(steps): p = 1 / (1 + np.exp(-(X @ w + b))) w -= lr * (X.T @ (p - y) / len(y) + l2 * w) b -= lr * (p - y).mean() return w, bLogistic regression on good features is a remarkably strong, fast, well-calibrated baseline. It is also precisely a one-layer neural network (Neural Networks).
Support vector machines
Section titled “Support vector machines”For separable data, choose the hyperplane with maximum margin: minimize s.t. . The margin is . Soft-margin SVM allows violations via slack :
the hinge loss plus regularization; trades margin width against training errors. Only points on or inside the margin (support vectors) determine the solution.
The dual form depends on the data only through inner products , enabling the kernel trick: replace with to get nonlinear boundaries without computing . The popular RBF kernel corresponds to an infinite-dimensional feature space; sets locality. Cost: kernel SVMs scale roughly to in training samples, so they suit up to tens of thousands of examples. For larger data use linear SVMs or other models. SVMs were dominant in the 2000s; today they are niche but instructive (margin, convexity, kernels reappear in Gaussian processes and the neural tangent kernel).
k-nearest neighbors
Section titled “k-nearest neighbors”Predict by the majority label (or mean) among the closest training points. No training phase; all cost at inference ( naive; use KD-trees, ball trees, or approximate nearest-neighbor indexes such as HNSW or FAISS in high dimension). Small gives low bias and high variance; large smooths. Requires scaled features and a meaningful distance. Suffers from the curse of dimensionality: in high dimensions distances concentrate and “nearest” loses meaning. Nearest neighbor search on learned embeddings is, however, the backbone of retrieval and RAG systems.
Naive Bayes
Section titled “Naive Bayes”Apply Bayes’ rule with the “naive” assumption that features are conditionally independent given the class:
Variants: Gaussian (continuous), multinomial (word counts), Bernoulli (binary features). Parameters are estimated by counting; use Laplace smoothing (add pseudo-count ) so unseen feature-class combinations do not zero the product. Work with log-probabilities. Independence is almost never true, yet predictions are often good because only the argmax matters, although probabilities are poorly calibrated. It is extremely fast, works with little data, and is a classic spam/text baseline.
Decision trees
Section titled “Decision trees”A tree recursively splits the feature space with axis-aligned tests (), predicting a constant (class majority or mean) in each leaf. At each node pick the split maximizing impurity decrease:
- Classification: Gini or entropy (information gain; see Information Theory).
- Regression: variance (MSE) reduction.
Pros: handles mixed types and missing values (implementation dependent), no scaling, captures interactions, interpretable when shallow. Cons: unstable (small data change gives a different tree), high variance, piecewise-constant, deep trees overfit. Control with max_depth, min_samples_leaf, or pruning.
from sklearn.tree import DecisionTreeClassifier, export_text# tree = DecisionTreeClassifier(max_depth=3).fit(X, y); print(export_text(tree))Ensembles: bagging and random forests
Section titled “Ensembles: bagging and random forests”Averaging reduces variance: for models with variance and pairwise correlation , the average has variance . So you want low-bias, high-variance, decorrelated models.
Bagging trains each tree on a bootstrap sample. Random forest also considers only a random subset of features ( for classification, for regression) at each split, which decorrelates trees. Predictions average (or vote). Benefits: strong out-of-the-box accuracy, few hyperparameters, parallelizable, built-in validation via out-of-bag error (the ~37% of samples left out of each bootstrap), feature importance estimates. Limits: cannot extrapolate beyond training range, large models, impurity-based importance is biased toward high-cardinality features (prefer permutation importance).
Gradient boosting
Section titled “Gradient boosting”Boosting builds an additive model sequentially; each weak learner (a shallow tree) fits the negative gradient of the loss at the current predictions:
For squared loss the residuals are just ; for logistic loss they are . This is gradient descent in function space. Boosting mainly reduces bias, so it uses shallow trees (depth 3 to 8) and needs regularization: small learning rate (0.01–0.1), subsampling rows and columns, limited leaves, on leaf values, and early stopping on validation.
Modern implementations: XGBoost, LightGBM, CatBoost (second-order gradient information, histogram binning for speed, native categorical handling in CatBoost/LightGBM, missing value handling). On tabular data, boosted trees typically match or beat deep networks with far less tuning; benchmark studies through the early 2020s found trees still competitive or superior on medium-sized tabular problems, while newer pretrained tabular foundation models (for example TabPFN variants) have narrowed or reversed this on small datasets. This field moves quickly, so verify on your own data.
from sklearn.ensemble import HistGradientBoostingClassifier# clf = HistGradientBoostingClassifier(learning_rate=0.05, max_depth=6,# early_stopping=True).fit(X_train, y_train)Practical default for a tabular project: logistic regression baseline, then LightGBM/XGBoost, with tuned learning rate, number of trees (by early stopping), depth, and subsampling.
Clustering
Section titled “Clustering”Unsupervised grouping of similar points.
k-means. Minimize within-cluster sum of squares by alternating (Lloyd’s algorithm): assign each point to the nearest centroid; recompute centroids as means. Converges to a local optimum, so use k-means++ initialization and multiple restarts. Assumes spherical, similar-size clusters; sensitive to scale and outliers; must be chosen (elbow method, silhouette score).
def kmeans(X, k, iters=50, seed=0): rng = np.random.default_rng(seed) C = X[rng.choice(len(X), k, replace=False)] for _ in range(iters): d = ((X[:, None, :] - C[None]) ** 2).sum(-1) z = d.argmin(1) C = np.array([X[z == j].mean(0) if (z == j).any() else C[j] for j in range(k)]) return z, CGaussian mixture models give soft assignments by fitting with the EM algorithm; k-means is the hard, spherical, equal-variance limit.
DBSCAN finds arbitrary-shaped dense regions and labels outliers as noise; needs radius and min-points parameters. Hierarchical (agglomerative) clustering builds a dendrogram by repeatedly merging the closest clusters. Evaluating clusters without labels is hard: silhouette, stability under resampling, and downstream usefulness are the practical criteria.
PCA and friends
Section titled “PCA and friends”PCA (Linear Algebra) projects centered data onto top eigenvectors of the covariance. Use it to reduce dimension before kNN/clustering, decorrelate features, denoise, or visualize. Keep components until explained variance reaches a target (e.g. 95%). Scale features first. For nonlinear structure use t-SNE or UMAP for 2D visualization only: distances between clusters and cluster sizes in those plots are not reliable quantitative evidence. Autoencoders (Generative Models) are a nonlinear generalization of PCA.
Choosing among models
Section titled “Choosing among models”| Situation | First choice |
|---|---|
| Tabular, medium data | Gradient boosting (LightGBM/XGBoost/CatBoost) |
| Need interpretability, fast baseline | Regularized linear/logistic regression, shallow tree |
| Very high-dimensional sparse (text, bag of words) | Linear models (logistic/linear SVM), naive Bayes |
| Small data, strong prior knowledge | Linear models, Bayesian methods, kernel methods |
| Images, audio, text, sequences | Deep learning (CNNs & RNNs, Transformers) |
| Need nearest-neighbor retrieval | Embeddings plus ANN index |
| No labels | k-means/GMM/DBSCAN, PCA; or self-supervised embeddings first |
The “no free lunch” theorem formalizes that no model dominates on all problems; inductive bias must match the data. Trees assume axis-aligned piecewise-constant structure; linear models assume additivity; CNNs assume locality and translation equivariance.
Pitfalls
Section titled “Pitfalls”- Not scaling features for kNN, SVM, k-means, PCA, and regularized models.
- Using accuracy on imbalanced data (ML Fundamentals).
- Tuning boosting without early stopping, or with a leaky validation set.
- Interpreting feature importances as causal effects, or relying on impurity importance with correlated features.
- Treating t-SNE/UMAP geometry as meaningful beyond local neighborhoods.
- Ordinal-encoding categorical variables for linear models; one-hot (or target encoding done within CV folds) is safer.
Key takeaways
Section titled “Key takeaways”- Linear and logistic regression are convex, fast, interpretable baselines; regularize them.
- SVMs maximize margin; kernels give nonlinearity at a quadratic-or-worse cost in .
- Single trees have high variance; random forests (bagging plus feature subsampling) reduce it, boosting reduces bias by fitting residual gradients.
- Gradient-boosted trees are the strong default for tabular data.
- k-means is simple but assumes spherical clusters; try GMM or DBSCAN when that fails.
- PCA is the linear dimensionality-reduction workhorse; t-SNE/UMAP are visualization tools.
Exercises
Section titled “Exercises”- Conceptual. Why does random forest subsample features at each split rather than only bootstrapping rows?
- Derivation. Show that for squared loss the negative functional gradient used in gradient boosting is the residual .
- Hands-on. On a tabular dataset (e.g. California housing or Adult), compare ridge, random forest, and a gradient-boosted model with 5-fold CV. Record RMSE and training time.
- Hands-on. Implement k-means from scratch and test on two concentric rings. Why does it fail? Fix it with DBSCAN or spectral clustering.
- Conceptual. Why can naive Bayes classify well even though its independence assumption is false?
Solutions (1, 2, 5)
- Without feature subsampling, strong predictors dominate the top splits of every tree, making the trees highly correlated ( large), which limits variance reduction in .
- , so .
- Classification only needs the correct ranking of class posteriors, not accurate probabilities. Correlated features inflate the evidence for all classes in a similar way, so the argmax is often preserved. Probabilities, however, become overconfident.
Further reading
Section titled “Further reading”- Hastie, Tibshirani, Friedman, The Elements of Statistical Learning (2nd ed., 2009).
- James, Witten, Hastie, Tibshirani, An Introduction to Statistical Learning (2nd ed., 2021).
- Breiman, “Random Forests” (2001).
- Friedman, “Greedy Function Approximation: A Gradient Boosting Machine” (2001).
- Chen and Guestrin, “XGBoost: A Scalable Tree Boosting System” (2016).
- Grinsztajn, Oyallon, Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?” (2022).
- Schölkopf and Smola, Learning with Kernels (2002).
- scikit-learn User Guide, supervised and unsupervised learning sections.