Classical machine learning

Overview

Classical machine learning is the set of models that predate deep learning and have not gone away: linear and logistic regression, decision trees and the ensembles built from them, support vector machines, nearest-neighbour methods, clustering and principal component analysis. They train in seconds on a laptop, need no GPU, and on tabular business data they are still usually the best choice. Grinsztajn et al. compared tree ensembles against tuned neural networks on 45 tabular datasets in 2022 and found the trees ahead on most of them, with far less tuning.

For a working engineer this hub is about choosing. A logistic regression that ships this week with coefficients the product team can read often beats a transformer that ships next quarter. Knowing which model fits which data shape, and what each needs from the features, is most of the skill.

Key ideas

Linear models are the baseline and often the answer

Linear regression fits a weighted sum of features to a number. Logistic regression passes the same sum through a sigmoid to get a probability. Both are convex, so training always finds the same optimum, and both are readable: a coefficient is the change in the output for a unit change in the feature, holding the others fixed. Ridge and lasso add a penalty on the weights, which controls variance and, for lasso, drives unhelpful coefficients to exactly zero.

What they need from you is scaled features, since the penalty treats all coefficients alike, and explicit interactions or transforms, since a straight line cannot bend by itself. With one-hot encoded categories, polynomial features and a bit of domain knowledge, linear models reach further than people expect, and they fail loudly rather than quietly.

Trees and the ensembles built from them

A decision tree splits the data on one feature at a time, choosing the split that most reduces impurity, and keeps splitting until the leaves are pure or a size limit is reached. One tree is readable and unstable: a small change in the data gives a different tree. Random forests fix the instability by averaging hundreds of trees, each trained on a bootstrap sample with a random subset of features at each split. Gradient boosting does something different and usually better: it trains small trees one after another, each fitted to the residual errors of the ensemble so far, scaled by a learning rate.

Tree ensembles need almost nothing from the features: no scaling, no one-hot encoding in LightGBM and CatBoost, and they tolerate missing values. The levers are tree depth, learning rate, number of trees with early stopping, and subsampling. Their weaknesses are extrapolation, since a tree cannot predict outside the range it saw, and very high-cardinality sparse inputs such as raw text, where linear models or neural networks do better.

Margins and neighbours

A support vector machine finds the separating boundary with the widest margin and, through the kernel trick, can do so in an implicit high-dimensional space without computing it. With an RBF kernel it is a strong choice for small, dense, scaled datasets, up to tens of thousands of rows, after which training time grows badly. k-nearest neighbours has no training at all: it stores the data and predicts from the k closest points at query time. It is the simplest possible model and the one behind similarity search, where the stored points are embeddings. Both are sensitive to feature scale and to irrelevant features, because distance treats every dimension alike.

Finding structure without labels

k-means partitions points into k clusters around centroids and is fast and easy to reason about, but assumes roughly spherical clusters of similar size and needs k chosen in advance. DBSCAN finds clusters of any shape by density and labels sparse points as noise. Hierarchical clustering builds a tree of merges you can cut at any level. All of them run on scaled features, and all of them will find clusters in pure noise, so inspect the result rather than trusting a silhouette score alone.

PCA rotates the data onto the directions of greatest variance, which is the singular value decomposition of the centred data matrix. The first few components give a two-dimensional view for plotting, a decorrelated input for a linear model, or a compressed representation that drops noise. The components are combinations of the original features, so read the loadings before giving one a name.

Where to start

Begin with Linear and logistic regression from scratch, then Decision trees and gradient boosting, which trains the ensemble in the diagram on a bundled dataset. Evaluation that does not lie covers how to compare the two fairly. Embeddings and similarity search is k-NN in modern clothing. The tabular classifier project takes a gradient boosting model to a running endpoint.

Tools and platforms

Common mistakes

• Feeding unscaled features to an SVM, k-NN, lasso or k-means. One feature measured in thousands dominates the distance or the penalty.
• Reading impurity-based feature importances from a forest as if they were causal. Prefer permutation importance on the validation set, and remember that correlated features share credit.
• Choosing the number of boosting rounds on the test set. Use early stopping on a validation fold and keep the test set for the final number.
• Letting a tree model extrapolate. A house-price model trained on homes under 1 M will predict about 1 M for a 5 M home.
• Trusting k-means output because the silhouette score looked fine. Clustering always returns clusters, plot them.
• Calling fit on the full dataset and then splitting, see data and features.

Going deeper

• Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, 2009, the standard reference for everything on this page.
• James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning, the gentler version with Python labs.
• Chen and Guestrin, XGBoost: A Scalable Tree Boosting System, 2016.
• Breiman, Random Forests, Machine Learning, 2001.
• Grinsztajn, Oyallon and Varoquaux, Why do tree-based models still outperform deep learning on tabular data?, 2022.
• The scikit-learn user guide, which documents each estimator with its assumptions.