Overfitting in the wild, and the knobs that fight it
Chapter 38 showed the memorizer in principle. Here it is with a real (tiny) model family, where one knob slides you between memorizing and generalizing.
Run it
k-nearest-neighbors predicts by looking at the k closest training points and voting. With k=1 the model is a memorizer: perfect on train (every point is its own nearest neighbor — including the noisy ones), notably worse on test. As k grows, votes average over neighbors, noise cancels, and the train/test gap closes.
The diagnostic is always that gap:
- train high, test low → overfitting → constrain the model.
- both low → underfitting → more capacity or better features.
- both decent and close → you're in business.
Regularization: the knob family
Every model family has knobs that limit how hard it can memorize — collectively, regularization:
- k in k-NN (bigger = smoother), tree depth and min-samples-per-leaf in trees/forests.
- Penalty strength in linear models — sklearn's
LogisticRegression(C=...): an extra cost on large weights keeps the model from contorting to fit noise. - In deep learning (chapter 43): dropout, weight decay, early stopping. Same idea, same reason.
More training data is regularization too — noise is harder to memorize when there's more signal to explain — and often the cheapest fix of all.
Tuning without cheating
Knob-tuning is a decision, so it must not touch the test set
(chapter 38's rule). The standard tool is cross-validation:
split the training data into folds, train on all-but-one, validate
on the held-out fold, rotate, average — sklearn's
cross_val_score(model, X_train, y_train, cv=5). Pick the knob by
CV score; open the test set once, at the end, for the number you
can actually report.
Where AI specifically gets this wrong
- Default hyperparameters presented as tuned. Generated code
ships
RandomForestClassifier()bare and reports test accuracy as if optimized. Defaults are a starting rung, not a result. - Tuning on test. A generated loop that tries ten settings and keeps the best test score has quietly spent the test set. CV on train; test once.
- "More epochs" as the answer to everything. Training longer helps underfitting and worsens overfitting — check which side of the gap you're on before turning that particular knob.