Overfitting, underfitting, and leakage: the three ways to fool yourself
A model's job is to work on data it has never seen. Every methodological rule in ML exists to protect one measurement: an honest estimate of that.
The memorizer, live
Run the editor. The "model" is a lookup table of the training set: 100% on training data, 50% (coin-flip) on new data. That gap is overfitting — the model learned the noise and specifics of its sample instead of the pattern. The opposite failure, underfitting, is a model too simple to capture the pattern at all: bad on training and test.
The classic mental model (bias/variance): simple models err by rigidity (high bias), flexible models err by mirroring their particular sample (high variance). You navigate between the two — and the only instrument that can tell you where you are is held-out data. Train score alone cannot distinguish brilliance from memorization; the editor just proved it.
The split is sacred
So: split before anything else. Train on one part, evaluate on a part the model — and you — never touched while making decisions. Two subtleties that bite:
- Tuning against the test set (trying ten ideas, keeping what helps on test) slowly overfits the test set through your own choices. Use a third slice (validation) for tuning; open the test set rarely.
- Time matters: for anything predictive, split by time (train on past, test on later) — a random split lets the model peek across time in both directions.
Leakage: overfitting's evil twin
Leakage is when information unavailable at prediction time
sneaks into training. You've met it twice: features summed across
the label window (chapter 36) and imputing with full-dataset
statistics before splitting (chapter 35). Add: duplicates straddling
the split, and "features" that are the label wearing a hat
(account_closed_date predicting churn). Leakage produces amazing
offline scores and production faceplants — the exact
confident-but-wrong failure this course keeps warning about.
Where AI specifically gets this wrong
- Preprocessing before splitting. Generated sklearn code loves to scale/impute on the full dataset, then split. Fit preprocessing on train only (chapter 39 shows the pipeline shape that enforces this).
- Reporting train accuracy. If a script prints one accuracy, ask which set it came from. The flattering one, usually.
- Suspiciously perfect scores. 99.8% on first try isn't genius; it's a leak until proven otherwise. Hunt the feature that knows the future.