Regression metrics and the residual habit
When the label is a number — minutes, dollars, demand — the confusion matrix doesn't apply. The questions stay the same though: how wrong are we typically, and where are we wrong?
MAE and RMSE, side by side
Run the editor:
- MAE (mean absolute error) — average size of the miss, in the label's own units. "Typically off by about 9 minutes" is a sentence a stakeholder understands.
- RMSE (root mean squared error) — squares the misses first, so big errors dominate. The one 45-minute miss barely moves MAE and inflates RMSE — which is exactly the signal: a large RMSE-to-MAE gap means your error budget is being spent on a few disasters, not spread evenly.
Which to optimize is, again, a product call: if one catastrophic mis-estimate costs more than ten small ones (delivery promises, capacity planning), weight RMSE. If errors cost linearly, MAE. (R², which you'll also meet, asks "how much better than predict-the-mean?" — chapter 38's baseline, rebuilt as a metric: 0 means no better, 1 means perfect.)
Residuals: the error's autopsy
The residual is actual − predicted, per row — the last line
of the editor. The habit that finds bugs: don't just average
residuals, look at their structure:
- Mostly positive residuals → systematic under-prediction (a bias the mean metrics can hide if big negatives offset it).
- Residuals growing with the prediction → the model's good on small orders, lost on large ones — slice by size (last lesson, regression edition).
- A few enormous residuals → read those rows by hand. In this editor's data, that 90-minute delivery is either a data error or the most informative example you own.
Where AI specifically gets this wrong
- Optimizing RMSE by deleting outliers. Generated cleanup drops the rows that hurt the metric — sometimes those rows are the product problem (the deliveries that go badly wrong). Investigate before deleting, and document either way.
- Reporting R² alone. "R²=0.84" with no unit-scale error number tells a stakeholder nothing about whether the misses are livable.
Above all: skipping the residual look entirely. One histogram and one residual-vs-prediction glance catch more model bugs than a week of hyperparameter tuning.