promptdojo_

Regression metrics and residuals — step 1 of 7

Regression metrics and the residual habit

When the label is a number — minutes, dollars, demand — the confusion matrix doesn't apply. The questions stay the same though: how wrong are we typically, and where are we wrong?

MAE and RMSE, side by side

Run the editor:

  • MAE (mean absolute error) — average size of the miss, in the label's own units. "Typically off by about 9 minutes" is a sentence a stakeholder understands.
  • RMSE (root mean squared error) — squares the misses first, so big errors dominate. The one 45-minute miss barely moves MAE and inflates RMSE — which is exactly the signal: a large RMSE-to-MAE gap means your error budget is being spent on a few disasters, not spread evenly.

Which to optimize is, again, a product call: if one catastrophic mis-estimate costs more than ten small ones (delivery promises, capacity planning), weight RMSE. If errors cost linearly, MAE. (R², which you'll also meet, asks "how much better than predict-the-mean?" — chapter 38's baseline, rebuilt as a metric: 0 means no better, 1 means perfect.)

Residuals: the error's autopsy

The residual is actual − predicted, per row — the last line of the editor. The habit that finds bugs: don't just average residuals, look at their structure:

  • Mostly positive residuals → systematic under-prediction (a bias the mean metrics can hide if big negatives offset it).
  • Residuals growing with the prediction → the model's good on small orders, lost on large ones — slice by size (last lesson, regression edition).
  • A few enormous residuals → read those rows by hand. In this editor's data, that 90-minute delivery is either a data error or the most informative example you own.

Where AI specifically gets this wrong

  • Optimizing RMSE by deleting outliers. Generated cleanup drops the rows that hurt the metric — sometimes those rows are the product problem (the deliveries that go badly wrong). Investigate before deleting, and document either way.
  • Reporting R² alone. "R²=0.84" with no unit-scale error number tells a stakeholder nothing about whether the misses are livable.

Above all: skipping the residual look entirely. One histogram and one residual-vs-prediction glance catch more model bugs than a week of hyperparameter tuning.