Mission: confusion-matrix triage
A confusion matrix you can compute; triage is deciding what to do about it. Build the smallest report that turns scores into decisions, run it against one lopsided eval set, then armor it against small-n overconfidence.
The job: a triage(scored_rows) report that takes model scores +
labels + segments and answers the three questions this chapter
taught, in one screen: how does it fail, for whom, and what should
the threshold be?
The stages:
- Matrix per slice — the starter: four boxes per segment. Extend with per-slice precision/recall (guard the zero denominators like the lesson code did).
- Threshold sweep — precision/recall at three thresholds, printed as a table; mark the row you'd ship for a recall-matters product and the row for a precision-matters one.
- Worst-slice detail — for the slice with the most FNs, print the actual missed rows (score, label) — the "read the failures" habit, mechanized.
- Stress case — feed it an all-negative segment (n small) and make sure the report prints n and refuses to editorialize on six rows.
- One improvement — an F1 column, a cost-weighted verdict (FP=$1, FN=$10 — total cost per threshold), or a majority- baseline row for shame comparison: pick one, note why.
In: a scored eval set. Out: a one-screen triage a product owner could act on. This is the report you'll wire into chapter 46's CI gate, so make it a function, not a notebook scroll.