Mission: the baseline model showdown
"Is the model any good?" deserves a scoreboard, not a debate. Stand up the smallest board that can host a fair fight, put three contenders on it, then stress-test the fairness itself.
The job: one scoreboard, every rung of the ladder, same test set — the artifact that makes "is the model any good?" a lookup instead of a debate.
The stages:
- The board — a
score(name, predict_fn)helper (the starter has it) so every contender is one line to add. - The rungs — majority class, one domain rule, and a simple trained model (a hand-fit threshold learned from train data counts here: pick the ticket-cutoff that maximizes train accuracy, then score it on test — that's a real trained model, one parameter big).
- Honesty checks — every rung scored on the same held-out test set; the train-picked cutoff also reported on train so you can see the gap (chapter 39.4).
- Stress case — rerun
make()with a new seed and watch the scores wobble (chapter 38's sampling noise, live). Report both runs. - One improvement — stratified reporting (accuracy on label=1 rows vs label=0 rows — a preview of chapter 41), or a second feature, or a CV-style repeated split: pick one, note why.
Run the editor: two rungs already on the board. Your mission is the trained rung, the gap report, and the wobble check. The contract for the artifact: a labeled dataset goes in; a scoreboard nobody can argue with comes out.