Wire the evals before the feature "works"
You already know this rhythm. Write the failing test, watch it fail, make it pass, keep it forever. TDD for agent features is the same move with one twist: the thing under test isn't your code's logic — it's the behavior of a system that includes a model, and "correct" is defined by cases, not types.
For the triage agent, the unit of correctness is a golden case: a real past incident plus the routing decision that was right for it, verified by whoever handled it.
("G2", "prod deploy failed healthcheck", "escalate")
That case is not hypothetical. That's the Tuesday prod went down and the on-call found out from a customer because some earlier automation filed a p3 instead of paging. The case encodes the lesson so the eval runner re-teaches it to every future version of the router, forever, for free.
Why "before it works" is the load-bearing part
The order of operations is the whole discipline:
- Collect four or five real incidents from CI history. Write each as
(id, job_text, correct_route). - Write the eval runner — a loop that routes each case and prints pass/FAIL with a summary.
- Now write the routing logic, and code until the runner says
4/4 — ship.
Do it in this order and "done" has a definition a script can check. Do it in the other order — feature first, evals someday — and "done" means "the three cases I tried by hand looked right," which is how the npm-503 case ends up routing to open_issue on launch day and you find out from the issue tracker's noise, not from a red eval.
There's a second-order effect too: writing the cases first changes the routing logic you write. Stare at "npm registry 503 during install → rerun" before coding and you'll never write a router that only checks for the word "timeout." The eval set is a spec you can execute, and specs you write after shipping are just documentation of the bugs.
What golden cases are not
- Not a benchmark. You're not measuring the model against the field; you're checking your feature still makes the calls your team already paid to learn.
- Not proof of future behavior. Four green cases mean the router handles four known situations. The queue will invent a fifth — that's what the ratchet at the end of this lesson is for.
- Not big. Four to a dozen cases per feature, each one earned by an incident. A thousand synthetic cases nobody can vouch for is worse than five real ones someone can, because when a synthetic case fails you argue about the case instead of fixing the router.
The eval run becomes your ship gate: green means deploy, red means the feature — not the test — needs work. Same as every other test suite you respect. The rest of this lesson builds the runner, turns it red, and makes it green the honest way.