Evals before it works — golden cases and the ratchet — step 1 of 8

Wire the evals before the feature "works"

You're in the agent studio. What you're doing this lesson: write a few real past incidents as test cases before you trust the helper to route new ones. Green on those cases is what "works" means. You do not need a test-driven-development habit to start.

An eval here is a scored test of the helper's behavior: given this past incident, did it pick the route a human already paid to learn? A golden case is one of those past incidents plus the routing decision that was right for it, verified by whoever handled it.

("G2", "prod deploy failed healthcheck", "escalate")

That case is not hypothetical. That's the Tuesday production (the live system customers use) went down and the on-call (the person whose phone rings) found out from a customer because some earlier automation filed a low-priority issue instead of paging. The case encodes the lesson so the eval runner re-teaches it to every future version of the router, forever, for free.

Why "before it works" is the load-bearing part

The order of operations is the whole discipline:

1. Collect four or five real incidents from CI history (the automated test runs). Write each as (id, job_text, correct_route).
2. Write the eval runner — a loop that routes each case and prints pass/FAIL with a summary.
3. Now write the routing logic, and keep going until the runner says 4/4 — ship.

Do it in this order and "done" has a definition a script can check. Do it in the other order — feature first, evals someday — and "done" means "the three cases I tried by hand looked right," which is how the npm-503 case (the JavaScript package registry was temporarily down) ends up opening a noisy issue on launch day instead of a quiet rerun.

There's a second-order effect too: writing the cases first changes the routing logic you write. Stare at "npm registry 503 during install → rerun" before coding and you'll never write a router that only checks for the word "timeout." The eval set is a spec you can execute, and specs you write after shipping are just documentation of the bugs.

What golden cases are not

• Not a benchmark. You're not measuring the model against the field; you're checking your feature still makes the calls your team already paid to learn.
• Not proof of future behavior. Four green cases mean the router handles four known situations. The queue will invent a fifth — that's what the ratchet at the end of this lesson is for.
• Not big. Four to a dozen cases per feature, each one earned by an incident. A thousand synthetic cases nobody can vouch for is worse than five real ones someone can, because when a synthetic case fails you argue about the case instead of fixing the router.

The eval run becomes your ship gate: green means deploy, red means the feature — not the test — needs work. Same idea as any test suite you already trust, aimed at the helper's behavior instead of your own code. The rest of this lesson builds the runner, turns it red, and makes it green the honest way.