A golden case is a past incident with a verdict
Run the code. Twelve lines of data, and every line has a story. G2 is the outage where a p3 issue quietly absorbed a prod incident. G4 is the 2 a.m. page for a registry blip that fixed itself before the on-call's laptop finished waking up. The golden set is your team's incident history, compiled to something executable.
Each case is three fields — id, input, expected route. The FORBIDDEN table adds the sharper edge: the route that must never fire, with the incident that taught you. Expected-route checks catch wrong answers; forbidden-route checks catch expensive wrong answers. "G4 should rerun" failing to open_issue is noise in the tracker. Failing to escalate is a human awake at 2 a.m. — same wrong answer count, very different postmortem, and your eval output should distinguish them.
Where cases come from, in practice
- Grep CI history for the incidents that had consequences. The failure that ate an afternoon, the page that shouldn't have fired, the flake that got a real bug ignored. Consequences are what make a case worth its slot.
- One case per failure mode, not per failure. You don't need forty timeout cases; you need one timeout, one registry 503, one prod healthcheck, one honest assertion failure. Coverage of modes beats volume of examples.
- The expected answer comes from the human who handled it. Not from running the router and freezing whatever it said — that's how you enshrine today's bugs as tomorrow's spec. If nobody can say what the right call was, it isn't a golden case yet; it's a discussion for standup.
The set only grows
One rule turns this from a test file into an asset: cases are never deleted to make the run green. A case that fails after a change is the eval doing its job; the router changes, not the history. Deleting G4 because it's inconvenient is deleting the memory of the 2 a.m. page — and booking a repeat.
Cases get added on a trigger you'll drill at the end of this lesson: every new incident becomes a new case the day it happens. Ship week one with four cases; a quarter of real traffic later you have fifteen, each one a mistake the feature can no longer repeat without a red run telling on it. That accumulation is the ratchet — and it's the difference between a feature that ages into reliability and one that stays exactly as trustworthy as its demo.