promptdojo_

Evals before it works — golden cases and the ratchet — step 8 of 8

Checkpoint

One last thing before we move on. Same surface as a write step — but the lesson doesn't complete until this passes.

Final checkpoint — the ship gate, run twice in one sitting, exactly the way it runs in real life.

Write ship_gate(golden):

  • For each case, route the job text and compare to the expected route. Match → print <id> pass.
  • Miss → check FORBIDDEN: if the route it did pick is that case's must-never route, print <id> FAIL (forbidden: <route>). Any other miss is plain <id> FAIL. Wrong is wrong, but the expensive kind of wrong gets named.
  • All green → final line N/N SHIP.
  • Any red → P/N HOLD (K failing) where K is the number failing.

Then the plot: run the gate on the four-case set (it's green — the 503 fix from earlier is in). A new incident lands: runner 7's disk has been full for three days, and each day the agent filed another issue instead of paging infra. Append ("G5", "runner disk full for the third day", "escalate"), record the route that caused the incident — FORBIDDEN["G5"] = "open_issue" — and run the gate again. The set only grows, and today's SHIP becomes HOLD with a FAIL line naming the exact mistake the router keeps making.

Expected output:

G1 pass
G2 pass
G3 pass
G4 pass
4/4 SHIP
G1 pass
G2 pass
G3 pass
G4 pass
G5 FAIL (forbidden: open_issue)
4/5 HOLD (1 failing)

full-screen editor opens — close anytime to keep reading.