promptdojo_

Evals before it works — golden cases and the ratchet — step 6 of 8

ch 57 · agent studio6/8

Every rule traces back to a failure

The fix you just shipped — the 503 clause — will be a mystery in six months. Someone will read or "503" in text, wonder if it's dead code, and delete it in a refactor. The eval case G4 will catch the regression, but now you're relitigating a solved incident. The ratchet log exists so solved stays solved.

Osmani's rule, lightly paraphrased — his exact wording in Agent Harness Engineering is "every line in a good AGENTS.md should be traceable back to a specific thing that went wrong" — and the discipline worth pinning above your desk: every line in a good system prompt traces back to a specific, historical failure. Same standard for your harness config, your tool contract, your routing rules. Run the code — each entry pairs a rule with the failure that earned it. Not "be robust to registry errors." A dated incident: forty issues, zero reruns, here's what changed.

Five fields, none optional

  • date — anchors the entry to the session where it happened, so you can find the trace (lesson 2 just made sure there is one) and replay the run.
  • failure — the specific narrated event, one sentence. If you can't write the sentence, you're about to add a cargo-cult rule. Go reproduce the failure first, or don't add the rule.
  • rule — phrased as an action with concrete strings, commands, values. "Be careful with reruns" is a vibe. "rerun_job max once per job per run" is enforceable, and you can tell at review time whether a change violates it.
  • artifact — where the rule lives: an AGENTS.md line, a hook, a validator branch, a golden case. The best entries land in two places — the text is the explanation for the model, the code is the enforcement. Note both entries above name a golden case or a cap as their enforcement half.
  • remove_when — the exit condition, and the most-skipped field in the industry. A rule with no exit condition is permanent, and permanent rules are debt: eighteen months from now half your config will be workarounds for model behaviors that no longer exist, each one taxing attention on every run. Write the exit the day you write the rule, even if it feels speculative.

The incident routine, end to end

New failure in production. The routine is four moves, maybe twenty minutes:

  1. Pull the trace. Find the step where it went wrong — verdict, result, spend.
  2. Add the golden case. The incident becomes ("G5", ..., correct_route). The eval run goes red. Good — red is the incident, preserved.
  3. Fix until green. Router, contract, caps — whichever artifact the trace implicates.
  4. Append the ratchet entry. Five fields, linking the failure to the rule to the artifact to the exit condition.

Append-only, audited quarterly, entries retired only when their remove_when comes true — and the retirement itself gets logged. Run this routine for a quarter and your harness is measurably better than at launch, no model upgrade required. Skip it and you have the same agent in month six that you had in week one, plus a pile of hotfixes nobody can explain in review.

read, then continue.