promptdojo_

Golden evals earn autonomy — every macro proves itself per intent — step 2 of 7

A macro is three things, and the third one is the job

Every helpdesk has macros — canned answers with merge fields, one per recurring intent. Most teams write them as one thing: the answer text. A macro you can trust an AI to reach for is three things:

  1. The intent — the one question this macro answers. Not "shipping stuff." One intent: where is my order. The narrower the intent, the easier everything downstream gets, because golden cases for a narrow intent actually cover it.
  2. The approved answer — with its KB citation, so the lesson-1 gate can check freshness. When KB-204 changes, this macro is automatically suspect. That's a feature.
  3. The never_when boundaries — the tickets that look like this intent but must never get this answer. Run the code: this macro refuses emotional language, legal or chargeback threats, third contacts, high-value orders, and anything multi-step.

Look at that never_when list again. It's Klarna's incident report, inverted. Emotionally charged, multi-step, edge-case: the three territories where the documented failures clustered are written into the macro as refusals. The boundary list isn't defensive boilerplate — it's the distilled cost of somebody else's outage.

Why "never_when" beats "when"

You might expect the opposite design: describe the tickets the macro should handle. But intents are recognized sloppily — by keyword, by classifier, by a tired agent at 4pm — and a "when" list quietly expands to fit whatever matched. The customer whose third where-is-my-order email ends "this is theft and I'm calling my bank" is asking where their order is. The intent matches. The ticket must still never get the cheerful tracking-link macro, because the actual situation stopped being about tracking two emails ago.

Boundaries are checked after intent matching, as a veto. Match loosely, veto strictly. That order of operations is what keeps a macro's blast radius small.

Status is earned, and it starts at the bottom

The last two fields are the autonomy machinery. status: "review-required" means every use of this macro passes full human review — and that's where every macro starts, no matter how obviously correct the answer text looks. golden_cases: 0 is why: this macro has proven itself against zero real tickets. Zero evidence, zero trust; the demo doesn't count.

The promotion path is the rest of this lesson. Golden cases come from the edit log you started keeping in lesson 1 — every logged correction is a real ticket where the drafter went wrong, with the known-good outcome attached. Five or more of those per macro, all passing, is the entry ticket to eval-passed status. Next steps: drill the boundary, then run the evals.