promptdojo_

Acceptance before the build — review against checks, not vibes — step 2 of 8

Evidence against the checks

Run it. Three checks, three recorded results, two BLOCKING flags — and only one of them is a recorded failure. C1 is the open-every-source check: a named reviewer ran it, it failed, and that line is the entire Deloitte case compressed to 40 characters. C2 is sneakier. The builder says the sections are all present, and the builder is probably right — but "probably right, says the thing being graded" is not evidence, so the gate downgrades the result to unverified and flags it anyway. The document could be gorgeous. Neither flag cares.

The structure has four parts worth internalizing:

  • Checks carry a blocking flag. Not all criteria are equal, and pretending they are is how teams end up relitigating scope at review time. "Sources are real" is blocking: fail it and the deliverable cannot ship, full stop. "Exec summary fits on 2 pages" is real but non-blocking: fail it and you ship with notes and a follow-up task. Deciding which is which at spec time is a negotiation you have once, calmly, instead of at 5pm on ship day, loudly.
  • Evidence is a record with a name on it, not a recollection. EVIDENCE maps check id → result and who produced that result — a reviewer, a script, the builder itself. Next lesson, this exact record becomes the input to your status roll-up — which is the trick of the whole chapter: status stops being what people say and starts being what the acceptance records contain.
  • A missing result is not a pass. Look at the .get(c["id"], ...) default — if nobody ran the check, the record says missing, and the gate treats anything that isn't pass as a failure. This is the polar opposite of how human review defaults work ("no one flagged it, so it's fine"), and it's the correct default when the thing being reviewed was generated by a system that fabricates fluently. Silence is not evidence. Only evidence is evidence.
  • The builder never attests to its own blocking checks. That's the C2 line, and it's a rule from the case, not a flourish: asking the AI that drafted the report whether its citations are real is asking the failure mode to certify itself. The gate encodes the rule — a blocking check whose only evidence is builder-attested comes out unverified, and unverified is not pass. Non-blocking checks (C3), delegate freely; if the builder fudges the exec summary's page count, the cost of catching it later is a shrug.

Who runs the checks

Not necessarily you — that's the bottleneck trap this chapter's title warns about. C2 (sections present) can be checked by a script, thirty seconds of diffing against the agreed outline — the point isn't that a human must do it, it's that the builder can't be the one whose word closes a blocking check. C1 needs a human opening sources, but it can be any named human with the checklist; the check is written so that a stranger can run it. Your job is designing the gate and making the call on the record it produces. The moment every check needs you personally to run it, you've rebuilt the bottleneck with extra steps.

One note before the drills: they consume the flattened form of this record — a plain {"C1": "pass"} map. That's not a loosening. It's the record after verification has run: by the time a pass gets written down, something other than the builder produced it, or the blocking check was downgraded on the spot exactly as above. In the drills, you're the review runner. The gate you're about to fix trusts the record because this step is where the record earned it.