promptdojo_

Numbers with receipts — every figure in the ops report traces to the export or it doesn't ship — step 2 of 8

Recompute, don't trust

Run it. The report claims average days-to-ship is 4.2. The export — all five rows of it — says 6.0. The draft was off by almost two days on the one number that decides whether this week's shipping performance was fine or a problem.

Notice how the invented number feels: 4.2 is exactly the kind of average this data could plausibly have. It's not absurd. It's not flagged. If you eyeballed the rows — 2, 9, 3, 4, 12 — would you have caught it? The 12 drags the mean up more than intuition expects; that's precisely the kind of arithmetic models fumble and humans don't re-do by hand. Which is why the check is code, not squinting.

Three details in this tiny verifier that carry the whole method:

  • The actual comes from the export, computed in front of you. sum(...) / len(...) over the rows. Not the model's stated methodology, not a second model asked to double-check the first — the same file the report claims as its source, re-aggregated. The receipt is only a receipt if you pull it yourself.
  • The comparison uses a tolerance, not ==. abs(claimed - actual) < 0.05 accepts honest rounding — a report that says 6.0 when the raw mean is 5.98 is fine. Float equality would flag legitimate rounding as fraud and bury the real catches in noise; a sloppy tolerance like ±0.5 would wave the fabrications through. The tolerance is a policy choice, so it's visible, in one place, on purpose.
  • The mismatch line names both numbers. report says 4.2, export says 6.0 is a sentence you can put in front of anyone — your boss, the model in a retry prompt, yourself in three weeks — and the correction is unambiguous. "This number is wrong" starts an argument; "the export says 6.0" ends one.

The pattern generalizes to every figure a report can claim: counts, maxima, percentages, rates. Each claim gets a recomputation; each recomputation is a few tokens of stdlib. The rest of the lesson scales this from one number to the whole report — and then puts a SEND/HOLD gate on the result, because a report with one unverified number isn't 95% done. It's not done.