The last door before your users
You've written evals (lesson 01), hired a judge and audited its biases (lesson 02). One question is left: who decides whether the new model actually ships? If the answer is "whoever feels good about it on Friday," everything upstream was decoration.
A validation gate is that decision, written down as code. It is exactly three things:
- A threshold — a number, chosen before you saw the candidate's score. Chosen after, it's not a threshold; it's a justification.
- An eval set — fixed, versioned, frozen. If the cases change every run, the score isn't comparable to last week's and the threshold is measuring nothing.
- A decision rule — what happens on failure. "Block the deploy" is a rule. "Post in Slack and see" is not.
Run the editor. The candidate scores 0.83 against a 0.85 floor, and
the gate says BLOCK — no meeting required.
Absolute gates and regression gates
An absolute gate compares the candidate to a fixed bar: accuracy at least 0.85, zero schema violations, refusal rate under 2%. It answers "is this good enough to face users?"
A regression gate compares the candidate to the last shipped model on the same frozen set: new score must be at least old score minus a small tolerance. It answers "is this at least as good as what users already have?" — which is the question your users are silently asking every time you deploy.
You want both. A candidate can clear a 0.80 floor while quietly scoring four points below the model it replaces. The absolute gate smiles; the regression gate blocks. You'll build exactly that case in step 3.
Overrides happen. Unrecorded overrides are the sin.
Sometimes shipping past a red gate is the right call — the regression is confined to a locale you're sunsetting, the fix ships tomorrow, the outage is worse than the dip. Fine. But the override is a decision by a person, and it gets recorded: who overrode, why, and which gates were red at the time. Same receipts culture as the judge lesson — a verdict without a critique is noise, and an override without a name and a reason is just a gate someone deleted quietly. Six months later, when someone asks why v9 shipped below the bar, the deploy log answers instead of nobody remembering.
The theater gate
Here's the failure mode that makes this lesson worth its slot: teams add a gate to satisfy a checklist, set the threshold at a level the worst model they've ever trained would clear, and enjoy years of green checkmarks. A gate that cannot fail is not protecting the deploy — it's decorating it. The test is brutal and simple: when did this gate last block anything? If the answer is "never, and it never could," you've built a ribbon to cut, not a door that locks. Step 5 has you fix one that computes its threshold from the candidate's own score — always green, forever, by construction.
Six steps ahead: spot the theater gate in a lineup, predict an absolute-passes-regression-fails split, wire the regression comparison, un-rig the rigged gate, build the gate runner, and finish with an override log that names names.