promptdojo_

GitHub Actions validation gates — step 1 of 7

ch 46 · model serving, ci/cd, and mlops1/7

CI validation gates: no evidence, no deploy

Chapter 17 introduced GitHub Actions as the place checks run on every push. For ML, the checks that matter aren't just lint and unit tests — they're the eval suite (chapter 21) and the data/feature contracts (chapters 37/45), run automatically, with deployment refused when they fail.

The gate, as code

Run the editor. A candidate model must (a) clear absolute floors — including per-slice floors, because chapter 41 taught you aggregates hide segment failures — and (b) beat the incumbent. Candidate v6 has the better headline accuracy and gets blocked anyway: its mobile recall collapsed. That's the gate doing the job dashboards don't: making regressions impossible to ship accidentally.

Wired into Actions

name: model-ci
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python check_contracts.py      # ch.37/45 gates
      - run: python run_evals.py            # ch.21/41 suite -> metrics.json
      - run: python gate.py                 # today's logic; exit 1 blocks merge

A failing step fails the workflow — and with branch protection configured to require this check (that part is a repo setting, not a default), a red workflow blocks the merge. Same mechanics as any CI, pointed at model quality. The pieces you've already built slot in directly: the quality checks are chapter 36's SQL gates, the eval suite is chapter 21's, the slice metrics are chapter 41's triage function, and a green run is what authorizes chapter 45's registry promote().

Two cautions before you wire this in. Evals cost money when they call paid models — run the cheap deterministic checks on every push and the expensive suite on a schedule or on demand (chapter 23's budget thinking). And gates are only as good as their eval set: chapter 21's Klarna lesson was precisely that a green demo-grade suite proves little — keep feeding failures back into the set.

Where AI specifically gets this wrong

  • CI that tests code but not the model. Generated workflows lint and unit-test, then deploy a model nothing evaluated. The eval step is the one that matters; ask for it explicitly.
  • Gates on the aggregate only. v6 ships. Slices in the gate, always.
  • Thresholds nobody owns. A floor of 0.80 someone typed in 2025 and forgot is chapter 30's stale-rule problem — date-stamp gate thresholds and revisit them like ratchet rules.