promptdojo_

Pipeline orchestration shape without the theater — step 1 of 7

ch 37 · dataset formats, ingestion, and validation pipelines1/7

The shape of an orchestrated pipeline

A dataset pipeline is not one script; it's a chain of stages — ingest → clean → features → dataset — where each stage has explicit inputs, explicit outputs, and a record of what it did. Get that shape right in a 40-line script and it survives the upgrade to a real orchestrator later.

The four properties of a stage worth having

  1. Explicit inputs and outputs. A stage reads files/tables and writes files/tables. No hidden globals, no "it also fixes that other thing on the side" — the same purity instinct as chapter 07's no-mutating-helpers rule.
  2. A run log. Run the editor: every stage reports rows-in and rows-out. That two-number trail is the fastest possible answer to "where did my rows go?" — the pipeline version of chapter 20's trace.
  3. Idempotent re-runs. Re-running a stage overwrites its own outputs; it never doubles them (lesson 02's rule).
  4. Restartable from any stage. Because outputs persist between stages, a bug in features doesn't force a re-ingest of 9,000 API pages.

Decorators

Look at the editor's @stage("clean") — that's the decorator pattern earning its keep: cross-cutting logging wrapped around every stage without touching stage logic. Orchestration frameworks (Airflow, Prefect, Dagster) are, at heart, this decorator grown up: you declare stages and dependencies, they add scheduling, retries, backfills, and a UI. The concepts you just ran are the concepts their docs assume.

When a script is enough

A daily 5-minute pipeline feeding one model is fine as a cron-scheduled script with stages, logs, and checkpoints. Reach for an orchestrator when you have many pipelines, dependencies between them, or backfills over history — the coordination is what you're buying, not magic. (Same buy-vs-build reasoning as chapter 16's framework lesson.)

Where AI specifically gets this wrong

  • One 400-line function. No stages, no restart points, no row-count trail. Ask Cursor for stages with logged in/out counts explicitly — it does it well when told.
  • Hidden state between stages. A global DataFrame mutated by three functions is chapter 07's aliasing bug wearing a pipeline costume.
  • Orchestrator-first. Standing up Airflow for one daily script is buying a train to cross the street.