Mission: API → dataset pipeline
Everything this chapter taught — pagination, checkpoints, contracts, stages — was one pipeline taken apart for study. Reassemble it: wire the smallest end-to-end version, crash it mid-run on purpose, then make the resume boring.
The job: assemble this chapter into one working pipeline — paginated ingest, checkpoint/resume, a schema contract with a quarantine sink, staged structure with a rows-in/rows-out log, and a final dataset you could hand to chapter 38.
The stages:
- Ingest — page until empty; checkpoint after each page.
- Contract — validate each record; quarantine rejects with the reason attached (never silently drop).
- Stage log — in/out counts per stage, printed at the end.
- Stress case — re-run from a mid-way checkpoint and prove the output has no duplicates (idempotency); then add a drifted record (new field, wrong type) and show it lands in quarantine, not the dataset.
- One improvement — a retry wrapper on
fetch, a run summary dict, or a persisted checkpoint file: pick one, note why.
Run the editor: the starter compresses stages 1–2 into fifteen lines. Your mission pulls them apart into named stages and makes the run report tell the whole story. What it proves is containment: a flaky, drifting API stays on one side, and what crosses over is a typed dataset, a quarantine, and a resumable run log.