skip to content
promptdojo
_
[ save your spot ]
[ follow on x ]
Pagination, retries, and checkpoints — step 2 of 7
ch 37 · dataset formats, ingestion, and validation pipelines
2/7
promptdojo
_
›
phase 08 · ai/ml engineering buildout
›
ch 37 · dataset formats, ingestion, and validation pipelines
lesson 2 of 5 · pagination, retries, and checkpoints
step 2 / 7
When should a pipeline persist the cursor for a page?
1
Before fetching the page, so the run looks faster.
2
After the page rows are validated and written.
3
Only after the entire pipeline finishes.
4
Never; retries should always start from page one.
check
Pagination, retries, and checkpoints — step 2 of 7
ch 37 · dataset formats, ingestion, and validation pipelines
2/7
promptdojo
_
›
phase 08 · ai/ml engineering buildout
›
ch 37 · dataset formats, ingestion, and validation pipelines
lesson 2 of 5 · pagination, retries, and checkpoints
step 2 / 7
When should a pipeline persist the cursor for a page?
1
Before fetching the page, so the run looks faster.
2
After the page rows are validated and written.
3
Only after the entire pipeline finishes.
4
Never; retries should always start from page one.
check
park a thought