promptdojo_

Batch vs realtime — the 50% discount for work nobody is waiting on — step 7 of 7

ch 23 · production tradeoffs7/7
Checkpoint

One last thing before we move on. Same surface as a write step — but the lesson doesn't complete until this passes.

Final drill: the lane decision as code. Build plan(job) that takes a job dict with calls_per_day, latency_budget_s, and repeat_rate, and returns a (lane, monthly_cost) tuple:

  • If latency_budget_s >= DAY_S (the batch API's 24-hour guarantee fits inside the budget), the lane is "batch": cost = calls_per_day × PER_CALL × DAYS × (1 - BATCH_DISCOUNT).
  • Otherwise the lane is "realtime" with a cache in front: paid calls = calls_per_day × (1 - repeat_rate), then cost = paid calls × PER_CALL × DAYS.

Five workloads run. Expected output:

weekly digest:         batch     $300/mo
support chat:          realtime  $288/mo
checkout fraud check:  realtime  $3,000/mo
embeddings backfill:   batch     $6,000/mo
faq answers:           realtime  $180/mo

Read the table your own function prints. The fraud check is 10× the price of the chat feature at 6× the volume — every transaction is unique, so the cache saves nothing and the lane can't change. FAQ answers run realtime too, but a 75% repeat rate makes them the cheapest line on the board. Same lane, opposite bills: the repeat rate did that.

full-screen editor opens — close anytime to keep reading.

open the editor to run it.