The cheapest inference infrastructure ever built is a cron job
Before you size an endpoint, answer one question: who is waiting for this result? Not "how fast could it be" — who, specifically, is staring at a spinner. If the answer is "nobody until tomorrow morning," you've just found a 50% discount.
Every major provider ships a batch API with the same deal: submit a file of requests, get results back within 24 hours, at half the price of the same calls made one at a time. There's no cleverness to unlock. The provider fills idle GPU capacity overnight and passes the savings to anyone whose latency budget is measured in hours instead of milliseconds.
Run the editor. Same model, same tokens, same predictions — $1.00 per thousand realtime, $0.50 per thousand batched. At 3 million predictions a month that's $3,000 versus $1,500, forever, for changing when the work runs rather than what it does.
Latency budget is a property of the workload, not the tech
Put real numbers on "who is waiting":
- Checkout fraud check — the buyer is mid-purchase. Budget: well under a second. Realtime, no debate.
- Support chat reply — a human is watching the screen. Budget: a few seconds. Realtime.
- Nightly ticket summaries for a 9 a.m. dashboard — nobody reads it before coffee. Budget: hours. Batch.
- Embeddings backfill over the archive — nobody is waiting at all. Budget: whenever. Batch, obviously, and yet teams run these through their chat endpoint constantly.
The trap is demo inertia: the prototype was a chat box, so the endpoint became the hammer, so every workload became a nail — and half of those workloads never had a user waiting on them. You pay the realtime premium to remove waiting. Paying it where nobody waits is a donation.
The third lever: most questions have been asked before
Realtime traffic has its own discount hiding in it. If 62% of today's queries are repeats of queries you already answered, a cache in front of the endpoint means you pay for 38% of the calls and serve the rest for free — at zero added latency. (This is the response-level sibling of the prompt caching you met earlier in this chapter, which discounts the prefix of calls you still have to make.) Batch discounts the calls nobody's waiting for; caching deletes the calls nobody needed to repeat. The rest of this lesson makes all three numbers — lane, latency, hit rate — do the arguing.