Most teams pick batch or streaming for AI workloads by habit, inheriting whatever they built first and applying it everywhere. That is how a nightly reporting pipeline ends up feeding a live chat feature, and how a real-time stack re-embeds a ten-million-row corpus one document at a time. The choice is really an economics decision: it sets your latency budget, your cost per request, and how gracefully you fail under load. Get it wrong and you either burn tokens on idle GPUs or overpay for real-time you never needed. This piece maps the decision to concrete signals.
The choice is an economics decision, not a preference
Batch and streaming are usually framed as an infrastructure style, so teams inherit one from whatever they built first and apply it everywhere. That is how a nightly reporting pipeline ends up feeding a chat feature, and how a real-time inference stack ends up re-embedding a ten million row corpus one document at a time. The right frame is narrower: for a given AI workload, what is the tightest latency the user actually perceives, and what does each request cost once you divide the bill by throughput.
Those two numbers move in opposite directions. Batch amortizes fixed costs across a large window, so it drives the lowest cost per unit of work, but it defers results by minutes to hours. Streaming pays for readiness, so it delivers a token or a result in under a second, but you carry idle capacity and pay a premium per request. The workloads that hurt are the ones sitting in the middle, where a team pays streaming prices for work no human is waiting on.
The trap is that both modes look fine on a dashboard. A streaming pipeline running a batch-shaped job still returns correct answers; it just does so at three to eight times the unit cost, and the overspend hides inside a per-token line item nobody reconciles against throughput. The decision deserves a deliberate pass, workload by workload, rather than a default inherited from the first thing you shipped.
Match the mode to the latency budget and the batch size
Start from the latency budget, defined as the longest delay the consumer of the output tolerates before the work loses value. A user watching a cursor blink has a budget under two seconds. A daily digest has a budget of hours. Everything else falls between. Once the budget is fixed, batch size and cost fall out of it. The table below shows how five common AI workloads land when you reason from the budget rather than from the stack you happen to run.
| Workload | Latency budget | Better mode | Why it wins |
|---|---|---|---|
| Interactive chat / copilot | Under 1s to first token | Streaming | Perceived latency is the product; idle GPU cost is worth it |
| Corpus embedding / re-index | Hours | Batch | Large windows cut cost per 1K embeddings by 4-8x |
| Nightly summarization / reports | Overnight | Batch | No user waiting; schedule against off-peak capacity |
| Fraud / moderation scoring | 50-300ms | Streaming | Decision blocks a transaction in the request path |
| Bulk classification backfill | Minutes to hours | Micro-batch | Groups of 100-500 balance throughput and freshness |
The middle rows are where most savings hide. Micro-batching, grouping requests into windows of 100 to 500 items flushed every few hundred milliseconds, captures most of the cost advantage of batch while keeping freshness inside a minute. It is the default that too few teams reach for.
Work a concrete case. Say you need to embed a 5,000,000 document corpus and keep it fresh as roughly 40,000 documents change per day. The naive design calls the embedding endpoint once per document on the real-time path: at about $2.10 per 1,000 calls that is $10,500 for the initial load, and the daily delta costs $84 a day, all while a synchronous service sits waiting on each round trip. Reframe it by latency budget. The initial load has an hours-long budget, so it belongs on the provider batch tier at roughly $0.35 per 1,000, dropping the load to about $1,750, a 6x cut. The daily delta has a minutes-long budget, so it belongs in micro-batches of 256, which lands near $0.60 per 1,000 with retries and brings the daily cost to about $24. Same vectors, same model, same freshness the product actually needs, at roughly one sixth of the bill, purely because each slice was matched to its budget rather than run real-time out of habit.
Size the pipeline from the numbers you can measure
- Write down the latency budget in milliseconds for each workload before choosing a mode. If you cannot name the number, you are not ready to pick.
- Measure cost per 1,000 requests in both modes on your real data, not a benchmark. A pipeline moving from single-call to batches of 256 commonly drops from roughly $2.10 to $0.35 per 1K embeddings.
- Set a maximum in-flight concurrency and a queue-depth alarm. Streaming without a backpressure ceiling turns a traffic spike into a token-cost spike.
- Use provider batch endpoints for any job with an hours-long budget. Discounts of 40-50 percent on the same model are common for asynchronous batch tiers.
- Instrument tail latency (p95 and p99), not the average. A 300ms average with a 4s p99 will define how the workload actually feels.
Where these pipelines quietly bleed money
- Paying streaming rates for work no user awaits. Fix: audit every real-time path for whether a human is blocked on the result, and move the rest to micro-batch.
- Batch windows so large they miss freshness targets. Fix: cap the window by time (for example flush every 60s) as well as by count, so a slow trickle still ships.
- No backpressure on the streaming path. Fix: add a bounded queue and shed or defer load past a concurrency ceiling rather than letting cost scale linearly with the spike.
- Retries that silently duplicate token spend. Fix: make batch jobs idempotent with a request key so a re-run reprocesses only what failed, not the whole window.
- Idle GPU reservations held around the clock for a bursty streaming workload. Fix: autoscale to zero off-peak, or route low-priority traffic to a shared batch tier.
Verify the mode fits before you scale it
- Every AI workload has a written latency budget in milliseconds.
- Cost per 1K requests is measured in both modes on production-shaped data.
- Any job with an hours-long budget runs on a batch or async tier, not real-time.
- Streaming paths have a concurrency ceiling and a queue-depth alarm.
- Batch jobs are idempotent and re-run only the failed slice.