Why your LLM's GPU is mostly idle—and how continuous batching fixes it

Most GPUs serving AI models are sitting idle. The fix isn’t a faster chip—it’s smarter scheduling. Batching groups requests together to share one expensive model-weight load, but the difference between a batching scheme built for uniform image-generation jobs and one built for the chaotic token counts of large language models is the difference between a cost center and a functional product.

Static batching is the oldest trick in the book: wait until you’ve got, say, eight requests, then run them all in one go. It’s efficient for offline, bulk inference, but it falls apart under live traffic. The first request waits for stragglers. Once the batch starts, five short prompts get held hostage by one that runs for a thousand tokens. There’s no latency bound, making it DOA for user-facing applications.

Dynamic batching patches the worst of this by adding a timeout window. Run the batch when it’s full, or when the clock runs out—whichever comes first. Triton benchmarks show this significantly improves throughput at the cost of a moderate tail-latency bump. But it still has a fatal flaw for LLMs: once a batch is running, every request is stuck until the longest one finishes. For an image generator with roughly fixed step counts, that’s fine. For a language model where completions can vary from 10 to 1,000 tokens, it’s a throughput killer.

That’s where continuous batching, also called in-flight batching, changes the game. It schedules at the token level, not the request level. The moment a short request finishes generating, it’s evicted from the batch and a new request from the queue can slide into its slot, immediately starting its own generation in the very next forward pass. No waiting for a full batch to finish. This keeps the GPU saturated with work even as requests of radically different lengths flow through the system, directly attacking the utilization problem that makes LLM inference so expensive at scale. It turns LLM serving from a batch-process problem into a real-time scheduling problem, and it’s quickly becoming table stakes for any production deployment.