Synchronous batching wastes 24% of GPU time: Here's how async cuts it to zero

If you’re running inference on an H200 at $5 an hour, every second the GPU sits idle is money you’re lighting on fire. The team behind Hugging Face’s transformers library just quantified exactly how much: nearly a quarter of total generation time. Their profiling of an 8B model generating 8K tokens with a batch size of 32 revealed a brutal 24% overhead. The GPU spent almost a minute doing nothing while waiting for the CPU to shuffle tokens and rebuild batches.

The culprit is the default synchronous loop. Continuous batching already solved the padding problem, packing requests tightly to avoid wasted compute. But it didn’t fix the seesaw. In a synchronous setup, the CPU prepares batch N, hands it off, and then twiddles its thumbs while the GPU crunches. Once the GPU finishes and samples new tokens, the CPU scrambles to update request states and schedule the next batch—while the GPU goes dark. They never overlap. Run that loop hundreds of times per second and those micro-gaps become a gaping hole in your throughput.

The fix is asynchronous batching, and the core insight is beautifully simple: prepare batch N+1 while batch N is still computing. You disentangle the two workloads using CUDA streams, which let you launch operations on separate queues that the GPU can execute concurrently. The CPU doesn’t have to wait for the GPU to finish before it starts its next bookkeeping cycle. This isn’t a kernel trick or a model architecture change. It’s just smarter orchestration of the hardware you already have. The projected result drops generation time from 300 seconds to 228—a free 24% speedup with no accuracy trade-off.

Of course, the devil’s in the details. You can’t prepare batch N+1 if you don’t know what tokens batch N will produce. The CPU needs those predictions to update the KV cache and decide which requests to evict or admit. Solving that requires careful data dependency management and a bit of speculation. But the payoff is real, and the transformers library now has a working implementation that proves it.