How continuous batching squeezes every drop of throughput from your AI chatbot

If you’ve ever watched a chatbot like Qwen or Claude compose a response, you’ve seen the bottleneck: a long pause, then a steady trickle of words. That’s the reality of autoregressive generation, where models predict one token at a time, each step demanding a pass through billions of parameters. The core of that cost isn’t just the math—it’s the attention mechanism’s quadratic complexity, which forces the model to re-read the entire conversation history for every new token.

Engineers don’t just throw more GPUs at the problem. One of the most impactful optimizations is continuous batching, a scheduling strategy that maximizes throughput when serving many users at once. The insight is simple but powerful. Traditional batching forces all sequences in a batch to wait for the longest one to finish. Continuous batching ditches that rigidity. As soon as a sequence generates its end-of-sequence token, it’s swapped out, and a new request can immediately take its slot. The GPU never idles waiting for stragglers.

The magic is in how it handles the two distinct phases of inference. During prefill, the model ingests the entire user prompt in one forward pass, computing keys and values for every token and caching them. Decode is a different beast: it generates one token at a time, appending new key-value pairs to the cache. Continuous batching lets these phases mix within a single batch. While some sequences are chewing through a long prefill, others are already in the lighter decode phase, keeping the hardware fed. This co-mingling of sequence lengths is why the blog post starts by building up attention from scratch. You need to understand that query, key, and value tensors can have mismatched token counts to grasp how the attention mask elegantly controls who talks to whom.

The result is a system that feels more responsive because it is. Server operators see higher throughput and lower latency for everyone, not just the user who got in first.