Transformers Run Twice: Why Inference Is Nothing Like Training

If you’re using the same PyTorch code for transformer training and inference, you’re leaving massive performance on the table. The difference is stark. Training is a brutish computational sprint, dominated by huge matrix multiplications and a gradient-crunching backward pass over fixed-length sequences. Inference is an entirely different beast. It’s a memory-bound marathon of repeated forward passes where you generate text one token at a time, and the naive approach is disastrously wasteful.

That naive approach, called autoregressive generation, feeds the entire growing sequence back into the model for every new token. It’s elegantly simple to code in a few lines with a for-loop and an argmax, but the math tells a grim story. If you start with a 1,000-token prompt and generate 100 more, you’re recomputing attention scores for the prompt tokens a hundred times over. The computational complexity balloons to roughly O(P²G + PG² + G³), a clear recipe for making your users stare at a spinner.

The industry’s escape hatch is splitting the process into two distinct phases: prefill and decode. Prefill happens once. You blast the entire prompt through the model in a single forward pass, generating the first token’s logits and—crucially—saving the attention keys and values for every layer. This is a heavy O(N²) operation, but it directly controls your “time to first token” metric. The decode phase then takes over, processing only the single newest token. It uses a key-value (KV) cache to avoid recomputation, attending the new query over all saved history. This chops the per-token cost down from O(N²) to O(N), turning a quadratic nightmare into a linear one and determining how fast tokens stream out.

The memory implications of that KV cache are now the bottleneck. For a model with L layers, H attention heads, and a hidden dimension D, storing two tensors (key and value) of size D for every token in a sequence of length N requires 2 * L * N * D floating-point numbers. That number gets terrifyingly large with long conversations or high-batch serving. It’s why a short prompt with a long answer hammers your decode throughput, while a long document summary task lives and dies by prefill speed. Optimizing one phase often constrains the other, and the entire field of inference serving systems is essentially a high-stakes game of trading memory for speed.