TNG runs 5,000+ LLM inferences per hour on 24 H100s—here's the prefill/decode trade-off

Most teams treat LLM inference as a black box. TNG doesn’t have that luxury. They’re running over 5,000 inferences per hour across 50 applications on a cluster of 24 H100 GPUs, generating north of ten million tokens daily. When you’re operating at that scale, the distinction between prefill and decode isn’t academic—it’s the difference between users sticking around or bouncing.

The prefill phase, which computes that first output token by processing every input token in parallel, is a compute-hungry beast. A single request with a long prompt can max out GPU utilization on its own. Decode, by contrast, is a memory-bandwidth game: each subsequent token taps the KV cache, adding one new set of key-value vectors at a time. It’s why commercial APIs charge you far less for input tokens than output tokens. The economics reflect the physics.

Latency targets are brutally tight for interactive apps. TNG flags 100-300 milliseconds per output token as the sweet spot—fast enough that users can skim text as it streams. Time to first token needs to land under three seconds, or people assume the thing is broken and leave. Hit five seconds of silence and you’ve lost them. Non-interactive workloads like code summarization or book translation don’t care about those numbers; they only care about total token throughput.

Here’s the rub: you can’t have it all. Batching requests during decode pushes throughput up—the memory-bound regime loves larger batch sizes—until GPU compute saturates and you hit a ceiling. But bigger batches mean any individual request waits longer. TNG’s setup makes that trade-off concrete. 24 H100s sounds like a lot of horsepower, and it is, but when you’re serving dozens of apps simultaneously, scheduling prefill and decode steps without starving one or the other becomes the real engineering challenge. The piece doesn’t go deep on their scheduling strategy, which is where I’d want to dig next. Continuous batching, prefill-decode disaggregation—those are the techniques that determine whether those latency targets survive real-world load.