Stop benchmarking LLMs wrong: why time.perf_counter() alone is a trap

Measuring the performance of a large language model service isn’t a single-number game. Anyone who’s spent time optimizing inference knows the sinking feeling of shipping a change that looks great on a throughput chart but makes the chat feel sluggish. The guide breaks this down with the precision of an engineer who’s been burned before. It pushes past vague latency talk and into the metrics that actually matter: time-to-first-token (TTFT) and time-per-output-token (TPOT).

A 2,000-token prompt with a 20-token answer is a completely different beast from a 20-token prompt generating a 2,000-token essay. Pretending they have the same performance profile because the total token count is similar is a recipe for disaster. The prefill phase in the first request will hammer compute, while the second is all about decode speed. This is why you must track prompt and output tokens independently. Tail latency is another trap. If your average looks great but p99 is ten seconds, you’ve built a service that’s fast for most but broken for some. The guide recommends using NumPy to pull p90, p95, and p99 numbers—a habit that separates a serious benchmark from a toy one.

The technical deep dive on measuring a single request is refreshingly honest. Instead of relying on the model’s black-box generate() method, it advocates manually stepping through prefill and decode. Using time.perf_counter() gives you high-resolution wall-clock timing, and the snippet explicitly uses use_cache=True to return the KV cache, feeding only the next token during decode. This isn’t just code; it’s a blueprint for exposing where the time actually goes. The real devil is in GPU synchronization. Launching a CUDA kernel is asynchronous, so a naive timer will stop long before the GPU finishes. Slapping torch.cuda.synchronize() in your timing blocks isn’t a nice-to-have—it’s the difference between measuring reality and measuring the speed of your Python loop. Without it, you’re just lying to yourself about how fast your model really is.