Nvidia and Cerebras are pitching inference performance using a metric that flatters their hardware while telling customers almost nothing about real-world behavior. Batch size 1 token generation — the first token out of the model — is the number both companies lead with. It’s fast. It’s also, for most deployments, irrelevant.
Think of it like a carmaker bragging about zero-to-sixty times for a vehicle you’ll spend most of your life in traffic with. Batch 1 measures latency without contention, without queuing, without the memory pressure that shows up when thousands of requests hit a cluster simultaneously. Real inference workloads run at high batch sizes because that’s how you get throughput and keep GPUs or accelerators busy. Nobody runs a production chatbot at batch 1 unless they enjoy burning money.
Cerebras has leaned hard on this framing because its wafer-scale engine genuinely excels at single-stream latency. The chip is enormous, memory is close, and there’s no interconnect bottleneck when you’re only pushing one request through. Nvidia’s Blackwell numbers look impressive under the same conditions. But the moment you ask what happens at batch 32 or batch 128 — the numbers that actually determine cost per million tokens — the conversation gets quieter. That’s not an accident.
This is the same playbook hardware vendors have used since the early days of deep learning: pick the benchmark that favors your architecture, then hope buyers don’t ask too many follow-ups. The difference now is that inference economics are existential for AI companies. Every percentage point of utilization matters. Buyers evaluating these platforms should demand sustained throughput under realistic load, not a drag race with no traffic on the road.
