Nvidia's 3,400 token/sec Groq 3 LPX record needs 64 chips to beat Cerebras's two

Nvidia has moved its Groq 3 LPX inference accelerator into full production, and the headline number is undeniably impressive: 3,400 tokens per second on Gemma 4 31B with a 100,000-token context window. That’s the fastest figure ever recorded for that model, according to Artificial Analysis, and Nvidia says it’s four times quicker than Cerebras’s 882 tokens per second. The chip, announced at Hot Chips 2026, extends the Vera Rubin platform and targets agentic AI workloads where token throughput directly determines how many reasoning steps an agent can complete before a user gets impatient.

The catch, as The Register was quick to point out, is what’s not in the comparison. Each Groq LPU carries just 500 MB of SRAM — 576 times less than a Rubin GPU’s 288 GB — so models have to be split across dozens of accelerators linked over Ethernet. The Gemma 4 31B benchmark is essentially a best-case scenario: a dense model that fits neatly in a single 256-LPU rack. Nvidia’s 3,400 tokens per second required at least 64 chips. Cerebras hit 882 with one or two. And the comparison ignores Cerebras’s newer CS-4 generation entirely.

Jonathan Ross, who founded Groq and joined Nvidia along with president Sunny Madra after the roughly $20 billion license deal in December, has long argued that SRAM-heavy dataflow architectures are the right tradeoff for inference. The LPX setup leans on GPUs for the compute-heavy prefill phase while LPUs handle the bandwidth-bound decode phase. That division of labor makes sense for sustained, repeated token generation — exactly what agents do when they’re checking files, writing code, and verifying results across hundreds of inference steps.

Where the architecture gets shaky is mixture-of-experts models. DeepSeek V3, for example, would require 1,342 LPUs — more than five full racks — to run. That’s a lot of hardware for a model that a handful of conventional GPUs can handle today. Nebius plans to be the first cloud provider offering the chip through its Token Factory, and Groq itself is among the early users. The real test won’t be a benchmark on a friendly model; it’ll be whether enterprises see enough agent-speed benefit to justify racks of specialized silicon that only shine under specific conditions. Nvidia says the chip goes live later this year. The market gets to decide if the math works.