DeepSeek R1's 20,000-token responses are breaking GPUs and rewriting the rules of evaluation

The open-source race to replicate DeepSeek R1 just hit its first major wall: the model’s runaway verbosity. The Open-R1 project, barely a week old, has successfully matched DeepSeek’s reported MATH-500 benchmark scores, a crucial first step that proves the reproduction is on solid ground. But the real shock came when engineers looked at the raw output. The average response clocks in at a staggering 6,000 tokens, with some completions blowing past 20,000 tokens. To put that in perspective, the average web page is about 500 words, and a token is slightly less than a word—meaning a single R1 response can easily span 10 pages or more.

This isn’t just a quirky observation. It fundamentally reshapes the engineering challenge ahead. Training via Grouped Relative Policy Optimization (GRPO) requires holding those long completions in GPU memory to store activations and gradients for the optimization step. The team at Hugging Face, which integrated GRPO into the latest TRL release, is now profiling memory usage to prevent the training pipeline from choking. The inference side is equally brutal. Running R1 on two 8xH100 nodes led to a GPU KV cache bottleneck so severe that only 8 parallel requests were possible before preemptions kicked in. The fix required quadrupling the hardware to 32 GPUs and scrapping batched inference altogether in favor of a streaming approach, which stabilized GPU utilization dramatically.

For the broader community, the verbosity points to a deeper tension in reasoning models. The very “thinking” that makes R1’s outputs so powerful—those long chains of internal deliberation—is also what makes it a logistical nightmare to work with. It’s a classic example of a capability improvement that comes with a hidden infrastructure tax. The Open-R1 team’s move to create a public leaderboard tracking reproduction efforts is a smart one; it turns what could be a proprietary black box into a transparent, communal slog. You can already see the synthetic data generation efforts scaling up to meet the demand, as the plan is to distill these reasoning traces into smaller, more efficient models.

What’s genuinely interesting here is how the open-source community is effectively building the cookbook that DeepSeek didn’t publish. The original R1 paper showed that fine-tuning smaller models on R1’s reasoning traces yields similar performance gains, but it left out the gritty details of how to actually generate that dataset at scale. The Open-R1 team’s tinkering with vLLM, their switch from batched to streaming inference, and their public documentation of memory constraints are filling a critical gap. This replication effort isn’t just about matching a benchmark number anymore—it’s about producing the engineering manual for the next generation of reasoning models.