Hugging Face squeezes 52% faster LLM inference by guessing when a draft model is wrong

The whole premise of speculative decoding is clever but wasteful. You let a tiny, fast draft model spit out a bunch of tokens, then a big, accurate model checks them all in parallel. The problem? Figuring out how many tokens the small model should guess before the big one steps in. Most implementations just pick a fixed number, like five, or use a simple heuristic—if the last batch was all correct, guess more next time. Both are leaving performance on the table.

Researchers at Hugging Face basically asked: what if we could see the future? They built an “oracle” that perfectly knows when the draft and target models will disagree, using a rejection sampling algorithm at zero temperature. When they ran it on a coding example from the MBPP dataset, the results were stark. A static lookahead of five tokens required 38 forward passes from the target model and 192 from the draft model. The oracle needed just 27 target passes and 129 draft passes. The oracle lookahead values bounced around wildly, never settling into a predictable pattern—proof that any fixed schedule is suboptimal.

So they built a simpler, practical version of that oracle. The key insight is brutally pragmatic: stop the draft model the moment it gets nervous. After generating each token, the system checks the assistant model’s softmax confidence. If it dips below a threshold they call assistant_confidence_threshold, generation halts immediately, even if the draft model hasn’t hit its maximum token limit. The half-finished sequence gets sent to the target model for verification right then and there. No waiting for a full block of guesses that are increasingly likely to be wrong.

When benchmarked against the existing heuristic approach, the dynamic method won across every task and model pairing tested. The most dramatic result came from pairing Llama 3.2 1B as the draft model with Llama 3.1 8B as the target. The dynamic approach delivered a 1.52x speedup, while the heuristic method showed no significant improvement at all. Another test with codegen-6B-mono actually ran slower using the heuristic, but saw a speedup with the dynamic method. That kind of inversion tells you the heuristic isn’t just suboptimal—it can be actively harmful for certain model combinations. This confidence-based approach is now available directly in Hugging Face’s Transformers library, making it a drop-in improvement for anyone already using speculative decoding.