Intel and Hugging Face crack 1.5x-2x faster LLM decoding across any model family

The dirty secret of assisted generation—that speed-boosting technique also called speculative decoding—has always been its pickiness. You needed a miniature version of your main model sharing the exact same tokenizer. For giants like Gemma-2-9b or Mixtral-8x22B, which lack a sufficiently tiny sibling, that was a dead end. Intel Labs and Hugging Face just blew that door open with Universal Assisted Generation (UAG).

The fix is a clever two-way tokenizer translation layer. When the tiny assistant model spits out tokens, UAG converts them back to plain text, then re-tokenizes that text using the target model’s own vocabulary for verification. After the target model gives its approval, the process reverses, translating the confirmed tokens back into the assistant’s format so it can pick up where it left off. It’s an iterative handshake that decouples the two models entirely. The team found that to get this re-encoding right, you need to feed the target’s tokenizer a small window of preceding context, not just the new tokens in isolation, to accurately align the different vocabularies.

What does this unlock in practice? The benchmarks are striking. Pairing the massive CodeLlama-13b with a completely unrelated, diminutive Vicuna-68m as its assistant delivered a 1.8x speedup. Even a Mixtral-8x22B-Instruct model, running on 4 A100 GPUs, saw its generation accelerate by 1.5x when paired with the same tiny Vicuna assistant. The team notes that the real gains kick in when the assistant is 50 to 100 times smaller than the target—a ratio that was simply impossible to achieve for many models until now. All of this comes with what the researchers call “almost zero overhead,” preserving the original model’s output accuracy.

The broader implication here is a real blow to hardware lock-in for fast inference. You’re no longer forced to use a specific model family just because you need a draft model that fits. Any of the 140,000 models on the Hugging Face Hub can now theoretically serve as the target, and you can grab the most efficient assistant you can find, regardless of its lineage. It turns the assisted generation problem from a model genealogy challenge into a simple performance optimization. The code is out there, and I suspect we’ll quickly see this folded into production serving stacks that have been stuck on slower single-model decoding paths.