Intel’s Gaudi processors now support assisted decoding through Optimum Habana, the Hugging Face library that optimizes AI workflows for Gaudi hardware. The technique, also called speculative sampling, pairs a smaller draft model with a larger target model. The draft model proposes K tokens at a time, the target model evaluates them, and rejected tokens trigger a fallback to standard autoregressive generation. The result: roughly 2x faster text generation on large transformer models, with no loss in sampling quality.
That quality guarantee isn’t marketing fluff. The authors of the original speculative sampling paper proved mathematically that the target distribution is recovered exactly, meaning you get the same outputs as you would from running the full model alone. The catch is that speedups depend on two factors: how much smaller the draft model is relative to the target, and how often the target model accepts the draft’s proposals. If the acceptance rate drops too low, the overhead of running two models eats into the gains.
What makes this notable for the hardware landscape is the positioning. Intel has previously claimed Gaudi delivers performance comparable to Nvidia’s H100, while pricing it closer to the A100 80GB. If that holds up in production, assisted decoding makes the value proposition sharper—you’re getting H100-class inference speeds at A100-class prices, now with an additional 2x acceleration layer on top.
The integration into Hugging Face Transformers is straightforward: the .generate() method accepts an optional assistant_model parameter. That’s the same interface Joao Gante built when he independently developed assisted generation for Transformers in May 2023. For teams already using Optimum Habana, enabling this is a one-line change. For everyone else, it’s another signal that Intel is serious about closing the inference gap with Nvidia—not just on raw silicon, but on the software ecosystem that makes that silicon usable.
