Intel Prunes Qwen3 Draft Model to Hit 1.4x Speedup on Local AI Agents

Intel’s latest experiments are putting the squeeze on draft models to make local AI agents feel less like a waiting game. Their engineers took Qwen3-8B, a model with native tool-calling chops that’s a natural fit for AI PCs, and set out to make it run faster on a Lunar Lake integrated GPU without gutting its brain.

The baseline was a 4-bit quantized version running through OpenVINO.GenAI. They then hooked it up with speculative decoding, using the smaller Qwen3-0.6B as a speedy draft model to propose tokens the larger model could verify. That netted a solid 1.3x speedup. But the team got greedy, in a good way. They theorized that the speedup depends heavily on the draft model’s latency, so making the draft model even lighter would let the larger model sprint ahead.

Here’s where it gets interesting. Instead of just using a smaller model off the shelf, they took a scalpel to the 0.6B draft, using a layer-pruning method that identifies and removes blocks of layers that aren’t pulling their weight, measured by angular distance. They cut 6 of the 28 layers. To patch the damage, they fine-tuned the pruned model on synthetic data generated by the Qwen3-8B model itself from a half-million prompts. The result? The speedup ticked up to roughly 1.4x over the baseline.

To prove it’s not just a benchmark stunt, they wrapped the accelerated model into a real agent with Hugging Face’s smolagents library. The demo task—summarizing Qwen3 features into a slide deck—showed the agent handling tool calls and multi-step reasoning with noticeably better responsiveness. It’s a practical signal that for developers building local agents on Intel hardware, attacking the draft model’s structure, not just its size, is a viable path to shaving off precious seconds.