Tencent open-sources AngelSpec to fix why one-size-fits-all draft models fail

Speculative decoding has a dirty secret: a single drafter model, no matter how well-tuned on benchmarks, breaks down under real-world traffic. The mix of chatty, high-entropy conversations and rigid, low-entropy code completions pulls the accuracy-speed tradeoff in opposite directions. Tencent’s newly open-sourced AngelSpec framework admits that tension outright and ships two specialized drafters instead of one compromise, integrating them into a PyTorch-native training pipeline.

The framework pairs an autoregressive multi-token prediction (MTP) model for open-ended dialogue with the block-parallel DFlash family, now upgraded to DFly, for code and math. The team found that a standard MTP drafter, trained with a single prediction layer, never learned to handle the recurrent self-unrolling used at inference. Errors compounded so severely that third-position acceptance cratered. Their fix applies a multi-depth training scheme where the block unrolls itself during training, feeding its own top prediction into the next step while the teacher target advances one position. Crucially, the target model’s backbone and LM head stay frozen and detached—improving the drafter’s proposals without polluting the distribution verification must preserve.

The numbers tell a sharp story. The new MTP regimen lifts mean acceptance from 52.8% to 66.4% at temperature zero, with the biggest gains concentrated exactly where the old method fell apart. On GSM8K, the third draft position’s acceptance rate rockets from 0.290 to 0.706. On HumanEval, it jumps from 0.387 to 0.757. The first position barely budges, proving the gains target the real failure mode.

On the block-diffusion side, DFly introduces two structural repairs. Instead of a one-size-fits-all context feature concatenated from target hidden states, it learns per-layer fusion weights as a residual refinement, giving each draft layer its own lens on the target’s hierarchy. A tiny sequential autoregressive head is then bolted on after the parallel backbone, converting position-wise guesses into prefix-conditioned distributions. The heavy lifting stays parallel; only a lightweight head runs left-to-right. AngelSpec doesn’t try to be clever for cleverness’s sake. It identifies exactly where existing drafters bleed acceptance rate and patches the holes with surgical, trainable fixes.