Meta's LayerSkip eliminates the draft model, cutting LLM memory use by up to 50%

Speculative decoding has been the go-to trick for speeding up large language models, but it always came with an annoying catch: you need a second, smaller draft model sitting in memory. Meta’s LayerSkip team just changed that. In a new paper and with ready-to-use Hugging Face checkpoints, they demonstrate how a single model can draft and verify its own tokens by reusing its early layers. The memory savings are immediate and substantial—for a Llama2 70B model, self-speculative decoding slashes memory consumption by roughly half compared to a traditional two-model setup.

Here’s how it works. The model generates candidate tokens using only its first few layers (the paper uses an early exit at layer 4), then the full stack of deeper layers verifies and corrects the sequence in parallel. This isn’t something that just works out of the box. The researchers baked in a special training recipe involving early exit loss and progressively increasing layer dropout, which forces those shallow layers to produce logits that are actually useful. The result is a collection of continually pretrained Llama2, Llama3, and Code Llama models—all available under the LayerSkip collection on Hugging Face.

The benchmarks tell a nuanced story. For models up to 13B parameters running on a single 80GB A100, self-speculative decoding consistently outruns the old two-model approach. But the 70B model, distributed across 8 GPUs, sees only limited speedups. The authors haven’t fully explained why, though communication overhead between nodes is a likely suspect when you’re sharding a model that large. Still, the raw memory efficiency is compelling. You can now get speculative-level throughput on smaller GPUs that simply couldn’t accommodate a separate draft model before.

For developers, the barrier to entry is almost nonexistent. If you’re already using the Transformers library, you add one parameter—assistant_early_exit—to your generate() call. Just don’t expect it to work on any random decoder model. The intermediate layer logits need to be trained for unembedding through the LM head, which is precisely what the LayerSkip recipe enables. This isn’t a universal accelerator, but for anyone deploying open-weight Llama models in production, it’s a genuinely practical shift that reduces the hardware you need to rent.