Meta drops Llama 4: 10M context window and a bold bet on NoPE layers

Forget the slow drip of model releases. Meta just dumped two Llama 4 variants on Hugging Face—Maverick and Scout—and they’re making some architectural bets that deserve a close look. Both are natively multimodal, early-fusion models that process text and images, and both use a Mixture-of-Experts (MoE) setup with 17 billion active parameters. The difference is scale: Maverick is a beast with ~400 billion total parameters across 128 experts, while Scout keeps things relatively lean at ~109 billion total parameters and just 16 experts. Hugging Face has baked in support from day one, with both models available under the custom Llama 4 Community License.

What jumps out immediately is the context length. The instruct-tuned version of Scout stretches to 10 million tokens—a number that makes Google’s Gemini feel a bit cramped. Maverick isn’t far behind at 1 million tokens. This is achieved through an architecture Meta is calling iRoPE, which interleaves standard RoPE layers with so-called NoPE layers that forgo positional encodings entirely and use a full causal mask. It’s a clever throwback to a 2022 concept, now apparently critical for extreme long-context work. To manage memory, the RoPE layers use chunked attention, a sparse approach that restricts how far back each token looks in its chunk.

On the deployment side, the practical considerations are clear. Scout is the workhorse for smaller shops—it can run on a single server-grade GPU with on-the-fly 4-bit or 8-bit quantization. Maverick ships with FP8 weights for those with more serious hardware. Hugging Face is also rolling these out with Xet storage, which reportedly achieves 25% deduplication on the base models and up to 40% on community fine-tunes, which translates to faster downloads and less wasted bandwidth for everyone iterating on top of these models.

The real test, as always, won’t be the spec sheet. It’s how well that 10M context holds up under actual retrieval pressure, and whether the NoPE layers introduce subtle degradation that benchmarks miss. The community will poke at those edges quickly now that the weights are live. If the long-context performance is solid, Scout in particular could become the default base for a wave of document-native AI applications.