Falcon Mamba: The First 7B Model to Ditch Attention Without Performance Loss

The team behind Falcon has released Falcon Mamba, a 7-billion-parameter model that stands as the first general-purpose, large-scale language model to completely abandon the transformer’s attention mechanism without taking a hit on benchmark performance. This isn’t a research toy. It’s a production-ready model released under the Apache 2.0 license, built on the Mamba state space architecture, and it’s competitive with the best transformers in its weight class.

Why does that matter? The attention mechanism that powers models like Llama 3 and GPT-4 is fundamentally broken for long sequences. Its memory and compute costs scale quadratically with context length. Falcon Mamba’s architecture sidesteps this entirely. During generation, it uses a constant amount of memory and takes a fixed amount of time to produce each new token, whether your prompt is a paragraph or a novel. On a single 24GB A10 GPU, the model can process prompts in float32 precision that would choke a comparable transformer.

The secret sauce isn’t raw novelty—it’s solid engineering. The team took the original Mamba architecture from the “Selective State Spaces” paper and added extra RMS normalization layers to keep a 7B-parameter model stable during a 5,500 GT training run. That data diet consisted mostly of their RefinedWeb corpus, spiked with high-quality technical and code data, followed by a final annealing phase with specially curated datasets to polish performance. The result is a model that excels on the new Hugging Face Open LLM Leaderboard, significantly outperforming Meta’s Llama 3 8B and nearly matching Gemma 7B.

There’s a catch with long sequences, and the team is upfront about it. To truly unlock infinite context, you have to feed the prompt in token by token—what they call “sequential prefill”—rather than in parallel like you would with a transformer. Parallel prefill is faster but stores hidden states for every token, reintroducing a memory penalty that scales with prompt length. For most real-world applications, it’s a manageable tradeoff. This release is a clear shot across the bow of transformer dominance, proving that the path to linear-complexity models isn’t just a theoretical exercise; it’s a viable, performant reality you can download and run right now.