Falcon 3's 10B model beats everything under 13B parameters — here's how

The Technology Innovation Institute just dropped the Falcon 3 family, and the headliner is a 10-billion-parameter model that punches well above its weight. It’s a clean sweep across five base models, from a tiny 1B variant to a pure state-space Mamba architecture. But the real story isn’t just the model sizes — it’s the training tricks that got them there.

The 10B behemoth didn’t start from scratch. The team trained a 7-billion-parameter model on a staggering 14 trillion tokens using 1,024 H100 GPUs, then depth up-scaled it by duplicating layers and continuing training on another 2 trillion high-quality tokens. The result is Falcon3-10B-Base, which posts a 59.7 on the BBH reasoning benchmark and an 83.0 on GSM8K math problems. For context, that’s state-of-the-art performance for anything under 13 billion parameters — a category that includes plenty of overhyped models that don’t deliver. The 7B base model, meanwhile, is neck-and-neck with Qwen2.5-7B, and the 3B variant built via knowledge distillation somehow outperforms Meta’s Llama-3.1-8B. That’s not just efficient; that’s embarrassing for much larger models.

What I find genuinely interesting here is the commitment to architecture diversity. While everyone else is chasing the Transformer dragon, Falcon3-Mamba-7B gets an additional 1.5 trillion tokens of training data and now leads as the top-performing state-space language model. It matches or beats standard Transformer LLMs at the 7B scale and supports a 32K context length, all without forcing users to change their existing integrations. Pure SSM models were supposed to be the quirky alternative; this one’s now a serious contender.

The practical upshot for developers is immediate compatibility. All the Transformer-based models use the Llama architecture, which means they slot right into existing tooling without the usual integration headache. You can grab them as Instruct variants, or in GGUF, GPTQ, and AWQ quantizations, which covers everything from cloud deployment to running on a laptop. The 10B model’s 86.3 score on BFCL hints at real-world tool-use capability, not just benchmark-chasing. It’s rare to see a research release this focused on making the models genuinely usable from day one.