The counterintuitive data trick that makes Mamba hybrids actually reason

The team behind Apriel-H1 discovered something that should make AI engineers reconsider their default approach to distillation. When they tried retrofitting a 15B reasoning model with Mamba layers for efficiency, their intuition told them to use broad pretraining data — after all, these new linear SSM layers had never seen data and needed to learn general-purpose token mixing from scratch. That approach failed. Reasoning quality cratered.

What actually worked was almost the opposite: training exclusively on the teacher model’s high-quality reasoning traces. Math proofs, coding tasks with clear logical dependencies, scientific analysis with step-by-step chains. Not pretraining data. Not a mix. Just concentrated examples of coherent multi-step reasoning.

The logic is specific to what’s being preserved. The base model already knows how to predict tokens. What’s fragile — and what attention mechanisms enable — is the chain of retrieval heads, induction heads, and long-range dependencies that connect premises to conclusions. Mamba’s linear recurrence disrupts those mechanisms, and the student needs explicit, visible reasoning structure to discover new computational paths to the same outcomes.

They also found reverse KL divergence consistently beat forward KL. When your teacher is confident and correct on these structured problems, mode-seeking behavior is exactly what you want — commit to those high-confidence predictions rather than hedging across a broader distribution. The result: Apriel-H1-15b-Thinker-SFT delivers 2.1x throughput while MATH500 actually improves slightly and GSM8k, GPQA, and AIME24 see only modest regression. The paper also details a staged conversion process using a dynamic heuristic called MIL-Mamba-Replacement that captures training dynamics rather than static layer importance, since layers that look unimportant in isolation become critical when multiple are replaced together.