The open AI community has a new obsession, and it’s not another dense parameter arms race. Mixtral 8x7B has pushed Mixture of Experts architectures into the spotlight, and the appeal is straightforward: you get dramatically faster pretraining and faster inference without paying the full compute cost of a model that size.
The core idea isn’t new. MoEs trace back to a 1991 paper called Adaptive Mixture of Local Experts, which proposed a system where separate networks each specialize on different subsets of the training data. A gating network decides which expert handles which input. That same architecture now applies to transformers: replace every dense feed-forward network layer with a sparse MoE layer containing multiple experts, typically FFNs themselves, plus a router that directs tokens to the right expert.
Here’s where the numbers get interesting. Mixtral 8x7B has 47 billion total parameters, not 56 billion, because only the FFN layers are replicated as experts while the rest of the model is shared. Yet during inference, if only two experts are active per token, the computational cost resembles running a 12B model. That’s the sparsity advantage. But there’s a catch: all 47 billion parameters still need to fit in VRAM, so memory requirements don’t shrink the way FLOPs do.
Training MoEs has historically been tricky. They pretrain efficiently but often struggle with fine-tuning, falling into overfitting. Recent instruction-tuning work looks promising, though the field hasn’t fully solved that tension. The router itself becomes a central design problem, with questions around load balancing and the recently introduced router Z-loss to stabilize training. For anyone watching the efficiency curve of large models, MoEs represent one of the more credible paths to scaling without proportional compute increases.
