Mixture of Experts (MoE)

Mixture of Experts (MoE) is an architecture where a gating network routes each input to a subset of specialized “expert” sub-networks. This allows models to have a very large total parameter count while keeping computational cost per token manageable.

Benefits of MoE:

• Efficiency: Only a fraction of parameters are used for each input
• Scalability: Can scale to trillions of total parameters
• Specialization: Different experts can learn different types of knowledge
• Performance: Often outperforms dense models at equivalent compute budgets

Notable MoE models include Mixtral 8x7B (Mistral), GPT-4 (estimated), Gemini 1.5, and DeepSeek V2. MoE is becoming increasingly popular as a way to scale model capability without proportionally increasing inference cost.