Moonshot AI drops MoonEP, claims 2.5x scaling boost for trillion-parameter models

Moonshot AI just open-sourced MoonEP, a communication library that tackles the messiest problem in distributed training: routing imbalance in mixture-of-experts (MoE) models. It’s not a minor tool. The team claims it’s a key piece behind a 2.5× improvement in scaling efficiency for Kimi K3, their new 2.8-trillion-parameter beast with native vision support. The MIT-licensed code arrived as part of a broader infrastructure dump alongside the K3 model weights, which also included the new AgentEnv toolkit.

The core headache MoonEP addresses is deceptively simple. When a router sends tokens to different experts across GPUs, some experts get slammed with far more work than others. Moonshot quantifies this skew with a metric they call ‘maxvio.’ A perfectly balanced system scores zero. Real-world routing is never zero, and that imbalance dictates the entire system’s speed because a collective operation can only move as fast as its most overloaded participant. The standard fixes involve painful per-layer host synchronization and fragmented GPU memory, a structural tax on every training step.

MoonEP’s solution is a hard invariant: every GPU rank receives exactly the same number of tokens, no matter how skewed the routing decisions are. It pulls this off by dynamically planning a few redundant expert copies on-the-fly, directly from the router’s output. These duplicated experts are prefetched before computation, and their gradients get reduced back home during the backward pass. The result is a system with zero-copy token placement, static buffer shapes that eliminate sync points, and a contract with the framework that demands a single contiguous weight tensor per projection. A neat trick is how they handle the extra memory for these prefetched experts—it’s drawn from a single process-global pool, not duplicated per layer.

How does it stack up against the incumbent? Moonshot ran head-to-head benchmarks against DeepEP v2 on H20 GPUs, sweeping from nearly balanced routing to extreme skew. The numbers on their GitHub page are stark. At a maxvio of 0.2—a best-case scenario—MoonEP is already 28% faster. At a maxvio of 1, the gap widens to 58%. And when the imbalance hits a catastrophic maxvio of 20, MoonEP reportedly delivers a 3.9× speedup. The library defaults to a specific config for training, requiring the number of prefetch slots to match the number of experts per rank, which guarantees every expert the GEMM kernel touches is local. For inference, you can get away with just 3 or 4 slots, making the memory overhead almost negligible. It’s a genuinely clever piece of engineering that treats imbalance not as a bug to fix, but as a physical constraint to route around.