AMD ships a 16B-param MoE model trained entirely on its own GPUs, but the license is research-only

AMD just dropped a Mixture-of-Experts model designed to prove a point: you can train a competitive large language model from scratch entirely on non-Nvidia hardware. Instella-MoE-16B-A3B packs 16 billion total parameters but only activates 2.8 billion per token, a lean setup that keeps serving costs in check — one high-memory accelerator can handle the roughly 32 GB of BF16 weights. AMD is publishing the whole recipe, not just the final cake. Weights from every training stage, data mixtures, configs, and inference code are all public.

The real story here is the systems engineering. AMD built two custom components to squeeze more performance out of its Instinct MI300X and MI325X clusters. Gated Multi-head Latent Attention adds a lightweight learned gate to the attention mechanism, applied before the output projection. Then there’s FarSkip-Collective, a technique that overlaps expert-parallel communication with computation by passing outdated and partial activations. AMD claims a 12.7% pre-training speedup and up to a 39.2% reduction in time-to-first-token during expert-parallel serving. Those aren’t paper gains; they’re infrastructure gains that make training on AMD silicon viable.

On benchmarks, the base model averages 76.7, edging out Moonlight-16B-A3B at 76.2 and leaving OLMo-3-7B in the dust at 70.1. It even leads on WinoGrande with an 86.5. The post-trained “Think” variant climbs to 73.22, ahead of Gemma-4-E4B and Qwen3.5-4B. That’s genuinely competitive territory, especially for a model nobody was asking for from a company better known for silicon than software.

There is, however, a massive asterisk. The weights are locked under a ResearchRAIL license — academic and research use only. So forget dropping this into a commercial product. The training codebase is MIT-licensed, which is the more reusable asset. For university labs, semiconductor R&D teams, and anyone wanting to study end-to-end MoE training on ROCm, this is a goldmine. For a startup that just wants a cheap hosted endpoint, it’s a non-starter. AMD seems to be playing the long game: seed the research ecosystem, prove the hardware stack works, and hope the commercial models follow later.