For years, XLM-R has been the untouchable gold standard for multilingual language models. Every new contender benchmarked against it, and none could consistently pull ahead. That changes today. A new model called mmBERT, built on the speedy ModernBERT architecture, is the first to decisively outperform XLM-R, and it does so across a staggering 1,833 languages.
This isn’t just a marginal gain. The team at Answer.ai trained mmBERT on over 3 trillion tokens, but the real innovation is how they fed it the data. They pioneered a “progressive language inclusion strategy.” Think of it as a multilingual curriculum. The model starts with a heavy diet of high-resource languages like English and Russian, then gradually adds more languages in phases—from 60 to 110, and finally to over 1,800. At the same time, they slowly flatten the data distribution so the model isn’t just memorizing English Wikipedia. Lead researcher Sam L’Huillier explained the core tension: “The key challenge in multilingual training is finding the right balance between high-resource and low-resource languages. Too much focus on English and you lose low-resource performance; too uniform and you degrade high-resource quality.”
They also got clever with the masking rate. Instead of a fixed 15% like BERT, mmBERT uses an inverse schedule that drops from 30% down to 5% across training. Combine that with an “annealed” temperature that shifts sampling bias, and you have a recipe that lets the model build a strong linguistic foundation before tackling the long tail of languages without overfitting. The base model packs 110 million non-embedding parameters, but the tokenizer, borrowed from Gemma 2, balloons the total to 307 million to handle the vastly expanded vocabulary.
The result is a model that’s not just good for a checkbox diversity metric; it’s genuinely better. It pushes past XLM-R on core natural language understanding benchmarks while also being faster, thanks to its ModernBERT backbone. The final model is actually a TIES merge of three specialized variants trained during the final decay phase. It’s a sophisticated engineering feat that opens up state-of-the-art NLP to thousands of languages that the field has largely ignored, without sacrificing the performance the English-speaking world demands.
