The Hugging Face Hub hosts roughly 50,000 public datasets, but here’s a number that should make anyone working in multilingual AI pause: only about 13% of them actually specify what language they contain. That means around 43,500 datasets are effectively invisible to anyone filtering by language — a massive blind spot for researchers building models for anything other than English.
A new project called Huggy Lingo is attacking that gap directly. The approach is twofold: first, use machine learning to detect the language of datasets that lack metadata, and second, deploy librarian-bots to submit pull requests adding those language tags to the dataset cards. The team avoids downloading entire datasets where possible by leaning on the dataset viewer API, which lets them sample text content without pulling gigabytes of data locally.
English dominance on the Hub is stark. Around 19% of datasets list English as their language, and that doesn’t even account for variations like ‘eng’ or ‘English’ — the actual figure is likely much higher. Excluding English, the language distribution shows a handful of dominant languages followed by a steep drop-off, which mirrors what we’ve seen in model training data biases for years. The 1,716 unique language tags currently in use also include duplicates caused by inconsistent formatting, so clean-up is as much a normalization problem as a detection one.
For anyone building open-source LLMs in Dutch, Swahili, or any language outside the top tier, this matters enormously. The Hub’s Dutch filter currently returns just 184 datasets, but the team’s own analysis suggests plenty of Dutch-containing datasets exist without proper tags. Better metadata means better discoverability, which means less duplicated effort and more realistic assessments of what training data actually exists for a given language. The bots are still an early experiment, but the underlying problem they’re solving is real, and it compounds as the Hub grows.
