Hugging Face's SmolLM hits above its weight with 1.7B params

Hugging Face just dropped SmolLM, a family of three small language models ranging from 135M to 1.7B parameters, and the early benchmark numbers suggest they punch well above their weight class. The models—trained from scratch rather than distilled from larger systems—beat competitors in the same size category on reasoning and world knowledge tests. That’s notable because the sub-2B model space has gotten crowded lately, with Microsoft’s Phi series, Alibaba’s Qwen2, and Meta’s MobileLLM all jockeying for position. Most of those teams keep their data recipes under wraps. Hugging Face is doing the opposite, releasing the full training corpus as SmolLM-Corpus.

The dataset itself is a three-part blend: Cosmopedia v2, a synthetic collection of 30 million textbooks and stories generated by Mixtral-8x7B-Instruct (28 billion tokens); Python-Edu, educational code samples from The Stack (4 billion tokens); and FineWeb-Edu, deduplicated educational web content (220 billion tokens). The Cosmopedia overhaul is where things get interesting. Version 1 relied on unsupervised clustering of FineWeb samples to determine topics—a method that produced a messy, uncontrollable topic list. Version 2 switched to the BISAC book classification system, a standard used by libraries and bookstores, giving the team a predefined list of 34,000 educationally focused topics.

Instead of clustering, they built a search tool that retrieves relevant web pages for each topic from 520 million FineWeb samples. The team experimented with swapping in more capable generators—llama3-70B-Instruct, Mixtral-8x22B-Instruct, Qwen1.5-72B-Chat—but found no meaningful improvement over the original Mixtral model. That’s a useful data point: better generation models don’t automatically produce better training data. Prompt quality matters more than model capability.

The implications go beyond this release. Small models that run locally solve real problems—lower inference costs, better privacy, offline functionality. But the field has been hampered by secrecy around data curation. Hugging Face’s decision to open-source both the models and the corpus means researchers can finally study what actually works at this scale without reverse-engineering proprietary pipelines. Whether SmolLM’s performance holds up outside benchmark tests remains an open question, but the transparency alone shifts the conversation.