The world’s smallest VLM is here at 256M parameters, beating 80B models from 17 months ago

Hugging Face just shrank its vision language models down to sizes that would have sounded like a joke a year and a half ago. The new SmolVLM-256M and SmolVLM-500M are now the tiniest VLMs you can get your hands on, with the 256M version claiming the title of the smallest VLM ever released. That’s smaller than some embedding models. And yet, here’s the kicker: the team says this 256M model already outperforms the massive Idefics 80B model they trained just 17 months ago. Let that sink in for a second.

The secret isn’t just aggressive pruning. The team made a counterintuitive call on the vision encoder, ditching the popular SigLIP 400M SO backbone used in SmolVLM 2B for a much smaller 93M-parameter SigLIP base patch-16/512. They found the bigger encoder offered only marginal gains, so they took the smaller one and cranked up its image resolution instead — a trick inspired by recent work from Apple and Google. The result is sharper visual understanding without the parameter bloat. They also threw in a tokenization trick that made training loss look worse on paper while significantly boosting performance on real-world benchmarks. Classic case of metrics not telling the full story.

On the data side, they leaned heavily into document understanding, which now makes up 41% of the training mix, alongside image captioning at 14% and new datasets like MathWriting. The 500M model gets a notable bump on benchmarks like DocVQA and MMMU and is more robust to prompting out of the box, which makes it the pragmatic pick if you’re thinking about production. But both models are designed to be fine-tuned, and both are loadable directly into transformers, MLX, and ONNX — with WebGPU demos already live. That browser-based inference angle isn’t just a gimmick. It’s where a lot of this is headed.

What’s really happening here is model parity with the SmolLM2 language family. You now have matched pairs of small LLMs and VLMs at 135M/256M, 360M/500M, and 1.7B/2B. For anyone processing huge volumes of documents or images, the cost argument is straightforward: these smaller models run at a fraction of the cost of the 2B version, and the throughput graphs on A100s show serious gains. The question now is how much capability users are willing to trade for speed and cost — and whether the community finds the 256M model genuinely useful or just impressively small.