Hugging Face drops nanoVLM: a 135M-param vision model you can train from scratch in pure PyTorch

Hugging Face just released nanoVLM, a deliberately minimal toolkit for training vision-language models that feels like a direct spiritual successor to Andrej Karpathy’s nanoGPT. The entire codebase is a handful of readable Python files—no sprawling abstractions, no black-box training frameworks. You clone the repo, run python train.py, and you’re off.

Under the hood, it marries Google’s SigLIP vision encoder with a SmolLM2-135M language backbone, bridged by a modality projection module that uses a pixel shuffle trick to compress image tokens before feeding them into the transformer decoder. That compression step isn’t cosmetic; it directly cuts the computational cost that makes most VLM training painful for anyone without a cluster. The default config targets visual question answering, keeping the objective narrow enough to actually finish a training run on a single GPU.

The repo structure is almost aggressively simple: models/vision_transformer.py, models/language_model.py, models/modality_projector.py, and a handful of data utilities. You can swap in different SigLIP or SmolLM2 variants if you want to experiment, but the defaults are chosen to be small enough to iterate on quickly. There’s even a Colab notebook so you can launch training with zero local setup.

What makes this worth paying attention to isn’t the model itself—135M parameters won’t set benchmarks on fire—but the educational wedge it drives into a field that’s become increasingly opaque. Most VLMs ship as giant black boxes with proprietary training recipes. nanoVLM lets you watch the entire pipeline from pixel to token, which is exactly how a generation of engineers learned transformers from nanoGPT. I wouldn’t be surprised if this becomes the default teaching tool in university vision courses within a year. The real test will be whether the community starts forking it to add modalities beyond vision and text.