AI’s new Swiss Army knives: VLMs are now reasoning, talking, and running on phones

The vision language model space has gotten weird — in the best way. When we last checked in on VLMs in April 2024, LLaVA was the open-source darling and the conversation centered on basic image understanding. A year later, the category has splintered into something far more interesting. We’re not just talking about better image captioning anymore. We’re talking about models that can see, hear, speak, reason through complex problems, and fit on a phone.

The most striking architectural shift is the rise of any-to-any models. These aren’t one-trick ponies — they take in any modality (text, image, audio) and output any modality. Meta’s Chameleon kicked this off, but Qwen 2.5 Omni is the current standard-bearer with a clever “Thinker-Talker” design that separates text generation from speech output, allowing it to stream spoken responses. DeepSeek’s Janus-Pro-7B takes a different approach with decoupled visual encoders for understanding versus generation. The bet here is that true multimodal learning — not just bolting a vision encoder onto a language model — is the path to deeper representations.

Meanwhile, reasoning has arrived in the VLM space. Qwen’s QVQ-72B was the lone open-source reasoning model for a while, but Moonshot AI just dropped Kimi-VL-A3B-Thinking. It’s a 16B-parameter Mixture-of-Experts model that only activates 2.8B parameters at a time, and it’s been tuned with long chain-of-thought and reinforcement learning. The result is a model that handles long videos, PDFs, and screenshots with agentic capabilities baked in. A smaller instruction-tuned variant is available too.

Perhaps the most practical development is the race to shrink these models. The SmolVLM family pushes the boundary to absurdly small sizes — 256M and 500M parameters — without sacrificing competence. The logic is straightforward: saturated benchmarks, diminishing returns from scaling, and a real need for models that run locally on consumer GPUs for privacy and cost reasons. The era of a capable VLM in your pocket, not just in a data center, is basically here.