Vision language models are having a moment. These systems — which take images and text as input and generate text — now handle everything from visual question answering to document understanding with zero-shot capabilities that would have seemed absurd two years ago. The open-source field is especially active, with at least eight serious contenders on the Hugging Face Hub, including LLaVA 1.6, Idefics2, DeepSeek-VL, and Qwen2-VL. Some are base models. Others are fine-tuned for chat. A few support “grounding,” which means they can output bounding boxes or segmentation masks and reduce hallucination by anchoring responses to spatial locations in the image.
Choosing the right model isn’t a matter of grabbing the most popular one. The Vision Arena leaderboard ranks models purely on anonymous human preference votes — users submit an image and prompt, compare two outputs, and pick the better one. The Open VLM Leaderboard takes a more quantitative approach, scoring models on benchmarks and letting you filter by size, license, and metric. For teams that want to run their own evaluations, VLMEvalKit powers that leaderboard, while LMMS-Eval offers a command-line interface to test any Hugging Face model against datasets like MME and MMBench.
Benchmarks matter here, and MMMU is the heavyweight. It contains 11,500 multimodal challenges requiring college-level reasoning across arts, engineering, and other disciplines. MMBench evaluates 3,000 single-choice questions spanning 20 skills, and its CircularEval strategy shuffles answer choices to ensure the model isn’t just pattern-matching. More specialized suites exist too: MathVista for visual math, AI2D for diagrams, ScienceQA for science questions, and OCRBench for document understanding.
What’s notable is how the leaderboards lag behind the actual model ecosystem. Both Vision Arena and the Open VLM Leaderboard only rank models that are explicitly submitted, meaning newer releases — like some of the models mentioned in this post — aren’t reflected. If you want the full picture, browse the image-text-to-text task on the Hub directly. The space is moving faster than the scoreboards.
