Hugging Face's Idefics2 shrinks to 8B parameters but now beats much larger vision models

Hugging Face has shipped Idefics2, an 8-billion-parameter vision-language model that does something genuinely surprising: it competes with systems four times its size. The team claims performance on visual question answering benchmarks matches or beats LLaVA-NeXT-34B and MM1-30B-chat, despite running on a fraction of the parameters. That’s not typical. Usually when you cut model size by 10x — which is exactly what happened from Idefics1’s 80B parameters — you expect to lose something. Instead, Idefics2 gained ground.

The architecture got a significant rework. Idefics1 relied on gated cross-attentions to blend visual and textual information, a design the team abandoned in favor of a simpler pipeline: images go through a vision encoder, then a learned Perceiver pooling layer, then an MLP projection before being concatenated with text embeddings. The model also now processes images at native resolutions up to 980x980 pixels using a NaViT strategy, avoiding the fixed-square resizing that has long been standard practice in computer vision. For very large images, optional sub-image splitting follows the approach used by SPHINX and LLaVA-NeXT.

OCR capabilities received particular attention. The team integrated training data requiring the model to transcribe text from images and documents, alongside chart and figure question-answering data. Pretraining drew on openly available sources including OBELICS interleaved web documents, LAION-COCO image-text pairs, PDFA and Rendered-text OCR data, and WebSight image-to-code examples. The model ships under Apache 2.0, which removes the licensing friction that has kept many capable multimodal models out of production pipelines.

Hugging Face also released The Cauldron, a compilation of 50 manually-curated datasets formatted for multi-turn conversations, used for instruction fine-tuning. The move addresses a real bottleneck: task-oriented training data for foundation models tends to be scattered across incompatible formats, and assembling it is tedious work most teams would rather not repeat. Idefics2 is available on the Hub with Transformers integration from day one, making fine-tuning straightforward for developers who need domain-specific multimodal behavior.