Hugging Face just dropped IDEFICS, an 80-billion-parameter multimodal model that accepts images and text as input and generates coherent text in response. The release is a direct shot at the closed-source Flamingo model from DeepMind — and according to Hugging Face, IDEFICS matches Flamingo’s performance across various image-text understanding benchmarks while being fully open. Two variants are available: the full 80B version and a smaller 9B one, plus fine-tuned instruct versions for conversational use.
The training data story is equally significant. Beyond publicly available sources like Wikipedia, the Public Multimodal Dataset, and LAION, the team built a new dataset called OBELICS: 141 million interleaved image-text documents scraped from the web, totaling 353 million images and 115 billion tokens. They’ve also published an interactive visualization tool built with Nomic AI so researchers can actually explore what’s in the training data — a level of transparency that’s rare for models at this scale. The team documented their technical mistakes and learnings along the way, which is the kind of candor that typically gets buried in corporate releases.
Licensing is where things get complicated. IDEFICS is built on top of two frozen models: laion/CLIP-ViT-H-14-laion2B-s32B-b79K (MIT license) and huggyllama/llama-65b, which carries Meta’s non-commercial research license. That means users must apply to Meta directly for permission. The newly trained parameters connecting the two, however, are released under MIT. It’s a hybrid arrangement that keeps IDEFICS open but not fully unrestricted — a tension worth watching as the open-source community grapples with what “open” actually means for models built on restricted foundations.
IDEFICS joins OpenFlamingo, another open Flamingo reproduction at the 9B scale, signaling a broader push to democratize multimodal AI. The practical implication: researchers who couldn’t afford or access proprietary models now have a credible alternative for image-text understanding, visual question answering, and multi-image storytelling. Whether the non-commercial license on the LLaMA backbone limits real-world adoption remains an open question — but for the research community, this is a significant step toward reproducibility in a field that has been largely opaque.
