Docmatix drops 9.5M QA pairs to close the DocVQA gap with closed models

The team behind Idefics2 just released Docmatix, a document visual question answering dataset that makes the previous standard look like a rounding error. We’re talking 2.4 million images and 9.5 million question-answer pairs pulled from 1.3 million PDF documents. That’s a 240X jump over DocVQA, which topped out at 10,000 images and 39,000 QA pairs — the dataset most open-source models have been stuck fine-tuning on.

The scale gap between open and closed vision-language models has been stubborn, and the Hugging Face crew thinks data scarcity is a big part of it. Docmatix builds on PDFA, an existing OCR dataset with 2.1 million PDFs. They took those transcriptions and fed them to a Phi-3-small model to generate the QA pairs, then ran a filtering pass that discarded 15% of generations flagged as hallucinations. Regular expressions caught code snippets, and any answer containing the word “unanswerable” got the boot. The result is a dataset where every row traces back to an original PDF in PDFA — no mystery provenance.

Prompt design turned out to matter more than you’d expect. The team ran ablation studies on a small batch and found that guiding Phi-3 to ask questions about specific document details — like “What are the titles of John Doe?” — nearly eliminated repetitive questions. They optimized for roughly four QA pairs per page, pushing back against both sparse coverage and overlapping questions. Human-like answer length was another constraint, avoiding one-word responses and rambling paragraphs alike.

The proof is in the fine-tuning. A Florence-2 model trained on just 20% of Docmatix’s images and 4% of its QA pairs, followed by one epoch on DocVQA for format alignment, saw a relative improvement of almost 20% on DocVQA evaluation. That’s a striking result for a fraction of the data. Even more telling: the 0.7B Florence-2 model came within 5% of the much larger 8B Idefics2, which was trained on a mixture of datasets. It’s a reminder that in multimodal training, data quality and scale often matter more than parameter count.

Docmatix won’t single-handedly close the gap with proprietary models like GPT-4V or Gemini, but it gives the open-source community something it’s never had before: a document VQA dataset large enough to actually test scaling laws. The processed images are already on the Hugging Face Hub at 150 dpi, saving researchers from the resource-intensive PDF conversion step. What remains to be seen is whether other teams can squeeze similar gains, or whether Florence-2’s architecture is just particularly well-suited to this kind of data. Either way, expect a wave of new DocVQA fine-tunes on the Hub in the coming months.