String-matching metrics are failing VQA models—even when answers are right

The way we score visual question answering models is quietly falling apart. On Docmatix, a synthetic document VQA dataset 100 times larger than anything previously available, models generate answers that are semantically spot-on yet receive dismal scores under traditional metrics like ANLS, BLEU, and CIDER. The reason is straightforward: exact string matching punishes any deviation in format, specificity, or interpretation. Zero-shot answers don’t parrot the reference text, so they get marked wrong.

A t-SNE visualization comparing zero-shot generated answers against reference captions shows the two clusters sitting close together in embedding space. The answers are similar. The scoring says otherwise. That gap is the whole problem. The researchers behind this analysis point out that this mismatch is especially acute when models trained or prompted on instruction-generated synthetic data are evaluated against human-curated benchmarks like DocVQA. Some teams have tried to force answer formats to align with reference strings, but that treats a symptom, not the underlying flaw in the metric itself.

The team tested MPLUGDocOwl1.5, a model that hits an 84% ANLS score on the original DocVQA test subset. They then ran it zero-shot on a 200-image Docmatix subset and scored the outputs using LAVE, an LLM-assisted evaluation method that uses Llama-2-Chat-7b as the judge. LAVE reframes evaluation as a rating task, asking the LLM to assign a 1-to-3 score based on whether an answer is incorrect, ambiguous, or correct, with the instruction to provide a rationale before the rating. Demonstrations in the prompt show the model what a good rating looks like, and the phrase “Provide only one rating” keeps it from drifting into sentence-by-sentence analysis.

The deeper question this work raises isn’t about any single model. It’s about whether the field should keep fine-tuning models to game brittle metrics, or finally build evaluation methods that track human perception. Human evaluation remains the gold standard, but it doesn’t scale. LLM judges like the one used here are cheap and fast, but they introduce their own biases and inconsistencies. The tension between scalability and reliability isn’t going away, and until the community settles on metrics that reward semantic correctness over string identity, zero-shot evaluation results will keep understating what these models actually know.