Most multimodal model evaluations test whether a system can count objects or list items in a photo. UCLA researchers argue that’s the easy part. Their new ConTextual benchmark instead probes something harder: context-sensitive reasoning over text-rich scenes, where getting the answer right means jointly understanding visual layout and textual meaning.
The dataset spans 506 human-written instructions across eight real-world domains, including time reading, shopping, navigation, mobile apps, webpages, and infographics. Each sample pairs a text-heavy image with a task and a reference response. Researchers released a 100-instance validation set publicly, while keeping the full test set answers hidden for leaderboard integrity.
Thirteen models faced the benchmark, split into three camps: augmented LLMs (GPT-4 fed OCR output and dense captions), proprietary LMMs like GPT-4V and Gemini Vision Pro, and open-source contenders including LLaVA-v1.5-13B and ShareGPT-4V-7B. Evaluation used GPT-4 as a judge, prompted with the instruction, reference answer, and model prediction. The team picked GPT-4 because it correlated most strongly with human judgments in their experiments.
Some results defy expectations. GPT-4V beat humans on abstract reasoning tasks, possibly thanks to training exposure to memes and quote-heavy images. But the same model stumbled on time-related infographics where people excel. Open-source models showed a stark divide: they earned acceptable human ratings on abstract and natural scenes, then cratered on time-reading, navigation, shopping, web, and mobile domains. That pattern suggests those domains are simply out-of-distribution for current open training data.
The augmented LLM approach fared worst of all. Converting visual information into OCR text or captions and feeding it to GPT-4 produced notably poor human approval ratings, reinforcing that joint reasoning cannot be bolted on after the fact. If the industry wants LMMs that can actually help with a grocery list, a bus schedule, or a confusing checkout page, training data diversity and architectural integration both need work.
