MoonshotAI’s PerceptionBench Exposes Where GPT-4o-Mini Still Can’t See Straight

A new open-source toolkit from MoonshotAI lets you systematically stress-test multimodal AI models on seven core visual skills—and the results are a reality check for anyone assuming today’s APIs just ‘get’ images. The PerceptionBench evaluation harness, designed to run in a single Colab notebook, measures fine-grained visual perception: OCR, counting, localization, contextual reasoning, comparison, depth understanding, and hallucination detection.

The system works by pulling a balanced dataset sample, decoding base64 images on the fly, and normalizing everything into a consistent record format. From there, it pings any OpenAI-compatible API—the default configuration targets GPT-4o-mini—or runs a local Hugging Face model like SmolVLM2-2.2B. The code then applies rule-based judging (with an optional LLM-assisted judge) to score responses, calculates bootstrap confidence intervals, and slices performance by difficulty. That last part matters. A model might nail simple counting but fall apart when localization gets tricky. Without per-skill breakdowns, you’re flying blind.

MoonshotAI included a leaderboard for comparison, and the default settings reveal something worth paying attention to: the blind-prior baseline is literally random chance. That’s the floor. Every model you test starts from zero. The harness then measures exactly how far above that floor a given system climbs—and on which skills it barely moves the needle. For tasks like hallucination detection, even frontier models often hover uncomfortably close to that baseline.

What’s genuinely useful here is the reproducibility. The entire workflow exports prediction artifacts and reporting charts. You can rerun it next week when a new model drops and get an apples-to-apples comparison. No vendor benchmarks, no cherry-picked examples—just the same 84 examples per capability (12 per category across 7 tasks) hitting the same evaluation logic. In an industry drowning in vague leaderboard claims, that kind of rigor is a quiet flex.