Moonshot AI just released PerceptionBench, and the results are brutal for anyone convinced multimodal models can actually see. The benchmark strips away reasoning and external knowledge to test ten atomic visual skills—counting, OCR, depth perception, fine-grained recognition—using questions a child could answer. Sixteen frontier models took the test. Not one cracked 60 percent.
GPT-5.6 Sol limped to first place at 59.7 percent, barely edging out Moonshot’s own Kimi K3 at 58.5 percent. Claude Fable 5 and Gemini 3.1 Pro followed at 57.2 and 56.2 percent. The open-source gap is even wider: Qwen3.5-397B-A17B managed 47.5 percent while GLM-4.6V collapsed to 32.5 percent. These are tasks like counting flowers in a red box or identifying where a symbol sits on a clock face—visual basics that humans handle effortlessly.
The category breakdown reveals something more interesting than the overall rankings. Hallucination is the universal weak spot. GPT-5.6 Sol, despite winning the overall crown, scored a dismal 26.9 percent on detecting objects that don’t exist, while the much weaker Gemini 3.5 Flash hit 50.6 percent on the same sub-test. Models with nearly identical aggregate scores diverge wildly on individual skills, which suggests the rankings hide as much as they reveal.
Moonshot built the benchmark from real failures, not theoretical categories. The team analyzed 42 open-source benchmarks, found their error profiles barely overlapped, and traced each model mistake back to its earliest failure point. From over 17,000 verified questions, they’re publishing 3,000. The implication is uncomfortable: many failures blamed on reasoning actually happen before reasoning starts—at the moment the model tries to read the image. This echoes earlier work from the same team, where WorldVQA showed Gemini 3 Pro couldn’t crack 50 percent on object recognition, and BabyVision found frontier models scoring 49.7 percent on tasks that humans complete at 94.1 percent. The gap between seeing and claiming to see remains enormous.
