Hugging Face's ScreenSuite stress-tests 5 AI models on pure vision — no DOM cheating

Evaluating AI that can actually use a computer like a person — by looking at the screen — remains surprisingly hard to measure. Hugging Face just dropped ScreenSuite, a new benchmarking kit they’re calling the most comprehensive suite for testing Vision Language Models on GUI agent tasks. It aggregates 13 different tests covering everything from basic screen perception to complex multi-step workflows like filling an Excel column or buying something online.

What makes this different from existing academic benchmarks is a deliberately stripped-down setup. “Our stack is intentionally vision-only,” the team writes. Most previous tests handed models an accessibility tree or raw DOM data — essentially giving them a cheat sheet of what’s on screen and where. ScreenSuite forces models to work purely from screenshots, the way a human would. They built custom Docker containers to spin up Ubuntu and Android environments locally, and for multi-step tasks they’re using their own smolagents framework as the orchestration layer. No extra metadata. Just pixels.

They’ve already run five leading models through the gauntlet: the Qwen-2.5-VL family (3B to 72B parameters), ByteDance’s UI-Tars-1.5-7B, H company’s Holo1-7B, and GPT-4o. The Qwen models have an edge here because they’re built with native coordinate outputs — they can say “click at x=130, y=540” directly rather than describing where to click. The team is careful to note that their scores won’t match previously published leaderboards for benchmarks like Mind2Web, where competitors gave models the accessibility tree. Removing that crutch makes tasks dramatically harder.

The real value here isn’t just another leaderboard — it’s that someone finally packaged a realistic, hard-mode evaluation that doesn’t require researchers to spin up their own Windows VMs and stitch together five different testing frameworks. For anyone building agents that need to navigate actual interfaces rather than APIs, this is the closest thing to a standardized driving test.