Turning a vision-language model with zero grounding ability into a GUI agent that can click and type is a monumental data problem. Hugging Face just open-sourced the entire recipe. Their new Smol2Operator project takes SmolVLM2-2.2B-Instruct, a small model that initially can’t locate a button on a screen, and teaches it to navigate interfaces through a two-phase post-training process.
Phase one is all about perception. The team fed the model a diet of normalized grounding data, standardizing a mess of conflicting action formats from different datasets. They built a parser to wrangle function calls like a raw pyautogui.click(x=0.8102) into a clean, unified action space with coordinates normalized to [0,1]. This stage alone pushed ScreenSpot-v2 benchmark performance from nothing to a functional 65%.
Phase two layers on cognition and agentic reasoning. Once the model can see, it needs to think. The second round of supervised fine-tuning uses the curated AGUVIS datasets to teach the model how to chain actions together, transforming high-level goals like “open the app” into sequences of precise clicks and swipes. The result isn’t a state-of-the-art behemoth but an open, reproducible blueprint.
The most refreshing part of this release is the philosophy. Hugging Face isn’t hoarding a secret sauce; they’ve published the datasets on their platform and the full code on GitHub. You can even use their utilities to adapt the entire pipeline to your own custom action space. It’s a direct challenge to the black-box approach of closed-source GUI agents. The question now isn’t whether small models can do this, but how quickly the community can take this foundation and push that 65% much higher.
