Hugging Face's Code Agent Tops GAIA Benchmark, Beating GPT-4 Turbo's 7%

Hugging Face engineers decided to stress-test their Transformers Agents library against GAIA, widely considered the most punishing agent benchmark out there. The result? Their code-based agent landed at the top of the public leaderboard, leaving GPT-4 Turbo’s sub-7% average in the dust and edging out a complex Autogen multi-agent system that had been sitting at 40%.

GAIA questions are deliberately brutal. One example asks which fruits from a 2008 painting appeared on a specific ocean liner’s October 1949 breakfast menu, listed clockwise from the 12 o’clock position. That single question demands multimodal image reading, chained web research across multiple steps, and rigid output formatting. The benchmark exposes exactly where LLM-based systems fall apart: high-level planning and rigorous execution.

The team kept their tool stack lean. They borrowed a Markdown-based web browser and file inspector from the Autogen submission, openly crediting that open-source work for saving weeks of development time. The browser compresses pages into markdown rather than taking screenshots for vision models, a tradeoff the team acknowledges can miss details but proved reliable overall. They note Selenium would be a future upgrade for handling JavaScript-heavy pages.

The differentiator is the code agent approach itself. Instead of outputting JSON dictionaries for tool calls, the agent writes and executes Python directly. Wang et al. (2024) showed this has real advantages, and Hugging Face’s implementation leans into it: no separate code interpreter tool is needed because code generation is the agent’s native language. This matters because GAIA’s chained reasoning demands precise, multi-step execution that JSON-based action formats handle clumsily. The benchmark win suggests code-native agents may be the more promising path forward for complex agentic systems than the JSON tool-calling approach OpenAI popularized.