Not all correct AI answers are equal: Why Hugging Face is benchmarking the path, not just the result

The way we build libraries is about to change, and it has nothing to do with making code faster or more elegant for humans. The new user is an AI coding agent, and it doesn’t get frustrated by a clunky API—it just burns through tokens and cash trying to work around it. Hugging Face just ran a fascinating experiment to prove this point, using their own transformers library as the test dummy.

They didn’t just check if an agent got the right answer. They measured the cost of getting there. In one example, two agents correctly classified a movie review’s sentiment. One wrote a 40-line Python script, imported PyTorch, and debugged a shape error. The other simply ran transformers classify --model ... --text "...". Both returned the right label, but their token usage, latency, and failure profiles were worlds apart. As the team puts it, if your evaluation only looks at the final string, you’re completely blind to whether a change you shipped actually made life easier for an agent.

The harness, running entirely on open models driven by the pi coding agent, swept across different model and library revision combinations on Hugging Face Jobs. The core philosophy is brutally straightforward: if it isn’t tested, it doesn’t work, and if it isn’t documented, it doesn’t exist. For an agent, that means an API must be discoverable, docs need to be structured for rapid access, and tooling should be built so a single command can replace a tangled script.

This isn’t just academic navel-gazing. The team wanted hard evidence before opening pull requests adding thousands of lines of agent-optimized code to transformers. Their prior work on the hf CLI showed agents using 1.3 to 1.8 times fewer tokens with a redesign. This new benchmark provides a repeatable framework for asking the question that will soon dominate software development: is your tool agentic enough, or is it silently burning money?