We’ve been benchmarking AI agents all wrong. That’s the blunt message from the team behind VAKRA, a new evaluation suite that ditches isolated skill tests for something far messier and more honest: actual enterprise workflows. Instead of asking a model to answer a trivia question or write a clean function, VAKRA forces agents to chain together API calls, read unstructured documents, and execute multi-step reasoning against real databases. The results aren’t just bad—they’re a wake-up call.
VAKRA isn’t a synthetic sandbox. It hosts over 8,000 locally run APIs across 62 domains, from sports analytics to business intelligence. Agents have to do things like filter a dataset about football teams by “build-up play speed” and “dribbling” values, then retrieve the team name. It sounds simple. It’s not. The benchmark includes four distinct tasks, each probing a different failure-prone capability: chaining generic data manipulation tools (called SLOT-BIRD), selecting from a swarm of highly specific endpoint-style APIs (REST-BIRD), and more. One task, API Chaining, alone contains 2,077 test instances where a single mistake in a 3- to 7-step tool call chain tanks the entire answer.
The design is clever about the real-world constraints that typically break agents. To avoid choking on massive data transfers, a special get_data function returns only a lightweight preview—column names, data types, and the first three values of each column—while the full dataset stays server-side. The agent then has to reason about which tools to use based on that sparse preview. As Harshit Joshi, one of the researchers, pointed out in the release, “models perform poorly on VAKRA.” The leaderboard is open, and the initial scores are a stark reminder that we’re far from having an AI that can reliably act as a digital employee.
What makes this sting is the specificity of the failure. This isn’t about a lack of world knowledge; it’s about a breakdown in the fundamental loop of observation, planning, and action. The benchmark is executable, meaning there’s no room for the model to bluff. The answer is right or it’s wrong, verified by a complete execution trace. For enterprise developers betting their workflows on agentic frameworks, VAKRA is the most honest performance review their tech stack will ever get—and it’s grading on a harsh curve.
