JSON plus Python beats raw code: structured agents gain 7 points on reasoning benchmarks

The way AI agents take action is getting a quiet but important upgrade. For years, the default has been JSON function calling — an agent picks a tool from a list and spits out a rigid blob of JSON. It works, but it’s brittle. Then came Code Agents, which let models write executable Python to call tools inside loops, conditionals, and functions. Far more expressive. The catch? Parsing code out of markdown is a mess — incomplete code fences and stray blocks still break things regularly.

New benchmarks from the smolagents team show a third path is the real winner: structured generation that wraps a Code Agent’s reasoning and executable code inside a clean JSON schema. The model is forced to output a “thoughts” field and a “code” field as structured JSON, with the code block parsed directly into Python. No markdown guessing games. No fragile regex.

The numbers are persuasive. On a composite of GAIA, MATH, SimpleQA, and Frames, the structured CodeAgent beat the regular CodeAgent by 2 to 7 percentage points on average across the most capable models. OpenAI’s models saw the biggest jump, especially on reasoning-heavy tasks. Claude 3.7 Sonnet posted particularly strong results with the structured approach. Even Qwen models improved, though smaller variants started feeling a “structure tax” — the extra overhead of the format can trip up less capable models.

This isn’t just a parsing fix. The enforced “thoughts” block means the agent gets a dedicated space to reason before executing, which the team says leads to clearer separation between planning and action. You get the unlimited flexibility of Python — loops, state, conditional tool calls — with the reliability of a schema that won’t silently fail. For anyone building production agents, that combination solves two headaches at once. The interesting question now is whether this becomes the new default for agent frameworks, or if we’ll see model providers bake structured code execution directly into their APIs. The parsing problem isn’t going away on its own.