When GPT‑5.6 Sol debuted on the ARC-AGI-3 benchmark, the results were baffling. The same model that solved the cycle double cover conjecture and beat Pokémon FireRed managed a measly 7.8% on a set of 2D puzzle games. GPT‑5.5 was effectively useless at 0.4%. It didn’t add up—until OpenAI engineers dug into the harness, not the model.
The culprit wasn’t a lack of intelligence. It was amnesia. The generic testing harness, designed for fair model comparisons, was discarding the model’s private reasoning after every single action. GPT‑5.6 Sol was forced to figure out each game from scratch, turn after turn, unable to remember its own plans or insights. Making matters worse, a rolling truncation window was silently deleting older actions from its memory as the game progressed. The model wasn’t failing to learn; it was being prevented from learning.
Switching to OpenAI’s Responses API changed everything. By simply retaining reasoning across tool calls, the model spent less time thinking before each move and started forming coherent strategies. Replacing the crude rolling truncation with intelligent compaction—summarizing rather than deleting—let it preserve hard-won knowledge across longer runs. An animation from the team shows the stark difference: with the old harness, the context window was a leaky bucket; with the new one, it became a reliable record.
The payoff was immediate and dramatic. With these two production settings enabled, the same GPT‑5.6 Sol model roughly tripled its score while slashing output token consumption by 6x. It’s a pointed reminder that benchmark scores measure a whole system, not just a brain. Changing the scaffolding can be as important as changing the model itself.
