Stop obsessing over model weights. The real bottleneck in autonomous AI isn’t the brain; it’s the body. A new benchmark called HarnessDev, proposed by researchers from ByteDance Seed and Georgia Tech, shifts the evaluation target from the answer a model produces to the runnable code harness it builds. The results are stark. On the Terminal-Bench 2.1 leaderboard, GPT-5 solves just 35.2% of tasks inside Terminus 2 but jumps to 49.6% inside Codex CLI using identical weights. The difference? The execution loop, tool integration, and state management surrounding the model.
In the Creation stage, six creator LLMs including Opus 4.8 and Gemini 3.1 Pro built harnesses from weak seeds. Opus 4.8 led with an average score of 67.8, nearing the human-engineered reference of 86.2. But domain matters. While Opus beat the reference on writing tasks (EQ-Bench3) and ML experimentation (MLE-bench), it trailed significantly on code and search. GPT-5.5 managed only 52.6 on BrowseComp against a 92.2 reference. Crucially, more code didn’t mean better performance. Gemini added just 1,006 lines yet led Terminal-Bench, while others bloated their harnesses with inert machinery. Of 108 code components, 18 never fired, and 124 writing features were dead code.
The Evolution stage revealed how fragile these systems are. Creators revised their harnesses using feedback from SWE-bench Pro and Terminal-Bench tasks. Progress was messy. Only two of nine final versions were optimal on held-out data, and feedback scores aligned with actual performance just 53.1% of the time. Swapping the executor model reshuffled rankings entirely. When Qwen’s harness ran on Gemini instead of its native creator, it gained 17.6 points on BrowseComp. Conversely, Opus 4.8’s SWE-Pro score collapsed from 69.3 to 33.0 because one harness hard-coded a step limit specific to its original executor.
This exposes a critical blind spot in current AI development. We treat agents as monolithic models, but they are actually complex software systems where small architectural choices dictate success. A single commit can swing performance by ±4.75 points. The clearest win came when Opus 4.8 diagnosed a premature completion bug, adding a gate that fixed 99 false positives. If we want reliable agents, we need to stop tuning prompts and start engineering robust execution loops.
