Every major model release ships a coding benchmark number. The number ticks upward. And the conclusion everyone rushes to—junior engineers are finished—is being drawn backward. People are leaping from a benchmark score straight to a labor market outcome without checking anything in between.
The first condition for replacement is reliability at real task lengths. METR’s time-horizon research shows frontier models succeed 50% of the time on tasks that take humans hours—a figure that has doubled roughly every seven months since 2019. Sounds decisive until you read the methodology. The 80% reliability horizon is dramatically shorter. And METR’s own framing admits their tasks are deliberately self-contained and well-specified, stripped of the context acquisition that defines a junior’s first six months on the job. A benchmark that removes the hard part isn’t measuring the job.
Condition two collapsed in February 2026 when OpenAI stopped reporting SWE-bench Verified and recommended others do the same. Their audit of a 27.6% subset found at least 59.4% of problems had flawed test cases rejecting correct solutions—plus contamination where frontier models reproduced gold patches verbatim. State of the art had climbed from 74.9% to 80.9% in six months, but on harder, cleaner suites like SWE-bench Pro, scores fall off a cliff.
Condition three has the cleanest evidence. METR’s randomized controlled trial with 16 experienced open-source developers across 246 real tasks found something brutal: developers predicted a 24% speedup, estimated 20% afterward, and were actually 19% slower. The perception gap is the durable finding. Stack Overflow’s 2025 survey of 49,000+ developers found 46% actively distrust AI output accuracy against 33% who trust it—only 3% report high trust. The fourth condition, the one that doesn’t require the other three to hold, is where the real worry should live. But that’s a conversation about who verifies the verifiers, and nobody’s staffing for it yet.
