The question isn’t whether AI can code anymore. It’s whether it can get paid for it. OpenAI’s new SWE-Lancer benchmark puts frontier models through 1,400 real freelance software engineering tasks pulled directly from Upwork, with total payouts hitting the $1 million mark. The tasks aren’t toy problems. They span everything from $50 bug fixes to a single $32,000 feature implementation, and they’re graded using end-to-end tests triple-verified by experienced engineers. No shortcuts.
The benchmark splits work into two categories: independent engineering tasks where the model has to actually write the code, and managerial tasks where it has to choose between technical proposals. That second part is clever. The managerial decisions are measured against choices made by the original human engineering managers who hired for those Upwork gigs. It’s not just about writing good code. It’s about knowing which code is worth writing.
Results are sobering. Frontier models still can’t solve the majority of tasks. The researchers don’t dress it up as progress. They say it plainly. The system includes a unified Docker image and a public evaluation split called SWE-Lancer Diamond, along with an updated dataset from July 2025 that removes internet connectivity requirements during execution. That last change eliminated what the team called a “primary source of variability” in how models performed.
The framing here matters more than the benchmark itself. By mapping model performance directly to dollars, OpenAI is pushing a specific kind of conversation about economic impact. Not “can it pass a test” but “can it do the job someone paid a human $32,000 for.” The answer right now is mostly no. But the yardstick is on the table, and it’s denominated in real money.
