OpenAI's o1-preview can snag Kaggle bronze, new MLE-bench reveals

How good are AI agents at doing the actual job of a machine learning engineer? Not just writing snippets, but wrangling messy data, tuning models, and submitting to a leaderboard. A new benchmark from OpenAI called MLE-bench puts that question to the test, and the results are a mixed bag — impressive in flashes, but mostly humbling.

The team curated 75 ML engineering competitions from Kaggle, a platform where humans regularly sharpen their skills against real-world problems. These aren’t toy tasks; they demand the full stack of an ML engineer’s workflow, from data preparation to experiment tracking. For each competition, they established a human baseline using Kaggle’s public leaderboards, creating a clear target for the AI agents.

So how did the bots do? The best setup paired OpenAI’s o1-preview reasoning model with a scaffold called AIDE. That combination managed to reach at least bronze-medal level performance — meaning it beat the bottom 60% of human competitors — in 16.9% of the competitions. That’s a non-trivial number. It suggests frontier models are starting to grasp the grunt work of ML engineering, not just the theory. But flip that stat around: in over 83% of cases, the agent couldn’t even scrape bronze.

The researchers also dug into what happens when you throw more compute at the problem and how much pre-training contamination might be inflating the scores. They’ve open-sourced the benchmark code to help the community track what’s coming next. Given the speed of progress, that 16.9% figure might not hold for long.