OpenAI's o1 model beats PhDs and cracks top 500 in Math Olympiad

OpenAI just dropped an early version of a new model, o1-preview, that doesn’t just incrementally improve on GPT-4o — it genuinely changes the game on hard reasoning tasks. We’re talking about a system that scores in the 89th percentile on competitive programming questions from Codeforces and places among the top 500 high school students in the US on the AIME, a brutal qualifier for the Math Olympiad. Those aren’t cherry-picked demos; they’re standardized benchmarks the industry has used for years.

The jump in math is particularly stark. On the 2024 AIME exams, GPT-4o could only manage a 12% average, solving about 1.8 out of 15 problems. o1, with a single sample per problem, hit 74%. Crank up the test-time compute with a consensus of 64 samples and you get 83%. Re-rank 1,000 samples with a learned scoring function and it nails 93% — a score of 13.9 out of 15, well above the cutoff for the USA Mathematical Olympiad. That’s not just beating a benchmark; it’s a genuinely impressive display of mathematical reasoning.

OpenAI is calling this a new paradigm, and they might be right. The secret sauce is a large-scale reinforcement learning algorithm that teaches the model to use a “chain of thought” productively. Think of it as the model learning to stop, think, catch its own mistakes, and try different approaches before blurting out an answer. The blog post is clear that this is highly data-efficient and that performance keeps improving the more compute you throw at it during both training and inference. The constraints on scaling this are different from standard LLM pretraining, and OpenAI admits they’re still figuring out the full picture.

The model even surpassed human PhD-level accuracy on the GPQA diamond benchmark for physics, biology, and chemistry problems — a first for any model. To be clear, OpenAI explicitly states this doesn’t make o1 more capable than a PhD in all respects, but it’s more proficient on some problems a PhD would be expected to solve. On top of that, with vision enabled, o1 scored 78.2% on MMMU, making it competitive with human experts, and it beat GPT-4o on 54 out of 57 MMLU subcategories. This isn’t just a language model getting slightly better at trivia; it’s a shift toward systems that can actually reason.