AI Benchmarks
Standardised tests used to measure and compare AI model capabilities, from knowledge and reasoning to coding and multilingual understanding.
AI benchmarks are the yardsticks of the field: fixed tasks with known answers that let researchers compare models and track progress. They range from multiple-choice knowledge questions to competitive programming problems, long-context retrieval tests, and agentic simulations.
Well-known examples:
- Knowledge and reasoning: MMLU and its successors, GPQA, HLE.
- Maths and science: MATH, AIME-style competitions.
- Coding: HumanEval, SWE-bench, LiveCodeBench.
- Long context: needle-in-a-haystack retrieval and long-document QA.
- Agents: tool-use and computer-use suites such as OSWorld and Terminal-Bench.
Benchmarks have limits. Once published, they leak into training data, inflating scores through contamination. Labs report results under varying conditions, making apples-to-apples comparison hard. And a high score on a static test may not translate to real-world usefulness.
The response has been live and private benchmarks that rotate questions, held-out test sets, human preference arenas, and task-specific evaluations built from actual user work. For anyone choosing a model, the practical advice is to evaluate on your own tasks — benchmarks narrow the shortlist; your workflow picks the winner.
Read more in Guide: How to Evaluate AI Models: A Practical Guide