The Open LLM Leaderboard is really just a wrapper around EleutherAI’s LM Evaluation Harness, running on Hugging Face’s spare compute cycles. That detail matters more than you’d think. When the LLaMA model topped the leaderboard with MMLU scores noticeably lower than what Meta’s paper claimed, the Hugging Face team started digging. What they found is a mess of implementation choices lurking beneath a single benchmark name.
Three different codebases evaluate the same MMLU dataset: the original UC Berkeley implementation, Stanford’s HELM benchmark, and the EleutherAI Harness. Same questions, same models, yet the numbers don’t just shift — they reshuffle the leaderboard rankings entirely. “These different implementations of the same benchmark give widely different numbers and even change the ranking order of the models,” the authors write. That’s not a rounding error. That’s a fundamental reproducibility problem.
The root cause lies in how language models handle multiple-choice questions. A model doesn’t “answer” a question — it predicts the probability of the next token. How you frame the prompt, how you compare the probabilities for each choice, whether you normalize by token count, all of it changes the result. MMLU has 57 knowledge domains and four choices per question, which sounds simple until you realize every implementation makes different decisions about those mechanical details.
This isn’t just an academic dispute. A model that looks state-of-the-art under one evaluation code can look mediocre under another. The Harness implementation was recently updated, suggesting the community is still converging on best practices. For anyone comparing models or building on leaderboard rankings, the message is uncomfortable: the number you’re trusting depends heavily on code you probably never read. The LLaMA paper’s numbers came from the Berkeley implementation, while the leaderboard uses EleutherAI’s harness — and those two don’t agree on what the same model actually knows.
