New leaderboard ranks which LLMs hallucinate least — and the results sting

Most LLM benchmarks reward models for sounding confident. The Hallucinations Leaderboard, an open project from the University of Edinburgh, flips that logic: it tests whether models can avoid making things up. The platform evaluates a wide range of language models against tasks specifically designed to expose hallucination-related failures, using the EleutherAI Language Model Evaluation Harness for zero-shot and few-shot evaluation via in-context learning.

The leaderboard splits hallucinations into two buckets. Factuality hallucinations occur when a model contradicts verifiable real-world facts — the project’s example is a model claiming Charles Lindbergh walked on the moon in 1951, when Neil Armstrong did it in 1969. Faithfulness hallucinations happen when generated content drifts from the user’s instructions or the given context, like a summarizer changing a conflict’s event date from October 2023 to October 2006. That distinction matters because a model can be factually correct yet still betray the source material, a failure mode that’s especially dangerous in news summarization, historical analysis, or health-related applications.

The benchmark suite is broad: closed-book open-domain QA (NQ Open, TriviaQA, TruthfulQA), summarization (XSum, CNN/DM), reading comprehension (RACE, SQuADv2), instruction following (MemoTrap, IFEval), fact-checking (FEVER), hallucination detection (FaithDial, True-False, HaluEval), and self-consistency (SelfCheckGPT). All metrics are normalized to a scale, so a TruthfulQA MC1 score of 0.8 means 80% accuracy. Experiments run on NVIDIA A100-40GB and A100-80GB GPUs at the Edinburgh International Data Facility and the university’s School of Informatics clusters.

The project is open and actively seeking contributions — new tasks, modified benchmarks, or compute resources. The team has also released a paper on arXiv with a Hugging Face page for community discussion. That openness is notable: hallucination evaluation has mostly been scattered across individual papers, and consolidating those benchmarks into one living leaderboard gives researchers a clearer picture of which models actually hold up under scrutiny. The code is a fork of the Hugging Face Leaderboard Template, which should make it familiar territory for anyone who’s worked with that infrastructure.

One thing to watch: the leaderboard’s value will depend on how quickly it incorporates new models and whether the community treats it as a neutral referee rather than another scoreboard to game. Hallucination benchmarks are notoriously tricky — a model can ace TruthfulQA and still fabricate with confidence in open-ended conversation. This platform won’t settle the debate, but it gives the field something it’s been missing: a common yardstick.