Vectara just released a new leaderboard for tracking LLM hallucinations, and here’s the interesting part: they built it using Hugging Face’s freshly released leaderboard templates. Those templates are lightweight, open-source versions of the Open LLM Leaderboard code, designed to be simpler to deploy than the original. Vectara’s team took the template, customized it for their Hughes Hallucination Evaluation Model (HHEM), and shipped the whole thing as a Hugging Face Space.
The HHEM itself has been around since late 2023. It’s an open-source classification model that measures how often LLMs generate text that’s nonsensical or unfaithful to source content. The original release covered a mix of models — Llama 2, Mistral 7B, GPT-4, Claude, and Gemini — and the differences in hallucination rates were stark. That’s the kind of data practitioners actually need when choosing between open-source and commercial models.
What’s new here is the infrastructure. The Vectara team describes setting up two datasets in Hugging Face: one for evaluation requests and one for results. They populated the results dataset with existing scores, updated the About and Citations sections, and for basic leaderboards that’s apparently all it takes. Their implementation got more complex because HHEM evaluation involves both a SummaryGenerator that produces summaries from a private dataset and an EvaluationModel that loads the proprietary HHEM to score factual consistency and hallucination rate.
The code is public in the Files tab of their leaderboard repo, and Vectara is actively encouraging the LLM community to submit new models for evaluation. This matters because hallucination tracking has historically been fragmented — individual papers release numbers, but there’s no central place to compare. The Hugging Face template lowers the barrier for anyone to spin up a focused leaderboard without rebuilding the plumbing from scratch. Whether that leads to a proliferation of niche leaderboards or genuinely useful standardization remains to be seen, but the open-source release of both the evaluation model and the leaderboard framework feels like a step toward making hallucination benchmarks as accessible as general capability benchmarks already are.
