The AI industry has quietly built its evaluation infrastructure on a shaky foundation. When a new model claims it’s “preferred to ChatGPT 80% of the time,” what that actually means is that another language model—usually GPT-4—said so. Human preference data is expensive and scarce, so the field has substituted LLM-as-judge evaluations without rigorously checking whether those judgments align with actual human opinions.
Hugging Face decided to test this directly. They curated a held-out set of 327 prompts spanning generation, brainstorming, question answering, summarization, commonsense, and coding tasks. Four open-source models—Koala 13b, Vicuna 13b, OpenAssistant 12b, and Dolly 12b—generated completions for each prompt. Then they asked Scale AI’s human annotators to rate the responses on a Likert scale from 1 to 8, following Anthropic’s preference modeling recipe. The same comparisons went through GPT-4 evaluations.
The humans produced bootstrapped Elo rankings, and the researchers compared those against GPT-4’s judgments. The gap between what GPT-4 considers good and what humans actually prefer is the core finding—and it’s a problem for anyone relying on LLM-based evaluation pipelines. The study expands the Hugging Face Open LLM Leaderboard to include automated academic benchmarks, professional human labels, and GPT-4 evals side by side.
This matters because RLHF itself depends on preference data. If the reward models trained on LLM-generated preferences don’t match human preferences, the entire training pipeline drifts. The efficiency gain from using language models to evaluate outputs means nothing if the measurements aren’t calibrated to what they’re supposed to approximate. The blog post zooms in on exactly where LLM labels can and cannot be trusted—and the answer isn’t uniform across task types.
