Hugging Face just made it dramatically easier to test large language models for hidden flaws — and the early results aren’t pretty. The company’s Evaluation on the Hub tool, powered by AutoTrain, now supports zero-shot evaluation for any causal language model with up to 66 billion parameters. No custom code. No GPU cluster. Just upload a dataset, pick your models, and wait for a pull request with results.
The infrastructure upgrade is significant. Previously, a 66-billion-parameter model took roughly 35 minutes just to load and compile before evaluation could even begin. That barrier kept serious testing in the hands of well-resourced labs. Now, running a zero-shot classification task with 2,000 sentence-length examples on a model that size takes about 3.5 hours — and it’s free. Anyone in the community can do it.
As a case study, the team evaluated several OPT models on WinoBias, a coreference benchmark designed to measure gender bias tied to occupations. The task asks models to choose between a stereotypical and anti-stereotypical pronoun — for instance, whether a “developer” should be referred to as “he” or “she.” The results showed a clear inverse scaling trend: smaller models were more likely to pick the anti-stereotypical pronoun, while larger models leaned harder into stereotypical associations between gender and occupation. This lines up with findings from BIG-Bench and earlier work showing that bigger, more capable models also generate more toxic text and display stronger biases across race, ethnicity, and nationality.
The tool’s flexibility is the real story for researchers. Any dataset that can be reshaped into a Winograd-style schema — where examples differ only by a few words — can be plugged in and run across many models simultaneously. That makes it practical to study questions like the inverse scaling problem: the counterintuitive pattern where larger models get worse at certain tasks even as their general capabilities improve. Hugging Face says support for models larger than 66 billion parameters is on the way, which will open the door to testing the very systems where these failure modes matter most.
