Most people assume an AI safety benchmark measures safety. BenchMIRT, a new auditing method from researchers analyzing 100 LLMs across 16 benchmarks and 34,000-plus questions, suggests that assumption is often wrong.
Take BBQ, a benchmark designed to catch social stereotypes. One question asks about a grandson and grandfather booking an Uber. It probes age bias, sure. But it also demands the model track who’s who and reason from evidence rather than assumptions. BenchMIRT found BBQ aligns far more with general reasoning than with safety. A low score might mean the model can’t parse the scenario, not that it’s biased. That’s a significant confound for anyone using BBQ to certify a model as safe.
The method adapts multidimensional Item Response Theory from psychometrics, the field behind standardized testing. Classic IRT treats questions as varying in difficulty and discriminative power. BenchMIRT extends that to multiple latent capabilities simultaneously, without being told which benchmarks measure what. It independently recovered two dominant dimensions: safety and general reasoning. The result held when the analysis was rerun from scratch, suggesting stability rather than a statistical fluke.
What else did it find? WMDP, which tests dangerous dual-use knowledge in biology, chemistry, and cybersecurity, tracks with general reasoning, not safety. Stronger reasoning actually correlates with lower WMDP scores because the benchmark rewards refusing to provide dangerous information. HarmBench’s standard and contextual prompts align with safety, but its copyright questions pull toward reasoning. WildJailbreak mixes both signals. The practical consequence: averaging these questions into one benchmark score hides what’s driving it. BenchMIRT gives researchers a way to separate those threads, and it’s the kind of measurement hygiene the field has needed since benchmarks became the de facto release currency.
