The team behind DecodingTrust, the NeurIPS 2023 Outstanding Paper winner, just released a public LLM Safety Leaderboard. It’s built on the Hugging Face leaderboard template, which means anyone can submit a model and see how it stacks up on eight trustworthiness dimensions: toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness against adversarial demonstrations, privacy, machine ethics, and fairness.
The evaluation isn’t a simple chatbot vibe check. The researchers built custom red-teaming algorithms for each perspective. For toxicity, they generate challenging prompts and pair them with 33 system prompts designed to trip up models through role-play, task reformulation, and respond-as-program scenarios. For privacy, they actively try to extract sensitive data like email addresses and credit card numbers. For ethics, they use the ETHICS and Jiminy Cricket datasets to build jailbreaking prompts. The stereotype bias test covers 24 demographic groups across 16 topics, prompting each model five times and averaging the bias scores.
Submitting a model is straightforward: convert weights to safetensors, make sure the model loads via standard AutoClasses, and then use the ‘Submit here!’ panel on the leaderboard. One notable limitation: models requiring use_remote_code=True aren’t supported yet, though the team says they’re working on it. That’s a real constraint for the growing number of custom-architecture models, but it keeps the evaluation pipeline clean and reproducible.
This matters because most public leaderboards measure capability—can the model solve math problems, write code, pass exams. Safety has been harder to benchmark consistently, especially across both open and closed models. DecodingTrust tried to fix that in 2023 with a unified framework, and now the leaderboard makes it accessible beyond the original paper. The fact that it won an Outstanding Paper Award at NeurIPS suggests the methodology has some academic weight behind it, not just marketing. The open question is whether model developers will actually submit their models voluntarily, or whether the leaderboard will mostly fill up with submissions from researchers who already care about safety. That self-selection bias is a problem every public benchmark faces, and this one will be no different.
