Most language model benchmarks are academic exercises—clean datasets, tidy metrics, and very little resemblance to what companies actually need. The new Enterprise Scenarios Leaderboard takes a different approach. Built around six tasks that came directly from conversations with companies using LLMs in production, it evaluates models on financial question-answering, legal confidentiality reasoning, creative writing, customer support dialogue, toxicity, and enterprise PII handling.
Two of the six datasets—FinanceBench and Legal Confidentiality—are open source. The remaining four are closed, a deliberate choice aimed at preventing the test-set contamination that has plagued other leaderboards. There’s been a growing pattern of teams fine-tuning models on benchmark test sets to climb rankings artificially, and keeping most of the evaluation data private is a straightforward countermeasure. A validation set is released for the closed tasks so users can still get a feel for what’s being measured.
Evaluation methods vary by task. FinanceBench uses 150 prompts and asks models to answer questions given retrieved document context, with gpt-3.5 acting as a few-shot evaluator to check whether generated answers match the labels. Legal Confidentiality pulls 100 prompts from LegalBench and measures exact-match accuracy on yes/no responses. Creative Writing leans on the EnDEX model, trained on 80,000 Reddit engagement samples, to score how engaging a story is. Customer Support Dialogue uses gpt-3.5 again to judge whether responses are actually helpful and relevant to the product in question.
The inclusion of creative writing alongside enterprise tasks like PII detection signals something worth paying attention to: companies don’t just need models that answer questions correctly. They need models that can write convincingly, handle sensitive data appropriately, and stay on-topic in multi-turn conversations. That’s a much messier evaluation problem than an academic benchmark. Whether the gpt-3.5-based evaluators introduce their own biases—an open question in the evaluation community—remains to be seen, but the leaderboard’s focus on production-style tasks is a meaningful departure from the status quo. If the closed datasets hold up against gaming attempts, this could become a more trustworthy reference point for teams trying to decide which model to deploy.
