The same model, the same question, the same answer choices — and wildly different scores. That’s the uncomfortable finding from a new analysis by the Leaderboards and Evals team, which tested eight prompt formats across five models on MMLU subsets. Qwen1.5-7B showed the most dramatic swing, dropping from 51.2% accuracy with one format to just 22.9% with another. The information was identical. Only the formatting changed.
The implications for model rankings are hard to overstate. When Gemma-7b can beat Mistral-7B-v0.1 simply by picking the right prompt template, leaderboards start to look less like scientific measurement and more like marketing. And because almost no one publishes their exact evaluation setup, model authors have quietly been exploiting this for years — cherry-picking the format that makes their numbers look best.
Few-shot ordering adds another layer of noise. Shuffling the same examples before the prompt produced score deltas of up to three points in extended experiments. That’s not catastrophic, but it’s enough to flip rankings between closely matched models. Sclar et al.’s FormatSpread tool tries to address this by evaluating across multiple formats and calculating variance, but it’s computationally expensive.
The folks at Dottxt, the team behind the Outlines library, are proposing a different angle: instead of trying to standardize inputs, constrain the output. Structured generation — forcing the model to respond in a specific format via regex or context-free grammar — was originally designed to make LLMs easier to use programmatically. But if output constraints can reduce this prompt-format sensitivity, it could solve the consistency problem from the opposite direction. That’s the experiment they’re digging into next.
