The Open LLM Leaderboard added DROP three weeks ago, and the results looked bizarre. Most pretrained models scored below 10 on f1, with no correlation to their overall performance. Falcon-180B and Mistral-7B underperformed expectations, while Yi-34B and Tigerbot-70B looked fine. Something was clearly wrong with the eval, not the models.
The first culprit: normalization. The scoring pipeline splits answers on separators, strips punctuation, and homogenizes numbers by casting them to floats. But if a number is followed by any whitespace other than a plain space — say, a line return — the cast fails. A model generating “10\n\nPassage: The 2011 census…” would have its answer normalized to the string “10 passage 2011.0 census recorded population of 1001360.0” instead of “10.0”. The bag-of-words comparison then sees zero overlap with the gold answer. Correct predictions were being scored as wrong.
That was only part of the problem. Zeno’s team dug into five representative models and found two more troubling patterns: not a single model got a floating-point answer right, and high-quality models that generated longer, prompt-faithful answers actually scored worse. Both issues trace to the same root cause: using the period as the end-of-generation stop token. Floating-point answers got cut off mid-number, and better models would generate “Answer\n\nPlausible prompt for the next question” before hitting that first period.
Swapping the stop token to a newline character fixes the correlation. Splitting generations on the first \n and recomputing scores brought DROP results in line with overall model performance. It’s an approximation — already-truncated floating-point answers stay broken — but it doesn’t advantage any model unfairly. The leaderboard team chose to publish this analysis rather than silently rerun everything, which is the right call for a benchmark that thousands of practitioners rely on to compare open models.
