New leaderboard ranks LLMs by how much chain-of-thought actually helps

Most AI leaderboards tell you which model is smartest. A new one from the Logikon project wants to tell you something arguably more useful: which models actually get better when you ask them to think step by step.

The Open Chain of Thought Leaderboard doesn’t score absolute accuracy. It measures the delta — the accuracy gain when chain-of-thought prompting is added versus when a model answers directly using loglikelihood scoring. The formula is simple: accuracy with CoT minus accuracy without it. That difference, the creators argue, is a cleaner signal of whether a model can genuinely reason through a problem rather than just recognizing the right answer from training data.

The evaluation uses LogiQA (both the original translation and version 2.0) and the LSAT dataset from AGIEval, covering analytical reasoning, logical reasoning, and reading comprehension. These tasks were chosen deliberately: they’re generic enough to solve with commonsense knowledge, yet still hard enough for frontier models that there’s meaningful headroom for improvement. The team also points out that static benchmarks are increasingly vulnerable to contamination — a model might score well because it memorized answers, not because it understands the underlying logic. Measuring the CoT delta, they argue, is more robust to that problem because knowing an answer doesn’t guarantee you can reason about it effectively.

The pipeline currently implements two prompting strategies: “Classic” (append “Let’s think step by step”) and “Reflect” (ask the model to reflect on the problem generally before solving). Both are modular, with more sophisticated chains — involving decomposition, aggregation, or revision — slated as extensions. That’s a sensible design choice given how sprawling CoT research has become; trying to benchmark every prompting variation at once would be a fool’s errand.

The implication for developers is practical. If you’re building an agent or app where reasoning quality matters, knowing which models show the largest CoT gains could directly inform your model selection — and your prompting budget, since reasoning traces consume tokens.