Why counting states still beats complex exploration in deep RL

For years, the consensus was clear: count-based exploration works beautifully in small, tabular settings but falls apart when state spaces get large or continuous. Most states appear once, so what’s left to count? The field moved on — toward optimism bonuses, curiosity-driven intrinsic rewards, and increasingly elaborate heuristics. But a paper from Tang, Houthooft, Foote, Stooke, Chen, Duan, Schulman, De Turck, and Abbeel suggests we gave up on counting too soon.

The team found that a dead-simple generalization of classic count-based methods can reach near state-of-the-art performance on both Atari games and continuous control benchmarks. The trick? Map states to hash codes, count occurrences in a hash table, and use those counts to compute a reward bonus — exactly as theory prescribes. No learned dynamics models, no sophisticated uncertainty estimates. Just hashing and counting. On Montezuma’s Revenge, the notoriously exploration-heavy Atari game that stumps most algorithms, their method scored 4,700 points on average. That’s not quite the best published result, but it’s within striking distance — and the approach is orders of magnitude simpler.

What makes a good hash function turns out to be surprisingly intuitive. The authors identify two essential properties: appropriate granularity and task-relevant encoding. Too fine-grained and every state looks novel; too coarse and you miss meaningful distinctions. A simple locality-sensitive hash using random projections already works well. But the real insight comes from learning the hash function. When they trained an autoencoder to compress states into codes that preserve information relevant to solving the MDP, performance improved further. A domain-specific learned hash that understands what matters for the task at hand outperforms generic compression.

The implications are bracing. While the field has spent years building ever more complex intrinsic motivation systems, a method that fits on a napkin — hash, count, bonus — holds its own across a wide range of benchmarks. It doesn’t win everywhere, and the authors are careful not to overclaim. But the paper reframes the exploration problem: maybe you don’t need to model uncertainty or predict future states. Maybe you just need to count better. For practitioners who want a simple baseline that’s easy to implement and tune, this is now a compelling starting point. The code is open-source, and the hash-based bonus plugs into any standard RL algorithm without much fuss. Sometimes the old ideas just needed a new coat of paint.