Curiosity finally cracks Montezuma's Revenge, a game that broke AI

There’s a reason reinforcement learning researchers have been obsessed with a dusty 1984 Atari game. Montezuma’s Revenge was the one that humiliated DeepMind’s original DQN back in 2013 — scoring a flat zero percent of the average human while mastering almost everything else thrown at it. Sparse rewards and brutal platforming meant simple exploration strategies got you nowhere. Now OpenAI has developed a curiosity-driven system that doesn’t just stumble through a few rooms. It beats the entire first level.

The method is called Random Network Distillation, or RND, and it works by turning unfamiliarity into a reward. The agent carries a neural network that tries to predict the output of another, fixed random network based on what it sees. When it encounters something new — a skull, a ladder, a key it hasn’t figured out yet — the prediction error spikes. That error becomes a bonus, pushing the agent toward the unknown without anyone having to program what ‘interesting’ looks like. It’s an elegant way to bottle curiosity.

The numbers are what make this a genuine breakthrough, not an incremental bump. Previous state-of-the-art efforts using count-based bonuses explored 15 rooms and averaged around 3,700 points. RND’s large-scale experiment, running across 1,024 parallel workers, hit a mean return of 10,000 points and discovered between 20 and 22 rooms across multiple runs. One extended experiment went even further, achieving a best return of 17,500 points, clearing all 24 rooms and passing the first level. That’s above human average performance, and it was done without peeking at the emulator’s RAM or cribbing from human demonstration data.

What’s particularly striking is the philosophical shift this represents. The team, which collaborated with UC Berkeley researchers, frames this as a move toward agents that can get competent in almost any environment without hand-crafted reward functions. When the only signal you need is ‘predict this random network,’ you don’t have to design a new incentive structure for every new game or simulation. I’m curious how well this generalizes beyond the pixel-perfect chaos of Atari to messier, real-world tasks where novelty isn’t always correlated with progress. Staring at a static wall of noise would generate endless curiosity but zero achievement.