Curiosity alone beats hand-crafted rewards in 54 games

Reinforcement learning has an obsession with rewards. For years, the playbook has been the same: carefully design a dense reward function, tune it, watch the agent exploit it in ways you never intended, then redesign it again. It’s tedious, brittle, and doesn’t scale. A new large-scale study from a team at OpenAI and UC Berkeley suggests we might be overthinking this whole exercise.

The researchers ran a pure curiosity-driven agent — one that receives zero extrinsic rewards from the environment — across 54 standard benchmarks, including the notoriously difficult Atari game suite. The agent is motivated solely by prediction error: it seeks out states where its internal model of the world fails, treating surprise itself as the reward signal. The results are striking. In many environments, the intrinsic curiosity objective aligns so well with the game’s actual goals that the agent learns competent, sometimes expert-level play without ever being told what “winning” means.

They also poked at what makes the prediction engine tick. One provocative finding: random features work just fine for many popular RL benchmarks. You don’t always need a carefully learned representation to generate useful curiosity signals — a randomly projected feature space creates enough variation in prediction error to drive exploration. But the story flips when you care about generalization. In Super Mario Bros., where the agent faces novel level layouts, learned features pulled ahead. Randomness can get you moving, but it doesn’t transfer.

Not everything sparkles, though. The paper is upfront about where curiosity crumbles — stochastic environments, where the world itself is noisy and unpredictable, become a nightmare. An agent chasing prediction error in a slot machine doesn’t learn to pull the lever; it just stares at the flashing lights forever, mesmerized by randomness it can never fully model. It’s a fundamental limitation that won’t be solved by better feature engineering alone. The full set of gameplay videos and code is available on the project site.