OpenAI launches retro contest to break RL's memorization habit

OpenAI is betting that classic Sega Genesis games can teach reinforcement learning algorithms something they’ve been terrible at: actually generalizing. The organization just launched the Retro Contest, a transfer learning competition that ditches the standard RL playbook of testing algorithms in the exact same environments where they were trained. Instead, contestants get a training set of Sonic the Hedgehog levels, and their algorithms are evaluated on completely custom levels they’ve never seen before. The contest runs from April 5 to June 5, and each algorithm gets just 18 hours — roughly one million timesteps — to figure out each unfamiliar level at test time.

That might sound like plenty of time for a machine, but the numbers tell a different story. Existing RL algorithms perform far worse than humans given this training budget. After two hours of practice on training levels and one hour of play on each test level, human players achieve scores significantly higher than any of the baseline algorithms, including ones that leverage transfer learning. To establish those baselines, OpenAI tested Rainbow DQN, PPO, and a random-action algorithm called JERK — the last one optimized specifically for Sonic by replaying its best action sequences more frequently as training progresses.

The results did contain one genuinely encouraging signal: pre-training PPO on the training levels and then fine-tuning on the test levels nearly doubled its performance, making it the strongest of the baseline approaches. OpenAI calls this “a large and reliable effect” — a phrase that’s measured but meaningful in a field where transfer learning success stories remain sparse. The contest is built on Gym Retro, a new platform that wraps classic video games as RL environments. This initial release includes 30 Sega Genesis titles and 62 Atari 2600 games from the Arcade Learning Environment. The Genesis was chosen deliberately — its games tend to have levels that share physics and object behaviors but differ in layout, making them natural testbeds for generalization.

OpenAI is also releasing a dataset of human gameplay recordings beating the Sonic levels, which researchers can use to expose agents to areas they might never reach by starting from scratch. Gym Retro itself is a second-generation attempt at building a large RL environment dataset, following the messy Universe project from late 2016. The key difference: Gym Retro lets you define environments through JSON files rather than C++ code, which should make integrating new games far less painful. Whether the contest actually produces algorithms that generalize well — or just reveals how far we still have to go — is the question hanging over the next two months.