OpenAI just open-sourced a toolkit designed to fix one of reinforcement learning’s messiest problems: nobody can agree on how to measure progress. The public beta of OpenAI Gym offers a standardized collection of environments—from classic control puzzles to Atari games and simulated 3D robots—along with a site for comparing and reproducing results. It’s a direct shot at the field’s reproducibility crisis.
Reinforcement learning is the branch of AI focused on sequential decision-making, where an agent learns to achieve a goal in an uncertain environment. It’s the tech behind DeepMind’s Atari-playing systems and AlphaGo. The problem, as OpenAI sees it, is twofold: existing open-source environment collections lack variety and are a pain to set up, and subtle, often undocumented tweaks to a reward function or action set can make comparing results across papers impossible. “Subtle differences in the problem definition…can drastically alter a task’s difficulty,” the team notes.
Gym tackles this head-on with a versioned environment system. Every task, like “Hopper-v0,” gets a version number. If the task definition ever needs to change, the version gets bumped, creating an entirely new benchmark. This means a result on Hopper-v0 will always mean the same thing, eliminating the guesswork when trying to reproduce a published study. The environments themselves span a wide difficulty curve, from simple toy text problems to complex board games like Go on a 19x19 board, played against a fixed opponent powered by the Pachi engine.
For the robotics tasks, Gym leans on the MuJoCo physics engine—proprietary software, but available with free trial licenses. The toolkit is framework-agnostic, playing nice with both TensorFlow and Theano out of the gate. For those who want to kick the tires, OpenAI has already posted a tutorial and is openly soliciting contributions from the research community. It’s a pragmatic move, but the real test will be whether the broader community actually adopts a single yardstick, or just forks it into a dozen new ones.
