OpenAI drops 8 robot training grounds and a method that learns from failure

OpenAI just released a suite of eight simulated robotics environments for Gym, paired with code for Hindsight Experience Replay (HER), a reinforcement learning trick that lets algorithms learn from failure. The environments aren’t cartoonish abstractions — four use the Fetch research platform and four use the ShadowHand robot, both modeled on real hardware, and they all demand agents solve practical manipulation tasks. These aren’t the MuJoCo continuous control problems that PPO can crush in an afternoon. Those are basically solved. The new tasks are genuinely difficult, in part because they default to sparse rewards: an agent gets -1 until it hits the goal, then a 0. It’s a much more realistic signal for real-world robotics than the dense, hand-shaped rewards researchers typically lean on.

That’s where HER earns its keep. The core idea is deceptively simple. If a robot tries to slide a puck to a target and misses, a standard RL algorithm learns nothing — just a flat -1. But HER says: okay, you didn’t hit the target, but you did land somewhere. What if we pretend that spot was the goal all along? By relabeling failed attempts as successes for a different, achieved goal, the agent suddenly has a learning signal. Replay that across episodes and the system bootstraps its way toward the original objective. OpenAI found that DDPG combined with HER and sparse rewards significantly outperforms every other configuration on these tasks. DDPG with dense reward manages to learn but does worse, and vanilla DDPG mostly sits there, stuck.

The results are striking, but the release is also a challenge to the research community. OpenAI is shipping a set of requests for robotics research alongside the code, similar to their previous RFR 2.0. They’re pointing at HER and saying: this works, but there’s huge room for improvement. The sparse-reward approach is more realistic, yet it’s far from solved across all the new environments. The accompanying technical report shows the trend holds broadly — HER with sparse rewards is the clear leader — but the performance curves aren’t all victories. Some tasks resist, and that’s the point.

What makes this release feel different from the usual model drop is the hardware grounding. These aren’t abstract benchmarks. They use models of actual robots, and OpenAI says they’ve trained policies in these environments that transfer to physical machines. That’s a high bar. It also means progress on these tasks isn’t just academic. If a researcher cracks a better method than HER for these environments, there’s a plausible path to that improvement showing up in a real robot arm. The door is open for anyone who thinks they can do better than learning from failure.