A quiet upheaval is underway in how we train artificial intelligence, and it leans on an idea most researchers had written off decades ago. OpenAI just demonstrated that evolution strategies (ES), a hill-climbing optimization method from the pre-backpropagation era, can match or beat sophisticated reinforcement learning (RL) algorithms like A3C on standard benchmarks including Atari games and MuJoCo physics tasks. The kicker? It’s dramatically simpler, faster, and scales with embarrassing ease.
The core insight is almost jarring. Standard RL trains a neural network policy by collecting thousands of gameplay episodes, then using backpropagation to nudge parameters so that actions leading to rewards become more probable. It’s a powerful but finicky process that struggles with sparse rewards and demands careful gradient management. ES tosses all of that out. It simply injects random noise into the policy’s parameters, runs a batch of episodes to see which noisy versions perform better, and nudges the central policy in that winning direction. No backprop, no value functions, no action-probability gradients. Just trial, error, and a small, calculated step.
The scaling numbers border on absurd. On a cluster of 80 machines packing 1,440 CPU cores, ES trained a 3D MuJoCo humanoid walker to a solid gait in 10 minutes. The A3C algorithm, a top-tier RL method, needs roughly 10 hours on 32 cores to do the same job. For Atari, 720 cores slashed training from a full day down to a single hour while hitting comparable performance. Because workers only need to share scalar rewards—not massive gradient vectors—the communication overhead is negligible, making it a near-perfect fit for distributed computing. OpenAI researcher Tim Salimans noted the method “resembles simple hill-climbing in a high-dimensional space based only on finite differences along a few random directions at each step.”
The discovery is part of a broader pattern of dusty ideas roaring back to life with modern hardware. Just as AlexNet revived convolutional neural networks in 2012 and Deep Q-learning reignited RL in 2013, ES is shattering the long-held belief that finite-difference methods can’t handle high-dimensional problems. It’s a humbling reminder that the bottleneck isn’t always algorithmic elegance—it’s often just raw compute, applied intelligently. The method isn’t a panacea; it still requires massive CPU farms and may be less sample-efficient on some problems. But for any team looking to train a policy without the headache of tuning backpropagation through time, the answer might be to stop calculating derivatives and start evolving.
