Evolved Policy Gradients: Teaching RL Agents How to Learn

Reinforcement learning agents are notorious blank slates. Drop one into a new environment and it flails around, relying entirely on sparse external rewards to slowly figure things out. That’s wildly different from how a human picks up a violin — you already have a feel for what progress sounds like, shaped by years of motor experience and a dash of evolution. OpenAI’s new experimental method, Evolved Policy Gradients (EPG), tries to close that gap by giving agents something humans take for granted: a learned loss function that encodes a prior notion of making progress.

The core idea is clever in its structure. EPG runs two optimization loops. An inner loop uses stochastic gradient descent to train an agent on a specific task — say, moving a gripper to a target at coordinates [50, 100]. But here’s the twist: the loss function that inner loop optimizes against isn’t hand-designed. It’s proposed by an outer loop, which uses Evolution Strategies to tweak the parameters of that loss function — itself a temporal-convolutional neural network — based on the final trained policy’s performance. The outer loop isn’t chasing short-term returns. It’s optimizing for whether the agent actually succeeded after learning, which forces the loss function to effectively internalize an RL algorithm.

The results point to something genuinely interesting about generalization. In one test, EPG evolved a loss function to train ant-like agents to walk to targets on the right side of an arena. Then they froze that loss and moved the targets to the left side — a task completely outside the training distribution. The ants learned to walk left anyway. That’s the kind of qualitative shift that other metalearning approaches, like training policy initializations or recurrent policies, struggle with. Those methods tend to overfit to the specifics of training tasks. A well-designed loss function, by contrast, can be generically applicable — the same reason PPO works across Atari games and robot control.

Speed matters too. The team found EPG could train agents faster than Proximal Policy Optimization, a standard off-the-shelf method. That’s not just academic. If loss functions really are the kind of object that generalizes across substantially different tasks — and this early evidence suggests they might be — then we’re looking at a path toward agents that don’t start from zero every time. The approach builds on prior work in evolving reward functions but goes further by evolving the complete loss, meaning the system has to learn an RL algorithm internally rather than just shaping a reward signal. It’s a conceptual step toward machines that know what learning feels like.