The brute-force approach that made DeepMind’s DQN a star—running millions of trials to master a single game—has always felt a little embarrassing when you compare it to a rat learning a maze in minutes. That gap between biological and artificial learners is exactly what researchers from OpenAI and UC Berkeley take aim at with RL², a method that embeds a fast reinforcement learning algorithm inside the weights of a recurrent neural network.
The trick is clever in its simplicity. Instead of hand-crafting an algorithm that can adapt quickly, the team lets a “slow” RL algorithm—trusty old TRPO, in this case—design one. The RNN is fed standard RL inputs: observations, actions, rewards, and termination flags. Crucially, it maintains its hidden state across entire episodes of a given MDP, effectively learning an internal representation of what kind of problem it’s dealing with. When the MDP switches, the RNN’s activations reset, forcing it to learn a general procedure for exploration and exploitation rather than memorizing a specific environment.
On randomly generated multi-armed bandits and small MDPs, the results are striking. After training, RL² performs nearly as well as theoretically optimal, human-designed algorithms like Gittins indices for bandits and UCB-VI for tabular MDPs. It’s not just mimicking them, either. The paper shows the RNN learns sensible exploration strategies—like periodically re-checking arms it hasn’t pulled in a while—without anyone coding that behavior. The system even handles a vision-based navigation task, proving it scales beyond toy problems.
I’m not ready to declare the sample-efficiency problem solved. The training process itself is still glacially slow, and the gap on harder MDPs remains significant. But reframing meta-learning as just another RL problem—one solved by a recurrent net—is the kind of conceptual shift that sticks. The real question is whether this approach can eventually train on a distribution of tasks broad enough to match the priors a rat is born with.
