RL agents learn to walk, then run through mazes

Reinforcement learning has a scaling problem. When a task demands thousands of individual steps, brute-forcing a solution by searching through low-level actions becomes absurdly inefficient. A team of researchers has now demonstrated a system that sidesteps this entirely by figuring out its own high-level skills—like crawling or directional walking—and chaining them together. They call it meta-learning shared hierarchies, or MLSH.

The core insight is hardly new: humans don’t think about individual muscle contractions when making breakfast. We sequence high-level actions like whisking or pouring. MLSH imports this logic into an RL agent. A master policy picks a sub-policy every N timesteps (the team used N=200), and that sub-policy executes for the full duration. What makes this different from prior hierarchical approaches is that no one hand-engineered those sub-policies. The system discovers them on its own by training across a distribution of tasks and keeping the sub-policies that let a freshly learned master policy rack up rewards fast on unseen problems. As the paper puts it, “a good hierarchy is one that quickly reaches high reward when training on unseen tasks.”

They tested it in a MuJoCo Ant environment, dropping the four-legged agent into 9 different mazes. The result: a diverse set of reusable locomotion primitives emerged entirely from interaction. The agent didn’t just memorize routes. It learned to walk and crawl in different directions, then sequenced those behaviors to solve mazes it had never seen before. A task requiring roughly 2,000 low-level actions compresses into a search over just 10 high-level choices.

The real leap here is what happens after training. Those discovered sub-policies transferred to a larger maze that exceeded anything in the original training set. The code and custom MuJoCo environments are now public, which means researchers can stress-test this approach on problems where planning over thousands of timesteps is the norm—not the exception.