A new framework called POLO — plan online, learn offline — shows how simulated robots can master genuinely difficult physical tasks, including humanoid locomotion and dexterous in-hand manipulation, with the equivalent of just a few minutes of real-world experience. The approach, which builds on the interplay between local trajectory optimization and global value function learning, tackles a persistent problem in robotics: how to act continuously in the world while simultaneously learning from those actions without needing prohibitive amounts of training data.
The core insight is a kind of virtuous cycle. Local model-based control handles the immediate demands of motion — like keeping a bipedal robot upright — even when the agent’s broader understanding of the task, captured by a value function, is still riddled with approximation errors. That same trajectory optimization doesn’t just execute; it generates high-quality training data for the value function during offline periods, which in turn reduces the planning horizon needed online. Approximate value functions let the planner see past myopic local solutions, making the whole system more strategic over time.
Exploration is where this gets particularly clever. Instead of random flailing, the trajectory optimizer performs temporally coordinated exploration — essentially planning sequences of actions specifically designed to reduce uncertainty in the value function approximation. That targeted curiosity means the system learns what it actually needs to know, fast. The result is stable, accelerated learning across the board.
The simulated results speak for themselves. Getting a humanoid to walk and manipulate objects dexterously has historically required enormous compute budgets. Cutting that down to minutes of equivalent experience suggests the POLO framework isn’t a theoretical curiosity. It’s a practical path toward robots that learn on the job without spending years doing it.
