Stochastic networks crack sparse RL rewards via skill reuse

Deep reinforcement learning has a problem. It crushes dense-reward games like Atari but falls apart when rewards are few and far between — exactly the kind of long-horizon, real-world tasks we actually want agents to solve. A new paper proposes a framework that sidesteps this by stealing a trick from how humans learn: master a bunch of basic skills first, then remix them for harder problems.

The approach, detailed by researchers from Mila and Google Brain, uses what they call Stochastic Neural Networks (SNNs) paired with an information-theoretic regularizer during a pre-training phase. This isn’t just throwing noise into the network for the sake of it. The stochasticity, combined with a carefully chosen proxy reward that requires almost no knowledge about the downstream tasks, pushes the agent to develop a wide repertoire of distinct, interpretable skills. You get diversity without having to hand-design a curriculum. The regularizer ensures the skills don’t collapse into a single boring strategy.

Once that skill library exists, a high-level policy is trained on top of it for the actual downstream task. This two-tier hierarchy fundamentally changes the exploration problem. Instead of fumbling around a sparse reward landscape, the high-level controller picks from pre-learned behaviors that already do something useful. The paper reports this boosts learning performance uniformly across a wide range of downstream tasks — not just a cherry-picked benchmark here and there.

The real punch here is sample efficiency. Pre-training a broad span of behaviors usually requires an enormous amount of environment interaction. By making the skill-acquisition phase efficient enough to be practical, this work chips away at one of the biggest barriers to using RL outside of simulation. It’s a technical step, sure, but one that nudges hierarchical RL from an elegant idea toward something you might actually deploy.