Parameter noise cuts RL training time in half on HalfCheetah

Throwing carefully calibrated noise at a neural network’s weights, rather than its actions, can dramatically speed up how fast robots learn to walk, run, and make decisions. OpenAI’s latest research shows that this technique, called parameter space noise, doubled the score of a reinforcement learning agent on the HalfCheetah Gym environment after just 20 episodes—hitting around 3,000 compared to 1,500 using traditional action noise.

This isn’t just random twiddling. Parameter noise injects adaptive randomness directly into the policy’s neural network parameters, making an agent’s exploration consistent across multiple timesteps instead of the jittery, moment-to-moment unpredictability you get from action space noise. The approach sits in a sweet spot between evolution strategies, which tweak parameters but don’t learn from individual actions during a rollout, and standard deep RL algorithms like TRPO and DDPG that only add noise at the action level.

Making this work reliably meant solving three practical headaches. Different network layers have wildly different sensitivities to perturbation, and those sensitivities shift as training progresses. Picking the right noise scale is also deeply unintuitive when you’re working in parameter space. The team used layer normalization to keep perturbed layer outputs within a sane distribution, then built an adaptive scheme that measures the perturbation’s effect in action space—a much more interpretable domain—and adjusts accordingly.

OpenAI is releasing baseline code that bakes this technique into DQN, Double DQN, Dueling DQN, Dueling Double DQN, and DDPG. An interesting footnote: early experiments required a separate policy head to prevent DQN’s Q-function perturbations from causing the agent to get stuck repeating the same action. Later testing revealed that improvement was an artifact of how noise rescaling was handled, not the extra head. Dropping it made the algorithm simpler, cheaper, and equally performant. It’s a blunt reminder that RL algorithms can fail in subtle, silent ways—and that clean code often beats clever architecture.