OpenAI scraps A3C's async trick, says ACKTR is 10-25% pricier but far better

OpenAI’s latest Baselines release buries a quiet admission: the “asynchronous” part of A3C, the influential algorithm that helped kickstart deep reinforcement learning, probably didn’t matter. A synchronous, deterministic variant called A2C performs just as well, if not better, and makes far more efficient use of GPUs. That’s the kind of detail that makes you wonder how many other widely-cited innovations are just implementation artifacts.

The bigger news, however, is the release of ACKTR. Developed with researchers from the University of Toronto and NYU, ACKTR uses Kronecker-factored Approximate Curvature (K-FAC) to optimize both the actor and critic. The algorithm takes steps in the natural gradient direction rather than the standard gradient, which means it’s optimizing for the biggest improvement in the policy’s output distribution per unit of change, constrained by a KL-divergence limit. This prevents the kind of catastrophic policy collapse that can happen when an update shifts behavior too radically.

The practical upshot is sample efficiency that beats both TRPO and A2C, while the computational cost per update is only 10 to 25 percent higher than a vanilla gradient step. That’s a fraction of what TRPO requires with its expensive conjugate-gradient calculations. In benchmarks across 49 Atari games, ACKTR outperforms A2C, PPO, and ACER—and the hyperparameters were tuned solely on Breakout, not per-game. The algorithm also scales well with batch size because it uses each batch to estimate local curvature in the parameter space, making it a strong candidate for large-scale distributed training.

One thing to watch: the video comparisons show ACKTR-trained agents crushing Q*Bert while A2C agents flail, but that’s a single-game anecdote. The full benchmark plots will tell the real story. Still, for anyone building on policy gradient methods, the message is clear: the async in A3C was a distraction, and K-FAC is the efficiency hack worth paying attention to.