Soft Q-learning and policy gradients are secretly the same

Reinforcement learning has a weird schism at its heart. On one side, you’ve got policy gradient methods like A3C, which directly optimize a policy. On the other, Q-learning methods like DQN learn a value function and derive a policy from it. The two camps rarely talk, and when they do, it’s usually to point out the other’s flaws. But a new paper from Schulman, Chen, and Abbeel suggests that divide is largely an illusion—at least when you add a bit of entropy regularization.

The core finding is elegantly simple: in the entropy-regularized setting, “soft” Q-learning is exactly equivalent to a policy gradient method. The math shows that the gradient of the soft Q-learning objective and the policy gradient are identical. This isn’t just a theoretical curiosity. It provides a partial explanation for a longstanding puzzle in deep RL: why does Q-learning work at all when its value estimates are often wildly inaccurate? If Q-learning is secretly implementing a policy gradient update under the hood, the quality of the value estimates matters far less than everyone assumed.

The researchers also draw a direct line between Q-learning and natural policy gradients, the more sophisticated cousin of vanilla policy gradients. This connection suggests that the success of both families of algorithms might stem from the same underlying principles. To prove the equivalence isn’t just textbook theory, they built a Q-learning agent that closely mirrors A3C’s learning dynamics—no target network, no epsilon-greedy exploration schedule required. On the Atari benchmark, the entropy-regularized versions of both Q-learning and policy gradients performed as well as or slightly better than their standard counterparts.

What’s genuinely surprising here is how long this connection remained hidden. The paper doesn’t introduce a new algorithm so much as it reframes how we should think about the ones we already have. If Q-learning and policy gradients are two sides of the same coin, the field’s tribal distinctions start to look a bit silly. The real question now is whether this equivalence holds beyond the entropy-regularized case—or if regularization isn’t just a trick, but the thing that makes deep RL coherent in the first place.