Here’s a problem that’s annoyed researchers for years: the L₀ norm — essentially counting how many non-zero weights a model has — is the truest measure of sparsity, but you can’t optimize it directly with gradient descent. It’s non-differentiable. Every time someone wants to prune a network, they resort to workarounds like magnitude-based pruning or L₁ regularization, which are proxies at best. A team of researchers just proposed a method that side-steps the entire issue, and the core insight is surprisingly elegant.
Instead of applying the L₀ norm to the weights themselves, they introduce a set of stochastic gates — one per weight — that randomly decide whether a weight gets used or clamped to zero. These gates flip on and off during training. The magic is in the distribution: they show that for certain gate distributions, the expected L₀ norm of the resulting model becomes smooth and differentiable with respect to the distribution’s parameters. You can now tune how sparse you want your network to be simply by adjusting a regularization coefficient.
The specific distribution they landed on is something they’re calling the “hard concrete” distribution. It starts with a binary concrete (a continuous relaxation of the Bernoulli), stretches it, and then pushes samples through a hard-sigmoid function. The result is a gate that mostly produces zeros and ones — but still allows gradients to flow during training via the distribution parameters. These gate parameters get optimized right alongside the network weights using standard SGD. No separate pruning pass. No fine-tuning stage. It just works.
What I find most compelling is the principled approach to conditional computation. By directly penalizing the expected number of active weights, the method learns which parts of the network to use for any given input. The experiments show they can prune networks aggressively during training, improving both speed and generalization. The connection to classical model selection criteria like AIC and BIC — which are special cases of L₀ regularization — gives this work a theoretical grounding that a lot of sparsity papers lack. It’s the kind of idea that makes you wonder why nobody tried it sooner.
