Deep linear networks are secretly nonlinear, and now we can train them

The assumption is so foundational it’s practically dogma: stack a bunch of linear layers without a nonlinearity like ReLU between them, and you’ve just built a very expensive single linear layer. Mathematically, they’re equivalent. But computers don’t run on math — they run on floating-point arithmetic, and at the extreme edges of that representation, things get weird. A team of researchers has now shown that the quirks of the IEEE float32 standard, specifically the way numbers underflow to zero near the 1e-38 scale, create an implicit nonlinearity that deep linear networks can exploit to perform genuinely nonlinear computation.

The trick is that near the smallest representable normal number, the gap between values becomes enormous compared to the gap between zero and that minimum. Addition stops behaving the way you’d expect — associative properties break down entirely. The authors provide stark examples where (a + b) + c gives a different result than a + (b + c). This isn’t a rounding error in the usual sense; the fundamental operation becomes nonlinear at this scale. Modern differentiation libraries are completely blind to these effects, which means standard backpropagation can’t leverage them. The team turned to evolution strategies instead, a gradient estimation method that doesn’t require symbolic differentiation.

When they pushed a deep linear network’s activations into float32’s nonlinear underflow range and trained it with evolution strategies on MNIST, the results were striking. A linear network trained via backpropagation reached 92% test accuracy. The same architecture, trained with ES to exploit floating-point nonlinearities, hit 96.7% test accuracy and over 99% training accuracy. Those extra percentage points come entirely from computation the differentiation library can’t see. The researchers note that TensorFlow compounds this behavior by building its primitives with denormals disabled — the ftz=true flag is set — meaning any non-matrix-multiply operation carries an implicit nonlinearity when computation operates near that 1e-38 threshold.

I’d be curious to see how far this scales. MNIST is a toy by modern standards, and deliberately forcing activations into underflow territory feels more like a clever exploit than a practical training strategy. The real question is whether this property can be harnessed without the brittle precision constraints — or whether it points to something more fundamental about how we should think about numerical precision in large models. The authors suggest recurrent networks and language modeling as next targets. If nothing else, it’s a reminder that the abstraction layer between mathematics and silicon is thinner than we like to pretend.