Sakana AI’s PC-ALM trains 1000-layer nets within 2% of backprop

Backpropagation has long been the gold standard for training neural networks, but it demands a global, synchronized dance of forward and backward passes that biological brains simply don’t perform. This mismatch has kept researchers hunting for local-learning alternatives like predictive coding (PC). The problem? Standard PC struggles in deep, narrow networks because the credit signal fades before it reaches the input layers. Sakana AI researchers have now introduced Augmented Lagrangian Predictive Coding (PC-ALM), a method that keeps updates strictly layer-local while recovering credit signals that align closely with backpropagation.

The core innovation lies in how PC-ALM handles constraints. Instead of just penalizing mismatches between layers, it attaches a Lagrange multiplier to each layer constraint. Think of it as adding an integral term to the proportional error correction already present in PC. The team describes this as a PI controller per layer. In linear networks, they prove these multipliers converge to exact backpropagation adjoints under specific stability conditions. This theoretical grounding explains why the method works where standard PC fails: it prevents the credit signal from vanishing in deep architectures by actively accumulating prediction errors.

The empirical results are striking. On MNIST and Fashion-MNIST, the team tested residual MLPs with widths and depths ranging from 8 to 128. While standard PC accuracy dropped sharply in deep, narrow configurations, PC-ALM matched backpropagation performance across the board. In a specific reference cell with width 32 and depth 32, PC-ALM achieved 77.75% test accuracy compared to backprop’s 78.66%, significantly outperforming standard PC’s 68.13%. Even more impressive, the method trained 1000-layer residual MLPs on MNIST within roughly 2 percentage points of backpropagation after five epochs. The gradient cosine similarity to backprop jumped from 0.604 for PC to 0.909 for PC-ALM.

This isn’t just a theoretical curiosity. The researchers released an MIT-licensed JAX reference implementation that runs on CPU and reproduces these findings. While currently limited to small image benchmarks like CIFAR-10 and Tiny ImageNet, the ability to train extremely deep networks without global synchronization opens new doors for neuromorphic hardware and energy-efficient training. It suggests we might finally be closing the gap between how machines learn and how brains might actually do it.