Algorithms, not hardware, are driving AI's 16-month doubling

The raw horsepower of GPUs gets the glory, but a new analysis from OpenAI suggests the real hero in AI’s breakneck pace is something much less tangible: algorithmic efficiency. The research finds that since 2012, the amount of computational firepower needed to train a neural network to the same performance on the ImageNet classification benchmark has been cut in half roughly every 16 months. For a concrete example, reaching the performance level of the seminal AlexNet model now takes 44 times less compute. To put that in perspective, if we’d only been riding the wave of classical hardware improvements a la Moore’s Law, we’d be looking at a mere 11x improvement over the same period.

OpenAI frames this as a way to adjust our thinking about compute, comparing it to how dollars need to be inflation-adjusted over time. A fixed amount of computational budget in 2019 simply accomplishes more. The team also observed this trend across other domains over shorter timescales. Three years after the Transformer architecture was introduced, it surpassed the older seq2seq model on an English-to-French translation task using 61 times less training compute. Similarly, a re-run of the Dota 2-playing OpenAI Five system surpassed the version that beat world champions with 5x less compute just three months later.

The researchers liken this to a “tick-tock” model of progress, borrowing from Intel’s old semiconductor playbook. The “tick” is the expensive, compute-hungry breakthrough that establishes a new capability. The “tock” is the subsequent wave of refinement that makes that same capability dramatically cheaper and more efficient to reproduce. These algorithmic gains are a force multiplier, allowing researchers to run far more experiments on the same budget, which in turn accelerates the cycle of discovery.

What’s conspicuously absent is any claim of a universal law. The team is refreshingly cautious, noting they have only a small number of data points and it’s unclear how well these trends generalize to other AI tasks. The million-dollar question they leave dangling is whether a true algorithmic equivalent to Moore’s Law exists. They suspect efficiency rates will be similar on tasks with comparable levels of investment, but the first time a capability is created—the leap from impossible to merely expensive—still resists clean measurement. For now, the evidence points to a quiet reality: clever math is outpacing brute force.