Sparse Transformers crack sequences 30x longer than before

OpenAI has unveiled a new neural network architecture called the Sparse Transformer that can process sequences with tens of thousands of elements—a roughly 30x jump over what was previously practical. The breakthrough comes from rethinking the attention mechanism that made Transformers so successful. In a standard Transformer, every output element calculates a weight for every input element, creating an attention matrix that balloons in memory usage for long sequences. The Sparse Transformer instead uses sparse attention patterns where each position only computes weights from a carefully chosen subset of inputs, reducing algorithmic complexity from O(n²) to O(n√n).

The team didn’t just stop at the sparse attention trick. They also used attention matrix recomputation during backpropagation, a known technique that trades compute for memory. Combined with adjustments to operation ordering and initialization, this allowed them to train networks up to 128 layers deep—dramatically deeper than typical Transformers. On CIFAR-10 images, those deeper networks consistently outperformed shallower ones.

What’s particularly clever is how they decided which attention patterns to sparsify. By first visualizing learned attention in deep Transformers on images, they spotted that many layers naturally developed structured, sparse patterns. Some layers, however, still needed to attend across the entire image. Their solution was a two-step factored approach: strided attention for data with 2D structure (like images) and fixed attention for sequential data like text. Surprisingly, the sparse models didn’t just match full attention—they achieved lower loss.

The results back up the claims. The Sparse Transformer set new state-of-the-art density estimation scores on CIFAR-10, Enwik8, and ImageNet 64×64, using configurations ranging from 30 million to 152 million parameters. For a field where longer context windows have been a persistent bottleneck, making attention scale to tens of thousands of elements without choking on memory feels like a genuine unlock—not just another incremental benchmark bump.