Longformer Cuts Transformer Memory from Quadratic to Linear for Long Docs

The quadratic memory cost of self-attention has been the single biggest obstacle to running transformers on long documents. Every token attends to every other token, so a 4,000-word input doesn’t just cost four times as much as a 1,000-word one — it costs sixteen times as much. Longformer, from Iz Beltagy, Matthew E. Peters, and Arman Cohan at Allen AI, attacks that bottleneck directly by swapping standard self-attention for a mix of local windowed attention and selective global attention.

What makes Longformer different from earlier long-range efforts like Transformer-XL or Reformer is a design choice that sounds almost mundane but matters enormously in practice: it’s a drop-in replacement for standard self-attention. That means you can take a pre-trained BERT or RoBERTa checkpoint, swap in Longformer’s attention pattern, and continue pre-training or fine-tuning on long sequences without starting from scratch. Previous architectures required building models around their constraints from the beginning. Longformer doesn’t.

For autoregressive language modeling, the paper introduces dilated windowed self-attention, where the window size grows through the layers. Memory consumption drops from O(n²) to O(n·w), with n being sequence length and w the window size — linear scaling when w stays small relative to n. The dilated pattern beat plain windowed attention on language modeling benchmarks and also outperformed Transformer-XL and adaptive span attention on downstream tasks. For encoder pre-training, Longformer combines local windowed attention with global attention on a handful of task-motivated tokens, like the CLS token or question tokens in QA. For sequence-to-sequence models, only the encoder gets the long-range treatment, since decoders rarely process massive inputs.

Reading this alongside the broader 2020 wave of efficient transformer papers — Reformer, Linformer, Performer, and the Long Range Arena benchmark — one thing stands out: Longformer’s bet on backward compatibility has aged better than most. The field has largely converged on the idea that you shouldn’t have to rebuild your entire pre-training pipeline just to handle longer inputs. Longformer made that argument early and with unusually direct engineering evidence. The open question is whether the global attention tokens, which require manual task-specific selection, will remain a practical burden as models get pushed toward more general-purpose long-context use.