Transformer models have a notorious problem: full attention scales quadratically with sequence length, which makes processing long documents computationally brutal. BigBird, introduced in a recent paper and now available in Hugging Face Transformers, tackles this by replacing BERT’s full attention with block sparse attention, allowing sequences up to 4096 tokens at a fraction of the compute cost.
The core question BigBird asks is simple: does every token really need to attend to every other token? The answer it proposes is no. Instead, BigBird combines three attention patterns. Sliding attention lets each token attend to its immediate neighbors—think of how the word ‘available’ in a sentence depends mostly on the words right next to it. For long-range dependencies, BigBird introduces global tokens that attend to everything and are attended by everything, acting as bridges between distant parts of a sequence. The third piece, random attention, gives each query a small set of random keys to attend to, which the authors show approximates full attention surprisingly well.
This matters most for tasks like long-document summarization and question-answering with extended contexts, where BigBird has achieved state-of-the-art results. In QA specifically, context tokens need to compare against the entire question—if they only attend to nearby context tokens, filtering relevant from irrelevant information becomes much harder. Global tokens help solve that without the quadratic memory blowup.
The honest caveat, which the Hugging Face post makes explicitly, is that BigBird’s attention is an approximation of BERT’s full attention. It doesn’t try to be better—just more efficient. If you had infinite compute and time, you’d still pick BERT’s full attention. BigBird simply makes it practical to run transformer models on sequences where BERT’s memory requirements would be unbearable.
