Informer Cuts Transformer Forecasting Costs to O(n log n) with Sparse Attention

The Informer model, the AAAI 2021 best paper, has landed in Hugging Face Transformers, bringing a much-needed efficiency upgrade to time series forecasting. If you’ve worked with the vanilla Time Series Transformer for probabilistic forecasting, you know the pain: self-attention’s quadratic complexity makes long sequence forecasting computationally brutal. Informer attacks that problem directly with two architectural changes that matter more the longer your sequences get.

First, there’s ProbSparse attention. The key insight is that canonical self-attention scores follow a long-tail distribution, where only a handful of “active” queries actually drive the attention output. The rest—the “lazy” queries—produce dot-products that generate trivial attention and mostly align with a uniform distribution anyway. Informer identifies the active queries using a sparsity measurement based on KL divergence from uniform, then builds a reduced query matrix Q-bar that selects only the top u queries, where u is controlled by a sampling factor c. That alone drops the attention computation from O(n²) to O(n log n).

The second fix targets the memory bottleneck when stacking layers. Vanilla Transformers need O(n²) memory across encoder and decoder stacks, which gets ugly fast with long sequences. Informer’s Distilling operation halves the input size between layers, slicing memory usage down to O((2 - ε) log n) and letting you go deeper without blowing up your hardware budget. The authors frame this as solving the LSTF problem—long sequence time-series forecasting—the same challenge that motivated Longformer and Sparse Transformer on the NLP side.

For the multivariate probabilistic forecasting use case, the modeling setup barely changes from the univariate version. The model still ingests a sequence of vectors; what shifts is the output side. Since modeling a full joint distribution over high-dimensional data gets expensive quickly, the approach here falls back to independent diagonal emissions across the supported distribution families. It’s an approximation, sure, but a pragmatic one—and it means you can plug Informer into existing probabilistic forecasting pipelines without redesigning everything. Given that this same architecture is now available as a drop-in within Hugging Face Transformers, teams already using the Time Series Transformer have a low-friction path to test whether sparse attention actually helps their specific forecasting workloads.