Time series forecasting is quietly undergoing the same shift that hit NLP and computer vision years ago: the move from local, point-estimate models to global, probabilistic ones. Classical methods like ARIMA get fitted to each series individually and spit out a single number per time step. That’s fine when you have a handful of series and only need a rough guess. But real-world decision pipelines — inventory planning, energy load forecasting, capacity management — need uncertainty estimates, not just point predictions. And when you’re working with thousands of related series, training one model across all of them lets the network learn shared patterns that individual fits would miss entirely.
Hugging Face’s latest tutorial walks through using the vanilla Transformer architecture from Vaswani et al., 2017 for exactly this task. The Encoder-Decoder structure maps naturally onto forecasting: the encoder ingests a context window of historical data, while a causal-masked decoder learns to predict the subsequent window step by step. At inference time, you can sample from the learned distribution autoregressively — a process known as ancestral sampling — to generate forecasts out to whatever horizon you need. It’s the same mechanism that powers text generation in models like GPT-2, just pointed at numbers instead of tokens.
Two Transformer properties make this especially practical. First, the attention mask trick that handles padding in NLP translates directly to handling missing values in time series, which are endemic in real sensor and business data. You don’t need to impute gaps before training. Second, the context window approach sidesteps the fact that you can’t shove a series with thousands of time points into a model with quadratic attention costs. You sample windows during training instead.
The catch is that quadratic complexity still caps how much history you can look at, and Transformers are notorious for overfitting when given enough capacity. The tutorial doesn’t pretend otherwise. But the broader point stands: probabilistic forecasting is becoming table stakes, and the toolkit that made Transformers dominant in text is proving just as adaptable to temporal data. Whether vanilla Transformers beat purpose-built time series architectures remains an open question — but the gap is narrowing fast.
