Models & Architecture
Transformer
A neural network architecture introduced by Google in 2017 that uses self-attention mechanisms to process sequential data. It is the foundation of virtually all modern LLMs.
The Transformer is a deep learning architecture introduced in the landmark 2017 paper “Attention Is All You Need” by Vaswani et al. Unlike previous sequence-processing models (RNNs, LSTMs), transformers process all tokens in parallel using a self-attention mechanism.
Key innovations:
- Self-Attention: Allows the model to weigh the importance of different parts of the input
- Parallel Processing: Much faster training than sequential models
- Scalability: Can be scaled to hundreds of billions of parameters
Transformers enable models like GPT, BERT, T5, and virtually all modern LLMs. The architecture has also been adapted for computer vision (ViT), audio processing, and multimodal AI.
Related Terms
Get smarter about AI
The sharpest AI news, curated daily. Delivered free to your inbox.