GPU (Graphics Processing Unit)
A processor with thousands of small cores designed for parallel arithmetic. Its ability to run many calculations at once made it the standard hardware for training and running neural networks.
Graphics Processing Units were built to render millions of pixels in parallel. That same architecture — many simple cores executing the same operation on different data — turns out to be close to ideal for the matrix multiplications at the heart of neural networks.
Key terms around AI hardware:
- VRAM: the memory attached to the GPU. Model weights, activations, and the KV cache must all fit; VRAM is often the binding constraint.
- Compute (FLOPS): raw arithmetic throughput, usually quoted in tera- or petaflops for AI accelerators.
- Bandwidth: how fast data moves from memory to compute. Many AI workloads are memory-bound, not compute-bound.
- Interconnects: NVLink and InfiniBand tie thousands of GPUs into one training cluster.
NVIDIA dominates AI training with its CUDA software ecosystem, though challengers include AMD’s ROCm stack and purpose-built accelerators from cloud providers and startups. GPUs are also central to inference, especially for large models and low-latency serving. Rising demand has made AI chips a strategic resource, shaping everything from data-centre construction to export controls.