Quantization

Quantization is a model optimization technique that reduces the precision of weights and activations in a neural network. By using fewer bits to represent each parameter, models become smaller, faster, and more memory-efficient.

Common quantization levels:

• FP16 / BF16: Half precision, widely used in training
• INT8: 8-bit integers, common for inference
• INT4: 4-bit quantization, enables running large models on consumer hardware
• NF4: Normal Float 4, optimized for LLM weights

Quantization enables running large models (like 70B parameter LLMs) on a single consumer GPU, or deploying models on edge devices and smartphones. Techniques like GPTQ, AWQ, and GGUF have made quantization practical for production use. The accuracy loss from quantization is typically 1-5% depending on the technique and model size.