Inference

Inference is the phase where a trained AI model is deployed and used to make predictions on new, unseen data. It is distinct from training, which involves learning patterns from labeled data.

In the context of LLMs, inference involves generating text token by token, with each token depending on all previously generated tokens (autoregressive generation). This makes LLM inference computationally intensive, especially for long outputs.

Optimizing inference is a major area of focus:

• KV-Cache: Storing intermediate computations to avoid recomputation
• Speculative Decoding: Using a smaller draft model to speed up generation
• Batching: Processing multiple requests simultaneously
• Quantization: Reducing precision for faster computation

Inference costs (per-token pricing) are a primary consideration when deploying LLMs at scale. Hardware improvements (GPUs, TPUs, NPUs) and software optimization continue to drive down inference costs.