Token

A token is the fundamental unit of text that language models use for processing. Tokenization is the process of converting raw text into tokens. Depending on the tokenizer, a token can represent a word, a subword, or even a single character.

Key facts about tokens:

• English text averages roughly 0.75 tokens per word
• A model’s context window defines how many tokens it can consider at once
• API pricing is typically per token (input and output)
• Different languages have different token efficiency

Most modern LLMs use subword tokenization (like Byte-Pair Encoding or SentencePiece), which handles rare words by breaking them into known subword units. This balances vocabulary size with coverage across diverse text.