Token
The basic unit of text that an LLM processes. Tokens can be words, parts of words, or individual characters. LLMs have a maximum context window measured in tokens.
A token is the fundamental unit of text that language models use for processing. Tokenization is the process of converting raw text into tokens. Depending on the tokenizer, a token can represent a word, a subword, or even a single character.
Key facts about tokens:
- English text averages roughly 0.75 tokens per word
- A model’s context window defines how many tokens it can consider at once
- API pricing is typically per token (input and output)
- Different languages have different token efficiency
Most modern LLMs use subword tokenization (like Byte-Pair Encoding or SentencePiece), which handles rare words by breaking them into known subword units. This balances vocabulary size with coverage across diverse text.