Context Window

A context window is the total number of tokens a language model can handle in a single forward pass. It determines how much text the model can “see” at once, which affects its ability to process long documents, maintain conversation history, and understand extended contexts.

Notable context window sizes:

• GPT-3.5: 4K tokens (~3,000 words)
• GPT-4 Turbo: 128K tokens (~96,000 words)
• Claude 3 Opus: 200K tokens (~150,000 words)
• Gemini 1.5 Pro: Up to 2M tokens

Larger context windows enable processing entire books, long codebases, and extended conversations. However, longer contexts increase computational cost and can sometimes reduce output quality due to the “lost in the middle” effect, where models perform worse on information in the middle of their context.