The industry’s obsession with massive context windows is running into reality. Sure, you can stuff an entire book into a prompt, but you’ll pay for it in API costs, sluggish response times, and a phenomenon researchers call “lost in the middle” — where models simply ignore data buried deep in a giant prompt. Working with small, deliberately managed context windows often produces better results by forcing the model to focus on what actually matters.
The simplest approach is the sliding window, which treats conversation history as a FIFO queue. As new messages arrive, the oldest ones get dropped. The payoff is absolute predictability: token usage stays flat, latency stays flat, and you never get surprised by a runaway API bill. The article includes Python code where you can adjust max_turns to see exactly how much memory gets injected into each prompt.
For RAG systems, token budgeting splits the context window into zones with hard limits — say 20% for system instructions, 20% for chat history, and 60% for retrieved documents. The retrieval loop halts as soon as it hits the budget ceiling, preventing a single bloated document from crowding out everything else. The example uses word count as a lightweight proxy for tokens, applying the common heuristic of 1 word equaling roughly 1.3 tokens.
Beyond those two, the article touches on rolling summaries (an auxiliary LLM condenses older history into a compact paragraph), prompt compression (algorithmic stripping of filler text), and observation masking (filtering tool outputs). Rolling summaries preserve long-term memory without token bloat but add API calls and latency — a tradeoff worth weighing carefully. The broader lesson here echoes what developers have learned with RAG itself: constraint isn’t a limitation, it’s a design tool.
