There is a default behavior that appears in almost every LLM application after a few months of iteration: the context window gets full. Not because anyone decided it should be full, but because every incremental improvement added something and nothing ever got removed.
A longer system prompt for edge cases. The full conversation history because truncating it felt risky. All ten retrieved chunks instead of the top three because more felt safer. A few examples added for format consistency. Each of these decisions made sense in isolation. Together, they produce a context window that is 80-90% full on a typical request -- and an input token bill that reflects it.
What context window stuffing actually costs
Token costs for input are lower than output on most models, but input volume is much higher. A single model call with 4,000 input tokens and 400 output tokens costs roughly 10x more on the input side than the output side per absolute token count -- and at current pricing for GPT-4o, 4,000 input tokens costs about $0.005. That's not alarming for a single call. At 100,000 calls per day, it's $500/day on input tokens alone.
The compounding effect is what makes this expensive. A context window that's 20% larger than it needs to be adds 20% to your input token bill for every request, across your entire traffic volume, indefinitely. It also degrades output quality, because models under high context load attend less precisely to instructions buried in long prompts.
What we measured in our own system
When we first started instrumenting our own LLM application with span-level tracing, we found that our typical input token count for the main generation call had grown from 920 tokens at launch to 2,340 tokens nine weeks later -- a 154% increase with no corresponding product change that explained it. The breakdown:
- System prompt: 340 tokens at launch, 890 tokens at measurement time. Instructions had been added for new edge cases without ever auditing the existing ones.
- Retrieved context: 380 tokens at launch (top-3 chunks), 780 tokens at measurement time (top-5, then top-6 after a retrieval experiment we forgot to revert).
- Conversation history: 200 tokens at launch (last 3 turns), 670 tokens at measurement (last 8 turns, changed during a session continuity improvement).
None of these changes were announced. They were the result of normal iterative development. Without per-span token tracking, we would not have known they were accumulating.
Where the quality impact shows up
The cost impact is the easy-to-measure part. The quality impact is subtler. When your context window is near capacity, several things happen:
Instruction following degrades for instructions that appear early in a long prompt. This is well-documented -- models under high context load are better at attending to recency-biased content. If your system prompt has important formatting instructions in the first 200 tokens and the total prompt is 3,000 tokens, those instructions compete with everything that follows.
Redundant retrieved content introduces noise. If you retrieve 8 chunks and 3 of them are about the same topic, the model may hedge its answer or introduce inconsistency from contradictory details across the chunks. Tighter retrieval -- fewer, more relevant chunks -- almost always produces better output than more retrieval.
Long conversation histories bring in irrelevant context. A user's question from eight turns ago that has been resolved is taking up tokens in your context window and contributing nothing except noise to the current generation.
Practical strategies for reducing context bloat
The most direct intervention is regular prompt auditing. If your system prompt has more than 500 tokens, schedule a monthly review. Delete instructions that are covered by other instructions. Remove examples whose failure modes they were added to address no longer appear in your data. Keep the prompt as short as it can be while still producing the output you need.
For retrieval, track retrieved-chunk count per request as a metric. Set a hard cap and measure output quality at different cap values. In most RAG applications, the top 3-4 chunks by relevance score outperform the top 8-10 because the additional chunks add noise without adding information.
For conversation history, consider summarizing old turns rather than appending them verbatim. A 50-token summary of turns 1-6 preserves continuity while consuming far fewer tokens than the raw text of those turns.
None of these strategies are new. What's changed is the ability to see, continuously and automatically, how much your context window is actually being used -- and to receive an alert when it starts growing for a reason you didn't intend. That visibility is what turns context window management from a periodic audit into a continuous practice.