The Context Window

Every transformer model has a context window — a fixed maximum number of tokens it can process in a single forward pass. Tokens are the basic units text is split into before a model reads it: roughly speaking, one token is about four characters of English, so 1,000 tokens is approximately 750 words. The context window sets an absolute ceiling on everything the model can see and reason about at once. It cannot attend to anything outside it.

Context window sizes have grown dramatically. Early GPT-2 had a 1,024-token limit. GPT-3 extended this to 4,096. GPT-4 Turbo reached 128,000 tokens — roughly 100,000 words, or the length of a full novel. Claude 3 supports up to 200,000 tokens. Some research models push into the millions. Despite this growth, the window remains finite, and understanding what goes inside it is critical for agent design.

What Lives Inside the Context Window

The context window is not a private notebook — it is the entire visible universe of a single model call. In an agent setting, that window typically contains several distinct layers of information, all competing for the same finite space. The system prompt occupies the first layer: instructions, persona definition, rules, and any background knowledge the developer injected at agent startup. This might run from a few hundred to several thousand tokens depending on how elaborate the agent's instructions are. Conversation history follows: everything the user said and the model replied, all the way back to the start of the session — unless the agent has truncated or summarized it. This grows without bound as the session continues. Tool call records come next: the structured output of every tool the agent invoked — web search results, database rows, file contents, API responses. These can be very large. A single web page retrieved by a browsing tool might consume 10,000 tokens. Finally, the current user message and any dynamic context the agent injects — retrieved documents, fresh facts — fill the remaining space before the model generates its response.

Hard Limits and Soft Degradation

When a prompt exceeds the context window, the API returns an error and the call fails entirely. This is the hard limit. But there is a softer failure mode that happens well before the limit is reached: attention degradation. Transformer attention is theoretically uniform across the window, but in practice models perform worse when critical information is buried in the middle of a very long context. Research has called this the 'lost in the middle' problem. Instructions given at the beginning and end of a long prompt tend to be followed better than identical instructions buried thousands of tokens deep. For agents injecting large tool outputs into the middle of the context, this is a real reliability hazard — the model may appear to 'ignore' a retrieved document even though it fits within the window.

Why Context Size Is Not a Free Lunch

Larger context windows are strictly better from a capacity standpoint, but they come with costs. Transformer attention has quadratic complexity with respect to sequence length in naive implementations — doubling the context requires four times the compute. Modern techniques like FlashAttention, sliding window attention, and sparse attention reduce this, but long-context processing is still significantly more expensive per token than short-context processing. For agents this creates a direct economic tradeoff: putting everything in the context is simple and correct but expensive. Selectively retrieving only the most relevant information saves cost and often improves reliability by reducing noise. This tradeoff — capacity versus cost versus relevance — drives the architectures you will study in the rest of this module.