Context window definition

A context window is the token budget a language model works within on a single request, covering the prompt, the conversation so far, any retrieved content, and the answer it writes back. Because that budget is finite, what you put in it is an engineering decision. Sanity Context, a hosted read-only Model Context Protocol server, serves an agent a compressed schema overview or a pre-built outline of your content first, and full entries only when the agent asks for them.

What counts toward a context window?

Everything in the request counts toward the context window, not just the question a user typed. According to Anthropic's documentation, the window covers the system prompt, every message in the conversation (including tool results, images, and documents), the tool definitions themselves, and the text the model generates in reply, including extended thinking tokens.

That last part surprises people. Output shares the same budget as input, so a request that fills the window with source material leaves less room for the answer. Vendors publish a separate max output tokens limit that caps how much a single reply can contain. OpenAI's API documentation lists GPT-5 with a 400,000 token context window and 128,000 max output tokens, which is a cap on one part of a budget the whole request shares.

A context window is also per request, not cumulative. The model does not accumulate a conversation over time. Each turn resends the accumulated history, and when that history outgrows the budget, something has to be dropped, summarized, or compacted.

How is a context window measured in tokens?

A context window is measured in tokens, which are subword chunks produced by the model's tokenizer, not in words, characters, or pages. A common word may be a single token, while a rare word, a long URL, or text in a language the tokenizer handles poorly can split into several.

Because tokenization is model-specific, the same paragraph costs a different number of tokens in different models, and folk conversion ratios between tokens and words are unreliable enough that it is better to count with the tokenizer you are actually using. Non-text inputs consume the budget too: Anthropic's documentation notes that a single request can include up to 600 images or PDF pages on its million-token models, and up to 100 on its 200,000 token models.

Published window sizes have grown quickly. When Google announced Gemini 1.5 Pro on February 15, 2024, it shipped with a 128,000 token standard window, with an experimental one million token window in private preview for a limited group. Numbers at the top end move fast, so treat any specific figure as something to check against the vendor's current model page rather than memorize.

Is a context window the same as a model's memory?

No. A context window is what the model can see during one request, while memory features in chat products are persistence layers that live outside the model. Stored facts, running summaries, and vector indexes are all held elsewhere and selectively re-injected into the context window on later turns. The model itself retains nothing between requests. Anything it appears to remember was put back in front of it.

A context window is also different from training data. Training data is what the model learned during training, fixed at its knowledge cutoff and baked into its weights. The context window is what it can reference right now, at inference time. OpenAI lists a September 30, 2024 knowledge cutoff for GPT-5, yet a model with a cutoff like that can still answer accurately about something published yesterday, provided that material is placed in the window.

One more limit gets conflated with the context window: rate limits, usually expressed as tokens per minute. Rate limits govern throughput across many requests. The context window governs the size of one.

Why does a bigger context window not always give better answers?

A bigger context window raises the ceiling on what a model can read, but models do not treat every token in that window equally well. Anthropic's own documentation names the failure mode directly: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot."

Two published results make the effect concrete. Context Rot: How Increasing Input Tokens Impacts LLM Performance, published by Hong, Troynikov, and Huber at Chroma in July 2025, evaluated 18 large language models and found that performance degrades as input length increases, "often in surprising and non-uniform ways," even on deliberately trivial tasks where task complexity was held constant. Earlier, Lost in the Middle: How Language Models Use Long Contexts by Liu and colleagues found a U-shaped curve: performance is highest when the relevant information sits near the beginning or the end of the context, and drops when it is buried in the middle.

The practical consequence is that the usable context window is smaller than the advertised one, and the gap widens as you fill it. This is why retrieval still matters on long-context models. Retrieval-augmented generation is a strategy for spending the window well, finding the few thousand relevant tokens rather than shipping the whole corpus, and a larger budget does not remove the reason for doing that.

How do you work within a context window?

Working within a context window means deciding, on every turn, which few thousand tokens deserve the space. That practice has a name, context engineering: the window is the constraint, and context engineering is the discipline of choosing what occupies it. In our experience, the useful habits are to send summaries or indexes before full documents, to place the most important material at the start or end of the prompt rather than the middle, to prune conversation history rather than let it accumulate unchecked, and to let the model request more detail instead of preloading it.

Sanity is the Content Operating System for the AI era, and its agent-facing surface is built on that last idea. Sanity Context exposes a hosted, read-only Model Context Protocol endpoint called Context MCP, which starts a conversation with an `initial_context` step: a compressed schema overview in GROQ mode, or a Knowledge Base outline in Knowledge Base mode. The outline lists every entry path with a one-line summary, and the documentation describes it as "small enough for an agent to hold in context for an entire conversation." The agent scans the outline, picks the entries that look relevant, and reads up to 20 of them in a single call.

The same budget logic shows up in how results come back. Large arrays inside returned documents are cropped individually rather than truncating the whole response, with a warning and a hint telling the agent what was left out, so it can read the full field on demand. Above 50 KB of JSON, schema exploration returns a navigator to page through instead of the raw schema. Knowledge Bases are currently in beta and opt-in, so features and limits may change before general availability.

Last updated: September 23, 2026