What Is a Context Window: A Plain-English Definition
Every time you call an LLM API, you send a block of text and receive a response. The context window is the maximum size of that combined input-and-output block, measured in tokens. A token is roughly 0.75 words in English, so a 128,000-token context window can hold approximately 96,000 words of combined input and output. The context includes everything the model sees: your system prompt, the conversation history, any retrieved documents, the user's current message, and the model's response. All of these compete for the same fixed-size window. If your system prompt is 1,000 tokens, your conversation history is 10,000 tokens, and your retrieved documents are 30,000 tokens, you have consumed 41,000 tokens before the user's message arrives. How you manage this allocation significantly affects both product quality and cost. The context window is not a persistent memory. When a conversation ends, the context is discarded. If a user starts a new conversation, the model has no recollection of previous sessions unless you explicitly include that information in the new context. This is one of the most common sources of user frustration with AI products: the expectation of persistent memory when the underlying architecture does not provide it by default. Large context windows have enabled entirely new product architectures. It is now feasible to send an entire 50-page contract to Claude 3.5 Sonnet for analysis in a single call, something that would have been impossible with earlier models. Document analysis, long-form content review, and code repository understanding are all unlocked by large context windows.
How Context Windows Work
The context window is a physical limit of the model's architecture. Transformer models process all tokens in the context simultaneously using attention mechanisms, which means the computational cost scales quadratically with context length. A context that is twice as long requires roughly four times the computation. This is why larger context windows come with higher latency and higher cost per token. In practice, how you fill the context window matters as much as its size. Models are not equally attentive to all parts of the context. Research has shown that LLMs pay more attention to information at the beginning and end of a long context than in the middle, a phenomenon called the 'lost in the middle' problem. If you place critical instructions or key documents in the middle of a very long context, the model may effectively ignore them. Consider a concrete example. A UK consulting firm builds an AI tool to analyse client engagement reports. Each report is around 20,000 tokens. They load the full report into the context and ask the model to identify risks. This works well for individual documents. But when they try to compare two reports simultaneously by loading both into the context, they hit 40,000 tokens, and the model begins to miss references to risks mentioned only in the middle sections of the second document. The solution is to use embeddings to identify the most relevant sections of each document and retrieve only those, reducing context usage while improving retrieval focus.
Why Context Windows Matter for AI Product Development
Context window size directly determines which product architectures are viable. For products that need to reason over entire long documents, such as contract analysis, patent review, or book summarisation, a large context window is necessary. For products with extensive conversation history, such as customer support assistants that need to remember everything discussed in a long session, context window size constrains how far back the model can see. Context window size also directly determines API cost. LLM providers charge per input token, per output token, or both. Sending a 100,000-token context for every user query can make your unit economics unworkable at scale. Careful context management, sending only the most relevant information rather than everything available, is a core engineering discipline in production AI systems. For multi-turn applications, context management requires explicit strategy. Common approaches include sliding window approaches that keep the most recent N turns, summarisation approaches that compress older conversation history into a compact summary, and retrieval-based memory that stores important information from past conversations in a vector database and retrieves it when relevant. Latency is the third dimension. Processing a 200,000-token context takes significantly longer than processing a 2,000-token context. For copilot and real-time suggestion use cases where latency must be under 500 milliseconds, this means designing deliberately small, focused contexts rather than relying on large context windows as a convenience.
Common Use Cases Shaped by Context Window Size
Document analysis and long-form review are the use cases most enabled by large context windows. Legal document review, financial report analysis, and academic paper summarisation can now be done in a single LLM call without chunking, provided the document fits within the context window. This simplifies architectures significantly compared to the chunking-and-aggregation approaches required with smaller context windows. Code understanding and review benefit from large context windows. Sending an entire Python module or a set of related files to a model for review, refactoring suggestions, or documentation generation is feasible with modern context window sizes. Conversational AI products with long session lifetimes, such as coaching applications or complex support workflows, need explicit context management strategies. Rather than trusting that the model can maintain coherence over a 50-turn conversation, well-architected products periodically summarise conversation history and inject the summary into the context. Agentic workflows that accumulate observations from tool calls over many steps need to manage context carefully. Each tool call result is added to the context. Over a long agentic session, the context can fill with intermediate observations, leaving less room for fresh information. Effective agent frameworks implement context pruning strategies that remove stale intermediate outputs. For applications processing sensitive personal data in long contexts, such as customer service tools with full conversation history, GDPR data minimisation obligations are particularly relevant. Including personal data in the context because it is convenient rather than because it is necessary for the task is not compliant with the data minimisation principle.
Related Concepts You Need to Know
Tokenisation is the process by which text is split into tokens before being sent to the model. Understanding tokenisation helps you estimate how many tokens your prompts consume and manage context window usage efficiently. Different languages and scripts tokenise differently: English text averages roughly 0.75 words per token, but other languages can consume significantly more tokens for the same semantic content. Retrieval-augmented generation is the architectural pattern that addresses context window limitations for knowledge retrieval. Rather than loading all potentially relevant knowledge into the context, RAG retrieves only the most relevant documents and includes those. This makes it possible to search knowledge bases far larger than any context window while keeping individual requests efficient. Large language models are the foundation. Understanding how different models compare on context window size, cost, and latency helps you make informed architecture decisions. As of 2025, context windows of 128K-1M tokens are available across frontier models, but larger contexts cost more and have higher latency. Few-shot learning and chain-of-thought prompting both consume context window space. The more examples and reasoning steps you include in your prompt, the less space remains for the actual task content. Optimising prompt length is a practical engineering concern in production AI systems. Inference cost is directly tied to context window usage. Since providers charge per input token, context management is not just a technical concern but a unit economics concern. Building cost-conscious context assembly, which means including only what is needed rather than everything available, is essential for AI products with healthy margins.