Unordered collection of legal files
AI Models & Assistants / Knowledge library
Context Window
The token span a model can process in one request
Why learn Context Window?
Larger windows allow more source material but increase cost and do not guarantee effective use of every token. Selection, ordering, and retrieval remain important.
Where it came from
A context window bounds the tokens available to a model for a forward pass, including instructions, conversation, retrieved material, and generated output according to the serving API's accounting.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Define the window: What unit normally limits a language model context window?
- Budget a request: A service has a shared input-plus-output token limit. What must the application reserve?
- Handle excess context: A conversation exceeds the available window. Which response preserves relevance best?
- Avoid a false assumption: A model accepts 100k tokens. What does that guarantee?
- Evaluate long-context use: How should a long-context application be tested?
Documentation and sources
- Lost in the Middle (arxiv.org)
Connect the ideas
Explore next
Real World Example
Legal Discovery Context Optimization
Corporate legal departments managing massive volumes of discovery documents.
The business problem
Legal teams often feed hundreds of case files into a single model request to find specific evidence. However, the model frequently fails to retrieve critical facts buried in the middle of these long documents, leading to incomplete legal summaries and missed discovery items.
How this topic solves it
By leveraging the model's context window, we implement a strategic document ordering protocol. Instead of appending files randomly, we place the most high-priority, high-probability documents at the very beginning and the very end of the prompt sequence, effectively sandwiching less critical information in the middle to mitigate retrieval degradation.
- Categorize all discovery documents by relevance score using metadata analysis.
- Sort the document list to place high-relevance files at the start and end of the input sequence.
- Construct the prompt by concatenating the sorted documents into a single text stream.
- Submit the structured prompt to the model's context window for processing.
- Verify the extraction of key facts to ensure the model successfully accessed the prioritized segments.
This diagram illustrates how documents are prioritized and arranged within the context window to maximize retrieval accuracy.
Assigns priority to documents
High-priority files at ends, low in middle
The input span processed by the LLM
- Document Pool → Input → Relevance Scorer
- Relevance Scorer → Sorts → Ordered Sequence
- Ordered Sequence → Injects → Model Context
Expected business outcome
Improved consistency in identifying key evidence across large document sets by aligning input structure with model retrieval tendencies.
Limitations and checks
The primary limitation is the 'Lost in the Middle' phenomenon where information density in the center of the context window remains prone to retrieval failure. To check for this, developers should perform periodic spot-checks on middle-positioned documents to ensure critical data is not being ignored.