Context Windows and Their Limits

Lab: Run the companion CPU exercise at /labs/track1/03-cache-and-gqa/ (context limits, truncation, evidence position).

Context Windows and Their Limits

Configured limit vs. effective context

A model’s configured context limit is the maximum number of tokens the architecture and tokenizer are designed to accept—often advertised as “8K,” “32K,” or “128K.” This is a hard ceiling set by the model’s position embeddings, attention implementation, and serving configuration.

The effective context is the amount of those tokens that are actually available for your use after accounting for what the model reserves or implicitly consumes. The gap between the two can surprise readers who assume prompt_tokens + output_tokens ≤ context_limit is sufficient.

Token-budget example

Suppose you are working with a model configured for an 8,192-token context window. You want the model to generate a response of up to 512 tokens. A naive check would verify:

prompt_tokens + 512 ≤ 8,192

If your user prompt is 7,200 tokens, the check passes: 7,200 + 512 = 7,712 ≤ 8,192. But the effective context may be smaller for two reasons:

1. Reserved output slots. Some implementations reserve a portion of the context for the model’s own generated tokens, meaning the prompt can only use 8,192 − 512 = 7,680 tokens before truncation kicks in. In this case, 7,200 tokens still fit, but a 7,700-token prompt would not.

2. System or instruction prompts. If your setup always includes a 1,000-token system prompt, the available prompt budget drops to 8,192 − 512 − 1,000 = 6,680 tokens. A 7,200-token user prompt no longer fits: it exceeds the budget by 520 tokens and must be truncated.

The practical takeaway: always subtract your reserved output tokens and any always-present prefix tokens from the configured limit before comparing against your prompt length.

Truncation

When a prompt exceeds the effective context, the serving system must truncate it. Common strategies include:

• Head truncation: Remove the earliest tokens, keeping the most recent (latest) tokens. This preserves the user’s final message but may lose the beginning of a long conversation or the system instruction.
• Tail truncation: Remove the most recent tokens, keeping the earliest. This preserves the system prompt and opening of a conversation but discards the user’s latest input.
• Middle-cut truncation: Keep the first N tokens and the last M tokens, discarding the middle. This can preserve both the system instruction and the latest user message, but loses the middle context.

Each strategy has different implications for downstream task quality, and the “best” choice depends on what information is most critical for the task at hand.

Model positional limits

Even within the configured context, not all positions are equally informative. Rotary Positional Embeddings (RoPE) and learned position embeddings give each token a positional identity, but attention scores vary with positional distance. Empirical work has shown that models tend to attend more strongly to information at the beginning and end of the context, with a relative dip in the middle.

Important: This observation is not a universal cutoff. Models can and do answer correctly on middle-positioned evidence when the task requires it. The “dip” is a tendency, not a hard rule. Do not cite it as a guaranteed boundary.

For any given model, the exact shape of positional effectiveness varies. The safest practice is to test your specific model and task rather than assume a fixed position window.

Evidence-position evaluation design

A reproducible way to assess how evidence position affects answer quality is to hold the question and evidence constant while varying where the evidence appears in the prompt. Here is a minimal design:

Evidence position	Question	Evidence (constant)	Answer supported?	Notes
Start (positions 1–5)	“What does the text say about X?”	Inserted at the beginning	Yes/No
Middle (positions 10–14)	“What does the text say about X?”	Inserted in the middle	Yes/No
End (positions N–N+4)	“What does the text say about X?”	Inserted at the end	Yes/No

Procedure:

1. Fix a question and a fixed-evidence passage (e.g., a 40-token span about a specific fact).
2. Construct three prompts that are otherwise identical, differing only in where the evidence passage is placed (start, middle, end of the context window).
3. Run the model on each prompt and record whether the answer is supported by the evidence.
4. Record observed results: which positions led to correct answers, which did not.
5. Distinguish these observed results from a proposed test: do not claim that a universal “middle-position cutoff” exists; instead, report what was observed for the model and prompt length tested.

This design can be replicated with different models, evidence lengths, and question types to build a position-sensitivity profile.

Connecting context length to KV-cache memory

Increasing the configured context window directly increases KV-cache memory, because each additional token position adds a key–value pair per attention layer. The relationship is linear: doubling the context window roughly doubles the cache memory for a given sequence length (see the KV-cache memory lesson for the byte-level derivation). When you plan a larger context window, consider the memory cost on your target hardware and whether the tradeoff in potential coverage is worth the additional GPU memory.

Decision task

You are building a RAG system that must handle user prompts of up to 6,000 tokens and generate responses of up to 512 tokens. Your model has a configured context limit of 8,192 tokens.

Choose a strategy and expected evidence:

1. Use the full 8,192-token context, setting aside 512 tokens for output.

   • Expected evidence: The prompt fits within the effective context (8,192 − 512 = 7,680 ≥ 6,000). No truncation needed. Record latency and quality metrics.

2. Truncate to 7,680 tokens (head truncation), reserving 512 tokens for output.

   • Expected evidence: If the user’s prompt is sometimes >7,680 tokens, record how often truncation occurs and whether answer quality drops for questions whose evidence was in the truncated region.

3. Use a 6,480-token effective prompt budget with a 1,200-token system prompt and a 512-token output reserve (6,480 + 1,200 + 512 = 8,192).

   • Expected evidence: Measure how often prompts exceed 6,480 tokens (and by how much) and whether the larger system-prompt reserve improves answer quality for instruction-following tasks.

Record the chosen strategy, the prompt-length distribution observed in production, and the resulting answer-quality impact. Use this data to decide whether the configured context limit or the effective context should drive your sizing decisions.

Key takeaways

• The configured context limit is an architectural ceiling; the effective context is what’s actually available after reserving output tokens and any always-present prefix tokens.
• Truncation strategies (head, tail, middle-cut) affect which information is lost; choose based on task-critical information.
• Evidence position can influence answer quality, but there is no universal cutoff—test with your model and task.
• A reproducible evaluation design holds the question and evidence constant while varying evidence position to measure positional sensitivity.
• Context window size has a linear impact on KV-cache memory; larger contexts cost more GPU memory.
• Use the effective context (configured limit minus reserved output and prefix tokens) to make sizing decisions, not the advertised ceiling alone.

Related lessons

• LLM Internals & Architecture · Generation, KV cache, and contextPretrained Weights and Text Generation12 min
• LLM Internals & Architecture · Generation, KV cache, and contextPrefill and Decode: Measuring the Two Phases11 min
• LLM Internals & Architecture · Generation, KV cache, and contextKV Cache Implementation and Memory Accounting14 min