A context window is the maximum number of tokens a language model can process in one go. It includes the system prompt, conversation history, and the model's own output. Inside the model, self-attention and a key-value cache enforce this limit, acting as the model's working memory. When the window fills up, older information falls out, and the model can no longer use it.
But here is the twist: the model's performance drops even when the window is not full. If you bury a crucial detail in the middle of a long prompt, the model often ignores it. And in a long chat, the assistant might start hallucinating long before you hit the advertised token limit.
In the last episode, we saw how training and inference are separate phases. Now we will look at the inference-time machinery that decides how much of your conversation the model can actually keep in mind.
What exactly is a context window?
Imagine you are reading a long recipe on a small phone screen. You can only see a few lines at a time. To follow the recipe, you scroll up and down, but you cannot see the whole thing at once. The visible area is your context window.
A language model works the same way. It reads text in chunks called tokens, not words. A token is a small piece of text, often a word fragment. The model can only “see” the tokens that fit inside its context window. If the recipe is 10,000 tokens and the window is 4,000, the model will only read the first 4,000 tokens (or the last 4,000 if you truncate from the beginning). It will miss the baking temperature at the end.
{
"type": "stat",
"title": "Context Window at a Glance",
"caption": "A typical LLM context window compared to human reading. Illustrative values based on common API limits and average reading speeds.",
"stats": [
{
"value": "128k",
"label": "Max tokens (GPT-4 Turbo)"
},
{
"value": "~100k",
"label": "Words that fits"
},
{
"value": "~1.3",
"label": "Tokens per word"
},
{
"value": "~5 min",
"label": "Human reading time"
}
]
}
Every message you send and every word the model generates consumes tokens from the same finite window. When you ask a chatbot for a recipe, the prompt plus the model's reply might use 500 tokens. If you then ask “what was the first ingredient?”, the model needs the earlier conversation to answer. But if the total token count exceeds the window, the earliest messages get pushed out, and the model literally cannot see them anymore. This is why long chats eventually lose track of what was said at the beginning. OpenAI tokenizer
How does the model “remember” everything within the window?
Think of the model as a reader who takes notes on a scratchpad. Instead of re-reading the whole book each time they write a new sentence, they jot down key points about every previous sentence. That scratchpad is the key-value cache, or KV cache.
When you first send a prompt, the model does a full read, called the prefill phase. It processes every token and stores two vectors for each token at every layer: a key and a value. These vectors capture what the token means and how it relates to others. Once the prefill is done, the model has a complete set of notes for the entire input.
When it generates the next word, it only needs to look at its cached notes and the new word. It does not re-process the whole history. This is why generation after the first token is fast. The KV cache grows with each new token, and it lives in GPU (graphics processing unit) memory. For a model with many layers and heads, the cache can easily become larger than the model weights themselves. Attention Is All You Need
A simplified view of the process looks like this:
# Pseudocode for autoregressive decoding with KV cache (simplified)
tokens = tokenize(prompt)
logits, kv_cache = model.forward_prefill(tokens)
next_token = sample_from_logits(logits[-1])
generated = [next_token]
for _ in range(max_new_tokens - 1):
logits, kv_cache = model.forward_decode(next_token, kv_cache)
next_token = sample_from_logits(logits)
generated.append(next_token)
text = detokenize(generated)
Why does the context window have a hard limit?
The attention mechanism is the reason. Attention lets every token look at every other token to decide what is important. If you have 4,000 tokens, the model computes 16 million pairwise comparisons per layer. Double the window to 8,000 tokens, and you quadruple the comparisons. This quadratic growth quickly becomes too expensive in time and memory.
Even with the KV cache, each new token still attends to all previous tokens. So the cost per generated token is linear in the sequence length, and the memory for the cache grows linearly too. A 128,000-token context can require hundreds of gigabytes of GPU memory just for the cache. That is why every model has a maximum sequence length baked into its architecture. Attention Is All You Need
{
"type": "comparison",
"title": "Attention Cost: Short vs. Long Context",
"caption": "Computational cost of the attention mechanism scales quadratically with sequence length. Values are illustrative for a single layer.",
"before": {
"label": "4k tokens",
"points": [
"16 million comparisons",
"~0.5 GB memory",
"Fast prefill"
]
},
"after": {
"label": "128k tokens",
"points": [
"16 billion comparisons",
"~64 GB memory",
"Slow prefill"
]
}
}
In the recipe analogy, if you had to compare every ingredient with every other ingredient to understand the recipe, a longer recipe would become impossibly slow. The context window is the model's way of saying “I can only handle this many comparisons at once.”
Why do long conversations get weird?
Two phenomena cause trouble: the “lost in the middle” effect and the mismatch between advertised and training context lengths.
Models pay the most attention to the beginning and the end of the context. If you put a crucial instruction in the middle of a long prompt, the model often ignores it. In a chat, a follow-up question that relies on something said 20 turns ago (now sitting in the middle of the window) may get a wrong answer. The model simply does not give that middle region the same weight. Lost in the Middle
{
"type": "donut",
"title": "Where Attention Goes",
"caption": "Models often over attend to the beginning and end of the context, causing the 'lost in the middle' effect. Illustrative distribution based on research findings.",
"data": [
{
"label": "Beginning (primacy bias)",
"value": 40
},
{
"label": "Middle (lost information)",
"value": 20
},
{
"label": "End (recency bias)",
"value": 40
}
]
}