Day 100: Context windows & tokenizers across model families
How much the model can see at once
The context window is the maximum number of tokens a model can attend to at once — prompt plus generated output. Exceed it and the model literally cannot see the earlier tokens. For RAG (soon), this bounds how much retrieved context you can stuff in. Different families offer different windows (4K to 128K+), and because tokenizers differ, the *same text* costs different token counts across models.
Bigger context is not free
Attention cost grows with context length, and the KV cache (Day 98) grows too — a 128K context uses far more memory and compute than 4K. 'Just use the huge context window' is a trap: it's slower, pricier, and models often attend poorly to the middle of very long contexts ('lost in the middle'). Retrieve the *relevant* few chunks (RAG) rather than dumping everything in.
Illustrative: memory/compute for the KV cache grows with context length — long context is a real cost, not a free feature.
Key terms
- Context window
- The maximum number of tokens (prompt + output) a model can attend to at once.
- Lost in the middle
- The tendency of models to attend less reliably to information in the middle of very long contexts.
- Token budget
- The finite token allowance a request must fit within the context window and cost constraints.
Why is retrieving a few relevant chunks (RAG) often better than dumping an entire catalog into a huge context window?
