What a Long Context Window Costs on Local Hardware

Setting a large context window costs you memory whether or not you fill it, and filling it costs you time on every single turn. On a hosted API those costs are someone else’s problem priced into a token rate; locally they come straight out of the VRAM you needed for weights and the seconds you spend waiting. Sizing context to what you actually use is one of the highest-value adjustments available.

Two separate costs

Keep these apart, because they have different fixes.

The allocation cost is memory. Runtimes reserve space for the KV cache — per-token intermediate state the model keeps for everything in the context window. Many reserve based on the context size you configured, not on what you’re currently using, so an ambitious setting takes its share at load time.

The usage cost is time. Every token in the context has to be processed before generation starts. A conversation that keeps growing re-sends its whole history on each turn, so the pause before each reply gets longer as the session goes on.

The memory cost is what steals VRAM from the weights and pushes layers onto the CPU — see GPU layer offload explained. The time cost is what makes a long chat progressively less pleasant.

What drives KV cache size

The cache scales with the number of tokens held, and per token it scales with properties of the model’s architecture — how many layers it has and how much attention state each keeps. So two things follow:

It’s roughly linear in context length. Doubling the context roughly doubles the cache. This makes it easy to reason about: if halving the context frees enough memory to fit your remaining layers on the GPU, that’s a straightforward trade.
It varies a lot between models of the same size. Modern architectures use attention designs that share state across attention heads specifically to shrink this cost, so a newer model may hold a much longer context in the same memory than an older one of similar parameter count. If long context is your priority, this is worth checking in the model card rather than assuming from parameter count.

For a concrete estimate, the reliable approach is empirical: load with a modest context, note the memory in use, then load with double and compare. The difference is your per-token cache cost on that model, and you can extrapolate from there. Any general figure quoted for “a 7B model” will be wrong for some 7B models.

The maximum is not the trained length

A separate trap. A model’s configuration may permit a very large context, and your runtime will let you set it, but the model may have been trained at a shorter length with an extension technique applied on top. Quality often degrades well before the stated maximum — models commonly attend less reliably to material buried in the middle of a very long input.

The practical version: a long context is a place to put information, not a guarantee the model will use all of it well. If accuracy matters, giving the model less but more relevant input usually beats giving it everything.

Model cards generally state the trained context length. That figure is more informative than the maximum.

Sizing it deliberately

Work from your actual inputs rather than from the maximum:

Estimate your real prompt length. Rough conversion for English prose: a token is a bit under a word on average, so a document’s word count divided by about three-quarters gets you in the neighbourhood. Code and unusual formatting run more tokens per word.
Add room for the answer. Output tokens go into the same window.
Add a margin, not a multiple. Some slack for a longer-than-typical input is sensible. Ten times your typical need is not.
Check what it did to memory, and whether all layers are still on the GPU. If context pushed you into partial offload, you traded a lot of speed for room you aren’t using.

For most day-to-day work — questions, drafting, short files — a modest context is plenty. Long context is a specific requirement (whole documents, large code files, long transcripts), and worth paying for when you have it.

Levers when you need more than fits

Quantize the cache. Several runtimes can store KV cache at reduced precision, buying length for some quality cost. Availability and naming differ by runtime and change between versions, so check current documentation rather than a flag you read somewhere.
Use a model with a cheaper cache. As above, architecture matters more than size here.
Step down the weights instead. A smaller quantization frees memory that context can use. If long context is the requirement, this is often the right direction — see is that quant good enough for how to check the cost.
Trim the input. Sending the relevant section of a document rather than the whole thing is faster, cheaper in memory, and frequently produces better answers.

Managing a long conversation

Because history is re-processed each turn, chats degrade in speed as they grow. Two habits help: start a new conversation when the topic changes rather than carrying an irrelevant history, and for long working sessions, paste a short summary into a fresh chat instead of continuing. You lose nothing the model was using well and you get the fast early-conversation behaviour back.

A practical takeaway

Set context to a little more than you use, then verify the whole model still sits on the GPU. Treat a large context window as a resource you’re spending rather than a capability you’ve enabled — on local hardware it’s paid for in the two currencies you have least of. If a chat that started fast has become slow, the length of its own history is the first thing to suspect; the diagnostic order covers the rest.