Working Within 8 GB of VRAM

An 8 GB GPU runs local models perfectly well — it just doesn’t run every model, and the difference between a good experience and a frustrating one is how deliberately you spend the memory. The main mistake is treating the number as a weights budget when it’s really a weights-plus-context budget, and the second is trying to squeeze in a model a size class too large.

Everything here applies to any tight memory budget; 8 GB is just the most common one. The same reasoning covers a 6 GB card or a laptop with a modest allocation.

What the 8 GB is actually spent on

Three claims compete for the same pool:

The weights, roughly the size of the model file on disk.
The KV cache, which grows with how much context you actually use.
Overhead — the runtime’s own allocations, plus whatever the operating system and your display are already holding.

That last one catches people out. A GPU driving your monitors has already given up some capacity before inference starts, and it isn’t a fixed amount — opening a browser with hardware acceleration takes more. Plan to leave a meaningful slice unspoken-for rather than budgeting to the last gigabyte.

The practical consequence: your usable weights budget is comfortably less than 8 GB. Treat the difference between the model file size and your real ceiling as the thing you’re managing, and use the estimates in how much VRAM and RAM you need to place a candidate model.

The size class that fits

At common 4-bit quantizations, small models in the single-digit-billion parameter range are the comfortable home for this budget — they leave room for context and overhead. Mid-size models around the low teens of billions of parameters are the boundary case: they may load, but with little left for context, which tends to produce exactly the partial-offload slowdown described in why your local model is slow.

The honest guidance is to stay one step below the largest thing that will load. A small model with room to breathe is more useful than a bigger one that spills, because spilling costs you far more than the capability difference gains you.

Large models — the dozens-of-billions class — are not an 8 GB proposition on the GPU. They can run with CPU offload or entirely in system RAM if you have enough of it, but you’re choosing a different performance regime; see CPU-only inference and when it works.

Where to spend a saved gigabyte

Once you’ve picked a model that fits with room left over, you have a genuine choice about the remainder. In rough order of how often it’s the right call:

1. Context length. For anything involving documents, code files, or long conversations, more context is usually the highest-value use of spare memory. A capable small model with generous context beats a slightly better model that can only see a fragment of your input. What a long context window costs covers the arithmetic and the levers.

2. A step up in quantization. If you’re on a very aggressive quant and have room, moving up a level recovers output quality — usually noticeably at the low end. Whether it’s worth it for your work is a question you can answer directly; see is that quant good enough.

3. Headroom. Genuinely a valid choice. Memory you haven’t committed is what lets you keep using the machine normally while a model is loaded, and it’s what absorbs the occasional long prompt without a crash.

Tactics that stretch a tight budget

Close the memory hogs. Browsers, other GPU applications, and previously-loaded models are the usual culprits. Checking what’s resident before blaming the model is a habit worth forming on a small card.
Don’t set context larger than you use. A big context allocation costs memory whether or not you fill it. Set it to what your actual prompts need.
Consider a smaller specialised model. If your task is classification, extraction, or formatting, a small purpose-trained model may do it better than a general model twice the size — and leave most of your budget unused.
Use quantized cache if your runtime offers it. Some runtimes can store the KV cache at reduced precision, which buys context length at some quality cost. Support and naming vary, so check your runtime’s current documentation rather than assuming a flag.
Accept partial offload knowingly, if you must. Running a model with a few layers on the CPU is a legitimate choice when the alternative is not running it. Just make it a decision rather than a surprise.

A practical takeaway

Pick the largest model that leaves a clear margin after weights, context, and desktop overhead — then spend what’s left on context rather than on model size. An 8 GB card handles drafting, summarising, code assistance, extraction, and everyday chat without complaint. What it won’t do is run a large model at full speed, and no amount of setting-tweaking changes that. Recognising the ceiling early is what makes the card pleasant to use.