What to Look For in a GPU for Local Inference

For running local models, GPUs are ranked almost entirely by one number, and it isn’t the one in the marketing. Memory capacity decides whether a model runs; memory bandwidth decides how fast it generates. Raw compute — the figure that sells gaming cards — is the least important of the three for inference.

This is a post about which properties matter, not a buying list. Specific cards and prices move constantly, and a recommendation would be stale within months. What follows should still be true when the model numbers have changed.

Capacity is a hard wall

VRAM capacity is binary in its effect. Either the model and its context fit, or layers spill to the CPU and you lose most of your speed advantage — the mechanism is in GPU layer offload explained. There is no gradual degradation and no compute figure that compensates.

This makes capacity the first filter and it makes the arithmetic simple: work out the memory footprint of the models you actually want to run, using the method in how much VRAM and RAM you need, add headroom for context, and that’s your minimum. A card with less capacity and much more compute is the wrong trade for inference, even though it may be the right trade for gaming.

Consequences worth internalising:

Each capacity tier unlocks a model class. The jump from one tier to the next isn’t a percentage improvement in speed, it’s access to models you previously couldn’t load. That’s why the tiers feel so discontinuous when you shop.
Headroom is not waste. Capacity beyond the weights buys you a longer context, which for many tasks is worth more than a bigger model.
A last-generation card with more VRAM often beats a current card with less for this one purpose. This is the most common mismatch between “good graphics card” and “good inference card.”

Bandwidth sets generation speed

Once everything fits, token generation is dominated by memory bandwidth rather than arithmetic throughput. The reason is structural: producing each token requires reading essentially the whole set of weights out of memory, and doing relatively little maths with each value. The chip spends its time waiting on memory.

Two practical implications:

Bandwidth is why a smaller model feels so much faster. Fewer weights to stream per token, directly fewer bytes moved. Dropping a model size or a quantization level improves speed for this reason, not because the maths got easier.

Memory type matters more than core count. When comparing two cards with the same capacity, the one with faster memory will generally generate faster, even if the slower one has more compute units. Bandwidth is usually on the spec sheet; it deserves the attention people give to core counts.

Prompt processing behaves differently — it is compute-bound and parallel, so raw throughput does help there. If your workload is long inputs and short outputs, compute matters more than it otherwise would. Prompt processing vs token generation covers that split.

Software support is a real constraint

A card that the runtime can’t use well is not a fast card. Before anything else, confirm that your intended runtime supports the vendor’s acceleration stack on your operating system, and how mature that support is. The dominant vendor’s tooling has the longest history and the widest support in local LLM projects; alternatives have improved considerably but the experience varies by platform and by project, and “supported” can mean anything from first-class to a build flag someone maintains occasionally.

Check the runtime’s documentation, not the GPU vendor’s. Vendor claims about AI capability frequently refer to frameworks that local inference tools don’t use.

Two GPUs, and whether it helps

Splitting a model across two cards is possible with several runtimes, and it’s the usual way people run large models on consumer hardware. What to expect:

Capacity does add up — this is the point, and it’s how a large model becomes runnable at all on consumer parts.
Speed does not add up. Layers are split across cards and executed in sequence for a single request, so you’re gaining room rather than throughput.
Mismatched cards run at the pace of the constraint, and the interconnect between them becomes a factor. Two identical cards is much less trouble than two different ones.

Worth doing if you need capacity you can’t otherwise reach. Not worth doing to make a model you already fit go faster.

The alternative worth considering

Before buying a discrete GPU, consider whether a unified-memory machine solves your problem instead. Systems where CPU and GPU share one large fast memory pool sidestep the capacity wall entirely, and the tiers available can exceed consumer graphics cards by a wide margin. The trade-off is generally lower bandwidth and less compute than a high-end discrete card, so it favours capacity over speed — see local LLMs on Apple Silicon for how that plays out.

A practical takeaway

Decide the largest model class you genuinely want to run, size its memory footprint with context headroom, and buy the cheapest well-supported card that clears that number with room to spare. Then, and only then, use bandwidth to choose between candidates at the same capacity. Buying for compute, or for a card’s reputation in other workloads, is how people end up with a fast GPU that can’t load the model they wanted.