How to Pick a Local Model for the Job You Have

There is no best local model, and any list claiming otherwise is out of date by the time you read it. What does hold up is a process: filter by what your hardware can run, shortlist two or three models built for the kind of work you’re doing, and decide between them on your own tasks rather than someone else’s leaderboard. That takes an evening and stays valid as models churn.

We’re deliberately not naming a winner here. Open-weight releases arrive constantly and the sensible choice moves with them — the method below is the durable part.

Step 1: filter by what fits

Start with the constraint that can’t be negotiated. Work out your usable memory budget and eliminate everything that doesn’t fit comfortably, using the arithmetic in how much VRAM and RAM you need.

This is a harsher filter than people expect, and that’s good news — it turns an unbounded question into a shortlist. A machine with a modest GPU is choosing among small models, full stop, and knowing that immediately saves you from downloading 40 GB to discover it crawls. Two notes:

• Filter on comfortable fit, not maximum fit. A model that technically loads but leaves no room for context will disappoint you in a way that looks like the model being bad.
• Include the quantization in the filter. The same model at a smaller quant is a different entry in your list, with different memory needs and slightly different output quality.

Step 2: shortlist by what the model was built for

Open-weight models are not interchangeable at the same size. Families are trained and tuned with intent, and that intent is usually stated plainly in the model’s own description. Broad categories worth distinguishing:

• General instruction-following — chat, drafting, summarising, rewriting. The default choice for assistant-style use.
• Code-specialised — trained heavily on source code, often with fill-in-the-middle behaviour for editor integration. Usually better at code and noticeably worse at prose than a general model of the same size.
• Reasoning-tuned — trained to work through problems in visible steps. Better on multi-step questions, but slower and more verbose, which matters more locally than in the cloud because you’re paying for every token in wall-clock time.
• Small purpose-built models — classification, extraction, embedding. If your task is “label this” rather than “write this,” a small specialised model often beats a large general one and runs in a fraction of the memory.

Read the model card. The people who trained it will tell you what it’s for, what languages it covers, and what context length it was actually trained on — which is frequently shorter than the maximum the runtime will let you set.

Step 3: decide on your own work

This is the step people skip, and it’s the only one that answers your question. Public benchmarks measure aggregate performance on standardised tasks; you care about one narrow job on your hardware.

Assemble ten to twenty prompts that look like your real work. Not clever test prompts — the actual things you’ll ask. If you want the model to summarise meeting notes, use three real sets of meeting notes. If it’s for code, use functions from your own repository. Keep them in a file so the set is reusable.

Then run the same set through each shortlisted model and compare. What to look for, roughly in order of how often it decides things:

1. Instruction adherence. Did it do the thing you asked, in the format you asked for? This separates models faster than anything else, and small models fail here first.
2. Failure style. Every model at this scale gets things wrong. Some hedge, some confidently invent. For most workflows a model that says less but stays accurate is worth more than a fluent one you have to fact-check.
3. Verbosity and stopping. Models that ramble or don’t know when to finish are exhausting to use locally, where every extra token costs you time.
4. Speed at your context length. Test with prompts the length you’ll really send. A model that’s snappy on a one-line question can feel very different with a few thousand tokens of input.

Score loosely — a three-point scale per prompt is plenty. You’re picking a winner, not publishing results.

What to do about size

The instinct is to run the largest model that fits. Resist it slightly. A smaller model leaves headroom for a longer context, responds faster, and lets you keep a second model loaded. For extraction, classification, formatting, and routine drafting, a small model that follows instructions well is usually the better tool.

Save the largest model that fits for the work that genuinely needs it, and be willing to run two: a fast small one for everyday calls and a bigger one you load when the task warrants it.

Re-run the process, don’t re-research it

Because the landscape moves, revisit this every few months — but only steps 2 and 3. Your memory budget hasn’t changed, your task hasn’t changed, and your prompt set is already written. Swapping a new candidate in is an hour’s work, and that’s the payoff for having built the set in the first place. When you’re comparing quantizations of the same model rather than different models, the narrower question in is that quant good enough is the one to ask.