CPU-Only Inference: When It's Genuinely Enough

Running a model entirely on the CPU is slower than running it on a GPU, and for a lot of real work that’s beside the point. If your task is small models, short answers, or anything that doesn’t need a person watching the output appear, CPU inference is a perfectly reasonable setup — and it’s the only option on plenty of machines, including cheap servers and older laptops.

The mistake is judging CPU inference by the interactive-chat experience, which is the one use case it’s worst at.

What decides CPU speed

Same principle as on a GPU, different numbers. Generation requires streaming the whole weight set out of memory for each token, so system memory bandwidth is the ceiling — not clock speed and not, past a point, core count.

That leads to a few conclusions that surprise people:

• More cores stop helping quickly. Once the memory bus is saturated, additional threads contend rather than contribute. Setting thread count above your physical core count usually makes things slower, and the best setting is often somewhat below the total.
• Memory configuration matters. Faster memory, and on multi-channel platforms actually populating the channels, affects inference speed more than a CPU generation jump does.
• Model size dominates everything. Halving the weights roughly halves the per-token work. On CPU this is the lever with by far the most leverage.
• Prompt processing is the compute-bound phase, and it’s where a CPU feels weakest relative to a GPU. Long inputs are expensive here — prompt processing vs token generation explains the asymmetry.

Sizing for CPU

Your memory pool is system RAM, which is usually far more generous than a consumer GPU’s VRAM — so capacity is rarely the binding constraint. Speed is. That inverts the usual advice: instead of “the largest model that fits,” aim for the smallest model that does your job.

Practical guidance:

• Small models are the sweet spot. Models in the low single-digit-billion parameter range, aggressively quantized, remain genuinely usable on CPU. Instruction-following at these sizes has improved a lot, and for extraction, classification, and formatting they’re often sufficient.
• Mid-size models are viable but patient work. Fine for batch jobs, wearing for chat.
• Large models will load and will be very slow. Technically possible, and occasionally the right answer if you need that specific capability and can wait. Not an interactive setup.
• Use the same sizing arithmetic, just against RAM — how much VRAM and RAM you need.
• Leave real headroom. If the model plus your applications exceed RAM, the OS swaps to disk and inference becomes unusable rather than merely slow. This is a cliff, not a slope.

Workloads that suit it

The pattern is: anything where latency doesn’t have a human attached to it.

Batch processing. Classifying a directory of files, extracting fields from records, generating summaries overnight. Total throughput matters, per-request latency doesn’t. A slow model that runs unattended for an hour does a lot of work.

Short-output tasks. Classification, routing, tagging, yes/no judgements, structured extraction. The expensive phase is generation, so tasks that generate a handful of tokens barely notice.

Non-interactive automation. A script that processes your inbox, tidies notes, or annotates data. Nobody is waiting.

Privacy-first work on modest hardware. If the reason you’re local is that the data can’t leave the machine, CPU inference at whatever speed beats not doing it — the calculus in local vs cloud shifts a long way when the requirement is absolute.

Development and testing. Building against a local endpoint doesn’t require the model to be fast, just present.

Setting it up

Nothing special is needed — the major runtimes all support CPU execution, and llama.cpp began as a CPU-focused project, which is why CPU support across the ecosystem is mature rather than an afterthought. Any of the options in choosing a local LLM runtime will work.

A few things worth doing:

• Set thread count deliberately rather than accepting a default that may assume more of the machine than you want to give it. Test a couple of values with the same prompt; the curve usually flattens or reverses before you reach your core count.
• Prefer smaller quantizations than you would on a GPU. The speed gain is proportional and the memory you save isn’t scarce, so the usual reason to keep fidelity is weaker.
• Expect thermal limits on laptops. Sustained CPU inference is a sustained full load, and it will settle to a lower clock.
• Serve it over HTTP if you’re automating — serving a local model over HTTP — since batch work is much easier to write against an endpoint than a chat window.

A practical takeaway

Don’t ask whether CPU inference is fast; ask whether your workload has a human waiting. If it doesn’t, a small quantized model on a CPU is a genuinely capable tool, and you already own the hardware. If it does, keep expectations to short answers from small models — and treat the GPU question as a speed upgrade rather than the entry ticket.