Model Load Time, Cold Starts, and Keeping a Model Warm

If your local model is slow only on the first request and fine afterwards, you aren’t measuring inference — you’re measuring the runtime reading a multi-gigabyte file into memory. That’s a cold start, it’s a completely different problem from inference speed, and the fix is about residency rather than tuning.

What loading involves

Two things happen before a model can answer: the weights are read from disk into memory, and the runtime allocates working space including the KV cache for your configured context.

The disk read dominates, and it’s bounded by storage throughput and file size. A fast NVMe drive moves a large model in seconds; a spinning disk or a network share can take a long while. This is one of the few places in local inference where storage speed matters at all — it does nothing for tokens per second, but it’s most of your cold start.

Two useful consequences:

Load time scales with the model file, so a smaller quantization loads faster as well as generating faster.
The operating system’s file cache helps. If you have RAM to spare and recently read the file, a reload may come largely from cache and be much quicker. This is why a second load often feels faster than the first even after a full unload — and why the effect disappears on a memory-constrained machine.

Why runtimes unload

Managed runtimes typically unload an idle model after a timeout. That’s deliberate and mostly correct: a resident model holds VRAM or RAM that you presumably want back for other work, and on a machine with unified memory it’s holding memory your applications want.

So there’s a genuine trade-off:

Keeping it loaded costs you memory continuously and buys instant first responses.
Letting it unload frees memory and costs a wait on the next request.

Which is right depends entirely on the workload, and the two cases are quite different.

Interactive use: keep it warm

For anything with a person waiting, a cold start is the worst-feeling delay in local LLM use, because it’s unpredictable — the same question is instant or takes many seconds depending on how long you were away. Options:

Extend the idle timeout. Most managed runtimes let you configure how long a model stays resident, either globally or per request. If you have the memory, setting this generously is the simplest fix. Configuration mechanisms and defaults change between versions, so check current documentation.

Keep the model loaded indefinitely. Some runtimes support pinning a model as resident. Reasonable on a machine dedicated to this, wasteful on your daily laptop.

Send a trivial warm-up request. A tiny prompt on a schedule, or when you open your editor, keeps the model resident without you noticing the load. Crude and effective, and it’s the portable answer when a runtime doesn’t expose residency controls.

Use a smaller model. Loads faster, so the cold start you can’t avoid costs less. If you’re switching between models often, this matters more than it looks.

Batch and automation: let it unload

For a nightly job or a script that processes a directory, a cold start is a rounding error against the total run — pay it once and let the model unload afterwards so the machine is free.

The thing to get right is client timeouts. A tool that expects a response within a couple of seconds will fail on a cold start and the error usually doesn’t say why. If you’re writing against a local endpoint, set generous timeouts and be prepared to retry — serving a local model over HTTP covers the other gaps between a local endpoint and a hosted one.

Better still: warm the model deliberately at the start of the script with a throwaway request, then run your real work against a loaded model.

Switching between models is the expensive pattern

The costliest habit is alternating between two models that don’t both fit in memory. Each switch means unloading one and loading the other, so you pay a full cold start on every alternation — and it’s easy to do accidentally, for example with an editor plugin using one model while a chat window uses another.

Ways out:

Run two small models rather than one large one, if both fit simultaneously. Two resident small models often serve better than one big one you keep swapping.
Consolidate onto one model for related tasks, even if it’s not optimal for each.
Batch your work by model — do the coding session, then the writing session, rather than interleaving.

Note that a model resident alongside another is also VRAM unavailable to either, which can quietly push you into partial offload — GPU layer offload explained.

Don’t mistake it for a performance problem

The reason this deserves its own attention is that cold starts get misdiagnosed constantly. Someone times one request, concludes the model is unusably slow, and starts changing settings — when the second request would have answered promptly. The test is trivial: ask the same thing twice. If the second is fast, stop tuning; you have a residency question, not a speed one. The rest of the diagnostic order is in why your local model is slow.

A practical takeaway

Decide whether each workload has a human waiting. If it does, keep the model warm and accept the memory cost. If it doesn’t, let it unload and make your client patient. And always benchmark on a warm model — a cold start folded into a measurement makes every comparison meaningless.