Running Locally: What Actually Leaves Your Machine
The strongest argument for local models is that your prompts never leave your machine — and that’s true of inference itself. It is not automatically true of everything around it. Model downloads, runtime update checks, optional telemetry, front-end integrations, and any tools you have wired in all generate traffic, and a couple of them can carry content. Worth knowing the real picture rather than assuming a blanket guarantee.
None of this undermines the case for local inference. It just means “private” is a property of a configuration you can verify, not of the word “local.”
What genuinely stays put
The inference itself. The weights are on your disk and the computation is on your hardware. Your prompt, the model’s response, and the context you supplied are processed in memory on your machine. No amount of misconfiguration sends a prompt to a model vendor, because there’s no vendor in the request path at all. This is the guarantee, and it’s a real one.
Whatever you feed it. Documents, code, and files you include in a prompt are read locally. For work with confidentiality requirements this is the whole point, and it holds.
What does go out
Model downloads. Fetching weights is a large download from a model host or a runtime’s registry, and it reveals which models you’re interested in and when. Harmless for most people, worth noting in a locked-down environment. It’s also the one unavoidable network event — though only once per model, and models can be copied between machines rather than re-downloaded.
Update and version checks. Runtimes commonly check whether a newer version exists, and desktop applications often check for library or catalogue updates. This is metadata, not content, but it is periodic outbound traffic. Most tools let you disable it.
Telemetry, where it exists. Some tools in this space collect anonymous usage data, some don’t, and policies change between versions. This is the item most worth checking yourself rather than taking anyone’s word for, including a blog post’s — read your specific tool’s current documentation and privacy statement, and look for a setting to turn it off if you’d rather.
Front-end and integration traffic. This is where content can actually leave. A chat UI, an editor plugin, or an agent framework in front of your local model is separate software with its own network behaviour. A tool that can also call a hosted model may fall back to one; a plugin may sync settings or history; an agent with web access will fetch pages, and what it fetches can be influenced by what’s in your prompt. The local model doesn’t leak — the scaffolding around it might.
Anything you deliberately wired in. Web search, retrieval from a hosted service, an API call in a tool definition. Obvious once stated, easy to forget you configured.
Where your prompts are stored, which is a separate question
Data can be exposed without going anywhere. Local setups accumulate content on disk:
- Chat history. Desktop applications and self-hosted front-ends keep conversations, usually in a local database, usually unencrypted. That’s a durable record of everything you’ve asked.
- Runtime logs. Depending on verbosity settings, logs may include prompt text.
- Backups. Whatever backs up your home directory now includes the above — and if that backup goes to a cloud service, your local prompts have just been uploaded by a completely unrelated process.
That last one catches people out and is worth checking once. It’s the most likely way a “local” conversation ends up on someone else’s server.
Verifying instead of assuming
You don’t have to take any of this on trust, and the checks are simple:
Watch the traffic. Run a prompt and observe your machine’s outbound connections with whatever tool you like. This directly answers the question for your setup, and it’s the only method that stays accurate as versions change.
Try it offline. Disconnect the network and use the model. Inference should work perfectly with a model already downloaded. Anything that breaks was talking to something, and now you know what.
Block outbound at the firewall. For a genuinely sensitive setup, allow the runtime network access while downloading models, then deny it. Heavy-handed and effective.
Read the settings. Look specifically for telemetry, update checks, and any “cloud” or “remote model” option in a front-end.
Threat models this changes
Confidential work documents. Local inference solves this properly. Check your front-end’s history store and your backup destination, and you’re done.
Regulated data. The offline test is the one to run, and to document. “It works with the network cable out” is a strong, demonstrable claim.
Avoiding training on your data. Local inference gives you this absolutely — there’s no provider to retain anything. The clearest advantage over the hosted case, and it stands regardless of the traffic above.
Air-gapped environments. Entirely feasible: download models on a connected machine, copy them across, disable update checks. This is a real use case that hosted models simply cannot serve, and it’s worth remembering when weighing local against cloud.
A practical takeaway
Inference is private; the ecosystem around it is a configuration you should check once. Run the offline test, look at your front-end’s settings and its history store, and find out where your backups go. Then, if you’ve made the endpoint reachable from other machines, the exposure questions in exposing a local model on your network apply too — that’s the change most likely to undo what running locally bought you.