Anything LLM

Run Your First Local LLM with Ollama

The fastest way to go from “I’ve heard you can run these things locally” to actually talking to a model on your own machine is Ollama. It bundles a model runtime, a downloader, and a small local server behind a single command, so you can pull a model and start chatting in a couple of minutes. This guide walks through that first run and a little past it — enough that you could wire a local model into your own scripts.

Ollama moves quickly, so treat exact model names and sizes here as examples and check the official docs for the current catalog.

Install Ollama

Ollama runs on macOS, Linux, and Windows. On macOS and Windows you download an installer; on Linux the one-liner from the site sets up the service:

curl -fsSL https://ollama.com/install.sh | sh

Once it’s installed, a background service (the Ollama server) listens locally on 127.0.0.1:11434. That server is what actually loads models and answers requests; the ollama command you type is a client that talks to it.

Pull and run a model

Everything starts with ollama run. Pick a small, well-known model for your first try — something in the 3B–8B parameter range downloads quickly and runs on modest hardware:

ollama run llama3.1:8b

The first time, Ollama downloads the model (a few gigabytes) and caches it. When it’s ready you get an interactive prompt. Type a question, press Enter, and the model streams a reply token by token. Type /bye to exit. Run the same command again later and it skips the download and starts instantly.

A few useful housekeeping commands:

ollama list          # models you've downloaded
ollama ps            # models currently loaded in memory
ollama pull <model>  # download without starting a chat
ollama rm <model>    # delete a model to reclaim disk

Pick a model that fits your machine

The model name usually encodes its parameter count — 8b means roughly 8 billion parameters. As a rough guide to what will run comfortably:

  • 3B–8B models run on most modern laptops, including CPU-only and Apple Silicon, and on any GPU with 8 GB of VRAM or so.
  • 13B–14B models want a GPU with 12–16 GB of VRAM, or a Mac with plenty of unified memory.
  • 30B and up start needing 24 GB+ of VRAM or serious system RAM, and get slow on CPU.

By default Ollama serves quantized versions of these models (compressed weights that trade a little quality for a much smaller memory footprint), which is why an “8B” model only needs a few gigabytes rather than tens. If a model runs but feels sluggish, it’s likely spilling out of your GPU into system RAM — step down to a smaller model or a smaller quant.

Talk to the local API

The interactive chat is convenient, but the reason Ollama is useful to developers is the HTTP API on the same local port. Nothing leaves your machine. A minimal chat request:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{ "role": "user", "content": "Explain quantization in one sentence." }],
  "stream": false
}'

You get back JSON with the model’s reply. There’s also a /api/generate endpoint for single-prompt completions. Because it’s just HTTP, you can call it from Python, Node, a shell script, or any language — no SDK required, though official client libraries exist to make it tidier.

Ollama also exposes an OpenAI-compatible endpoint at /v1, so a lot of tooling written for the OpenAI API will point at your local server if you change the base URL to http://localhost:11434/v1. That’s often the quickest way to try a local model inside an app you already use.

Customize behavior with a Modelfile

If you want a model to always behave a certain way — a fixed system prompt, a set temperature — you don’t have to repeat those settings on every call. A Modelfile captures them into a reusable named model:

FROM llama3.1:8b
SYSTEM "You are a terse assistant for a senior developer. Prefer code over prose."
PARAMETER temperature 0.4

Save that as Modelfile and build it:

ollama create terse-dev -f Modelfile
ollama run terse-dev

Now terse-dev carries that personality and settings wherever you use it. This is handy for giving a project its own tuned assistant without touching application code.

What to expect (and what not to)

A local 8B model is genuinely useful for drafting, summarizing, reformatting, code snippets, and offline Q&A. What it won’t do is match the reasoning depth of the largest frontier models — those have hundreds of billions of parameters and won’t fit on consumer hardware. Setting that expectation early saves disappointment: local models win on privacy, cost, and availability, not on being the single most capable model in existence.

Where to go next

Once you’ve got a model answering prompts, the natural next questions are hardware and formats: how much memory do I actually need, and what do all those quantization labels mean. Those are their own posts. For now, you have a model running entirely on your own machine, reachable over a local API, with no key and no bill — which is the whole point.

Because the ecosystem changes fast, always cross-check current model names, sizes, and flags against the official Ollama documentation before relying on them.

getting-startedollamalocal-vs-cloud