Tagged “serving”
-
What a Tool Call Looks Like From the Model's Side
Local models don't run code or reach the network. Tool calling is a text protocol plus a loop you write — and small models fail at it in specific ways.
-
Serving a Local Model Over HTTP to Your Own Apps
How local LLM runtimes expose an HTTP endpoint, why the OpenAI-compatible shape matters, and how to wire your own scripts and tools into a local model.
-
Model Load Time, Cold Starts, and Keeping a Model Warm
The first request after an idle period pays to read gigabytes from disk. Why that happens, when to prevent it, and what keeping a model resident costs.
-
What Happens When Two Requests Hit Your Local Model at Once
Local runtimes handle concurrency very differently from hosted APIs. What queueing and batching cost you, and how to plan for more than one caller.
-
Running a Local LLM in a Container: What's Worth the Trouble
Containers make a local model reproducible and portable — but GPU passthrough and model storage are the two things that decide whether it's worth it.
-
Exposing a Local Model on Your Network Without Regretting It
Local runtimes bind to loopback for a reason. What changes when you make one reachable from other machines, and how to do it deliberately.
-
Running Locally: What Actually Leaves Your Machine
Inference staying local doesn't mean nothing goes out. The traffic a local LLM setup really generates, and how to verify rather than assume.