What a Tool Call Looks Like From the Model's Side

A model running on your hardware cannot do anything except produce tokens. It cannot read a file, run a command, or fetch a URL. Every setup where a local model appears to do those things is really a loop somebody wrote around it: the model emits some text that means “call this”, your code notices, executes the actual work, and pastes the result back into the conversation. Understanding that loop is the difference between wiring up tools successfully and being confused about why a 7B model keeps hallucinating function output.

The whole mechanism, plainly

You give the model a list of callables in its context — names, descriptions, and parameter schemas. Text, at the end of the day; the runtime formats it into whatever the model was trained to expect. When the model wants one, it emits a structured blob instead of prose, typically JSON with a name and arguments. Your code parses it, actually runs the thing, and appends the result to the conversation as a new message. Then you run inference again, and now the model has the answer in context and can write a real reply.

Two things fall out of this that matter more than they sound:

The model never executes anything. It writes a request. Your loop decides whether to honour it. Every bit of what a tool can reach is a choice you made in code, which is the security property to hold onto — an untrusted prompt cannot make a local model do something you didn’t already implement.

Every tool call costs two inference passes minimum. The one that produces the call, and the one that reads the result. On a hosted API that’s two billable requests. Locally it’s two rounds of prompt processing over a context that just grew, which is a real wall-clock cost worth budgeting — and prompt processing over a long context is usually the slow half.

Tools versus attachments

There is a useful distinction that gets muddled, and it becomes concrete the moment you touch Model Context Protocol (MCP) — the emerging convention for exposing capabilities to a model over a standard interface rather than as bespoke glue per application. MCP servers can offer both model-invoked tools and host-attached content, and the two are not interchangeable. As Serply’s note on picking between MCP resources and tools frames the test: ask who knows the parameter. If the model has to work it out from the conversation — a search query, a URL it only just discovered — it has to be a tool, because no user interface can pre-select an argument that doesn’t exist yet.

Applied to a local setup, that’s the line between “I dragged a file into the chat” and “the model decided to go look something up.” The first needs no model capability at all. The second needs the model to reliably produce well-formed calls, which brings us to the part that actually breaks.

Why small local models are worse at this

Tool calling is a trained behaviour, not an architectural feature. A model learns during fine-tuning to emit the call format when the situation calls for it, and that skill is one of the first things to degrade as parameter count drops. The specific failure modes you’ll meet:

Calls that never happen. The model answers from its own weights instead of reaching for the tool, confidently and wrongly. Common when the tool description is vague or when the model is small enough that instruction-following is shaky.

Malformed arguments. Right tool, JSON that doesn’t parse, or a parameter invented out of nothing. Structured-output enforcement in the runtime — constrained decoding against a grammar — fixes the parsing half of this outright and is worth turning on if your runtime supports it.

Fabricated results. The nastiest one. The model emits a call and then writes what it imagines the answer was, in the same turn, without waiting. Your loop returns the real result into a conversation that already contains a fake one.

Template mismatch. The model supports tool calling, but the GGUF you pulled ships a chat template that doesn’t render the tool block correctly, so the model never sees the tool list in the form it was trained on. This is the one that looks like a model quality problem and isn’t — it’s the same class of issue as running a GGUF your runtime’s library doesn’t offer, where the weights are fine and the packaging is not.

Quantization interacts here too. Aggressive quants tend to lose the crisp formatting behaviours before they lose general fluency, so a model that chats acceptably at Q3 may emit broken JSON at the same level. If you’re leaning on tools, add a tool call to the fixed prompt set you use for deciding whether a quant is good enough.

Wiring it up without a framework

You don’t need an agent library to try this. If your runtime already speaks the OpenAI-compatible shape described in serving a local model over HTTP, the tool fields are part of that shape, and thirty lines of script will do it:

messages = [user prompt]
loop:
  reply = POST /v1/chat/completions {messages, tools}
  if reply has no tool_calls: return reply.content
  for each call: result = dispatch(call.name, call.arguments)
  append the assistant reply and each result to messages

Cap the loop iterations. A model that gets stuck re-calling the same tool will happily do it until you run out of context, and the failure is silent — it just gets slower and then truncates.

Choosing a model for it

If tool use is the point of your setup, treat it as a selection criterion rather than a bonus. Check the model card for explicit tool-calling or function-calling support, prefer the instruct-tuned variant, and then test it the way you’d test anything else here: a handful of your own prompts, some of which should obviously need the tool and some of which obviously shouldn’t. A model that calls the tool for “what’s 2+2” is as broken as one that never calls it. This slots directly into the process in picking a local model for the job.

The practical takeaway

Tool calling is a text protocol plus a loop you control, not a capability the model has. That means the reliability you get depends on three separate things — whether the model was trained for it, whether the chat template renders it correctly, and whether your loop is defensive about malformed output. Verify each one independently and the whole thing stops feeling magical, which is exactly what you want from something running on your own machine.