Running Qwen3-32B as a Local AI Agent: Fixing the Tool Calling Bug That Burned Every Token

How we got a locally hosted 32B parameter model to reliably control a headless browser and act as a fully autonomous agent, and the invisible bug that nearly killed it.

Why an offensive security firm cares about local AI agents

Part of our work at Coastline is evaluating whether local open-weight models can replace frontier APIs for autonomous reconnaissance and browser-driven security workflows. If a 32B model on a single GPU can reliably call tools, navigate pages and chain actions, it changes what is practical for always-on security automation without sending data to a third party.

The goal

We run OpenClaw, an AI agent platform that orchestrates multiple specialist agents through Discord and WhatsApp. Our chief agent handles general tasks, delegates to specialists, and needs to browse the web, run commands, and use a growing library of skills.

The chief ran on Claude Sonnet. Reliable, but expensive. We wanted to see if we could replace it with a locally hosted model running on an RTX 4090 (24GB VRAM) to eliminate API costs for the always-on agent.

The candidate was Qwen3-32B at Q4_K_M quantization, about 20GB, served through Ollama.

Step 1. Wiring Ollama and Qwen3 into the agent platform

OpenClaw has native Ollama support through auto-discovery, but we hit the first wall immediately. The auto-discovery requires an auth profile mapped to environment variables, and “ollama” is not in the built-in provider mapping. The gateway returned:

Unknown model: ollama/qwen3:32b

The fix was to define a custom provider in openclaw.json:

"models": {
  "mode": "merge",
  "providers": {
    "ollama": {
      "baseUrl": "http://127.0.0.1:11434/v1",
      "apiKey": "ollama",
      "api": "openai-completions",
      "models": [
        {
          "id": "qwen3:32b",
          "name": "Qwen3 32B",
          "contextWindow": 40960,
          "maxTokens": 8192
        }
      ]
    }
  }
}

Setting api: "openai-completions" tells OpenClaw to use Ollama’s OpenAI-compatible endpoint at /v1/chat/completions. The model was recognized, config validated, gateway started.

We restarted Ollama and confirmed all 65 layers were offloaded to CUDA:

llm_load_tensors: offloaded 65/65 layers to GPU
llm_load_tensors:        CUDA0 buffer size = 18430.02 MiB

First Discord message from qwen3. “I’m ready to assist.” We were in business. Or so we thought.

Step 2. The browser problem

When a user asked the chief to read a webpage, instead of using the browser tool (headless Chrome controlled over CDP), qwen3 responded with:

“The error you encountered is because the web_search tool requires a Brave Search API key…”

It was hallucinating a solution for a tool it couldn’t use, instead of calling the browser tool that was right there in its tool list. We tried:

• Detailed browser instructions in the agent’s system prompt (SOUL.md)
• Denying web_search in the tools config to force browser usage
• Exact JSON parameter examples
• Clearing sessions to force a reload

Sometimes it worked. Sometimes it didn’t. The first test against httpbin.org was perfect. Correct tool call, correct parameters, clean response. Then subsequent attempts failed at random. A classic intermittent bug.

Step 3. Finding the root cause, the invisible token drain

We dug into the actual API responses. The smoking gun:

Content: ''
Tool calls: none
Usage: {'prompt_tokens': 17, 'completion_tokens': 200, 'total_tokens': 217}

200 completion tokens generated. Zero content returned. Where did 200 tokens go?

We switched to Ollama’s native /api/chat endpoint and found the answer:

{
  "content": "4",
  "thinking": "Okay, so the user is asking '2+2?' and wants a brief answer. Let me think. First, 2 plus 2 is a basic arithmetic problem..."
}

Qwen3’s thinking mode was on by default. Every response started with a hidden <think> block, internal chain-of-thought reasoning that consumed the majority of the output token budget. The model was literally thinking too hard about every request, burning tokens on invisible reasoning, leaving nothing for the actual response or tool calls.

The OpenAI-compatible API strips the <think> tags and returns only the content field. When all tokens are spent on thinking, content comes back empty. OpenClaw sees an empty response, retries, gets another empty response, and times out at 180 seconds, which triggers the Claude fallback.

That is why the behavior was intermittent. Sometimes the thinking was short enough to leave room for a tool call. Sometimes it was not. On complex requests like “read this URL”, the model would reason for hundreds of tokens about how to approach the task, exhaust its budget, and return nothing.

Step 4. The fix, patching the chat template

Ollama’s native API supports "think": false to disable thinking:

curl http://127.0.0.1:11434/api/chat -d '{
  "model": "qwen3:32b",
  "messages": [{"role": "user", "content": "Read https://example.com"}],
  "think": false,
  "tools": [...]
}'

With think: false, tool calling was perfect. Correct JSON, correct parameters, 4.5 seconds, 29 tokens. But OpenClaw uses the OpenAI-compatible API, which does not support the think parameter. We could not just flip a switch.

The solution was to patch qwen3’s chat template in the Ollama Modelfile. Here is how qwen3’s template handles thinking:

{{- if and $.IsThinkSet (eq $i $lastUserIdx) }}
   {{- if $.Think -}}
      {{- " "}}/think
   {{- else -}}
      {{- " "}}/no_think
   {{- end -}}
{{- end }}

When $.IsThinkSet is true (only when the think API parameter is explicitly passed), it appends /think or /no_think to the user message. When it is not set, as with the OpenAI API, the template does nothing, and thinking defaults to on.

We created a custom Modelfile that always forces no-thinking mode:

FROM qwen3:32b

# Force /no_think on every user message (bypass $.IsThinkSet check)
# Prefill empty think tags in assistant response
TEMPLATE """
... (modified template that always appends /no_think and prefills empty think tags) ...
"""

PARAMETER temperature 0.3
PARAMETER top_p 0.8
PARAMETER top_k 15
PARAMETER num_ctx 40960

Three critical template changes:

1. Always append /no_think to the last user message, removing the $.IsThinkSet conditional.
2. Always prefill empty think tags before the assistant response. This tells the model reasoning is “done”.
3. Strip thinking from replayed messages so past reasoning does not waste context.

Then:

ollama create qwen3-tools -f /tmp/Modelfile.qwen3-nothinker

Step 5. The results

With the patched model we ran 5 consecutive browser tool-calling tests:

=== Test 1 === browser(navigate, https://test1.com) ok  4.6s  30 tokens
=== Test 2 === browser(navigate, https://test2.com) ok  4.8s  30 tokens
=== Test 3 === browser(navigate, https://test3.com) ok  4.9s  30 tokens
=== Test 4 === browser(navigate, https://test4.com) ok  4.9s  30 tokens
=== Test 5 === browser(navigate, https://test5.com) ok  4.8s  30 tokens

5 out of 5 clean. Every call produced the correct JSON schema:

{"action": "navigate", "targetUrl": "https://test1.com"}

Before and after:

Metric	Before (thinking on)	After (thinking off)
Success rate	about 20% intermittent	100%
Response time	26 to 180 seconds, often timeout	4.5 to 5 seconds
Tokens per call	200, all wasted on thinking	29 to 30
Tool call accuracy	random	perfect JSON every time

The end-to-end test worked. We asked the chief to read a target URL, and qwen3-tools:

1. Called browser(action="navigate", targetUrl="https://example.com/")
2. Called browser(action="snapshot")
3. Returned a detailed summary of the page content

All running locally on a single RTX 4090.

Additional tuning

Beyond the thinking fix we also tuned:

• Temperature 0.6 to 0.3. More deterministic outputs mean more consistent tool call JSON. The default 0.6 added unnecessary randomness to structured output.
• top_p 0.95 to 0.8, top_k 20 to 15. Tighter sampling reduces the chance of the model drifting into wrong tool names or parameter formats.
• Skills expansion. Opened the allowBundled list and installed additional CLIs to give the agent more capabilities.
• System prompt improvements. Added explicit tool calling examples with the exact parameter format. Qwen3 benefits more from in-context examples than Claude does.

Lessons

1. Thinking mode is a silent killer for tool calling. If you are running qwen3, or any model with default-on reasoning, through an OpenAI-compatible API, the thinking tokens are invisible but consume the entire output budget. You will see no error, just empty responses.
2. The native API and OpenAI-compatible API behave differently. Ollama’s native /api/chat returns thinking in a separate field and respects think: false. The OpenAI-compatible /v1/chat/completions strips thinking tokens silently. If your framework uses the OpenAI API, you need to handle this at the model template level.
3. Lower temperature matters more for smaller models. Claude at 0.7 still produces valid tool call JSON. Qwen3-32B Q4 at 0.6 occasionally drifts. Dropping to 0.3 eliminated the inconsistency without noticeably hurting response quality.
4. Model loading time is the remaining bottleneck. First call after a restart takes about 140 seconds because the 20GB model needs to load into VRAM. Subsequent calls are fast at 4 to 5 seconds. Keep the model warm with Ollama’s keep-alive setting or periodic pings.
5. In-context examples beat system prompt instructions. For qwen3, showing the exact JSON {"action": "navigate", "targetUrl": "..."} in the system prompt was more effective than describing the parameter schema in words. Smaller models learn better from examples than specifications.

The stack

• Hardware. RTX 4090, 24GB VRAM.
• Model. Qwen3-32B, Q4_K_M quantization, about 20GB.
• Inference. Ollama v0.15.4.
• Agent platform. OpenClaw 2026.2.3.
• Browser. Headless Chrome 144 over CDP.
• Channels. Discord and WhatsApp.

Final thoughts

Running a 32B parameter model as a fully autonomous agent, with browser control, skill execution, and multi-channel messaging, on a single consumer GPU is genuinely impressive. The model is fast, capable, and free to run 24/7.

But the gap between “model works in a chat” and “model works as an agent” is real. The thinking mode issue cost us hours of debugging because everything looked fine. The model was generating tokens, no errors were thrown, the API returned 200 OK. The bug was invisible by design.

If you are building agents on local models, test your tool calling through the exact same API path your framework uses, check your token counts obsessively, and never assume the model is doing what you think it is doing. Sometimes it is just sitting there, thinking very hard about nothing.

To see how we apply this kind of research to autonomous security testing, take a look at AI Penetration Testing vs. Manual Penetration Testing or try a free scan on the Coastline Autonomous Pentest portal.