Ollama 支持工具调用流式响应

May 28, 2025

Ollama now supports streaming responses with tool calling. This enables all chat applications to stream content and also call tools in real time.

Models that support using tools:

Qwen 3; Devstral; Qwen2.5 and Qwen2.5-coder; Llama 3.1; Llama 4; and more tool calling models

Example of simple tool calling (weather)

Example of web search

Download the latest version of Ollama

cURL

An example of Ollama using the weather tool to answer the prompt What is the weather today in Toronto?

Output

Python

Install the latest version of the Ollama Python library:

An example of Ollama using a mathematical function:

Output

JavaScript

Install the latest version of the Ollama JavaScript library:

An example of Ollama using a mathematical function:

Output:

How tool parsing works in Ollama

Background

We’ve built a new parser that focuses on understanding the structure of a tool call rather than simply looking for JSON.

Previously, when tools were passed into the model, the system had to wait until the entire output was generated and then parse it as JSON to determine whether it contained a tool call or normal content. Users had to wait for the complete generation before seeing any streamed token. This approach was reliable against malformed output, but blocked streaming because a tool call might occur at any point in the text.

Ollama supports a wide range of models, some trained with tool-specific tokens and some without. The parsing logic would needs to stream user content while being able to detect, suppress, and parse the tool call tokens.

Incremental Parser

The new parser directly references each model’s template to understand the prefix of the tool call. This is necessary for Ollama to understand and separate the tool calls and content.

When a model is not directly trained on tool usage directly (trained with a prefix/tool token), it may still be able to output valid tool calls based on the sheer amount of knowledge it has. In this case, the parser is able to handle the partial prefixes output by the model and correctly separate tool calls and content.

Some models also elect to output a tool call without a prefix, even though they were trained on using a prefix for calling tools. Empirically, this behavior happens at the start of a model output only. To address this, the parser can fallback to parsing JSON as a tool call when it recognizes the start of a JSON. If the JSON does not match the tool call format for the model, the JSON will be returned.

Accuracy

In some cases, a model will reference the tool call it made previously when the results are passed into the model. Previously, this would return in multiple extra tool calls. By implementing prefix matching and correctly managing states where JSON is parsed to detect tool calls, tool calling reliability should be improved.

Previously, a model output like the example below would result in two tool calls instead of one:

Model Context Protocol (MCP)

With the improvements, developers can now stream chat content and tool calls when using Ollama with MCP (Model Context Protocol). Anecdotally, using a context window of 32k or higher improves the performance of tool calling and also the result of the tool call.

Example increasing the context window in Ollama to use 32k

Please note, memory usage increases with a longer context window.