Ollama 推出多模态模型新引擎，支持 Llama 4、Gemma 3、Qwen 2.5 VL 等视觉模型

May 15, 2025

Ollama now supports multimodal models via Ollama’s new engine, starting with new vision multimodal models:

Meta Llama 4; Google Gemma 3; Qwen 2.5 VL; Mistral Small 3.1; and more vision models

General Multimodal Understanding & Reasoning

Llama 4 Scout

(Note: this is a 109 billion parameter, mixture-of-experts model.)

Example of asking location-based questions about a video frame:

You can then ask follow-up questions:

Gemma 3

Example of giving multiple images and asking about their relations:

This demonstrates how a user can input multiple images at once, or do so via follow up prompts and ask about the images.

Document scanning

Qwen 2.5 VL

Example of using Qwen 2.5 VL for character recognition:

Example understanding and translating vertical Chinese spring couplets to English:

Ollama’s new multimodal engine

Ollama has so far relied on the ggml-org/llama.cpp project for model support and has instead focused on ease of use and model portability.

As more multimodal models are released by major research labs, the task of supporting these models the way Ollama intends became more and more challenging.

We set out to support a new engine that makes multimodal models first-class citizens, and getting Ollama’s partners to contribute more directly to the community via the GGML tensor library.

What does this mean?

To sum it up, this work is to improve the reliability and accuracy of Ollama’s local inference, and to set the foundations for supporting future modalities with more capabilities - i.e. speech, image generation, video generation, longer context sizes, improved tool support for models.

Let’s break down a couple specific areas:

Model modularity

Our goal is to confine each model’s “blast radius” to itself, thereby improving reliability and simplifying model integration for the model creators and developers.

Today, ggml/llama.cpp offers first-class support for text-only models. For multimodal systems, however, the text decoder and vision encoder are split into separate models and executed independently. Passing image embeddings from the vision model into the text model therefore demands model-specific logic in the orchestration layer that can break specific model implementations.

Within Ollama, each model is fully self-contained and can expose its own projection layer, aligned with how that model was trained. This isolation lets model creators implement and ship their code without patching multiple files or adding cascading if statements. They no longer need to understand a shared multimodal projection function or worry about breaking other models—they can focus solely on their own model and its training.

Examples of how some models are implemented are available on Ollama’s GitHub repository.

Accuracy

Large images produce large number of tokens which may exceed the batch size. Processing this correctly with the right positional information is challenging, specifically when a single image crosses boundaries.

Ollama adds metadata as it processes images to help improve accuracy.

Some examples:

Should causal attention be on / off?; Is it possible to split the image embeddings into batches for processing, and if possible, what are the boundaries when accounting for quality of output, and the computer being used for inference? If an image is split in the wrong place, the quality of output goes down. This is usually defined by the model, and can be checked in its paper

Many other local inference tools implement this differently; while a similar result may be achieved, it does not follow how the models were designed and trained.

Memory management

Image caching

Once an image is processed, Ollama caches it so later prompts are faster. The image remains in cache while it is still being used and is not discarded for memory-cleanup limits.

Memory estimation & KV cache optimizations

Ollama collaborates with hardware manufacturers and an operating system partner to make sure the correct hardware metadata is detected to better estimate and optimize for memory usage. For many firmware releases, partners will validate/test it against Ollama to minimize regression and to benchmark against new features.

Ollama has some KV cache optimizations to improve how memory can be efficiently used. Ollama configures causal attention at the individual model level instead of configuring as a group.

Examples:

Google DeepMind’s Gemma 3 leverages sliding window attention, and Ollama can leverage that to allocate a subset or a portion of the model’s context length to improve performance, and because of the resulting memory efficiency, we can increase the context length of the model on the same system or use the remaining memory for higher concurrency; To uniquely support Meta’s Llama 4 Scout and Maverick models, Ollama has implemented chunked attention, attention tuning to support longer context size, specific 2D rotary embedding, and in the mixture-of-experts type of model

If a model’s attention layer isn’t fully implemented, such as sliding window attention or chunked attention, it may still ‘work’. However, because this isn’t how the model was trained, the end user may begin to see erratic or degraded output by the model itself over time. This becomes especially prominent the longer the context / sequence due to cascading effects.

What’s next

Support longer context sizes; Support thinking / reasoning; Tool calling with streaming responses; Enabling computer use

Acknowledgements

Thank you to the teams working to release openly available and powerful vision models - Google DeepMind, Meta Llama, Alibaba Qwen, Mistral, IBM Granite, and the many countless people & labs in the community.

GGML

Thank you to the GGML team for the tensor library that powers Ollama’s inference. Being able to access GGML directly from Go has given a portable way to design custom inference graphs and tackle harder model architectures not available before in Ollama.

Hardware partners

Thank you to hardware partners for collaborating and helping to improve inference on a large number of devices - NVIDIA, AMD, Qualcomm, Intel, and Microsoft.