Tang Jie-authored paper discloses Zhipu's internal multimodal model GLM-Omni

After the stunning debut of GLM-5.3 and GLM-5.3-Flash (although the two models belong to the same series by name, they were trained on different base models), Zhipu has continued to improve the models' capabilities in coding, long-horizon agents, and more. But how exactly will the next-generation GLM be upgraded? That remains an important question everyone is watching.

Beyond stronger reasoning and lower cost, there is another direction worth watching: native multimodality.

Recently, researchers from Tsinghua University and Zhipu jointly published a paper on arXiv titled "Can Visual Generation Expand What Unified Multimodal Models Know?" The corresponding author is none other than Zhipu's chief scientist and Tsinghua University professor Tang Jie.

The paper introduces a previously undisclosed internal Zhipu model, GLM-Omni, and attempts to answer a somewhat counterintuitive question: can knowledge a model learns through image generation be transferred into language reasoning ability?

In the experiments, GLM-Omni was able to correctly draw a specified jersey number in 72% of digital tasks, but when asked about the same number using only text without providing an image, its accuracy was only 2%.

The researchers then adjusted the training approach so that the model placed generated images, observed images, and text answers into the same training sequence, raising the model's average text question-answering accuracy from 12.57% to 26.86%. On this basis, allowing the model to first generate an image and then observe the image before answering further raised accuracy to 41.14%.

This study is not direct evidence that the next-generation GLM will adopt a certain architecture, but it offers a research clue worth tracking: after Zhipu already achieved native multimodal understanding on GLM-5.3-Flash, researchers are still exploring how to establish a closer connection between visual generation and language reasoning.

Zhipu Is Just Waiting for You

To understand the significance of this paper, we first need to clarify a concept: native multimodality does not mean the model can generate images on its own.

In August this year, Zhipu launched GLM-5.3-Flash and called it the first native multimodal model in the GLM-5 series. It uses a brand-new pretraining base, supports multimodal inputs such as images and video, and can understand interfaces, observe rendering results, and continue modifying code based on interactive feedback within Coding Agent workflows.

But judging from the publicly available model interface, GLM-5.3-Flash's main output is still text. Its ability to understand visual information does not mean it can already natively generate images within the same model.

This is also an important technical dividing line for multimodal models today.

One class of models mainly addresses unified understanding of multiple types of information, enabling the model to process inputs such as text, images, and video simultaneously;

The other class goes further and attempts to unify understanding and generation, enabling the same model to both understand visual content and directly generate images, while exploring knowledge sharing between the two capabilities.

The latter faces problems that are not just about how to draw images well, but also about whether knowledge the model learns from generation tasks can be invoked by other tasks.

When Tang Jie previously discussed native multimodality, his focus was not merely on adding another input or output capability. In his view, unified perception of information such as vision, sound, and even touch is an important foundation for models to further understand the environment and execute long-horizon tasks.

For agents that need to continuously observe the environment and adjust actions based on feedback, multimodality is not just a feature expansion; it also concerns whether models can truly enter complex real-world tasks.

In June this year, a netizen asked Musk on X: when can Chinese models reach Fable's level? Musk answered, possibly in the first quarter. Tang Jie replied: it won't take too long.

On the evening of October 21, the official operators sent a compensation notice to coding plan users. The notice ended with: "After the new model is released, we will additionally gift another reset card," which amounted to a preview that an update was coming soon.

The Hidden Capability of Multimodal Output

The paper's corresponding author, Tang Jie, is Zhipu's chief scientist. The three co-first authors Li Wenkai, Hong Wenyi, and Yang Zhuoyi completed this research during their internships at Zhipu.

The model GLM-Omni uses Zhipu's open-source GLM-4.1V-Thinking as its base, with 9B parameters, making it a relatively small test model. The research team proposed a testing method called Knowledge Transfer Probing (KTP), whose core is to distinguish between two things:

Whether the model learned a certain piece of knowledge during image generation;

Whether it can express this knowledge through language.

This is also a potential value worth exploring in unified multimodal models. It can not only let the model look at and draw images at the same time, but may also allow knowledge learned in different modalities to complement each other rather than work independently.

To verify this, the researchers built a fictional world called MagicWorld, in which entities such as teams, characters, and badges do not exist. For example, a fictional team's jersey number is 10, and during training the model is only taught to generate images of jerseys printed with the number 10, while the text prompt does not provide this number.

The dataset contains 7 categories of knowledge, including color, quantity, number, shape, and spatial position, with a total of 350 entries, which were expanded into 8,552 groups of image training samples. This is intended to eliminate as much as possible the possibility of the model answering questions by relying on existing textual memory.

In addition to Zhipu's internal GLM-Omni, ByteDance's BAGEL and SenseTime's SenseNova-U1 also participated in the test. The three models use different visual generation architectures, and the researchers separately checked whether they could correctly draw the target attributes and whether they could answer with text when no image assistance was available.

The first round of experiments already revealed a clear case of "can draw but cannot say."

GLM-Omni's average image generation accuracy reached 54%, but its accuracy in answering the same knowledge directly with text was only 14.57%. Among these, for numerical tasks, image accuracy was 72%, while text question-answering accuracy was only 2%. SenseNova-U1's gap on numerical tasks was even larger, at 88% and 0%, respectively.

This means that the model can already reproduce certain knowledge during image generation, yet it is difficult to invoke that knowledge directly through language. In other words, putting visual generation and language understanding into the same model does not mean that the knowledge mastered by the two capabilities will automatically become mutually accessible.

So where exactly is the problem? The researchers next tried two different training methods.

The first was text alignment: supplementing some fictional entities with text question-answering, letting the model learn how to answer related questions, and then testing whether it could transfer to entities for which it had not seen textual answers.

But the effect was unstable. GLM-Omni's average text accuracy even dropped from 14.57% to 12.57%. This shows that simply adding text question-answering is not enough to stably transform visual knowledge into language ability.

The second method is more interesting, called Interleaved Image–Text Alignment.

The researchers no longer treated image generation and question answering as two independent tasks, but placed them in the same training sequence. For example, when facing a jersey number question, the model first generates a relevant image, then observes the image, and finally gives a textual answer.

As a result, in text tests where GLM-Omni was not allowed to draw images on the fly, its average accuracy rose from 12.57% to 26.86%.

To determine whether the improvement came from image generation, the researchers also conducted a control experiment: keeping the same training samples and text supervision but removing only the loss signal from the image generation part of interleaved training. As a result, accuracy dropped again to 13.71%.

This reveals the paper's first important conclusion:

1) Knowledge sharing in a unified model depends not only on whether the architecture is unified, but also on whether the training signals of different tasks can be effectively connected.

2) Image generation provides more than just an output function; it may itself also help the model build knowledge associations that can be used by other tasks.

However, even after joint training, the model still had a large amount of visual knowledge that could not be directly expressed in text. The researchers therefore took a different approach: since the model can draw the answer, why not have it first draw the relevant content when answering a question?

This is the Visual Chain-of-Thought adopted in the third round of experiments.

Unlike the previous improvements to training methods, this time what changed was the model's actual reasoning process when answering questions: first generate an image, then observe the image, and finally organize a textual answer.

The effect was even more pronounced. GLM-Omni's average text question-answering accuracy further increased from 26.86% to 41.14%, an increase of 14.28 percentage points. Among these, numerical tasks rose from 0% to 36%.

This also brings the paper's second important finding: image generation may not only help the model learn knowledge during training, but can also serve during inference as an intermediate representation for invoking knowledge. For some questions that are difficult to answer directly in language, the model can first present the relevant information by generating an image, and then extract the answer from it.

Overall, this paper does not prove that unified multimodality is necessarily superior to a combination of independent models, but it does provide a technical value worth further verification: visual generation is not just about drawing images for users; it may also become a new pathway for models to learn, organize, and invoke knowledge.

This may also be an important direction that Zhipu has noticed when training its next-generation models.

Multimodal route choice

Unified multimodality: how will the next-generation GLM choose? Since letting the model draw images itself may also improve reasoning ability, why have leading vendors not fully adopted this architecture?

In March 2025, OpenAI integrated native image generation into GPT-4o, emphasizing that image generation should become a core capability of language models. But in subsequent products, OpenAI shifted back to having specialized models such as GPT Image handle image generation. Google has also formed a product division of labor between the general-purpose Gemini and the Nano Banana image model.

The practical consideration behind this is cost and efficiency. Compared with having the flagship model take on all tasks, specialized models are easier to optimize separately for image quality, speed, and cost.

Zhipu is the same: currently GLM-5.3-Flash is mainly responsible for multimodal understanding, while GLM-Image specializes in image generation.

Interestingly, as early as December 2025, Tang Jie said that text, multimodal understanding, and multimodal generation "may still be most effectively developed separately," but that combining the three is also worth exploring, though it requires sufficient capital support.

So what exactly can the extra cost paid by a unified model buy?

This paper gives a preliminary answer: image generation is not just adding an output function; it may also allow the model to learn information that was not present in language training and convert it into other capabilities during training and inference. However, the research has not yet proven that this benefit is enough to offset the cost of a unified architecture.

And Zhipu's layout for next-generation models is clearly more than this. Its latest financing arrangements involve the next-generation GLM, fully self-training, and computing infrastructure construction. At Zhipu's August 31 earnings communication meeting, Tang Jie proposed that GLM-6.0 will explore the model's self-evaluation and continuous improvement capabilities.

The same announcement also mentioned that Zhipu aims to improve the effective scale of foundation models, native context length, and "native multimodal unified modeling capability."

From the native multimodal understanding of GLM-5.3-Flash to GLM-Omni's exploration of knowledge sharing between visual generation and language reasoning, one can begin to sense an important direction for the next-generation GLM—not just whether it can directly generate images from text, but whether multimodality can truly become a new source of growth for model capabilities.

This article comes from the WeChat public account "AI Frontline" (ID: ai-front), authors: Siyue, Feiye, published by 36Kr with authorization.