AI Trends

Weitu AI Streaming Multimodal Video Brain: Capable of Memory and Thinking, Scores 69.9 on OVBench, Surpassing Gemini 3.1 Pro

Sina TechUpdated Nov 16 14:26

AI summary

Weitu AI has launched a streaming multimodal video brain. Through the Dual Sparse Memory two-layer sparse memory architecture and the Long-term Memory Module, it enables camera-equipped devices to continuously perceive their environment, update memories, and support complex reasoning. It achieves an overall score of 69.9 on OVBench, 11.1 higher than Gemini 3.1 Pro's 58.8; on the Meta S-EMBER benchmark it scores 58.23, 3.82 higher than Gemini 3.1 Pro's 54.41. This capability targets scenarios such as AI glasses, wearable cameras, and industrial inspection, and can serve as the perception and memory entry point for general-purpose large models.

Body · AI translation

Add crowdsourcing promotion copy Add crowdsourcing promotion copy end Add App-exclusive traffic routing

Published by Machine Heart

Before heading out wearing AI glasses, ask the AI, "Where did I just put my keys?"; after a meeting ends, have the AI assistant review "what changed on the whiteboard compared with the beginning"; when a maintenance worker wearing a wearable camera device is performing an inspection, the video brain reminds them, "The action of closing the valve just now was not completed, please check again." These daily life and industrial needs all require AI to connect the scene in front of it with prior content. The streaming multimodal video brain built by Weitu AI is precisely intended to enable any device with a camera to continuously, efficiently, and accurately understand its environment, and to turn the information it has seen into memories that can be used for recall and complex reasoning and decision-making.

The Hong Kong multimodal AI company Weitu AI currently serves dozens of enterprises worldwide, as well as about 6 million individual users globally. The team hopes to further bring multimodal technology into daily life, office, and industrial scenarios, so that multimodal devices in various scenarios have the ability to continuously perceive, reason, and make decisions.

According to the Weitu AI team, the core problem that the streaming video brain must solve is how, as video continuously arrives, to efficiently understand changes in the scene, continuously generate memories, and use real-time understanding and continuously updated memories to achieve complex reasoning and decision-making. For users, this means that with the application of such a streaming video brain, all kinds of multimodal devices that need to continuously perceive the environment and interact will be the first to encounter the ChatGPT moment of the physical world.

An efficient learning architecture with spatiotemporal compression brings continuous multimodal perception

Video in the real world is continuously generated, yet the model's computational resources and memory capacity are very limited. If all frames are retained for a long time and repeatedly involved in computation, the processing burden increases; excessive frame sampling or retaining only a very short time window may miss key temporal changes. For example, the action of "the keys being put into the side pocket of the backpack" lasts only a moment, and this detail becomes especially important only when the user looks for the keys ten minutes later.

Weitu AI has built a video-native multimodal architecture. Such an architecture can achieve efficient compression and perceptual learning of streaming video signals. The design idea is to let computation revolve around the changes that currently need to be understood. When building video-native representation and memory mechanisms, the team constructed the Dual Sparse Memory learning architecture: first automatically filter out relevant segments from the temporal dimension, then extract key spatiotemporal features from within the segments, avoiding large amounts of irrelevant visual information occupying learning computational resources.

The first step is temporal block-level sparse learning, that is, compressed sensing in the temporal dimension. Faced with retrospective questions such as "what was written on the whiteboard at the start of the meeting," the model first focuses attention on the relevant historical time segments, greatly narrowing the scope of subsequent computation. The second step is sparse learning in the spatial domain: after entering these segments, the model automatically focuses on and extracts key visual details (such as the main content of the scene and its changes), rather than letting all information in the entire frame participate in computation with equal intensity.

This mechanism puts both "when it was seen" and "which details are useful" into the network for learning. Its goal is to retain key features while controlling attention computation and cache overhead, making continuous video understanding more suitable for long-term operation on various hardware devices. For example, for various hardware systems, computational efficiency is directly related to response experience, deployment cost, and the feasibility of device-cloud collaborative solutions.

Continuously updated memory helps complex reasoning

Efficient compressed sensing and computation solve the problem of continuous understanding, while the continuously updated memory module can help the model solve more complex reasoning. In addition to understanding the current scene, Weitu AI's model also includes a Long-term Memory Module, which compresses earlier visual and spatiotemporal information into compact representations and continuously updates them as new frames arrive.

This type of memory carries historical information that the model can use, including changes in object positions, the sequence of actions, and important states in the scene. The model must remember both that "the keys were once on the table" and retain the information, with temporal and spatial changes, that "they were later put into the side pocket of the bag." For example, in industrial scenarios, this capability can be used to understand the steps of past operation demonstrations. Schematic of the memory architecture of the streaming video brain

The Weitu AI team explained that the team regards memory as an intelligent representation that has been selected and compressed. What the memory module needs to learn is not only the compressed perception of current information, but also information that can support future understanding and reasoning. Just as humans, when thinking, often recall memories to find information for reasoning. This provides inspiration for the memory design of video models.

For example, when a user asks, "Were the keys and the book put together?", the system needs to connect the object positions and storage actions seen at different times, and then determine whether they were placed in the same location. This allows memory to participate in complex reasoning and judgment. Compressed memory cannot retain all the details of the original video, and the model needs to learn and continuously update within a limited capacity, so that the memory module can participate as much as possible in subsequent complex reasoning.

Not only that, the Weitu AI team also explored new scenarios for the streaming video brain, such as predicting and judging video states, and making accurate predictions about the state of ongoing events or actions. The Weitu AI team mentioned, "For example, when an AI system is watching the coffee brewing process, it needs to accurately distinguish between 'pouring water' and 'this step is already complete' in order to answer questions about step order and whether an action is complete. Incorporating state changes together with temporal relationships into memory helps understand how an event or action occurs and when it enters the next stage. This type of problem is also a very interesting new research question, namely how to predict the temporal boundary of an action that has not yet been completed; this is equivalent to predicting a future state."

Demonstrating advantages in streaming video and memory evaluations

Recent evaluations show that the Weitu AI streaming video brain demonstrates significant architectural advantages in the field of continuous video understanding. On the authoritative benchmark OVBench [1], which examines the three major temporal dimensions of "past memory, current perception, and future prediction" across a total of 16 subtasks, the Weitu AI streaming video brain achieved a comprehensive score of 69.9, 11.1 higher than the industry's fully validated general multimodal strong baseline Gemini 3.1 Pro (58.8); at the same time, on the Meta S-EMBER benchmark [2], which is highly close to the continuous perception scenarios of wearable devices such as AI glasses (this test uses first-person perspective daily videos and requires the model to answer only based on content seen before the moment of the question), the Weitu AI streaming video brain achieved a score of 58.23, also 3.82 higher than the benchmark-aligned Gemini 3.1 Pro (54.41).

These results show that Weitu AI's model demonstrates advantages in both streaming video understanding and benchmarks with memory evaluation. The product value this brings can enable more AI devices to better understand constantly changing scenes and to reason and make decisions by combining the current frame with memory.

The two examples in the figure demonstrate the model's "spatiotemporal memory" question-answering capability: in a continuous video stream, the model can respond to user questions at any time, integrating current visual information with relevant historical memory frames to answer questions accurately.

Bringing continuous perception and memory into more application scenarios

The application space of the streaming multimodal brain covers various devices and products that need to continuously understand the environment and respond. From AI glasses and wearable cameras to other visual terminals, this capability is expected to be integrated into daily life, office, education, retail, and industrial scenarios, providing users with immediate assistance closely related to their current activities.

For example, when learning origami or cooking, an AI assistant can prompt the next step based on the steps already completed; when shopping, it can compare the product in front of the user with styles, specifications, and labels seen earlier; after an office presentation ends, it can help users review the sequence of operations and changes in the plan; and during industrial maintenance, it can combine previous operations with the current equipment status to indicate the links that require further inspection. These scenarios share the same capability requirement: linking current observations with historical memory, understanding the progress of the task, and providing a basis for the next judgment.

Such capabilities can also be applied to video creation. In the process of continuous video generation and editing, the model can combine the current frame, creative progress, and previous content to continuously evaluate how well the work fits the creative goal, propose optimization suggestions around shot transitions, narrative rhythm, and visual style, and dynamically plan subsequent generation, editing, and revision steps. By integrating continuous perception, memory, and reasoning into the creative process, it is expected to reduce the time and computational cost caused by repeated generation and modification, making works better match the creator's expressive intent while maintaining content coherence and stylistic consistency.

Providing general large models with an entry point for continuously perceiving the world

The progress of general large models is expanding the space for AI to participate in complex tasks. OpenAI's GPT-6 series has made progress in complex task handling, computer operation, and cost efficiency; Anthropic, in the release of Claude Fable 5.2, demonstrated the model's ability to control robots to complete tasks such as unscrewing bottle caps and drawing.

In Weitu AI's view, as the reasoning and task planning capabilities of general models strengthen, how to efficiently and in real time perceive environmental changes and bring relevant historical information into current judgments will become an important foundation for robotic applications. For example, when looking for items or tidying a room, a home robot needs to continuously confirm whether items have moved, whether operations are complete, and whether task conditions have changed, and adjust its actions based on previous observations. Continuously updated perception and memory can provide the necessary context for these judgments.

Weitu AI hopes to promote its streaming multimodal brain as an entry point for various general large models to continuously perceive the world: the video brain continuously processes frames, tracks scene states, and updates memory, providing timely, compact, and task-relevant visual and multimodal information for complex reasoning and task planning. This is equivalent to providing general large models with a pair of "eyes" that can continuously observe and retain relevant memories, and is expected to reduce their burden of repeatedly processing lengthy visual storage.

This positioning also gives the streaming multimodal brain two types of application value: it can directly support specific scenarios, and it can also serve as a perception and memory module that works in concert with other general models. In the future, when this capability is further combined with a robot's motion control and execution systems, it is expected to help robots continuously understand task processes in home, office, and service environments, and adjust subsequent actions according to environmental changes.

Starting from video, integrating more modal information

Weitu AI explains that the streaming video brain is the team's starting point for exploring multimodal interactive intelligence. In the future, the team hopes to continue integrating more modal information into this model architecture, so that perception, memory, and reasoning capabilities can adapt to the needs of different devices and scenarios.

For example, in industrial scenarios, video can be combined with sensor signals such as temperature, pressure, and vibration to jointly reflect equipment operation and environmental changes. Different signals provide complementary information: images present the operating process and appearance changes, while sensors reflect temperature fluctuations, pressure changes, or vibration anomalies. The model needs to correlate this information in time and space, understand the connections between them, and form memories that can be used for subsequent judgments.

Continuously perceiving changes, compressing important information into spatiotemporal memory, and then combining current perception with historical memory for reasoning and decision-making is the core approach of Weitu AI in building the streaming multimodal brain. From today's cameras, AI glasses, and wearable devices to future robots and industrial multimodal systems, the team hopes that through a scalable model architecture, more devices will gain the ability to continuously understand the environment, and also provide general large models with an entry point for quickly and accurately perceiving the world.

Weitu AI welcomes partners from all industries to discuss cooperation and continuously explore applications of the streaming video brain in more scenarios around the world. Commercial users who wish to apply for experience and trial of the model, please use your corporate email to send your request to business@weitu.ai, and we will contact you as soon as possible.article_adlist[

To follow more progress from Weitu AI, welcome to subscribe to the official account below. ]article_adlistarticle_adlist[

[1] Huang Z, Li X, Li J, et al. Online video understanding: Ovbench and videochat-online [C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 3328-3338.

[2] Wang X, Zhao X, Rodriguez P, et al. S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval [J]. arXiv preprint arXiv:2607.02689, 2026. ]article_adlistarticle_adlist[

For reprinting, please contact this official account for authorization

Submissions or report requests: liyazhou@jiqizhixin.com ]article_adlist Statement: This is an exclusive article of Sina.com. Reproduction without authorization is prohibited. tech_web_article_v2015_block_bottom Non-targeted 300*250 button 17/09 wenjing begin

Non-targeted 300*250 button end

Source

Sina Tech