English text is being prepared.
English text is being prepared.
Why read this
AI score 69
Duo v1 integrates interactive decision-making and audiovisual generation into a unified timeline. Its modular design, training methods, and evaluation shortcomings provide concrete reference points for bidirectional world models.
Topics
Body · AI translation
This world model has “come alive”: You hold out a bottle of water, and an AI-generated virtual person takes it and drinks Download the Zhidx app for an exclusive first look November 18, 2026, 23:00:24, from Beijing
Zhidx reported on November 18 that on November 17, the Ant Technology Research Institute and Hangzhou-based large-model company Westlake Xinchen released and open-sourced Duo v1, a Reciprocal World Model (RWM). Through two tightly coupled modules, Intent and Realization, it internalizes intelligent agents as part of the world.
In scenes generated by Duo, virtual agents can continuously receive users’ audio and video input, independently decide when to speak and when to listen, and generate the corresponding audio and visuals in real time to interact with users. For example, after exercising, a virtual agent says it is thirsty. When the user holds out a bottle of ice water, the virtual agent can take the bottle and drink from it.
Traditional world models generally simulate only physical scenes, including lighting, materials, and motion. Users must issue instructions using a keyboard, mouse, or text input. Although the worlds generated by these models look realistic, they lack agents capable of responding proactively.
Reciprocal world models, by contrast, encompass both physical reality and intelligent agents. In the Duo v1 framework, the virtual agent and the world share the same interactive space-time: it continuously receives users’ audio and video input and generates consistent audiovisual behavior. Within this reciprocal relationship, the virtual agent is capable of perception, reasoning, and expression.
In human evaluations involving 30 sets of real interactions, Duo v1 passed 168 of 180 checks, for an overall pass rate of 93.3%. Its pass rates for language compliance, speech intelligibility, and visual identity stability were all 100%, while its content responsiveness rate was 73.3%. Content responsiveness and long-range consistency remain the main challenges for the current version.
Duo v1’s technical report, training code, and model weights are now publicly available. Developers can download the model weights and build further upon them.
Technical report:
Official website:
Code:
Model:
ModelScope:
Hugging Face:
I. Deciding every 0.7 seconds whether to speak or wait, with the Intent and Realization modules jointly generating audio and video
Duo consists of two modules, Intent and Realization, which handle interaction decisions and audiovisual expression, respectively.
Intent acts as the virtual agent’s “brain,” enabling continuous interactive reasoning and decision-making along a unified timeline.
Within each roughly 0.7-second chunk, Intent independently decides, based on user input and interaction history, whether to speak or listen and wait, while simultaneously generating the corresponding linguistic content.
What Intent passes on is not decoded text but the model’s internal hidden states, which directly guide the Realization module.
According to the team, this design gives the virtual agent native interactive rationality: it can decide when to join a conversation, when to remain silent, and how to respond after being interrupted.
Realization serves as the virtual agent’s “expression,” converting decisions into audiovisual behavior. By combining Intent’s hidden states, user input, and its own previously generated content, it can generate matching audio and video.
To ensure causal correctness in streaming interactions, Duo v1 strictly aligns user input, Intent decisions, and Realization output along a unified timeline. Attention masks in the different modules restrict the scope of information access, preventing the model from using input that has not yet occurred. Previously generated audio and video are used as historical context when generating subsequent chunks.
According to the team, this causal synchronization is achieved through the model’s own inference rhythm. Each chunk lasts approximately 0.7 seconds, allowing Intent reasoning and Realization generation to connect seamlessly within a unified pipeline.
II. Three-stage training aligns intent understanding with audiovisual generation, followed by distillation into a four-step generator
Duo v1’s training is divided into three stages: Intent training, Realization training, and distribution-matching distillation. These stages progressively align “intent understanding” with “audiovisual generation.”
In the first stage, the Intent module is trained for causal participation and text prediction. In each chunk, the model predicts whether to speak or wait, along with the corresponding linguistic content. Because most chunks contain no new speech from the character, the model can easily develop a tendency to keep waiting. The team uses action-aware rebalancing to sample and weight training objectives from different categories, mitigating this “excessive waiting” bias.
At the same time, the training team truncates or repeats historical text to teach the model how to recover from incomplete or duplicated context.
In the second stage, the Realization module is trained for causal joint audiovisual generation while the Intent module remains frozen. Based on the hidden states extracted by Intent, recorded media history, and user input, Realization learns to generate the corresponding latent audio and video representations.
During this stage, attention masks control access to information: noisy media representations may access only the reference images, global descriptions, clean historical media, and the current Intent hidden states. They cannot access their corresponding clean training targets or future chunks. These causal constraints ensure consistency between training and inference.
The third stage uses distribution-matching distillation to compress the model into a four-step generator.
The key issue to address in this stage is the distribution gap between training, when Realization uses recorded media history, and actual inference, when it must rely on history generated by the system itself.
To this end, the team introduced Dual Rollout to address distribution shift: within each training window, the frozen Intent module autoregressively generates the character’s text, while Realization continues generating based on these Intent features and its own synthesized audiovisual history. By training on self-generated data, the model further adapts to the input distribution encountered during actual interactions.
III. A 60-Second Generation Test Mitigates Visual Degradation, but Content Responsiveness Remains a Weakness
Duo v1’s training data includes both synthetic and recorded data.
The synthetic data is produced through a two-stage pipeline. First, a large language model generates a structured script from a control card, and then a video generation model generates the audio and video chunk by chunk. The control cards cover dimensions such as duration, language, character profile, and interaction type, ensuring a diverse range of scenarios.
One challenge in video synthesis is error accumulation. If each chunk uses only the final frame of the preceding chunk as a reference, visual errors may gradually compound, causing image quality to deteriorate continuously.
To mitigate this issue, Duo v1 consistently uses the opening video of the first chunk as a quality reference. In a continuous-generation test comprising 10 chunks with a total duration of 60 seconds, the baseline method’s brightness value fell from 103.8 to 69.6, a decrease of 34.2. With this method, the brightness value remained at approximately 99, while consistency in facial appearance was also maintained.
After generation, the team filters out low-quality samples using four types of automated quality checks: first-frame consistency, unexpected audio, camera zoom-ins, and large-scale visual changes.
The recorded data comprises six types: single-person videos, synchronized dual-view videos, asynchronous dual-view videos, livestream interactions, full-duplex audio, and filtered open-source datasets. All data undergoes automatic speech recognition (ASR), voice activity detection (VAD), speaker diarization, and timeline alignment before ultimately being standardized into a causal interaction format.
In terms of performance, Duo v1 passed 168 of 180 checks in a human evaluation involving 30 real-world interactions, achieving an overall pass rate of 93.3%. Language adherence, speech intelligibility, and visual identity stability all reached 100%; audiovisual emotional consistency reached 96.7%, audiovisual synchronization 90.0%, and content responsiveness 73.3%.
The team said content responsiveness is the most challenging metric. The model must make decisions while the user’s audio and video inputs are still ongoing, inferring the user’s intent in real time. Understanding complex intentions and maintaining long-range consistency will be key priorities for optimization in future versions.
Conclusion: Virtual Agents in World Models Begin to Participate Autonomously in Interactions
Duo v1 attempts to place interaction decision-making and audiovisual generation within a single model framework, enabling virtual characters to continuously receive information from their environment, autonomously decide when to listen and when to respond, and generate the corresponding audiovisual behavior.
The current version still has room for improvement in content responsiveness and stability during extended conversations. With the training code and model weights now publicly available, this approach of incorporating intelligent agents into world models will also be tested across a broader range of scenarios.
“Special Notice: The content above (including any videos, images, or audio) was uploaded and published by users of Dafeng Hao, a self-media platform operated by Phoenix New Media. This platform merely provides information storage services.
Notice: The content above (including the videos, pictures and audios if any) is uploaded and posted by the user of Dafeng Hao, which is a social media platform and merely provides information storage space services.”
AI Trends