蚂蚁 inclusionAI 开源 Realtime-Venus 全双工交互系统

inclusionAI; Realtime-Venus

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork11 (11); Star149 (149)You must be signed in to star a repository

Repository files navigation

README; Apache-2.0 license

A full-duplex interaction system with asynchronous delegation

English | 简体中文

Overview

See, listen, and respond in real time—with background tasks running alongside the conversation. Realtime-Venus combines full-duplex conversational models with an asynchronous execution framework. You can ask the assistant to work on a task and continue talking while it runs. The result returns to the same conversation for spoken delivery.

The project brings together three components:

What Realtime-Venus enables

Continuous audiovisual interaction. Omni observes streaming camera and microphone input and can initiate a response when the evolving scene calls for it; Full-duplex conversation. The models process incoming speech while speaking, learning to distinguish acknowledgments and background speech from interruptions that call for a revised response; Asynchronous delegation. A model can hand a natural-language task to Harness for reasoning or tool execution while live interaction continues; Results in the same conversation. Harness prepares a reply; the conversational model handles its timing and native speech output. Work tracking separates task completion from spoken delivery

The paper describes the model family, dual-loop runtime, training data, and evaluation. This source release includes the Omni model integration, the reusable Harness package, and a browser demo using a Codex task backend. The demo's microphone-only mode also uses Omni.

Quick start

1. Install dependencies

Standalone inference requires Python 3.10, CUDA, and FFmpeg. Run the following commands from the source repository root. First install the download helper's dependencies:

The unified downloader reads the root config.yaml in the single Hugging Face repository inclusionAI/Realtime-Venus to locate the model directories. Choose one download:

Omitting --model defaults to all. The downloader saves the root config.yaml and each selected model's complete directory under --local-dir, displays the standard Hugging Face Hub progress bars, and reuses cached files on later runs.

The same downloader is available from Python. It returns the selected local model directories as Path objects:

The ModelScope mirror is an optional alternative, downloaded directly with its own CLI:

Then install the dependencies for your selected model:

For the browser demo, run the installer on Linux. It creates an isolated Python 3.11/3.12 environment and installs the model, web service, and Harness dependencies:

Then choose standalone inference or the online experience below.

2. Frontend Usage

Use the local checkpoints downloaded above. With --local-dir ., they are in ./Realtime-Venus-Omni/ and ./Realtime-Venus-Audio/ for the models you selected. Replace the /path/to/ placeholders below with those directories, or with your chosen download location.

Realtime-Venus-Omni

Offline video chat. Generate text and speech responses based on the input video. The two examples demonstrate basic video chat and chat with long-video memory.

Full-duplex video interaction. Stream the visual and audio content of a recorded video into the model as it generates text and speech responses. The three examples demonstrate text questions, spoken questions, and long-video memory.

Duplex examples save subtitled videos. See the Omni guide for inputs and output paths.

Realtime-Venus-Audio

Offline audio understanding. Provide a complete audio clip for the model to understand and respond to in text.

Full-duplex audio interaction. Stream recorded audio into the model and generate speech responses while it continues listening. Save the output as a 24 kHz WAV file.

Both scripts accept --system-prompt and --prompt; see the Audio guide for decoding options.

3. Talk to Realtime-Venus

The online Duplex demo requires at least one NVIDIA A100 GPU. Configure the frontend model in root config.json, and configure task execution in harness/config.json. The Demo file references the Harness file through harness.config.

Follow the Demo configuration guide and Harness configuration guide, then start:

Choose model.type: "audio" for microphone and audio upload, or "video" (also accepts "omni") for camera with microphone and video upload. Set model.path to the corresponding downloaded checkpoint. Complete Codex login if prompted; default ports are 8031 for the model and 8032 for the web service.

On your local computer, keep this tunnel open, replacing user@server with your SSH login:

Open http://localhost:8032, complete Settings and start a conversation. See the Demo guide for service management.

4. Android Demo (Beta)

Download the Android demo: Realtime-Venus-0918.apk — Beta. This is a Beta release for research and demonstration. See the release page for package details.

Results

Figure 1. Video and audio understanding results from the paper.

Figure 2. Full-duplex interaction results from the paper.

The source code in this repository is licensed under the Apache License 2.0, except for components with separate license notices. Third-party fonts and paper figures retain their respective licenses; see the license files in demos/static/fonts/ and assets/README.md.

Realtime-Venus is a research project by Venus Team, in collaboration with Tsinghua University.

The content on this page is for research and demonstration purposes only.