Tsinghua University, NVIDIA and other institutions jointly release StreamEval, a real-time interactive video evaluation benchmark, with 38 models participating

Over the past six months or so, related models have broken out of their niches in rapid succession. Alibaba launched HappyOyster, Kunlun Wanwei released Matrix-Game 4.0, Aishi Technology released PixVerse R2, Shengshu Technology launched Vidu S1 in July and followed up with Vidu S2 in September...

Capital has likewise been rushing in, with several companies completing new funding rounds—world models, digital humans, real-time editing... players are pouring in from every direction.

But behind the excitement, this track is in a state of chaos.

The chaos starts with definitions. For the same real-time interactive video, some call it an "interactive world model," some call it a "real-time interactive multimodal model," and some understand it as a third path independent of video generation and world models. There isn't even consensus on what the track is called, let alone on evaluation standards.

Existing evaluation tools still follow the old framework: give the model a prompt, generate a video, and check whether the image is clear, the motion is reasonable, and the style matches the description. Mainstream benchmarks such as VBench and EvalCrafter all work this way, but for real-time interactive video systems, this approach doesn't test the hardest part at all.

The core challenge of real-time interactive video is that user instructions are continuous: the model needs to respond promptly to new instructions without interrupting generation, while keeping character identity, the existing scene, and all previously valid modifications unchanged. Existing evaluation tools barely touch any of this.

Recently, Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton, NUS and other universities and institutions at home and abroad jointly launched a new evaluation benchmark—StreamEval.

This is the industry's first global streaming interactive video generation evaluation leaderboard, covering five task categories—digital human dialogue, virtual human performance, world generation, instruction editing and reference editing—with 300 test cases in total and 38 models included.

Compared with existing benchmarks, StreamEval tests not only "generation quality" but also "interaction capability," bringing instruction execution, state consistency, visual quality and audio-visual coordination into a single quantifiable evaluation system, so that models on different technical paths can be tested under unified conditions.

At present, the code, dataset and scoring rules are all open source, and developers can independently re-test and verify the scoring process according to the public workflow. The leaderboard is continuously updated, and any developer can plug in their own model to participate in testing.

What exactly does StreamEval test? How does it test?

Evaluating a real-time interactive system requires running it under near-real conditions, with instructions arriving in batches, the scene continuously evolving, and the model responding continuously as time flows.

StreamEval is designed on exactly this logic, covering five task categories—digital human dialogue, virtual human performance, world generation, instruction editing and reference editing—each containing 60 test cases and covering five durations of 15 seconds, 30 seconds, 60 seconds, 120 seconds and 300 seconds, with very comprehensive evaluation dimensions. Each case is run independently three times by the same model to reduce the impact of chance factors on the results.

The overall StreamEval framework. The benchmark covers five task categories, involving digital human interaction, world generation and real-time video editing.

The full dataset contains 1,347 online events. Speech, text instructions, reference images and source video frames are released step by step along the timeline; the model can only see the inputs already released at the current moment, cannot know subsequent requirements in advance, and has no access to the expected answers used for scoring.

Composition of the benchmark dataset. (a) Task and function distribution; (b) preset input types; (c) number of input events in each test session; (d) time intervals between adjacent preset input events. The figure shows 200 test cases, containing 562 online input events in total, excluding initialization inputs and continuously input source video frames. When computing input intervals, event pairs involving trigger events were excluded.

The five task categories each have their own focus:

Digital human dialogue examines whether the model can understand user speech, give context-appropriate responses, and at the same time maintain character identity and dialogue state;

Virtual human performance requires the character to complete actions such as gaze, facial expressions, posture and gestures according to real-time instructions;

World generation focuses on scene changes, camera movement and spatial continuity under continuous control;

Instruction editing and reference editing require the model, while the source video keeps playing, to make modifications according to text or reference images and preserve content that was not specified.

Coverage and evaluation durations of the five task categories. Dlg. denotes digital human dialogue, Perf. denotes virtual human performance, World denotes world generation, and Instr./Ref. denote real-time video editing based on text instructions and reference images, respectively. ✓ indicates the task is included, × indicates it is not.

StreamEval breaks performance down into four dimensions and ten metrics.

Task execution focuses on dialogue performance, instruction following and complete editing; state consistency examines image stability, temporal continuity and preservation of non-target content; visual fidelity covers image quality, character identity and reference image matching; audio-visual coordination mainly evaluates audio-visual synchronization.

Each metric uses a 0–100 scale. For each task, only the applicable metrics are calculated, and then equally weighted to obtain a composite score within the task. For digital human tasks, systems that support custom avatars receive an additional 5 points. The five task categories are ranked separately and are not merged into a single cross-task overall leaderboard.

38 models submitted results, and continuous instructions are the hardest to score on

The paper evaluates 38 models in total, forming 49 model-task configurations. The results show that visual quality, character identity preservation, and real-time control are several relatively independent capabilities; a model being able to stably generate video does not mean it can reliably execute constantly changing instructions.

In the digital human conversation task, Vidu S2-Avatar's base overall score (Q-Overall) is 88.98, and after adding 5 points for custom avatar capability, its overall score (Overall) reaches 93.98, achieving the highest result among the evaluated models that support custom avatars.

Digital human conversation task evaluation results.

Moving to the virtual human performance task, Vidu S2-Avatar likewise achieves the highest overall score at 81.71, with its audio-visual synchronization, stability, and identity consistency reaching 99.60, 97.25, and 86.12 respectively. However, its instruction following (Following) is only 36.90. This score is already the highest among the seven evaluated models.

This means that the character can remain stable and the lip movements can keep up with the voice, yet specific actions may still fail to be completed as required. Continuous interaction imposes more complex demands on models: they must not only understand instructions but also make the right object change at the right time.

World generation also shows clear capability divergence.

MemFlow achieves the highest overall score at 33.36, Krea Realtime obtains the highest instruction following score at 25.93, and Matrix-Game 3.5 achieves the highest continuity (Continuity) score at 34.50. HappyOyster 1.0 Adventure's image quality and stability reach 58.34 and 56.67 respectively, both the highest values for this task, but its instruction following score is only 5.56, and its continuity score is only 2.78.

World generation. Model configurations under text conditions and image conditions are ranked together; when the same model is evaluated under both input conditions, only its evaluation results under the text condition are reported.

These results indicate that visual appearance, instruction response, and long-duration continuity need to be judged separately; a model looking more stable does not mean it follows commands better.

In the real-time editing section, Vidu S2-Editing achieves an overall score of 45.43 in the instruction editing task, with instruction following and edit success rate (Edit Success) at 72.10 and 57.20 respectively; JoyAI-Video-Edit's overall score is 42.39, but its content preservation (Preservation) score reaches 83.33, higher than Vidu S2-Editing's 60.93. The former is higher on instruction-execution-related metrics, while the latter is better at preserving non-target content in the source video.

Reference editing is more difficult. Vidu S2-Editing achieves the highest Overall at 39.02, with Preservation at 64.44 and Edit Success only 15.00. This score is already the highest among the four reference editing configurations publicly reported in the paper.

Reference image switching, modification scope control, preservation of existing state, and undo/restore need to be accomplished together within the same continuous generation. A model may successfully change a character's appearance while destroying the background; it may also preserve the original video content without adopting the new reference appearance. Only when all conditions are satisfied simultaneously can a full editing score be obtained.

Take the example where the system first loads a red velvet cloak, switches to a gold satin cloak at the 10th second, and at the 20th second requests removal of the edit: the model must change the outfit in time, maintain the character's movements and background, and finally precisely restore the original state of the source video.

Reference image editing during input changes. Reference image A is preloaded, reference image B is input at the 10th second, and at the 20th second an instruction to remove the reference image is issued. Each column pairs and displays the source video frame and the model output frame at the corresponding session time point. The local enlarged images show the boxed cloak region. The results show that in the two sets of outputs during the stage when reference image B is in effect, the cloak remains gray and does not turn gold satin as required.

Therefore, the focus presented by StreamEval is not who leads comprehensively across all tasks. Different models have their own advantages, and the common challenge facing the industry is how to stably execute continuous instructions during continuous generation and correctly maintain already-established state.

Conclusion

The value of StreamEval is not just ranking models.

Behind it are top academic and industrial institutions such as Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton, and NUS. The joint participation of multiple research forces determined that from the outset it is a public evaluation framework oriented toward the entire industry, serving the common needs of this field.

More critically, it is establishing a reproducible and sustainably evolving evaluation standard for real-time interactive video. Public test data and scoring rules allow developers to precisely locate models' weaknesses, and progress along different technical routes thus has a clear and verifiable basis. In a track where even definitions have not yet been unified, being able to provide such a public, transparent, and verifiable benchmark is itself rare.

The value of a leaderboard depends on how many models participate and whether the evaluation can be continuously updated. StreamEval plans to maintain the leaderboard over the long term, continuously incorporating new models and the latest evaluation results every month. As more research teams and developers join, test records will continue to accumulate, helping the entire industry more clearly judge where the technology has reached and which problems remain unresolved.

StreamEval's participation method is straightforward, in three steps.

Step one, obtain the code and data. Developers can view the evaluation process, model integration examples, and scoring rules through GitHub, and download the media files required for evaluation through the public dataset. The project provides complete task definitions, test timelines, and expected results, supporting reproduction of the evaluation under unified conditions.

Step two, integrate your own model. The project provides clear model interface specifications and currently supports two integration forms: in-process model interfaces and WebSocket services. For specific technical requirements, refer to the Add a Model documentation on GitHub.

Step three, submit model information and evaluation results. Research institutions, enterprise model teams, and individual developers can all submit model information through the "Evaluate your model" entry on the StreamEval official website (https://streameval.ai). Participants who have completed local testing according to the public process may also attach their own evaluation results and technical feedback. After review and verification, the results will be included in subsequent leaderboard releases.

The next step for real-time interactive video requires continuous breakthroughs from models and also an evaluation standard that can withstand scrutiny. If you are developing digital human, interactive world, or real-time video editing models, you might as well run StreamEval yourself, let your model undergo testing, and let more real results enter this continuously updated leaderboard, jointly contributing to the progress of the industry.

This article comes from the WeChat public account "Machine Heart" (ID: almosthuman2014), author: Yang Wen, published by 36Kr with authorization.