InternLM 发布 OVO-S-Bench：面向多模态 LLM 流式空间智能的分层基准（EMNLP 2026 Oral）

InternLM; OVO-S-Bench

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork0 (0); Star56 (56)You must be signed in to star a repository

Repository files navigation

README; MIT license

🔥🔥OVO-S-Bench is accepted by EMNLP 2026!🔥🔥

†Equal Contribution *Project Leader

Overview of OVO-S-Bench. The benchmark evaluates streaming spatial understanding across four levels, from instantaneous egocentric perception and spatiotemporal context tracking to generative spatial reasoning and global topological mapping. The right panel summarizes representative model behavior across task families.

Abstract

Multimodal agents in robotics, AR, and autonomous driving must reason about places and layouts from continuous egocentric streams, often using evidence outside the current view. Existing benchmarks either evaluate offline over full videos or target events rather than spatial structure. We introduce OVO-S-Bench, a fully human-annotated benchmark for streaming spatial intelligence, comprising 1,680 questions over 348 source videos. Annotation involves 12 trained annotators (each also serving as a blind cross-reviewer) across roughly 804 person-hours of multi-round quality assurance. Each question carries a query timestamp and an evidence interval, and at evaluation, the model sees only the prefix preceding the query. Questions span four levels of increasing abstraction: instantaneous egocentric perception, spatiotemporal context tracking, spatial simulation and reasoning, and allocentric mapping. Across 38 proprietary and open-source MLLMs, Gemini-3.1-Pro trails human experts by 27 points (59.2 vs. 86.6), with allocentric mapping as the dominant bottleneck. Notably, streaming and spatially fine-tuned MLLMs underperform their own backbones. We further find that chain-of-thought reasoning amplifies spatial errors when ungrounded in the stream. By exposing these limitations, OVO-S-Bench establishes a demanding testbed for next-generation streaming spatial MLLMs.

Four-Level Streaming Spatial Taxonomy

OVO-S-Bench organizes questions into four cumulative levels by the spatial state a model must access at query time. The progression goes from evidence directly available in the current view to allocentric map queries that require cross-viewpoint integration, reflecting a gradient of persistence and abstraction.

The released benchmark comprises 1,680 questions over 348 source videos from 9 datasets, organized into 30 canonical task types across four levels. Mean prefix at query time: 8.8 min. Evidence-span medians by level: L1 2.0 s · L2 36.8 s · L3 2.0 s · L4 278.7 s — reflecting the spatial persistence each level demands.

Representative OVO-S-Bench examples. Each card pairs a spatial question with visual evidence, illustrating the progression from current-view perception to allocentric mapping.

Taxonomy and benchmark statistics. Left: four-level spatial taxonomy. Right: task-family counts, source distribution, and evidence-interval lengths by level.

L4.3 trajectory-matching questions ship their option images embedded inline as base64 data URIs in the parquet's options column. No separate image-asset download is needed.

Benchmark Construction

Video sources. OVO-S-Bench draws from 9 publicly available or accessible sources covering five regimes: indoor walkthroughs (RoomTour3D), egocentric activities (Ego4D), outdoor/world scenes (Sekai, OmniWorld, YouTube walking tours), driving videos (CODa, Honda HDD), and spatially annotated 3D environments (ARKitScenes, VSI-Bench); Human annotators write every item. Annotators with 3D-vision backgrounds choose clips with stable motion, clear viewpoints, and enough spatial variation for the target level. For each item, they record the video, task label, question, options, answer, query timestamp, and evidence interval; Each item follows the streaming setting. The answer must be derivable from the video prefix before the query timestamp. Annotators mark the shortest interval that contains the needed evidence and write distractors that are plausible under the visual context but wrong under the annotated evidence; Quality control removes shortcuts. A text-only LLM probe flags items that leak the answer through wording, common sense, or option asymmetry. A second annotator then cross-reviews each item without seeing the original answer, checking that the answer and evidence interval are sufficient. Recurring problems are folded back into the annotation guideline

Key Findings

Six observations about the current state of streaming spatial intelligence, from 38 evaluated systems on OVO-S-Bench:

Leaderboard

Main results under the streaming protocol (multiple-choice accuracy). Numbers replicated from the paper's main results table. The public dataset release is linked above; a live submission portal will follow.

Interactive version with sorting + per-category drill-down: https://internlm.github.io/OVO-S-Bench/

The complete annotations parquet and source videos are currently available from ModelScope:

Layout you should see after download:

Configure API keys

Run

API model (single-process, threaded):

Open-source MLLM via vLLM (multi-GPU auto-sharded; default 128 uniformly-sampled frames per prefix):

Both scripts call inference.py then score.py. Inference auto-resumes on rerun — interrupted jobs pick up from the last checkpoint.

Add a new model

See docs/adding_models.md. The contract is BaseModel.inference(frames: List[PIL.Image], prompt: str) -> str; add a config entry to config.yaml::MODELS and register the wrapper in models/api_models.py or models/vllm_models.py.

Custom API endpoint

Set *_BASE_URL in .env to point at your proxy or local vLLM server. API providers (openai, gemini-native, anthropic) honor the standard *_BASE_URL env var convention.

Evaluation Protocol

All systems are evaluated under a unified streaming protocol: each source video is truncated at the annotated query timestamp $t_q$, and the model receives 128 frames uniformly sampled from the resulting prefix together with the question and multiple-choice options. For streaming-architecture models that implement a native sequential ingestion path, we instead feed the video at each model's published streaming rate and query the resulting compressed state. No model sees frames after $t_q$. Answers are extracted by regular expression without further post-processing.

Per-query responses land under results/ /ovo_s_bench.json. score.py aggregates accuracy by main category and subcategory:

For sharded multi-GPU runs, merge_results.py consolidates *_rank{N}.json shards into a single result file (auto-invoked by launch.py).

OVO-S-Bench draws videos from publicly released datasets — many thanks to: Ego4D, RoomTour3D, CODa, OmniWorld, VSI-Bench, Sekai, ARKitScenes, Honda HDD, and selected YouTube walking tours.

The evaluation framework integrates wrappers for several token-compression and streaming MLLM research repos: HERMES, FluxMem, StreamingTOM, InfiniPot-V, InfiniteVL, Flash-VStream, StreamForest, Spatial-MLLM, Cambrian-S, SenseNova-SI, Spatial-TTT.

Project page template adapted from OVO-Bench, which itself builds on the Nerfies template.

License

MIT — see LICENSE. Annotations released under CC-BY-4.0; source videos retain their original licenses.