InternLM 发布 STAR-Bench：评测音频 4D 智能的时空推理基准

InternLM; StarBench

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork5 (5); Star46 (46)You must be signed in to star a repository

Repository files navigation

README; MIT license

* Equal Contribution. †Corresponding authors.

📖arXiv | 🌐Homepage | 🤗Dataset

🚀 [10/28/2025] We have released the STAR-Bench 🏠repository and 🌐homepage; 🚀 [10/28/2025] STAR-Bench v1.0 is now available on 🤗HuggingFace! Compared with v0.5 (introduced in our arXiv paper), v1.0 features revised and refined Questions & Answers for improved clarity and quality for spatial tasks. 📌 Please cite this version (v1.0) when reporting results going forward. The leaderboard will be updated soon

🌈Overview

We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space, and introduce a STAR-Bench to measure it. STAR-Bench combines a Foundational Acoustic Perceptionsetting (six attributes under absolute and relative regimes) with a Holistic Spatio-Temporal Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories.

Benchmark examples are illustrated below. You can also visit the 🌐homepage for a more intuitive overview.

📊Results and Analysis

Evaluation results of various models on STAR-Bench v0.5 are shown below. The leaderboard for v1.0 will be released soon.

💡 Key Insights

🔥A clear capability hierarchy between the two groups. Closed-source models are bottlenecked by fine-grained perception, while open-source models lag across perception, knowledge, and reasoning; 🔥 Enhancing dense audio captioning. Open-source models struggle to produce dense, fine-grained captions, which limits their perceptual sensitivity and ability to extract embedded knowledge. Bridging this gap is a crucial first step; 🔥 Improving multi-audio reasoning. Open-source models lag significantly in comparing, integrating, and grounding information across multiple audio clips; 🔥 Moving beyond channel-averaged audio preprocessing. The common practice of averaging multi-channel audio into a mono signal is a major bottleneck for spatial reasoning. Developing architectures that natively process multi-channel cues is essential for unlocking genuine spatial awareness

⚙️Data Curation

🛠️ Test Your Model!

The ALMEval_code/ is partially adapted from VLMEvalKit and Kimi-Audio-Evalkit.

It provides a unified evaluation pipeline for multimodal large models on STAR-Bench.

Step 1: Prepare Environment

Step 2: Get STAR-Bench v1.0 Dataset

Download STAR-Bench v1.0 dataset from 🤗HuggingFace

Currently supported models include: Qwen2.5-Omni, Qwen2-Audio-Instruct, DeSTA2.5-Audio, Phi4-MM, Kimi-Audio, MiDashengLM, Step-Audio-2-mini, Gemma-3n-E4B-it, Gemini and GPT-4o Audio.

To integrate a new model, create a new file yourmodel.py under the models/ directory and implement the function generate_inner().

✅ Example: generate_inner()

Step 4: Configure Model Settings

Modify the configuration file: /models/model.yaml.

For existing models, you may need to update parameters such as model_path to match your local model weight path.

To add a new model variant, follow these steps:

Create a new top-level key for your alias (e.g., 'my_model_variant:'); Set 'base_model' to the NAME attribute of the corresponding Python class; Add any necessary arguments for the class's __init__ method under init_args

Evaluation results will be automatically saved to the ./eval_results directory.

You can also evaluate specific subtasks or their combinations by modifying the --data argument. The full list of available task names can be found in ALMEval_code/datasets/__init__.py.

Example: Evaluate only the temporal reasoning and spatial reasoning tasks:

✒️Citation

📄 License

Usage and License Notices: The data and code are intended and licensed for research use only.

Acknowledgement

We sincerely thank 2077AI for providing the platform that supported our data annotation, verification, and review processes.