InternLM 发布 SciDocBench：面向科学文档理解的 workflow 基准

InternLM; SciDocBench

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork0 (0); Star6 (6)You must be signed in to star a repository

Official repository for SciDocBench: a workflow-centered benchmark for scientific document understanding.

Repository files navigation

README

* Equal contribution. † Corresponding author.

2026-09-28: We released the supporting code for benchmark authoring and review, SciDocIR construction, training-data generation, and evaluation, with an offline quick start and paper-to-code map; 2026-09-15: We released the complete SciDocBench benchmark and document assets on Hugging Face; 2026-09-12: We released the SciDocBench SFT and RL training data on Hugging Face; 2026-09-04: Our SciDocBench paper is available on arXiv; 2026-09-04: The official repository is initialized; 2026-09-04: SciDocBench is integrated into VLMEvalKit. We recommend using VLMEvalKit for standardized inference and evaluation

Overview

Scientific papers combine text, equations, figures, tables, appendices, citations, code, and datasets. Reliable scientific-document assistants must therefore do more than retrieve visible text: they must locate evidence, verify numerical and logical relations, recover scientific structure, integrate information across documents, and produce reusable outputs.

SciDocBench evaluates selected evidence-grounded operations and their explicit outputs, from locating support to cross-paper synthesis. It contains:

124 manually designed and difficulty-screened questions; 7 scientific-document capability groups and 19 subtasks; 5 scientific domains; 2 question languages: English and Chinese; 2 document representations: All Images First and Markdown Interleaved; 496 matched evaluation instances in total; 3 evaluator families: rule-based, LLM-as-a-judge, and execution-based evaluation

Each question is instantiated under four matched settings while preserving its task semantics and evaluation criteria:

Capability Taxonomy

Leaderboard

Overall scores are computed over all 496 evaluation instances on a 0-100 scale. Failed or unusable responses receive zero.

Claude Opus 5 currently ranks first with 62.60, followed by GPT-5.6-Sol with 61.00 and Gemini 3.6 Flash with 59.87. No evaluated model reaches 63. Capability leaders are distributed across model families, and overall scores can conceal substantial differences between scientific-document skills.

Benchmark Data

The complete SciDocBench benchmark is publicly available on Hugging Face. The release contains 496 evaluation instances derived from 124 semantic questions, with 124 instances in each language-representation setting. It includes 7,052 image references and 2,758 unique document images after SHA-256 deduplication.

The repository provides a Hugging Face Parquet test split, a portable VLMEvalKit-compatible SciDocBench.tsv, content-addressed image assets, and integrity manifests with file and image checksums. Download it with:

For an offline VLMEvalKit installation, materialize the benchmark under LMUData:

Training Data

The SciDocBench Training Data release provides benchmark-oriented multimodal data for supervised fine-tuning and reinforcement learning:

The SFT data contain 981 seeds, each expanded into four language-representation settings. Across SFT and RL, the release contains 55,603 unique images and 120,489 image references. It preserves ordered multimodal conversations, bilingual questions, document-input variants, SFT reasoning traces, and RL solutions used by the reward implementation. JSONL files use portable, content-addressed image paths; images are distributed in tar shards.

Download and materialize the dataset with:

The preparation script verifies assets and writes machine-local training files under runtime/, including LLaMA-Factory registration and ms-swift RL inputs. The portable source files remain under data/sft/ and data/rl/. See the dataset card for field definitions and licensing notes, and experimental settings for the training mixtures used in the reported experiments.

Construction and Training Code

The supporting materials provide benchmark authoring and review interfaces, document processing, task generation, training-data assembly, and the SciDocBench evaluation adapter.

The paper-to-code map identifies the implementation and required inputs for each pipeline stage. To run the portable construction demo without an API or GPU:

The demo uses synthetic fixtures. Document assets, model weights, and API credentials are configured separately for full construction and training workflows.

Evaluation with VLMEvalKit

SciDocBench is available in VLMEvalKit, which is the recommended evaluation entry point. After downloading the benchmark automatically or materializing the offline release as shown above, run:

Replace with a model registered in VLMEvalKit. The command performs inference and evaluation. Configure the credentials required by the selected model and semantic judge according to the VLMEvalKit documentation.

The evaluation guide documents final-answer extraction, question-specific rule and judge routing, and the execution evaluator. The supporting adapter pins the public benchmark revision and checksum. Review local changes before copying it into an existing VLMEvalKit checkout.

Paper

Shenxi Wu, Yuhong Liu, Haosong Zhang, Tongjin Zou, Yanxun Zhang, Gaochang Chen, Dun Liang, Jiaqi Wang, Zhecan James Wang, Yuhang Zang, and Dahua Lin.

Please cite our paper as follows:

Acknowledgements

We thank the VLMEvalKit team for providing the standardized evaluation framework.

License

See the supporting-code notices for third-party components and attribution.

Official repository for SciDocBench: a workflow-centered benchmark for scientific document understanding.