Research from the Jiangsu Provincial Key Laboratory of Language Computing and Applications Selected for ACM MM 2026: Enhancing Multimodal Large Model Capabilities with Tool-Augmented Reasoning

Recently, the 34th International Conference on Multimedia, ACM Multimedia 2026, announced its acceptance results, and several papers from the Jiangsu Provincial Key Laboratory of Language Computing and Applications were accepted by the conference. ACM MM is hosted by the Association for Computing Machinery (ACM) and is a Class A international academic conference recommended by the China Computer Federation. It has long been regarded as one of the most internationally influential academic conferences in the multimedia field, covering frontier directions such as video, audio, human-computer interaction, and multimodal intelligence. This year's conference will be held in Rio de Janeiro, Brazil, from November 10 to 14, 2026, and its accepted results collectively reflect the development direction of cross-modal information processing and multimodal large model technology.

The Jiangsu Provincial Key Laboratory of Language Computing and Applications (hereinafter referred to as "the Laboratory") is led by AISpeech Co., Ltd. and jointly built with Shanghai Jiao Tong University and Soochow University. It is an important strategic scientific and technological force in Jiangsu Province's layout in the field of general artificial intelligence. Relying on AISpeech's industrial platform, the Laboratory brings together top scientific research resources from Shanghai Jiao Tong University and Soochow University, forming an innovation community with deep integration of "industry, academia, research, and application." It focuses on core language computing technologies and connects the entire chain of innovation from basic theory and key algorithms to industrial implementation.

The accepted results focus on two core technical systems: multimodality and interaction, and large models and agents. Through tool-augmented reasoning, multimodal large models can see accurately and calculate correctly on complex visual information, continuously strengthening the Laboratory's core capabilities in fine-grained visual understanding, reliable tool invocation, and parameter-efficient modeling. The related progress strengthens the understanding and execution capabilities of AISpeech's dialogue foundation model DFM and the 1+N distributed agent system for mixed image-text information, supports intelligent interpretation of meeting materials and business reports in smart office scenarios, and provides a more trustworthy technical foundation for data analysis services for customers in finance, government, and other industries. The representative accepted results are introduced below:

Tool-Augmented Multimodal Chart Reasoning

This result addresses the pain point that charts in reports and presentation materials in office and government-enterprise scenarios are difficult to read and calculate accurately, directly supporting the customer value of more precise data interpretation and more trustworthy analytical conclusions.

Chart reasoning requires both precise localization of details in dense layouts and accurate calculation of numerical values. However, existing multimodal large models mostly rely on pure text reasoning, are easily affected by language priors and produce hallucinations, and high-quality training data is also relatively scarce. This work makes simultaneous breakthroughs at both the data and reasoning ends: on the one hand, it proposes a dual-source data construction pipeline, DuoChart, which integrates code-synthesized charts with real charts from scientific papers, and uses multiple agents to collaboratively generate and verify question-answer pairs, constructing a training set containing 100,000 high-difficulty question-answer pairs; on the other hand, it equips the model with two types of tools, image cropping and code calculation, and uses agent reinforcement learning to enable the model to autonomously invoke tools during reasoning, first focusing on local details and then completing precise calculations, while flexibly selecting the required tools according to task characteristics.

Experiments show that CharTool achieves consistent improvements on six chart reasoning benchmarks. On the real-world benchmark CharXiv reasoning task, CharTool-7B improves by 8.0 percentage points over the base model, and on ChartQAPro it improves by 9.78 percentage points. Its average score across the six benchmarks exceeds proprietary models such as GPT-4o and is slightly higher than open-source models with about ten times its parameter scale; it also achieves improvements on visual mathematical reasoning tasks without specialized training, demonstrating good generalization ability.

This result benefits AISpeech's core technical system at two levels. In the multimodality and interaction technical system, it further extends multimodal understanding from audio-visual signals such as speech, lip movement, and speakers to high-information-density visual content such as charts and reports, completing fine-grained perception and numerical reasoning capabilities in mixed image-text scenarios. In the large model and agent technical system, the tool-augmented reasoning of first focusing and then calculating is consistent with DFM's technical route that emphasizes tool invocation and reliable task execution: the model obtains visual evidence and calculation results by invoking external tools rather than relying on language priors to answer directly, which helps reduce hallucinations and improve the traceability of conclusions; achieving performance comparable to larger models at the 7B scale also aligns with AISpeech's technical direction of coordinating large and small models and achieving reliable execution at lower computing cost.

Product implementation:

From content organization to reliable execution, empowering smart office and industry agents

Smart office: A large amount of key information in meetings and presentations is presented in the form of charts and reports. The related capabilities can be further integrated into smart office products such as AISpeech AI office notebooks, as well as the OfficeClaw office architecture based on the distributed agent framework, so that meeting minutes not only record the content of speeches but also understand the data in meeting materials, automatically extract key indicators, compare trend changes, and generate directly usable key points, promoting the upgrade of office intelligence from content organization to reliable execution.

Industry services: For smart operation service customers in finance, government, and other industries, chart data in business reports and statistical materials can be accurately read, compared, and calculated, and the analysis process is traceable. The parameter-efficient feature helps lower the computing threshold for private deployment and provides safer and more trustworthy intelligent analysis services for data-sensitive government and enterprise customers.

Agent platform: The training paradigm of tool-augmented reasoning can provide reusable tool capabilities such as image reading and calculation for vertical-domain agents in the 1+N distributed agent system and the DUI middle platform 2.0, and is expected to extend to image and interface information understanding in smart mobility and smart IoT terminals, improving the reliability of task execution in multiple scenarios.

Relying on the related capabilities, AISpeech will further improve the speed of scenario adaptation and delivery efficiency in multi-industry projects, providing customers with more precise and trustworthy intelligent information services.

The Laboratory will continue to promote the transformation of high-level scientific research results into product capabilities, polish implementable, scalable, and continuously optimizable multimodal and language computing technical capabilities around real business scenarios, and help AISpeech serve key scenarios such as smart mobility, smart office, and smart IoT.