CAICT, Lenovo, and the AIIA Data Committee Release the "Guidelines for Building High-Quality Datasets for Agent Capability Construction (2026)"

In October 2026, at the AI Dataset and Data Annotation High-Quality Development Forum of the Artificial Intelligence Industry Conference, the Institute of Artificial Intelligence of the China Academy of Information and Communications Technology (CAICT), together with Lenovo (Beijing) Co., Ltd. and the Data Committee of the Artificial Intelligence Industry Alliance of China (AIIA), officially released the "Guidelines for Building High-Quality Datasets for Agent Capability Construction (2026)".

At present, artificial intelligence is accelerating its transition from being "driven by model capabilities" to being "driven by application capabilities". As an important vehicle connecting large model capabilities with complex application scenarios, AI agents are gradually becoming an important form for promoting the large-scale deployment of artificial intelligence and deep industrial integration. As agents evolve from knowledge question answering toward complex capabilities such as task planning, reasoning and decision-making, tool invocation, multi-turn interaction, and autonomous execution, traditional datasets based mainly on static samples can no longer fully meet the needs of agent capability construction, and data systems are showing new characteristics such as multimodal integration, process orientation, scenario-driven development, and continuous evolution. At the same time, the industry still faces problems such as a mismatch between data supply and application demand, data silos, a lack of annotation specifications, an outdated quality evaluation system, and security and compliance issues. The construction of high-quality datasets urgently needs to shift from traditional data production to systematic construction oriented toward capabilities and scenarios.

Focusing on agent development trends and the new requirements they place on data construction, the report systematically analyzes the data needs for agent capability construction, proposes an overall framework, key methods, and implementation paths for building industry high-quality datasets for agent capability construction, and, drawing on typical practices in fields such as enterprise operations and maintenance, healthcare, communications, intellectual property, automobiles, industrial parks, and scientific research, provides reference basis and implementation paths for all parties in the industry to build high-quality agent datasets. Core viewpoints of the report

1. Agent development is accelerating the transformation of the paradigm for building high-quality datasets

The report points out that as AI development evolves from being driven by model capabilities to being driven by application capabilities, the role of data in agent capability construction is extending from supporting model training to supporting the execution of complex tasks. Compared with traditional model training data, the data required by agents must not only provide knowledge information but also support task understanding, reasoning and decision-making, tool invocation, and result feedback, driving the construction of high-quality datasets to shift from being mainly knowledge-supply oriented to being oriented toward specific tasks and serving practical applications.

2. The development of agents' multidimensional capabilities places systematic requirements on high-quality datasets

Focusing on six core capabilities—environmental perception, task planning, tool invocation, multi-turn interaction, safety and controllability, and industry application—the report systematically analyzes the data types and construction priorities required for building different capabilities. Among them, environmental perception requires multi-source heterogeneous contextual data, task planning requires reasoning chains and decision trajectory data, tool invocation requires tool usage and API interaction data, multi-turn interaction requires context management and dialogue data, safety and controllability require risk scenario and value alignment data, and industry application requires domain knowledge and real-scenario data.

3. Building high-quality datasets requires coordinating data quality, capability fit, and application feedback

The report proposes that dataset construction for agent capability building should run through key links such as data collection, cleaning, annotation, evaluation, and optimization, focusing not only on basic quality such as data completeness, accuracy, and consistency but also on examining the supporting effect of data on improving agent capabilities and completing specific tasks. On this basis, by combining the actual application performance of agents with data quality evaluation, and through task performance observation, quality problem identification, data optimization iteration, and feedback verification, a continuous optimization mechanism in which data construction and application effects reinforce each other is formed, promoting the evolution of dataset quality management from static evaluation to dynamic iteration.

4. The construction of industry high-quality datasets is developing toward multimodal integration, dynamic evolution, and service-oriented application

The report believes that industry high-quality datasets will continue to evolve in aspects such as data form, update mechanisms, application methods, and organizational methods. As agents gradually penetrate industry business processes, industry datasets will integrate multiple types of data such as images, curves, formulas, models, and literature, and data updates will also shift from periodic maintenance to continuous iteration combined with application feedback. At the application level, the value of datasets will extend from storage, archiving, and retrieval to question answering, recommendation, analysis assistance, and business process support, promoting the development of industry data resources from decentralized storage toward structured governance, standardized organization, and controllable sharing.

5. Promoting the development of high-quality agent datasets requires building a multi-party collaborative industrial ecosystem

The report believes that the construction of high-quality agent datasets involves multiple links such as data supply, technology research and development, annotation processing, quality evaluation, security governance, and application operations, and requires the coordinated advancement of policies, standards, industry, and enterprises. It is necessary to strengthen the overall planning of data resources around key industries and key application scenarios, improve standards systems for dataset construction, quality evaluation, annotation specifications, and security governance; promote industry entities to strengthen data engineering capacity building, while strengthening data collaboration and ecosystem cooperation between industries and between enterprises, and gradually form a full-chain support system covering construction, evaluation, application, and iteration, so as to consolidate the data foundation for the large-scale application of agents in industry scenarios. Report table of contents

III. Overall framework and key methods for building industry high-quality datasets for agent capability construction

The main research directions are identification of the data annotation industry, development of methods and tools for high-quality dataset quality evaluation, and construction of standards systems. Led the drafting of more than 10 AI industry standards, led and supported the drafting of a number of ministerial policy documents, and completed a number of research projects and research reports in the AI field.

Li Sun, Deputy Director and Senior Engineer of the Platform and Engineering Department of the Institute of Artificial Intelligence at the China Academy of Information and Communications Technology (CAICT)

Director of the Data Committee of the China Artificial Intelligence Industry Development Alliance. Mainly engaged in research on AI policy, standards, and industry, focusing on fields such as multimodal perception and trusted data governance. Has participated in industry research and planning projects of national ministries and commissions, led and participated in the drafting of multiple international and industry standards for AI technical services, and has published/applied for more than 10 academic papers, patents, and software copyrights in total.

Fan Wei, Senior Engineer of the Institute of Artificial Intelligence at CAICT

Deputy Director of the Data Committee of the China Artificial Intelligence Industry Development Alliance. Has long been engaged in research on AI, high-quality datasets, and policies, standards, and industry related to data annotation, focusing on fields such as AI data governance. Has led and supported the drafting of multiple ministerial policy documents, completed multiple research projects and research reports in the AI field, published more than 30 academic papers in total, and received one Beijing Municipal Science and Technology Award.

Cao Feng, Director and Senior Engineer of the Platform and Engineering Department of the Institute of Artificial Intelligence at CAICT

Chair of the Internet Application General and Artificial Intelligence Working Group of the China Communications Standards Association. Has long supported the work of ministries and commissions such as the Ministry of Industry and Information Technology, the National Development and Reform Commission, and the National Data Administration, participated in the drafting, formulation, and promotion of implementation of multiple national-level AI policies and industry reports, and led more than 60 international, industry, and association standards on AI.

Compilation Notes

This research report was initiated in April 2026 and is divided into five stages: preliminary research, framework design, draft writing, solicitation of comments, and revision and improvement. This guide focuses on the new data requirements for agent capability construction, addresses the problems faced in building agent datasets, systematically proposes the overall framework, construction process, key technologies, and quality assurance methods for building high-quality datasets for agent capability construction, and forms practical references in combination with typical application scenarios, providing methodological guidance and implementation references for carrying out the construction of high-quality datasets for agents.

This report was jointly written by the Institute of Artificial Intelligence of CAICT and Lenovo (Beijing) Co., Ltd.

Participating units: Neusoft Group, Dongfeng Motor Finance, Inspur Zhuoshu, China Telecom Digital Intelligence, iFlytek, Shanghai Institute of Microsystem and Information Technology of the Chinese Academy of Sciences, Hefei Data Space Research Institute, Shanghai Advanced Research Institute of the Chinese Academy of Sciences, School of Economics and Management of Beijing University of Posts and Telecommunications, Liying Digital Intelligence, Speechocean, Whale Cloud Technology, Suzhou Rail Transit Group Co., Ltd., Guangzhou CEPREI Certification Center, Beijing Changan Automobile Engineering Technology Research Co., Ltd., DataBaker (Qingdao) Technology, Aerospace Yunlu, CAIC Intelligent Manufacturing, East Money, Yangzhou Municipal Data Bureau, Beijing University of Technology Research Institute, Beijing Kejie Technology Co., Ltd., Guangyu Mingdao Digital Technology Co., Ltd., Anhui Feishu Information, China Judicial Big Data Research Institute, Zhongke Siou (Hefei), Jijia Inheritance and Innovation (Beijing) Technology Co., Ltd., and Data Elements Society.

Source: China Academy of Information and Communications Technology, Lenovo (Beijing) Co., Ltd., and the Data Committee of the China Artificial Intelligence Industry Development Alliance

Contact information of the writing team:

CAICT Institute of Artificial Intelligence, Ms. Li, liwenjing1@caict.ac.cn, 13713513537 Report download

Guidelines for Building High-Quality Datasets for Agent Capability Construction (2026).pdf www.smartcity.team

Disclaimer: Please indicate when reprinting that the source is "Smart City Industry Analysis (www.smartcity.team)" and the relevant author. Except for original content, some information on this site comes from the Internet. If the relevant content involves infringement of rights, please contact the contact information at the bottom to delete it.

[Digital Technology Special Research] Community

In-depth peer exchange, focusing on research and material accumulation in the smart city and digital technology industries.

Continuously updating the latest industry white papers, industry research, policy developments, and implementation solutions to help industry practitioners efficiently accumulate knowledge and share experience.

Massive premium content is continuously iterated, providing frontier perspectives, connecting industry talent, carrying out case co-creation and resource exchange, and jointly building a high-quality industry exchange atmosphere.

#post-##

Recommended for you

CAICT Planning Institute's "Research Report on the Development of Intelligent Computing Infrastructure for Token Services (2026)": systematically reviews the paradigm reconstruction process of intelligent computing infrastructure from "computing power factories" to "Token factories"

Token Economy Development Research Report (2026), with an interpretation PPT by Li Wei, Director of the Cloud and Big Data Institute

CAICT Cloud and Big Data Institute's "Engineering of Steering: Research Report on Empowering Intelligent Native Software Engineering (2026)"

China Embodied Intelligence Data Industry Analysis Report 2026: detailed explanation of the current status, ecosystem, business models, typical enterprises, and future development trends of China's embodied intelligence data industry

Industrial Agent Development Report: led and released by the China Academy of Industrial Internet, detailing the industrial agent system architecture, core technologies, innovative practices in five major links, and key work

Interpretation of Huawei and the State Information Center's "AI CITY Development Research Report 2026" — building an AI-native smart city and creating a new paradigm for urban evolution