AI News

Tsinghua proposes SkillMaster: training self-evolving skills into the Agent's brain, paper accepted by NeurIPS 2026

Sina TechUpdated Nov 16 20:59

AI summary

Li Yong's team at Tsinghua University proposes SkillMaster, which incorporates skill creation, updating, and retention into the agent's own policy and trains it together with task execution using reinforcement learning; the paper has been accepted by NeurIPS 2026. Experiments based on Qwen2.5-7B-Instruct show that the task success rates on ALFWorld and WebShop reach 98.7% and 82.0%, respectively, improving by 8.8 and 9.3 percentage points over SkillRL. The overall score on SearchQA is 48.0%, 0.9 percentage points higher than SkillRL.

Body · AI translation

Add crowdsourcing testing promotional copy Add crowdsourcing testing promotional copy end Add App-exclusive traffic guidance

With the rise of Recursive Self-Improvement (RSI) research, the long-term capability evolution of agents has attracted more attention. Skills can distill interaction experience into reusable operational procedures and task strategies, allowing the gains from one task to serve subsequent tasks. Enabling agents to learn to autonomously create and improve these skills provides a concrete path for their continuous evolution.

Figure 1 Left: External modules manage skills, and the agent retrieves and uses skills. Right: SkillMaster incorporates skill management into the agent's own policy, learning jointly with task execution.

However, skill creation and updating in existing methods usually rely on external teacher models, preset rules, or auxiliary modules (Figure 1 Left). Agents that execute tasks learn to use skills, but are rarely trained directly to improve skills. To achieve autonomous improvement along this path, it is also necessary to make "how to evolve skills" a capability that the agent itself can learn (Figure 1 Right).

Professor Li Yong's team at Tsinghua University proposed SkillMaster, and the related paper has been accepted by NeurIPS 2026. The first author is Min Yang, a doctoral student at Beijing Zhongguancun College.

The research starts from an intuitive insight: valuable skill improvements should make similar tasks that previously failed more likely to succeed later, and make tasks that already succeeded complete in fewer steps. SkillMaster turns these two types of effects into learning signals for skill editing, training the same agent to create, update, and retain skills based on interaction experience, so that skill evolution and task execution jointly enter the reinforcement learning process.

Author: Min Yang, Jinghua Piao, Xu Xia, Xiaochong Lan, Jiaju Chen, Yongshun Gong, Yong Li

Core insight: make failed tasks easier to succeed

make successful tasks more efficient to complete

To train an agent to improve skills, it is first necessary to know what a modification brings to subsequent tasks. Using only whether the task ultimately succeeds as the reward makes it difficult to distinguish execution ability from skill quality, and even harder to provide clear feedback for a particular skill edit.

To this end, SkillMaster focuses on the actual impact of skill modifications on subsequent related tasks. If a skill fills a gap in existing knowledge, it should increase the chance of success on tasks that previously failed; if it improves an existing operational strategy, it should reduce unnecessary steps for tasks that can already be solved. These two observable changes provide a basis for measuring the effect of skill modifications.

Method framework:

Jointly learning task execution and skill evolution

To enable an agent to learn to autonomously create, revise, and retain skills, three problems need to be solved: on what basis to propose modifications, how to evaluate the value of modifications, and how to jointly optimize task execution and skill management. SkillMaster implements these three components through trajectory-based skill review, downstream utility rewards, and DualAdv-GRPO.

Figure 2 The three components of SkillMaster: trajectory-based skill review, counterfactual skill utility rewards, and DualAdv-GRPO joint optimization.

Learning skill management from interaction trajectories

In the skill review stage, the task goal, retrieved skills, action and observation trajectories, and final feedback are provided to the model together. The model performs creation, update, and retention operations through three types of tool calls: propose_skill, update_skill, and keep_skill. For example, when it finds that existing skills do not cover a certain search strategy, it can add a new skill; when it observes that actual feedback conflicts with existing operational rules, it can revise the original skill.

Task actions and skill tool calls are generated by the same policy, so skill management is also directly incorporated into model training. SkillMaster first uses supervised fine-tuning to give the model basic task execution and tool-calling capabilities, and then uses skill utility feedback to optimize decisions on creating, updating, and retaining skills through reinforcement learning.

Evaluating skill utility through counterfactual comparison

For a candidate skill edit, the system selects a set of related probe tasks and executes them using the original skill library and the temporarily modified skill library, respectively, directly comparing the downstream changes brought by the modification. Whether previously failed tasks turn into successes and whether previously successful tasks reduce execution steps are both reflected in the utility score, providing direct training feedback for skill editing.

Probe tasks come from the training task pool and are selected based on task category, product category, or question semantic similarity. The system computes the utility reward by considering the average score change of probe tasks as well as the number of tasks that improve and degrade, and optimizes skill editing decisions accordingly. During training, only modifications with positive utility rewards are written into the persistent skill library.

DualAdv-GRPO jointly optimizes two types of decisions

Task execution mainly relies on feedback from environmental outcomes, while skill management uses feedback on tool-call effectiveness and downstream utility; the two types of rewards have different scales and meanings. To this end, SkillMaster proposes DualAdv-GRPO, which separately computes the advantages of task execution and skill management, then assigns them to the corresponding generated content to jointly update the same policy. The model thereby simultaneously learns to execute tasks effectively and to improve reusable skills.

Experimental results:

From task performance to skill evolution capability

The experiments are based on Qwen2.5-7B-Instruct and cover ALFWorld, WebShop, and SearchQA. Among them, ALFWorld contains six types of simulated household tasks, WebShop examines multi-step product search and purchase, and SearchQA covers seven single-hop and multi-hop question answering datasets. Table 1 Main experimental results on ALFWorld and WebShop.

As shown in Table 1, SkillMaster achieves the best performance among the compared methods on both ALFWorld and WebShop, with task success rates reaching 98.7% and 82.0%, respectively, improving over SkillRL by 8.8 and 9.3 percentage points. Four of the six ALFWorld task types reach a 100% success rate, and the remaining two both exceed 95%. Table 2 Experimental results on the seven SearchQA datasets.

As shown in Table 2, on SearchQA, SkillMaster achieves an overall score of 48.0%, the highest among the compared methods, improving over SkillRL by 0.9 percentage points. It outperforms SkillRL on five of the seven datasets, demonstrating the effectiveness of this framework in search-augmented question answering tasks. Figure 3 Left: component ablation results. Middle: changes in the proportion of skill management operations. Right: changes in the size of the skill library.

As shown on the left side of Figure 3, removing the skill utility reward, switching to merged advantage normalization, or randomly selecting probe tasks all reduce performance, supporting the roles of utility feedback, separate advantage estimation, and related task selection. The middle and right sides of Figure 3 show changes in skill management behavior and the size of the skill library, respectively.

During ALFWorld training, creation and update operations are relatively active in the early stage, and in the later stage, the proportion of retained skills rises from about 81.25% to 97.66%. This process reflects the behavioral shift of the model from supplementing and revising skills to gradually retaining accumulated skills. The authors also fix the trained model parameters to test its ability to build skills during test-time interaction. At this point, counterfactual probing and utility feedback are not used; legitimate skill management operations are executed directly and reused for subsequent tasks. On WebShop, starting from an empty skill library and enabling both skill construction and retrieval, the success rate rises from 65.63% to 71.88%, indicating that the trained policy can generate and reuse useful skills based on new interactions.

Case analysis:

Creating and revising skills based on execution evidence Figure 4 The agent discovers skill gaps from task trajectories, corrects existing rules, and verifies the effect of modifications on related tasks.

As shown in Figure 4, in an ALFWorld cooling task, the agent spends a large number of steps on cabinets and drawers with a low probability of containing food, and ultimately fails to complete the task. During review, it identifies that the existing skills lack search priority rules, so it creates a new skill to guide subsequent searches to prioritize high-probability locations related to the target object.

Another case demonstrates the revision of an existing skill. According to ALFWorld's interaction interface, heating needs to be performed while the agent is holding the target object, but the old skill required placing the object into the microwave first. The model revises this operating condition based on feedback from the trajectory. The comparison on the right side of Figure 4 shows that in related probe tasks, the heating step is shortened from 8 steps to 2 steps, eliminating six redundant operations, directly reflecting the downstream utility of the skill modification.

Research outlook:

Exploring recursive self-improvement through learning skill evolution

Starting from skills as a reusable carrier of knowledge, SkillMaster explores one path toward recursive self-improvement of agents. The agent discovers knowledge gaps or inefficient strategies during interaction, changes subsequent behavior through skill editing, and then learns more valuable editing decisions based on actual effects. Task execution and skill improvement are interconnected, enabling the agent to both accumulate skills and improve its ability to improve skills through training.

The current method has been validated in structured interaction environments and search-augmented question answering. To further move toward open environments, it is necessary to explore more effective probe task selection and lower-cost utility evaluation, and to extend to skill deletion, cross-agent sharing, and continual learning under distribution shifts. These directions are expected to enable the skill library and the agent's capabilities to co-evolve with long-term interaction.article_adlist[

]article_adlistarticle_adlist[

For reprinting, please contact this official account for authorization

Submission or reporting requests: liyazhou@jiqizhixin.com ]article_adlist Statement: This is an exclusive article of Sina.com; unauthorized reproduction is prohibited. tech_web_article_v2015_block_bottom non-targeted 300*250 button 17/09 wenjing begin

non-targeted 300*250 button end

Source

Sina Tech