东锡推荐 TreeOPSD 长程智能体强化学习论文

(1/5) 新预印本：TreeOPSD，面向长程 LLM 智能体的无 critic 强化学习 🌳 长程智能体在经历多轮交互后只收到一个最终结果奖励，而每个 rollout 批次会依次进行多次小批量更新，此时策略已经偏离了采样时的策略。 TreeOPSD 将终端奖励沿 rollout 树回传，并把每次更新拆分为两步：为每一轮构造一个目标，然后用前向 KL 让当前策略匹配该目标。 这带来了： • 闭式、有界的轮次目标，且与树回传可交换，因此共享前缀只训练一次且不损失信用分配 • 在第一个小批量上精确的树策略梯度，且更新在到达目标时停止、在过冲后反向 • 相比 GRPO、GSPO 和匹配的树基线取得一致增益，在最长程任务上增益最大 在 Deep Research 上，TreeOPSD 将 Qwen3-30B-A3B-Thinking-2507 的平均准确率从 16.23（GRPO）提升到 21.20。 标题：Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning 论文：https://t.co/kOqeI03FnB 与 @TencentHunyuan 合作 #LLM #RLVR #AIAgents

查看原文

(1/5) New preprint: TreeOPSD, critic-free RL for long-horizon LLM agents 🌳 Long-horizon agents receive a single outcome reward after many turns, and each rollout batch feeds several sequential mini-batch updates while the policy drifts from the one that sampled it. TreeOPSD backs terminal rewards up a rollout tree and splits each update into two steps: construct a target for every turn, then match the current policy to it with forward KL. This leads to: • closed-form, bounded turn targets that commute with tree backup, so shared prefixes are trained once with no loss of credit • the exact tree policy gradient on the first mini-batch, and updates that stop at the target and reverse after overshoot • consistent gains over GRPO, GSPO and a matched tree baseline, largest on the longest-horizon tasks On Deep Research, TreeOPSD lifts Qwen3-30B-A3B-Thinking-2507 from 16.23 (GRPO) to 21.20 average accuracy. Title: Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning Paper: https://t.co/kOqeI03FnB Together with @TencentHunyuan #LLM #RLVR #AIAgents