Dongxi recommends the TreeOPSD long-horizon agent reinforcement learning paper
Dongxi recommends reading the preprint TreeOPSD: a critic-free reinforcement learning method for long-horizon LLM agents that propagates terminal rewards back along the rollout tree and uses forward KL to align the current policy with the target of each turn. According to the original post, on Deep Research this method raises the average accuracy of Qwen3-30B-A3B-Thinking-2507 from 16.23 with GRPO to 21.20.
View original post(1/5) New preprint: TreeOPSD, critic-free RL for long-horizon LLM agents 🌳 Long-horizon agents receive a single outcome reward after many turns, and each rollout batch feeds several sequential mini-batch updates while the policy drifts from the one that sampled it. TreeOPSD backs terminal rewards up a rollout tree and splits each update into two steps: construct a target for every turn, then match the current policy to it with forward KL. This leads to: • closed-form, bounded turn targets that commute with tree backup, so shared prefixes are trained once with no loss of credit • the exact tree policy gradient on the first mini-batch, and updates that stop at the target and reverse after overshoot • consistent gains over GRPO, GSPO and a matched tree baseline, largest on the longest-horizon tasks On Deep Research, TreeOPSD lifts Qwen3-30B-A3B-Thinking-2507 from 16.23 (GRPO) to 21.20 average accuracy. Title: Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning Paper: https://t.co/kOqeI03FnB Together with @TencentHunyuan #LLM #RLVR #AIAgents
View original
(1/5) New preprint: TreeOPSD, critic-free RL for long-horizon LLM agents 🌳 Long-horizon agents receive a single outcome reward after many turns, and each rollout batch feeds several sequential mini-batch updates while the policy drifts from the one that sampled it. TreeOPSD backs terminal rewards up a rollout tree and splits each update into two steps: construct a target for every turn, then match the current policy to it with forward KL. This leads to: • closed-form, bounded turn targets that commute with tree backup, so shared prefixes are trained once with no loss of credit • the exact tree policy gradient on the first mini-batch, and updates that stop at the target and reverse after overshoot • consistent gains over GRPO, GSPO and a matched tree baseline, largest on the longest-horizon tasks On Deep Research, TreeOPSD lifts Qwen3-30B-A3B-Thinking-2507 from 16.23 (GRPO) to 21.20 average accuracy. Title: Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning Paper: https://t.co/kOqeI03FnB Together with @TencentHunyuan #LLM #RLVR #AIAgents
Body · AI translation
Recommended reading!
TreeOPSD: