JetBrains 论文：编码智能体基准应匹配真实任务流
New JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types
JetBrains 研究论文提出编码智能体基准应匹配用户真实任务流：分析 4,782 个真实会话发现功能开发与缺陷修复占比远低于一半，而 SWE-Bench Pro 中这两类任务占 93.8%。团队构建 SWE-TaskFlow，将每个基准问题拆成多轮并加入可校验的代码问题，在 700 个 SWE-Bench Pro 任务上拆分使智能体成本约翻倍，成功率无稳定变化。
正文
当前显示原文，中文全文译文尚未就绪。
New JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types across many messages.
Real developers using coding agents ask questions, plan, review, and switch tasks mid-session, but benchmarks like SWE-Bench Pro hand the agent 1 detailed issue and grade 1 final patch. In that benchmark, new features and bug fixes make up 93.8% of tasks.
JetBrains researchers studied 4,782 real agent sessions, where features and bug fixes were well under half of requests. In longer sessions, users kept switching between questions, planning, reviews, and edits.
They built SWE-TaskFlow, which splits each benchmark issue into several turns and adds checkable questions about the code. On 700 SWE-Bench Pro tasks, splitting roughly doubled agent cost with no stable change in success rate.
If you evaluate coding agents, first measure what your own users ask for, then shape the benchmark's conversation to match it.
– arxiv. org/abs/2610.09633