微软 CABRA 研究：编码智能体瓶颈在读懂代码而非改代码
New Microsoft paper finds that coding agents trip up when they have to understand a lot of code, not when they have to edit a lot of it, so test them on reading
微软论文发现编码智能体在需要理解大量代码时出错，而非在大量编辑时出错，因此应以阅读和比较代码而非 diff 大小来评测。团队构建 CABRA 生成合成编码任务，每次只提升一种难度，用 8 个 LLM 和 6 个智能体跑了 6,840 个任务，并把每次工具调用标注为阅读、分析、搜索、编辑或测试。纯 LLM 随任务变难而变差，智能体靠 grep 等工具保持接近满分；在 SWE-bench Verified 上，阅读与分析调用次数比编辑行数更能预测智能体失败，相关系数为 -0.200 对 -0.159。
正文
当前显示原文，中文全文译文尚未就绪。
New Microsoft paper finds that coding agents trip up when they have to understand a lot of code, not when they have to edit a lot of it, so test them on reading and comparing code instead of diff size.
Microsoft researchers built CABRA, which generates synthetic coding tasks and raises 1 kind of difficulty at a time. They ran 8 LLMs and 6 agents on 6,840 tasks and labeled each tool call as reading, analyzing, searching, editing, or testing.
Plain LLMs got worse as tasks grew, but agents stayed near-perfect by using tools like grep. On SWE-bench Verified, the count of reading and analysis calls tracked agent failures better than lines edited, with correlations of -0.200 versus -0.159.