ServiceNow 与 McGill 论文：AI 评判长程电脑操作任务仍会漏掉细微错误
New ServiceNow and McGill paper finds that AI judges still miss subtle mistakes in long computer-use tasks, and giving them tools does not reliably help.
ServiceNow 与 McGill 的新论文发现，AI 评判在长程电脑操作任务中仍会漏掉细微错误，给评判模型配工具也不能稳定改善表现。研究团队构建了 AgentHorizon，从人类屏幕录制中整理出 1,373 个评判任务，通过录制近乎相同但互换指令的任务对，让每次失败都对应一个略有差异的请求。表现最好的 GPT-5.5 在 Codex 中于困难子集取得 80.9% 平衡准确率（全判通过为 50%）；Qwen 3.6 27B 直接读完整运行记录得 77.6%，作为智能体运行时降至 70.4%。作者建议在信任 LLM 评判智能体运行前，先在实际使用的智能体环境中用近义失败样本测试它。
推荐理由
AI 评分 58
论文用近义任务对构造评测集，量化了 LLM 评判长程智能体任务时的漏判程度，并给出工具未必有效的对照结果。
主题
正文
当前显示原文，中文全文译文尚未就绪。
New ServiceNow and McGill paper finds that AI judges still miss subtle mistakes in long computer-use tasks, and giving them tools does not reliably help.
Teams use LLMs to grade agent runs for training and evaluation. A run of 100+ steps can look finished while it drew the wrong shape or posted to the wrong Slack channel.
They built AgentHorizon, 1,373 judging tasks from human screen recordings. People recorded pairs of near-identical tasks and swapped the instructions, so each failure is a clean run of a slightly different request.
The best judge, GPT-5.5 in Codex, scored 80.9% balanced accuracy on the hard split, where always saying yes gets 50%. Qwen 3.6 27B did worse with tools, scoring 77.6% reading the whole run directly but 70.4% as an agent.
Before you trust an LLM judge with agent runs, test it on near-miss failures inside the exact agent setup you will use.