微软发布 ThinkingBox:智能体工具调用成功不等于数据库状态正确
微软 Copilot Studio 团队联合 Toloka 等发布 ThinkingBox 智能体沙箱与 ThinkingBox-Bench 基准,主张最终回复和合法工具调用只是代理指标,只有数据库终态才是证据。在覆盖 12 个 LLM、121,680 次有效试验的消融中,79,853 次未通过可执行检查,其中 67.24% 仍干净终止、调用了改状态工具且未报最终工具错误,可执行检查在这些失败中发现 77.61% 字段值错误、43.30% 多余副作用、25.36% 缺失必要效果。每个任务在隔离后端上独立运行 20 次,Claude Opus 5.5 以 67.16% pass@1 领先,但只有 GPT-6 Astra、Claude Opus 5.5 和 Claude Opus 5 保留了大部分单次成绩;Kimi-K3 覆盖最广(507 个任务中 476 个至少成功一次),却只有 13.41% 任务 20 次全过。环境与数据集已在 Hugging Face 开源,代码 MIT 许可。
nd your agent? Figure 3: Breadth and consistency pull apart. Twelve of the eighteen models are shown; six below 33% pass@1 are omitted for legibility. Kimi-K3 has the broadest coverage of any model we tested. It solves 93.89% of the benchmark at least once: 476 of 507 tasks. Only 31 tasks defeat it entirely, the lowest count in the field. On retail workflows it leads outright at 82.24% pass@1, ahead of every proprietary model. Kimi-K3 is also among the least consistent. Just 68 of 507 tasks, 13.
匹配字段正文
