Anthropic 详解 AI 智能体评测:从评分器设计到测试集维护
Anthropic 工程团队发布文章,系统讲解如何为 AI 智能体设计评测,涵盖任务、trial、评分器、transcript、outcome 等核心概念,以及代码、模型、人工三类评分器的组合方式。文章区分能力评测与回归评测,并分别给出编码智能体和对话智能体的评测方法,例如 SWE-bench Verified 与 Terminal-Bench 的确定性评分思路。文中还建议以评测驱动开发,由专门团队维护核心基础设施、产品与领域专家贡献评测任务,并附录了 Harbor、Braintrust、LangSmith、Langfuse、Arize 等评测框架。
te: does the code run and do the tests pass? Two widely used coding agent benchmarks, SWE-bench Verified and Terminal-Bench , follow this approach. SWE-bench Verified gives agents GitHub issues from popular Python repositories and grades solutions by running the test suite; a solution passes only if it fixes the failing tests without breaking existing ones. LLMs have progressed from 40% to 80% on this eval in just one year. Terminal-Bench takes a different track: it tests end-to-end technical ta
匹配字段正文



