Madhu Guru：评估智能体要看步骤而非结果

如何构建出色的评估——第10部分 衡量步骤，而不仅仅是结果。 就像高中数学一样，仅仅得出正确答案是不够的，达成答案的步骤至关重要。 两条智能体轨迹可能得出相同的答案（42！！）。 但其中一条搜索了正确的来源，检索到了正确的文档，进行了4次干净的工具调用并计算出结果。另一条进行了17次调用，对同一内容搜索了3次，从2次错误中恢复，最终才得出答案。 哪一条更好显而易见。 以下是你需要做的： 1/ 清晰定义你的整个工作流 2/ 定义每个步骤中的任务 3/ 思考如何衡量每个步骤——是单独的评估，还是更大评估中的一个切片 4/ 定义你的中等难度和高难度任务——在评估中体现它们 现在，每当你查看评估结果时，先研究步骤，再看最终结果。

查看原文

How to build great evals - part 10 Measure the steps, not just the result. Much like high school math, it isn’t sufficient just to get the right answer, the steps to get there are critical. Two agent trajectories might produce the same answer (42!!). but one of them searches the right sources, retrieves the right document, makes 4 clean tool calls and calculates the result. The other makes 17 calls, searches the same thing 3 times, recovers from 2 errors and eventually gets there. It’s clear which one is better. Here’s what you need to do: 1/ clearly define your whole workflow 2/ define the tasks in each step 3/ think through how you measure each step - separate evals or is it a slice of a bigger eval 4/ define your median and hard tasks - reflect them in your evals Now any time you look at eval results, study the steps first and the final results next.