如何构建优秀的评估体系第 10 部分：衡量过程，而不只是结果
How to build great evals - part 10
构建智能体评估体系时，应先衡量执行步骤，再看最终结果，因为得到相同答案的轨迹可能在工具调用次数、检索质量和错误恢复上差异显著。具体可先定义完整工作流及各步任务，再确定分步评估方式，并把中等难度与高难度任务纳入评估。
查看原帖How to build great evals — Part 9 The Eval Roadmap Problem Most evals fail because teams treat them as static artifacts while their users expectations and behaviors have evolved. Your evals need a roadmap that evolves with your product and actual usage patterns. Take a financial research agent. Here is how use cases will evolve over time. Early…
展开完整原文
How to build great evals — Part 9 The Eval Roadmap Problem Most evals fail because teams treat them as static artifacts while their users expectations and behaviors have evolved. Your evals need a roadmap that evolves with your product and actual usage patterns. Take a financial research agent. Here is how use cases will evolve over time. Early user: Summarize this 5-page earnings report. 3 weeks later: review the last 5 earnings reports and explain their growth story. 2 months later: Here are 15 filings, earnings transcripts and research reports. Build an investment thesis. Eventually: Monitor my stock portfolio and tell send me alerts when something materially changes my thesis. Each stage requires different capabilities and hence different evals. Your evals should change with usage patterns: short-context -> long-context single-turn QA -> multi-turn passage citations -> doc and line citations Simple QA -> complex synthesis reactive chat -> proactive agent If your evals are stuck in week 1 while users are in week 3, it will show in your product and churn metrics. Here’s a practical way to build the roadmap: 1/ Map the dimensions along which usage will evolve (eg # turns, document size, tool use, autonomy, journey coverage). 2/ Prioritize the use cases and dimensions that matter most for your product. 3/ Talk to your users, mine production traces to look for shifts. 4/ Build P0 evals for the next stage of usage. 5/ Run the evals, find failure modes (remember the taxonomy from Part 3), and hill-climb (from part 6). The goal is to stay ahead of usage patterns. PM 101. Drop your questions and I will address in future posts. Share this with your teammates who think your evals are perfect.
正文
当前显示原文，中文全文译文尚未就绪。
How to build great evals - part 10
Measure the steps, not just the result.
Much like high school math, it isn’t sufficient just to get the right answer, the steps to get there are critical.
Two agent trajectories might produce the same answer (42!!).
but one of them searches the right sources, retrieves the right document, makes 4 clean tool calls and calculates the result. The other makes 17 calls, searches the same thing 3 times, recovers from 2 errors and eventually gets there.
It’s clear which one is better.
Here’s what you need to do: 1/ clearly define your whole workflow 2/ define the tasks in each step 3/ think through how you measure each step - separate evals or is it a slice of a bigger eval 4/ define your median and hard tasks - reflect them in your evals
Now any time you look at eval results, study the steps first and the final results next.