English translation is being prepared.

English translation is being prepared.

View original

How to build great evals — Part 9 The Eval Roadmap Problem Most evals fail because teams treat them as static artifacts while their users expectations and behaviors have evolved. Your evals need a roadmap that evolves with your product and actual usage patterns. Take a financial research agent. Here is how use cases will evolve over time. Early user: Summarize this 5-page earnings report. 3 weeks later: review the last 5 earnings reports and explain their growth story. 2 months later: Here are 15 filings, earnings transcripts and research reports. Build an investment thesis. Eventually: Monitor my stock portfolio and tell send me alerts when something materially changes my thesis. Each stage requires different capabilities and hence different evals. Your evals should change with usage patterns: short-context -> long-context single-turn QA -> multi-turn passage citations -> doc and line citations Simple QA -> complex synthesis reactive chat -> proactive agent If your evals are stuck in week 1 while users are in week 3, it will show in your product and churn metrics. Here’s a practical way to build the roadmap: 1/ Map the dimensions along which usage will evolve (eg # turns, document size, tool use, autonomy, journey coverage). 2/ Prioritize the use cases and dimensions that matter most for your product. 3/ Talk to your users, mine production traces to look for shifts. 4/ Build P0 evals for the next stage of usage. 5/ Run the evals, find failure modes (remember the taxonomy from Part 3), and hill-climb (from part 6). The goal is to stay ahead of usage patterns. PM 101. Drop your questions and I will address in future posts. Share this with your teammates who think your evals are perfect.