引用Madhu Guru · @realmadhuguru
How to build great evals — Part 9
The Eval Roadmap Problem
Most evals fail because teams treat them as static artifacts while their users expectations and behaviors have evolved.
Your evals need a roadmap that evolves with your product and actual usage patterns.
Take a financial research agent. Here is how use cases will evolve over time.
Early user: Summarize this 5-page earnings report.
3 weeks later: review the last 5 earnings reports and explain their growth story.
2 months later: Here are 15 filings, earnings transcripts and research reports. Build an investment thesis.
Eventually: Monitor my stock portfolio and tell send me alerts when something materially changes my thesis.
Each stage requires different capabilities and hence different evals.
Your evals should change with usage patterns:
short-context -> long-context
single-turn QA -> multi-turn
passage citations -> doc and line citations
Simple QA -> complex synthesis
reactive chat -> proactive agent
If your evals are stuck in week 1 while users are in week 3, it will show in your product and churn metrics.
Here’s a practical way to build the roadmap:
1/ Map the dimensions along which usage will evolve (eg # turns, document size, tool use, autonomy, journey coverage).
2/ Prioritize the use cases and dimensions that matter most for your product.
3/ Talk to your users, mine production traces to look for shifts.
4/ Build P0 evals for the next stage of usage.
5/ Run the evals, find failure modes (remember the taxonomy from Part 3), and hill-climb (from part 6).
The goal is to stay ahead of usage patterns. PM 101.
Drop your questions and I will address in future posts.
Share this with your teammates who think your evals are perfect.