Netflix Research Scientist Shares a Roadmap for Evaluating Agents on Subjective Tasks
Cameron R. Wolfe, a senior research scientist at Netflix, has published a roadmap for evaluating subjective tasks, proposing a three-step process: human evaluation, offline automated evaluation, and online monitoring. The article presents statistical tools such as power analysis, confidence intervals, and repeated sampling, along with five criteria that evaluation sets must meet: diversity, realism, sufficient difficulty, discriminative power, and accuracy. The overall process is: offline evaluation screens ideas; promising versions are handed to human evaluation; after A/B testing, they are fully launched and continuously monitored, with the results feeding back into the evaluation set.
Why read this
AI score 60
The original article breaks subjective task evaluation into three layers—human, offline, and online—and provides reusable statistical practices such as power analysis and confidence intervals.
Topics
View original postI just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence… Blog is here: https://t.co/uo0FsWtkYF Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process. Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components: - Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality. - Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model. - Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent. Multi-stage evaluation. These components form a tiered evaluation process: Offline evaluation → Human evaluation → Production A/B test → Full deployment We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop. Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets. Handling noise. We should leverage statistics throughout the evaluation process: - Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements. - Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results. - Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals. - Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences. Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.
View original
I just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence… Blog is here: https://t.co/uo0FsWtkYF Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process. Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components: - Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality. - Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model. - Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent. Multi-stage evaluation. These components form a tiered evaluation process: Offline evaluation → Human evaluation → Production A/B test → Full deployment We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop. Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets. Handling noise. We should leverage statistics throughout the evaluation process: - Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements. - Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results. - Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals. - Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences. Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.
Body · AI translation
Subjective tasks can also be evaluated reliably: a three-step approach of human evaluation, offline automated evaluation, and online monitoring, plus a set of statistical methods.
Shared by @cwolferesearch, a senior research scientist at Netflix; this article is also the write-up corresponding to his talk at the AI Engineer conference: https://t.co/A9LIAVqU7C
Why is evaluating subjective tasks hard? Agents can do a wide range of things, task chains are long, evaluation is costly, and the same task produces different results each time it is run. Subjective tasks add another layer: people often disagree with each other, and LLM judges often disagree with people. Evaluation is essentially "statistically measuring the pass rate on a limited set of tasks," so whether the conclusions are reliable depends on two things: whether the tasks are realistic and representative, and whether noise has been correctly accounted for.
Three statistical tools · Power analysis: figure out how many samples you need before evaluating. The smaller the improvement you want to detect and the larger the score variance, the more samples you need. · Confidence intervals: obtain a 95% interval by taking the mean plus or minus 1.96 times the standard error. If multiple tasks come from the same codebase or the same document, use clustered standard errors; otherwise you will underestimate uncertainty. When comparing two agents, have them run the same batch of tasks and look at the pairwise differences; the intervals will be narrower. · Repeated sampling: have the agent run the same task several times and have the judge evaluate several times, then average to reduce within-task variance. But this cannot eliminate differences between tasks.
How to choose an evaluation set (five criteria) Diverse (use a taxonomy to ensure coverage, e.g., GDPval), realistic (drawn from real usage scenarios, e.g., CursorBench, SWE-Lancer), hard enough (filter for hard problems using model pass rates, then add human review), discriminative (able to separate different models, measurable with item response theory, IRT), and accurate (have an audit agent check failed run records to find out whether the problem lies in the question or the scoring itself, or whether the model really got it wrong).
Three-step roadmap 1. Human evaluation (the ultimate standard): work with a small number of domain experts to define quality standards: a small number of scoring dimensions, binary or Likert-scale scoring rules, plus examples. Have multiple people annotate repeatedly, measure agreement with Fleiss' kappa and Krippendorff's alpha, and revise the standards based on disagreements. If a dimension cannot reach agreement no matter how you adjust it, simplify it, split it, or just remove it. Calibration must be ongoing to prevent standards from drifting over time. 2. Offline automated evaluation: use LLM judges or rubrics in place of human scoring. The prerequisite is to first validate the judge against human-labeled "gold data"; the goal is for judge-human agreement to reach human-human agreement. Then use it to build benchmarks and continuously add new items and fix old ones. 3. Online monitoring: look at implicit metrics (session duration, abandonment rate) and explicit metrics (likes/dislikes, preference feedback), run A/B tests first and then roll out fully. Find new scenarios and failure cases from online interactions, then feed them back into the evaluation sets from the first two steps, forming a closed loop.
The overall process is: offline evaluation quickly screens a large number of ideas, promising versions go to human evaluation, then after A/B testing they are rolled out fully, and after launch they are continuously monitored and feed back into the evaluation set.