GitHub 发布 ReviewBench：面向 AI 代码审查的开放基准

Agentic code review is becoming an essential piece of how development happens. It helps you inspect pull requests, catch issues, and decide what deserves attention before code ships.

But the quality of existing AI reviewers can be hard to measure, and you need to know the strengths of a reviewer before you know if it will help you. Some reviewers surface more issues, some produce less noise, and some are stronger at catching critical problems while others surface smaller improvements, too. You may need code review to do different things within your workflow.

That makes it important to understand how reviewers actually compare: what different systems catch, what they miss, and the tradeoffs they make. A good code review benchmark should reflect the diversity of real pull requests, capture a broad set of review findings, and support meaningful breakdowns by severity, category, and precision-recall preferences. For teams building code review agents, the benchmark should also provide an offline signal that reliably tracks whether changes are likely to improve the experience in production. Existing benchmarks often make tradeoffs between label quality, coverage, and how well they represent real-world code review, leaving a gap for a rigorous and reproducible evaluation methodology that brings these pieces together.

We built ReviewBench , a new code review offline benchmark , to address that gap, and it is available for you to use today. It follows the language, repo size, and size distribution of pull requests, modeled after over 100 million real pull requests on GitHub. It uses a multi-source golden set and a consistent evaluation rubric and has been independently validated by senior engineers. Just as important, with the help of ReviewBench, our offline evaluation of Copilot code review (CCR) has become more effective at anticipating the direction of production experiments, giving us greater confidence that measured improvements reflect meaningful gains for users.

In this post, we’ll walk through how ReviewBench is constructed, how it establishes reliable ground truth and scoring, and how to onboard your own code review system and submit results.

Definitions of terms used in this blog post

Benchmark : A standardized evaluation that tests code reviewers on a common set of pull requests using the same scoring methodology.

Finding : A specific issue surfaced during code review.

Golden set : A validated collection of known findings for each pull request, used as a reference for evaluating what a reviewer catches or misses.

Precision : Of the issues a reviewer surfaces, the proportion that are valid. Higher precision generally means less noise.

Recall : Of the known valid issues, the proportion the reviewer finds. Higher recall means broader coverage.

F1 score : A single score that balances precision and recall equally.

Fβ score : A variation of F1 that lets you put more weight on either precision or recall, depending on your review preference.

ReviewBench at a glance

What we built

A realistic, comprehensive benchmark for AI code review agents