Cognition Releases FrontierCode 2.0 Coding Benchmark, Top Score Only 45%
Cognition launches FrontierCode 2.0, a coding benchmark that examines whether models can go beyond specifications and make judgments about whether a PR should be merged. The new version features more diverse tasks, a broader scope, and higher merge standards, and deliberately uses under-specified prompts to let models demonstrate judgment. The benchmark is far from saturated, with the top-scoring model achieving only 45%.
Body
Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.