Understand this story
On November 3, 2026, Hugging Face announced the CWE-bench v2 evaluation results and stated that the dataset, scorer, and harness are all publicly available and can be run directly from the Hub via Harbor. This evaluation covered 11 models: Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, followed closely by Claude Opus 5.5 with 67.3% pass@1. At least one model solved 95 tasks, but the top model still fails 19% of the time on each attempt. In terms of cost, DeepSeek V4.1 Flash, with 57.0% pass@1 at $0.04 per rollout, forms the Pareto frontier together with Claude Opus 5.5 and Gemini 4 Argon.
AI-generated from public reports
Latest development · Report
CWE-bench v2: Comprehensive improvements in coverage, quality, and transparencyDevelopments
1 confirmed developments
CWE-bench v2 public set evaluation results are tracked on the Hub
Source:Hugging Face
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 3
Hugging FaceFirst-party
CWE-bench v2: Comprehensive improvements in coverage, quality, and transparency
CWE-bench v2 publishes evaluation results: Among 11 models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, followed closely by Claude Opus 5.5 with 67.3% pass@1. 95 tasks were solved by at least one model, but the top model still fails 19% of attempts. On cost, DeepSeek V4.1 Flash, with 57.0% pass@1 and $0.04 per rollout, forms the Pareto frontier together with Claude Opus 5.5 and Gemini 4 Argon; the dataset, scorer, and harness are all publicly available and can be run directly from the Hub via Harbor.