CWE-bench v2: Comprehensive improvements in coverage, quality, and transparency
CWE-bench v2 publishes evaluation results: Among 11 models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, followed closely by Claude Opus 5.5 with 67.3% pass@1. 95 tasks were solved by at least one model, but the top model still fails 19% of attempts. On cost, DeepSeek V4.1 Flash, with 57.0% pass@1 and $0.04 per rollout, forms the Pareto frontier together with Claude Opus 5.5 and Gemini 4 Argon; the dataset, scorer, and harness are all publicly available and can be run directly from the Hub via Harbor.
Body
Results
Results are tracked in two places on the Hub. The leaderboard on the CWE-bench dataset page covers open-weight models and updates on its own: labs and the community add a result by opening a PR on the model's repo with a results file ( how ). For the full comparison, including API-only models such as Gemini, Claude and GPT, see the CWE-bench leaderboard Space , where new models are added through a PR on the Space.
Open-weight models, from the dataset page:
All evaluated models, including API-only ones:
Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4). 95 tasks are solved by at least one model, an indication that the tasks are indeed solvable. However, even the leader fails 19% of the set on every attempt, so the long tail is far from solved. The public set, the grader and the harness are open. Run your model, and tell us where it stands.
On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92.
Run it from the Hub
Every task is a self-contained Harbor task (a Dockerized environment, an instruction, and a deterministic verifier), and the dataset ships a registry.json . Harbor can therefore run the whole benchmark straight from the Hugging Face repo, without cloning anything yourself:
-k 4 runs 4 attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated: accept its access conditions on the dataset page first. Harbor fetches the tasks with git, so also save your HF token to git's credential helper once:
The tasks are also tagged rl-environment : the same environments and verifiers can provide rewards for training defensive agents, not just for evaluating them.
Datasets mentioned in this article 1
collinear-ai/cwe-bench
Spaces mentioned in this article 1
CWE-bench Leaderboard
Leaderboard for the CWE-bench v2 public set
Datasets mentioned in this article 1
collinear-ai/cwe-bench
Spaces mentioned in this article 1
CWE-bench Leaderboard
Leaderboard for the CWE-bench v2 public set