CWE-bench v2：覆盖、质量与透明度全面提升

Results

Results are tracked in two places on the Hub. The leaderboard on the CWE-bench dataset page covers open-weight models and updates on its own: labs and the community add a result by opening a PR on the model's repo with a results file ( how ). For the full comparison, including API-only models such as Gemini, Claude and GPT, see the CWE-bench leaderboard Space , where new models are added through a PR on the Space.

Open-weight models, from the dataset page:

All evaluated models, including API-only ones:

Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4). 95 tasks are solved by at least one model, an indication that the tasks are indeed solvable. However, even the leader fails 19% of the set on every attempt, so the long tail is far from solved. The public set, the grader and the harness are open. Run your model, and tell us where it stands.

On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92.

Run it from the Hub

Every task is a self-contained Harbor task (a Dockerized environment, an instruction, and a deterministic verifier), and the dataset ships a registry.json . Harbor can therefore run the whole benchmark straight from the Hugging Face repo, without cloning anything yourself:

-k 4 runs 4 attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated: accept its access conditions on the dataset page first. Harbor fetches the tasks with git, so also save your HF token to git's credential helper once:

The tasks are also tagged rl-environment : the same environments and verifiers can provide rewards for training defensive agents, not just for evaluating them.

Datasets mentioned in this article 1

collinear-ai/cwe-bench

Spaces mentioned in this article 1

CWE-bench Leaderboard

Leaderboard for the CWE-bench v2 public set