VulnBench v3 scores 27 models on patching 200 real CVEs. Each model sees the vulnerability and source, never the reference fix, and a pinned cross-vendor judge panel decides whether the patch fixes the root cause. Claude Opus 5 leads at 64.3% — roughly double any other model.
Every ranked model ran three independent times; the score is the mean pass rate with a 95% confidence interval. All rows are artifact-free (no API or judge failures counted as model errors).
Claude Opus 5 fixes 64.3% of real CVEs (95% CI 60.4%–68.1%), roughly double the next model.
Claude Opus 5 leads GPT-5.6 Sol (42.2%) with non-overlapping confidence intervals — a statistically clear #1.
Claude Opus 5 produced a passing patch for 80.5% of vulnerabilities within three attempts.
The median model fixes only 19.7% of real CVEs — blind security repair remains hard across the field.
Frontier, coding-specialist, and open-weight families evaluated on the identical Protocol v2 harness, every row artifact-free.
Rebuilt after an 81-finding audit: uniform token budget, a pinned cross-vendor judge panel, and mean-of-3-runs with confidence intervals.
Mean pass rate across 3 runs on the curated 200-CVE set. 95% CI is the Wilson interval over pooled trials; pass@3 is the share solved in at least one run. Judged by a pinned Claude Opus 4.8 + GPT-5.5 panel with a Gemini 3.5 Flash tie-breaker — no model judges its own patches.
| # | Model | Pass Rate | 95% CI | pass@3 | Mean Score |
|---|---|---|---|---|---|
| 1 | Claude Opus 5 Anthropic | 60.4%–68.1% | 80.5% | 0.638 | |
| 2 | GPT-5.6 Sol OpenAI | 38.3%–46.2% | 56.5% | 0.505 | |
| 3 | GPT-5.3 Codex OpenAI | 31.9%–39.6% | 50.5% | 0.496 | |
| 4 | GPT-5.6 Terra OpenAI | 29.7%–37.2% | 47.5% | 0.457 | |
| 5 | GPT-5.5 OpenAI | 28.5%–36.0% | 53.0% | 0.570 | |
| 6 | Gemini 3.1 Pro Google | 27.6%–35.0% | 45.0% | 0.430 | |
| 7 | Gemini 3.5 Flash Google | 25.0%–32.2% | 41.0% | 0.376 | |
| 8 | Gemini 3.6 Flash Google | 23.3%–30.3% | 40.5% | 0.320 | |
| 9 | GPT-5.6 Luna OpenAI | 22.3%–29.3% | 36.5% | 0.404 | |
| 10 | Claude Fable 5 Anthropic | 21.4%–28.3% | 33.0% | 0.267 | |
| 11 | Grok Build 0.1 xAI | 20.4%–27.2% | 37.0% | 0.349 | |
| 12 | Grok 4.5 xAI | 18.7%–25.3% | 40.0% | 0.296 | |
| 13 | Claude Opus 4.8 Anthropic | 17.8%–24.3% | 35.0% | 0.422 | |
| 14 | Claude Sonnet 5 Anthropic | 16.7%–23.0% | 33.0% | 0.357 | |
| 15 | Claude Sonnet 4.6 Anthropic | 14.4%–20.4% | 29.0% | 0.315 | |
| 16 | DeepSeek V4 Pro DeepSeek | 13.3%–19.1% | 26.5% | 0.294 | |
| 17 | GLM 5.1 Z.AI | 12.1%–17.7% | 27.5% | 0.158 | |
| 18 | GLM 5.2 Z.AI | 11.2%–16.7% | 26.0% | 0.268 | |
| 19 | MiniMax M3 MiniMax | 9.9%–15.2% | 21.0% | 0.229 | |
| 20 | Grok 4.3 xAI | 9.6%–14.8% | 19.0% | 0.266 | |
| 21 | Gemini 3.5 Flash Lite Google | 9.5%–14.7% | 19.0% | 0.245 | |
| 22 | Nemotron 3 Ultra NVIDIA | 8.6%–13.6% | 19.5% | 0.243 | |
| 23 | GPT-5.4 Mini OpenAI | 8.1%–13.0% | 19.5% | 0.258 | |
| 24 | Claude Haiku 4.5 Anthropic | 6.8%–11.4% | 12.0% | 0.221 | |
| 25 | Step 3.7 Flash StepFun | 6.8%–11.4% | 15.5% | 0.238 | |
| 26 | DeepSeek V4 Flash DeepSeek | 5.9%–10.3% | 15.5% | 0.205 | |
| 27 | Mistral Medium 3.5 Mistral AI | 4.2%–8.0% | 9.0% | 0.164 |
v3 was rebuilt after an 81-finding audit of the earlier leaderboards. Every model runs under one frozen protocol so rankings reflect capability, not harness artifacts.
Each model gets a real vulnerability and its source context, but never the reference fix, and must emit a minimal patch.
Identical 16k-token budget for every model — the fix for the artifact that zeroed out reasoning models in earlier reports.
Three independent runs per model; the headline is the mean pass rate with a 95% Wilson interval, plus pass@3.
A pinned Opus 4.8 + GPT-5.5 panel scores each patch, a third-vendor judge breaks split votes, and no model ever judges its own output.
200 curated CVEs from 200 repositories, balanced across three difficulty tiers, spanning 48 CWE types and 7 package ecosystems. Advisory text is sanitized to remove fix pointers before a model sees it.
The repository ships the curated CVE set, every model's patches, per-judge decisions, the statistics code, and the full audit trail behind Protocol v2.
$ ./run_v3_200.sh Set VulnBench-200 Runs mean of 3 + 95% CI Judges Opus 4.8 + GPT-5.5 (+ tie-breaker) Self-judge disabled Leader Claude Opus 5 Pass Rate 64.3% (CI 60.4%–68.1%) pass@3 80.5%