CVE-Bench is here: Let AI Agents fix real CVEs and see who’s truly capable

A New Benchmark That Scores With Real Repositories

This week, a benchmark called CVE-Bench has been quietly spreading in the developer community. Unlike those overused ones such as SWE-Bench or HumanEval, this one asks a very specific question: Given a real CVE vulnerability report and the corresponding project’s source code of that version, can an AI Agent produce a patch equivalent to the one human maintainers committed back then?

It might look like a small thing, but the angle is sharp. Over the past year, discussions about "AI and security" have been almost entirely focused on the offensive side—from Google Project Zero’s Naptime/Big Sleep to the ICML 2025 paper with the same name CVE-Bench (note: that’s a different one, measuring an Agent’s ability to exploit web vulnerabilities, where even state-of-the-art Agent frameworks achieved only a 13% success rate).
On the defensive side—especially the simple, everyday task of "patching vulnerabilities"—there has long been no proper yardstick.

This week’s CVE-Bench fills that gap.

Completely Different From the "Offensive" CVE-Bench

Let’s clear up the naming confusion first.

If you search "CVE-Bench", you’ll most likely first hit arXiv:2503.17332 — a paper by the UIUC team submitted to ICML 2025. That one evaluates Agent attacks against web applications in a sandbox environment: it includes a collection of containerized web apps with critical CVEs, letting Agents attempt to exploit them, and automatically checks whether they succeed. That paper produced a striking result: even the strongest Agent framework at the time only achieved around 13%.

This new CVE-Bench (authored by Giovanni Gatti, hosted on GitHub Pages) is doing the exact opposite:

• Input: CVE description, source snapshot of affected version, and optionally, exploit/reproduction samples
• Task: Agent autonomously analyzes, locates, and generates patch
• Evaluation: The benchmark uses the actual historical fix commit from the project maintainer as the ground truth, plus a set of PoC test cases for that vulnerability, to verify whether the Agent’s patch both closes the hole and doesn’t introduce regressions

One benchmark tests offense, the other defense—the name collision is purely coincidental. Viewed together, they complete a full circle. If an Agent can exploit X% of vulnerabilities in the attack version of CVE-Bench but can fix Y% in the patch version, then (X–Y) roughly represents the net security risk the model currently poses to human engineers.
That difference is much more instructive than either number alone.

How the Dataset Was Selected

This is the most interesting part.

The most common pitfall in vulnerability-fixing benchmarks is data leakage: CVEs are public, patches are public, and the model almost certainly saw them during pretraining. If you test GPT-5 on something like Log4Shell from 2019, the code it "fixes" will likely be memorized from training, not actual reasoning.

CVE-Bench tackles this with several measures:

1. Prioritize CVEs published after the model’s cutoff date. For each tested model, only vulnerabilities published later than its training cutoff are kept. This means that within the same leaderboard, different models actually run different subsets—the metric becomes "how well can you fix what you’ve never seen".
2. Mandatory PoC verification. Comparing diffs isn’t enough, because there are many valid ways to fix a vulnerability. Each sample includes test cases that reliably trigger the vulnerability before the patch and fail afterward. The Agent’s patch is automatically validated: what was red must turn green, and what was green must stay green.
3. Covers multiple languages and project types. Current public samples include Python, JavaScript/Node, PHP, Java, and Go projects—from web frameworks to parsers to CLI tools—without bias toward any one "model-friendly" code style.
4. Full repository context preserved. Not just a single file or snippet, but the complete project structure. The Agent must use tools to grep, read, and explore the git history on its own—ensuring the benchmark measures true Agent capabilities rather than one-shot code completion.

Why Now

The timing is delicate.

In the past six months, major research labs have started showing tangible progress with security Agents: Anthropic integrated Claude’s Computer Use into CTF pipelines; Google DeepMind’s Big Sleep publicly claimed to have discovered multiple real 0-days in mainstream open-source projects; and OpenAI included "autonomous vulnerability discovery" results in GPT-5 release materials.
The problem is—all those numbers are self-reported.

Cross-comparable, third-party, tamper-resistant benchmarks are practically nonexistent. CVE-Bench fits nicely into this gap: it doesn’t aim to answer "Can an Agent become an AI Red Team?" but instead asks a very pragmatic question—"Can it handle the backlog of GitHub Security Advisories that Dependabot can’t auto-upgrade and require you to write a patch by hand?"

That question is extremely down-to-earth for enterprise developers. Any moderately sized team has been there: an old internal dependency suddenly gets a CVE, the original author’s long gone, and you have to fork and patch it yourself. Such work is low-margin, repetitive, but high-responsibility—making it a perfect landing zone for Agents.

What the Early Scores Reveal

The author released the first batch of results (still updating). Some takeaways:

• Fix success rates are much higher than exploit success rates. On the initial dataset, leading frontier models (Claude, GPT, Gemini) achieved patch pass rates well above the 13% seen in the ICML attack version. This makes sense: fixing known issues is a goal-directed task with verification signals, while discovering new bugs is open-ended.
• Failure modes are concentrated. Most failures come from over-editing rather than under-editing—Agents tend to "clean up" extra code they dislike, causing unrelated regressions. This is a known issue from SWE-Bench, but is far more dangerous here: in production, a "fix" that changes behavior is a P0 incident.
• Cross-language gaps are bigger than expected. The same model might hit 50%+ success on Python but drop sharply on legacy PHP projects (e.g., WordPress plugins, old Laravel)—mirroring training data distributions. This warns against using "average scores"; per-language evaluation is much more meaningful.

Some Critiques

A few caveats after the praise.

First, the dataset is still too small. The current public size is about a hundred samples—insufficient for something aspiring to be the "SWE-Bench of security." Even SWE-Bench Verified’s 500 samples were criticized as sparse. Given the long tail of vulnerability types—SQL injection, prototype pollution, deserialization RCE—each category needs enough examples for stable metrics.

Second, PoCs may not cover the full attack surface. For example, an SSRF bug’s PoC might only test http gopher:// paths, while the Agent patch only blocks HTTP, leaving gopher still exploitable. Since the PoC passes, the benchmark gives credit—this kind of "false positive fix" is unacceptable in real security engineering. The benchmark needs stronger, multi-payload validation.

Third, missing "decoy" tasks. A mature fix benchmark should include some "false alarms"—reports that look vulnerable but aren’t—to test if an Agent blindly patches everything. The current version uses only confirmed CVEs, meaning Agents always assume “there must be a bug.” That diverges from real workflows where you must first determine whether a report is valid.

These are all solvable issues. The author explicitly published a roadmap inviting the community to contribute new samples.

What It Means for Developers

In the short term, CVE-Bench won’t change your workflow directly. But a few things are worth noting:

1. A real reference point for model selection. Next time you want to plug an "auto CVE-fixing Agent" into your internal SRE chain, you’ll have neutral, cross-vendor numbers to compare—Claude, GPT, Gemini, DeepSeek, Qwen and others will all get benchmarked.
2. "Agent engineering" details will matter more. The same underlying model can differ by double-digit percentage points depending on tools: whether it has git blame, access to PR history, or the ability to run tests. The benchmark will force teams to reverse-engineer those design choices.
3. Multi-model usage will become routine. In reality, no single model dominates all languages or vulnerability types. Claude might best handle Python deserialization bugs, Gemini might excel at Java XXE, and GPT might outperform others when large historical context is required.
   This pattern keeps appearing in test results. For domestic teams, this explains the recent surge in multi-model routing via platforms like OpenAI Hub—calling GPT, Claude, Gemini, DeepSeek under one API key is cheaper and easier than juggling multiple overseas accounts.

Final Thoughts

Security is always asymmetric—attackers only need one breakthrough; defenders must cover everything.
The arrival of CVE-Bench (the patching edition) matters because someone is finally measuring, systematically and reproducibly, how capable AI really is on the defense side—instead of just showing flashy "AI hacker" demos.

Everyone in AI safety evaluation knows one truth: benchmarks drive progress; without them, it’s all hype.
SWE-Bench took two years to raise "AI code modification" accuracy from 2% to over 70%. CVE-Bench may now stand at the start of an equally important story.

It’s worth keeping an eye on.

References

• arXiv 2503.17332 - CVE-Bench (Offensive) Paper — Proposed by the UIUC team; note this one evaluates Agents’ ability to exploit web vulnerabilities.