Claude Mythos’ First Battle with curl: Scanning 176,000 Lines of Code Revealed Only 1 Real Vulnerability, While 4 Were False Positives or Regular Bugs

When Anthropic hyped up Mythos as the “world’s most dangerous AI” in April, the entire security community got a little tense. The company claimed at the time that its new model possessed “dangerously powerful” source code vulnerability detection capabilities — so powerful that they decided not to release it publicly, only granting trial access to a few large institutions. Combined with rumors in early May about an unreleased version being hacked, the whole situation felt like a ticking time bomb.

Last weekend, curl creator Daniel Stenberg defused that bomb — and the outcome was rather awkward: after scanning 176,000 lines of C code, Mythos reported 5 vulnerabilities. Upon manual review by the curl security team, only 1 turned out to be real, and it was low severity. Of the remaining 4, three were false positives, and one was just a regular bug.

This marked the first real-world test of Mythos on an open-source project — and the results stood in sharp contrast to Anthropic’s earlier marketing pitch.

A Report Shorter Than Expected

The collaboration was arranged under the Linux Foundation’s Alpha-Omega program, with researchers assisting the curl team in running the Mythos scan. On May 6, curl received the first analysis report.

In his blog, Stenberg said the team had expected “a long list of issues,” given how much buzz Anthropic had generated in recent months. But when they opened the report, there were only 5 items labeled as security vulnerabilities.

After manual review, the results broke down as follows:

• 1 item confirmed as a real vulnerability (low severity; to be fixed in curl 8.21.0, scheduled for late June)
• 3 items were false positives — AI misjudgments
• 1 item was not a security issue but a regular bug
• An additional 20 bug reports, almost none of which were false positives

If you look only at vulnerabilities, that’s 1 out of 5, or 20% accuracy — not exactly flattering, especially for a model marketed as the “most advanced code security audit system” ever.

However, the 20 regular bug reports were nearly all valid, showing Mythos performed adequately in that category — though Anthropic wasn’t selling a bug detector; it was selling a vulnerability detector.

curl Wasn’t a Fair Target

To be fair, Mythos deserves some credit. Stenberg himself pointed out that comparing discovery counts alone is not fair.

Anyone who’s done security audits on C projects knows curl is a particularly tough target. Before Mythos came in, the curl team had already been using a suite of AI tools for code scanning:

• AISLE – an AI system dedicated to open-source security auditing
• Zeropath – an automated vulnerability discovery platform
• OpenAI Codex Security – OpenAI’s own product focused on code security

Thanks to these tools, curl’s team had patched 200–300 bugs and issued over a dozen CVEs in the past 8 to 10 months. In other words, all the “low-hanging fruit” that AI could find had already been picked.

Under such conditions, Mythos still managed to surface 1 real vulnerability and 20 valid bug reports. That’s not bad — just not the “quantum leap” Anthropic’s hype suggested.

Stenberg’s conclusion was measured: “Mythos didn’t find any new vulnerability types or memory-related flaws, and its detection capability wasn’t significantly better than existing AI tools.”

The Gap Between Anthropic’s “Hype vs. Reality”

Looking back at the events in April, Anthropic’s marketing narrative for Mythos unfolded like this:

1. First, Fortune Magazine leaked that Anthropic’s upcoming Mythos showed “major performance improvements” in software coding, academic reasoning, and cybersecurity testing.
2. Second, the company amplified the message, emphasizing Mythos’s “dangerously powerful” vulnerability detection capability — powerful enough that they would not make it public yet.
3. Third, in early May, news broke that a hacker group allegedly bypassed Mythos’s security controls to access an unreleased version, reinforcing the myth that the model was “so powerful that even Anthropic couldn’t contain it.” Reports claimed the attackers guessed online storage locations based on Anthropic’s typical naming patterns and exploited permissions from a third-party contractor.

Through this sequence of events, Mythos was effectively mythologized as a nuclear-grade weapon in AI security.

However, Bernstein Research had already published a note tempering market expectations, stating the public had “misread” Mythos’s real capability and purpose. The curl test now confirms Bernstein’s assessment.

What This Test Really Shows

Beyond the marketing layer, Mythos’s performance actually revealed several insights valuable for developers:

1. AI still hasn’t passed the “human review bottleneck” in code security auditing.
Four out of five reported vulnerabilities needed to be manually discarded — meaning AI output still functions more as a lead generator than a conclusion provider. Handing Mythos’s reports straight into remediation pipelines would waste engineers’ time chasing phantom issues.

2. Bug detection and vulnerability detection are two separate challenges.
In the 20 bug reports, Mythos had almost zero false positives — but in the vulnerability set, the false positive rate was 60% (3/5). This asymmetry suggests Mythos is good at sensing “something looks off” in code, but still struggles to determine whether that irregularity is actually a security exploitability issue.

3. AI tools face diminishing returns on mature projects.
For veteran codebases like curl, which have been extensively scanned, new tools can rarely uncover major new flaws. Conversely, projects that haven’t undergone systematic AI-assisted auditing will likely see impressive results in their first round. If you maintain an older open-source project that hasn’t had an AI security audit yet, now’s a good time to start.

4. Closed release does not equal superior capability.
Anthropic used the “too dangerous to release” narrative to hype Mythos. But results show its performance doesn’t significantly surpass open tools like AISLE or Zeropath. The security community should build immunity to this kind of marketing rhetoric.

Practical Takeaways for Security-Focused Developers

If you’re evaluating AI code security tools, the curl test offers a relatively clean comparison sample. Here are some directly applicable insights:

• Don’t bet on a single tool — the curl team ran AISLE, Zeropath, Codex Security, and Mythos together; combined use beats any single solution
• Treat AI reports as priority queues, not fix lists — always allocate time for human review
• False positive rates are notably higher for vulnerability reports than for general bugs — prioritize review resources accordingly
• Watch for whether the tool finds new vulnerability types or memory-related flaws — that’s the key metric for real breakthroughs, and Mythos found none

The one low-severity vulnerability Mythos identified will be included in curl 8.21.0, expected in late June. Consistent with curl’s practice, the CVE ID and impact details will be disclosed when the patch goes public.

In the end, Mythos’s real-world performance wasn’t flattering for Anthropic — but it’s good news for the industry. It brings AI security auditing down from “PowerPoint promises” to measurable engineering benchmarks. Next time some company claims its AI is “dangerously powerful,” developers will at least have a ruler to measure against.

References

• AI Security Model Mythos’ First Real-World Test on curl Project: 1 Vulnerability and 20 Bugs - linux.do — Summary of the event with key excerpts from Daniel Stenberg’s blog
• AI Security Model Mythos Tested with curl Project - iThome — Full iThome report including data on curl’s past AI-assisted fixes