Sangfor AI Ranks Among CyberGym’s Top Four

Chinese Model Solves 86.3% of Vulnerability Challenges

On July 28, Sangfor disclosed Sangfor AI’s latest results on the CyberGym cybersecurity benchmark: using the Chinese-developed GLM-5.2 model as its fixed foundation, the system successfully solved 1,301 of 1,507 vulnerability tasks spanning ARVO and OSS-Fuzz, achieving an overall success rate of 86.3%. This placed it among the global top four and first among participating Chinese teams.

This result deserves attention for more than its leaderboard position.

CyberGym does not test whether a large language model can recite CVE descriptions, nor does it ask the model to answer a few multiple-choice security questions. Instead, it places an agent inside real open-source projects, requiring it to read code, locate flaws, construct trigger paths, and ultimately generate a proof of concept (PoC) capable of reproducing the vulnerability. Put simply, it requires the model to move from “knowing where a problem might exist” to “producing executable evidence.”

For the security industry, that final step is what truly matters.

Why CyberGym Is Harder Than Conventional Benchmark Leaderboards

CyberGym was originally proposed by a research team affiliated with the University of California, Berkeley. It contains 1,507 historical vulnerabilities from the real world, covering 188 widely used open-source projects, including foundational software such as OpenSSL and FFmpeg. Its task data primarily comes from vulnerability and fuzzing systems such as ARVO and OSS-Fuzz.

What distinguishes it from code benchmarks such as HumanEval and SWE-bench is that its objective is not to complete a function or fix a clearly described issue, but to reproduce a real vulnerability. A typical workflow includes at least the following steps:

1. Read the vulnerability description and determine the input surface and potential trigger conditions;
2. Locate the relevant modules and call chains within a large code repository;
3. Understand cross-file state, memory layout, or business logic;
4. Configure dependencies and compile the target project;
5. Construct an input and execute the PoC;
6. Use crashes, Sanitizer logs, or anomalous behavior to confirm that the vulnerability has actually been triggered.

An error at any step in this chain can cause the task to fail. If the model finds apparently dangerous code but cannot compile the project, the task is not considered solved. Likewise, generating text that looks like exploit code but cannot actually trigger the vulnerability does not count.

CyberGym is therefore more like a judged vulnerability range than a security knowledge exam. Its execution-based validation mechanism significantly reduces the scope for results that “sound plausible in words but fail completely at runtime.”

In the early CyberGym paper, the strongest model-and-agent combination achieved a success rate of only around 20%. Subsequent research showed that increasing the number of attempts on the same task could significantly improve performance: for example, success rates from a single run and from 30 runs could differ by more than twofold. This also shows that CyberGym has never measured only the intelligence of the foundation model; it also evaluates the agent framework, search budget, tool use, context management, and failure-retry strategy.

86.3% Is Not the Result of GLM-5.2 Running on Its Own

It is important to distinguish between the “Chinese foundation model” and the “complete system.”

The key information disclosed by Sangfor is that Sangfor AI uses GLM-5.2 as its fixed foundation model and completes tasks through a security agent system. According to the public description, the system employs an Agent Swarm-style collaboration mechanism. In other words, GLM-5.2 provides code comprehension, reasoning, and generation capabilities, but the final result comes from the combined effect of the foundation model, task orchestration, toolchain, and validation mechanism.

The workflow can be abstracted as follows. This is only an architectural illustration, not Sangfor’s complete publicly disclosed engineering configuration:

benchmark: CyberGym
tasks: 1507
base_model: GLM-5.2
model_policy: fixed

agent_roles:
  - repository_analysis
  - vulnerability_reasoning
  - poc_generation
  - build_and_execution
  - evidence_verification

success_condition:
  - project_builds
  - poc_executes
  - target_vulnerability_is_reproduced

This design better reflects real-world vulnerability research than having a single agent handle the entire process from start to finish.

Large codebases can quickly exhaust the context window. A single agent must remember the vulnerability description, track dozens of files, and deal with compilation errors and runtime logs, making it easy for the reasoning trace to become overwhelmed by noise. A multi-agent architecture can divide the work: some agents perform repository reconnaissance, some trace data flows, some generate PoCs, and others specialize in validating results and eliminating invalid paths.

It is more like a small security research team than an omnipotent chatbot.

What Sangfor’s result truly demonstrates is that Chinese foundation models can already support intensive, long-chain security-agent workflows. The industry previously tended to assume that complex code reasoning required proprietary flagship models from OpenAI or Anthropic. Sangfor AI has at least demonstrated that, with GLM-5.2 fixed as the underlying model, a Chinese model can also enter the top tier when paired with an engineering system designed specifically for security scenarios.

Conversely, however, this result cannot simply be reframed as “GLM-5.2 has an 86.3% vulnerability-solving capability.” Without Sangfor’s agent orchestration, sandbox environment, tool use, and validation mechanisms, calling the foundation model alone would most likely fail to reproduce the same result.

“Global Top Four” Cannot Yet Be Judged by a Single Percentage

The 86.3% figure is high, but its leaderboard position should still be interpreted cautiously.

As of July 27, a third-party compilation of the public CyberGym leaderboard showed Sakana AI’s Fugu Cyber in first place at 86.9%, with systems such as OpenAI GPT-5.6 Sol and Google Gemini 3.5 Flash Cyber also near the top. However, this public table and Sangfor’s claim of being “among the global top four” cannot be directly combined into a strictly comparable leaderboard: different pages may use different snapshots, evaluation tracks, context limits, numbers of attempts, and proprietary-system configurations.

The following variables require particular clarification:

• How many attempts are allowed per task;
• Whether the reported metric is pass@1 or the best result across multiple runs;
• The token and time budget for each task;
• Whether internet access, code search, or external knowledge bases are permitted;
• Whether multiple agents are used for parallel search;
• How many retry rounds are allowed after compilation failures;
• Whether all 1,507 tasks are covered;
• Whether different systems use the same versions of containers, patches, and evaluation scripts.

Historical CyberGym data has already shown that increasing the number of trials can substantially improve success rates. Succeeding on the first attempt and succeeding once across 30 attempts represent entirely different costs, speeds, and levels of reliability in an enterprise production environment.

Sangfor’s 86.3% result can therefore be confirmed as a very strong system-level performance, but until the full configuration is disclosed, it would be inappropriate to conclude that it has consistently surpassed any specific proprietary model. A more accurate statement is that, under its disclosed fixed-model configuration and full-task evaluation scope, Sangfor AI has entered the leading global tier.

This is not nitpicking, but a shared challenge now facing agent benchmarks. Model leaderboards can generally control variables using fixed prompts and sampling parameters, while agent leaderboards encompass models, frameworks, tools, concurrency, and compute budgets. Comparing agent systems is increasingly like comparing racing teams rather than engines alone.

The Value for Development Security Is More Tangible Than the Leaderboard Position

If Sangfor AI can transfer its benchmark capabilities to enterprise codebases, its practical value will primarily lie in three areas.

1. Adding “Understanding” and “Validation” to SAST

Traditional static application security testing relies on rules, syntax trees, and taint analysis. It excels at identifying clearly patterned issues such as SQL injection, dangerous function calls, and hard-coded secrets. However, when dealing with authorization flaws, authentication bypasses, and cross-module business-logic vulnerabilities, it can often see local code but cannot understand the complete business process.

The advantage of security agents is their ability to keep asking questions along the call chain: Where does this parameter come from? What validations does it pass through? Under what state can those validations be bypassed? Can the result ultimately form an executable attack path?

More importantly, an agent can attempt to generate a PoC for validation. Traditional scanners generally report a “suspected vulnerability,” while an agent may be able to go further and say, “I have reproduced it in a sandbox.” If the validation is reliable, the hundreds of alerts that development teams face every day could be reduced to a small number of issues that genuinely require attention.

2. Shifting Code Auditing Earlier into CI/CD

Manual penetration testing generally occurs before a release, when the cost of rework after discovering a problem is high. If security agents can operate reliably, they can be integrated into code commit and merge request workflows:

• After a developer submits code, the agent analyzes the incremental changes;
• It performs cross-file tracing of high-risk call chains;
• It automatically constructs test inputs in an isolated environment;
• It outputs vulnerability evidence, the scope of impact, and remediation recommendations;
• After the fix is completed, it reruns the PoC to confirm that the issue is gone.

This is much closer to a usable engineering feedback loop than displaying a message in the IDE saying, “This may be insecure.”

3. Shortening the Vulnerability Exploitation Window

Generative AI is also lowering the barrier for attackers to analyze patches and develop PoCs. After a vulnerability is disclosed, the time from a patch diff to usable exploit code is shrinking. If defenders still rely on manual asset investigation and validation, they will be at an increasing disadvantage in response speed.

Agents capable of automatically locating, reproducing, and validating vulnerabilities can help enterprises quickly determine whether their versions are affected, whether existing protections can block the exploit, and whether a patch is truly effective.

This is also the practical significance of CyberGym’s results. It tests not only “vulnerability discovery,” but also the most labor-intensive parts of the automated vulnerability research pipeline.

Four Hurdles Remain Before Real-World Deployment

A high leaderboard score does not mean an enterprise can directly connect an agent to its production repository. What Sangfor must demonstrate next is the system’s unit cost and controllability in real development environments.

First, the false-positive rate. CyberGym provides the location and description of a known target vulnerability, making it a clearly defined reproduction task. Enterprise code auditing, by contrast, involves finding unknown issues within vast amounts of normal code. The former is like searching for evidence with clues from a case, while the latter is more like identifying a crime that has not yet occurred from routine surveillance. The two are not equally difficult.

Second, compute costs. An Agent Swarm can improve success rates through parallel search, but it also increases token consumption, sandbox resource usage, and execution time. If every code commit requires hours of runtime and dozens of agents, broad deployment will be difficult outside high-value core systems.

Third, private code security. Enterprises are unwilling to send core source code, secrets, and internal dependencies to external proprietary APIs. Using a Chinese model as the foundation does provide more options for private deployment and supply-chain control, but the system must still address compliance across the entire pipeline, including model weights, inference frameworks, log storage, and sandbox isolation.

Fourth, the security of the agent itself. Code repositories are not trusted inputs. Malicious comments, build scripts, and test data could all induce an agent to leak information or execute dangerous instructions. When an agent is allowed to compile and run unknown code, container isolation, network permissions, credential management, and resource limits must be far stricter than those used for ordinary coding assistants.

Our Assessment: A Systems Engineering Victory, Not Merely a Model Victory

The most valuable signal from Sangfor AI’s entry into CyberGym’s global top four is not that “a Chinese model has topped another benchmark,” but that a Chinese foundation model can now support a sophisticated vulnerability-reproduction agent.

In tasks of this kind, model capability is only the starting point. Final success rates also depend on code retrieval, context compression, task decomposition, compilation tools, failure retries, and evidence validation. As a security vendor, Sangfor’s advantage lies not only in its model, but also in knowing how vulnerability research workflows should be decomposed and what constitutes a genuinely valid result.

The 86.3% result still requires more complete configuration disclosure to establish cross-system comparability, including the number of attempts, compute budget, context size, and cost per task. If these metrics are also competitive, the significance of this result will clearly exceed that of the leaderboard position alone.

At least based on the current result, competition among security agents has moved beyond “which model knows more about security” and into “who can organize models into an executable, verifiable system that can be integrated into development workflows.” The latter is precisely the contest Sangfor has now entered.

References

• CyberGym paper page: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale: Introduces the 1,507 real-world vulnerability tasks, the execution-based evaluation methodology, and experimental results from early model-and-agent combinations.
• Discussion of Chinese Large-Model Capabilities and Industry Impact: Provides additional background on the development of Chinese foundation models and industry competition.