40,000 Tests Expose Blind Spots in Agent Approval Processes

Industry Brief | Human Approval Is Becoming the Weak Link in the AI Agent Security Chain

Requiring “human approval for critical operations” in AI Agents is not as reliable as many assume.

A recent test published by Scalex found that across approximately 40,000 game runs, human reviewers missed about one-third of malicious commands. In other words, even when high-risk actions are explicitly presented to humans, and even when the final decision remains in human hands, a substantial number of dangerous actions are still approved.

The value of this data is not that it proves “humans are worse than models,” but that it exposes a more practical problem: adding an approval pop-up to an Agent does not amount to establishing a security boundary.

Missing One-Third Means the Approval Mechanism Is Not a Safety Fuse

The test used a gamified environment to simulate an Agent performing tasks and requesting permissions. The Agent proposed a series of commands, some of which the system submitted for human review. Reviewers then decided whether to allow or reject them. The test did not focus on whether the model could directly carry out an attack, but on whether humans could accurately identify threats in these requests.

The results were not encouraging: about one-third of malicious or dangerous commands were not blocked.

On the surface, this may appear to be merely a problem of insufficient human judgment accuracy. In production systems, however, the consequences are magnified. Agents typically do not execute just one command; they run continuously for minutes, hours, or even days. A single mistaken approval may authorize only a low-risk action, but several low-risk actions chained together can result in unauthorized access, credential theft, data exfiltration, or supply-chain tampering.

Human approval is like checking whether each turn a vehicle makes is reasonable. The real risk introduced by an Agent, however, often emerges only after it has made dozens of turns in succession.

The test also shows that human approval failures do not necessarily stem from carelessness. More common causes include:

• The command itself lacks context: The reviewer sees a Shell command, an API call, or a file access request, but cannot see where it fits within the complete task chain.
• Permission requests look routine: Reading a configuration file, creating a temporary directory, or accessing an internal service may all seem reasonable in isolation.
• Agents act far faster than humans can review: Humans can process only a limited amount of information per second, while an Agent can generate and submit numerous consecutive requests within seconds.
• Risk alerts can easily cause alert fatigue: When the system repeatedly displays approval requests, reviewers gradually shift from “analyzing” to simply “clicking Allow.”
• Malicious intent is broken into separate steps: A dangerous objective is decomposed into multiple legitimate actions, making it difficult for step-by-step approval to detect the ultimate goal.

This is why “human in the loop” is often misinterpreted as “a safety-conscious human in the loop.” Having a reviewer does not mean that the reviewer has sufficient information, time, or control granularity.

The Real Risks of Agents Are Hidden in Their Action Traces

Traditional permission systems are primarily designed around the question of “who can access what.” For human users, this model is relatively mature: users have identities, roles, organizational affiliations, and fixed permissions, while systems can record logins, actions, and audit logs.

Agents behave very differently.

Based on a natural-language objective, an Agent may dynamically select tools, create temporary files, read environment variables, access external websites, delegate subtasks, and then write results back to a database. Each step may appear to comply with permission rules, while the complete trace may not align with the business intent.

For example, suppose a coding Agent is granted permission to read a repository and run tests. It first reads the project configuration and discovers deployment information. It then accesses the build service before calling an external API to verify dependency versions. If the system only checks whether each action is “authorized,” all of these steps may be allowed. Taken together, however, they may have already exposed internal credentials to an external service.

These issues can be summarized in three layers:

1. Single-step risk: Whether a command is inherently dangerous, such as deleting a directory, modifying an IAM policy, or reading a secret.
2. Combined risk: Whether multiple seemingly normal actions form an attack chain.
3. Goal drift: Whether the Agent has deviated from the task originally authorized by the user.

Many approval products currently address only the first layer. A few systems are beginning to address the second, while the third remains extremely difficult. While completing a task, a model may independently expand its search scope, invoke tools that the user did not explicitly authorize, or take a path that is “technically feasible but unacceptable from a business perspective.”

Permission approval therefore cannot stop at asking, “Is this command dangerous?” It must also ask:

Why is this command appearing now? Which earlier actions is it related to? If approved, what new capabilities will it give the Agent?

Why Human Approval Fails Systematically

First, Approval Interfaces Give Humans a False Sense of Security

Many systems implement approval as a permission-style pop-up that displays the tool name, parameters, and risk level, followed by two buttons: “Allow” and “Deny.” This design appears clear, but it can easily reduce a complex security decision to a binary choice.

If reviewers cannot see the call chain, data flows, credential sources, and downstream impact, they are not actually reviewing the Agent’s behavior. They are merely reviewing a packaged request.

For developers, this is equivalent to seeing only the final line of a Git diff and being asked to determine whether the entire commit is safe.

Second, Agent Permissions Tend to Expand Continuously

Agents in production environments rarely have full capabilities from the outset. An Agent may initially be connected only to a ticketing system, then receive read-only database access, followed by access to a CRM, cloud platforms, code repositories, and message queues. Every new workflow introduces new tools and expands the scope of permissions.

The problem is that permissions are easy to add but rarely revoked automatically.

An Agent granted write access to a production database during a pilot may retain that capability months after the project ends. No one deliberately expanded its permissions, and no one specifically revoked the old ones. The permissions simply accumulated like sediment.

This kind of permission sprawl resembles what happens with traditional service accounts, but Agents pose a greater risk: they can not only exercise permissions but also plan their next actions based on feedback from the environment.

Third, Attackers Do Not Necessarily Need Direct Control of the Model

The Agent security attack surface has expanded beyond prompts to include tools, plugins, MCP services, web content, code repositories, and external APIs.

Attackers can influence an Agent’s decisions through malicious documents, poisoned dependencies, forged tool responses, or indirect prompt injection. The dangerous action may ultimately still be initiated by an Agent that “possesses legitimate credentials and invokes legitimate tools,” making it difficult for traditional network defenses to determine from identity alone whether the Agent has been manipulated.

This differs from traditional malware. Malware usually manifests as an explicitly anomalous process or suspicious connection. A manipulated Agent may fully comply with tool protocols, use the correct API key, and leave behind extensive logs filled with seemingly reasonable actions.

What This Test Means for Enterprises

First, human approval can no longer be treated as the sole control for high-risk actions. At most, it is an additional layer of defense. It cannot replace least privilege, sandbox isolation, tool-level policies, or real-time revocation.

Second, enterprises should treat an Agent as an independent non-human identity rather than an extension of an ordinary employee account. At a minimum, the following information must be clearly defined for each Agent:

• Who is responsible for it from both a business and security perspective;
• Which tools it may invoke;
• Which actions it may perform with each tool;
• How long its permissions remain valid and when they are automatically revoked;
• Whether it may create sub-Agents or delegate tasks;
• Which data it may read, modify, or transmit externally;
• How its credentials can be revoked immediately if an anomaly occurs.

Third, permissions should be refined from “role-based authorization” to “task-based authorization.” A data analysis Agent should not be able to read an entire database by default merely because it has database read access. It should be able to access only the tables, fields, and time ranges required for the current task.

For high-risk operations, short-lived, revocable, and conditional authorization is preferable to granting an Agent a long-lived, all-powerful token. Permission policies can be bound simultaneously to:

• Task ID;
• Responsible user or service owner;
• Data scope;
• Network egress;
• Number of operations;
• Time window;
• Budget or resource limit.

More Effective Approval Should Review “Plans,” Not “Commands”

If the system still requires human participation, the object of approval should also change.

Approving individual commands is suitable for simple, low-frequency automation scripts, but not for Agents capable of planning autonomously. A more reasonable approach is to have the Agent first generate a reviewable action plan explaining its objective, the tools it intends to invoke, the data it expects to access, the possible side effects, and its rollback path if an anomaly occurs.

Humans approve the “boundaries of the plan,” and the system then automatically validates each execution step. If the Agent deviates from the plan—for example, by suddenly accessing a new data domain, requesting higher privileges, or changing its network egress—the system should automatically pause it rather than continuing to display pop-ups and wait for a human click.

The control mechanism can be divided into three layers:

1. Before Execution: Limit Capabilities

• Apply least privilege and default-deny policies;
• Create short-lived credentials for different tasks;
• Require tools to declare their inputs, outputs, and side effects when registered;
• Prevent Agents from accessing environment variables, secret directories, and production control planes by default;
• Set separate controls for external network access, file writes, and code execution.

2. During Execution: Observe the Trace

• Record the complete call chain rather than only the final command;
• Limit the rate, frequency, and scope of tool calls;
• Detect anomalous privilege escalation, lateral access, and data flows;
• Analyze combinations of multiple low-risk actions;
• Automatically interrupt behavior that deviates from the task objective.

3. After Execution: Enable Accountability and Rollback

• Assign a clearly identified human owner to each Agent;
• Review permissions regularly and make them expire automatically;
• Preserve model inputs, tool responses, decision rationales, and execution results;
• Provide rollback capabilities for database modifications, code commits, and infrastructure changes;
• Support one-click revocation of all temporary credentials and subtask permissions.

Do Not Simply Blame the Problem on Models “Not Being Smart Enough”

This test can easily be interpreted as evidence that “humans can also be deceived by AI.” But what truly deserves attention is not the intelligence ranking of models—it is how the control system is designed.

More capable models may be better at identifying complex threats, but they may also be better at probing permission boundaries, bypassing rules, and combining legitimate tools. Less capable models make more basic mistakes, but more powerful Agents can execute longer chains of actions, increasing the potential blast radius of errors or malicious behavior.

Security architecture therefore cannot depend on a model being “smart enough” or “obedient enough.” The model is only one component in the decision chain. What truly needs to be controlled is what it can see, what it can invoke, how long it can continue operating, and whether it can be stopped quickly when it deviates.

This is also the dividing line between today’s Agent products and traditional chatbots. When a chatbot generates incorrect text, a human generally still has to act on it. Once an Agent has credentials, it can directly change the state of a system. The former is primarily a content risk; the latter is already a system risk.

OpenAI Hub Observation: An Approval Button Is Not an Agent Security Solution

For developers, the most practical conclusion is this: do not relax permission design simply because “human confirmation” has been added. Whether the underlying model is GPT, Claude, Gemini, DeepSeek, or another model, switching models does not automatically solve the Agent authorization problem.

OpenAI Hub provides multi-model API aggregation and OpenAI-compatible integration, making it easier for teams to test capabilities and costs across different models. However, once a model is placed inside an Agent framework with tool-calling, file-access, and system-write capabilities, permission isolation, log auditing, and operational rollback must still be handled at the application layer.

Going forward, what Agent security products should truly compete on is not how polished their approval pop-ups look, but whether they can answer three questions:

1. What exactly is the Agent doing right now?
2. Why is it authorized to do this?
3. If it starts doing the wrong thing, can the system stop it before its next step?

Missing one-third of threats across 40,000 test runs is enough to demonstrate that human approval can participate in control, but it cannot bear the entire burden of control. As Agents continue to act faster, enterprises must shift from asking “Should we consult a human for every operation?” to asking “Are permissions sufficiently narrow, are action traces explainable, and can the system recover from a loss of control?” That is the minimum security standard for production-grade Agents.

References

• Official Hugging Face Blog: A resource for learning about practices related to Agents, model security, and open-source AI infrastructure.
• OWASP LLM Application Security Project on GitHub: Provides risk classifications, threat-modeling guidance, and defensive approaches for large-model applications and Agents.
• Scalex, “Humans missed 1 in 3 threats approving AI agent commands across 40k game runs”: The source of the core test data cited in this article—approximately 40,000 runs, with human reviewers missing about one-third of threats.