120 Organizations Equip Agents with Black Boxes

AI Agents Are Finally Getting a Unified Incident Reporting Framework

According to the latest news on August 12, more than 120 companies and organizations—including NVIDIA, Cisco, CrowdStrike, Hugging Face, and Red Hat—are promoting a security incident reporting framework for AI agents.

The framework is called SAFE, short for Shared AI Findings Exchange. Developed by the Open Secure AI Alliance, it is still in the public comment phase and is not yet a mandatory standard.

SAFE is intended to address an increasingly real problem that has long lacked a unified approach: when an agent accesses a database without authorization, leaks customer information, or is induced by prompt injection to keep probing production systems, who should document the incident, what should be recorded, how quickly should it be reported, and how can other companies learn from similar incidents?

When traditional software fails, companies at least have CVEs, threat intelligence, indicators of compromise, and mature incident response processes. Agents are different. A single loss-of-control incident often spans models, prompts, memory, identities, tools, cloud services, and third-party interfaces, making it difficult to attribute the failure to a single line of code.

SAFE’s value lies in its attempt to create a cross-company “black box” for such incidents.

Which Incidents Need to Be Reported

According to the currently disclosed draft, participating members must report at least three categories of security incidents:

• An AI system accesses or exploits a third-party system without authorization;
• An agent leaks confidential information, sensitive data, or protected content;
• An agent continues probing production targets after an operator has already suspected that its behavior is unauthorized.

The third category is particularly noteworthy.

Traditional automation usually behaves according to the principle that “if the rule is wrong, it will keep doing the wrong thing.” Agents, however, may dynamically adjust their approach in pursuit of a goal. For example, if an operations agent lacks permission to read server configurations, it may instead invoke a search tool, query historical support tickets, and then try another set of credentials. A single tool call may not appear malicious, but the complete task trajectory may reveal a sustained attempt to probe systems without authorization.

SAFE therefore focuses not only on incidents that have already caused damage, but also requires the recording of certain events that “almost happened”—what the security industry commonly calls a near miss.

For example:

1. An agent attempts to send internal code to an external analysis service, but is blocked by an egress gateway;
2. An agent gains access to an administrator-level tool, but is stopped by a human approval step before it can actually perform a deletion;
3. A prompt embedded in a malicious webpage successfully influences the agent’s planning, but the target system happens not to permit write access;
4. An identity inheritance error occurs during multi-agent collaboration, but the final request fails because the token has expired.

These events cause no actual damage, but they reflect real-world risk more closely than ordinary vulnerability scan results. They show that the attack path was viable, but the final door happened to remain closed.

Aviation safety systems have long emphasized near misses for a straightforward reason: waiting until an aircraft actually crashes before learning from the event is far too costly. At the Black Hat conference, Justin Boitano, NVIDIA’s vice president of enterprise computing, also compared SAFE to NASA’s Aviation Safety Reporting System, expressing the hope that flight-recorder-like data could be used to investigate agent incidents.

The analogy largely holds, but AI systems are more difficult to investigate than aircraft. Aircraft control systems, sensors, and operating procedures are highly standardized. An agent’s model may be switched at any time, prompts may be assembled dynamically, memory may be compressed, and tools may come from different vendors. The same input is not even guaranteed to produce the same decision. Whether SAFE can be implemented successfully therefore depends less on how comprehensive its reporting forms are and more on whether companies routinely retain sufficient execution context.

Four Days, Thirty Days, and Ninety Days

The SAFE draft proposes a relatively clear incident response timeline:

• Notify as soon as possible: After discovering an incident, affected organizations should be informed as quickly as possible;
• Within four business days: Submit an initial confidential report to SAFE;
• Within thirty days: Publish an initial factual report where appropriate;
• Within ninety days: Provide updates on remediation measures, repair progress, and subsequent findings.

This cadence clearly draws on mature cybersecurity incident response practices: first contain the impact and notify affected parties, then verify the facts, and finally publish reusable defensive lessons.

Four business days is not especially generous. The supply chain involved in an agent incident is often lengthy. A single anomalous tool call may pass through an agent framework, a model API, an identity provider, an MCP server, a cloud function, and the target database. If a company lacks unified trace IDs and time synchronization, simply reconstructing a complete timeline could take several days.

SAFE requires the preservation of evidence including:

• User prompts and system prompts;
• Agent execution traces and intermediate decisions;
• Tool invocation parameters, returned results, and execution status;
• The identity that initiated the invocation and its role;
• The permissions and authorization policies in effect at the time;
• The credentials used and their relevant context.

The final item is particularly sensitive. “Preserving credentials” should not be interpreted as writing API keys, access tokens, and passwords to logs in plaintext. A more appropriate engineering approach would be to record the credential type, issuer, scope, hashed identifier, validity period, and rotation version, while separately encrypting and sealing any necessary sensitive evidence. Otherwise, in the course of investigating one agent-related leak, a company could create an even more concentrated point of credential exposure.

SAFE Is Not a Guardrail, but an Incident-Learning Layer

One easy misconception is that joining SAFE will automatically make agents safer.

It will not.

SAFE will not automatically block prompt injection, configure least-privilege access on a company’s behalf, or press the stop button before an agent deletes a database. It operates at the incident response and industry collaboration layer, addressing how to preserve evidence after an incident, share recurring patterns, and avoid repeating the same mistakes.

A complete agent security system requires at least four layers of capabilities:

1. Prevention layer: Tool allowlists, least privilege, sandboxing, network isolation, and approval for high-risk operations;
2. Detection layer: Prompt injection detection, anomalous behavior detection, sensitive data identification, and invocation rate controls;
3. Response layer: Pausing tasks, revoking credentials, isolating agents, rolling back operations, and notifying affected parties;
4. Learning layer: Preserving evidence, analyzing root causes, sharing incident patterns, and updating controls.

SAFE fills the gap at the fourth layer, while also indirectly pressuring companies to improve the first three. If a company cannot answer “which identity invoked which tool under what permissions,” it cannot even submit a valid incident report.

This is also SAFE’s most practical impact on development teams: agent observability will evolve from a performance feature into security infrastructure.

In the past, developers implementing agent tracing mainly focused on token consumption, model latency, task success rates, and tool errors. SAFE-style incident investigations also require identities, permissions, policy versions, and data flows to be tracked. Simply recording a stream of natural-language conversations is no longer enough. Logs must be able to reconstruct what the agent saw at the time, why it took a particular action, and whose permissions it used to execute that action.

The Real Challenge Is Cross-Vendor Attribution

An agent is not merely a model; it is an entire system.

After purchasing an agent product, an enterprise may find that it calls a third-party model underneath, stores memory in a vector database, relies on another vendor for identity management, and connects to tools through MCP or internal APIs. When an incident occurs, each party can easily offer a completely different explanation:

• The model provider argues that the deployer exposed dangerous tools;
• The tool provider argues that the agent submitted a legitimate request;
• The deployer argues that the model failed to follow the system prompt;
• The identity platform argues that the token itself was not stolen, merely misused;
• The agent framework developer argues that the customer failed to configure a human confirmation checkpoint.

If SAFE can establish a common evidence format, its greatest value will not be making incident reports easier to write, but reducing this accountability vacuum in which “every layer worked normally, yet the combined system still failed.”

However, the draft still needs to address several issues.

1. Is Voluntary Sharing Enough?

The alliance argues that the cybersecurity industry already has a tradition of sharing threat intelligence and that companies will be willing to participate. However, agent incidents may directly expose product capability flaws, the scope of customer data, and internal permission designs. The commercial risk is far greater than that of sharing a malicious IP address.

Without clear confidentiality boundaries, legal safe harbors, and anonymization mechanisms, companies will most likely report minor incidents that are easy to disclose, while their legal departments block the genuinely valuable, serious cases.

2. Will Incidents Be Classified Consistently?

The severity of the same behavior can vary completely across different scenarios. It is normal for an agent to read a public document, but reading an unauthorized payroll table is an incident. Scanning a test environment is part of a security research agent’s job, while scanning a customer’s production subnet may constitute an intrusion.

SAFE needs more than event types. It also requires dimensions such as the environment, scope of authorization, business impact, recoverability, and degree of autonomy. Otherwise, it will end up with a collection of cases that cannot be meaningfully compared.

3. How Should Prompts and Privacy Be Handled?

Complete prompts may contain customer data, trade secrets, personal information, or even attack payloads. Directly sharing raw traces is neither practical nor necessarily compatible with contractual and data governance requirements.

A more feasible approach would be tiered disclosure: affected parties receive the necessary raw evidence, SAFE analysts access controlled data, and public reports retain only sanitized attack paths, failed control points, and remediation recommendations.

4. How Can Model Randomness Be Reproduced?

Traditional vulnerabilities can usually be reproduced by following a fixed set of steps. Agent incidents, however, are affected by model versions, sampling parameters, context windows, memory states, and responses from external tools. If the same prompt is tested again a month later, the model may have been upgraded and produce a completely different result.

Companies therefore need to record more than the model name. They must also capture the model version, inference parameters, system prompt version, tool descriptions, policy configurations, and critical external responses. Otherwise, so-called “traceability” may lead only to an execution environment that no longer exists.

Developers Should Not Wait for the Standard to Be Finalized

SAFE is still only an RFC draft, and its final fields and processes may change. However, development teams do not need to wait for the standard to be finalized before improving their logs, because incident forensics for agent systems cannot be solved simply by adding an SDK.

At a minimum, teams can take five steps now:

• Assign correlatable unique identifiers to every agent task, subtask, and tool invocation;
• Bind identities, permissions, and policy versions to execution traces instead of storing only conversation content;
• Add human confirmation, quotas, allowlists, and revocable authorization for high-risk tools;
• Apply tiered redaction to prompts, tool parameters, and returned data to prevent logs from becoming a new source of leaks;
• Regularly rehearse credential revocation, agent isolation, task termination, and incident notification procedures.

Teams should be especially careful to avoid pseudo-observability: having extensive logs that still cannot answer important questions. A mature recording system should at least be able to quickly determine who initiated a task, what context the agent received, which tools it invoked, what permissions it used, which controls were triggered, and where the ultimate impact occurred.

The Industry Needs a Black Box, but It Cannot Rely on the Black Box Alone

SAFE is a direction worth pursuing.

As agents enter production environments, the security question has shifted from “Will the model say something wrong?” to “Can the model do something wrong while acting with the company’s identity?” The former may produce a bad response; the latter may actually send emails, modify code, transfer data, or operate infrastructure. Without cross-company incident records and shared lessons, every enterprise will be forced to pay the same tuition by repeating the same mistakes in its own production systems.

But SAFE cannot become a substitute for security accountability. No matter how polished an incident report may be, it cannot compensate for long-lived administrator privileges, high-risk tools without approval requirements, or execution chains that cannot be stopped.

More precisely, SAFE is not an airbag for the age of agents; it is a flight recorder and an accident investigation board. It cannot guarantee that the aircraft will never crash, but it can help the next aircraft avoid making the same mistake.

If the framework ultimately enables vendors to share real incidents, standardize evidence structures, and convert recurring failures into actionable security controls, SAFE will be more useful than yet another high-level declaration of AI principles. Conversely, if the alliance receives only low-risk cases polished by public relations teams, it may ultimately deteriorate into a compliance box-ticking exercise.

The real question to watch is not how many more companies join, but whether the first members are willing to provide sufficiently specific incident examples—and whether SAFE can establish credible boundaries between transparency, trade secrets, and legal risk.

References

• ITHome: NVIDIA, Cisco, and 120 Other Organizations Propose an AI Agent Incident Tracking Mechanism: Introduces the organizations participating in the SAFE draft, its reporting scope, evidence retention requirements, and proposed timeline.