Frontier Models Fail Collectively: ITBench-AA Exposes Real Weak Spots in Enterprise IT Agents

On May 26, IBM Research and independent benchmarking agency Artificial Analysis jointly released ITBench‑AA on Hugging Face—the industry’s first benchmark for agentic capabilities in enterprise IT operations scenarios. The results are disappointing: more than a dozen frontier models tested, and not a single one scored above 50 percent overall.

This isn’t just another benchmarking contest; it’s more like a real “enterprise health check.” While model vendors obsess over SWE‑bench, AIME, and MMLU, pushing scores past 90+, IBM shifted the focus to real‑world enterprise production environments—SRE incident response, FinOps cost governance, and CISO security compliance—where frontier models instantly reverted to amateur status.

I. Why Such a Benchmark Is Needed

Over the past year, “enterprise‑grade AI agents” have become high‑frequency buzzwords in virtually every big‑tech earnings call and product launch. Microsoft introduced Copilot Studio, Salesforce released Agentforce, and IBM offers watsonx Orchestrate plus CUGA—the champion at last year’s AppWorld. Yet within developer circles there’s an open secret: between a lab demo and real deployment lies a Mariana Trench.

The root problem is the evaluation methodology. Existing agent benchmarks—AppWorld, WebArena, OSWorld, SWE‑bench—are either simulated environments, open‑source code repositories, or consumer‑web tasks. They tell you whether a model can use a browser or modify Python code, but they can’t answer the question that really keeps CTOs up at night: “When the production Kubernetes cluster blows up at 3 a.m., can the model locate the root cause like an on‑call SRE?”

ITBench‑AA aims to fill precisely that gap. Its design is rigorously pragmatic:

• Scenarios based on real trouble tickets: incidents, alerts, and logs sampled from IBM internal and client environments
• End‑to‑end executability: each task comes with a reproducible containerized sandbox; the agent must actually operate in real Kubernetes, Prometheus, Grafana, and cloud consoles
• Verifiable outcomes: it’s not about what the model says but what it changes—did configuration files, resource policies, and IAM rules actually fix the issue?

The underlying message couldn’t be clearer: enterprises don’t pay for fine explanations; they pay for problems being solved.

II. Three Tracks, Three Ways to Fail

ITBench‑AA currently covers three major enterprise IT scenarios, each with different pain points and exposing different weak spots in models.

1. SRE (Site Reliability Engineering)

This is the classic track. The task: given a misbehaving microservice cluster amidst a flood of alerts, you must identify the root cause, propose a fix, and execute it within a set time limit.

Example task:

# Simplified ITBench-AA SRE task description
incident:
  symptom: "checkout-service 99th percentile latency spikes from 200ms to 8s"
  blast_radius: "affects 23% of order requests"
  available_tools:
    - kubectl
    - prometheus_query
    - jaeger_trace
    - loki_logs
success_criteria:
  - root_cause_identified: true
  - mitigation_applied: true
  - slo_restored_within: 15m

A common flaw across frontier models: they can read logs but can’t “analyze” logs. Skilled SREs know which dashboards and metrics correlations to check first; models often start by spamming kubectl describe, quickly filling their context window, then hallucinating service names.

2. FinOps (Cloud Cost Governance)

A relatively new area. Tasks involve taking an abnormal cloud‑billing report, finding the cost‑spike root cause, and formulating a remediation plan.

This track reveals a model’s “engineering taste.” The cheapest plan isn’t always the right one—downgrading every GPU instance indeed saves money, but if training jobs then can’t finish, that’s an even bigger waste. ITBench‑AA’s scoring function jointly evaluates savings, SLO impact, and implementation risk.

One curious discovery from IBM: models tend to propose “aggressive” actions, such as shutting down seemingly idle instances—but those might be end‑of‑month batch‑processing nodes. This seemingly competent yet reckless behavior is catastrophic in production.

3. CISO (Security & Compliance)

This track stresses long‑chain reasoning. A typical task: detect a suspicious IAM role privilege escalation, determine whether it’s a real attack, its scope, whether to revoke permissions immediately, and whether revocation would disrupt normal operations.

Here, frontier models almost uniformly fail. The issue isn’t reasoning power but context dependency—enterprise security hinges on organizational context. The same permission change could be routine CI/CD in a test environment but a red flag in production. Lacking such context, models resort to pattern‑matching guesses.

III. Whose Score Looks Less Bad

IBM shared all results openly. Overall scores (weighted across three tracks) roughly break down as follows:

• Tier 1 (40 %–49 %) : GPT‑5, Claude Opus 4.5, Gemini 2.5 Pro Deep Think—three nearly neck‑and‑neck, differences within statistical noise
• Tier 2 (30 %–40 %) : Claude Sonnet 4.5, DeepSeek‑V3.5, Qwen3‑Max, Grok 4
• Tier 3 (below 30 %) : open‑source mid‑size models and versions without agent‑specific fine‑tuning

Notable insights:

First: Reasoning model ≠ model that gets the job done. Systems like OpenAI o‑series or Gemini Deep Think excel in math and programming contests, yet their advantage on ITBench‑AA is minor. Enterprise IT bottlenecks lie not in thinking deeply but in acting correctly—even the deepest thought fails if tool calls or context handling break down.

Second: The Claude line shows surprising strength in SRE tasks. Opus 4.5 scores slightly higher than GPT‑5 for SRE, consistent with Anthropic’s heavy investment in tool use and long‑chain agentic loops.

Third: Open‑source models aren’t as far behind as expected. DeepSeek‑V3.5 and Qwen3‑Max trail the top tier by only 8–12 points; considering their cost—fractions of GPT‑5’s—this price‑performance ratio is appealing for budget‑sensitive enterprises.

IV. Why Even GPT‑5 Can’t Break 50 %

IBM’s blog lists several common failure patterns—all worth noting for agent‑framework developers.

1. Breakdown in long‑range context management
Enterprise IT tasks often require dozens of sequential steps. By step 30, models forget key info from step 5. All frontier models show instability in ultra‑long agent loops—not due to small context windows, but to weak attention recall for an old log line seen hours earlier versus a human engineer’s working memory.

2. Tool‑use hallucination
Models fabricate nonexistent kubectl subcommands or invent cloud‑API parameter names. In a sandbox this means failure; in production it could be an outage.

3. Lack of “stop” judgment
Human engineers pause and ask “Can I safely touch this?” when uncertain; models charge ahead. ITBench‑AA offered an “ask for help” option, but most models rarely use it.

4. Weak verification loop
After applying a fix, models seldom verify real effectiveness, assuming success and moving on. That’s heresy in ops—the SRE golden rule is “always verify.”

V. What This Means for Developers and Enterprises

The release timing of ITBench‑AA is subtle. In recent months, enterprise attitudes toward AI agents have cooled after 2024’s exuberance. Gartner’s report—“40 % of agentic AI projects will be canceled by 2026”—already doused many CIOs with cold water. ITBench‑AA adds quantitative reinforcement to that caution.

But pessimism isn’t the benchmark’s intent. Seen differently, it gives “enterprise‑grade agents” a measurable yardstick for the first time. This means:

• Model vendors now have concrete optimization targets—future model releases will likely highlight ITBench‑AA scores
• Enterprise buyers gain a neutral reference instead of relying on glossy demos
• Agent frameworks (LangGraph, AutoGen, CrewAI, etc.) get authentic stress tests

For developers, the takeaway is straightforward: don’t worship “a stronger model will fix it.” Data from ITBench‑AA shows only limited improvement from GPT‑4 to GPT‑5 in enterprise IT tasks. The decisive factor is the agent framework itself—the memory module, validation loop, and human‑in‑the‑loop design define the deployment baseline.

Incidentally, for teams looking to compare multiple models on their own workloads, aggregator platforms like OpenAI Hub are handy—one API key can pull GPT‑5, Claude Opus 4.5, Gemini 2.5 Pro, DeepSeek‑V3.5, and Qwen3‑Max into a single test script, saving the hassle of per‑vendor quota management.

VI. What’s Next

At the end of its blog, IBM outlined future plans for ITBench‑AA:

• Expand to more domains: DevOps, database operations, network operations
• Add multi‑agent collaboration tasks to test team coordination
• Introduce adversarial testing simulating real production noise and misinformation
• Open‑source additional sandbox environments for community‑driven task creation

If ITBench‑AA gains SWE‑bench‑like de facto‑standard status, the next 1–2 years of enterprise‑agent evolution could become much clearer. As frontier models climb from 50 % to 70 % and then 90 %, that trajectory mirrors enterprise AI shifting from experimental investment to infrastructure investment.

Today’s 49 % may be unsatisfactory—but it’s an intriguing starting point full of possibilities.

References

• ITBench‑AA: Frontier Models Score Below 50 % on the First Benchmark for Agentic Enterprise IT Tasks — Original IBM Research and Artificial Analysis blog on Hugging Face, including full methodology and ranking data