Orca-Bench Brings Agents into the On-Call Room

Released in late July, Orca-Bench moves the testing ground for language model agents away from customer service conversations, web navigation, and code repair—and into a more dangerous setting: the production on-call room.

The question it asks is not “Can the model explain an error message?” but something much closer to an enterprise procurement decision: When a service triggers an alert in the middle of the night, context is incomplete, tool outputs are full of noise, and a single mistaken action could worsen the incident, can an agent truly take over the shift on its own?

This is where Orca-Bench matters. Over the past two years, agent benchmarks have increasingly resembled specialized exams: SWE-bench evaluates code repair, WebArena evaluates web navigation, and τ-bench evaluates multi-turn tool use under policy constraints. Orca-Bench pushes the focus into online incident response—a task that simultaneously demands diagnosis, planning, tool use, risk control, and state tracking. It is also an unavoidable test on the path from agents that “can give demos” to agents that “can stand watch.”

The paper’s answer is not reassuring: current models can already handle some troubleshooting work, but they are not yet ready to receive operational access to production systems without supervision and guardrails. More precisely, they resemble newly assigned on-call engineers who have read a vast number of operations manuals and react quickly, but have not participated in enough incident postmortems.

On-Call Is Not a Log Q&A Task

Many existing evaluations reduce incident diagnosis to static question answering: give the model a log excerpt, several monitoring screenshots, or an incident description, then ask it to identify the root cause. Such tasks can test knowledge and reasoning, but they omit the most difficult aspects of real on-call work.

A production incident is not a puzzle that stands still and waits for someone to answer it.

A typical sequence might look like this: a latency alert appears first; the on-call engineer checks service metrics and discovers that the number of database connections has increased; they then review deployment records and notice that a deployment completed ten minutes earlier; after a rollback, the error rate declines, but the backlog of requests remains; at that point, they must verify that the rollback has actually taken effect while avoiding a second disruption caused by repeated actions. Every step changes the system state that will be visible in the next step.

An on-call agent therefore needs to handle at least five things simultaneously:

• Build a working hypothesis from alerts, logs, metrics, topology, and change records;
• Select the appropriate diagnostic tools and correctly specify time windows, service names, and filters;
• Update its judgment based on new evidence rather than clinging to its initial guess;
• Distinguish read-only investigation from high-risk operations and control the boundaries of actions such as rollbacks, restarts, and scaling;
• Continue verifying after the system recovers rather than declaring victory as soon as a single metric turns green.

This is much harder than ordinary tool use. Choosing the right function name only shows that the agent knows where the hammer is; true on-call capability also requires deciding whether the object in front of it is a nail, which nail to strike, and whether doing so will rupture the pipe next to it.

Orca-Bench’s approach is to place these intertwined capabilities in a single executable evaluation environment, rather than separately testing log comprehension, tool selection, and natural-language responses.

The Focus Shifts from “Saying the Right Thing” to “Finishing the Job”

Orca-Bench continues an important recent shift in agent evaluation: instead of comparing only the textual similarity between a final response and a reference answer, it examines the agent’s action trajectory and whether the system actually returns to the target state.

This distinction is crucial.

A model might correctly state that “a recent deployment introduced a regression and should be rolled back,” yet select the wrong service, roll back to the wrong version, or forget to check dependent services during execution. Conversely, another trajectory may not follow the human troubleshooting manual word for word, but if it takes safe, evidence-based actions and restores the critical service, it should be considered a valid solution.

What benchmarks of this kind truly care about is therefore not an eloquent incident summary, but signals such as the following:

1. Whether the task was completed: Was the impact mitigated, and did the service return to an acceptable state?
2. Whether the root cause was identified: Did the agent restore the service by chance, or did it form a judgment supported by evidence?
3. Whether tool calls were correct: This includes the tool, parameters, target, and time range—not merely whether the call was syntactically valid.
4. Whether actions were safe: Did the agent perform any unnecessary, unauthorized, or potentially impact-expanding operations?
5. Whether the process was efficient: Did it fall into repeated queries, ineffective retries, or context bloat?
6. Whether the outcome was verified: Did it confirm that metrics declined, instances were healthy, and user impact was resolved rather than stopping prematurely?

This also explains why a simple function-calling success rate cannot represent on-call capability. Having a model generate valid JSON is merely equivalent to an on-call engineer knowing how to type a command; what enterprises actually care about is whether the engineer will run that command against the wrong cluster.

The Hardest Part Is Not Knowledge, but Closing the Loop

Judging from the recurring problems exposed by existing agent evaluations, the most important question in Orca-Bench is not whether a model remembers a particular Kubernetes command, but whether it can reliably complete the “observe—hypothesize—act—verify” loop.

Premature Assumptions Compound Across the Toolchain

Language models are good at quickly completing a story from a small number of clues. In conversation, this makes their answers sound fluent; in incident response, it can become a source of risk.

For example, when an agent sees a CPU spike and a recent deployment, it can easily jump straight to the conclusion that “the new version introduced a performance regression.” But the CPU increase could also be caused by amplified retries due to downstream timeouts, while the deployment record is merely a temporal coincidence. If the model locks onto a root cause too early, every subsequent query may search only for evidence that supports the original hypothesis.

This kind of failure is not a single wrong answer, but a complete, coherent, and seemingly professional incorrect trajectory. The agent may even cite genuine logs to build a self-consistent explanation for the wrong conclusion.

Parameter Errors Are More Subtle Than Choosing the Wrong Tool

In real systems, an agent may know that it should “check the logs” without being able to correctly handle namespaces, instances, regions, and time windows. It may query the wrong environment for the correct service, or use filters that are too narrow and conclude that “nothing is wrong.”

Parameter errors are particularly dangerous because the tool will often return results normally. The absence of an error does not mean that the query was correct. Both automated evaluations and production monitoring must audit tool parameters and query coverage rather than recording only whether the call succeeded.

Recovery Actions Require Causal Judgment

Restarting instances, increasing replica counts, and rolling back deployments can all improve metrics in the short term, but “improvement” does not necessarily prove that the root cause has been resolved. Scaling out may merely dilute traffic pressure temporarily, while a restart may clear the state that triggered the problem.

A reliable on-call agent should know which actions provide mitigation and which provide a fix, and clearly distinguish between them in the incident record. Otherwise, it can easily mistake “the alert has temporarily stopped” for “the problem has been solved.”

Declaring Success Too Early Remains a Common Agent Failure

Many agents stop after receiving a single positive signal: the error rate falls, a pod resumes running, or one health check passes. But production recovery usually requires observing multiple metrics and waiting through a sufficiently long stability window.

A large part of an experienced human on-call engineer’s expertise lies in knowing “when the job is not yet done.” This kind of conservative judgment is difficult to measure through single-turn question answering, yet it directly determines whether an agent is ready for production.

Orca-Bench Is Closer to Production Than Traditional Agent Benchmarks, but It Is Not Production

Compared with τ-bench, Orca-Bench focuses more heavily on infrastructure and reliability engineering. τ-bench primarily examines whether an agent can converse with users and correctly modify backend state under business rules in domains such as retail and aviation. Orca-Bench asks whether an agent can take a sequence of actions based on observable signals from a running technical system.

Compared with SWE-bench, it is also not simply “code repair for operations.” SWE-bench typically provides a relatively well-defined issue, code repository, and test results, with the goal of generating a patch that passes the tests. The boundaries of on-call incidents are less clear: an alert may be only a symptom, the root cause may span multiple services, and mitigating the incident is often different from permanently fixing it.

Orca-Bench therefore fills an important gap, but it should not be misread as evidence that “a high score means an agent can automatically take over production.” No matter how realistic a benchmark environment is, it is difficult to fully reproduce the accumulated complexity of an enterprise’s internal systems, including permission models, internal abbreviations, incomplete documentation, specialized deployment processes, and implicit constraints that exist only in senior engineers’ memories.

There is also a more practical issue: incident distributions have a pronounced long tail. Common failures can be covered by templates, runbooks, and historical data, while incidents with truly severe impact often result from combinations of multiple low-probability conditions. Average success rates on a static test set cannot be directly converted into safety guarantees for production environments.

Orca-Bench is therefore better suited as an admission test and regression test than as an autonomous-driving license. It can eliminate agents that cannot even complete the investigation loop, and it can compare different models, prompts, tool designs, and safety strategies. But the decision to grant write access must still incorporate each enterprise’s own incident replays and shadow-mode operating data.

Enterprises Should Not Start with “Unstaffed On-Call”

The most immediate lesson from Orca-Bench is that on-call agents should be deployed first on low-risk, high-frequency, and verifiable tasks rather than jumping directly to fully autonomous incident response.

A more sensible staged path would be:

• Stage 1: Information aggregation. The agent automatically collects relevant alerts, deployment records, log anomalies, and service dependencies, then generates a timeline without making any changes;
• Stage 2: Investigation recommendations. The agent proposes queries and diagnostic steps, and invokes read-only tools only after engineer approval;
• Stage 3: Controlled remediation. The agent may execute only preapproved, reversible runbooks with limited impact, using parameter allowlists;
• Stage 4: Limited autonomy. The agent automatically mitigates specified failure types for specified services within an explicit budget, while preserving the ability for humans to abort;
• Stage 5: Expanded coverage. Permissions increase gradually only after the agent has demonstrated long-term reliability in replays, shadow on-call operations, and real incidents.

The most important architectural choice here is not adopting a larger model, but designing the permission system to distrust the model by default. Diagnostic queries and change operations should use separate credentials; high-risk actions such as deletion, traffic switching, and database changes should require secondary confirmation; every tool call should record its parameters, return values, and decision rationale; and an independent checker should verify the system state after the incident ends.

In other words, a production-grade agent is not “a model plus dozens of tools,” but a control system built around an unreliable decision-maker.

Stability Matters More Than Model Rankings

Agent teams also need to be wary of single-run success rates. On-call work is not an exam where one can repeatedly draw responses and select the best answer. When an incident occurs in the middle of the night, the system usually has only one window in which to act.

If an agent sometimes performs the correct rollback for the same incident, sometimes scales out incorrectly, and sometimes stops halfway through the investigation, its average score may look respectable even though it cannot be trusted. Rather than looking only at pass@1, teams should test consistency across multiple independent runs and separately track worst-case outcomes and the rate of high-risk actions.

At minimum, the following metrics should be retained:

• Stable success rate across repeated runs of the same incident;
• Number of ineffective, repeated, and high-risk tool calls;
• Time and token cost required to achieve the first effective mitigation;
• Rate of incorrect root-cause judgments and missing evidence;
• Proportion of unverified recoveries, premature exits, and incorrect escalations;
• Additional cost required for humans to correct the agent’s mistakes after taking over.

The final metric is often overlooked. An agent that is “useful most of the time” may provide less overall value than a less capable system that always remains in read-only mode if its failures create configuration drift that is difficult to recover from.

The Dividing Line for On-Call Agents Is Knowing When Not to Act

The timing of Orca-Bench is significant. Current frontier models already have sufficient tool-use, long-context, and code-comprehension capabilities to deliver impressive operations demos. But what truly prevents them from joining the on-call rotation is no longer simply whether they are “smart enough”; it is their reliability, auditability, and risk boundaries.

The benchmark’s greatest value lies in advancing the industry discussion from “Can the model find the root cause?” to “Can the model safely complete an incident response?” The former demonstrates capability; only the latter carries production responsibility.

Our assessment is that language model agents are already suitable as on-call copilots, particularly for automatically organizing context, conducting read-only investigations, matching historical incidents, and generating remediation recommendations. In general-purpose scenarios, however, they are not yet suitable as independent on-call operators. Models can quickly learn runbooks, but they still handle conflicting evidence, permission risks, and recovery verification outside those runbooks inconsistently.

The factors that ultimately create differentiation may not be limited to foundation model scores. Clear semantics in tool interfaces, complete observability data, support for dry runs and rollbacks in the execution layer, and independent verifiers can all significantly affect final performance. If enterprises compare only model leaderboards while handing agents chaotic internal APIs and excessively broad production permissions, what they end up measuring may not be intelligence, but the blast radius of the next incident.

Orca-Bench has brought agents into the on-call room, but there is still a long way to go before human engineers can be taken off the call list.

References

• τ-bench: A New Benchmark for Evaluating the Reliability of Language Agents in Real-World Interactions: Explains how τ-bench evaluates agent consistency through multi-turn conversations, business rules, and tool use, providing context for the evolution of evaluations that led to Orca-Bench.

Note: The core information in this article comes from the paper “Orca-Bench: How Ready Are Language Model Agents for Oncall?”, published in July 2026. Because the reference links were restricted to the specified domain, the paper’s original webpage is not included in the references.