Wall Street Puts Agents to the Test: Qwen Tops Office Tasks

Qwen Office Takes First Place in a Wall Street Test

On August 19, Wall Street investment bank Jefferies released the results of a hands-on test of leading AI agents worldwide. Across a suite of office productivity tasks involving eight products, Alibaba’s Qwen Office ranked first, outperforming mainstream U.S. agent tools such as Claude Cowork and Codex.

This was not a leaderboard that merely compared the accuracy of model responses. Jefferies placed the products in environments that more closely resembled real-world work, requiring them to read files, search the web, operate browsers, create presentations, and produce marketing posters based on reference images. In other words, the test was not about “which agent is better at chatting,” but “which agent can finish an assigned task.”

The results showed that Qwen Office performed particularly well in complex office tasks, web browser control, and multimodal content generation. It was also the only agent product in the evaluation to score above 90 in every category.

These results deserve attention, but they should not be simplistically interpreted as “the Qwen model comprehensively defeated U.S. models.” An agent’s final performance is often not determined by the underlying model alone. The model is responsible for understanding tasks, planning steps, and generating content, while the product itself must provide tools, context, permissions, feedback mechanisms, and error recovery capabilities. What truly determines the user experience is whether the model and this operating framework can work effectively together.

The Competition Is Not About Answering Questions, but Completing Tasks

Jefferies set up five categories of real-world office tasks:

• Produce a summary of a company’s annual report based on multiple files;
• Search online for and compare companies’ operating data;
• Operate a real desktop browser to retrieve information and generate documents;
• Create an English-language PowerPoint presentation based on data;
• Generate a marketing poster based on reference images.

These tasks cover several of the most common stages involved when enterprises use agents in their day-to-day operations: understanding internal materials, obtaining external information, operating software, organizing structured output, and producing text-and-image content.

Traditional chatbots usually stop at the first step: the user asks a question, and the model returns a block of text. What enterprises actually need, however, is an end-to-end process that goes from “finding information” to “delivering results.” For example, an investment researcher may not simply want an agent to summarize several annual reports. They may want it to identify changes in revenue growth, compare peer data, flag anomalies, and organize the conclusions into pages that can be inserted directly into a presentation. If anything goes wrong at any stage, the final result may lose its value.

This is also one reason Qwen Office stood out in the test. Rather than relying solely on the model itself, it combines file processing, web operations, office document creation, and image generation into a single workflow. For users, the difference is not “how polished the model’s answer is,” but whether they must constantly copy and paste on its behalf, switch windows, supply additional context, and correct formatting.

The Harness Is Becoming the Second Core Variable for Agents

Agent products involve an often-overlooked concept known as the harness. It can be understood as the complete working system built around the model, encompassing task instructions, context management, tool use, operational boundaries, feedback and error correction, and organizational governance.

The model is like a highly capable new employee, while the harness is analogous to the company’s management system and workstation. Even a smart new employee will struggle to perform tasks well without access to data, system permissions, or a clear understanding of procedures. Conversely, a model that is not top-tier may still achieve better results on specific tasks if it has clear task decomposition, a stable toolchain, and effective error feedback.

Jefferies’ test effectively sends an important signal to the industry: competition among agents is no longer just competition among large models. It is now a systems-level competition involving “models + tools + data + processes.”

This also explains why model leaderboards alone often fail to predict how an agent will actually perform in an enterprise. A model may score highly on public benchmarks, but that does not mean it can reliably open a browser, recognize page states, fill in forms, save files, and pass the results to the next step. The difficulty of real-world work often lies not in generating a block of text, but in whether an agent can execute dozens of consecutive actions with fewer mistakes, detect errors, and recover from them.

Qwen Office’s Advantage Lies in Its Closed-Loop Office Workflow

Qwen Office entered public beta on August 3. Both individual and enterprise users can access it through its official website. The web version and standalone PC client are already available, while access through DingTalk’s PC and mobile apps is being rolled out. The product integrates previous agent products including QoderWork, MuleRun, and Wukong, and incorporates Alibaba’s latest flagship model, Qwen3.8.

From a product positioning perspective, Qwen Office is not simply a model-based application. It is competing to become the gateway to enterprise productivity. It covers desktop agents, cloud agents, and enterprise collaboration agents, seeking to bring personal tasks, computer operations, and organizational collaboration into a single product.

Qwen Office has already established initial integration with collaboration environments such as DingTalk. It can handle tasks including summarizing group chats, creating documents and spreadsheets, and sending and receiving messages and emails. For enterprise users, the value of these capabilities does not lie in occasionally generating a summary, but in whether they can be continuously integrated into everyday workflows: identifying action items in group chats, adding tasks to spreadsheets, accessing internal data, and then generating results that can be approved or sent.

Alibaba’s long-standing experience in enterprise software also gives Qwen Office certain advantages in workplace scenarios. To deploy office agents effectively, organizations generally need to connect them to instant messaging, documents, spreadsheets, email, databases, and permission systems. Model capability is only the starting point. What enterprises truly care about is whether data can be connected, permissions can be controlled, operations can be audited, and employees are willing to keep using the product in their daily work.

These capabilities are not easy for users to see in a single demonstration, but they determine whether a product can move from “looking impressive” to “actually being used by the company.”

Claude Cowork and Codex Have Not Lost Their Value

Qwen Office’s first-place ranking does not mean that products such as Claude Cowork and Codex are at a disadvantage in every type of work. Different agents do not have identical product orientations.

Claude Cowork focuses more on long-document comprehension, knowledge work, and continuous task processing. It remains attractive for complex text analysis, research-oriented writing, and scenarios requiring longer context windows. Codex, meanwhile, is more closely aligned with software development workflows, focusing on code comprehension, modification, test execution, and engineering collaboration. For developers, an agent deeply optimized for code repositories cannot be judged solely by the results of office software tests.

A more accurate interpretation of the leaderboard is therefore that Qwen Office completed a more comprehensive closed-loop workflow across the office tasks defined by Jefferies. It reflects product fit and systems engineering capability, rather than absolute model capability across every scenario.

Development teams should likewise avoid looking only at overall rankings when selecting an agent. They should first break down their own tasks. If their work mainly involves reading materials, conducting research, and writing reports, an office-oriented agent may be more suitable. If it mainly involves modifying code, running tests, and handling issues, a coding agent may have the advantage. If the agent must be integrated into enterprise processes, teams should focus on connectors, permissions, auditing, private data isolation, and failure recovery.

Cost May Determine Whether Agents Can Be Commercially Viable

The test also produced a conclusion more commercially relevant than the rankings: cost is becoming an essential variable in agent commercialization.

To complete a task, an agent may require multiple rounds of model inference, dozens of tool calls, visits to numerous web pages, and access to large numbers of files, followed by replanning if it fails. The user sees a single task, but the service provider bears a series of inference and execution costs. If every task uses the most expensive model, the longest context window, and the most complex toolchain, the product can easily encounter a situation where “the more it is used, the more money it loses.”

This differs from the cost structure of traditional SaaS. Traditional office software is generally priced by account, storage capacity, or feature module, with relatively controllable marginal costs. An agent is more like a digital employee that consumes resources according to its workload: the more complex the task, the more model tokens, computing power, and external tool resources it consumes. A free tier can help attract new users, but it is difficult for it to support unlimited, high-frequency automation tasks.

Qwen Office currently offers free, personal standard, and personal premium subscription plans, indicating that agent products have begun to shift from providing a “model experience” to pricing a “service.” The industry may subsequently adopt more granular billing models, such as charging by the number of executions, task complexity, or team seat, or pricing model calls, tool calls, and enterprise connectors separately.

For enterprise customers, the real calculation should not be “how much does one account cost per month,” but the total cost of completing a task and how much human labor it replaces. For example, if an industry data compilation task that would take an analyst half a day can be completed by an agent in a few minutes with an acceptable error rate, it may have commercial value even if the cost per call is relatively high. Conversely, if the agent frequently requires manual review and repeated execution, and the time saved is insufficient to offset the cost, then even a low price is meaningless.

Controllability Is the Dividing Line for Commercialization

Once agents enter the enterprise, their risks increase accordingly. They may read sensitive files, send emails, modify business data, or even trigger real transaction processes. Simply increasing model parameters cannot solve these problems.

Enterprises need a controllable execution mechanism: which data may be accessed, which tools may be called, which actions require human confirmation, whether errors can be rolled back, and whether every action is logged. For developers, an agent’s permission system should more closely resemble a service account and workflow engine than a chat window with unlimited privileges.

From this perspective, the boundaries, feedback, and governance capabilities within the harness may have a greater influence on enterprise purchasing decisions than the quality of an answer in a one-off demonstration. The focus of future competition may not be who can make an agent complete one impressive task, but who can keep it running continuously for a month while remaining stable, auditable, and predictable in cost.

What This Means for Developers

The implications of this test for developers are straightforward: when building agents, the underlying model is only one component. Other areas requiring substantial engineering effort include:

• Breaking tasks into observable, retryable steps instead of asking the model to complete everything in a single pass;
• Designing clear input and output protocols for tool calls to reduce the model’s discretion in edge cases;
• Adding approval and human confirmation mechanisms for high-risk operations;
• Preserving task traces, tool-call results, and causes of failure to facilitate debugging and evaluation;
• Dynamically selecting models according to task difficulty rather than using the highest-cost configuration for every request;
• Building evaluation sets with real business data and focusing on completion rates, rework rates, latency, and per-task costs rather than only general benchmark scores.

Teams already using multiple models such as GPT, Claude, Gemini, DeepSeek, or Qwen can also switch between them and compare costs through a unified OpenAI-compatible API. OpenAI Hub supports unified access to leading models and is suitable for practical stress testing across model capability, pricing, and stability. However, such platforms can only solve problems at the model access layer. Enterprises must still build their own systems for permissions, data security, and business process governance.

Overall, Qwen Office’s first-place finish in Jefferies’ hands-on test shows that Chinese vendors have become competitive in office agent product integration and enterprise scenario adaptation. More importantly, the test has redirected industry attention away from “which model is the smartest” and back toward two more difficult questions: whether agents can reliably complete real work, and how much completing that work actually costs.

In 2026, agent competition is moving from capability demonstrations at product launches to enterprise balance sheets. Models determine the ceiling, the harness determines deployment, and cost determines scale.

References

• ITHome: Alibaba’s Enterprise Agent Product “Qwen Office” Enters Public Beta: Introduces Qwen Office’s public beta, product formats, and integration with Qwen3.8.
• ITHome: Provides publicly available reporting on China’s technology industry and AI product developments.