Qwen-UI-Agent Enables AI to Truly Use Screens

Alibaba officially released Qwen-UI-Agent today. It is not merely a vision model that can recognize buttons and simulate clicks, but a GUI agent foundation designed for real digital devices. It aims to enable models to complete cross-application, cross-device, long-horizon tasks on phones, computers, browsers, and in DeepSearch environments.

According to data disclosed by Qwen officials, Qwen-UI-Agent achieved leading or highly competitive results across tests covering mobile devices, computers, browsers, web search, and GUI grounding. Among them, the 27B version achieved an 82.1% success rate on the MobileWorld test, 92.2% on the real-device MobileWorld-Real benchmark, and 97.5% on AndroidDaily. On computers, it scored 79.5% on OSWorld-Verified, 73.6% on WebArena, and 81.5% on ScreenSpot-Pro.

The significance of these results is not simply that another model has topped a leaderboard. Over the past two years, the primary battleground for AI agents has largely remained browsers and API tool calls. But software in the real world does not always provide structured interfaces for models. Online banking, corporate intranets, desktop clients, mobile apps, and legacy business systems can often still be operated only through screens. Whoever can use these interfaces reliably will be closer to becoming a truly general-purpose digital executor.

From “Understanding Pages” to “Completing Workflows”

The basic idea behind GUI agents is not complicated: the model reads a screenshot or interface structure, determines whether the next step should be clicking, typing, scrolling, dragging, or opening an application, and then continues making decisions based on the results of its actions.

The difficulty is that real-world tasks are rarely as simple as “click a button once.” A user might say:

I’m meeting a friend for coffee at Tianmuli today. Help me search Amap for the detailed address, open Dianping to find cafés within 1 kilometer, choose the most popular one, then go to Xiaohongshu and summarize the first five posts to see which coffee everyone recommends.

This instruction contains at least several stages: address lookup, cross-app switching, geographic filtering, ranking and comparison, content retrieval, information summarization, and result retention. A popup, login state, page layout change, or network delay in any one of the apps could interrupt the task.

Another example is arranging a business trip: check 12306 for the earliest high-speed train arriving at Hangzhou West, look up the subway travel time from Hangzhou West to Alibaba’s Xixi campus, calculate the arrival time at the office, and then create a meeting in DingTalk. This task requires more than language understanding. The model must maintain context across multiple applications, perform time calculations, and confirm the participants, subject, and reminder settings before finally creating the meeting.

The traditional tool-calling approach requires developers to first integrate an API or MCP service for each application and then design parameters and permissions for the model. Its advantages are clear structure and strong controllability, but its drawback is the high cost of coverage. Many applications do not have usable interfaces at all. The value of GUI interaction is that it bypasses this barrier: as long as a person can operate something on the screen, the model has an opportunity to learn how to operate it.

Qwen-UI-Agent aims to extend this capability from a single platform into a unified execution framework. Phones, computers, and browsers have different interaction methods, but the model can switch between them within the same task without requiring a completely separate agent to be trained for each platform.

A Unified Action Space: GUI and CLI Are No Longer Either-Or

One key design of Qwen-UI-Agent is its unified action space for GUI operations and CLI commands.

In desktop tasks, the model can click windows, enter text, and drag files. It can also invoke the command line when appropriate to process files, convert text, or perform batch operations. According to the technical materials, around 40% of actions in computer tasks are output as batch actions, while CLI actions account for nearly half.

The reasoning behind this is sound: GUI is well suited to tasks requiring visual understanding and contextual judgment, while CLI is better for clear, repetitive, and verifiable operations. Having the model simulate mouse clicks at every step is like asking a programmer to open a calculator and enter numbers one by one. It works, but it is neither efficient nor particularly stable.

For example, suppose a user asks the agent to move receipts from a phone’s photo album to a receipts folder on a remote desktop, rename them by date, and generate an Excel summary. Filtering images and switching applications may require GUI interaction, while renaming files, creating directories, and generating spreadsheets are better suited to CLI or structured tools. If the agent can autonomously choose between the two types of actions, its execution trajectory does not have to be completely constrained by screen clicks.

The technical report also mentions that the model can generate multiple actions in a batch within a single decision. For tasks such as continuous scrolling, entering a block of text, or opening multiple pages at fixed locations, batch actions can reduce the back-and-forth overhead of “observe once, act once.” This is similar to turning one-step driving into planning a section of a route in advance, but it requires the model to know which actions can safely be executed consecutively and which require another confirmation in the middle.

This is also where GUI agents differ from ordinary RPA. RPA typically relies on fixed coordinates, DOM selectors, or explicit rules, and can easily fail when a page changes. GUI agents attempt to adapt to changes through visual and semantic understanding. But the stronger the adaptability, the harder the behavior is to predict completely, which means security strategies cannot be treated as an auxiliary feature.

Real-Device Training Closes the Gap Between Simulated Environments and Reality

Alibaba built a mobile execution environment covering more than 100 real phones and over 150 applications for task construction, trajectory collection, training, and evaluation. It also developed the MobileWorld-Real real-device benchmark, which contains more than 400 tasks and over 100 applications.

This step matters. Many GUI models perform well in simulators or standardized webpages, but their scores drop significantly once they run on real phones. The reasons include unpredictable system permission popups, notifications covering pages, differences between application versions, network instability, page-loading delays, differences in screen size, and inconsistent account logins and device states.

A simulated environment is like a standardized circuit in a laboratory, while real devices are more like the old computers, popups, and temporary updates users face every day. If a model learns only in a clean simulator, it may mistake “the page always appears as expected” for a rule of the world. Real-device data forces it to learn how to handle exceptions rather than merely memorize ideal workflows.

The mobile results disclosed officially show that Qwen-UI-Agent-27B achieved 82.1% on MobileWorld, 8.9 percentage points higher than the next-best general-purpose model among the reference models. It achieved 92.2% on MobileWorld-Real, surpassing several flagship models, and reached 97.5% on AndroidDaily, close to a perfect score.

However, real-device benchmark scores still need to be interpreted within the context of their test configurations. Success on one task does not mean success across all devices, accounts, and network conditions. Task length, permission scope, failure-retry mechanisms, and the way human intervention is handled in a benchmark can all affect the results. For enterprise developers, average success rate is not enough. They also need to know the P95 execution time, model cost per task, whether failures can cause side effects, and whether tasks can be audited and replayed.

Long-Horizon Capability Depends on “Remembering What It Has Done”

The Qwen-UI-Agent technical report mentions that online reinforcement learning supports trajectory training with more than 100 rounds, while rollouts are accelerated by more than 10,000 concurrent environments. The model also participates in task and environment construction, failure diagnosis, and subsequent training planning through a data flywheel similar to AutoResearch.

This mechanism targets one of the most difficult problems currently facing agents: errors accumulate during long tasks.

If the model enters the wrong path in the fifth step, it may not discover that the result is wrong until the tenth step. If the user asks it to “schedule a meeting exactly on the hour after arrival,” and the model fails to correctly record the train’s arrival time, it may create an incorrect meeting in DingTalk. Long-horizon tasks are not simply a matter of chaining short tasks together. They require the agent to continuously maintain state, periodically verify intermediate results, and roll back or replan when deviations occur.

According to the reference materials, Qwen-UI-Agent focuses during training on exploration loops, popup interference, and step-level errors. It improves trajectory-level behaviors such as “verifying whether a task is complete” and “cross-modal collaboration” through Action RL and Online RL. This indicates that Alibaba is concerned not merely with whether the model can identify a button, but whether it can still understand the current goal, completed items, and next-step risks after dozens of execution steps.

That said, some caution is necessary. Training on trajectories of more than 100 rounds does not mean the model can reliably perform 100 rounds of real operations in deployment. Training has controlled environments, recoverable states, and explicit rewards. User devices may suddenly lose network connectivity, log out, display a CAPTCHA, or receive new notifications while the model is operating. Long-trajectory training addresses the upper limit of capability; production environments still require state management, exception recovery, and human takeover mechanisms.

Leading Several Benchmarks Does Not Mean a Mature General-Purpose Agent

The results currently disclosed for Qwen-UI-Agent cover several categories of capability.

• Mobile operation: 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily.
• Computer use: 79.5% on OSWorld-Verified and 40.0% on OSWorld-v2 Partial, with a claimed 58% reduction in execution steps compared with the baseline.
• Browser and search: 73.6% on WebArena, ranking first among the compared models; 75.0% on BrowseComp-ZH.
• GUI grounding: 81.5% on ScreenSpot-Pro, along with new best results on four other grounding benchmarks.
• General capabilities: Officials state that GUI post-training did not significantly compromise the model’s original general and agentic capabilities, and that it outperformed GUI-specialized models on relevant tasks.

Together, these results form a relatively complete capability profile. The model can not only see screens but also locate interface elements; it can operate not only phones but also desktops and webpages; it can execute GUI actions while incorporating command-line operations; and it attempts to handle not only short sequences but also long, cross-application workflows.

Compared with models that optimize for a single benchmark, this breadth is closer to product requirements. What developers actually need is often not “the top score on a particular webpage test,” but a model that can work continuously across office software, browsers, file systems, and mobile apps.

However, the benchmarks cannot simply be added together or compared directly. OSWorld, WebArena, and MobileWorld define their environments, tasks, action budgets, and evaluation methods differently. Claims of “leading” generally hold only under the corresponding test configurations. Browser automation in particular already benefits from mature tools and structured page information, while real-device mobile tasks are more susceptible to device and account state. Summarizing all the scores as “全面超越旗舰模型” is not sufficiently precise for product decisions.

More important is whether Qwen-UI-Agent will make its weights, inference interface, deployment requirements, and complete evaluation scripts public. The technical report says that the 27B model achieves a balance among capability, real-device reliability, and deployment efficiency. For developers, however, GPU memory usage, quantization performance, average tokens per task, visual input costs, and concurrent throughput will determine whether it can enter real-world systems.

Security Boundaries Determine Whether It Can Enter Production

A model that can operate a screen inherently has more privileges than an ordinary question-and-answer model. It can open applications, read messages, move files, and, in some scenarios, make payments, delete data, and send notifications. Therefore, Qwen-UI-Agent emphasizes integrating security judgments throughout the entire task-execution process.

The strategy described officially is as follows: when faced with illegal or high-risk requests, the model refuses to perform interface operations. For sensitive actions such as payments, data deletion, and privacy authorization, it stops at critical steps, explains the situation to the user, and continues only after obtaining explicit confirmation.

This principle basically meets the minimum requirements for an agent product: dangerous tasks must not be executed, and sensitive actions must not be decided autonomously. In actual deployments, however, “explicit confirmation” cannot simply mean that the model asks “Shall I continue?” in a chat window. More reliable designs generally also require:

• Independent permissions and secondary confirmation for payments, deletion, external sharing, authorization, and other actions;
• Displaying the specific changes that are about to occur before execution, rather than showing only an abstract task name;
• Recording complete operation trajectories to support replay, undo, and accountability;
• Minimizing the handling of personal data transferred across applications;
• Forcing human takeover when CAPTCHAs, login pages, privacy screens, or unknown popups appear;
• Setting clear trigger conditions, time windows, and operational boundaries for proactive services.

The “proactive service” feature deserves particular attention. The technical report mentions that the Harness layer supports maintaining stateful workflows across phones and computers and proactively initiating useful services. For example, after detecting that a flight has been canceled, the agent could search for alternative flights or high-speed trains, combine the options with the user’s schedule, and present a plan before continuing after confirmation.

This is more valuable than passively answering questions, and more dangerous. Proactive services mean that the model can initiate searches and operations without the user issuing step-by-step instructions. It needs to understand not only “what does the user want to do?” but also “when may it act?”, “at what point must it stop?”, and “which outcomes must be left for a person to decide?” Without a detailed permission system, proactivity can easily become an amplifier for erroneous actions.

What It Means for Developers

The emergence of Qwen-UI-Agent may change the technology stack choices for agent applications.

First, GUI will continue to be an important complement to APIs and MCP. For applications without open interfaces, visual operation is a realistically viable integration method. For systems that do have APIs, GUI can serve as a fallback path for handling the small number of workflows that cannot be called in a structured way.

Second, a single model covering multiple platforms can reduce the engineering complexity of cross-platform agents. In the past, developers might have needed to maintain separate agent logic for Android, desktop browsers, and remote desktops. Now they can try using unified task descriptions and action abstractions while leaving platform differences to the model. This does not mean platform adaptation work disappears. Reliable systems still require device management, session isolation, permission controls, and fault recovery.

Third, GUI agents will not replace structured tools. For orders, payments, inventory, databases, and enterprise processes, API calls are generally faster, more stable, and easier to audit than screen clicks. A more reasonable architecture is “API first, GUI as a complement”: use structured interfaces wherever possible, and use GUI when the system needs to understand a page or operate a legacy system.

Fourth, evaluation must expand from “was the task completed?” to “was it completed correctly?” Developers need to track the rate of erroneous operations, the rate of irreversible actions being triggered, recovery capability after task interruptions, cross-device state consistency, and the frequency of human takeover. An agent with a very high success rate that occasionally deletes files by mistake is still unacceptable in an enterprise environment.

My Assessment: The Direction Is Right, but Productization Still Has Three Hurdles

The value of Qwen-UI-Agent lies first in bringing real devices, mixed GUI and CLI actions, long-trajectory reinforcement learning, a data flywheel, and a cross-device Harness into a single system. Compared with merely “teaching a model to click webpages,” this approach is closer to what users actually want from a digital assistant: provide a goal, and let the system complete the work across multiple applications.

Its mobile performance is especially worth watching. The mobile app ecosystem is highly fragmented, pages and interactions change rapidly, and real-device environments are more difficult to standardize than browsers. If the MobileWorld-Real and AndroidDaily results can be reproduced across more devices, account states, and longer tasks, Qwen-UI-Agent could become an important foundation model for mobile agents.

But it cannot yet be equated with “general-purpose computer use has been solved.” Benchmark scores do not answer questions about deployment costs, consequences of failure, cross-version generalization, or long-term operational stability. Training on 100-round trajectories does not automatically translate into 100 rounds of production-grade execution. The triggering and authorization mechanisms for proactive services also require extensive validation in public products.

For developers, the most pragmatic current approach is to treat it as a GUI execution layer rather than allowing it to take over an entire business independently. Start with low-risk, reversible, and easily verifiable tasks, such as collecting information across applications, archiving files, filling out page drafts, and operating in test environments. Processes involving payments, deletion, external communication, and account permissions must retain human confirmation.

Qwen-UI-Agent will also push model competition beyond “whose conversational ability is stronger” toward “who can use existing software more reliably.” If Alibaba later opens up more complete model, inference, and device-runtime tools, domestic developers will have another deployable GUI agent option. OpenAI Hub currently aggregates models such as GPT, Claude, Gemini, and DeepSeek and is compatible with the OpenAI format. For applications that need to combine a GUI execution model with general reasoning, coding, or search models, a unified interface can be used to build multi-model collaboration, but the specific models and capabilities available should still be determined by the platform’s real-time list.

The final winner in this competition will not be determined solely by who can locate buttons most accurately in screenshots. It will come down to who makes fewer mistakes in the real world, can explain what happened, can recover from failures, and knows when to stop and wait for human confirmation at critical moments. Qwen-UI-Agent has advanced the question from “can the model click?” to “can the model complete a task?” The next question is whether “users dare to entrust tasks to it.”

References

• ITHome: Alibaba Releases Qwen-UI-Agent, Focused on Enabling Models to Truly “Use” Every Screen — Summarizes Qwen’s official disclosures on the model’s positioning, benchmark results, and real-device environment.
• Zhihu Column: Alibaba’s Qwen-UI-Agent No Longer Only Clicks Screens: It Can Run Hundred-Step Tasks Across Phones, Webpages, and CLIs — Introduces the technical designs for mixed GUI and CLI actions, batch actions, proactive services, and cross-device workflows.