OpenClaw Major Update: The Agent Grows "Eyes and Hands"

OpenClaw has just rolled out a low-key but monumental version update. This time, it’s not just bug fixes—it’s giving the Agent “eyes and hands” – the ability to see screen contents and control the mouse and keyboard. Together with the newly added Dreaming mechanism, the Agent now has the ability to think offline.

What does this mean? AI Agents are evolving from “conversational tools” into “system-level intelligences.” They no longer just answer questions—they can proactively execute tasks, work continuously, and self-optimize.

Visual Interaction: The Agent Can Finally “Understand” Interfaces

In the past, an Agent’s way of operating a computer was primitive: either through API calls or via command line. When it encountered software without APIs, it was helpless; complex GUIs were even more of a challenge.

This update gives the Agent visual capability. It can capture screen content, understand interface layouts, and recognize the positions and functions of buttons, input boxes, and menus. It can then simulate mouse clicks and keyboard input to perform operations.

What’s the value of this capability? For example:

• Traditional approach: If you want the Agent to organize your emails, it needs the email provider’s API and must handle all kinds of authorization and authentication.
• Visual-interaction approach: The Agent directly opens your email client and browses, categorizes, and archives emails just like a human.

Technically, this combines the visual understanding ability of multimodal models with the operational capability of RPA (Robotic Process Automation). But the real breakthrough is that the Agent doesn’t need to know an interface’s structure in advance—it can understand and adapt to any UI in real time.

This expands the Agent’s applicability by several orders of magnitude. In theory, if a human can operate a piece of software, the Agent can learn it too.

Dreaming Mechanism: The Agent Begins to “Think Offline”

Another core of this update is the Dreaming mechanism. The name is fitting—when the Agent isn’t working, it’s “dreaming”: organizing memories and optimizing strategies.

Memory Becomes “Cognition” Instead of Just “Storage”

Previously, the Agent’s memory mechanism was crude: it stored all conversation histories and operation logs, retrieving them as needed. Problems included:

1. Context confusion: The more it remembered, the harder it was to find useful information.
2. Lack of growth: It made the same mistakes repeatedly—no learning ability.
3. Poor generalization: Experience from scenario A didn’t transfer to scenario B.

The Dreaming mechanism turns raw memory into structured cognition. Specifically:

Raw Memory:

User: Book me a flight to Shanghai for tomorrow
Agent actions: Open Ctrip → search flights → choose time → fill in info → pay
Result: Booking successful

Structured Cognition:

Task pattern: Flight booking
Key steps:
  1. Confirm origin, destination, and time
  2. Compare prices and times
  3. Confirm passenger info
  4. Complete payment
Notes:
  - Confirm ID info beforehand
  - Double-check time before payment
  - Save the order number

When idle, the Agent automatically runs the Dreaming process, extracting reusable task templates from scattered operation logs. Next time it encounters a similar task, it can invoke the template directly—greatly improving efficiency and accuracy.

From “Executor” to “Operating System”

At its core, the Dreaming mechanism adds a “runtime” to the Agent. This is crucial because it marks the Agent architecture’s evolution toward an operating system form.

Traditional Agents are stateless:

• Receive command → execute → return result → end
• Each run starts from scratch, with no continuity

Under the new architecture, the Agent can run persistently:

• Foreground tasks: Respond to user instructions and perform operations
• Background tasks: Organize memories, optimize strategies, pre-load resources
• Scheduling: Allocate computing resources by priority

It’s like upgrading from a single-threaded program to a multitasking OS. The Agent is no longer a “use-and-discard” tool but a continuously running, evolving system.

Technical Implementation: Dual Engine of Skills + APIs

OpenClaw’s capability framework revolves around two cores—Skills and APIs.

API: Connecting to the External World

APIs answer the question “What can it do?” Through the MCP (Model Context Protocol), OpenClaw can call various external services:

• File systems: Read and write local files
• Network services: Access third-party APIs
• Databases: Query and update data
• Messaging platforms: Integrate with Feishu, WeCom, QQ, etc.

The MCP’s design is clever: it’s not a simple API aggregator—it provides a unified context management mechanism. The Agent can pass states among services, enabling cross-platform workflows.

Skills: Defining Execution Logic

Skills answer the question “How does it do it?” A Skill is essentially a reusable task template, including:

• SKILLS.md: Task description—tells the Agent “who you are” and “what you do”
• scripts: Executable scripts providing concrete tools
• references: Reference documents defining execution flow
• assets: Supporting materials like configuration files or templates

For example, a “Code Review” Skill might include:

# SKILLS.md
You are a code review assistant responsible for checking code quality and potential issues.

## Review Process
1. Read code files
2. Check syntax errors
3. Analyze code style
4. Identify potential bugs
5. Generate a review report

## Notes
- Focus on security vulnerabilities
- Check for performance issues
- Ensure code readability

The Skills marketplace is key to the OpenClaw ecosystem. Developers can publish their own Skills, and users can install them with one click. Tencent’s SkillHub, launched in March, is a Skills community optimized for domestic users, offering fast downloads and Chinese search.

Ecosystem Evolution: From Tech Circles to Mainstream Use

OpenClaw’s development path is clear:

Phase 1 (Jan–Feb 2026): Tech users self-deploy

• Tencent Cloud launched Lighthouse OpenClaw
• Alibaba Cloud released LEC OpenClaw
• Users had to configure and debug manually

Phase 2 (Feb 2026 – Present): Model vendors release plug-and-play versions

• KimiClaw (Moonshot AI)
• MaxClaw (MiniMax)
• AutoClaw (Zhipu)
• Integrated proprietary models to lower the entry barrier

Phase 3 (Underway): Integration into mainstream platforms

• Entrances via WeChat, QQ, and WeCom
• From “standalone apps” to “system capabilities”
• Mass adoption begins among consumers

This trajectory is reminiscent of early mobile internet development: first tech geeks tinker, then big firms release standardized products, and finally the technology becomes infrastructure.

Impact on Developers: Test Boundaries Are Being Redefined

For developers, the most significant impact of this update isn’t a few new features—it’s a paradigm shift in testing.

From Function Validation to Behavioral Stability

Traditional software testing focuses on:

• Does input A produce output B?
• Are edge cases handled correctly?
• Are error messages shown for exceptions?

But Agent systems are different. The same input might produce several acceptable execution paths. You need to verify:

• Whether the Agent’s reasoning is sound
• Whether execution processes are stable
• Whether results meet expectations

It’s like testing a person, not a program.

From Interface Testing to Decision Chain Testing

An Agent’s workflow is dynamic:

Input → Reasoning → Tool Call → Memory Update → Reasoning Again → Tool Call → ...

Each step can affect subsequent decisions. Traditional interface testing only covers individual steps, not the overall decision chain.

You need new testing methods:

• Scenario testing: Simulate real-world use cases and observe the full behavior
• Adversarial testing: Give ambiguous or contradictory commands to test responses
• Long-term testing: Let the Agent run continuously to observe memory and strategy evolution

From Coverage to System Evolution Capability

The hardest part is verifying the Agent’s learning ability. The Dreaming mechanism enables self-optimization, but that also brings uncertainty:

• Could the Agent learn incorrect patterns?
• Are the optimized strategies genuinely better?
• How do we ensure the Agent doesn’t “go out of bounds”?

AI testing is shifting from “verifying outcomes” to “constraining behavior.” You must define the Agent’s behavioral boundaries, not predict each move.

Controversies and Challenges: Will Software Disappear?

The biggest controversy surrounding OpenClaw is: if the Agent can operate all software, does software itself still need to exist?

From GUI to API: Redefining Software Value

Traditional software value lies in interface and interaction. Users must learn how to use it, and software companies build moats through UX optimization.

But in the Agent era, users no longer directly operate software—they just tell the Agent what they want, and the Agent handles the rest.

That means:

• Metrics change: From “active users” to “API calls”
• Business models change: From “selling interfaces” to “selling infrastructure”
• Competition focus changes: From “UI/UX” to “service capability”

The Battle for Entry Points: Whoever Controls the Agent Controls the Traffic

Agents are becoming the new entry points. Users no longer open dozens of apps—they simply @ different Agents in one chat interface.

This redistributes “entry value”:

• Old model: Users actively search for apps → browse functions → complete tasks
• Agent model: Users describe needs → Agent invokes services → returns results

WeChat, QQ, and WeCom are all racing to claim that entry. Whoever creates the best Agent experience will control user attention.

Data and Context: The New Moat

An Agent’s capability depends on two factors:

1. Data breadth: How many services and data sources it can access
2. Context depth: How well it understands user needs

Both require long-term accumulation. As an open-source project, OpenClaw lacks advantages in data and context. The real battle will be fought among major platforms:

• Tencent: social data and enterprise services
• Alibaba: e-commerce data and payment capability
• ByteDance: content data and recommendation algorithms

Whichever platform best integrates its data with Agent capabilities will form the strongest moat.

What’s Next: Who Will Be Eliminated?

OpenClaw’s latest update reveals one trend: AI Agents are shifting from “tools” to “platforms.”

Those likely to be eliminated:

• AI assistants that only provide simple conversations
• Closed software without API capabilities
• Traditional services incompatible with Agents

Those likely to prevail:

• Infrastructure offering rich APIs
• Platforms with unique data and services
• Developers who can adapt quickly to the Agent ecosystem

In essence, OpenClaw is redefining what “software” means—from user interfaces to service endpoints, from single interactions to continuous workflows, from passive responses to active execution.

This isn’t just a feature upgrade—it’s a paradigm shift, like the move from PC to mobile. Many rules are being rewritten.

For developers, now is the best time. The ecosystem is still young, standards are still forming, and whoever seizes this opportunity will gain a head start in the next era.

---

References

• OpenClaw quietly releases major update: the lobster grows eyes and hands – QbitAI – Detailed explanation of OpenClaw’s visual-interaction technology
• OpenClaw is evolving from “personal AI assistant” to “Agent Operating System” – Zhihu – Analysis of OpenClaw’s architecture and version evolution
• After OpenClaw’s major update, Agents begin evolving into “system-level intelligences” – InfoQ – In-depth look at the Dreaming mechanism and memory system design