Skills Now Come with Illustrated Manuals: openJiuwen Launches the Multimodal Skill Paradigm Skill-Omni

Yesterday, openJiuwen dropped Skill-Omni. In one sentence: starting today, an Agent’s Skill library is no longer just a text manual, but a visual-textual experience base with illustrations and reference images.

That may sound abstract, but for people building Agents, it’s a signal. Over the past year, the entire field has been racing toward the same goal: “making Agents get smarter the more they are used.” Anthropic’s Skills, NVIDIA’s recently open-sourced robotics skill library, and the self-evolving RL infrastructure behind AReaL 2.0 are all pushing in the same direction: turning one-off task execution into reusable long-term experience. openJiuwen’s differentiation this time is straightforward — while others are still writing experience in markdown, they’ve already started attaching images to it.

Why Pure Text Skills Aren’t Enough

Anyone who has worked on browser Agents or GUI Agents has probably hit this problem before: you ask the model to summarize an operational workflow, and it writes a perfectly logical description — “click the settings button in the top right corner, enter account management, and locate the API Keys option.” Sounds flawless. But during actual execution, the Agent can’t find the button because the page layout changed, or because “top right corner” isn’t even the same location under different resolutions.

Video-related tasks are even more troublesome. If you want an Agent to learn a tricky editing workflow in video software or reproduce a game strategy sequence, pure text descriptions simply cannot convey the kind of spatial relationships humans understand at a glance. Humans learn new things half through images and half through text — why should Agents only read words?

Skill-Omni’s approach is to restore that missing visual layer. According to openJiuwen, it can extract and structure visual knowledge from web pages and videos, then store it alongside textual descriptions as part of the Skill library. The next time the Agent encounters a similar task, it won’t just have procedural instructions — it will also have reference images for alignment.

From “Readable” to “Visible”

Their slogan is actually pretty accurate: from “readable” to “visible.”

Breaking it down, Skill-Omni operates across several layers:

• Visual knowledge extraction: identifying key UI elements, operational paths, and state transitions from webpage screenshots and video frames
• Experience structuring: organizing aligned image-text information into Skill entries, including prerequisites, execution steps, reference images, and exception handling
• Multimodal retrieval: when encountering a new task, the Agent can match historical experience based on visual similarity rather than just textual similarity
• Continuous evolution: after each task execution, new screenshots and new workflows are fed back into the Skill library

That last point is especially important. openJiuwen has consistently emphasized “self-evolution” inside AgentOS — both tools and Skills should iterate in real time. Skill-Omni fills in the multimodal layer. Previously, self-evolution mainly occurred at the text level; now visual experience enters the loop as well.

Comparing It Horizontally with Competitors

Putting together the major Skill-related projects from the past six months:

| Project | Positioning | Modality | Evolution Mechanism | |------|------|------|----------| | Anthropic Skill | Reusable capability units for Claude | Primarily text | Static registration | | NVIDIA Robotics Skill Library | Action primitives for embodied scenarios | Actions + vision | Requires simulation training | | AReaL 2.0 | RL foundation for self-evolving Agents | Text | Online RL | | Skill-Omni | Visual-textual experience base for general-purpose Agents | Multimodal image + text | Automatically accumulated during task execution |

Skill-Omni occupies a fairly unique position. It doesn’t focus on low-level training frameworks, nor is it limited to embodied scenarios. Instead, it targets the question of how general-purpose Agents can accumulate visual experience during everyday task execution. That space was effectively empty before — Anthropic’s Skills are too text-centric, while NVIDIA’s stack is too embodied-focused.

For teams building browser Agents, office Agents, or GUI automation systems, Skill-Omni’s paradigm may be a much better fit than the others.

Physical Skills Are the Next Step

openJiuwen also revealed its next direction: Physical Skills for Physical AI scenarios.

This direction is not hard to predict. Embodied intelligence this year has broadly been moving toward “skill libraries.” When NVIDIA open-sourced its robotics skill library, Jim Fan’s statement that “the paradigm has changed” referred exactly to this shift — instead of training one giant end-to-end model to solve every task, reusable actions, perception modules, and planning capabilities are encapsulated as Skills that Agents can compose on demand. Fei-Fei Li’s team recently made a similar point in their Real2Sim work: when real-world data is insufficient, reverse-generate real scenarios into simulation. At its core, it’s still about accumulating experience.

Skill-Omni starting with visual-textual data and evolving toward Physical Skills is a natural progression. The real challenge is this: visual-textual experience libraries can still be directly understood by VLMs, but there is no standard answer yet for how physical-action Skill libraries should be stored, retrieved, or transferred across different robotic embodiments.

What This Means for Developers

First, the downside: Skill-Omni currently looks more like a capability module within openJiuwen’s own AgentOS ecosystem. There’s no standalone SDK or out-of-the-box integration method yet. Anyone wanting to use it directly will likely have to wait.

Now the upside: the paradigm itself is worth borrowing. If you’re building your own Agent system — especially for web, video, or GUI-heavy scenarios — you can adopt Skill-Omni’s organizational approach and evolve your experience base from pure markdown into a multimodal image-text structure. The technical stack is not especially complicated: an MLLM capable of visual understanding, a multimodal vector database with embeddings, and a pipeline that extracts image-text pairs from task traces are basically enough to get started.

A few ideas that can be implemented directly:

• Add screenshot fields to Skill entries: store screenshots of key successful execution steps, then use them as few-shot examples in prompts for similar future tasks
• Multimodal retrieval: don’t rely solely on text embeddings for historical Skill retrieval — incorporate visual similarity as well, especially for UI-heavy tasks
• Store failed experiences too: a Skill library shouldn’t only contain “how to do it right”; it should also include negative examples of “what these pitfalls look like,” where images are even more intuitive

One Takeaway

Viewed in isolation, Skill-Omni is not revolutionary. But it is moving in the right direction. Over the past two years, model capabilities have improved rapidly, while the Agent-side bottleneck has become increasingly obvious: experience accumulation. Models repeatedly start from zero in understanding tasks, then discard everything afterward. That approach is clearly unsustainable. The entire industry is now trying to solve this problem — Anthropic, NVIDIA, openJiuwen, and the AReaL team are all pursuing different paths toward the same goal.

Whoever turns the Skill layer into a de facto standard will control a highly strategic entry point in the Agent ecosystem. openJiuwen has planted the multimodal flag first. The next thing to watch is how quickly it can push real-world adoption and how open the ecosystem becomes. If it remains confined within its own AgentOS, the impact will be limited. If it evolves into an open Skill description standard, that becomes a very different story.

Worth watching.

References

• Zhihu: Enabling Multimodal Agents to Continuously Accumulate Experience and Become Smarter Through Skills — Technical discussion on experience accumulation for multimodal Agents