Tencent Open-Sources OpenSearch-VL: Revealing the Training Formula for Multimodal Search Agents

On May 6, Tencent Hunyuan uploaded a training scheme called OpenSearch-VL to arXiv, in collaboration with the University of California, Los Angeles, and the Chinese University of Hong Kong. This isn’t just a set of model weights—it’s a complete open-source recipe covering everything from data construction and tool environment to reinforcement learning algorithms. The goal is straightforward: to tackle the toughest bottleneck in multimodal search agent training—high-quality data.

A day later, the related code and datasets began to roll out. For developers who have been tracking GPT-4V’s search capabilities or eyeing Gemini Deep Research, this is probably the most worthwhile technical report to read in the past six months.

Why Now, Why This Project

First, let’s clarify what a multimodal search agent is. Simply put, you give it a picture and a question—it must be able to interpret the image, perform searches, call OCR or cropping tools to process visual content, and hop across multiple sources to verify evidence before delivering an answer. In product terms, this corresponds to offerings like Perplexity’s visual search, ChatGPT’s “image-based questioning + internet connectivity,” or Google’s Deep Research.

The problem is that all capable systems in this field are closed source. How commercial companies obtain their data, what tool-calling trajectories look like, how reinforcement learning reward functions are designed—these are all black boxes. Academic teams wanting to reproduce results hit a dead end without data. For systematic research in this area, there’s not even a single open-source baseline to reference.

That’s what motivated the OpenSearch-VL project. The research team frankly states in the paper: the biggest bottleneck hindering progress in advanced multimodal search agents is the lack of high-quality training data. They aim to open up what commercial companies have been keeping under wraps.

Core Innovation: Wikipedia Isn’t Just for Searching—it’s for “Creating Questions”

The most ingenious part of the whole design lies in the data pipeline.

Traditionally, to train a search agent, you start with existing visual question-answering datasets such as OK-VQA and teach the model to use search tools. But here’s a hidden trap: many questions can be solved with just one retrieval step, which risks teaching the agent lazy shortcuts—spot the most obvious element in the image, perform one search, and stop. Such models falter when faced with questions requiring multi-hop reasoning.

OpenSearch-VL reverses this logic—intentionally constructing questions that require multi-hop reasoning to solve.

In practice, the team uses Wikipedia’s hyperlink graph to sample multi-hop entity paths. For example, starting from a photo of the Eiffel Tower, traverse links to the designer Gustave Eiffel, then to the Statue of Liberty (which he helped design), and finally to a specific riveting technique used in its structure. This path naturally forms a question requiring at least three search hops to answer correctly.

But the paths alone aren’t enough. The team introduces two mechanisms to close shortcuts:

• Intermediate entity obfuscation: Entities along the path aren’t directly named; they’re rewritten descriptively. “Designer of the Eiffel Tower” becomes “the French engineer known for a famous tower who also contributed to the design of an iconic American statue.”
• Anchor entity grounding to the image source: The starting entity appears only through an image, with its name omitted from the text. The agent must first identify the image content before initiating the search chain.

These two tactics effectively block single-step shortcuts. The final data pipeline yields the SearchVL-SFT-36k dataset, with each trajectory averaging 6.3 tool calls—a figure an order of magnitude higher than most visual QA benchmarks, demonstrating deeper tool interactions.

Tool Environment: Teach the Agent to “See Clearly” Before Searching

Another noteworthy design is the expanded tool environment.

Most existing search agents have just two tools—text search and image search. OpenSearch-VL greatly broadens this by adding OCR, image cropping, sharpening, super-resolution, and perspective correction to the visual toolkit.

The reasoning is intuitive: in real-world scenarios, user-provided images are often blurry, skewed, or contain critical text hidden in corners. If the agent dives straight into searching without cleaning up the image, it’s essentially wasting effort.

To reinforce this “think while processing” behavior, the team deliberately degrades 10% of the training images (blurring, downsampling, etc.) and pairs them with appropriate enhancement tools—effectively teaching the model: “Encounter a bad image? Don’t rush to search. Fix it first.”

This integrated “active perception + knowledge acquisition” approach appears for the first time in an open-source recipe of this kind. By contrast, many so-called multimodal agents in academia still treat tool use as little more than calling a simple image search API wrapper.

Training Method: SFT Foundation + RL Refinement

The training process combines classic two-stage methods but with noteworthy engineering details.

The first stage is supervised fine-tuning (SFT), leveraging the SearchVL-SFT-36k dataset. This teaches the model basic tool usage formats and multi-step reasoning rhythms.

The second stage applies reinforcement learning (RL). The reward function is straightforward—answer correctness + tool usage efficiency—but the team put effort into ensuring rollout stability. Multimodal long-trajectory RL training is resource-intensive: each rollout involves dozens of tool calls and tens of thousands of tokens. Training crashes are common in this domain.

The released representative model, OpenSearch-VL-30B-A3B, is built on Hunyuan’s MoE architecture with 3B active parameters. It’s usable for inference on consumer-grade hardware, though full-scale training still requires a significant compute cluster.

Results: Baseline 47.8 → 61.6

The numbers speak for themselves.

Across multiple multimodal search benchmarks such as VDR and MMSearch, OpenSearch-VL-30B-A3B raises the average score from 47.8 to 61.6—a nearly 14-point gain. That’s a substantial improvement compared to peer open-source models.

The ablation study is even more informative:

• Removing source anchoring: -8.2 points
• Removing entity obfuscation: -9.5 points
• Removing staged filtering: -11.5 points

Each core component contributes roughly 8–12 points, indicating the pipeline’s strength lies in integrated design rather than single clever tricks. This kind of structural improvement is far more credible than simple gains from scaling parameters or data volume.

Industry Context

OpenSearch-VL occupies an interesting spot in the recent half-year industry timeline.

On one hand, open progress in the Agent field is accelerating. From DeepSeek’s public release of R1’s RL training details early this year, to Qwen’s continued investment in tool use, and now Tencent Hunyuan’s full disclosure of multimodal search training recipes—Chinese teams are becoming increasingly proactive about making advanced capabilities reproducible. Academic partnerships are gradually filling gaps commercial players won’t disclose.

On the other hand, the business value of multimodal search is clear. Internal enterprise knowledge retrieval, photo-based shopping searches, and combined text+figure research queries are all directly monetizable. Once a usable open-source baseline emerges, we can expect to see a wave of fine-tuned versions in vertical niches.

Technically, my view is that OpenSearch-VL’s lasting contribution isn’t the 30B model itself—model weights iterate quickly and may be outdated in six months. The real value lies in the data construction methodology. Using Wikipedia’s hyperlink graph for multi-hop sampling combined with obfuscation to prevent shortcuts could well become a standard paradigm for future multi-hop reasoning agent training—beyond just search tasks.

A cautious note: the size of SearchVL-SFT-36k is modest by today’s standards—36k trajectories × 6.3 tool calls each gives only a few hundred million tokens total. That’s tight for cutting-edge model training. However, since the team’s focus was methodological demonstration rather than scaling, community follow-up to expand the dataset is entirely feasible.

How Developers Can Use It

For developers looking to build on this framework, several practical paths:

• Use SearchVL-SFT-36k for fine-tuning directly: If you already have a multimodal backbone (e.g., Qwen2.5-VL or InternVL), applying this dataset for SFT should yield noticeable search capability enhancements.
• Reuse the data pipeline to generate domain-specific data: Replace Wikipedia hyperlinks with your own knowledge graph or product document links to create industry-specific training datasets.
• Integrate the tool environment into your agent framework: The OCR, super-resolution, and perspective correction toolkit can be incorporated into existing LangChain or LlamaIndex projects to enrich visual input preprocessing.

Incidentally, the Tencent Hunyuan series models are available on OpenAI Hub and support OpenAI-compatible APIs. Teams wanting to validate prototypes with the closed-source version initially can skip some environment setup hassle.

Final Thoughts

OpenSearch-VL isn’t one of those dazzling new products—it’s more like pulling back the curtain on an area long kept under wraps. Two years of commercial engineering distilled into one open-source report that explains the methodology thoroughly.

The next six months are likely to be lively in this field—reproductions, expansions, and domain adaptations from multiple directions. For teams still debating whether to enter the multimodal agent track, the current starting point is already much higher than it was half a year ago.

References

• Tencent Open-Sources OpenSearch-VL, Overcoming the Bottleneck of Multimodal Search AI Agent Training – IT Home: IT Home’s May 7 debut report on OpenSearch-VL, detailing technical content and benchmark results
• Survey of ~500 LLM Papers – Zhihu Column: Comprehensive review of LLM research covering search agents and RL training directions
• GitHub Trending Project Crawler: Reference repository tracking open-source AI and multimodal model projects