New Open-Source Multimodal RAG Project: The Combined Power of Zilliz + Qwen-VL

Text-based RAG Has Been Overdone, but Multimodal RAG Has Always Been Half-baked

Just after the May Day holiday, an open-source project called Multimodal-RAG appeared on linux.do, created by liangdabiao — known for developing the e-commerce bestseller replication pipeline and Perler Beads AI, one of the more prolific contributors in the community. The technology stack of this new project isn’t complicated: multimodal embedding + Zilliz vector database + Qwen visual understanding, but together they tackle a long-standing problem that's plagued the RAG field for over two years — how to make retrieval-augmented generation truly “see” images.

In the past, what most people called “multimodal RAG” were actually pseudo-multimodal: when a PDF came in, they ran OCR to extract text, used a caption model to generate text descriptions for charts, then dumped everything into a text index. This process is fine for pure text documents, but it crumbles when faced with exploded product diagrams, bar charts in financial reports, or medical imaging — because most of the information in the images gets lost in the text conversion step.

What This Project Actually Does

Looking through the repository, its core idea is to put images and text into the same vector space for retrieval, rather than handling them separately. The author used Cohere’s multimodal embedding model for vectorization — the key is that the image and text embeddings are inherently aligned. You can search for “slimming red dress” and directly hit the corresponding product image in an image database, and the reverse works as well.

Choosing Zilliz for the vector database is no surprise. Over the past couple of years, Milvus/Zilliz has practically become the default choice for multimodal retrieval scenarios — for good reason: it supports hybrid retrieval combining sparse vectors, dense vectors, and scalar filtering, and its HNSW index can deliver millisecond-level latency even with tens of millions of image-text pairs. In comparison, Chroma is lightweight but struggles in production — index partitioning and streaming writes become bottlenecks; Qdrant is pretty good on performance, but its community and ecosystem lag behind Zilliz’s in China.

On the generation side, the author uses Qwen’s visual understanding model (Qwen-VL series). This is the most crucial design in the whole pipeline: the retrieved item isn’t a text snippet but the original image plus contextual text, which gets fed directly into the VLM to “talk about the picture.” That means the system doesn’t depend on pre-generated captions and is much better suited to “a picture is worth a thousand words” scenarios like charts, UI screenshots, and medical images.

Smart Design Choices in the Technical Details

Carefully reading through the code and documentation reveals several clever decisions worth highlighting:

First, hybrid retrieval strategy. The system doesn’t rely solely on vector retrieval; instead, it fuses multimodal vector similarity with BM25 text-based recall for ranking. By 2025, this design has become the hallmark of advanced RAG implementations — pure vector search tends to miss proper nouns and exact matches, while BM25 misses semantic relations. Combining both yields much stronger robustness.

Second, preserving layout information when chunking documents. The PDF parser doesn’t just split by page; it treats images, tables, and text paragraphs as separate chunks while keeping their spatial relationships within the document. So when retrieval hits an image, it can also bring along related explanatory text. This is far more sophisticated than the brute-force chunk_size=512, overlap=50 approach common in many RAG frameworks.

Third, Qwen-VL serves as a secondary filter. The Top-K results from retrieval are re-evaluated by the VLM to check relevance — essentially replacing a cross-encoder reranker with a visual language model. This adds some latency, but for image-heavy retrieval tasks, accuracy improves significantly because the VLM can understand details like “the bar chart in this image represents Q3, not Q2,” which text embeddings simply cannot capture.

How It Compares to Existing Solutions

Current multimodal RAG solutions generally fall into three categories:

• LlamaIndex/LangChain multimodal modules: well-integrated, but weak out-of-the-box performance — require heavy parameter tuning to get good results
• ColPali-style end-to-end systems: directly encode PDF pages as images to skip parsing, but demand high compute and have expensive inference costs
• Cloud-hosted services (like those offered by certain domestic model providers): turnkey and easy, but with little privacy or customization flexibility

This open-source project takes a more pragmatic position — you can self-host, tune parameters, or swap models freely. The embedding model can be changed from Cohere to JinaCLIP or BGE-M3; the VLM can be swapped from Qwen-VL to any other OpenAI-compatible visual model; and theoretically, the vector database can switch from Zilliz to a self-hosted Milvus instance.

Incidentally, since the system uses OpenAI-compatible APIs throughout, it integrates smoothly with aggregator platforms like OpenAI Hub — one API key can cover Qwen-VL, Cohere embeddings, and even replace generation with Claude 3.5 Sonnet later, saving the hassle of applying for multiple quotas. For developers building quick prototypes, this route is far more convenient than registering each API individually.

To Be Honest, There Are Some Limitations

The project isn’t quite mature yet, so several pitfalls need mentioning:

Limited evaluation data. The provided demos focus on e-commerce and product manual scenarios. There’s no benchmark for specialized fields like medical imaging or engineering drawings. Multimodal RAG performance is highly domain-sensitive — don’t plug it directly into mission-critical workflows.

Not cheap. Multimodal embeddings per thousand images and VLM inference token costs are about an order of magnitude higher than text-only RAG. For commercial deployment, you’ll need a clear cap on per-query costs.

Lacks production-grade infrastructure. Incremental indexing, version management, and caching layers — all essential for production — are missing for now. Building a commercial product on top of this would require substantial engineering work.

Signals Worth Watching

Stepping back, the timing of this project’s appearance is revealing — by 2026, RAG isn’t about “whether it works” anymore, but about how to integrate multimodal, agentic, and long-context paradigms. The old one-way retrieve-read workflow of naïve RAG is obsolete for complex tasks; agentic RAG, multimodal RAG, and iterative retrieval are becoming the new baseline.

For indie developers and small teams, open-source projects like this — prepackaged with the core capabilities — are ideal learning resources. The code base is small, the core logic fits in just a couple hundred lines, yet it encapsulates the best practices the industry has refined over the past two years. For learning multimodal RAG, it’s far more efficient than digging through papers.

The project is still evolving, and the author mentioned plans to add support for more embedding models and agentic retrieval modules. Definitely worth keeping an eye on.

References

• GitHub - liangdabiao/Multimodal-RAG — Main repository with full source code and deployment instructions
• linux.do open-source announcement — Author’s original community post and discussion
• GitHub - liangdabiao/perler-beads-ai — Author’s earlier open-source Perler Beads AI project, useful for understanding their engineering style
• How to Build a Multimodal RAG System from Scratch - Zhihu — A reference implementation for multimodal retrieval using JinaCLIP + Milvus