SenseTime Open-Source SenseNova-U1: A New Multimodal Paradigm Without Codecs

Cutting Out All Intermediate Encoders: SenseTime Rewrites the Multimodal Game Rules

On April 27, SenseTime officially released and open-sourced its new-generation native multimodal large model, SenseNova-U1. This is not just another model stacking parameters on top of existing paradigms — its underlying NEO-unify architecture completely removes the long-standing visual encoder (VE) and VAE from multimodal models.

On the same day, Higon DCU announced that it had completed full adaptation to SenseNova-U1, making it one of the first domestic chip manufacturers to successfully run the model.

This is worth a closer look.

Why Remove Encoders/Decoders?

Let’s review the mainstream approach to current multimodal models.

Over the past few years — whether GPT-4V, Gemini, or open-source projects like LLaVA and Qwen-VL — multimodal models have largely followed a default paradigm:

• On the vision side, use a pre-trained visual encoder (e.g., CLIP ViT or SigLIP) to compress images into tokens
• Add a projection layer or adapter to map visual tokens into the language model’s semantic space
• Have the language model handle understanding and reasoning
• For image generation, connect a VAE or diffusion model for decoding

This works, but problems have become increasingly obvious:

1. The visual encoder is pre-trained and frozen, so its representational power caps the model’s visual understanding ceiling. If you use CLIP as the encoder, the model inherits CLIP’s biases and blind spots.
2. Understanding and generation are disconnected. Understanding goes through the encoder, generation through the decoder — the two paths lack deep interaction. "Understanding" an image and "drawing" one rely on entirely different representational systems.
3. This stitched architecture makes end-to-end training difficult. Encoders, projection layers, language models, and decoders all have different gradient flows and learning paces, resulting in high engineering complexity for joint optimization.

SenseTime’s conclusion: instead of patching within the current paradigm, start over from the architecture level.

What Exactly Does NEO-unify Do?

The core idea of NEO-unify can be summed up in one sentence: let a unified Transformer directly process raw signals of all modalities — no need for any external encoders/decoders.

Sounds simple, but implementing it requires solving several key issues.

Native Visual Primitives

Traditionally, images are split into patches, passed through a ViT, and converted into visual tokens. NEO-unify takes a different route — it innovates at the attention mechanism and positional encoding level, creating “native primitives” that allow the model to process visual information directly at the patch level, without a separate visual encoder preprocessing step.

Think of it as removing the translator: the traditional approach “translates” images into a format the language model understands before processing; NEO-unify teaches the model to “see” on its own, no translation needed.

Unified Semantic Space

Without encoders/decoders as intermediate bridges, how to align visual and language semantics?

NEO-unify redesigns semantic mapping: through a unified tokenization scheme and shared attention mechanism, visual tokens and language tokens interact in the same semantic space. The model no longer needs a two-step process — “first understand the image, then describe it in language” — instead, cross-modal fusion happens at every processing layer.

Unified Understanding and Generation

This is NEO-unify’s most ambitious part.

Most current models excel at either understanding (image captioning) or generating (text-to-image), but few do both well. This is because understanding and generation rely on different modules with different optimization goals.

NEO-unify unifies understanding and generation in the same model, with the same parameters. The model can both comprehend an image and answer questions about it, as well as generate images from text descriptions. These abilities share the same underlying representation — meaning visual "understanding" can directly guide "generation," and vice versa.

Improved Training Efficiency

A notable statistic: according to SenseTime's earlier disclosures, NEO architecture training requires only one-tenth the data of traditional approaches.

Not hard to see why. In traditional setups, the visual encoder needs massive image-text pairs for pretraining, the projection layer needs fine-tuning for alignment, the language model needs instruction data — each module has its own data requirements. NEO-unify trains end-to-end with a single dataset, naturally improving data utilization.

Performance: Open-Source SOTA

SenseTime claims SenseNova-U1 achieves SOTA in open-source models.

Specific benchmark data awaits community reproduction, but given the architectural soundness, the claim is plausible. NEO-unify tackles structural bottlenecks in multimodal models rather than simply piling on more data or parameters. Remove the architectural bottlenecks, and the model’s capability ceiling indeed rises.

Of course, “SOTA” in 2026 is somewhat diluted — new models claim it weekly. The real measure is performance in actual use. Community evaluations and feedback over the next few weeks will give a more objective answer.

Higon DCU Adaptation: A Signal for Domestic Compute Ecosystem

Higon DCU completing full SenseNova-U1 adaptation on Day 0 is notable.

For the past two years, domestic AI chips have faced an awkward gap: hardware closing in on global standards, but software ecosystems lagging. Many open-source models launch supporting only NVIDIA GPUs, with domestic chips needing weeks or months for compatibility.

Higon achieving Day 0 adaptation suggests:

1. SenseTime considered domestic chip compatibility during model development — a deliberate ecosystem strategy
2. Higon DCU’s software stack maturity is improving, at least with solid Transformer model support

For teams deploying multimodal models on domestic compute, SenseNova-U1 + Higon DCU offers an out-of-the-box option.

How Does It Compare to Competitors?

In today’s multimodal model landscape:

Closed-source: GPT-4o and Gemini 2.5 have native multimodality, but details are undisclosed and they cannot be deployed locally.

Open-source: mainstream solutions remain “visual encoder + language model.” Qwen-VL series, InternVL, LLaVA-OneVision follow this route — advantages are maturity and community support; disadvantages are that architecture ceilings depend on the visual encoder’s capability.

SenseNova-U1 takes a different path. Its strengths lie in unified architecture and efficient end-to-end training; risks are that brand-new architecture means less community experience, tools, and best practices.

Developer recommendations:

• For primarily visual understanding tasks (image QA, document parsing), mature existing solutions (Qwen-VL, InternVL) remain reliable
• If you need unified understanding + generation or care about training efficiency, SenseNova-U1 merits careful evaluation
• If you operate in a domestic compute environment, this may be among the most hassle-free choices

Open-Source Strategy Considerations

SenseTime’s decision to open-source SenseNova-U1 has clear business logic.

As a company focused on AI platforms and solutions, SenseTime needs a developer ecosystem. Open-sourcing a technically differentiated model attracts developers to build on SenseTime’s stack, promoting usage of its cloud services and toolchains.

From a technology dissemination perspective, architectural innovations like NEO-unify are hard to validate and refine without open source. Open access lets researchers worldwide experiment on the architecture, accelerating iteration.

SenseTime’s collaboration with Nanyang Technological University is also notable — academic participation ensures research rigor and publication transparency, key to establishing credibility.

Impact on the Multimodal Field

Broadly speaking, SenseNova-U1 represents a direction in multimodal model architecture evolution: from stitched to native.

This direction is not unique to SenseTime. Meta’s Chameleon and Google’s Gemini also move toward native multimodal. But in open source, SenseNova-U1 is among the earliest to achieve “encoder/decoder-free” end-to-end multimodality.

If NEO-unify proves effective through community validation, it could influence design choices in upcoming open-source multimodal models. At minimum, the question “do we still need a separate visual encoder” now has a real-world counterexample.

Of course, bridging the gap from paper to production is long. Robustness, inference efficiency, and performance in edge cases all require time to verify — but SenseTime has provided a convincing starting point.

Summary

The release of SenseNova-U1’s key significance is not “another open-source multimodal model” but the NEO-unify architecture’s challenge to existing paradigms. Removing encoders/decoders, unifying understanding and generation end-to-end, reducing training data needs to one-tenth — if these hold up in community reproduction, it’s meaningful architectural innovation.

Higon DCU’s Day 0 adaptation is another positive sign, reflecting accelerated ecosystem synergy for domestic AI chips.

Now we wait for community evaluations.

---

References

• Farewell to VE and VAE! SenseTime Hard-Core Reconstruction of Multimodal: Cutting All Intermediate Encoders — Zhihu column, detailed analysis of NEO-unify architecture design concepts and technical details