Tsinghua’s Open-Source Spatial Model Beats Gemini: Learning While Watching 120-Minute Long Videos

Putting "Continuous Learning" Back into Spatial Intelligence — Tsinghua Makes Gemini Look Less Impressive This Time

In late June, the Tsinghua team announced a project accepted by ECCV 2026: an open-source spatial intelligence model focused on "continuous learning in a changing world." The most direct selling point is — give it a 120-minute continuous video, and it can encode spatial information into its state while watching, rather than chopping the video into dozens of frames and forcing them through the context window like mainstream VLMs do.

On multiple spatial reasoning benchmarks, this model outperformed the latest closed-source Gemini series. For an open-source model led by an academic lab with significantly fewer parameters, this result is uncommon.

What Problem Does It Actually Solve

To understand the significance, we first need to see how current spatial intelligence models process video.

Mainstream approaches are roughly twofold:

• Sparse sampling: Extract 8, 16, at most 32 frames from a video and feed them to a VLM. Gemini, GPT-4o, Qwen-VL all use this route. The obvious issue — if you sample 16 frames to understand a 2-hour room tour video, you lose far more information than you keep.
• Injecting 3D priors: Use geometric information reconstructed from point clouds, depth cameras, or SfM as an additional modality for the model. This route works well but is expensive, relying on extra sensors and reconstruction pipelines, which isn’t cleanly end-to-end.

Tsinghua’s ECCV paper takes a third route: treat the video as a time stream and let the model maintain an “online, incremental” spatial memory. Conceptually, this is closer to the human brain — when you walk into a new room, you don’t reconstruct a 3D point cloud in your head every second; you gradually form a hierarchical spatial representation like “living room to the left, kitchen to the right, cup on the dining table,” and keep updating it as observations continue.

As the paper puts it, true spatial intelligence is not about “watching once and answering,” but about continuous learning in a changing world. This isn’t a new idea — LeCun’s JEPA and “world model” camp keep chanting this — but in the specific scenario of long video spatial reasoning, implementing it to the point of beating Gemini is indeed a first.

How Does It “Watch and Remember” a 120-Minute Video

The key is in how state is maintained. Attention mechanisms are naturally bad at this — as context gets longer, KV cache blows up, and compute cost rises quadratically. So approaches in this line usually use some kind of “compress-and-retain” mechanism, giving the model a fixed-size working memory.

From the publicly available information, Tsinghua’s approach breaks down into several layers:

1. Streaming encoder: Video frames enter in time order, each being encoded into an continuously evolving spatial state rather than tokenized independently waiting for attention.
2. Continuous learning loss: During training, the model does self-supervision over long videos — at time t, it must predict not only “what’s in this frame right now,” but also, when asked at t+k, recall spatial details from t accurately. This explicitly brings the traditional “forgetting problem” into the loss.
3. Spatial anchor mechanism: Significant geometric/semantic elements are “anchored,” preventing key spatial structures from being diluted over time.

The most direct benefit of this is GPU memory usage during inference stays roughly constant regardless of video length. This is entirely different from forcing a 120-minute video into a 1M context window — Gemini’s approach accumulates tokens as the video progresses, ultimately requiring truncation or becoming unusably slow.

Compared to Gemini — What’s Better

Spatial reasoning has a few tough benchmarks now: VSI-Bench (video spatial intelligence), EmbSpatial-Bench (embodied spatial), MMSI-Bench (multi-image spatial), plus some dedicated long video QA.

Tsinghua’s model shows the strongest edge in long-video subsets. Understandably — benchmark questions like “Where was the red cup you saw 30 minutes ago?” are almost a death sentence to sparse sampling models. Even Gemini with a 1M context must downsample a continuous video stream, and if the key frame isn’t sampled, the answer is a guess.

Tsinghua’s streaming architecture gradually cements temporal details into its state, avoiding the binary “sampled/not sampled” problem.

Some key comparison dimensions:

| Dimension | Mainstream VLMs (Gemini, etc.) | Tsinghua Spatial Model | |------|---------------------|------------| | Video processing method | Sparse sampling + long context | Streaming + continuous state | | Length limit | Context window bound | Theoretically unlimited | | Memory growth | Approx. linear/quadratic | Approx. constant | | Long-range spatial consistency | Relies on sampling luck | Guaranteed by training objective | | Open-source? | ❌ | ✅ |

This architectural difference means it’s not just “higher scores,” but able to do things Gemini simply can’t — e.g., after a robot roams a room for half an hour, being asked “Is the cup in the kitchen still on the table?” is a natural query for a streaming model but basically guesswork for a sampling model.

Why This Matters More than “Leaderboard Climbing”

Stepping back, in the first half of 2026 the AI field’s excitement for “world model” and “embodied intelligence” remains high. The ZhiYuan Conference just finished, with the World Model Forum packed with players from academia to industry — Skywork, Extreme Vision, Independent Variable, Xingyuan Intelligence... all trying to answer the same question: How can a model truly understand the physical world.

The mainstream routes split into two camps:

• Generation camp: Learn physics implicitly with video generation models. Sora, Veo, Hunyuan World follow this line.
• Understanding camp: Make VLMs stronger in 3D/spatio-temporal dimensions. Spatial-MLLM, SenseNova-SI, GEM (previously co-developed by Tsinghua and Tencent) belong here.

Tsinghua’s ECCV work can be seen as a new branch within the “understanding” camp — making “continuous learning” one of the core definitions of spatial intelligence. This is a sharp stance: it implies that no matter how beautiful your video generation or how precise your 3D reconstruction, if the model can’t continuously maintain state along a time stream, it doesn’t truly possess spatial intelligence.

This is crucial for the next year of robotics and embodied intelligence. Think of what UR5, Unitree, Independent Variable, etc. are doing — robots working continuously in a household for hours or even days, perceiving environmental changes every second. Using a sparse sampling VLM as the brain won’t cut it. The streaming continuous learning paradigm is theoretically the correct foundation for VLA (Vision-Language-Action) models.

The Chain Reaction from Open-Sourcing

With both model code and weights open-sourced, the impact goes far beyond benchmark scores.

Some expected directions:

• Rapid integration into embodied intelligence stacks: Independent Variable, Galaxy General, Xingyuan Intelligence, etc. will likely have someone hook this streaming encoder into their VLAs within weeks to run experiments.
• Pressure on closed-source vendors: Gemini has long been the closed-source benchmark for video stream processing. Being surpassed by an open-source model in long video spatial reasoning puts product positioning pressure on Google.
• Promoting new benchmarks: Current benchmarks are weak in assessing “continuous learning.” Following this work, new evaluations around long-term temporal consistency and cross-scene memory retention will likely emerge.

For developers, the more practical question is usage. The open-source repo includes full training and inference code, with weights directly on Hugging Face. Local inference has modest hardware requirements — one benefit of streaming architecture is a single 24GB GPU can handle videos of any length, which is a completely different scale compared to approaches requiring an 80GB H100 just to fit long contexts.

Unresolved Issues

Some cold water to pour as well. This architecture has a few obvious open questions:

1. Training cost: Streaming continuous learning requires long videos with fine-grained temporal annotations, which are currently limited. The paper’s training set size is still smaller than closed-source giants.
2. Error accumulation: Will state drift during long-term maintenance? Public demos look stable at 120 minutes, but what about 10 hours, 100 hours?
3. Relationship to generative world models: Understanding improves, but the model itself can’t “imagine” future scene changes, which leaves a gap for planning tasks.

The team frankly noted these in the paper as next-step work.

In One Sentence

The greatest value of Tsinghua’s work is not in surpassing Gemini, but in putting “continuous learning” back at the center of spatial intelligence discussions. Over the past two years, everyone focused on longer context windows and stronger 3D priors, but a model that forgets cannot truly understand the world. In this sense, this ECCV 2026 paper marks a noteworthy turning point.

The open-source repo is already available — for teams working in embodied intelligence, VLA, or long video understanding, it’s worth cloning and running — it will likely change your default assumptions about “how video should enter a model.”

References

• Tsinghua Spatial-MLLM: Crushing GPT-4o/Gemini with Only 16 Frames of Video (Zhihu column) — contextual background on Tsinghua’s prior work in spatial multimodal models