SILX AI launches Quasar-Preview: 18B MoE supports 5 million token context

Today, SILX AI has released the first public preview of its long-anticipated Quasar series base model—Quasar-Preview—directly on Hugging Face under the MIT license. Unlike the recent months’ releases that often claim to “dominate the leaderboards,” SILX is upfront this time: this thing isn’t here to compete on benchmarks, it’s here to validate the architecture.

This kind of honesty is actually rare. Over the past year, from DeepSeek-V4-Pro to Baidu’s Wenxin 5.1 to Zhipu’s GLM-5.1, top domestic teams’ launch events have mostly focused on scores, activation parameter efficiency, and pricing. SILX has chosen a relatively academic approach: open-sourcing a mixed architecture that hasn’t finished training yet, so researchers can take it apart themselves.

18B total parameters, 2B activation — but that’s not the main point

Let’s go over the specs first:

• Total parameters: ~18B, Mixture-of-Experts (MoE) architecture
• Activation parameters: ~2B level
• Context window: Experimental 5M tokens
• Training tokens: Currently 1T–1.5T total, with less than 1B on the long-context extension path
• License: MIT, fully open source

The 18B/2B combo isn’t unusual in 2026. Qwen3, the DeepSeek series, and even the recently open-sourced Keye-VL-2.0-30B-A3B by Kuaishou have all played the “small activation powering a big model” trick. What’s really worth highlighting are the next set of architecture terms: Loop Transformer + Quasar hybrid attention + Quasar/Raven/GLA hybrid layers + sparse MoE routing.

It may look flashy, but it has one objective—making long context no longer a “just stuff it in” pseudo-capability.

5M context — how it’s achieved and what it means

The 5M token figure needs unpacking. It’s not just a matter of tweaking RoPE frequencies and then declaring “support.” SILX uses a Safe NoPE / DrOPE-style phased extension method—a path increasingly validated in recent long-context research. The core idea is to gradually “degrade” or “drop” position encoding in stages so the model doesn’t collapse on positional priors in ultra-long sequences.

Industry comparisons:

• In March 2024, when Kimi achieved lossless context up to 2 million characters, it was already viewed as a “scale breakthrough.”
• Gemini series has long kept context between 1M-2M.
• New-gen multimodal models like Keye-VL-2.0 from Kuaishou consider 256K “ultra-long.”

Quasar-Preview’s claim of 5M sounds bold, but SILX clearly states: the long-context extension path has so far been fed less than 1B tokens. Meaning, this 5M is more like an architectural run-capacity limit than an engineering-level “usable quality.”

That’s why the team keeps emphasizing: Preview does not represent Quasar’s final quality.

Loop Transformer and hybrid attention — betting on a “memory system”

The most intriguing architectural choice is the Loop Transformer. It’s a relatively niche but distinctive direction—using looped unfolding to have the same parameter set repeatedly process the same context segment during inference, thereby extracting depth without increasing parameter count.

With Quasar’s own hybrid attention, plus GLA (Gated Linear Attention)—a linear attention variant already validated by multiple teams—the tradeoff logic is fairly clear:

1. Sparse MoE widens capacity—18B total parameters provide breadth of knowledge
2. GLA and other linear attention extend context—making 5M workable in VRAM
3. Loop Transformer deepens inference—enhancing reasoning without stacking parameters
4. Quasar/Raven hybrid layers retain full-attention accuracy—avoiding capability loss from full linearization

In short, SILX wants a foundation tailored for memory-based systems. This aligns with actual needs in agents, long-range tasks, and personal assistants—you don’t need the model to re-understand 5 million tokens every time, you need it to “store” a long state and access it repeatedly in loops.

Only 1.5T training tokens — what that implies

Current flagship open-source models often start at 10T-20T tokens, so Quasar-Preview’s 1T-1.5T clearly places it in an “early” stage. SILX’s listed roadmap confirms this:

• Iterative subnet training and knowledge distillation
• Longer training cycles and stronger post-training
• Further long-context extension training and architecture updates

From an engineering pace perspective, this Preview is more like an “executable version of an architecture whitepaper.” Researchers can use it for ablation studies, test attention mechanism failure modes, validate Loop Transformer stability under MoE routing—all things difficult to run in papers, but worth verifying in industrial training.

Viewed alongside the recent “small activation MoE” wave

So far in 2026, open-source’s main trends are twofold: activation parameters getting smaller (DeepSeek-V4-Pro slashed API prices to 1/4 of their original rates, thanks to activation efficiency), and context windows getting longer (Kuaishou’s Keye with 256K, Zhipu’s GLM-5.1 enhancing long-text scenarios).

Quasar-Preview bets on both, but more aggressively:

| Dimension | Mainstream approach | Quasar-Preview | |-----------|---------------------|----------------| | Activation parameters | 3B-7B | 2B | | Context | 128K-1M | 5M (experimental) | | Attention | Full + sparse | Quasar/Raven/GLA hybrid | | Depth strategy | Layer stacking | Loop Transformer looping | | Training maturity | Fully pre-trained | 1T-1.5T early stage |

This comparison explains why SILX repeatedly lowers expectations—betting on four unvalidated directions means this model likely won’t outperform mature models of the same scale on zero-shot benchmarks. But as an open-source architectural prototype for community dissection, it offers far more informational value than a conventional model scoring two points higher.

Practical advice for developers

If you plan to pull Quasar-Preview today, set your expectations right:

1. Don’t compare its dialogue quality to GPT or Claude—training volume’s the reason
2. You can try the 5M context, but don’t expect stable retrieval beyond 4M—long-context path has been fed under 1B tokens
3. Architecture validation value > direct deployment value—best suited as a research or fine-tuning starting point, not immediate production
4. Track iteration pace in later versions—MIT license plus full open-source means likely releases of v0.2, v0.3 in the future

For domestic developers, you can directly pull the weights from Hugging Face for local inference; if you normally use aggregated APIs (like OpenAI Hub that lets you call GPT, Claude, Gemini, DeepSeek with one key), early-stage open-source models like Quasar likely won’t show up in hosted form soon—wait for training volume to reach usable territory.

In closing

By 2026, large model launch events increasingly resemble scripted productions—release, benchmark, PR, price cut, API, open-source. Quasar-Preview stands out in this context: no polished benchmarks, no dazzling demos, not even fully trained. SILX chose to throw out a half-finished product, betting on community interest in the architecture itself.

This gamble might not win, but at least it’s honest. For a model series aiming to be “memory-native,” publicly sharing the experimental path to 5M context lets researchers help validate directions too costly for a single team to complete—this route is arguably smarter than quietly training to 10T tokens and then releasing “Quasar-1.0.”

Whether this architecture works, whether Loop Transformer with MoE has potential, whether 5M context can actually land—answers will come when SILX pushes training volume past 10T. Today’s Preview is just laying the cards on the table for everyone to see.

References

• SILX AI officially releases Quasar-Preview: early preview of 18B MoE architecture with 5M context length - linux.do — Original Chinese community discussion with full specs
• silx-ai/Quasar-Preview · Hugging Face — Model weights and official documentation, MIT open source
• AI Morning 2026 Long-Context Model Overview - Zhihu — Recent horizontal comparison of MoE and long-context models