SILX AI Releases Quasar-Preview: 18B MoE Pushes Hard for 5 Million Context

SILX AI Brings 5M Context to an 18B MoE

In early June, a team called SILX AI quietly uploaded Quasar-Preview to Hugging Face— the first public checkpoint in the Quasar base model series. With 18B parameters, a MoE architecture, and only about 2B active parameters per token inference path, it sounds like a standard small MoE. But what really made this release stand out was the number on its model card: a 5,000,000 token context window.

Yes, 5 million tokens. Not 200K, not 1M—5M.

According to the model card, Quasar-Preview is not a final product, but rather an “architecture preview + base checkpoint.” SILX AI defines this release as the first step in the Quasar roadmap—proving that this combination of sparse MoE + hybrid recurrent/attention layers + experimental long context can run at real scale and be trainable. They plan to continue development via decentralized training, distillation, and architecture improvements on Bittensor SN24.

This “here’s the architecture for the community to see, it works but don’t expect leaderboard dominance” release posture is actually quite rare. At a time when GPT-5, Claude 4.5, and Gemini 3 are all competing toward multimodal reasoning, someone willing to go back and tackle the underlying architecture for long context is worth discussing.

18B with Only 2B Activated: MoE’s Math Logic

Let’s start with parameters. Quasar-Preview is set at ~18B total parameters, ~2B active parameters, with an activation ratio of about 1:9. Compared with MoEs already validated by the market like Mixtral 8×7B (47B total / 13B active) and DeepSeek-V3 (671B total / 37B active), Quasar-Preview is more sparse, with compute usage for single inference closer to that of a dense 2B model.

The ratio has practical considerations:

• Small activation path = cheaper inference. 2B active parameters means single-token FLOPs are close to Qwen2.5-1.5B, and a single 24GB VRAM card (with quantization) can run it.
• 18B total parameters = enough knowledge capacity. This size can handle base language capability and general knowledge, avoiding the common-sense pitfalls of 1-3B dense models.
• MoE routing is a natural match for long context. Expert specialization allows the model to only activate relevant experts over ultra-long sequences, avoiding compute blow-up from full parameter participation.

In other words, this is a MoE form optimized for long context, not just parameter stacking to chase benchmarks.

Hybrid Recurrent/Attention Layers: Transformer Is No Longer Alone

The really interesting part is at the architecture level. Quasar-Preview uses hybrid recurrent/attention layers—alternating between recurrent structures (similar to RWKV or Mamba’s state-space/linear attention) and traditional attention layers. The model card says this is built on the Loop architecture.

Background: pure Transformers have an unavoidable flaw for long context—KV cache grows linearly with sequence length, attention compute grows quadratically. When extending windows from 128K to 1M or 5M, just piling on VRAM and FlashAttention optimizations won’t save you.

Industry solutions over the past couple years have gone two directions:

1. Sparse / block attention: Gemini 1.5’s approach, essentially attention-based but relying on engineering optimization.
2. Linear attention / SSM: Mamba, RWKV, Jamba, replacing part of the attention with O(n) recurrent mechanisms.

Quasar-Preview takes the second route’s hybrid version—keeping attention layers’ precise recall capabilities, while using recurrent layers to reduce compute for long-range dependencies. This “Hybrid” idea is in the same family as AI21’s Jamba or Nvidia’s Hymba, but Quasar pushes both scale and context length to a more extreme place.

Safe NoPE / DrOPE: A New Solution for Long Context Positional Encoding

The model card also mentions a detail: Quasar-Preview adopts a Safe NoPE / DrOPE style phased long-context expansion strategy.

Positional encoding is a notorious challenge in long-context scenarios. RoPE decays during extrapolation, ALiBi loses precision scores, and interpolation methods like YaRN/NTK-aware are essentially after-the-fact fixes. NoPE (No Positional Encoding) has been shown in some studies to learn implicit positional info at large scales, but stability is an issue—hence “Safe NoPE.”

DrOPE leans toward randomly dropping part of the position signal during training so the model learns to stay consistent even when positional encoding is missing. SILX wraps this combo into phased expansion: train stably on short contexts first, then gradually push the window to 5M.

Frankly, the practical use of the 5M figure is questionable—no downstream benchmark yet truly verifies effective information retrieval at 5M tokens (needle-in-a-haystack struggles even at 1M). SILX admits this is an “experimental config for future memory-based systems,” more like laying groundwork for memory-augmented agents and long-range tasks.

How to Run It

Quasar-Preview is already open-sourced on Hugging Face (silx-ai/Quasar-Preview), supporting Transformers, vLLM, SGLang, and Docker Model Runner deployment. Since it uses custom architecture code, you need trust_remote_code=True when loading:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "silx-ai/Quasar-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Explain hybrid recurrent-attention layers in one paragraph."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Hardware-wise, the official recommendation is at least 24GB VRAM (bf16 + 2B active path). If you want to approach the 5M window, KV cache + recurrent layer states will consume massive VRAM, requiring 80GB-class cards or multi-card sharding. Tests show that 256K–512K context runs comfortably on A100 80GB, beyond which vLLM + paged attention is needed.

For production deployment, vLLM is recommended. Example:

vllm serve silx-ai/Quasar-Preview \
    --trust-remote-code \
    --dtype bfloat16 \
    --max-model-len 524288 \
    --gpu-memory-utilization 0.92

Note: don’t set max-model-len to 5M immediately—start with 256K–512K to avoid KV cache pre-allocation OOM.

Bittensor SN24 and Decentralized Training

Another less noticeable but notable detail: the Quasar series’ subsequent training will continue on Bittensor SN24 subnet.

For developers unfamiliar with Bittensor—it’s a decentralized ML network with token incentives, and SN24 (Subnet 24) focuses on long context and memory research tasks. Placing Quasar on SN24 means future iterations will leverage distributed compute contributors, not the traditional “self-built cluster behind closed doors” route.

This mode has mostly been seen in research projects like Hivemind or Petals, and hitting 18B MoE scale with usable checkpoints is still rare. If the Quasar series can complete a full train-distill-evaluate loop on SN24, it will be a significant case for decentralized training.

Of course, decentralized training has its costs: high communication overhead, node heterogeneity, complex gradient synchronization strategies. SILX’s MoE + recurrent architectures are likely chosen with MoE’s natural affinity in distributed contexts—experts can be split by node, and recurrent layers’ state passing produces less KV sync overhead than full attention.

Looking Sideways: Where Quasar Fits in the Ecosystem

Placed on the 2026 open-source model landscape, Quasar-Preview’s positioning is rather subtle:

• Competing in general capability: It can’t beat mainstream dense/MoE like Qwen3, DeepSeek-V4, Llama 4.
• Competing in ultra-small size: It’s not as good as finely tuned 1-4B models like Gemma 3 or Phi-5.
• Competing in long context: The 5M window is more hype than practical, but the 256K–1M range can really leverage the hybrid architecture.
• Competing in architectural novelty + openness: This is Quasar’s true selling point—a genuinely downloadable, finetunable, continuously trainable hybrid MoE checkpoint the community can use as a starting point for Mamba+MoE experiments.

So who gets value from this model?

1. Architecture researchers: If you want to verify hybrid recurrent/attention + MoE behavior at 10B+ scale, Quasar-Preview is one of the few directly downloadable samples.
2. Teams building long context applications: Even at just 256K window, hybrid architecture offers more inference speed and VRAM friendliness than pure Transformer.
3. Developers following decentralized training: Progress on Bittensor SN24 is worth watching.

For regular application developers, stick with GPT, Claude, Gemini, or DeepSeek via platforms like OpenAI Hub—Quasar-Preview, as a “preview base model,” is essentially research material, not a production tool.

Some Reservations

Finally, some less polite points.

I personally take a cautiously optimistic view toward the 5M context number. Hybrid architecture theoretically supports ultra-long sequences, but actual information utilization rate is key. We’ve seen too many models claiming “supports 1M context” whose recall collapses after 100K. SILX AI has yet to publish full long-context benchmarks (RULER, LongBench, NIAH, etc.), so we’ll have to wait for third-party testing.

Also, the “first public checkpoint” + “architecture preview” positioning means this model has almost zero optimization for instruction following, conversation ability, and safety alignment. Using it directly for Chat applications will likely result in poor experiences. It’s better suited as a starting point for continued pretraining / SFT.

The true value of Quasar-Preview will only be clear once later Quasar models—especially Instruct versions—are released. But as a clear signal of “we’re going all-in on hybrid architecture + long context,” it’s already strong enough.

References

• silx-ai/Quasar-Preview · Hugging Face — Official model release page with full architecture description, usage, and roadmap.
• SILX AI officially releases Quasar-Preview · linux.do — Chinese community summary and discussion of the release, including Safe NoPE / DrOPE technical details.