JetBrains Releases Mellum2: 12 Billion-Parameter MoE Model Now Live

JetBrains has just launched Mellum2 on Hugging Face—a 12-billion-parameter Mixture-of-Experts (MoE) model. This marks another major move by JetBrains in AI-assisted programming, following its debut code generation model last year.

Unlike traditional dense models, Mellum2 employs an MoE architecture with eight expert networks. Only two experts are activated for each inference, giving it an effective computational load equivalent to a 2-billion-parameter model. This design allows it to maintain strong performance while achieving inference speeds several times faster than equally sized dense models.

MoE Architecture: The Design Philosophy of Specialization

The core concept behind Mixture-of-Experts models is “divide and conquer.” Instead of having one large neural network handle all tasks, an MoE model delegates work to multiple specialized sub-networks (experts). A gating network decides which experts to activate for each input.

In Mellum2, this mechanism manifests as follows:

• 8 expert networks: each expert is an independent feed-forward neural network (FFN), specializing in particular code patterns or language features
• Top-2 routing strategy: the gating network scores the affinity of each input token and selects the top two experts to process it
• Sparse activation: only 25% of parameters (2 out of 8 experts) are activated per inference, dramatically reducing computation cost

This architecture was first introduced into Transformers by Google’s GShard paper in 2020, and later validated in models such as Switch Transformers (1.6 trillion parameters) and Mixtral 8x7B. JetBrains’ adoption of this design clearly targets two core benefits MoE offers in code generation: faster inference and stronger multitask generalization.

Performance: Dual Optimization for Code Generation and Multilingual Tasks

JetBrains shared Mellum2’s benchmark results in its official blog. The data shows solid performance on code completion, code explanation, and multilingual translation tasks.

Specifically:

• HumanEval (Python code generation): pass@1 = 68.3%, surpassing CodeLlama 13B (65.8%)
• MBPP (basic programming problems): pass@1 = 72.1%, roughly equal to StarCoder 15B
• MultiPL-E (multilingual code generation): average score of 61.5% across Java, JavaScript, and C++, an 8-point improvement over Mellum1

Equally impressive is inference speed. JetBrains’ tests show that under identical hardware conditions (single A100 GPU), Mellum2 achieves 3.2× higher throughput and 60% lower latency than a dense model of equal parameter count. That’s a key indicator for IDE scenarios requiring real-time code completion.

However, MoE models come with trade-offs. Although only 2 billion parameters are active during inference, total parameter size remains 12 billion, meaning memory usage doesn’t shrink. Running Mellum2 on consumer hardware requires at least 24GB VRAM (FP16 precision) or 12GB (INT8 quantization)—a significant barrier for individual developers.

Training Details: From Data to Optimization Strategy

The Mellum2 training corpus contains 1.2 trillion tokens spanning over 50 programming and natural languages. JetBrains hasn’t disclosed data sources, but given its strong multilingual performance, the dataset likely includes extensive open-source repositories, technical documentation, and Stack Overflow Q&As.

Key optimization strategies used during training:

1. Load balancing loss — prevents overuse of certain experts and idling of others by adding an auxiliary term encouraging balanced utilization. This standard MoE technique was first introduced in the Switch Transformers paper.
2. Expert capacity limit — each expert has a maximum token capacity (capacity factor). Tokens exceeding the limit are dropped or rerouted, avoiding bottlenecks but potentially sacrificing optimal handling of some inputs.
3. Layered learning rates — higher learning rate for the gating network to quickly learn routing, and lower for expert networks to avoid overfitting.
4. Mixed-precision training — trained in BF16 (Brain Float 16) format to reduce memory usage while maintaining numerical stability, crucial when handling 12 billion parameters.

JetBrains reports Mellum2’s training was conducted on 256 A100 GPUs for three weeks, consuming about 500,000 GPU hours—significantly less than the 700–800K GPU hours typical for dense models of similar size, showcasing MoE’s efficiency advantage.

Challenges of MoE: More Than Just Splitting a Model

While elegant in theory, the MoE paradigm still faces practical challenges.

Training instability is one of the biggest. Because the gating network must learn task allocation and expert networks depend on its decisions, their interdependence can cause instability. If the gating network makes poor early choices (e.g., repeatedly selects the same expert), other experts remain undertrained, degrading performance. The Switch Transformers paper notes frequent “loss spikes” requiring careful tuning of learning rates and initialization.

Load imbalance is another common issue. Ideally, each expert processes roughly equal token counts, but in practice, some may be heavily used while others rarely activated. This wastes compute and weakens generalization. Solutions include auxiliary loss terms, dynamic capacity adjustments, or even forced expert rotation during inference.

Communication overhead is especially significant in distributed training. Because experts may reside on different GPUs, the gating network must exchange data across devices. As models scale and expert count rises, this cross-device traffic becomes a bottleneck. Google’s GShard addressed this with customized sharding strategies—effective but complex.

Memory usage during inference also remains nontrivial. Even though only part of the model activates, all expert weights must be loaded into VRAM. For a 12B model like Mellum2, INT8 quantization still demands at least 12GB VRAM—challenging for local deployment.

Open-Source Strategy: JetBrains’ AI Ambition

JetBrains’ decision to open source Mellum2 on Hugging Face is telling.

As an IDE-focused company, JetBrains could have kept Mellum2 proprietary, bundling it into IntelliJ IDEA or PyCharm as a paid feature. Instead, they chose open access—perhaps for several reasons:

1. Ecosystem building — open sourcing invites developer participation, fostering a community around Mellum2 and expanding JetBrains’ influence in AI-assisted coding.
2. Data feedback — public availability enables broader usage data and feedback, essential for improving performance on real-world code tasks.
3. Competitive pressure — rivals like GitHub Copilot, Cursor, and Codeium are iterating fast; open sourcing demonstrates JetBrains’ technical strength.
4. Exploring business models — while the model itself isn’t monetized, it underpins JetBrains AI Assistant and related paid services offering higher speed, reliability, or enterprise support.

Mellum2 saw over 5,000 downloads within 24 hours of its release on Hugging Face. Developers are already sharing experiences—some integrating it into VS Code plugins, others using it for code review tools.

The Future of MoE: More Than Parameter Stacking

From GShard’s introduction in 2020 to Switch Transformers’ trillion-scale in 2022 and Mixtral 8x7B’s current buzz in 2024, MoE architectures have advanced rapidly. But are they the ultimate answer for large models?

Not necessarily.

The main advantage—sparse activation—keeps computational cost low but doesn’t reduce total parameters or memory footprint, still limiting use on consumer hardware. Moreover, MoE models demand complex engineering for training and deployment compared to dense models.

Other paths are emerging:

• State Space Models (e.g., Mamba) — linear-complexity sequence modeling showing promise for long texts
• Retrieval-Augmented Generation (RAG) — reduces parameter size by leveraging external knowledge bases
• Distillation and quantization — allow large models to run efficiently on smaller hardware
• Multimodal fusion (e.g., GPT-4V, Gemini) — expands applicability to vision and speech tasks

MoE is less a paradigm shift than an optimization within the scaling-law era—it enables training larger models with less computation but doesn’t change the “more parameters, better performance” rule.

True breakthroughs may stem from rethinking architectures, training approaches, or even task definitions. As Rich Sutton noted in The Bitter Lesson, simple methods plus large-scale computation often outperform clever ones—but that doesn’t mean innovation should stop.

What Developers Should Know

Mellum2’s release offers several takeaways for developers:

1. Local deployment feasibility — though 12 billion parameters sound hefty, MoE and quantization make consumer-level runs possible. With a GPU of 24GB VRAM (e.g., RTX 4090 or 3090), local deployment for code completion or review is achievable.
2. Multilingual support — Mellum2 performs well across 50+ programming languages, great for teams managing multilingual codebases. Applications include cross-language translation, API docs generation, or unit test creation.
3. Fine-tuning flexibility — JetBrains open-sourced weights and training code, enabling domain fine-tuning on your own repositories for in-house style or logic alignment. Be mindful of MoE-specific complexities like load balancing and capacity management.
4. Performance vs cost trade-off — MoE models excel in low-latency use cases (e.g., real-time autocomplete), while dense or smaller MoEs (e.g., Mixtral 8x7B) may suit batch tasks with tighter budgets.
5. Toolchain maturity — Hugging Face Transformers supports MoE models, yet inference optimization, quantization, and distributed deployment tooling remain less polished. Production users should expect a learning curve.

Conclusion

Mellum2 marks another milestone for JetBrains in AI-assisted programming. With 12 billion parameters, eight experts, and an open-source release, it reflects both technical vision and commitment to the developer community.

While the MoE approach isn’t new, applying it effectively to code generation—and making it accessible—is noteworthy. Whether Mellum2 can stand out against competitors like GitHub Copilot or Cursor remains to be seen.

What’s certain is that AI-assisted coding is far from settled. Architectures, methodologies, and use cases are evolving rapidly, with breakthroughs every few months. For developers, staying curious and hands-on is the best way to ride the wave.

Mellum2 is now open source on Hugging Face. Interested developers can download and experiment. JetBrains says more language-specialized versions and deeper IDE integrations will follow in the coming months.

---

References

• Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains – official JetBrains blog introducing Mellum2’s architecture, performance, and training details
• Mixture-of-Experts (MoE) Explained – Chinese Blog – Hugging Face Chinese blog detailing MoE principles, training, and applications
• Mixture of Experts (MoEs) in Transformers – Hugging Face English blog diving deep into MoE implementation and optimization in Transformers