HuggingFace Releases EMO: Enabling MoE to Develop Modularization on Its Own

MoE Is Not Just About Saving Compute—It Conceals the Model’s “Modular Gene”

In early May, AllenAI published a post on the HuggingFace blog titled EMO (Pretraining Mixture of Experts for Emergent Modularity). From the name, it sounds like just another MoE variant, but upon reading it, you realize the authors are probing a deeper question: During pretraining, do large models spontaneously develop “functional partitions”? If so, can MoE make this process explicit and controllable?

This perspective is far more interesting than simply scaling parameters. Over the past two years, the MoE narrative has been dominated by works like Mixtral, DeepSeek-V3, and Qwen-MoE, tying MoE tightly to the engineering goal of “building larger models under equivalent compute.” Experts have been treated purely as tools for capacity expansion. EMO refocuses attention on a set of foundational questions from around 2017 about “conditional computation” and “modular neural networks” — experts should correspond to capabilities, not just FLOPs.

What EMO Does: Treating “Emergent Modularity” as a Training Objective

According to AllenAI, EMO’s core principles can be summarized in three points:

1. In the later stages of pretraining, dense Transformers exhibit clear functional clustering within FFN layers — some neurons activate only for code, some are sensitive to multilingual text, some respond only to mathematical symbols. This has been repeatedly verified in works like Skill Neurons (2022) and MoEfication (2023).
2. Since this kind of “modularity” emerges naturally, instead of waiting for it to appear and then post-process it (as MoEfication does by slicing a trained dense model into an MoE), we might as well use an MoE structure during pretraining to guide and amplify this modular tendency.
3. Modularity shouldn’t be measured solely by whether the routing distribution is balanced (the traditional MoE concern of load balancing); more importantly, each expert should stably correspond to a consistent semantic/task class. Consequently, EMO introduces new evaluation metrics focused on the “functional specificity” of experts.

In this framework, the formerly indispensable auxiliary loss from traditional MoE training (which enforces balanced expert loads) becomes something of a villain—it forcibly distributes tokens evenly among experts, which disrupts naturally forming functional clusters. EMO relaxes this balance constraint and introduces a mutual-information-based objective that encourages experts to diversify in representation and complement one another functionally.

Technical Details: Routing, Losses, and Expert Granularity

From the details provided in the blog, several of EMO’s key design choices merit closer examination.

1. Fine-grained Experts + Shared Experts

EMO adopts the same design lineage as DeepSeekMoE — make experts smaller and more numerous, while keeping one or two “shared experts” for general knowledge. Coarse-grained experts (e.g., Mixtral’s 8 experts) easily degrade into “each expert does a bit of everything,” making modularity impossible; fine-grained setups (dozens to hundreds of experts) provide room for functional specialization.

The shared experts are a pragmatic compromise: since some computations (e.g., syntax processing, frequent token embeddings) are task-agnostic, separating them allows sparse experts to focus on “differentiated” capabilities.

2. Mutual Information–Driven Auxiliary Objective

This is EMO’s most fundamental departure from traditional MoE. In models like Switch Transformer or GShard, the load-balancing loss is essentially an entropy regularizer—aiming for uniformly distributed routing. EMO replaces that objective with one that:

• Encourages activation patterns within the same expert across tokens to be highly consistent (expert cohesion)
• Encourages different experts’ activation patterns to be as orthogonal as possible (expert decoupling)

This objective resembles a classic clustering loss more than a balancing one. The trade-off is stability—it’s harder to train, prone to “winner-takes-all” collapse, requiring careful temperature scheduling and router initialization.

3. Quantifiable Evaluation of Modularity

EMO proposes evaluating each expert’s activation distribution over probe tasks (code, math, multilingual, commonsense reasoning, etc.) to derive an “expert-capability” correspondence matrix. The closer this matrix is to a permutation matrix (each expert cleanly maps to one skill cluster), the higher the modularity.

This evaluation fills a long-standing gap in MoE research: everyone talks about expert specialization, but few have systematically proven that such specialization actually exists—and aligns with human-understandable capability dimensions.

Why This Matters

By 2026, MoE has become the default option for large models. DeepSeek-V3, Qwen3-MoE, and Llama 4 Maverick all use sparse architectures, with hundreds of billions of parameters and tens of billions active per step. Yet engineering maturity masks an awkward truth: we know almost nothing about what each expert is actually learning. Most visualizations stop at “Expert 3 likes punctuation marks,” far from true interpretability or controllability.

EMO’s value lies not in topping a benchmark but in restoring “modularity” to the core of MoE research. This has at least three potential benefits:

• Interpretability: If experts truly map to distinct skills, we finally get something like an internal “map” of the model that’s more informative than attention visualizations.
• Editability: Modularity enables “capability-level surgery”—e.g., disabling a “code expert” for safety testing, or fine-tuning a few specific experts to enhance a targeted ability, without touching the whole model.
• Continual Learning: Modular structures naturally support incremental training—new capabilities can be added as new experts without overwriting old weights, potentially mitigating the catastrophic forgetting problem.

Mild Reservations

Of course, some caution is warranted. The experiments disclosed so far use relatively small-scale models (tens of billions of parameters), not in the same league as practical giants like DeepSeek-V3 with 671B total params. Whether the mutual-information regularizer scales stably to trillion-level models remains uncertain.

Another concern is a possible trade-off between modularity and overall performance. Enforcing strict specialization may sacrifice the “feature reuse” typical of dense models—many abilities overlap (e.g., mathematical reasoning draws on linguistic capacity, code generation draws on logic), and hard separation could hurt performance. AllenAI’s blog candidly reports that EMO underperforms a standard MoE of equal size on some composite tasks—a point in its favor academically, showing transparency.

There’s also a practical issue: current industry MoE optimizations all assume “balanced load + high-throughput inference.” Frameworks like vLLM, SGLang, and TensorRT-LLM implement expert parallelism based on roughly even load distribution. EMO’s deliberately unbalanced routing will skew expert utilization, complicating deployment. That’s an engineering problem—but a nontrivial one.

Relation to Contemporary Work

EMO didn’t emerge in isolation. Viewed in the context of MoE research circa 2025–2026, the picture becomes clearer:

• DeepSeek series: Perfected the fine-grained + shared-expert design, focused on training stability and inference efficiency.
• Qwen3-MoE: Introduced more context-aware routing strategies, still primarily performance-driven.
• Anthropic’s interpretability work: Explores sparse autoencoders (SAEs) to probe latent features externally—an “outside-in” approach, in contrast to EMO’s “endogenous modularity.”
• EMO: Internalizes interpretability goals into pretraining itself, bridging the two paradigms.

In short: DeepSeek cares about faster training, Anthropic cares about model cognition, and EMO tries to answer both.

Practical Takeaways for Developers

If you’re engineering or researching MoE systems, EMO’s approach offers several takeaways:

• Even without adopting mutual information losses directly, you can relax load-balancing weight in later training phases to observe whether experts self-differentiate.
• When evaluating MoEs, don’t stop at perplexity or benchmarks—add probe tasks to inspect expert-level functionality distributions; it reveals hidden issues.
• The fine-grained + shared-expert setup is already validated; if you haven’t tried it, it’s worth experimenting with—it delivers major gains in small-scale MoEs.
• For tasks like model editing, safety alignment, or capability isolation, more modular MoEs offer far greater manipulability than dense models—by an order of magnitude.

EMO’s model weights and training code are already released on HuggingFace, sized for single-machine or small-cluster replication—an excellent starting point for research.

In Closing

MoE has come far in engineering, but theory and interpretability have lagged. EMO isn’t the kind of project that grabs headlines—it doesn’t chase leaderboards or scale records, and even loses slightly on some metrics—but it brings back a neglected question: What do we actually want from MoE? Just cheaper compute—or more structured intelligence?

There’s no standard answer—but asking the question is valuable in itself.

References

• EMO: Pretraining mixture of experts for emergent modularity (HuggingFace Blog) — AllenAI’s original EMO blog post on HuggingFace, including technical details and experiments
• A Detailed Introduction to Mixture-of-Experts (MoE) (HuggingFace) — A systematic Chinese overview of MoE architecture, routing mechanisms, and training techniques
• Understanding Mixture-of-Experts MoE in One Article (Zhihu) — A Chinese interpretation of MoE core components and representative works
• In-depth Analysis of MoE Architectures in Large Language Models (Zhihu) — A long-form article tracing the historical evolution of MoE concepts