MRU Update: An independent developer's alternative to Attention is now running successfully

A Refurbished Attention Alternative

On June 20, a small but interesting update appeared on Reddit’s r/MachineLearning: an independent developer revisited their Matrix Recurrent Units (MRU) project released a year ago, announced that they had solved the previous training instability issues, and shared details of the improved implementation.

A year ago, this might have gone unnoticed—just another grassroots attempt at challenging Transformer. But in mid-2026, the context is very different: from Kimi Linear’s open-source release last autumn, to Qwen3-Next’s hybrid linear approach, and Minimax M2’s rollback from M1’s linear scheme back to Full Attention, the industry’s discussion around “how to get rid of O(N²)” has moved from “should we do it” to “specifically how to do it” in a more refined stage. At such a moment, an independent developer’s public recap of a year’s worth of trial and error can be more valuable than a big-company paper—because it covers the kinds of failures that never make it into publications.

What MRU Actually Does

Let’s be clear—MRU isn’t a ground-breaking new paradigm. Mechanically, it fits within the broad family of Linear RNN / Linear Attention.

The core process has only three steps:

• Dimension expansion: reshape or transform each token’s embedding vector into an input state matrix
• Cumulative multiplication: multiply these matrices sequentially along the sequence dimension to produce the output state matrix
• Dimension reduction: transform the output state matrix back into a vector as this step’s output

Sounds pretty straightforward, but two key points determine whether it’s “usable”:

First, matrix multiplication is associative—i.e., (A·B)·C == A·(B·C). The author leveraged this to write a parallel scan implementation, enabling MRU to avoid degrading into the disastrous serial training speeds typical of RNNs on GPUs. This is aligned with Mamba’s selective scan and the chunk-wise parallelism approach in Linear Attention—without parallel scan, all RNN-like architectures are dead on modern GPUs.

Second, cumulative matrix multiplication is essentially a state transition. At each step, multiplying the current “state matrix” by the new input matrix is equivalent to hidden state updates in RNNs—except the state is a matrix, and the transition operation is matrix multiplication instead of non-linear activation. This gives MRU theoretically stronger expressive power than scalar-gated linear attention, but makes stability harder to control—precisely where the author stumbled last year.

The Two Fatal Issues From a Year Ago

In 2025, MRU achieved good results on toy datasets like shakespeare-char, but two problems raised in the comment section put it back to square one:

1. Matrix states were unbounded. If eigenvalues of the cumulative product matrices exceeded 1, states exploded exponentially; below 1, they decayed exponentially to zero. This “matrix multiplication” flaw is the original sin of all such architectures—RWKV, Mamba, DeltaNet, and KDA in Kimi all address it differently.
2. Training instability. With more complex text datasets, the model simply wouldn’t train.

These are really two sides of the same coin—if states are unbounded, gradients inevitably blow up.

In the latest update, the author said their main focus was on the “how to construct the input state matrix” step. The original approach was to simply reshape vectors into matrices, with no constraints on the spectral properties. The improved version tries different parameterizations aimed at producing matrices with inherent stability at the multiplication level—for example, restricting them to certain matrix groups, or using structured decomposition (similar to DPLR) to make the cumulative multiplication controllable.

This Idea Isn’t Alone

This brings us to what academia and industry have been doing lately—because MRU’s value lies in its collision with mainstream directions.

Kimi Linear (open-sourced late October last year by Moonshot AI) essentially adds fine-grained structural constraints to the state transition matrices of Linear Attention. Its Kimi Delta Attention (KDA) introduces channel-level forget gating, updates state based on an improved Delta Rule, and uses an engineering trick called Diagonal-Plus-Low-Rank (DPLR)—splitting the state transition matrix into a “diagonal block + low-rank patch,” preserving expressiveness while ensuring GPU-efficient parallelism. Kimi Linear ultimately adopts a 3:1 hybrid layer design: one Full Attention layer for every three KDA layers.

According to Yang Songlin, one of Kimi Linear’s paper authors, in a podcast, this 3:1 ratio is becoming an industry consensus. Qwen3-Next uses a similar structure. The reason is simple: pure linear attention has inherent weaknesses in multi-hop reasoning and needs some full attention layers for semantic aggregation.

A counterexample is Minimax. The M1 version used Lightning Attention (a linear scheme), but M2 reverted to Full Attention. Why? The podcast’s explanation: the real competitor to linear attention isn’t Sparse Attention (like DeepSeek), but Sliding-Window Attention. Under fair comparisons, linear attention’s advantage over sliding-window isn’t as big as imagined, yet its engineering complexity is significantly higher.

Looking back at MRU’s “pure linear, pure matrix multiplication” approach, it sits somewhere between Kimi Linear’s “constrained linear attention” and a “pure Mamba-style SSM”—its survival depends entirely on whether the author can get the input state matrix design right.

The Significance of an Independent Developer Doing This

This deserves some elaboration.

At this point in time, the bar for linear attention research has been raised very high by teams like Kimi, Alibaba, and Minimax. An independent developer using toy datasets to compete on absolute metrics with industrial-grade schemes is meaningless and unfair. MRU’s value isn’t in “can it beat Kimi Linear,” but in:

• It fully exposes an extremely simple baseline. Code, pitfalls, and improvements are all on GitHub—transparency that industrial papers can’t match
• It experiments with different methods of input state matrix construction—which is essentially the same underlying question as Kimi DPLR and Qwen3-Next’s structured explorations
• It uses associativity for parallel scan implementation details, making it a nice entry point for developers wanting to understand the internals of FLA (flash-linear-attention) and similar libraries

Some Observations for Developers

If you’re following progress along this line, the following points might help:

1. The Linear Attention story has moved from “can it work” to “how to tune it better”. Kimi Linear is the first scheme to fully surpass Full Attention under fair comparison, but only with hybrid architecture
2. Structuring the state transition matrix is the core battlefield. Whether it’s DPLR, Delta Rule, or experiments like MRU, the fundamental question is “how to keep cumulative multiplication stable while retaining expressiveness”
3. The 3:1 hybrid ratio is becoming the default recipe, at least for medium-scale models
4. NoPE (no explicit position encoding) + data-dependent state transition is replacing RoPE, since gating already encodes positional information
5. Multi-hop reasoning is linear attention’s biggest weakness right now, relying entirely on hybrid layers’ full attention to compensate

Finally

Back to MRU itself—it’s still far from being production-ready, and the author admits this is just an “update to a hobby project.” But after big companies have pushed linear attention to a mature engineering stage, seeing independent developers retrace the path in the simplest way possible—and honestly share their failures and improvements—feels like a sign that the community is still healthy.

As a side note, mainstream open-source linear attention models—Kimi Linear, Qwen3-Next series—are now accessible via OpenAI Hub, with a single API key, compatible with OpenAI format, saving the hassle of deploying vLLM yourself. You can directly compare different architectures’ practical long-context performance.

References

• An Update on Matrix Recurrent Units, an Attention Alternative - Reddit — the original post by the MRU author with improvement details and discussion
• Kimi Linear Study Notes: Making Attention Fast and Good - Zhihu — a deep Chinese interpretation of the Kimi Linear paper, providing context for understanding MRU’s technical lineage
• Kimi-Linear-48B-A3B-Instruct - Hugging Face — the official model and technical report of Kimi Linear