An Alternative Player with 232M Parameters: Hierarchos Packs RNNs and Memory Slots into a Small Model

On July 2, a technical report signed by Makhi Burroughs (netcat420), Lost Time, and the Hierarchos project team was posted to r/MachineLearning. They trained a 232M-parameter model called Hierarchos from scratch. Instead of following the Transformer route, they combined an RWKV recurrent backbone, a hierarchical manager/worker control loop, differentiable slot-based long-term memory (LTM), and a deterministic suffix automaton into a single architecture.

At this scale, it obviously is not competing with GPT-3.5. The authors themselves lowered expectations in the TL;DR: this is not a GPT-3/3.5-level model, but rather a proof that this hybrid architecture can run, avoid training collapse, and maintain coherence on short instructions. But viewed from the vantage point of 2026, it becomes interesting precisely because of that — while everyone else is piling toward trillion-parameter models, some people are still seriously experimenting with memory architectures at the 232M scale.

Why a 232M model is worth paying attention to

First, the numbers. 232M parameters, 13 epochs, trained on a rented RTX 6000 Blackwell 96GB GPU. The dataset was the authors’ own curated netcat420/Experiment_0.1 in Alpaca format. By today’s standards, this setup is basically equivalent to a PhD student thesis budget.

But the problem Hierarchos is trying to solve is not small. It attempts to answer a question that has been repeatedly discussed in the AI agent space but never cleanly resolved: can a small model compensate for its lack of parameter scale through architectural memory mechanisms?

Over the past two years, approaches to AI agent memory have steadily diverged. A major 2025 survey from Peking University, Fudan University, and others categorized the field into a three-dimensional “form-function-dynamics” framework. Mainstream methods mostly rely on external RAG systems, vector databases, and graph databases, treating memory as a service outside the model itself. MemOS went a step further by proposing a three-layer system of “explicit memory, activated memory, and parametric memory,” attempting to manage an agent’s knowledge state like an operating system manages memory. Projects like EverOS followed a pipeline-oriented route, extracting facts and events from conversations into persistent storage.

What all these approaches have in common is this: the model itself remains largely unchanged, while memory capability is handled by external systems.

Hierarchos makes the opposite choice. It burns the memory mechanism directly into the architecture itself.

Architectural breakdown: four components, four responsibilities

The report describes the model as a non-Transformer hybrid with four core components:

• RWKV backbone: Responsible for sequence modeling. RWKV is an RNN variant that has been discussed in niche circles for years. During training it supports Transformer-style parallelization, while during inference it maintains constant-level VRAM usage like an RNN. For a 232M model, this is a practical choice, especially if you want long-context handling without KV cache memory exploding.
• Hierarchical manager/worker loops: This is the part of the architecture that feels most “agent-like.” The manager handles planning and action decisions, while the worker executes tasks. This structure is not new in LLM agent frameworks, but embedding it directly into the model’s internal loop instead of constructing it through prompts is a different approach.
• Differentiable slot-based long-term memory: This is Hierarchos’ key differentiator. The LTM is implemented through a set of learnable memory slots, with fully differentiable read/write operations. Gradients can backpropagate through memory access into the model parameters. This is fundamentally different from external vector databases, whose retrieval operations are non-differentiable. In vector DB setups, the model never truly learns “how to use memory”; it only learns “how to encode queries.”
• Deterministic suffix automaton: This is the most counterintuitive component. In an architecture otherwise dominated by soft, continuous, differentiable modules, the authors inserted a hard deterministic suffix automaton. Its role is likely precise matching and retrieval acceleration, compensating for neural components’ weakness in exact recall. This idea — embedding symbolic structures inside neural networks — has appeared sporadically in retrieval-augmented systems over the past two years, but is still relatively rare in small models.

Packing all four of these components into 232M parameters and still getting the model to train stably is itself a significant engineering effort.

The most honest part of the report: the list of pitfalls

The most valuable part of technical reports is often not the architecture diagram, but “what went wrong.” Hierarchos states this very directly in its TL;DR:

Most of our breakthroughs came from fixing subtle train/inference parity mismatches and numerical stability bugs.

In other words, most of the progress did not come from brilliant moments of model design, but from aligning training and inference behavior and fixing numerical stability bugs.

That honesty deserves credit. In hybrid architectures, train/inference parity is a hidden killer. RNN-style models often use teacher forcing or parallelized paths during training, but pure autoregression during inference. If the numerical behavior between the two paths diverges, training metrics may look great while deployment performance falls apart. Add hierarchical manager/worker control flow, differentiable memory read/write operations, and deterministic branches from the suffix automaton, and even a tiny mismatch in one component can cause the entire system to drift unpredictably during inference.

The authors say they managed to “survive training, avoid collapse, and maintain short-form instruction coherence.” Those three claims carry weight — training collapse, mode collapse, and broken instruction-following are the three common ways small models fail, and Hierarchos avoided all of them. But notice the qualifier “short-form.” It frankly tells you that long-form generation and complex reasoning are not yet within this model’s scope.

Compared with mainstream approaches, what is Hierarchos betting on?

Viewed within the context of early 2026, Hierarchos is effectively betting on several things:

First, it is betting that the cost-performance curve of small models plus strong architecture is not exhausted yet. Most companies building agents today follow the route of “large general-purpose model + RAG + tool calling”: expensive, high-latency, but with a relatively low engineering barrier. Hierarchos is betting on the opposite direction: giving small models architectural memory capabilities so that 200M-scale models can handle some stateful tasks, such as personal assistants or persistent conversations on edge devices.

Second, it is betting that the inference-cost advantages of RNN-style approaches will become important again. Transformers’ KV cache problem is becoming increasingly severe in long-context scenarios. Hybrid SSM/RNN approaches such as Mamba and RWKV have continued gaining attention in academia over the past year. By using RWKV as its backbone, Hierarchos is placing another bet on that branch.

Third, it is betting that differentiable memory is more worthwhile than external memory systems. This is the biggest bet of all. External RAG systems benefit from modular decoupling and easy upgrades, but the tradeoff is that the model never truly learns “what to remember” or “when to use memory.” Differentiable memory slots allow gradients to propagate through the entire memory access path, meaning the model could theoretically learn much more sophisticated memory strategies. The cost is dramatically increased training difficulty and greater challenges in scaling.

Some grounded conclusions

To be clear: Hierarchos is not currently a production-ready model. A 232M-parameter model with coherent short instructions still falls far behind models like Llama 3 8B, Qwen 2.5 7B, or even the Phi series in absolute capability. The authors themselves make no such comparison claims.

Its value lies elsewhere:

• Architectural validation: It demonstrates that the combination of RWKV + hierarchical control + differentiable memory + deterministic automata can be trained from scratch, converge, and maintain basic instruction-following ability. This is a useful feasibility report for future researchers.
• Engineering handbook: The pitfalls around train/inference parity and numerical stability have already been explored once by the authors, and the lessons in the report are valuable for any team experimenting with hybrid architectures.
• Contribution to the open-source ecosystem: Among increasingly large open-source models, having another project focused specifically on memory architectures for small models is beneficial for the community.

If you are building edge-side agents, domain-specific small models, or are interested in non-Transformer architectures, Hierarchos is worth following. If you simply want a ready-to-use assistant model, there are already many mature options on the market.

What to watch next

The project is still at the “preliminary findings” stage. It has not yet run mainstream benchmarks or performed systematic comparisons with similarly sized models. Both are necessary next steps; otherwise it will be difficult to determine how much net gain this architecture actually provides over a Transformer baseline with the same parameter count.

Another important direction to watch is scaling behavior. Hybrid architectures are most vulnerable to the problem of “working at small scale but collapsing when scaled up.” The behavior of hyperparameters such as the number of differentiable memory slots, the depth of manager/worker hierarchies, and the granularity of suffix automaton integration at 1B or 3B scale will determine whether this path has long-term viability.

In the short term, Hierarchos is more of a proof of concept. But it at least serves as a reminder that beyond the parameter arms race, the path of “small models + architectural innovation” still has room left — and is far from exhausted.

References

• Hierarchos: Preliminary Findings From a 232M Recurrent Memory-Augmented Assistant Model - Reddit: Original project technical report post, including architectural details, training configuration, and summaries of engineering lessons learned.
• Interpretation of the Latest 2025 “AI Agent Memory” Survey - Zhihu: Analysis of the AI agent memory survey jointly released by institutions including Peking University and Fudan University, introducing the “form-function-dynamics” framework that provides useful context for understanding Hierarchos’ architectural choices.
• hello-agents Chapter 8: Memory and Retrieval - GitHub: An open-source project breaking down engineering implementations of working memory, episodic memory, semantic memory, and perceptual memory, useful for comparison with Hierarchos’ architecture-integrated approach.