Training a 1B model for $1,500 – why did HRM-Text make the Bengio team place their bets?

Over the weekend, the AI community was abuzz with a training cost report: a base language model with 1 billion parameters, pre-trained from scratch, running for 1.9 days on 16 GPUs, with an estimated total compute cost of around $1,500. The model is called HRM-Text. HuggingFace CEO Clément Delangue directly retweeted and recommended it on X, and Yoshua Bengio’s team also stated they are following up on this line of research.

What does $1,500 mean? For traditional pre-training at the same scale, using mainstream cloud GPU prices at full capacity, the cost starts at five figures; compared to Meta’s LLaMA-7B training costs back then, the difference is several orders of magnitude. So the question isn’t "Is HRM-Text powerful?", but rather "How can it be this cheap?"—and whether this path is viable.

First things first: What exactly is HRM?

HRM stands for Hierarchical Reasoning Model. This architecture wasn’t invented yesterday—it had prototypes in earlier experiments with reasoning-focused small models. HRM-Text applies this hierarchical concept to base language model pre-training, which is typically the most expensive stage, and proves that it works.

In a nutshell: Stop treating every token equally—process them at different granularity levels over different time scales.

Specifically, HRM divides the network into two loops:

• Lower level (fast module): High-frequency updates, handling local, fine-grained language patterns—similar to what a traditional Transformer does.
• Upper level (slow module): Low-frequency updates, advancing one step only every few iterations, integrating long-range context, abstract semantics, and global planning.

The two layers exchange information through gating mechanisms. The result: within the same compute budget, the model spends more effective computation where it matters, instead of maxing out on every token. This is somewhat reminiscent of the brain’s dual systems (System 1/System 2), but closer to the “hierarchical time scale” hypothesis in classic cognitive science.

For training data, the HRM-Text team didn’t use several terabytes of tokens scraped from the entire web, but instead did aggressive filtering and curriculum learning: start with well-structured, high-density data (textbooks, Wikipedia, code, synthetic reasoning chains), and gradually introduce noisier web data. This “less but better” strategy, combined with the architecture’s efficiency, is key to bringing the cost down to $1,500.

Why is this catching attention?

Frankly, over the past two years, “low-cost strong model” stories aren’t rare. The most famous is Fei-Fei Li’s team’s s1 last year—replicating o1-level reasoning ability for $50—but that was distillation + fine-tuning, essentially building on top of Qwen’s already-trained foundation. $50 for s1 doesn’t buy you a from-scratch base model.

HRM-Text is different—it’s from scratch pre-training. This is fundamentally different:

• Fine-tuning / Distillation: Relies on a strong teacher model—essentially knowledge transfer and compression.
• Pre-training: Starts from random initialization and must build a world model from data.

The latter’s cost curve has long been thought difficult to reduce, as “emergent abilities” seem to scale power-law with training compute, repeatedly confirmed by Kaplan and Chinchilla scaling laws. HRM-Text’s significance is that it provides a small-scale counterexample: architecture improvements can pry open the scaling law.

That’s why Bengio’s team is interested. Bengio has spent the past two years publicly questioning the “brute force compute” path, advocating for more structured, cognitively aligned architectures. HRM-Text fits right into that direction, serving as a verifiable small-scale sample.

Performance—don’t be blinded by $1,500

Let’s temper expectations: the 1B parameter HRM-Text’s results on general benchmarks are “mixed” compared to same-size Pythia-1B and TinyLlama—meaning it hasn’t broken limits, just reached similar levels using less money.

Specific results:

• HellaSwag: Slightly above TinyLlama, below Pythia-1.4B.
• ARC-Challenge: Roughly tied with models of the same size.
• GSM8K: Significantly better—this likely relates to its layered architecture excelling at multi-step reasoning.
• Long text perplexity: Outperforms same-size baselines beyond 2k-token windows.

So, can this 1B model be production-ready? Probably not—it can’t replace your current Qwen-1.5B or Phi-3-mini. But is the methodology worth following? Absolutely.

The bigger question is scaling: Can HRM’s layered mechanism be scaled to 7B, 70B and still retain efficiency gains? We have no public experimental data yet—only a “future work” note in the paper. This is the biggest uncertainty. Many architectures that shine at small scale (e.g., various Linear Attention variants) get crushed by standard Transformer implementations when scaled up.

The logic behind that $1,500 number

Let’s see how the team cut costs:

1. GPU choice: Not H100s, but cheaper A100s or even older generation. A 16-card cluster on cloud spot pricing is only a few dollars per hour.
2. Training duration: 1.9 days—about 45 hours.
3. Data volume: Much smaller than Chinchilla’s optimal mix—architecture efficiency traded for less data.
4. No R&D labor cost: The $1,500 is just “electricity after hitting Enter”—architecture tuning, data cleaning, hyperparameter search aren’t included.

Point #4 is important: Every time a “$XX / $X thousand strong model” story hits, comments point out—researcher salaries, failed experiments’ compute, data preprocessing costs are excluded. This is true—$1,500 is the replication cost, not the invention cost.

But replication cost reduction is itself hugely valuable. It means:

• Small academic labs can afford comparative experiments.
• Startups can do domain-specific pre-training (medical, legal, code) cheaply.
• Education can let students actually train a model rather than just read papers.

Some in the open-source community are already forking the code to attempt 1.5B and 3B replications. Expect the first unofficial benchmarks in two weeks.

Industry impact: another challenge to scaling laws

In recent years, many works have challenged scaling laws:

• Data side: The Phi series shows “textbook quality” data can achieve more with less.
• Architecture side: Mamba, RetNet, and other linear architectures try to break attention’s quadratic complexity.
• Training side: MoE uses sparse activation to spread inference cost.
• Post-training side: DPO, RLHF variants cut alignment costs to the thousands.
• Distillation side: DeepSeek keeps improving the efficiency of transferring big-model capabilities into small models.

HRM-Text is a “architecture + data” combo, tackling pre-training—the hardest nut to crack. Its existence won’t overturn scaling laws, but will make everyone reexamine: Of the compute we burn, how much genuinely teaches the model something new, and how much is wasted on ineffective computation?

If half is the latter, then a significant portion of the past three years’ compute arms race is waste. Considering that top players like OpenAI, Anthropic, and xAI spend billions annually on compute, the answer is worth a lot of money.

How developers can use it

HRM-Text is already open-sourced on HuggingFace, with weights and training code available. Tips:

• For inference: Load directly via transformers, but install a custom modeling file—layered structure isn’t in the standard implementation.
• For domain fine-tuning: Authors recommend LoRA—full-parameter fine-tuning may upset the coordination between layers.
• For training replication: Code is a fork of nanoGPT—readable, but still requires 16×A100 hardware.
• For architecture innovation: Layer update frequency ratios, gating mechanisms, curriculum order—all remain open hyperparameter space.

OpenAI Hub has integrated HRM-Text into the open-source model router, letting you call it via an OpenAI-compatible interface without self-deployment.

Unanswered questions

Some points worth following up:

1. Scalability: Can HRM above 7B retain efficiency gains?
2. Multimodal: Hierarchical time scales seem natural for vision/audio—will we see HRM-Vision, HRM-Audio?
3. Inference efficiency: Training is cheap—what about inference? KV cache impact hasn’t been fully evaluated.
4. Alignment: Base model’s out—when will instruct and RLHF versions arrive?
5. Relationship to SSMs (Mamba): Both aim to break the Transformer unified architecture—will they merge or replace each other?

Likely, these will be answered within the next two to three months—that’s open-source speed.

A quick judgement

Taken alone, HRM-Text is a clever small-scale project. In the bigger picture, it’s another signal in the “post-scaling-law” era: While top players burn money to push frontier models like GPT-5 or Claude 4, another route is maturing—using architectural efficiency to reduce compute costs, making base model training accessible to mid-sized teams.

This route won’t replace the mainstream quickly, but it will chip away at the narrative that “you need tens of thousands of GPUs to play with AI.” For developers, this is good news: Over the next three years, open-source base models that are affordable, modifiable, and understandable will only increase.

Will HRM-Text itself become the next mainstream architecture? Honestly, many similar “challengers” appeared in the past two years, few survived. But each challenger pushes toward the same goal: making AI cheaper, more interpretable, less dependent on brute force. That goal is valid.

References

• HuggingFace - HRM-Text model page — Hosted weights, configs, and example code for HRM-Text, ready to load and use.
• Reddit r/MachineLearning - HRM-Text discussion thread — Community technical discussion on training cost and architecture design, includes author Q&A.
• Zhihu - Interpretation of HRM architecture — In-depth Chinese community analysis of hierarchical reasoning model principles.