Neo Lab Bets on Large Models That Can Grow

Large Models Need to Learn Not Only How to Reason, but Also How to “Grow Up”

On July 21, Mind Lab released and open-sourced Macaron-V1 under its Neo Lab technology brand, bringing to the forefront a concept that has been discussed in academia for years but has rarely made its way into real-world product systems: Continual Learning.

Founded last October, the company currently has a team of more than 30 people, with backgrounds spanning xAI, DeepMind, DeepSeek, ByteDance Seed, MIT, and Tsinghua University. Founder Chen Kaijie previously built the AI interactive storytelling platform MidReal and the Personal Agent product Macaron, while his team has also participated in Agent research projects such as FireAct.

Neo Lab’s assessment is straightforward: no matter how many parameters today’s large models have, they are still fundamentally “frozen models” produced through one-time training. They can remember what just happened within the context window and retrieve new information through RAG, but once a session ends, their model weights remain unchanged. The next time around, they may make exactly the same mistake again.

Macaron-V1 attempts to solve this problem differently. Rather than frequently retraining the entire foundation model, it attaches multiple LoRA expert modules to the base model and continuously updates these lightweight parameters based on tasks and user feedback. This approach is called Mixture-of-LoRA, or MoL for short.

This is not simply about “adding memory” to a model, nor is it merely another Agent framework layered on top. What it truly seeks to change is the model’s post-deployment lifecycle: going live no longer marks the end of training, but the beginning of the next round of learning.

How Macaron-V1 Works: Freeze the Brain, Update the “Skill Plug-ins”

Neo Lab first released Macaron-V1-Preview last month. According to information disclosed by the team, the Preview version is built on GLM-5.1 and comes with five external LoRA expert modules, each containing approximately 1 billion parameters.

LoRA itself is nothing new. It uses low-rank matrices to approximate incremental changes to some of a model’s weights, allowing developers to perform domain adaptation without modifying every parameter. The foundation model can be thought of as a massive operating system, while LoRA is more like a driver that can be installed, upgraded, and uninstalled independently.

MoL goes a step further in two ways:

1. It separates different capabilities into different experts. Code, reasoning, interaction preferences, and domain-specific knowledge do not all have to be written into the same set of incremental parameters.
2. It routes dynamically at inference time. The system selects a more suitable LoRA based on the current task and combines multiple experts when necessary, rather than sending every request through the same fine-tuned model.

This is similar to traditional MoE, but it operates at a different layer. MoE typically divides the feed-forward network into multiple experts during pretraining, with a router deciding which parameters each token should pass through. MoL, by contrast, is more like adding a replaceable capability layer outside an already trained model. The former focuses on expanding model capacity while controlling compute, whereas the latter is better suited to low-cost post-training, tenant isolation, and rapid iteration.

An even more critical step is user-specific LoRA. Data generated through long-term interaction can be distilled into independent adapters, enabling the model to gradually learn the real preferences of a particular user, team, or business. For example, if a frontend team repeatedly asks the model to use a specific component library, directory structure, and testing conventions, the system does not need to place dozens of rules in the System Prompt every time or immediately modify the shared base model. Instead, it can gradually write stable feedback into a team-level LoRA.

The value here goes beyond simply reducing token consumption. Rules in a prompt are only temporary constraints, and the model may “see them without following them.” A trained LoRA, however, changes the output distribution, turning correct behavior from an external reminder into a more stable internal tendency.

Continual Learning Does Not Mean Letting Models Randomly Modify Their Weights Online

The idea of a “model learning as it is used” can easily evoke a system that performs a gradient update immediately after receiving every piece of user feedback. Truly deployable continual learning systems are generally not this aggressive.

A negative rating does not necessarily mean the model was wrong. The user may have made a mistake, expressed themselves incompletely, or even attempted to poison the model deliberately. If every piece of feedback is immediately incorporated into the model, an attacker would only need to construct a batch of malicious conversations to potentially alter the system’s behavior. For a multi-tenant API service, this would also constitute a serious data isolation failure: Company A’s input must not become part of Company B’s model.

A production-grade continual learning system therefore looks more like a layered pipeline:

• Collect explicit feedback, task completion rates, manual edit records, and other signals online;
• Deduplicate and de-identify the data, while identifying anomalous users and adversarial samples;
• Write short-term preferences into external memory, prompts, or temporary adapters first;
• Train LoRAs in an offline or near-real-time environment;
• Evaluate them against fixed regression sets, safety sets, and historical tasks;
• Distribute new adapters to a small portion of traffic through canary releases;
• Quickly unload the LoRA and roll back to the previous version if metrics deteriorate.

In other words, “continual” describes the model’s entire lifecycle and does not necessarily mean millisecond-level online updates. To developers, the latter may sound cooler, but the former is what is actually feasible from an engineering perspective.

Why the Industry Is Revisiting This Now

The dominant theme of the past two years has been inference scaling: longer chains of thought, more test-time compute, and reinforcement learning based on verifiable rewards. This addresses whether a model can think more deeply about the problem currently in front of it, but it does not fully address whether the model can learn from yesterday’s failures.

At the opening-day main forum of the recently held WAIC 2026, reinforcement learning researcher Richard Sutton once again emphasized the importance of experience-driven learning. His core assessment was that approaches relying on static labeled data for training are nearing their limits, and that the next generation of AI must acquire experience through interaction with its environment. DeepSeek has also recently identified continual learning as a problem to be addressed after Agents.

This is no coincidence. Agents have brought models out of chat windows and into browsers, code repositories, enterprise systems, and real business workflows. For the first time, they have also generated large volumes of verifiable outcome data: whether code passed its tests, whether a website redesign improved conversion rates, whether a support ticket was resolved, and which proposal the user ultimately adopted.

In the past, model training data and product usage data flowed through two separate pipelines. Agents make a closed loop possible. The model performs a task, the environment returns a result, the system extracts a reward, and effective experience is then used for post-training. Without Agents, continual learning lacks high-quality environmental feedback; without continual learning, every task an Agent performs feels like its first day on the job.

This is also where Neo Lab’s approach offers the most value. Rather than building another general-purpose base model with even more parameters, it is attempting to turn product interactions into training infrastructure. For a startup, this is far more realistic than competing head-on in trillion-parameter pretraining.

Why LoRA May Become the Key Interface for Continual Learning

The cost and risk of full-parameter updates are simply too high. The larger the model, the more extreme the gradients, optimizer states, and checkpoints that must be synchronized in each training round. Even more problematic, once the entire model has been modified, it becomes difficult to determine precisely which new training task damaged an existing capability.

Experiments disclosed by Neo Lab show that when performing LoRA reinforcement learning on trillion-parameter models such as Kimi K2, GPU requirements can fall to approximately 10% of those required by conventional full-parameter RL. The relevant experiments were completed using 64 NVIDIA H800 GPUs, rather than the hundreds of GPUs that full-parameter training at a comparable scale might require. This figure still needs to be interpreted in light of batch size, sequence length, and training configuration, but the direction is clear: continual learning can only happen continuously if it is affordable enough.

Another interesting result is that a large model with a small number of trainable parameters may be more effective than full-parameter training on a smaller model. The reason is not mysterious. Reinforcement learning generally does not teach a model a skill from scratch. Instead, it reinforces more effective pathways among the knowledge and reasoning patterns the foundation model already possesses. Larger models have a broader range of “candidate capabilities,” and only relatively small parameter changes may be needed to bring them out.

From a systems-design perspective, LoRA also offers several inherent advantages:

• File sizes are relatively manageable, making version control and distribution easier;
• Adapters can be isolated by user, organization, task, or time period;
• Problematic adapters can be unloaded without rolling back the entire foundation model;
• Multiple experts can be trained independently, reducing direct interference between tasks;
• When the base model is upgraded, adapters can be evaluated and migrated rather than rebuilding all accumulated preferences from scratch.

However, these advantages do not come for free. As the number of experts grows, the router itself becomes a new bottleneck. If routing is wrong, even the best expert cannot perform effectively. Loading multiple LoRAs at the same time may also cause parameter interference, higher VRAM usage, and fluctuations in inference latency. Whether MoL can ultimately scale depends on routing quality and adapter scheduling, not merely the training results of individual LoRAs.

The Benchmarks Look Impressive, but It Is Too Early to Declare That It “Surpasses Closed Models”

According to results published by Neo Lab, Macaron-V1-Preview outperforms the GLM-5.1 base model across multiple benchmarks and leads models such as GPT 5.4 and Claude Opus 4.6 on some tasks. Outperforming the base model is significant: at the very least, it shows that external LoRAs are not merely being used for stylistic customization, but may deliver measurable capability gains.

That said, cross-model rankings should still be treated with caution.

Results can be significantly affected by whether different models use the same prompts, reasoning budgets, tool environments, and sampling parameters. A model that has undergone task-specific post-training may outperform general-purpose closed models on the relevant benchmark, but that does not mean its overall capabilities, stability, and safety are comprehensively superior. For continual learning systems in particular, a single leaderboard score is not the most important metric.

What truly matters is a curve over time: after the 10th or 100th update, does the model’s capability on new tasks continue to improve while its performance on old tasks remains stable? Can it adapt more quickly when the distribution shifts? After a user’s data is deleted, can the related capabilities truly be removed?

In other words, a static model can prove itself through a single exam, while a continual learning model must prove itself through a long-term operational record.

Neo Lab disclosed that Macaron-V1-Preview reached $10 million in ARR two weeks after commercialization began. This figure demonstrates market demand for personalized models and post-training services, but ARR is an annualized run rate based on current subscriptions or contract value. It does not equal confirmed annual revenue, much less serve as a substitute for retention or gross margin. Whether continual learning can become a viable business depends on whether the model genuinely gets better after customers have used it for several months, rather than merely benefiting from the novelty of the deployment phase.

Three Walls Remain: Forgetting, Loss of Plasticity, and Early Bias

Continual learning is not a newly discovered problem. The reason it has failed to become mainstream for so many years is precisely that three longstanding challenges remain unresolved.

1. Catastrophic Forgetting

After learning a new task, a model’s capabilities on old tasks may decline rapidly. Training a model to become better at mathematics may not directly cause it to forget how to code, but parameter drift caused by repeated updates can indeed damage existing capabilities. LoRA reduces this risk by isolating parameters, but it does not eliminate it entirely, especially when multiple adapters are combined and may still interfere with one another.

2. Loss of Plasticity

After many rounds of training, a model may find it increasingly difficult to learn new things. Parameter structures formed during early updates can constrain the room available for subsequent optimization. This may manifest as training loss no longer declining effectively or as the need for more aggressive learning rates to achieve further improvement. Continual learning is not an unlimited refill; the model itself can also become “set in its ways.”

3. Primacy Bias

The data a system encounters earliest may exert excessive influence over its subsequent strategy. For a personal model, this means choices a user casually makes early on may be misinterpreted as long-term preferences. For an enterprise model, the usage patterns of a small number of employees during the pilot phase may define the default behavior of the entire organization.

Some recent research has begun drawing on human fast and slow memory mechanisms: short-term experience is first written into rapidly changing “fast weights” or external memory, then gradually transferred into stable “slow weights” after filtering, replay, and verification. MoL can play part of this role—user-level LoRAs handle rapid adaptation, while general experts and the base model retain long-term capabilities—but determining what is worth preserving remains a core unresolved problem.

What Developers Truly Need Is a Model Operations System

If continual learning enters production environments, MLOps will change along with it. In the past, model versions might have been updated once every six months. In the future, new adapter versions may be generated every day or even every hour. What each user sees as “the same model” will actually be a combination of the base model, expert LoRAs, personal LoRAs, prompts, memory, and tool permissions.

This requires development teams to establish at least the following capabilities:

• Data lineage: It must be possible to trace and delete the interaction data used to train each adapter;
• Capability regression: Testing must cover not only average scores, but also key users, long-tail tasks, and safety boundaries;
• Adapter registry: LoRA versions, base-model dependencies, training configurations, and compatibility relationships must be recorded;
• Routing observability: Teams must know which experts were invoked for each request and the routing confidence;
• Incremental release: The system must support shadow testing, canary traffic, and automatic rollback;
• Tenant isolation: Enterprise data, gradients, and adapters must not leak across organizations;
• Anti-poisoning mechanisms: The system must detect anomalous feedback, coordinated manipulation, and reward hacking;
• Forgetting and compliance: After a user withdraws authorization, the system must be able to locate and remove the corresponding training impact.

The final point is especially difficult. Deleting a row from a database is easy; deleting a specific piece of information from already trained parameters is not. Modular LoRAs make the problem somewhat more manageable: if the data entered only a particular user adapter, the entire adapter can simply be deleted. Once the experience has been merged into a shared expert or even the base model, however, the cost of machine unlearning rises rapidly.

Assessment: This Approach Has More Potential Than “Building Yet Another Large Model,” but It Is Far from the End State

Neo Lab deserves attention not because it has produced another model that leads on several leaderboards, but because it has chosen a problem that is more relevant to the next stage of product competition: as foundation model capabilities become increasingly commoditized, who can turn real-world usage experience into model capabilities faster and more safely?

MoL is currently a relatively pragmatic implementation. It uses a frozen base model to preserve general capabilities, small-scale expert parameters to absorb incremental experience, and dynamic routing to control the division of labor between tasks. It also leaves engineering interfaces for rollback, isolation, and personalization. Compared with continuously piling on prompts, infinitely expanding context windows, or stuffing every historical record into a vector database, this approach is more likely to produce stable behavioral change.

However, it still cannot be equated directly with “self-evolving AI.” True continual learning must answer at least three questions: what signals the model should use to determine that it is performing better, which experiences are worth retaining over the long term, and how to prove that it has not learned harmful behavior after an update. The first two involve rewards and memory, while the last concerns safety and governance. Each is more complex than training a LoRA.

Over the next year, continual learning will most likely evolve from a research concept into a standard capability of Agent platforms. But most products will not allow models to update themselves entirely autonomously. Instead, they will adopt a controlled closed loop of “online collection, offline training, automated evaluation, and canary release.” It may look less futuristic than real-time self-evolution, but it is far more likely to be deployed in practice.

The first half of the large-model era was about training a model smart enough to begin with. The second half will be about enabling it to keep growing without forgetting, losing control, or leaking data. Neo Lab is betting on the second half—and this time, the direction may matter more than any single benchmark leaderboard.

References

• NVIDIA Megatron-Bridge: Open-source infrastructure for large-scale model training and format conversion. Related LoRA training capabilities are being incorporated into this type of foundational toolchain.
• ByteDance verl: An open-source training framework for reinforcement learning with large language models, useful for understanding the engineering implementation of low-cost post-training and distributed RL.