AReaL 2.0 Open Source: Equipping Agents with an Online Learning Feedback Loop

On July 2, the open‑source reinforcement learning infrastructure AReaL officially released version 2.0. This upgrade extends the scope from “how to efficiently train an Agent” to “how can a running Agent continue to learn.”—it sounds like just a one‑word difference, but the engineering complexity is orders of magnitude apart.

Readers familiar with the project may recall that AReaL v1.0 was released in March this year, featuring large‑scale asynchronous RL training and one‑click Agent integration. Version 2.0 clearly shifts its focus: it no longer obsesses over training speed, but instead faces a real and awkward production issue—Agents work every day but don’t actually learn anything.

Agents freeze after going online—this is a real problem

Over the past year, Agents have truly entered production. Enterprises use them to write code, gather information, control tools, and run workflows—all increasingly complex tasks. Yet an ironic phenomenon persists: these Agents generate massive amounts of interaction data daily—failed tool calls, user intent changes, misaligned planning steps—but most of these experiences are simply logged and discarded.

Traditionally, improving an Agent involves: manually constructing training data → running offline RL → redeploying online. This process is complex and slow, and there’s always a gap between offline simulation and real business operations. When the codebase updates, workflows change, or internal APIs shift, the Agent still perceives the world as it did during its last training session.

The AReaL team’s technical report is quite direct: the bottleneck of self‑evolving Agents lies not solely in the model or the RL algorithm, but in the absence of an online RL infrastructure that serves real Agents. I mostly agree with this conclusion—many open‑source RL frameworks exist, but most are built for research benchmarks and assume conditions far removed from production reality.

Core concept: turn RL infrastructure into microservices

AReaL 2.0 can be summed up in one sentence: restructure the compute units that serve rollout and training into deployable, pluggable, and replaceable Agent‑compute microservice components.

In plain terms—you don’t need to rewrite your Agent or rebuild your business system. Simply redirect the Agent’s inference requests from the model endpoint to AReaL 2.0’s managed inference gateway; it then takes over the backend process—logging dialog interactions, collecting trajectories, binding rewards, and orchestrating asynchronous training.

This design choice is smart. Many Agentic RL approaches either require developers to simulate the online Agent as an offline training task or to rewrite the Agent loop according to the training framework’s specification. The former often yields models that underperform in real environments, and the latter demands so much migration effort that most developers give up. AReaL 2.0’s choice to unify at the inference layer preserves existing planning, tool‑calling, sandbox, and memory modules of the Agent—an approach that feels right from an engineering perspective.

Three pillars: ATDP, Data Proxy, Evolution Control Plane

According to the technical report, AReaL 2.0 abstracts the self‑evolving Agent system into three modules:

• Agent Trajectory Data Protocol (ATDP) — defines what should be recorded. Multi‑turn dialogs, tool calls, execution results, and reward signals must all be uniformly structured for downstream training.
• Agentic Data Proxy — handles how to record in real production systems. This layer focuses on access control, data masking, isolation, and auditing—since enterprise Agents may touch codebases, customer data, or internal systems, training data flow must be strictly controlled.
• Agent Evolution Control Plane — manages online policy updates and overall orchestration.

Of these, I believe the Data Proxy is the most commercially valuable yet easily overlooked component. Academic Agentic RL research often assumes unrestricted data flow, but in enterprise reality, deployment feasibility depends less on algorithms and more on data compliance policies. Building this layer into the architecture from the start makes AReaL 2.0 more pragmatic than most counterparts.

The team acknowledges that AReaL 2.0 does not yet provide a full foundation for self‑evolving Agents but instead focuses on online policy model updates based on real deployment trajectories. This boundary choice gives 2.0 a clear identity.

From Hermes to Claude Code: two reusable examples

Architecture alone is abstract; AReaL 2.0 provides two runnable examples.

The Hermes Agent example demonstrates integration. Hermes continues handling tasks, planning, and model inference as usual, while AReaL 2.0 records key interaction steps in the background. It combines end‑of‑task feedback signals with real trajectories for subsequent training. Developers can replace Hermes with their own Agent and task environment—the integration pattern remains the same.

The Claude Code Agent RL example serves as a near end‑to‑end reference for software engineering Agents, covering data processing, agent infrastructure, and algorithmic training pipelines. For teams building coding Agents, this saves a substantial amount of engineering work.

The sample code is available in the GitHub repository’s examples/ directory:

• examples/hermes — Hermes Agent online RL integration example
• examples/swe — Claude Code direction software‑engineering Agent training example

Reusability is the most tangible highlight of version 2.0. Teams working on Agentic RL often reinvent similar wheels—data collection, trajectory storage, reward alignment, asynchronous training—all tasks that must be repeated. By open‑sourcing these as microservices, AReaL 2.0 helps reduce redundant engineering efforts across the industry.

Project ownership and ecosystem

The AReaL project was initiated in 2024 by teams from Ant Group, Tsinghua University, and Hong Kong University of Science and Technology, with core researchers including former OpenAI scientists and the Wing Wu team from Tsinghua’s Institute for Interdisciplinary Information Sciences. In May, AReaL spun off from Ant’s InclusionAI incubator to become an independent open‑source community and joined the PyTorch Foundation Ecosystem project.

Since then, partners such as Huawei Cloud and MindLab have joined. The roadmap lists three directions: deeper exploration of online RL, automated evaluation, and multimodal Agent training—the third is expected to be the focus of version 3.0.

What this means for developers

If you’re building production‑level Agent applications, AReaL 2.0 offers at least two benefits:

1. A ready‑to‑use online RL infrastructure—no need to build rollout workers, asynchronous training schedulers, or trajectory managers from scratch.
2. An engineering paradigm for converting real online data into model capabilities—previously either proprietary to big tech companies or scattered across research codebases, never forming a complete workable system.

Of course, it’s not a silver bullet. Running online RL still requires computational resources, so smaller teams must calculate the costs carefully. The training outcome also largely depends on how well reward signals are designed—AReaL 2.0 doesn’t help much here; that’s up to the business implementer.

The underlying model choice is open. AReaL 2.0 trains policy models, so developers can select any base model they wish to optimize. If you simply want to test your Agent’s reasoning pipeline without base model training, you can use aggregation platforms like OpenAI Hub to benchmark models such as GPT, Claude, Gemini, or DeepSeek first, then pick one for RL fine‑tuning—a practical approach.

A brief assessment

For the past two years, the Agent field has had an unspoken awkward truth—everyone talks about Agents that self‑evolve and improve with use, but once deployed, most remain static. AReaL 2.0 may not fix this instantly, yet it shifts the conversation from theory to engineering reality: continual Agent learning is not a concept, but a concrete system problem involving inference gateways, trajectory protocols, data proxies, and asynchronous training.

For China’s open‑source community, delivering an RL infrastructure project that joins the PyTorch Foundation Ecosystem is rare. What’s worth watching next is whether the community can build an ecosystem around the ATDP protocol and attract more Agent frameworks to connect—only then will “self‑evolving Agents” have a real engineering foundation.

References

• AReaL Project GitHub Repository — Project homepage containing v2.0 code, technical report, and Hermes & SWE integration examples
• “Making Agents Stronger Through Use: AReaL 2.0 Gives Smart Agents a Growth System” (Juejin) — Detailed Chinese technical community analysis covering design principles and enterprise adaptation