<think>Translating content with markdown preservation</think>

SmoothRL Enables Robots to Learn While They Act

Astribot’s Foundation Model Team recently released SmoothRL, an online reinforcement learning framework whose full name is Online Reinforcement Learning During Asynchronous Execution. It is not designed to solve the problem of whether a robot “knows how to perform” a task, but rather how, after the robot has already learned to perform it, to make its actions more accurate and stable during real-world execution—without having to stop and wait for a large model to finish inference.

At first glance, this appears to be a training framework problem. In practice, it corresponds to a critical bottleneck in the deployment of embodied intelligence: the inference speed of large models is becoming a new bottleneck in the robot control loop.

In a traditional online reinforcement learning process, the robot executes an action, waits for feedback from the environment, then has the policy model calculate the next step, after which the model is updated. This process is suitable for discrete, low-frequency tasks that can be paused, but not for continuous dynamic operations such as throwing, grasping, swinging, and obstacle avoidance. Once a robot begins moving, its actions often cannot be paused simply because the model has not finished computing.

The core idea behind SmoothRL is to split robot execution and policy learning into two processes that can advance in parallel: the robot continues moving according to the current policy, while the model asynchronously receives execution trajectories and environmental feedback and updates the policy. In other words, the training process no longer requires every action to be strictly queued in sequence; instead, the robot is allowed to act while the model learns.

Why Robots Cannot Stop and Wait for the Model

Over the past few years, robot foundation models have primarily addressed the problem of “capability coverage.” By expanding data scale, increasing model parameter counts, and introducing vision-language-action models, robots can understand more instructions and perform basic operations involving a wider range of objects and scenarios.

After real-world deployment, however, the most common problems are often not complete failures, but near misses:

• The grasping point is off by a few millimeters, causing the object to slip from the gripper;
• The release is half a beat too late, completely changing the throwing trajectory;
• The target position shifts by a few centimeters, causing a previously stable action to begin shaking;
• The robotic arm suddenly pauses during continuous motion, interrupting the entire action sequence;
• The model generates a better policy, but its inference latency exceeds the control cycle, preventing the policy from being deployed in time.

For robots, time is not merely an ordinary input variable—it is part of the action itself. Executing an action at 100 milliseconds versus 150 milliseconds may already produce two different outcomes. This is especially true in throwing, rapid grasping, and dynamic-contact tasks, where the control system faces a continuously changing environment. Waiting means missing the window of opportunity.

Asynchronous inference for large models is becoming the norm in real-world deployment. A policy model may gradually output actions in a generative manner, while the vision module, planning module, and low-level controller may each operate at different refresh rates. If reinforcement learning continues to follow a strictly synchronous sample–evaluate–update process, an awkward situation arises: the model becomes increasingly capable, but the training loop actually becomes slower.

SmoothRL’s value lies in adapting the update mechanism of online reinforcement learning to this kind of asynchronous execution environment.

What Exactly Does SmoothRL Change?

The basic logic of online reinforcement learning is not complicated: the policy produces an action, the environment returns a result, the system evaluates the action based on its reward, and the policy is then adjusted. But when policy generation, robot execution, and parameter updates are no longer synchronized, the problem quickly becomes more complex.

First, the action currently being executed by the robot may have been generated by an older version of the policy. By the time the trajectory enters the training queue, the model parameters may already have been updated several times. If the old trajectory is used directly for training, the policy will encounter significant off-policy bias. What the trainer learns may not reflect the behavior that the current policy would actually take.

Second, rewards in dynamic tasks are usually delayed. In throwing, for example, the robotic arm’s acceleration, angle control, and release action may occur continuously over tens to hundreds of milliseconds, while the final reward often cannot be determined until the object lands in the target area. How to assign the final result to the preceding sequence of actions is one of the central challenges of online learning.

Third, data arrival rates are unstable in asynchronous systems. Visual perception, action execution, and model inference each have different latencies, requiring the trainer to handle trajectories that arrive out of order, are delayed, or have become stale. If the system prioritizes data freshness too heavily, throughput will decline; if it allows too much old data into training, stability will suffer.

Based on publicly available information, SmoothRL was designed around an online reinforcement learning process for asynchronous execution. It is not simply a matter of converting the training thread into a multithreaded system. Rather, it redefines which executed actions can enter the learning process, when they should enter it, and how to handle the discrepancy between the execution policy and the policy currently being trained.

This distinction is important. Ordinary asynchronous systems generally prioritize throughput: as long as data continues to flow in, the GPU should not remain idle. Robot training, however, cannot be judged solely by how many trajectories are generated per second. It must also consider whether those trajectories are consistent with the current policy, whether rewards can be correctly attributed, and whether the updated policy will undermine capabilities the robot already possesses.

First Validation in a Highly Dynamic Throwing Task

The focus of SmoothRL’s public validation is a highly dynamic throwing task.

Throwing is a classic scenario for testing a robot’s online learning capabilities. It combines three characteristics: continuous acceleration, precise release, and the inability to pause. The robotic arm must continuously adjust its state while moving, and the release moment is highly sensitive to velocity, angle, and position. Once the timing is missed, there is almost no opportunity for correction.

With a traditional synchronous process, the robot may need to stop after each round of action and wait for the model to complete evaluation and updating. With fully offline training, meanwhile, it is difficult for the model to cover real-world mechanical errors, object variations, and environmental changes. SmoothRL attempts to combine the two: allowing the robot to execute continuously while correcting the policy based on the results of real-world execution.

This indicates that online reinforcement learning is moving from “high-precision fine-tuning” toward more dynamic manipulation tasks. In the past, online learning was more commonly used to compensate for robotic-arm positioning errors, correct grasping deviations, or adapt a policy to a fixed environment. A throwing task, by contrast, requires the system to learn under continuous time and irreversible actions, making it substantially more difficult.

Of course, the publicly available information currently focuses primarily on the framework’s underlying ideas and task validation. It has not yet disclosed sufficiently complete details on training scale, hardware configuration, sample efficiency, success-rate curves, or detailed comparisons with synchronous online reinforcement learning baselines. Therefore, SmoothRL is currently better viewed as an important validation of a systems approach rather than as a framework that has already demonstrated universal advantages across all robot tasks.

It Is Not the Same as Asynchronous Training for Large Models

The industry has recently also been discussing asynchronous reinforcement learning for large language models—for example, decoupling rollout generation from model training so that a trajectory can enter the update process as soon as it is complete, rather than waiting for an entire batch of samples to be generated. These methods primarily address GPU idling, batch-processing efficiency, and long-horizon agent trajectories in language-model training.

SmoothRL deals with a different layer of asynchrony. Its goal is not simply to make a training cluster run faster, but to adapt policy training to physical execution in the real world, where actions cannot be paused.

The two share similar systems-level ideas: both emphasize decoupling data production from data consumption, and both need to handle off-policy bias and trajectory freshness. But robotic environments impose an additional hard constraint: actions must be completed in real time, and control cycles, sensor latency, actuator inertia, and safety boundaries cannot be ignored.

If a language model takes several dozen milliseconds longer to generate a token, the result is usually just a slower response. If a robotic arm is delayed by several dozen milliseconds while moving at high speed, it may collide with equipment or miss the release window entirely. For robots, asynchronous training is not an engineering optimization; it is a prerequisite for closing the loop.

This is also what makes SmoothRL worth watching: it advances reinforcement learning from “the model waits for feedback from the environment” to “the model and the environment move forward simultaneously.” For future embodied intelligence systems, this decoupling of training and inference may become infrastructure rather than a technique for a specific task.

Pretraining Solves “Knowing How to Do It”; Online Learning Solves “Doing It Reliably”

From a technical perspective, SmoothRL represents the next stage of post-training for embodied models.

The pretraining stage typically relies on large-scale demonstration data, video data, or offline trajectories to teach the model the basic structure of actions. For example, the model learns how to approach a cup, close the gripper, and move an object to a target position. This stage prioritizes breadth of capability and generalization.

Once the model enters a real environment, the problem becomes policy refinement: What is the current friction coefficient of the robotic arm? How do the weight and material of objects vary between batches? Does the camera have latency? When the target position changes, how should the original action be adjusted? It is difficult to cover all this information through pretraining data alone; it must be obtained through real-world execution.

Online reinforcement learning allows the model to continuously correct itself using this feedback. It is more like a driver continuing to adjust the steering wheel after becoming familiar with a vehicle than relearning how to drive from scratch. For a pretrained policy that already has basic capabilities, the goal of online RL is not to start over, but to reduce errors, improve stability, and adapt to a constantly changing environment.

There is also a risk here: online updates may damage existing capabilities. In pursuit of short-term rewards for a particular task, a robot may learn a set of actions that work in a specific environment but fail when transferred to another. Therefore, whether SmoothRL can support broader policy updates and a wider range of task distributions, as well as how it balances exploration and safety, will determine whether it can move from experimental tasks to large-scale deployment.

SmoothRL’s Real Significance: Making Asynchrony the Default Architecture

The robotics industry has traditionally discussed models, control, and hardware separately. Model teams focus on policy performance, control teams on real-time responsiveness, and hardware teams on execution accuracy. But as large models enter robotics, the boundaries among the three are disappearing.

A larger vision-language-action model may provide better generalization, but it may also introduce longer inference latency. A more complex generative policy may improve action flexibility, while making real-time control more difficult. More frequent online updates may allow a robot to adapt to its environment faster, but may also cause policy oscillation and safety risks.

SmoothRL’s direction essentially acknowledges these contradictions at the systems level and coordinates them through an asynchronous architecture:

1. Execution does not wait for training: The robot continuously completes actions according to the currently available policy, reducing pauses caused by model updates.
2. Training does not block inference: Policy updates run in the background, preventing the training process from occupying the real-time inference path.
3. Data enters learning with temporal information: The system needs to know which policy version generated a trajectory and assess how current that trajectory remains.
4. Rewards must be adapted to continuous actions: The system cannot score only the final outcome; it must also handle credit assignment across long action sequences.
5. Updates must comply with safety boundaries: Online learning cannot come at the expense of equipment safety or basic capabilities.

If these mechanisms can be further standardized, future robot systems may develop continuously operating online learning loops similar to those in modern recommendation systems: devices work in the field, data flows back in real time, policies are continuously iterated, and updates are gradually deployed through version control and safety evaluation.

That would be closer to industrial value than simply improving a benchmark success rate.

What to Watch Next

Astribot’s team said it will next explore broader policy updates, a wider range of task distributions, and the integration of asynchronous execution with end-to-end optimization of generative policies.

These three directions correspond to three key questions.

First, broader policy updates mean that the system would not merely correct a few millimeters of error at the end of an action, but could potentially adjust the entire action policy. However, the broader the update, the greater the risk of damaging existing capabilities, making stability constraints a key focus.

Second, a wider range of task distributions means that SmoothRL will need to expand from a single throwing task to grasping, organizing, tool use, and multistep operations. The more complex the task, the more difficult reward design and trajectory attribution become, while the data time lags introduced by asynchrony also become more pronounced.

Third, end-to-end optimization of generative policies could further integrate high-level decision-making, action generation, and low-level control. But it would also concentrate issues involving inference latency, control frequency, and training stability into the same pipeline.

Therefore, the most important thing to watch about SmoothRL at present is not the emergence of a new term, but an architectural judgment: once robots begin using large models for continuous decision-making, online reinforcement learning must adapt to asynchronous inference and asynchronous execution. Conversely, robot training systems can no longer assume that the environment will patiently wait for the model to finish computing.

For developers working on embodied intelligence, robot foundation models, and Agent RL, the central question has shifted from “How do we make a model learn an action?” to “How do we make a model continue learning while actions are constantly taking place?” SmoothRL provides at least one clear answer: run execution, sampling, inference, and updating in parallel, then use algorithms to address the resulting issues of timeliness and stability.

This approach still requires validation through more publicly available data before it can achieve large-scale commercial deployment, but the direction is sound. Future robots will not be products that run in a fixed manner after training is complete. They will be more like systems capable of continuously absorbing experience under safety constraints. Whoever can make this closed loop fast and stable enough will have a better chance of bringing embodied intelligence from demonstration videos into real production environments.

References

• GitHub SmoothRL project search page: Used to track whether SmoothRL code, paper implementations, or subsequent engineering materials have been made public. As of the time of publication, publicly available reports had not provided a confirmed official repository URL.
• Hugging Face SmoothRL search page: Used to track whether related models, datasets, and experimental weights have been publicly released.
• Factual basis of this article: Public reports released on September 4, 2026, and information published by Astribot’s Foundation Model Team, with a focus on SmoothRL’s asynchronous online reinforcement learning design and validation in a highly dynamic throwing task.

Note: OpenAI Hub supports OpenAI-compatible APIs for leading models including GPT, Claude, Gemini, and DeepSeek. SmoothRL is an update to a robotics reinforcement learning framework; this article does not provide API usage examples.