Ascend Achieves Training-Inference Consistency for RL

On August 6, Huawei Computing announced that Ascend now supports training-inference consistency in large-model reinforcement learning scenarios. In tests on the Qwen3-30B MoE model, Logdiff between the training engine and the Rollout inference engine reached 0. Combined with FA operators optimized for Ascend, the solution also delivered performance gains of 20% to 60% in Eager mode.

This is not a routine model adaptation, nor is it merely a matter of getting a particular operator to run. Reinforcement learning post-training is evolving from an experimental project within model teams into a standard production workflow for finance, coding, mathematics, agents, and other use cases. Numerical discrepancies between training and inference directly undermine the reliability of policy updates. In severe cases, the training curve may appear normal even though the actual optimization direction has already drifted off course.

What Ascend has addressed is an inconspicuous yet challenging problem in RL infrastructure: given the same weights and the same sequence of tokens, can the probabilities calculated during inference truly match those recomputed during training?

Why the Same Model Can Produce Two Different Answers

A typical large-model reinforcement learning workflow can be roughly divided into four steps:

1. The Rollout engine generates responses using the current model;
2. A reward model or rule-based system scores the responses;
3. The training engine recomputes token logprobs, KL divergence, and the advantage function;
4. Algorithms such as PPO and GRPO update the model parameters accordingly.

The problem is that the first and third steps usually do not use the same execution system.

The Rollout side prioritizes throughput and low latency, commonly using continuous batching, KV Cache, PagedAttention, or specialized inference operators. The training side, meanwhile, needs to retain activations and participate in backpropagation, and therefore typically uses FlashAttention, tensor parallelism, and fused operators provided by the training framework. Even with exactly the same weights, the two sides may produce slightly different logprobs because of differences in operator implementations, tensor partitioning, and accumulation order.

This discrepancy is generally measured using Logdiff. It may appear to be nothing more than a difference in the least significant digits of a floating-point number, but in RL training, logprob is not merely a metric for display. It is a direct input to policy ratios, KL constraints, and loss functions.

Take PPO-style algorithms as an example. During training, the probabilities assigned to the same token by the old and new policies are compared. Before the model parameters have been updated, this ratio should theoretically be close to 1. If the old probability recorded on the Rollout side inherently differs from the result recomputed on the training side, the optimizer no longer sees a clean on-policy sample. Instead, it receives a signal contaminated by infrastructure-induced error.

A single error may be small, but after being propagated through long sequences, multiple rounds of sampling, and thousands of update steps, it may no longer remain small. The larger the model and the more complex the parallelization strategy, the more likely such discrepancies are to be amplified.

The Root Cause Is Not Randomness, but Non-Order-Preserving Floating-Point Accumulation

The fundamental cause of training-inference inconsistency is that floating-point arithmetic does not strictly satisfy the associative law. Mathematically, (a+b)+c and a+(b+c) should be equal. In finite-precision FP16, BF16, or FP32 computation, however, the two paths may produce different results because rounding occurs at different points.

Attention computation contains a large number of reduction operations, while Softmax involves finding the maximum value, calculating exponentials, and performing block-wise summation. As long as training and inference use different block sizes, parallel partitions, or reduction trees, the final mantissas may differ.

The issue becomes even more complex with MoE models. In addition to tensor parallelism, they may introduce expert parallelism, data parallelism, and cross-device collective communication. Which experts a batch of tokens is routed to, the order in which results are merged across devices, and the path selected by the communication library can all alter the actual execution order.

Training-inference consistency, therefore, cannot be achieved simply by adding a deterministic switch to a single operator. It requires the framework, operators, parallelization strategy, and communication paths to jointly follow alignable execution semantics. Huawei describes this as a system-level engineering effort, and that characterization is no exaggeration.

What Exactly Did Ascend Change?

According to information disclosed by Huawei, Ascend has provided a set of alignment operators based on Ascend C that cover the entire training-inference pipeline. The core changes focus on four areas.

1. Unifying Attention Semantics

During training, the entire sequence is usually processed at once, with a causal mask limiting the context visible to each token. During inference, decoding proceeds token by token, and each step can naturally see only the content that has already been generated.

Both are mathematically forms of causal attention, but their specific implementations may not iterate over exactly the same set of valid positions. If the training-side masked Softmax and the inference-side incremental attention differ in boundary handling, padding, or the set of valid tokens, their results will diverge from the outset.

Ascend has aligned the valid computation ranges of the two paths, ensuring that training and inference use the same attention semantics rather than merely requiring them to use the same data type.

2. Fixing the Reduce Accumulation Order

The templates for training-side FA and inference-side PFA or PagedAttention can be similar, but the two sides often use different partitioning methods to accommodate different sequence lengths, cache layouts, and hardware resources.

For the same reduction dimension, one side may first sum the first half and then merge the results, while the other performs parallel reduction over multiple smaller blocks. Although the mathematical formulas are identical, the floating-point results may differ. Ascend constrains the partitioning and reduction order of critical reduction steps so that both sides accumulate values along the same numerical path.

This is the most critical step in achieving a Logdiff of 0, as well as the one most likely to hurt performance. High-performance operators generally allow the scheduler to freely choose a partitioning scheme, while deterministic computation necessarily restricts that freedom.

3. Aligning Online Softmax Blocking Strategies

Long-sequence attention does not first generate an entire massive attention matrix. Instead, it uses online Softmax to compute results incrementally while reading the data. Local maximum values and normalization factors must be maintained across blocks and merged at the end.

If the training and inference sides use different block sizes, or merge multiple blocks at different intervals, the final results may differ in their least significant digits. Ascend has unified the relevant blocking and reduction strategies, effectively ensuring that the two engines not only solve the same problem but also use the same step-by-step computational process.

4. Aligning Precision Along Critical Paths

In mixed-precision systems, not every step uses the same precision. Matrix multiplication may use lower precision to improve throughput, while sensitive steps such as Softmax accumulation are often promoted to higher precision.

If the training side converts to FP32 at a particular step while the inference side continues accumulating at lower precision, it is difficult to obtain strictly identical results even if every other part of the process is the same. Ascend has therefore unified precision-conversion strategies along critical paths such as Softmax accumulation, reducing discrepancies caused by mixed-precision implementations.

Together, these four changes address the complete pipeline—from semantics and scheduling to numerical precision—rather than applying a patch at a single point.

A Logdiff of 0 Matters More Than the Maximum 60% Gain

From a publicity perspective, a performance gain of up to 60% is clearly more eye-catching. For teams actually conducting RL training, however, a Logdiff of 0 is the more valuable result.

Performance can continue to be improved by adding accelerators, batching, and engineering optimizations. But if the policy probabilities themselves are unreliable, the assumptions underlying the training algorithm are undermined. This is especially true in workflows emphasizing true on-policy training: the more closely the Rollout behavior policy matches the reference policy on the training side, the more meaningful the policy ratio, KL estimate, and clipping mechanism become.

Of course, a Logdiff of 0 must also be understood in the context of the test criteria. It means that the two pipelines achieved numerical consistency under Huawei's disclosed Qwen3-30B MoE test configuration. It does not automatically mean that every model, sequence length, parallel scale, and operator combination is now unconditionally consistent.

Different systems may also use different Logdiff aggregation methods, comparison precision, and tolerance definitions. To verify production readiness, developers still need to test their own training configurations across:

• Different input lengths and long-tail sequences;
• Precision combinations such as BF16 and FP16;
• Changes in tensor-parallel and expert-parallel scale;
• Dynamic batching and different levels of KV Cache utilization;
• Different attention backends and operator versions;
• The stability of KL, entropy, and reward curves after multiple rounds of training.

In other words, a Logdiff of 0 is a highly significant ticket for entry, but it is not yet a universal conclusion for all production environments.

A 20% to 60% Gain Does Not Mean Overall Training Is 60% Faster

Huawei stated that Ascend-optimized FA operator enhancements and improved dispatch scheduling delivered performance gains of 20% to 60% in Eager mode. This shows that determinism and high performance are not necessarily mutually exclusive. Even while constraining the accumulation order, performance can still be recovered through hardware-aware blocking, memory access, and scheduling design.

However, the currently available information does not fully specify the hardware model, accelerator count, sequence length, batch size, parallelization strategy, comparison baseline, or whether the performance metric refers to operator latency, training throughput, or end-to-end iteration time. The figure therefore cannot be directly extrapolated to mean that all RL workloads will run 60% faster overall.

The Eager-mode qualifier is particularly important. Eager mode facilitates debugging and dynamic execution, but its optimization path differs from that of graph or compiled modes. These results are better understood as evidence that Ascend has improved operator efficiency in flexible execution scenarios; they cannot substitute for comparative testing across different execution modes.

A common situation in the industry is that a particular attention operator becomes 60% faster, but the end-to-end training workflow still includes Rollout, reward computation, parameter synchronization, and optimizer updates, diluting the final benefit. Conversely, if FA was already the primary bottleneck, the end-to-end improvement could still be substantial. Without a complete test configuration, the 20% to 60% figure should be treated as a measured range under specific conditions rather than a universal guarantee.

What This Means for Developers

This support will first benefit teams running online reinforcement learning workflows such as PPO and GRPO on Ascend. In the past, many teams treated the separation of training and inference as a purely engineering issue: as long as weight synchronization succeeded and the two sides produced roughly the same results, that was considered sufficient. As training scales continue to grow, this permissive assumption is becoming increasingly difficult to sustain.

A set of alignable foundational operators can provide at least three benefits:

• Lower troubleshooting costs. When rewards become abnormal or KL suddenly rises, teams can spend less time suspecting low-level numerical errors between training and inference;
• More credible algorithm experiments. Differences between RL recipes are more likely to come from the algorithms themselves rather than discrepancies between inference and training backends;
• Lower migration risk. When deploying the Rollout engine and training framework on different clusters, teams have a better chance of achieving stable behavioral consistency.

However, developers should not equate infrastructure support with automatically achieving true on-policy training. Weight synchronization frequency, asynchronous Rollout, sample staleness, experience replay, and sampling parameters can all introduce policy discrepancies. Operator consistency addresses the low-level numerical path, not every consistency issue in an RL system.

Ascend Is Strengthening Its Software Stack, Not Merely Competing on Compute Power

Discussions of Chinese AI hardware have often focused on peak compute performance, memory capacity, and cluster scale. Yet the parts of large-model training that are truly difficult to replace often lie in the long-accumulated details of the software stack: operator coverage, framework compatibility, communication stability, debugging tools, and—as in this case—determinism and training-inference consistency.

Reinforcement learning exposes weaknesses in the software stack more readily than pre-training. Pre-training primarily revolves around forward passes, backpropagation, and optimizers. RL additionally introduces high-throughput inference, frequent weight synchronization, dynamic sampling, reward computation, and multiple execution engines. Being able to run a model on the hardware only means the first step has been completed. The ability to run long-duration RL workloads reliably is a much closer test of production readiness.

From this perspective, the value of Ascend's update is not that it adds another item to a support list, but that it begins addressing the deeper numerical-consistency issues involved in operationalizing RL. A Logdiff of 0 on Qwen3-30B MoE at least demonstrates that this approach can be implemented on a complex mixture-of-experts model.

Our assessment is that this is an infrastructure update whose technical significance exceeds the amount of attention it has received. It will not directly cause a leap in model capabilities, nor will it automatically replace a mature RL software ecosystem, but it can eliminate a long-underestimated source of training noise. For teams that need to conduct large-scale post-training on Ascend clusters, this is more useful than merely announcing that another model has been adapted.

The next thing to watch is not a higher single-point peak, but whether Ascend can publish a more complete testing methodology and extend consistency coverage to more models, parallel scales, inference backends, and RL frameworks. Only when developers can reproduce the results reliably will a Logdiff of 0 evolve from a demonstration result into a platform capability.

References

• ITHome: Huawei Ascend Announces Support for RL Training-Inference Consistency, Achieving Performance Gains of Up to 60% in Tests — Reports on Ascend's training-inference consistency solution, Qwen3-30B MoE test results, and four core operator changes.