Domestic Computing Power Successfully Completes Full Training of 1.6 Trillion Parameters

Shenzhen Hetao Academy, Harbin Institute of Technology (Shenzhen), Shenzhen Big Data Research Institute, together with relevant Huawei teams, completed full parameter post-training of DeepSeek-V4-Pro (1.6 trillion parameters) on an Ascend 910C cluster. This marks the first time a third-party organization has successfully run a complete training process for a model of this scale on a domestic computing power platform.

The significance of this is not about training yet another large model — it is about proving that domestic computing power can withstand the full-chain pressure of training trillion-parameter models. Previously, domestic AI chips were mainly used for inference or small-parameter fine-tuning, and true deep training — especially the post-training phase — relied almost entirely on Nvidia GPUs. This successful run means that the entire closed loop from data input to model optimization output can now be handled by domestic computing power.

Post-training is harder than pre-training

Many people think pre-training (training a model from scratch) is the most costly and difficult. In reality, post-training places more demanding requirements on hardware.

Pre-training is like feeding knowledge into a model — data in, gradients out, parameters updated — the process is relatively fixed. Post-training, however, uses supervised fine-tuning (SFT) and reinforcement learning (RLHF) to make the model “obedient” and “aligned with human preferences,” involving large amounts of dynamic computation:

• More complex memory usage: RLHF needs multiple model copies loaded simultaneously (policy model, value model, reference model), resulting in more than double the memory pressure compared to pre-training
• More irregular communication patterns: Expert routing in a MoE architecture triggers All-to-All communication, with traffic dozens of times higher than dense models; if any card lags, the whole cluster stalls
• More dynamic computation graphs: During RL rollouts, the computation path may change at every step, rendering static optimizations ineffective

This is why DeepSeek-V4-Pro could previously run inference on domestic computing power, but nobody had publicly completed full-parameter post-training — not because of insufficient computing power, but because of the engineering difficulty.

How the thousand-card cluster held up

This training used a cluster of over one thousand Ascend 910C cards. Throughout more than 1,500 training steps, there was not a single interruption, model FLOP utilization (MFU) exceeded 30%, and key operator efficiency improved by 14%. Behind these numbers are three layers of optimization:

1. Distributed Load Strategy

A 1.6 trillion parameter model cannot fit on a single card, so the team designed a detailed tensor partitioning scheme. DeepSeek-V4-Pro uses MoE architecture; while active parameters are only 37 billion, the total parameters remain large, requiring separation of expert weights and attention matrices at each layer.

Traditional data-parallel + model-parallel approaches struggle with MoE because expert routing is dynamic — one step may activate experts 1, 3, 7, the next step 2, 5, 9; moving parameters repeatedly between cards consumes all bandwidth. The team’s approach:

• Static expert allocation: Assign 64 experts to fixed card groups to reduce cross-node communication
• Pipeline optimization: Stagger forward pass and backward gradient computation to avoid mutual waiting between communication and computation
• Memory reuse: Recalculate certain intermediate activations instead of storing them, trading compute for memory

This kept utilization of each 910C (64GB memory) steady at over 85%, with no waste or memory overflow.

2. Load Balancing

MoE models are vulnerable to “uneven expert load” — some experts get activated frequently, maxing out the corresponding cards, while others idle. DeepSeek-V4-Pro’s routing policy includes load balancing, but skew still occurs in practice.

At the framework level, the team implemented:

• Dynamic load monitoring: Real-time tracking of each expert’s activation frequency; if a card’s utilization stays 20% above average, trigger expert migration
• Communication topology optimization: Using Ascend 910C’s HCCS (Huawei Cache Coherence System) interconnect, the team remapped experts to cards based on physical topology, placing high-frequency communication experts in the same node where possible

Result: All-to-All communication latency dropped from a peak of 120ms to a stable 40ms, virtually eliminating communication bottlenecks.

3. Fault Tolerance and Observability

With thousands of cards computing in parallel over 1,500 steps, hardware failures, network jitter, and process crashes are statistically inevitable. Traditional practice is periodic checkpointing — if an issue occurs, roll back — but checkpointing a trillion-parameter model takes over 10 minutes, and rollbacks incur unacceptable time costs.

The team built a layered fault-tolerance system:

• Card-level redundancy: Reserve 3% spare cards at startup; if a card fails, immediately replace with a spare and rapidly catch up by recomputing recent gradients
• Gradient verification: After each batch, compute a hash of global gradients; if NaN or Inf appears, trigger rollback immediately to prevent contaminated data from polluting the model
• Real-time monitoring dashboards: Visualize temperature, power, utilization, and communication bandwidth for each card, with early warnings for anomalies

In the end, there was not a single full rollback; only 7 card-level hot swaps occurred, amounting to effectively fault-free operation.

What 30% MFU means

Model FLOP utilization (MFU) measures training efficiency as the proportion of actual effective computation to theoretical peak hardware performance. Achieving 30% on a trillion-parameter MoE model is already industrial-grade.

For comparison:

• Meta’s training of Llama 3 (dense model, 405B parameters) had MFU in the 40-50% range
• Mistral’s training of Mixtral 8x22B (MoE, 176B parameters) achieved MFU between 25-35%
• DeepSeek’s official training of V3 (671B-parameter MoE) reached MFU of 35-40%

MoE MFU is inherently lower than dense models due to high communication overhead and dynamic computation paths. V4-Pro, at 1.6T parameters, is over twice as large as V3; achieving 30% shows communication optimization and scheduling strategies are on point.

The 14% key operator efficiency gain came mainly from operator fusion and mixed precision optimization. The Ascend 910C’s Da Vinci architecture is specially optimized for matrix operations; the team fused multiple small operators (LayerNorm, GELU, Dropout) into a single large operator, reducing memory read/write counts. They also lowered some computation from FP32 to BF16 without losing accuracy, doubling throughput directly.

The true position of domestic computing power

Completing full-parameter post-training doesn’t mean domestic computing power has caught up to Nvidia, but it proves the technical path works.

What it can do:

• Handle entire process of inference, fine-tuning, post-training for trillion-parameter models
• Achieve industrial-grade training efficiency in MoE architectures
• Support stable operation of thousand-card clusters

What is still lacking:

• Ecosystem toolchain: Nvidia has CUDA, cuDNN, Megatron-LM — a mature toolset; Ascend’s CANN framework and MindSpore are still rapidly iterating, with many optimizations requiring manual tuning
• Pre-training proof: Post-training has been demonstrated; but starting from scratch (tens of millions of GPU hours) lacks a public case — this remains the real challenge
• Single-card performance: Ascend 910C FP16 performance is 320 TFLOPS vs. H100’s 1000 TFLOPS, with an absolute gap filled only via cluster size and software optimization

For large-model training, single-card performance is never the sole bottleneck. Meta’s Llama 3 training used H100s but MFU was only a bit over 40%, with the remainder lost to communication, scheduling, and fault tolerance. Domestic computing power can close this gap through continued system engineering optimization.

Industry impact

The most direct impact of this breakthrough is breaking the perception ceiling that “domestic computing power can only do inference.” Many large-model teams previously avoided domestic computing not out of patriotism issues, but because of high technical risk — what if training stalls halfway? What if model quality is subpar?

Now, with a third-party public verification, at least for post-training, domestic computing power is a viable option. Especially for teams with pre-trained models needing vertical domain adaptation, an Ascend 910C cluster can fully handle the work.

For Huawei, this is a key battle for the Ascend ecosystem. AI chip competition is not about the chip itself, but about running real production workloads. This joint task force was not internal to Huawei — it involved universities, research institutes, and industry partners — showing that Ascend’s toolchain and ecosystem support is mature enough for external teams to complete complex tasks independently.

Looking long-term, this success marks domestic AI infrastructure moving from “usable” to “easy to use.” Pre-training remains the next major hurdle, but the direction is clear: not to build an exact Nvidia replica, but to follow a path leveraging system-level optimization, hardware-software synergy, and engineering innovation tailored to domestic computing power’s characteristics.

For developers, this means there is now another truly competitive option for training platforms. If your model is between tens of billions and a trillion parameters, needs post-training or vertical fine-tuning, and you have a tight budget, an Ascend 910C cluster is a serious candidate. While the toolchain does have a learning curve, you no longer have to fear “halfway through training you discover it doesn’t work.”

What to watch next

This success is a first step. Next, focus on three things:

Can pre-training be run?: Post-training’s compute load is only a fraction of pre-training; the real test is starting from zero on a trillion-parameter model. This involves not only computing power, but also data engineering, training stability, and hyperparameter search — each step is a hard battle.

Can cost advantages be realized?: Domestic computing power is cheaper, but if training efficiency is low and debugging costly, total cost of ownership (TCO) may not be competitive. Future commercial cases must be examined for end-to-end cost competitiveness.

Can the ecosystem grow?: Most open-source model training code is written for PyTorch + CUDA, requiring adaptation to run on Ascend. If Huawei can run and open-source training scripts for mainstream models, lowering migration barriers, the ecosystem can truly flourish.

Regardless, this breakthrough proves one thing: In the large-model training race, the question for domestic computing power is no longer “can it be done?” but “how to do it better?” — a qualitative change.

References

• Domestic Computing Power Milestone: Thousand-card Ascend 910C Runs DeepSeek 1.6 Trillion Model Full Parameter Post-training - Linux China Open Source Community — Official technical details and training data disclosed by the joint team
• 1.6 Trillion Parameter Large Model Post-training, Domestic Computing Power Completes Key Step - Xinhua News — Authoritative coverage of the breakthrough by Xinhua
• Domestic Computing Power Successfully Completes Trillion-scale AI Model Full Parameter Post-training - Tencent News — Detailed analysis of the training process and technical challenges by 21st Century Business Herald