Huawei storage boosts large model inference speed by 372%

Huawei and Hubei Mobile have just delivered a performance report: in a live carrier network environment, the token throughput rate for long‑sequence large model inference increased by up to 372%.

This figure comes from the joint test results released by the two parties during MWC Shanghai on June 24. The test subjects were two mainstream large models, MiniMax M2.5 and GLM‑5.1, with sequence lengths ranging from 8K to 190K — basically covering the most common long‑context scenarios in current AI applications.

Where the Problem Lies

To understand the significance of this test, we first need to clarify why long‑sequence inference is difficult.

When performing inference, large models must maintain a data structure called a KV Cache, which stores the key‑value pairs of processed tokens to avoid redundant computation. The longer the context, the larger the KV Cache becomes. The problem is that GPUs/NPUs have limited high‑bandwidth memory (HBM), usually only a few dozen to a hundred GB. Once the KV Cache overflows the HBM, either old data is discarded (resulting in quality loss) or frequent swapping slows down performance.

This is especially fatal in the era of AI Agents. Agents need to handle extremely long code, multi‑turn conversation history, and complex document analysis, often exceeding tens or hundreds of thousands of tokens in context. Traditional solutions struggle badly in such scenarios.

Huawei’s Solution: Moving the KV Cache to Storage

Huawei’s UCM (Unified Cache Manager, inference memory data management) technology, launched in late 2025, has a straightforward idea: if HBM is insufficient, extend it with external storage.

Specifically, UCM does the following:

• Capacity Expansion: Uses OceanStor A800 storage to provide PB‑level KV Cache space, completely breaking the HBM capacity ceiling
• Tiered Management: Applies full‑lifecycle tiered scheduling to KV Cache — hot data stays in HBM, warm data goes to DRAM, cold data sinks into storage
• Cross‑Session Reuse: Allows multi‑turn conversations to reuse historical KV Cache, avoiding recomputation from scratch each time

While this approach is not new in academia, Huawei has engineered it into practical deployment in a live carrier network and achieved measurable benefits for the first time.

Test Details

The test hardware consisted of Huawei Ascend A3 super‑nodes and OceanStor A800 storage, with the vLLM‑Ascend inference framework running at the software layer. The test focused on two key metrics:

• TTFT (Time To First Token): First‑token latency, measuring response speed
• TPS (Tokens Per Second): Per‑card token output efficiency, measuring throughput

MiniMax M2.5 Scenario

| Sequence Length | TTFT Optimization | TPS Increase | |----------------|------------------|--------------| | 64K | 26%~62% | 58% | | 128K | 26%~62% | 78% |

GLM‑5.1 Scenario

| Sequence Length | TTFT Optimization | TPS Increase | |----------------|------------------|--------------| | 64K | 51%~93% | 313% | | 128K | 51%~93% | 372% |

Interestingly, the longer the sequence, the greater UCM’s advantage. For GLM‑5.1, TPS improvement was 313% at 64K, and jumped to 372% at 128K. This shows UCM truly addresses the KV Cache bottleneck in long‑sequence scenarios, rather than inflating numbers in short‑sequence cases.

Technical Architecture Breakdown

The full solution stack can be divided into three layers:

Compute Layer: Ascend A3 super‑nodes — Huawei’s latest AI inference hardware, with significant improvements in NPU computing power and interconnect bandwidth.

Framework Layer: vLLM‑Ascend, the Ascend‑adapted version of vLLM. vLLM is known for efficient memory management — its PagedAttention mechanism allows more flexible KV Cache control, naturally fitting UCM’s tiered scheduling.

Storage Layer: OceanStor A800 — Huawei’s flagship all‑flash storage with low latency and high bandwidth, able to match inference data throughput needs.

Coordination between these layers is critical. If storage latency is too high, swap‑in/out overhead will cancel out compute gains; if the framework layer doesn’t support fine‑grained cache management, UCM’s tiered scheduling cannot be fully leveraged.

Significance for the Industry

Although this test was conducted in Hubei Mobile’s environment, the validated technical approach has reference value industry‑wide.

For cloud providers and carriers: AI business for carriers is growing rapidly. Hubei Mobile’s own China Mobile Intelligent Computing Center (Wuhan) already offers 1500P compute, with plans to expand to 6800P. Inference acceleration solutions can directly improve the utilization and service capability of these computing resources.

For model vendors: MiniMax and Zhipu are top domestic large model companies. Their models performed well on this solution, indicating UCM’s general applicability to mainstream models, not just optimizations for a specific one.

For enterprise users: Long‑context capability is a hard requirement for AI deployment. Code generation needs the full project context; customer service chatbots need to remember past interactions; document analysis needs to process dozens of pages. Lower inference costs make the ROI for these scenarios viable.

Huawei’s AI Infrastructure Layout

In the bigger picture, Huawei’s AI infrastructure layout is already comprehensive:

• Chips: Ascend AI processors for both training and inference
• Compute Platforms: Atlas servers and super‑nodes
• Frameworks: MindSpore deep learning framework, plus adaptations of open‑source frameworks like vLLM‑Ascend
• Storage: OceanStor all‑flash storage optimized for AI
• Software: System‑level optimization technologies like UCM

The logic: AI inference bottlenecks keep shifting — from pure compute power, to memory bandwidth, and now to storage I/O. Chips alone, or storage alone, are not enough. Only by optimizing the entire data path can full performance be unlocked.

Comparison with Other Approaches

Current solutions for long‑sequence inference issues generally include:

1. Model‑level: Reduce KV Cache size with techniques like sparse attention or sliding windows, e.g., Mistral’s Sliding Window Attention. Pros: no hardware changes required; Cons: potential loss in model capability.

2. Algorithm‑level: KV Cache compression, quantization, selective dropping — also model‑side trade‑offs with similar pros/cons.

3. System‑level: Huawei’s demonstrated approach — using larger‑capacity storage for KV Cache. Pros: transparent to the model, no output quality loss; Cons: requires joint optimization of storage and compute, with higher technical barriers.

4. Distributed Inference: Splitting a long sequence across multiple cards for parallel processing — suitable for ultra‑long sequences, but with high communication overheads and costs.

Huawei’s solution falls into category three, following a “hardware + system” co‑optimization path. This route is heavier but has a higher ceiling — model‑level optimizations inevitably hit quality limits, while theoretically storage expansion can scale infinitely.

Remaining Challenges for Deployment

Naturally, there’s a gap between testing and large‑scale commercial rollout.

Cost: The OceanStor A800 is enterprise‑grade all‑flash storage and not cheap. For small‑ to medium‑scale inference scenarios, ROI needs careful calculation.

Generality: This test was within the Ascend ecosystem; users of NVIDIA GPUs will have to wait for similar adaptations.

Operational Complexity: Introducing external storage complicates system architecture, increases points of failure, and demands higher operational skill sets.

These issues are more commercial and ecosystem‑related than technical. Huawei’s choice to collaborate with a carrier for the first live‑network test is also about paving the way for scaled deployment.

Carriers’ AI Ambitions

Hubei Mobile’s role extended beyond providing the test environment. Judging from their actions in the past two years, carriers are investing aggressively in AI.

In January 2024, the China Mobile Intelligent Computing Center (Wuhan) began operations, covering Hubei, Hunan, and Jiangxi, as China Mobile’s first operational regional center node. By the end of 2024, capacity expansion to 6800P is planned, making it the largest intelligent computing center in Central China.

In November 2024, Hubei Mobile inaugurated the Yangtze River Research Institute and China Mobile Large Model Industry Innovation Base (Hubei), with research directions including AI, 5G‑A/6G, BeiDou and satellite internet, and the low‑altitude economy.

Carriers have natural advantages in AI: infrastructure, network coverage, and government/enterprise client resources. What they lack are software and algorithm strengths — exactly where cooperation with Huawei is complementary.

Final Thoughts

The 372% boost is an impressive number, but more noteworthy is the underlying tech trend: AI inference is shifting from “compute‑driven” to “system‑driven.”

The era of simply stacking GPUs is ending. As model scale and context length continue to grow, memory walls and storage walls will become bottlenecks in turn. Players who can coordinate compute, memory, storage, and networking will have an edge in inference efficiency.

Huawei’s showcased solution is an effective validation in this direction. Whether it becomes an industry‑standard approach will depend on future cost optimization and ecosystem adoption.

---

References

• ITHome: Huawei and Hubei Mobile Complete China’s First Live Network Test of an AI Inference Acceleration Solution for Carriers — Original news report with complete test data