Huawei Cloud Open Sources KVarN: Native KV Cache Quantization Backend for vLLM

Today, Huawei Cloud open-sourced KVarN (KV Cache Quantization for vLLM on Ascend NPU) on GitHub — a KV cache quantization backend purpose-built for vLLM. The project gained 600+ stars shortly after launch, mainly because it addresses a very real pain point: during large model inference, KV cache consumes more than half of the GPU memory, while existing quantization solutions either suffer heavy accuracy loss or only support specific hardware.

What makes KVarN special is that it’s a native backend for vLLM, not an external patch-style solution. It supports four quantization precisions — int2, int4, int8, fp8 — allowing developers to freely trade off between accuracy and performance based on scenarios. On Huawei Ascend 910B NPU, int4 quantization can increase inference throughput by more than 40%, and cut memory usage by 60%.

Why KV Cache Quantization Matters So Much

Large model inference has two phases: prefill (processing the full input) and decode (generating output token by token). In the decode phase, each generated token’s Key and Value vectors are stored in the cache, and later retrieved for Attention computation. The issue is that as generation length increases, KV cache grows linearly.

For example: in the Llama-3-70B model, when processing a 32K context, model weights occupy 140GB, while KV cache consumes 80GB. For long conversations or RAG tasks, KV cache can easily become the bottleneck — either running out of memory (OOM), or being forced to reduce batch size, which directly halves throughput.

Existing quantization solutions mainly fall into two categories: general backends like quanto and HQQ, which support multiple hardware but offer mediocre performance; and NVIDIA’s FP8 KV Cache, which has great performance but only runs on Hopper architecture (H100/H200). For domestic compute chips to integrate with the vLLM ecosystem, they either have to write quantization logic from scratch or make do with community solutions.

KVarN fills this gap. As an official vLLM backend interface implementation, developers only need to add a --kv-cache-dtype parameter when launching vLLM to use it — no need to change model code or recompile.

Technical Details: How to Compress Without Losing Accuracy

The core challenge in KV cache quantization is: Attention computation is highly sensitive to numerical precision, and improper quantization can directly degrade generation quality. KVarN uses several key designs to balance accuracy and performance:

Dynamic Scaling Factor (Per-Token Scaling): Instead of applying a fixed quantization range to the entire KV cache, each token’s K/V vector gets its own scaling factor. This adapts to different token value distributions and reduces quantization error.

Mixed Precision Support: Allows Key and Value to use different quantization precisions. For example, in some tasks, Value is more sensitive to accuracy — you can use int8 for Value and int4 for Key. This saves memory without noticeably affecting results.

Hardware Native Operators: KVarN’s quantization and dequantization operations use native CANN operators on Ascend NPU, avoiding extra CPU-GPU data transfer overhead. Quantized KV cache is stored directly in HBM, and dequantized during decode for Attention computation.

Optional Calibration Strategies: Supports two calibration modes — static calibration (pre-determining quantization parameters using a small sample set) and dynamic calibration (calculating parameters in real time). Static calibration suits offline deployment, while dynamic calibration is more flexible but slightly more computationally expensive.

According to Huawei Cloud’s internal test data, int4 quantization has accuracy loss within 1% on benchmarks like GSM8K (math reasoning) and HumanEval (code generation) — basically negligible. In long text tasks (e.g., 32K-context document QA), memory usage can drop from 180GB to 72GB, directly determining whether it can run on a single card.

Part of Huawei Cloud’s Large Model Inference Stack

The open-sourcing of KVarN is not an isolated event. In the first half of this year, Huawei Cloud’s Storage Innovation Lab, together with Peking University and Nanjing University, had three papers accepted at USENIX ATC, ICML, and ACL — all focused on large model inference optimization:

• DEEPSERVE (ATC 2025): Serverless architecture large model inference platform, natively adapted for Ascend NPU, supporting large-scale concurrency and elastic scaling. It has been running stably for more than a year in Huawei Cloud’s MaaS service.

• EPIC (ICML 2025): Position-Independent Caching mechanism, breaking traditional prefix cache limitations. Previously, KV cache could only be reused for identical prefixes; now, arbitrary context fragments can be reused, boosting reuse rates 3–5x in RAG and Few-Shot scenarios.

• RaaS (ACL 2025): Inference-aware Attention sparsification algorithm. Targets explosive KV cache growth in long inference tasks (math reasoning, code generation) by identifying “key inference tokens” to selectively keep KV cache, reducing memory complexity from O(N) to O(L) (L = context length, N = generation length), while maintaining accuracy.

KVarN is a practical engineering complement to this inference stack. EPIC solves cache reuse, RaaS solves long inference memory challenges, and KVarN minimizes memory usage per inference. Combined, they can theoretically support higher concurrency and longer contexts on the same hardware.

Huawei Cloud CEO Zhou Yuefeng mentioned in a June 5 media interview that Huawei Cloud will not compete with other vendors on compute scale or total tokens consumed; the core goal is to “develop a second compute plane” — building complete AI infrastructure based on domestic compute power (Ascend NPU). Open-sourcing KVarN is part of this strategy: exposing low-level optimization capabilities so ecosystem partners and developers can achieve performance on Ascend close to, or even exceeding, NVIDIA GPUs.

What This Means for Developers

Lower Deployment Barrier: Previously, running vLLM on Ascend NPU meant using default fp16 KV cache (memory tight) or writing CANN operators for quantization (too high threshold). Now simply adding --kv-cache-dtype int4 in vLLM’s launch parameters enables quantization, as easy as NVIDIA’s FP8 KV Cache.

More Flexible Precision Choices: Different tasks have different accuracy requirements. Chatbots and content generation tolerate lower precision — int4 or even int2 is fine; code completion and math reasoning require higher precision — int8 or fp8. KVarN supports runtime precision switching without reloading the model.

Possibility of Multi-Hardware Support: While KVarN currently optimizes mainly for Ascend NPU, its architecture is general. The community can implement corresponding backends for other domestic chips (like Hygon DCU, Cambricon MLU) using the same interface. This is good news for interoperability in domestic compute ecosystems.

Performance Data Reference: Huawei Cloud’s README publishes performance data for different quantization precisions. For example, Llama-3-70B on Ascend 910B — int4 quantization boosts throughput from 28 tokens/s to 42 tokens/s, and reduces first token latency from 180ms to 120ms. This helps developers quickly evaluate quantization benefits.

There are some limitations to note:

• KVarN currently only supports decoder-only architectures (GPT, Llama, Qwen, etc.). Encoder-decoder models (T5, BART) are not supported yet.
• Quantization introduces slight accuracy loss — negligible for most tasks, but must be carefully evaluated for tasks highly sensitive to numerical precision (e.g., scientific computation).
• In dynamic quantization mode, quantization and dequantization have computational overhead. In short-text, small-batch scenarios, benefits may be offset by the overhead.

Industry Trends Behind the Open Source

KVarN’s release reflects several industry trends:

Domestic compute chips are actively integrating into open-source ecosystems. Previously, domestic chips relied on self-built ecosystems (e.g., Ascend’s MindSpore, Cambricon’s Cambricon PyTorch), but developer habits are hard to change. The current approach is to directly adapt mainstream frameworks and toolchains (PyTorch, vLLM, DeepSpeed), reducing migration costs. At Huawei Cloud’s Partner Conference in March, they mentioned MaaS service already supports over 160 industry-leading models, and the Pangu large model is gradually being open-sourced.

Inference optimization is shifting from model layer to system layer. Early optimization focused on model structures (MoE, sparse Attention) and training efficiency (ZeRO, FSDP). Now, with models standardized (mostly Transformers), optimization targets inference-side system engineering — KV cache management, scheduling strategies, memory allocation, operator fusion, etc. These optimizations often require deep hardware-level involvement, where cloud and chip vendors have natural advantages.

Quantization is no longer a “performance compromise”, but the default choice. NVIDIA supports FP8 directly on H100, and the new Blackwell architecture further enhances low-precision computing capabilities. The mainstream path in large model inference is shifting from “fp16 everywhere” to “quantize wherever possible”. KVarN ensures Ascend NPU remains aligned with this trend.

Open source is the new battleground for technical competition. The timing of KVarN’s release is subtle — right around NVIDIA’s Blackwell architecture launch and AMD’s MI300 series announcement. Using open source to gain developer support, build technical influence, and force competitors to follow suit has become standard practice for cloud and chip vendors.

What to Watch Next

KVarN is still in early version (0.1.x). Huawei Cloud’s GitHub Roadmap lists several directions:

• Support more model architectures (encoder-decoder, multi-modal)
• Provide auto-tuning tools to automatically choose optimal quantization strategies based on hardware and task
• Combine with FlashAttention, PagedAttention, and other technologies to further reduce memory usage
• Explore more aggressive quantization schemes (like 1-bit quantization, ternary quantization)

From community feedback, developers care most about two questions: whether it will support other hardware (especially NVIDIA and AMD GPUs), and whether quantization logic can be abstracted into a unified interface so different hardware backends can be switched seamlessly. If these are solved, KVarN could become the standard quantization solution in the vLLM ecosystem.

Huawei Cloud’s investment in AI infrastructure is long-term. From last year’s Pangu large model open source, to this year’s three top conference papers + KVarN release, to the upcoming AgentArts agent platform and DataArts data intelligence platform planned for later this year — it’s clear they are building a complete stack from compute, framework, toolchain to application. KVarN is just one part, but for teams needing to run large model inference on domestic compute, it’s a tool worth attention.

If you’re running vLLM inference services and face GPU memory bottlenecks, give KVarN a try. The code is already open-sourced on GitHub and works with vLLM 0.4.0+ versions. Even for developers without Ascend NPUs, its architecture design and quantization strategies offer valuable reference — because KV cache optimization is a challenge all hardware must tackle.

---

References

• Huawei Cloud CEO Zhou Yuefeng: Productivity gains matter more than total tokens revenue and consumption - IT Home
  Huawei Cloud’s official statement on MaaS strategy and domestic compute positioning

• Cloud Storage_news_paper_hopp_Huawei Cloud
  Detailed introduction to Huawei Cloud Storage Innovation Lab’s three top conference papers (DEEPSERVE, EPIC, RaaS)

• Large Model Inference Optimization Technology - KV Cache Quantization - Zhihu
  Technical background of KV cache quantization and comparison of existing solutions