Tencent Hunyuan Stem Algorithm Selected for ICML-26: 25% Compute Power Approaches Full Precision

Tencent Hunyuan announced today that its Stem sparse attention algorithm has been accepted by ICML-26, a top-tier machine learning conference. This algorithm can achieve accuracy close to dense attention using only 25% of the compute power, reducing first-token latency by 3.6x with a 128K context. Both the paper and the HPC operator library are now open source.

For developers, this is not just another “theoretically elegant” optimization proposal. The core value of Stem lies in actually transforming the theoretical acceleration ratio of sparse attention into true end-to-end measured performance, rather than staying in the realm of FLOPs counting on paper.

Revisiting Block-Level Sparsity From the Perspective of Causal Information Flow

The attention mechanism in Transformers is a compute power black hole during long-context inference. The common industry approach is to use block-sparse attention to reduce computation, but traditional solutions either suffer significant accuracy loss or fail to achieve theoretical acceleration on real hardware.

Stem’s approach is to return to the essence of causal information flow: in autoregressive generation, later tokens depend on earlier tokens, but this dependency is not uniform. It introduces two key innovations:

Token Position Decay (TPD): The earlier a token is, the weaker its influence on the current generation, allowing it to be processed with coarser-grained attention. This is not simple distance decay, but a decay pattern based on causal linkages.

Output-Aware Metric (OAM): Instead of statically determining which tokens are important, it dynamically adjusts the sparsity pattern according to actual output needs. This enables Stem to maintain accuracy close to dense attention across different tasks and generation stages.

Near Lossless Accuracy with 25% Compute Budget

The data from the paper is straightforward: with a 25% compute budget, Stem’s performance in multiple benchmark tasks is close to dense attention. Here, “close” doesn’t mean 90%—the difference is within the statistical margin of error.

More importantly, this accuracy was measured on Tencent Hunyuan’s Hy3 preview (W8A8-FP8) quantized model, not under ideal FP16/BF16 conditions. Quantization itself introduces accuracy loss, so the fact that Stem maintains performance on a quantized model shows its design is robust.

From a developer’s perspective, this means you can cut inference costs to a quarter of the original while maintaining output quality. For applications that require processing long contexts—code completion, document analysis, multi-turn dialogue—this is a tangible cost optimization.

HPC Operator Library Brings Acceleration Ratio Into Reality

Anyone with GPU optimization experience knows that theoretical FLOPs reduction and actual latency reduction are two different things. Memory bandwidth, kernel launch overhead, and irregular access patterns can each render your theoretical acceleration meaningless.

Stem’s companion open-source HPC operator library addresses this issue. It provides CUDA implementations of Stem and BSA (Block-Sparse Attention), with specific optimizations targeting H100/H800 hardware.

For a 128K context, the 3.6x reduction in first-token latency is measured end-to-end, including kernel launches, data transfer, and all overheads encountered in real inference scenarios. This is much more reliable than solutions that only report single operator performance.

Benchmarking Against Other Sparse Attention Solutions

Sparse attention is not a new concept. FlashAttention, PagedAttention, and various sliding window schemes all tackle similar problems. Where does Stem differ?

FlashAttention primarily optimizes memory access patterns but is essentially still dense attention. It is very efficient with short contexts but cannot solve the compute issue for long contexts.

Sliding window schemes (e.g., Longformer, BigBird) use fixed patterns for sparsity, which is simple but results in notable accuracy loss. Performance is poor on tasks requiring global information (e.g., document QA).

Learning-based sparsity schemes (e.g., various learned sparsity methods) are theoretically more flexible but come with high training costs and usually require retraining from scratch or large-scale fine-tuning.

Stem’s advantage is that it can be directly applied to existing models without retraining, while achieving a better balance between accuracy and efficiency. The design of TPD and OAM allows it to capture global information while maintaining compute efficiency.

Engineering Implementation Details

The open-source code has several noteworthy aspects:

1. Chunking strategy: Stem does not simply split sequences into fixed-size blocks but adjusts block size and sparsity patterns dynamically based on TPD and OAM. Earlier token chunks are larger, later ones are smaller.

2. Mixed precision processing: In W8A8-FP8 quantization scenarios, Stem applies different precision strategies to blocks with different sparsity levels. Important blocks keep higher precision; less important blocks use more aggressive quantization.

3. Kernel fusion: The HPC operator library fuses multiple operations into a single kernel to reduce memory round trips. This is especially important in sparse scenarios, where irregular access is prone to bandwidth bottlenecks.

4. Prefetching and pipelining: Long-context inference is most vulnerable to memory stalls. The operator library uses aggressive prefetching strategies and multiple pipelines to overlap computation and data transfer as much as possible.

What ICML-26 Acceptance Signifies

ICML is one of the top three machine learning conferences, with strict reviews and rejection rates consistently above 70%. Stem’s acceptance indicates that its theoretical contributions have gained recognition within academia.

More importantly, this is not purely a theoretical paper. A complete open-source implementation, real hardware test data, and validation on quantized models—all these demonstrate engineering value. Academia and industry have been exploring sparse attention for years, and Stem’s contribution is narrowing the gap between theory and engineering in a big way.

From the perspective of Chinese large model vendors, this is also a signal: beyond model scale and training data, system optimization and algorithm innovation are equally important. OpenAI and Anthropic have invested heavily in inference optimization; if domestic vendors focus only on parameter counts and benchmark scores, their long-term competitiveness will be at risk.

Practical Significance for Developers

If you are using Tencent Hunyuan’s API, Stem’s optimization may already be active in the backend, manifesting as faster responses and lower costs in long-context scenarios.

If you are building model inference services, the HPC operator library is a tool worth investigating. It is not tied to a specific model and can theoretically be integrated into any Transformer-based inference engine. However, note that the current implementation is mainly optimized for NVIDIA H-series cards, and performance may be reduced on other hardware.

If you are researching sparse attention, Stem’s TPD and OAM designs provide new ideas. Since the code is open source, you can experiment on top of it without implementing a sparse attention scheme from scratch.

Open Source Code and Resources

Tencent has open-sourced the complete implementation of Stem:

• Paper: arxiv.org/abs/2603.06274 – Detailed description of the design principles, theoretical analysis, and experimental results of TPD and OAM
• Algorithm implementation: github.com/Tencent/AngelSlimHPC – Includes Python interface and reference implementation of Stem
• HPC operator library: github.com/Tencent/hpc-ops – CUDA kernels and performance testing tools

The code quality is good, with complete documentation and usage examples. If you want to integrate Stem into your own project, starting with the HPC operator library is the most direct route.

The Future of Long-Context Inference

Sparse attention is just one direction for optimizing long-context inference. Other parallel technical routes include:

Architectural innovation: Models like Mamba, which use state space architectures to fundamentally avoid the quadratic complexity of attention mechanisms

Quantization and compression: More aggressive quantization schemes (e.g., 4-bit, even 2-bit), or dynamic precision allocation

System optimization: Better memory management (PagedAttention), more efficient scheduling strategies, specialized hardware accelerators

Stem goes deep down the sparse attention track, but it is not the only answer. The future may be a combination of technologies: using Mamba for ultra-long contexts, sparse attention for medium-long contexts, dense attention for short contexts, coupled with aggressive quantization and specialized hardware.

From this perspective, Stem’s value is not just as a specific algorithm, but as proof that within the existing Transformer architecture, algorithm and system co-optimization can still extract significant performance gains. This is good news for the entire industry, as rewriting all model architectures is too costly; optimizing on existing foundations is a more realistic path.

---

Reference Resources

• Tencent Hunyuan introduces Stem sparse attention algorithm, first-token latency reduced by 3.6x – IT Home – Official release information and key data
• Stem algorithm open-source repository – GitHub – Python implementation and usage documentation
• Stem HPC operator library – GitHub – CUDA kernels and performance testing tools