Tencent Hunyuan HPC-Ops Upgrade: Five Key Operators Target Inference Pain Points

Tencent Adds Five More Operators to HPC-Ops, This Time Targeting Real Pain Points in Inference Engineering

On June 11, Tencent’s Hunyuan AI Infra team released an upgraded version of the open-source HPC-Ops operator library, dropping five new operators at once. To understand the weight of this upgrade, we need to roll the timeline back to February of this year — that’s when HPC-Ops was first open-sourced. Built entirely from scratch with CUDA and CuTe, it boosted the Hunyuan model’s inference QPM in real production scenarios by 30%, with DeepSeek also seeing a 17% improvement. At the single operator level, Attention achieved up to 2.22x vs. FlashInfer/FlashAttention, GroupGEMM achieved up to 1.88x vs. DeepGEMM, and FusedMoE achieved up to 1.49x vs. TensorRT-LLM. These numbers were already enough to raise eyebrows in the industry at the time.

Four months later, this new wave of upgrades has a much clearer goal: plugging those engineering bottlenecks that only reveal themselves after going live.

Why Five Operators This Time, Not a Grand Narrative

Anyone who’s done inference deployment knows: a beautiful benchmark run and holding up under production load are two very different things. An operator may have low average latency in the white paper, but if the P99 tail latency wobbles, the entire service SLA can collapse. There’s a key phrase in the official HPC-Ops upgrade statement — “to further meet the inference system’s adaptability to dynamic business workloads.” In plain terms: the previous version crushed benchmarks under static, orderly workloads, but real traffic is dynamic, irregular, and a mix of long and short — meaning there’s still room to squeeze out performance in those scenarios.

This time, Tencent specifically pointed out three hard engineering problems they’re aiming to solve:

• Attention tail latency: In variable-length sequence and continuous batching scenarios, long sequence requests can blow up the latency distribution for the whole batch
• Memory transfer overhead: The gap between modern GPU compute power and HBM bandwidth is widening; often operators are bottlenecked by transfers, not compute
• Cross-GPU communication: In single-machine multi-GPU or multi-machine multi-GPU tensor parallel / expert parallel scenarios, the overlap between AllReduce, All-to-All communication primitives and compute directly determines throughput

These three directions match the roadmap previously published by the HPC-Ops team — sparse Attention, mixed precision quantization, compute–communication co-optimization. The five new operators essentially advance that roadmap by a step.

Treating the Operator Library as an Engineering Project, Not a Paper Project

The biggest difference between HPC-Ops and other open-source operator libraries is its origin. The FlashAttention series and FlashInfer follow an academic route — paper first, academic benchmark runs; DeepGEMM is DeepSeek’s open-source FP8 GEMM; TensorRT-LLM is NVIDIA’s official offering, mostly closed-source kernels. HPC-Ops is positioned more like “we hit countless production pitfalls, and extracted the solutions to open-source.”

That origin determines several traits:

First, abstracted engineering architecture. Operator development has traditionally been done with low-level template libraries like CUTLASS, or by directly writing CUDA kernels — the former has a high barrier, the latter is hard to maintain. HPC-Ops builds an abstraction layer on top of CuTe so upper-layer operator developers don’t have to wrestle with tile, layout, swizzle concepts every time. This lowers the development barrier and speeds up iteration.

Second, deep microarchitecture adaptation. Hopper (H100/H800) and Blackwell (B200) GPUs introduce new hardware features like TMA and WGMMA, and leveraging them well directly impacts how close operators can get to hardware peak performance. HPC-Ops has gone deep here, especially in Attention and GroupGEMM, fully exploiting TMA’s asynchronous load and WGMMA’s warp group collaboration.

Third, instruction-level extreme optimization. This is the dirty, exhausting work — register allocation, bank conflict avoidance, instruction scheduling at the PTX level, squeezing out performance bit by bit. Most teams lack the patience and opportunity cost for this. Hunyuan has its own real models and external DeepSeek deployments pushing them, so the ROI justifies the effort.

Competition and Comparison

Let’s line up the competitors. FlashAttention 3 has already pushed Attention close to theoretical peak on H100; HPC-Ops achieving 2.22x on top of that sounds incredible, but the reason is clear — FlashAttention 3 excels at structured long sequences, and its optimization for variable-length scenarios is conservative. HPC-Ops has obviously created dedicated paths for variable-length, sparse, and tail-heavy cases, so for certain shapes it can open such a big gap. This reminds us: when looking at benchmarks, check the exact config; “up to 2.22x improvement” is peak, not average.

GroupGEMM vs. DeepGEMM is similar. DeepGEMM focuses on FP8, and is a core operator in DeepSeek’s MoE path. HPC-Ops achieving 1.88x likely comes from scenarios with large group counts and small M per group — i.e. real MoE inference with uneven expert activation. FusedMoE vs. TensorRT-LLM at 1.49x is heavier weight, because TRT-LLM is NVIDIA’s official product — getting 50% over it shows HPC-Ops has meaningful ideas in MoE fusion path design.

Which Scenarios the Five Operators Cover

While full technical details wait for the code repo update and future blog posts, based on the three directions disclosed officially, we can roughly guess the distribution of the five operators:

• Sparse/Dynamic Attention: For long-context scenarios, reducing KV Cache memory access pressure. Now large-model contexts are often 128K or 200K tokens, and Attention’s memory bottleneck is the real issue — compute is comparatively abundant.
• Quantized GEMM/MoE: Core operators for mixed precision. 4-bit weights + 8-bit activations, W4A8 combos are increasingly common in production, and open-sourcing high-performance kernels for this is very friendly for small teams deploying large models.
• Compute–communication co-ops: Embedding AllReduce, All-to-All into GEMM or MoE kernels so communication runs in the shadows of compute. This is the toughest challenge in distributed inference, and success here can save lots of cross-GPU wait time.

# Typical HPC-Ops operator usage (illustration)
from hpc_ops import fused_moe, attention

# Fused MoE call
out = fused_moe(
    hidden_states,
    expert_weights,
    topk_ids,
    topk_weights,
    quant_type="w4a8",
    overlap_comm=True,  # Overlap compute and communication
)

# Variable-length Attention call
out = attention(
    q, k, v,
    cu_seqlens,
    sparse_pattern="block_sparse",  # Sparse mode
    kv_cache_layout="paged",
)

What This Means for Developers

If you’re a small or medium team running inference frameworks on your own hardware, can you use HPC-Ops directly? Answer: Yes, but with a threshold.

The benefits are obvious:

1. Direct cost savings: 30% QPM improvement means the same GPU handles 30% more traffic — or the same traffic requires 30% fewer GPUs. That’s not pocket change.
2. Friendly to domestic models: DeepSeek also gained 17%, showing HPC-Ops isn’t just private optimization for Tencent’s models — it genuinely contributes to the open-source ecosystem.
3. Production-grade stability: Hunyuan is used internally at Tencent, and externally by customers running DeepSeek, meaning Tencent has already done the pitfall work for you.

But the thresholds are worth noting:

• High hardware demand, best optimized for Hopper and above; runs on A100/A800 but gains reduced
• Integrating into your inference framework requires engineering skills; it’s not as simple as pip install
• Roadmap still iterating; stability of sparse Attention, etc., needs time to prove

A Bigger Picture

In the AI Infra world of 2026, several clear trends: MoE models becoming the mainstream large-model form, surging demand for long-context inference, FP8/INT4 low-precision inference moving from experimental to standard, multi-GPU/multi-machine deployment becoming the norm. Against this backdrop, competition between inference engines has moved from “can it run” to “how fast, how stable, how cheap.”

The open-source upgrade of HPC-Ops is Tencent playing a card in this competition. The logic is clear: open-source the low-level operators to let the ecosystem use them, thereby boosting deployment efficiency and influence of its own models. DeepSeek played the same script with DeepGEMM; NVIDIA has done it for decades with cuBLAS/cuDNN. Tencent’s entry means domestic vendors now have something at the AI Infra core level that can stand toe-to-toe with international open-source projects.

For developers, this is good news — more choices, more performance competition, lower deployment costs. Whether HPC-Ops becomes the standard operator library for next-gen mainstream inference engines will depend on community acceptance and sustained iteration. But at least judging from the five operators released today, Tencent’s Hunyuan AI Infra team is genuinely serious about this — not just releasing a PR for attention.

Notably, OpenAI Hub (openai-hub.com), as an AI API aggregation platform, has consistently provided full support for domestic mainstream models like Hunyuan and DeepSeek, with domestic direct connections and OpenAI-format compatibility. If you want to directly call inference services optimized underneath with HPC-Ops, it’s a relatively convenient path.

References

• Tencent Hunyuan AI Infra Core Technology Open-Sourced, Inference Throughput Improves by 30% - IT Home: IT Home’s report on HPC-Ops’ first open-source release, with official performance data and roadmap
• Tencent Hunyuan AI Infra Core Technology Major Open-Source Release: Inference Throughput Improves by 30% - Zhihu Column: Zhihu column’s interpretation of HPC-Ops technical details, including CUDA/CuTe build path analysis