AMD Launches vLLM-ATOM Plugin to Directly Accelerate DeepSeek and Kimi Inference

Yesterday (May 11), AMD released the vLLM-ATOM plugin, a dedicated inference acceleration solution for Instinct GPUs. Its key selling point: no change to your existing vLLM commands, APIs, or workflows—the plugin takes over optimization in the background to directly improve inference performance for models like DeepSeek-R1, Kimi-K2.5, and gpt-oss-120B.

For developers, this means you can smoothly migrate existing vLLM-based services to AMD GPUs without rewriting code or learning new frameworks. AMD packages this as “zero learning cost”, though the real results will vary depending on scenario and model.

AMD’s Native Solution for the vLLM Ecosystem

vLLM is one of the most popular open-source frameworks for large model inference, focusing on optimizing throughput and memory efficiency under high-concurrency service scenarios. It’s not a simple “one-off inference” tool—it emphasizes request scheduling, KV cache management, and continuous batching, making it ideal for long-running online model services.

AMD’s previous support for vLLM mainly relied on the ROCm backend, but performance lagged behind NVIDIA’s mature CUDA ecosystem. With the vLLM-ATOM plugin, AMD has built its own optimization layer tailored to the Instinct GPU architecture and integrated it into the vLLM ecosystem as a plugin.

Architecturally, vLLM-ATOM consists of three layers:

• Top Layer – vLLM: Handles request scheduling, KV cache management, continuous batching, and OpenAI API compatibility. This layer remains unchanged—developers continue using the same interfaces.
• Middle Layer – ATOM Plugin: Manages platform registration, model implementation, attention backend routing, and kernel tuning. This is AMD’s optimization core.
• Bottom Layer – AITER: Provides GPU kernels, integrating fused MoE, Flash Attention, quantized GEMM, and RoPE fusion—operators optimized for AMD GPU instruction sets.

The benefit of this layered design is that AMD can inject its optimizations without modifying vLLM’s core code. For the vLLM community, this is also a friendly collaboration mode—mature optimizations can later be upstreamed to the native ROCm backend, benefiting the whole ecosystem.

Performance: MI355X Single GPU Exceeds 1,000 Tokens/s

AMD’s published performance data shows that on the DeepSeek-R1-0528 model, the MI355X with ATOM framework achieves single-GPU throughput of over 1100 tok/s, with about 26 seconds latency. By comparison, an NVIDIA B200 with TensorRT shows significantly lower throughput at similar latency.

These numbers come from AMD’s own testing, including FP8 precision, specific batch sizes, and sequence lengths. Real-world performance will vary by model architecture, input distribution, and concurrent request volume—but this indicates that AMD’s inference optimizations on Instinct GPUs have reached a level to directly compete with NVIDIA.

ATOM’s performance gains mainly come from:

1. Paged KV Cache: A core vLLM optimization that ATOM further tunes for AMD’s GPU memory hierarchy.
2. Segmented Compilation & Graph Capture: Uses eager mode during prefill and graph replay during decode to reduce kernel launch overhead.
3. Fused Operators: MoE routing, Flash Attention, and quantized GEMM are fused to reduce memory access frequency.
4. Multi-GPU Parallelism: Supports tensor and pipeline parallelism, compatible with MI350 and MI400 setups.

Currently, ATOM mainly targets high-end GPUs: Instinct MI350, MI400, and MI355X. Radeon Pro and older MI series cards may not enjoy the full performance boost.

Supported Models and Scenarios

vLLM-ATOM currently supports mainstream open-source large models, including:

• Dense Models: Llama 2/3/3.1, Qwen series
• MoE Models: Qwen3-MoE (128 experts, top-8 routing), Mixtral (8 experts, top-2 routing), DeepSeek V2/V3 (MLA attention), GLM-4-MoE
• Hybrid Architectures: Qwen3-Next (full attention + Gated DeltaNet), GLM-5 (MLA attention, similar to DeepSeek V3.2)
• Multimodal Models: Supports text-plus-vision VLM scenarios

AMD especially highlights support for the latest models DeepSeek-R1-0528, Kimi-K2.5-MXFP4, and gpt-oss-120b. These models are either recent releases or have unique inference needs (like Kimi’s MXFP4 quantization format).

From the supported list, AMD’s strategy clearly prioritizes models popular among Chinese developers. DeepSeek, Kimi, and GLM are widely used domestically; supporting them helps AMD quickly tap into the market. In contrast, although NVIDIA’s TensorRT-LLM is powerful, it often lags in supporting new models—giving AMD an entry advantage.

Deployment Threshold: Is It Really Zero Learning Cost?

AMD markets vLLM-ATOM as “zero learning cost”, but deployment does involve a few practical considerations.

First, you must install the ROCm environment. AMD recommends ROCm 6.0 or higher, which itself is a significant setup effort. ROCm configuration can be more complex than CUDA—multi-GPU environments might encounter driver, kernel module, or container runtime issues.

Second, vLLM-ATOM can run as a standalone inference server or as a backend plugin for vLLM. For the plugin mode, you must specify ATOM as the backend in vLLM’s configuration. While this is theoretically seamless, version compatibility may cause issues.

Third, performance tuning still requires experience. Although ATOM provides default optimization settings, the best batch size, KV cache strategy, and parallel mode depend on the model and hardware. AMD offers a benchmark dashboard to examine latency, throughput, and quality under different setups—but you’ll still need testing and adjustment.

Finally, ecosystem maturity remains an issue. Most vLLM contributors and users are within the NVIDIA ecosystem, meaning resources and discussions around CUDA are far richer. AMD is actively promoting its platform, but community support is still weaker.

Practical Implications for Developers

For developers, vLLM-ATOM’s release has several practical takeaways:

1. One More Hardware Option: If you’re building large-model inference services, you can now consider AMD GPUs—especially for cost-sensitive scenarios, where AMD may have better price-performance.
2. Supply Chain Risk Mitigation: NVIDIA H100/H200 supply remains tight; AMD Instinct GPUs can serve as an alternative. Though its ecosystem maturity lags, it’s at least a viable choice.
3. Competitive Pressure: AMD’s entry may push NVIDIA to offer better price and performance, which benefits the industry long term.

However, limitations exist:

1. Ecosystem Gap: CUDA’s maturity, tooling completeness, and community activity remain unmatched for now.
2. Model Coverage: While ATOM supports mainstream models, niche or custom ones may still require manual adaptation.
3. Long-Term Commitment: Whether AMD will sustain its AI investment remains uncertain—future strategy shifts could make current migrations obsolete.

Relationship with vLLM’s Native ROCm Backend

A common confusion is: how does vLLM-ATOM differ from vLLM’s native ROCm backend?

vLLM already supports ROCm via a community-maintained native backend. vLLM-ATOM is AMD’s own optimized solution provided as a plugin. The two can coexist—developers can choose either.

AMD’s approach is to rapidly iterate optimizations within ATOM, verifying results before gradually upstreaming them to the native ROCm backend. This ensures AMD’s improvements can be adopted quickly while also benefiting the open-source community—a pragmatic strategy.

For developers, if you want the latest performance optimizations, use vLLM-ATOM; if you prefer stability and community support, stick with vLLM’s native ROCm backend. Since both share compatible APIs, switching costs are minimal.

Opportunities for Chinese Developers

vLLM-ATOM’s focus on supporting DeepSeek, Kimi, and GLM—popular domestic models—is great news for Chinese developers. These models are heavily utilized in China, but NVIDIA’s TensorRT-LLM support often lags. AMD’s quicker follow-up offers more flexible options.

In cost-sensitive inference scenarios—such as consumer AI apps or large-scale inference services—AMD GPUs could prove especially cost-effective. If vLLM-ATOM maintains its advertised performance, Chinese cloud providers and AI firms may be more willing to adopt AMD solutions.

Another notable aspect is quantization support. vLLM-ATOM supports FP8 and MXFP4 formats, which are critical for reducing inference costs. Kimi-K2.5-MXFP4 is one example—it uses aggressive quantization to significantly cut memory and computation demands while maintaining quality. AMD’s strong quantization support could influence domestic developers’ choices in this area.

Summary

vLLM-ATOM represents AMD’s major push into large-model inference. Technically, its layered architecture, plugin integration, and targeted optimizations are sound. Performance-wise, the MI355X shows results competitive with NVIDIA. Strategically, AMD’s rapid iteration and upstream-sharing approach is pragmatic.

Whether it can truly challenge NVIDIA’s dominance depends on several key factors: continuous improvement of the ROCm ecosystem, AMD’s long-term commitment to AI, and developer adoption. In the short term, vLLM-ATOM is more of a complementary option than a replacement.

For Chinese developers, this is a promising direction. If you’re running large-model inference with DeepSeek, Kimi, or GLM, it’s worth trying vLLM-ATOM to see how it performs. Having more choices is always good—especially amid hardware supply constraints and cost pressures.

---

References

• AMD Launches vLLM-ATOM Plugin to Accelerate DeepSeek, Kimi and Other AI Inference Performance - IT Home – AMD official release and technical details
• AMD ATOM Inference Engine: How Paged KV Cache and Segmented Compilation Double Large-Model Performance - Zhihu – Deep dive into ATOM architecture and performance data
• vLLM-ATOM GitHub Repository – Open-source code and documentation