DeepGEMM Major Update: 300 Lines of Code Make Matrix Multiplication 2.7× Faster

DeepSeek recently released a major update to its open-source matrix computation library, DeepGEMM. This library, designed specifically for FP8 precision General Matrix Multiplication (GEMM), has a core kernel of only about 300 lines of code but achieved up to 1358 TFLOPS of computational power and 2.7× the performance of the optimized CUTLASS baseline on H800 GPUs.

For developers focused on AI infrastructure, this is not a number to be ignored.

First, what is DeepGEMM?

Matrix multiplication is the absolute foundation of large model computation. Whether it’s the attention mechanism in a Transformer or the linear transformations in feedforward networks, it ultimately comes down to multiplying matrices. GPUs can train large models because their core capability lies in parallelizing massive amounts of matrix computations.

For years, everyone has defaulted to using NVIDIA’s official CUDA libraries (cuBLAS) or CUTLASS for this task. These libraries are mature and highly optimized. However, during the training of the V3 and R1 models, the DeepSeek team discovered significant room for optimization for FP8 precision and certain model structures.

DeepGEMM is the result of that optimization. It’s not a general-purpose math library, but a highly focused tool — designed specifically for FP8 matrix multiplication, and it comes with fine-grained scaling support as used in DeepSeek-V3.

To use an analogy: cuBLAS is a Swiss Army knife — it can do anything. DeepGEMM is a scalpel — it does one thing, but does it extremely well.

What does this update bring?

This major update further improves both performance and engineering usability. Several key points are worth expanding on.

Performance numbers: not a tweak, but a leap

The DeepSeek team used the NVCC 12.8 compiler on H800 GPUs to systematically test all matrix shapes used in DeepSeek-V3/R1 inference. The results were impressive:

• Peak compute power: 1358 TFLOPS
• Peak memory bandwidth: 2668 GB/s
• Compared to the CUTLASS 3.6 optimized baseline: up to 2.7× speedup

Performance varied significantly across different matrix shapes. For dense model GEMM tests, for example:

| M | N | K | Compute Power | Memory Bandwidth | Speedup | |---|---|---|----------------|------------------|----------| | 64 | 2112 | 7168 | 206 TFLOPS | 1688 GB/s | 2.7× | | 64 | 7168 | 16384 | 336 TFLOPS | — | — |

What does "2.7×" mean? Suppose training a model originally required 100 H800s running for a week. If matrix multiplication becomes 2.7× faster, the overall training time and cost would drop significantly. Of course, the actual gain depends on how much of the computation graph is occupied by matrix multiplication — but for Transformer architectures, that proportion is typically very high.

It’s important to note that 2.7× is the peak speedup under ideal conditions, which occurs at smaller M values (e.g., M=64). This precisely corresponds to typical inference-stage matrix shapes — when batch size is small, one matrix dimension tends to be short. In other words, DeepGEMM is especially optimized for inference scenarios.

Targeted optimization for MoE models

DeepSeek-V3 uses a Mixture of Experts (MoE) architecture, which requires a large number of grouped GEMM operations. Ordinary matrix multiplication libraries often don’t handle these scenarios well.

This update to DeepGEMM provides about a 1.2× stable performance improvement for MoE scenarios. While 1.2× might sound less impressive than 2.7×, there are two important points:

1. The 1.2× is stable, meaning it’s consistent across various matrix shapes, not just a peak in a special case.
2. MoE models already involve massive computation, so even a 1.2× consistent gain accumulates into a major efficiency impact. DeepSeek-V3 has 256 experts, requiring huge numbers of grouped GEMMs during inference. This optimization directly speeds up the model’s core compute path.

The 300-line engineering philosophy

Perhaps even more striking than the raw performance is DeepGEMM’s code size — the core kernel consists of just about 300 lines.

Anyone who’s done CUDA development knows how complex writing a high-performance GEMM kernel is. CUTLASS runs into tens of thousands of lines of code, full of template metaprogramming, pipeline scheduling, and memory hierarchy management. That DeepGEMM achieves higher performance in just 300 lines is an impressive feat of engineering decisions.

Two key design choices made this possible:

1. Abandon generality; focus solely on FP8.
   By not supporting FP16, FP32, INT8, or other datatypes, code size was dramatically reduced.

2. Use JIT (Just-In-Time) compilation.
   DeepGEMM doesn’t precompile kernels; instead, it compiles them dynamically at runtime based on actual matrix shapes. This allows the compiler to perform more aggressive optimizations — like constant folding and loop unrolling — for specific parameters. The tradeoff is a compile overhead on first run, but for long-running training or inference workflows, it’s negligible.

This design philosophy aligns with DeepSeek’s usual style: not "big and comprehensive", but "small and perfect" for specific scenarios. The 300 lines also mean extremely low maintenance costs and very high readability, making it far easier for other teams to understand, customize, or adapt to their use cases.

A bigger picture perspective

To grasp the importance of this update, it helps to view DeepGEMM within DeepSeek’s overall open-source strategy.

In early 2025, DeepSeek launched an "Open Source Week", releasing five core technical components over five days:

1. FlashMLA (Day 1): An MLA decoding kernel optimized for Hopper GPUs, achieving 3000 GB/s throughput and 580 TFLOPS on the H800
2. DeepEP (Day 2): An EP communication library for MoE models, improving inter-GPU token routing and aggregation
3. DeepGEMM (Day 3): The FP8 matrix computation library
4. and the subsequent components that followed

Together, these components form the core of DeepSeek’s training and inference technology stack. They’re not isolated tools but a cohesive system: FlashMLA optimizes attention, DeepEP optimizes inter-expert communication, and DeepGEMM optimizes matrix multiplication — tackling the three main bottlenecks in large model computation.

This major DeepGEMM update shows that DeepSeek’s open-source week wasn’t just a one-time PR stunt — they’re continually iterating and improving these base components. That’s a highly positive signal for the open-source community.

What it means for developers

Now some practical takeaways.

If you’re doing large model training, especially with FP8 precision (increasingly common on H100/H800 GPUs), DeepGEMM is well worth evaluating seriously. A 2.7× peak speedup translates directly into cost savings. Even if the average gain is "only" 1.5–2×, that’s still substantial — given training costs often reach into the millions of dollars.

If you’re focused on inference optimization, DeepGEMM might be even more valuable. Inference matrix shapes (small batch size, i.e., small M) align perfectly with DeepGEMM’s strongest speedups. For teams running large-scale model inference services, this means lower latency and higher throughput.

If you’re only using model APIs and not dealing with the underlying computation directly, DeepGEMM’s impact is indirect — it reduces inference costs for model providers, potentially lowering API prices. DeepSeek is already known for its ultra-low API pricing, with low-level optimizations like DeepGEMM being a key cost advantage.

For developers wanting to call DeepSeek models via API, you can access them directly through OpenAI Hub, compatible with the OpenAI SDK format — no separate integration needed:

from openai import OpenAI

client = OpenAI(
    api_key="Your OpenAI Hub API Key",
    base_url="https://api.openai-hub.com/v1"
)

response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the benefits of FP8 precision in large model training."}
    ],
    temperature=0.7
)

print(response.choices[0].message.content)

# Using curl works too
curl https://api.openai-hub.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YourOpenAIHubAPIKey" \
  -d '{
    "model": "deepseek-chat",
    "messages": [
      {"role": "user", "content": "What are the advantages of DeepGEMM’s JIT compilation mechanism?"}
    ]
  }'

Comparing with competitors

Current options in the FP8 matrix computation field include:

• NVIDIA cuBLAS: The official library, most general-purpose, but less optimized for specific use cases
• CUTLASS 3.6: NVIDIA’s open-source template library, flexible and widely adopted — DeepGEMM’s main benchmarking baseline
• Triton: OpenAI’s compiler that allows writing GPU kernels in Python — developer-friendly, but typically doesn’t match hand-tuned CUDA in peak performance
• DeepGEMM: Focused on FP8, minimalistic code, highest peak performance, but narrow scope

Which to choose depends on your context:
If you need multiple datatypes and hardware support, CUTLASS remains the safer bet.
If you’re targeting FP8 on Hopper GPUs — particularly running DeepSeek-series models — DeepGEMM is likely the best-in-class solution.

A noteworthy trend: more teams are going “narrow and deep” — building specialized optimizations rather than relying solely on general-purpose libraries. DeepGEMM’s success validates that direction. Expect to see more such specialized libraries emerge in the future.

Some details worth noting

A few points from community discussions:

Hardware compatibility.
DeepGEMM is currently optimized mainly for NVIDIA Hopper architecture (H100/H800). If you’re on A100 or older GPUs, FP8 itself isn’t supported, so DeepGEMM isn’t applicable. That’s a hardware limitation.

Precision concerns.
FP8 precision loss has long been an industry concern. DeepGEMM’s built-in fine-grained scaling (from DeepSeek-V3’s design) helps mitigate it, though not completely. For precision-sensitive tasks, careful validation is still needed.

Ease of integration.
Thanks to JIT compilation and minimal design, integrating DeepGEMM is straightforward. No complex build configurations — a simple pip install suffices. Much friendlier than CUTLASS, whose setup can be a headache.

Open-source license.
DeepGEMM is released under the MIT license — unrestricted for commercial use. Excellent news for teams looking to integrate it into their products.

Final thoughts

This DeepGEMM update once again proves a key point: in AI infrastructure, “small and elegant” specialized optimizations often beat “big and general” solutions. Achieving 2.7× acceleration with 300 lines of code is an astonishing ROI, even by CUDA ecosystem standards.

DeepSeek is redefining the standard for open-source AI infrastructure through real engineering results. From FlashMLA to DeepEP to DeepGEMM, each component is not a toy demo but an industrial-grade, production-validated tool. The openness and quality are rare, especially among domestic AI companies.

For developers, this is an exciting time — foundational compute efficiency is improving rapidly, model capabilities keep evolving, and the barrier to leveraging these advances continues to drop. Whether you work on low-level optimization or application-level development, DeepSeek’s open-source progress is well worth following.

---

References

• DeepSeek releases major DeepGEMM update — discussion on Linux.do community (Community discussion of this update)