DeepMind open-sources DiffusionGemma: H100 achieves 1000 TPS

DeepMind Puts Image Generation Architecture into Text Models

On June 10, Google DeepMind open-sourced DiffusionGemma—an experimental open model that applies the diffusion architecture to text generation. The official figures are striking: over 1000+ tokens/sec on a single NVIDIA H100, over 700+ tokens/sec on an NVIDIA RTX 5090, with decoding speed improved by about 4× compared to an autoregressive Gemma of the same scale. NVIDIA also published a blog the same day, announcing targeted optimizations from the GeForce RTX and RTX PRO platforms to DGX Spark.

Hearing the combination of "diffusion + text," the first reaction from those who’ve worked with models is likely: here we go again. Over the past two years, academia has occasionally attempted to bring diffusion to LLMs—from SSD-LM to Mercury to the batch of models from Inception Labs—but those that can match autoregressive baselines in both throughput and quality in industrial-grade scenarios are few and far between. DeepMind’s move to release weights and give reproducible 1000 TPS numbers pushes this technical route a big step forward from the paper stage toward engineering mainstream.

Where Exactly the Speed Comes From: Moving from "Memory Bandwidth Bottleneck" to "Compute Bottleneck"

To understand why DiffusionGemma is fast, we first need to clarify where autoregressive models get stuck.

Traditional Next Token Prediction (NTP) architecture, for every token generated, reads billions of parameters from GPU memory for a full forward pass. The problem is that the calculation per token is tiny, and most of the time the GPU’s TensorCores are waiting for memory—this is the classic memory-bound scenario. An H100 has nearly 1000 TFLOPS of theoretical FP16 compute, but when running inference with a 7-billion-parameter autoregressive model, compute utilization is often only single-digit percentages. The rest of the time is spent waiting for HBM.

Diffusion models take a different approach. Instead of emitting tokens one by one from left to right, they start with a fixed-length sequence of blank (noisy) tokens, predict the entire segment at once, and then use multi-step denoising iteration to refine the output until clean.

This mechanism brings two direct results:

• Parallelism increases. Each denoising step operates on the entire block simultaneously, greatly increasing matrix operation density, so the GPU finally gets fully loaded.
• The bottleneck shifts from bandwidth to compute. Previously idle TensorCores are now saturated, allowing the H100’s hardware capabilities to truly shine.

Thus, the 1000 TPS is not simply a result of “making the model smaller” or “quantizing,” but from aligning the architecture with the GPU’s strengths. This is a different approach from last month’s Gemma 4 MTP (Multi-Token Prediction) drafter—MTP is essentially speculative decoding, using a small model to draft and a large one to verify in parallel, achieving at most 3× speed-up; DiffusionGemma changes the underlying paradigm.

Understanding the Numbers: What Does 1000 TPS Mean?

Let’s give some comparative benchmarks. Currently, running Llama 3 8B or Gemma 2 9B autoregressive inference on the H100 in a single-user scenario usually sits in the 150–250 TPS range. With optimizations like vLLM and PagedAttention, hitting around 300 TPS is considered excellent. So what does 1000 TPS mean?

• From a user perspective, it’s almost “instant answers.” A 500-token reply is done in 0.5 seconds.
• Particularly friendly for agent scenarios. Multi-round tool calls, ReAct loops, code execution feedback—chains that used to be painful due to latency stacking can now keep each hop under a second.
• Consumer-grade GPUs can handle it. The RTX 5090’s 700 TPS is more critical, meaning this isn’t just a privilege of data centers; local dev machines can now deliver experiences that previously required A100/H100 clusters.

NVIDIA’s optimizations here are also worth mentioning. They’ve fine-tuned kernels for Blackwell architecture, fusing the repeated attention computations and denoising steps in the diffusion process to avoid intermediate memory reads/writes. DGX Spark’s product line will likely use this model as one of its selling points.

And the Quality? That’s the Key Question

Speed is easy to talk about, quality less so. The biggest historical controversy with diffusion-based text models is whether generation coherence, reasoning ability, and long-text consistency can match autoregression.

DeepMind’s blog claims DiffusionGemma’s performance on mainstream benchmarks is “comparable to same-scale autoregressive Gemma.” But how wide that “comparable” range is, isn’t yet clear from the public details. Based on preliminary community testing:

• Short contexts, deterministic tasks (classification, extraction, summarization, code completion): basically no gap; speed advantage directly translates into user experience.
• Multi-step reasoning, long chain-of-thought: diffusion’s iterative denoising is inherently less suited to “tightly logical chain reasoning,” which needs more benchmark verification.
• Ultra-long generation: diffusion requires pre-determined output length, making it less flexible than autoregression for open-ended long text. This is a natural constraint at the paradigm level.

Therefore, seeing DiffusionGemma as a cure-all isn’t realistic. It’s more like an alternative prepared for latency-sensitive, throughput-first scenarios—customer service, real-time translation, IDE inline completion, agent tool invocation—where an extra 200 milliseconds can cause user drop-off, and deep reasoning isn’t the core requirement.

What This Means for Developers

Some tangible impacts from my perspective:

1. The ceiling for local inference is raised. Previously, running a 7B model on RTX 4090 maxed out around 200 TPS; now RTX 5090 gets 700 TPS—this is a qualitative, not just quantitative, leap. Local Copilot and local agent playability increases significantly.
2. Inference service providers’ cost structure will be reassessed. If the same GPU can deliver 4× the throughput, the per-token cost is theoretically 1/4. This benefits inference platforms like Together and Fireworks, and also teams running in-house inference.
3. Architectural diversity returns. Over the last three years, the LLM track was heavily “Transformer + autoregression,” with routes like Mamba, RWKV, and Diffusion always in fringe exploration. DeepMind’s open-source release gives diffusion text models a strong boost, likely to trigger a wave of follow-up in the next six months.

As a side note, the OpenAI Hub (openai-hub.com) aggregation of open-source models will also integrate DiffusionGemma, allowing developers to switch between mainstream models like GPT, Claude, Gemini, DeepSeek with the same key, compare different architectures’ actual performance in their business contexts, and save the trouble of managing multiple APIs.

Questions Still Unanswered

This is an experimental release, and DeepMind itself says it’s “experimental.” Some things need further observation:

• Long context support. The current version’s max context length hasn’t been strongly promoted; diffusion’s support cost for ultra-long contexts may be higher than autoregression.
• Fine-tuning ecosystem. Can tools like LoRA and QLoRA be directly applied? Best practices for adding adapters into the diffusion process haven’t yet formed.
• Inference framework adaptation. Mainstream frameworks like vLLM, TensorRT-LLM, llama.cpp currently have only rudimentary support for diffusion text models; community follow-up speed will determine practical deployment range.

That said, open-sourcing it is itself the biggest signal—letting the community fill gaps, experiment, and expand. This has been Gemma series’ consistent strategy, explaining why since last year downloads skyrocketed to 60 million within weeks. DiffusionGemma has been out only a few days, yet the HuggingFace download curve is already steep.

Whether it will eventually become a mainstream route alongside Transformer, or remain another experiment with good benchmark performance but limited engineering adoption, will be clear if we look back after six months.

References

• linux.do - Google open-sources Diffusion Gemma, can hit 1000 TPS on H100 - First-hand discussion and test feedback from the Chinese community
• Reddit - Google says Multi-Token Prediction makes Gemma 4 faster - Practical discussion of Gemma 4 MTP, serves as a speed optimization comparison reference
• Zhihu - Gemma 4: Google’s open-source byte efficiency king - Analysis of overall architecture and positioning of the Gemma series open-source models