Google open-sources DiffusionGemma: H100 reaches 1000 TPS

Google DeepMind dropped something interesting yesterday — DiffusionGemma, open sourced under the Apache 2.0 license, with weights already hosted on Hugging Face. The biggest selling point of this model isn’t the size of its parameters or its benchmark scores, but the fact that it is not an autoregressive model at all. It outputs 256 tokens in one go — an H100 single card can reach 1000 tokens/s, and even an RTX 5090 can push above 700 tokens/s.

Note, these are latency figures for a single user, single request, not throughput from cloud batch processing. For developers running local inference, this matters much more than it might seem.

Time for Autoregressive Models to be Challenged

Let’s recap the background. Starting from GPT-2, mainstream LLMs have all followed the autoregressive route: predicting one token at a time from left to right. This paradigm is very cost-effective in cloud batch serving — an H100 can handle dozens or hundreds of requests simultaneously, maxing out GPU utilization and driving the per-token cost extremely low.

But in local, single-user, single-request scenarios, autoregression becomes a waste. Your 4090 or H100 might have tens of TFLOPS of computing power, yet for each generated token the entire model’s weights must be read from VRAM. The bottleneck is entirely on memory bandwidth, so most of the compute sits idle. That’s why locally running LLMs never achieve high token/s rates — it’s not a lack of compute power, it’s an architecture mismatch.

Google’s idea here isn’t actually new — diffusion models have proven themselves in image generation long ago. Stable Diffusion starts from a completely noisy image, iteratively denoising until a clear picture appears. DiffusionGemma applies this to text: the model first fills a “text canvas” with 256 random placeholder tokens, then gradually denoises and corrects them through iterative steps, finally producing the whole passage at once.

Google’s analogy is quite apt: Autoregressive is a typewriter, one key at a time; diffusion is a printing press, a whole page at once.

26B MoE Architecture, but Only 3.8B Activated

Key specs:

• 26 billion total parameters, MoE architecture
• 3.8 billion activated per step
• Built on the Gemma 4 series and Gemini Diffusion research
• A newly designed diffusion head
• Bidirectional attention mechanism
• Quantized model can run on consumer GPUs with 18GB VRAM

The MoE choice is important. 26B parameters sounds impressive, but each forward pass only activates a few expert subnetworks, resulting in just 3.8B parameters in use. This separates model capacity from inference cost — you can leverage the entire knowledge base without paying the cost of full activation.

Bidirectional attention is another noteworthy aspect. Autoregressive models can only see tokens to their left, which is a big drawback for generating code, Markdown, Sudoku — tasks that require consistency front and back. If something is wrong earlier, later tokens just carry forward the mistake. Diffusion models are different — in each iteration, every token can see all other tokens in its context and can fix mistakes in subsequent rounds. That’s why Google emphasizes its suitability for code completion, in-line editing, mathematical formulas, amino acid sequences — structured tasks.

Interpreting the Performance Numbers

NVIDIA’s measured results:

| Hardware | Single request throughput | |-------------|---------------------------| | H100 | ~1000 tokens/s | | DGX Station | ~800 tokens/s | | DGX Spark | ~150 tokens/s | | RTX 5090 | >700 tokens/s |

Compared to equivalent autoregressive models, in single-user, single-GPU scenarios it’s about 4× faster.

But note: this 4× speedup comes with conditions. Google was quite honest about the limitations:

1. High QPS cloud services see diminished advantage. Cloud batch serving already feeds the GPU fully; the marginal gain from parallel decoding in diffusion quickly drops — you’ve already squeezed the compute dry, so layering diffusion adds little.
2. Shared memory architectures (like Apple Silicon) see modest benefit. Unified memory Macs don’t have bandwidth bottlenecks for autoregressive runs, so diffusion’s advantage won’t stand out.
3. Output quality is lower than standard Gemma 4. This is an experimental model that deliberately sacrifices quality to maximize speed.

Google makes it clear: for quality use Gemma 4; for speed and low-latency interaction, play with DiffusionGemma.

Its Actual Target Scenarios

Beyond the benchmark numbers, let’s think about where this model truly shines.

First: Local real-time editors. When writing code or documents, completion/rewrite assistants in IDEs are most annoying when latency is high. Tools like Cursor or Copilot could, in theory, return complete rewrites almost instantly with diffusion — instead of outputting word-by-word and making you watch the cursor anxiously.

Second: Non-linear text generation — an underappreciated direction. Traditional LLMs writing JSON, YAML, or Markdown tables essentially use natural language sequences to fit structured outputs, which causes frequent issues like missing brackets in JSON or misaligned tables in Markdown. Diffusion’s view of the whole canvas at once makes it more suitable for tasks with global constraints.

Third: Fast iteration experiments. Researchers tweaking prompts or running ablation studies hate waiting for inference. If producing 256 tokens takes about a second, the entire dev loop speeds up several times.

As for chat — diffusion models aren’t necessarily the best choice. Chat is naturally streamable; autoregressive can compute and output simultaneously, making latency imperceptible to the user. Diffusion waits for the whole block to finish before outputting, which can feel awkward.

Deployment Ecosystem: vLLM, MLX, Unsloth Suite

Google’s open-source release is quite comprehensive — no need to reinvent the wheel. Supported toolchains:

• vLLM — server-side inference
• MLX — runs on Apple Silicon (though gains are limited)
• Hugging Face Transformers — standard interface
• Hackable Diffusion / Unsloth / NVIDIA NeMo — fine-tuning
• llama.cpp — official support “coming soon”

Hardware optimization was done jointly with NVIDIA, covering RTX 5090, 4090, Hopper, Blackwell, DGX Spark, DGX Station, RTX PRO.

For local developers, the minimum VRAM requirement is 18GB — 4090 (24GB) and 5090 (32GB) are more than enough. Given this is a 26B MoE model, the VRAM usage is quite restrained.

Does the Text Diffusion Path Have Potential?

My judgment: this is a memorable milestone, but not a revolution.

Text diffusion isn’t new — Inception Labs’ Mercury has long sold similar productized solutions, and Google’s own Gemini Diffusion research published reports earlier this year. But as a truly open-source, downloadable, fully tooled SOTA-level model, DiffusionGemma is the first. Apache 2.0 licensing is also key — no restrictions on commercial use.

It addresses a very specific engineering problem — low GPU utilization in local single-user scenarios. This is a real issue in PC-side AI, on-device inference, and Copilot-like products. That’s why DiffusionGemma’s positioning is clear: not competing with cloud general models, but specializing in local real-time interaction.

Will diffusion replace autoregression? For now, far from it. Quality gap aside, cloud batch serving’s business model still favors autoregression. The flagship models of OpenAI, Anthropic, and Google won’t switch paths anytime soon.

A more likely trajectory is hybrid architectures — core generation with autoregression for quality, but specific modules (like code block generation, JSON output, long document rewrites) switched to diffusion mode for speed. This approach already exists in multimodal tasks, and could be adapted for text.

Practical Tips for Developers

If you want to try it today, here are some tips:

• Don’t use it in production yet. It’s experimental, quality below Gemma 4 standard.
• Prefer NVIDIA discrete GPUs. Macs will disappoint you here.
• Follow vLLM’s implementation progress. Batch serving to exploit diffusion’s advantages has a completely different scheduling logic from autoregression.
• Prioritize structured output scenarios — code, JSON, Markdown tables are its sweet spot.
• Fine-tuning path is ready — Unsloth + LoRA can be applied directly, low barrier.

By the way, mainstream closed models (GPT, Claude, Gemini series) can be called via a single API key on OpenAI Hub, with direct connection in China, compatible with OpenAI format — so you could use DiffusionGemma locally for real-time editing, and send to the cloud via OpenAI Hub for quality validation. This “local + cloud” hybrid workflow is worth setting up.

Conclusion

DiffusionGemma isn’t a shiny launch-stage model — it didn’t break any leaderboard records. But it challenges the almost decade-old assumption that text generation must be autoregressive.

Local AI has been a hot topic in recent years, from llama.cpp to Ollama, with evolving tooling — but underlying architectures never escaped the “memory bandwidth bottleneck” curse. DiffusionGemma proves at least one thing: by changing the generation paradigm, the ceiling of local inference can be pushed higher.

How far this path goes will depend on what the community creates with it in the coming half year.

References

• DiffusionGemma on Hugging Face — Google’s official model repo; weights and model card downloadable on Hugging Face