Diffusion Models Break Into Text Generation: Google Open Sources DiffusionGemma, 4× Speed Boost

Last week, Google DeepMind released DiffusionGemma on Hugging Face under the Apache 2.0 license, with fully open weights. This is not yet another “stronger, bigger” model release — Google openly stated in the official documentation: its output quality is not as good as Google’s own Gemma 4.

But it's fast. Fast enough to hit over 1000 tokens/s on a single H100, and 700+ tokens/s on a consumer-grade RTX 5090. That’s roughly 4× the speed of an autoregressive model of the same level.

CEO Sundar Pichai personally posted about it, comparing it to a “racehorse” — winning speed by generating whole blocks of text at once instead of predicting word-by-word.

This is a formal debut for diffusion models in text generation, and it’s arriving in open-source form usable on consumer-grade hardware.

Not a Typewriter, but Developing a Photo

To understand why DiffusionGemma is so fast, you first need to see how current mainstream large models generate text.

GPT, Claude, Gemini, Llama — all of these are autoregressive architectures. When generating text, they work like a typewriter, producing tokens from left to right one at a time. Before generating the 100th character, they must produce the preceding 99. Every time they generate a token, the GPU has to reload the entire model weights from memory.

This causes one problem: Most of the GPU’s time is spent waiting for data transfers, while its compute capacity sits idle.

In cloud batch processing, this problem can be alleviated by processing multiple requests in parallel. But in local deployment, single-user scenarios, GPU utilization is very low, latency high, and energy consumption large.

DiffusionGemma switches things up. It borrows the idea from Stable Diffusion, Midjourney image generation: start with a noise-filled canvas, then progressively denoise it until the content “develops”.

Specifically:

1. The model first places 256 random placeholder tokens on the “canvas”.
2. Through multiple rounds of denoising, the entire text block is processed at once.
3. Every token position sees all other positions on the canvas.
4. Until readable content emerges.

Processes 256 tokens at a time instead of 1 per step. This shifts the inference bottleneck from memory bandwidth to compute capability — and compute power is exactly what GPUs excel at.

Test Data: It’s Really Fast

Google and NVIDIA both released test data:

| Hardware | Throughput | |------|--------| | NVIDIA H100 (FP8) | 1000-1100+ tokens/s | | NVIDIA DGX Station | Up to 2000 tokens/s | | NVIDIA DGX Spark | ~150 tokens/s | | GeForce RTX 5090 | 700+ tokens/s |

By comparison, Gemma 4 with the same architecture hits around 303 tokens/s under the same conditions. DiffusionGemma is nearly 4× faster.

The key is the hardware threshold. DiffusionGemma has about 26B total parameters but uses a MoE (Mixture of Experts) architecture, activating only 3.8B parameters during inference. After quantization, the model takes only about 18GB of VRAM, so consumer GPUs like RTX 4090 and 5090 can run it locally.

The speed advantage fits specific use cases. Google states clearly in the docs:

• ✅ Local deployment, single-user, low-concurrency scenarios
• ✅ Latency-sensitive interactive applications
• ❌ High-QPS cloud-scale services (parallel decoding’s marginal benefit diminishes)
• ❌ Shared memory architectures (e.g. Apple Silicon Macs, where autoregressive models aren’t heavily bandwidth-bound)

In other words: this is a model specifically optimized for local inference.

Trading Quality for Speed: Google’s Own Low Score

4× faster — what’s the cost?

Google is upfront about it. The official docs include a comparison chart:

| Benchmark | DiffusionGemma | Gemma 4 | |----------|----------------|---------| | AIME 2026 (Math) | 69.1% | 88.3% | | MMMU Pro (Multimodal) | 54.3% | 73.8% | | Multilingual QA | Slightly lower | Higher | | Code generation | Slightly lower | Higher | | Scientific knowledge | Slightly lower | Higher |

Math ability is nearly 20 percentage points lower, multimodal about 20 points lower. This is not something fine-tuning can fix.

Google’s attitude is pragmatic:

This is an experimental model designed to “push speed to the extreme by deliberately compromising quality.” For high-quality, production-grade outputs, keep using Gemma 4.

But lower quality doesn’t mean useless. The diffusion architecture has strengths in some tasks that autoregressive models fundamentally struggle with.

Bidirectional Attention: Unique Advantages of Diffusion Models

In autoregressive text generation, each token can only “look left” — it sees all previously generated content, but not what comes next. This is fine for most tasks, but some inherently require “looking both ways.”

Sudoku is a typical example. Each cell is constrained by row, column, and box. An autoregressive model fills cell-by-cell left to right — if it later realizes an earlier cell is wrong, it’s too late to change.

DiffusionGemma’s diffusion architecture supports bidirectional attention — during denoising, each position on the canvas sees information from all other positions. If a token’s confidence drops, the sampler can revert it to noise and regenerate.

Google demonstrated a fine-tuned DiffusionGemma solving Sudoku: denoising across the entire board in parallel converges in a few steps, rather than filling cell-by-cell.

Similar tasks include:

• Code infilling: inserting content into existing code, needing to consider both before and after.
• Inline editing: modifying a sentence in the middle of a paragraph while keeping coherence.
• Markdown format closure: diffusion models can pair brackets/tags correctly in one go, rather than realizing halfway they haven’t closed them.
• Amino acid sequence generation: protein structural constraints are global.
• Math diagram construction: geometric relations require overall coordination.

Google calls these “non-linear text generation” tasks. This is where diffusion architectures truly shine, not just being “fast but rough.”

Technical Details: Block Autoregressive & MoE

DiffusionGemma is not purely a diffusion model — it uses a block-autoregressive strategy:

1. Generate a 256-token canvas.
2. Denoise it over multiple iterations until content forms.
3. Write these 256 tokens into the KV cache.
4. Generate the next canvas, continuing the text.

This preserves the parallelism advantage while enabling arbitrary-length generation.

Architecturally, DiffusionGemma is based on Gemma 4’s 26B parameter MoE setup. The core idea of MoE: the model contains multiple “expert sub-networks”, activating only the most relevant few for each inference, instead of running the entire model.

26B total parameters, only 3.8B active during inference. That’s the reason it fits into consumer GPUs.

NVIDIA also put effort into adaptation:

• Native support for NVFP4 (4-bit floating point), near-lossless precision with further throughput gains.
• CUDA stack ready to run on release day, no special optimization required.
• Hugging Face release includes both BF16 precision and NVFP4 lightweight editions.

Ecosystem Support: Ready Out of the Box

This release is highly complete. Google provided full ecosystem support:

• Hugging Face Transformers: standard integration.
• vLLM: OpenAI-compatible local deployment command — download weights and start service.
• Unsloth: efficient fine-tuning support.

NVIDIA made a free online demo available on build.nvidia.com on release day.

Model address: google/diffusiongemma-26B-A4B-it

For developers wanting local deployment, vLLM’s process is roughly:

1. Download model weights (~18GB quantized version).
2. Start vLLM service, specifying model path.
3. Call via OpenAI-compatible API.

Exact commands & parameters: see the Hugging Face model card for up-to-date instructions.

Competitive Landscape of Diffusion Text Generation

The idea of using diffusion models for text isn’t new, but has so far stayed in papers and closed-source products.

Inception Labs’ Mercury series has commercialized the fastest — Mercury 2 (Feb 2026) claimed “up to 5× acceleration for relatively speed-optimized models”, with early material even mentioning “up to 10× throughput.” But only the 1.3B parameter version was truly open-sourced, too small to be practical.

Google’s own Gemini Diffusion was demoed at Google I/O in May 2025, achieving 1479 tokens/s, faster than DiffusionGemma. But weights were never released, so developers couldn’t use it.

DiffusionGemma’s uniqueness is all three boxes ticked:

1. Produced by a leading lab (DeepMind)
2. Fully open weights (Apache 2.0)
3. Usable locally on consumer hardware (18GB VRAM)

This is the first time diffusion text generation tech has reached the developer community with such completeness.

Applicable Scenarios: When to Use It

According to Google’s guidelines and actual characteristics, DiffusionGemma suits these cases:

Recommended

• Local interactive apps: IDE plugins, CLI tools, local chat UIs.
• Fast iteration workflows: scenarios needing repeated attempts and instant feedback.
• Inline editing & code infilling: insertion into existing content.
• Latency-sensitive demos & prototypes: delivering “instant reply” experiences.
• Non-linear structure generation: Sudoku, crosswords, constraint-solving tasks.

Not Recommended

• Production-grade content output: high-quality required — use Gemma 4.
• High-concurrency cloud services: diffusion’s advantage diminishes at high QPS.
• Apple Silicon Macs: in shared-memory setups, advantage is negligible.
• Math reasoning & complex logic: explicit weakness here.

Worth Exploring

• Fine-tuned vertical tasks: Google’s Sudoku fine-tune demo suggests other constraint-solving tasks may yield surprises.
• Hybrid pipelines with autoregressive models: fast draft + high-quality refinement.
• Edge device deployment: 18GB VRAM allows more hardware choices.

Industry Significance

Diffusion models dominate image generation — Stable Diffusion, Midjourney, DALL-E 3 all use diffusion. But in text generation, autoregressive has been the only option.

DiffusionGemma represents the beginning of architectural diversity.

It won’t replace GPT, Claude, and other autoregressive models — at least not given the clear quality gap. But it proves:

1. Diffusion architectures are viable for text generation, with unique strengths.
2. Speed–quality trade-offs can be explicit — choose architecture per task.
3. Local deployment scenarios deserve dedicated optimization — not all models need to target the cloud.

For developers, this means one more tool in the toolbox. If your app scenario is “local, low-latency, quality drop acceptable,” now there’s a serious open-source option.

Google labels it “experimental”, but the release completeness (open weights, multi-framework support, consumer hardware usability) is beyond an experiment. It’s more like a proof-of-concept for the whole industry: diffusion text generation can be done like this.

Now it’s worth seeing if other vendors will follow.

---

References

• google/diffusiongemma-26B-A4B-it - Hugging Face - Model weights and official documentation
• Exploring a New Paradigm in Text Generation: Google Open Sources Experimental Diffusion Language Model with 4× Speed Boost - Zhihu - Technical principles explained in Chinese
• Google Deepmind just dropped DiffusionGemma - Reddit - Community discussion and feedback