Diffusion Models Enter Text Generation: Google Open Sources DiffusionGemma, 4× Speed Boost

This month, Google DeepMind open-sourced DiffusionGemma, an experimental model that uses diffusion techniques for text generation. In tests on a single H100, output speed exceeded 1000 tokens/s, 4× faster than autoregressive models of the same class.

This isn’t just a minor tweak. It directly challenges the underlying architectural logic of mainstream large models like GPT, Claude, and Gemini.

The Autoregressive Bottleneck: The Typewriter Problem

Let’s spell out where the problem lies.

Almost all current mainstream LLMs are built on autoregressive architecture—the model works like a typewriter, generating tokens one by one from left to right. GPT‑4o, Claude 3.5, and Gemini 2.5 all follow this route.

This architecture is highly efficient when doing batch inference in the cloud, since multiple requests can be batched for processing together. But when running locally on a single machine, the problems become apparent:

• High latency: Generating 1000 tokens requires 1000 forward passes
• Low GPU utilization: With a single request, much of the compute power sits idle
• Energy waste: Each token requires a full attention computation

In simple terms, autoregressive models are designed for large-scale cloud services. When used for local real-time interaction, they are at a disadvantage.

The Diffusion Approach: From Typewriter to Printing Press

DiffusionGemma takes a different approach.

It borrows from the diffusion model paradigm used in image generation—think Stable Diffusion, DALL‑E 3—but replaces continuous pixel space with discrete token space. Google calls it “Discrete Text Diffusion.”

How it works in practice:

1. Initialization: Place 256 random placeholder tokens on a “canvas”
2. Iterative denoising: Through multiple forward passes, progressively convert noisy tokens into meaningful words
3. Parallel output: Each forward pass processes the entire 256‑token block at once

Analogy: Autoregressive models are typewriters, producing one letter at a time; DiffusionGemma is a printing press, outputting 256 characters in a batch.

This brings a key advantage—maximum parallelism. A single forward pass produces an entire chunk of text, allowing the GPU’s parallel compute capacity to be fully utilized.

Test Data: It's Really Fast

NVIDIA ran a benchmark:

| Hardware | Single-request output speed | |----------|-----------------------------| | H100 | ~1000 tokens/s | | DGX Station | ~800 tokens/s | | GeForce RTX 5090 | >700 tokens/s | | DGX Spark | ~150 tokens/s |

For comparison, autoregressive models of similar parameter count achieve about 200–300 tokens/s on an H100. DiffusionGemma is 3–4× faster.

The RTX 5090 results are notable: consumer GPUs reaching 700 tokens/s means local developers can enjoy smooth interaction without renting a cloud GPU.

Architecture Details: MoE + Quantization Down to 18GB VRAM

DiffusionGemma nominally has 26 billion parameters, but only 3.8 billion are active during inference.

This is thanks to its MoE (Mixture of Experts) architecture. The model contains multiple “expert” subnetworks, and routing is dynamic—only the most relevant experts are activated for computation, while others remain idle.

Combined with quantization, the entire model can fit into 18GB of VRAM. This means high-end consumer GPUs like the RTX 4090 and 5090 can run it—no need for professional A100/H100 hardware.

Another highlight is bidirectional attention. Autoregressive models can only look forward—when generating the Nth token, they can only reference N‑1 preceding tokens. DiffusionGemma can view the entire 256‑token window at once during generation.

This is particularly useful for certain tasks:

• Inline editing: Modify a word in the middle without regenerating an entire paragraph
• Code completion: Fill in logic in the middle of a function, referencing both preceding and following context
• Structured generation: JSON, tables, and other non-linear formats

Trading Quality for Speed: The Cost You Need to Know

In its announcement blog, Google unusually poured cold water on expectations:

“This is an experimental model deliberately compromising quality to push speed to the limit.”

Plainly put: DiffusionGemma’s output quality is inferior to the standard Gemma 4.

This is not a bug—it’s a design trade-off. The iterative denoising process of diffusion models inherently introduces “fuzziness”—each step is probabilistic, so the final output is less deterministic than autoregressive generation.

Specific effects:

• Accuracy drops in complex reasoning tasks
• Long-text coherence is weaker than autoregressive models
• Possible hallucinations in domain-specific knowledge

Google’s advice is straightforward: for production outputs and scenarios requiring high quality, stick with the standard Gemma 4. DiffusionGemma is suited for:

• Rapid prototyping and iteration
• Draft generation for manual refinement
• Real-time interaction with strict latency requirements
• Localized development and debugging

Another Limitation: Cloud Batch Scenarios Lose the Edge

Its speed advantage diminishes in certain scenarios.

As noted earlier, DiffusionGemma’s acceleration comes from generating entire token blocks in parallel. But in cloud services handling many simultaneous requests, autoregressive models can batch tokens from multiple requests together, similarly maximizing GPU utilization.

In such cases, DiffusionGemma’s parallel decoding advantage is diluted. Google is explicit: “In high-QPS large-scale cloud services, DiffusionGemma’s advantage declines rapidly.”

Another mediocre performance scenario: Apple Silicon Macs.

With a unified memory architecture, CPU and GPU share memory bandwidth; autoregressive models are not heavily constrained by VRAM bandwidth here. DiffusionGemma’s acceleration gains are less pronounced than on discrete GPUs.

Toolchain Support: Already Running

For open-source models, ecosystem support is key. DiffusionGemma launched with support for three mainstream frameworks:

1. Hugging Face Transformers

The official model is already on Hugging Face Hub and can be loaded directly with the transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("google/diffusiongemma-26B-A4B-it")
tokenizer = AutoTokenizer.from_pretrained("google/diffusiongemma-26B-A4B-it")

2. vLLM

The vLLM team has released a DiffusionGemma-adapted version; Red Hat has provided quantized model weights for easier enterprise deployment.

3. MLX

Apple Silicon users can run locally via the MLX framework. While unified memory architecture limits acceleration compared to discrete GPUs, it’s still functional for development and testing.

Technical Significance: More than Just Another Model

DiffusionGemma’s importance lies beyond “one more open-source model.” It validates a different technical path.

In recent years, nearly all major breakthroughs in the LLM space have happened within the autoregressive framework: bigger models, longer context, better alignment. Diffusion models have been seen as exclusive to image generation.

Google has now shown: diffusion can be used for text, and in certain scenarios, it’s better suited than autoregressive approaches.

This opens a door.

We can expect more teams to explore this path:

• Can diffusion models improve quality while maintaining speed?
• Can they be hybridized with autoregressive models to combine strengths?
• Can they outperform in specific tasks like code generation and structured data output?

What It Means for Developers

Practical tips:

Local development and debugging: DiffusionGemma is currently one of the best choices. 700+ tokens/s means you can quickly iterate prompts and test different approaches without waiting.

Production deployment: Hold off for now. The quality gap is evident; unless your scenario requires extremely low latency and can tolerate lower output quality, stick to other options.

Research and exploration: Worth digging into. Features like bidirectional attention and iterative denoising may offer surprises in certain tasks.

Hardware choice: If you have an RTX 5090 or equivalent GPU, DiffusionGemma is a great way to leverage its compute power. If you only have a Mac, expect diminished results.

In Closing

Trading quality for speed—Google has been very honest with this choice.

They’re not hyping DiffusionGemma as the next GPT-killer, but clearly positioning it as an “experimental model” for specific needs in specific contexts. Such restraint is increasingly rare in AI.

For developers, more options are always good. Autoregressive isn’t the only answer; diffusion can also do text. The future may bring better architectures combining the best of both.

The model is open-sourced under the Apache 2.0 license—those interested can try it right away.

---

References

• google/diffusiongemma-26B-A4B-it - Hugging Face – official model weights and usage docs
• Exploring a New Paradigm for Text Generation: Google Open Sources a 4× Faster Experimental Diffusion Language Model - Zhihu – technical principles explained in Chinese