Google Open Sources DiffusionGemma: Diffusion Models Break into Text Generation, Speed Soars 4×

Late at night on June 10, Google DeepMind quietly dropped a bombshell into the Gemma family — DiffusionGemma. This 26-billion-parameter MoE model, open sourced under the Apache 2.0 license, does not follow the well-trodden “autoregressive typewriter” path of GPT or Gemini, but instead pulls the diffusion model approach—which has been hugely successful in the image generation domain—straight into the text generation arena. The trade-off is slightly compromised output quality, but the payoff is stunning speed: over 1,000 tokens per second on a single H100, and over 700 tokens/s on an RTX 5090, representing up to a 4× speed boost compared with similarly sized autoregressive models in low-latency local scenarios.

Google CEO Sundar Pichai put his stamp on it on X with a playful remark: “Fast as a racehorse 🏇.” DeepMind’s official blog was unusually blunt — “We want to push generation speed to the limit.”

1. No More “Letter-by-letter”: How Diffusion Models Rewrite Text Generation Paradigms

To understand the disruptive nature of DiffusionGemma, we first need to review how mainstream large models work today.

GPT-5, Claude, Gemini, DeepSeek… almost all “conversational” LLMs are built on the autoregressive framework. They function like an old-fashioned typewriter: the first token must be produced before it can be added back into the context to predict the second token; after the second comes the third… repeating this strictly serial left-to-right sequence.

This paradigm works brilliantly in high-concurrency cloud scenarios — the server can batch thousands of user requests together, saturating the GPU to maximum utilization. But in single-user local inference, problems appear immediately:

• GPU/TPU spends most of the time “waiting for the next key,” wasting compute resources
• The decoding bottleneck is memory bandwidth, not actual compute units
• If users write something and edit a sentence, the model often regenerates the entire section, creating discontinuity
• Even strong edge hardware can’t squeeze out full throughput

DiffusionGemma flips this: it spreads 256 random placeholder tokens across the “canvas” in one go, then iteratively denoises — locking in accurate tokens as context, continuously refining the remaining “noise tokens,” layer by layer until the whole text converges to a coherent final result.

This logic mirrors Stable Diffusion generating images from random noise — except pixels are replaced by text tokens. Each forward step can process all 256 tokens in parallel; each token can “see” all others; the hardware receives a large compute block each time, not a lonely single-token prediction.

2. 26B Total Parameters, 3.8B Activated: MoE Cuts Cost to 18GB VRAM

Another keyword for DiffusionGemma is MoE (Mixture of Experts).

The official parameter configuration is as follows:

• Total parameters: ~26B
• Active per step: ~3.8B
• Parallel token block size: 256
• VRAM usage after quantization: ~18GB
• License: Apache 2.0

MoE’s core idea is “multiple expert subnets in parallel, model calls them on demand,” avoiding waste by not activating all 26B parameters every step. This allows DiffusionGemma, after quantization, to fit in a high-end consumer GPU — yes, RTX 4090 / 5090 can run it — making it highly friendly for independent developers and small teams.

DeepMind also worked with NVIDIA to optimize across the entire hardware stack, with native NVFP4 kernel support to maximize performance on Hopper (H100, H200) and Blackwell (B200, RTX 50 series) architectures, along with adaptations for desktop DGX Spark, DGX Station, and professional RTX PRO series.

NVIDIA official benchmark results:

| Hardware | Single-request generation speed |
| --- | --- |
| NVIDIA H100 | ~1,000+ tokens/s |
| DGX Station | ~800 tokens/s |
| GeForce RTX 5090 | ~700+ tokens/s |
| DGX Spark | ~150 tokens/s |

Compared to the 100–200 tokens/s typical for Llama 3, Gemma 2, and other autoregressive models of similar size on a single consumer GPU, DiffusionGemma shows a clear 3–4× speed advantage in local inference scenarios.

3. More than Just “Fast”: New Use Cases Unlocked by Bidirectional Attention

Speed is the surface story — the real intrigue is that DiffusionGemma naturally fits certain tasks that traditional autoregressive models struggle with.

1. In-line Editing and Real-time Completion

When users edit partway through writing, autoregressive models regenerate everything after the edit point; DiffusionGemma treats the entire text block as a refinable canvas and can patch edit in place, ideal for IDE code completion, live Markdown rendering, and in-line text polishing.

2. Intelligent Self-correction

The model can scan the entire output at once during iteration, detecting contradictions and correcting them immediately, rather than “living with” mistakes like autoregressive models do. This benefits long-form logical consistency and factual constraint.

3. Non-linear Structure Generation

Code blocks, mathematical graphs, amino acid sequences, SVG — these “later tokens constrain earlier tokens” structures are traditionally weak spots for autoregressive models. DiffusionGemma’s bidirectional attention thrives here. Hugging Face has already released a demo showing the model generating 3D SVG graphics directly from text descriptions.

4. Globally Constrained Tasks like Sudoku

Open-source finetuning framework Unsloth finetuned DiffusionGemma into a Sudoku solver. In Sudoku, each cell is interdependent with all others — autoregressive models mostly guess blindly; DiffusionGemma elegantly handles such global constraints with its bidirectional attention.

4. Honest Trade-offs: Where Not to Use It

This time DeepMind was unusually upfront: DiffusionGemma is an experimental model built deliberately for speed at the expense of quality, overall output quality lower than standard Gemma 4.

The official boundaries are clear:

Suited for:

• Local, low-concurrency, single-user real-time interaction
• In-line editing and code completion in IDEs
• Local workflows requiring repeated fast iteration
• Non-linear structured text (SVG, Markdown, Sudoku, math graphs, etc.)

Not suited for:

• High-QPS large-scale cloud services: Autoregressive models can already batch GPUs fully; parallel decoding gains diminish in this context, possibly raising cost per token
• Production environments requiring very high text quality: Despite ~3.65× single-request speed boost over Gemma 4, quality is slightly worse — standard Gemma 4 is recommended for formal products
• Shared memory architectures (e.g., Apple Silicon Mac): M-series chips aren’t bandwidth-limited; autoregressive models already run fast in unified memory, meaning DiffusionGemma’s speedup is much smaller

This “clearly state unsuitable scenarios first” approach is rare in big-company model releases, showing DeepMind has a pragmatic intent — this is aimed at researchers and enthusiasts exploring the frontier, not at immediately replacing existing tools.

5. Model Weights and Getting Started

Weights are available on Hugging Face; the instruction-tuned version is DiffusionGemma 26B A4B-it (26B total parameters, ~3.8B active per step, instruction-tuned). The Transformers library already includes relevant support, and the community can run it now.

# Download model weights
huggingface-cli download google/diffusiongemma-26B-A4B-it

# Install latest transformers (with diffusion head support)
pip install -U transformers accelerate

Recommended local hardware requirements:

• Minimum: RTX 4090 (24GB VRAM), NVFP4 quantization required
• Recommended: RTX 5090 (32GB VRAM) for higher precision
• Enterprise-grade: H100 / H200 / B200, capable of 1,000+ tokens/s per card

6. How Far Can Diffusion Text Models Go This Time?

Diffusion models entering text generation isn’t new. In works like Diffusion-LM, SSD-LM, researchers repeatedly attempted to transfer image diffusion’s success to text — but natural language’s heavy reliance on grammatical order, contextual coherence, and factual constraints kept autoregressive models dominant.

Fast generation doesn’t equal stable writing — this is why diffusion text models haven’t gone mainstream for years.

DiffusionGemma’s significance lies not in toppling GPT instantly, but in being the first open-weight industrial-scale model to show tangible speed advantages with the diffusion approach, bringing this path out of academic papers and onto an RTX 4090. Whether it matches autoregressive models in long-text generation, complex QA, or factual accuracy still awaits independent community evaluation.

One thing is clear:

As demand grows for edge AI, local deployment, and privacy-sensitive scenarios, “finding a new balance between generation quality, inference speed, and hardware cost” is becoming one of the most important competition dimensions for the next phase of large models.

Autoregressive won’t be replaced, but monopoly of a single architecture may be loosening. DiffusionGemma is more like Google throwing out a question — giving developers a 26B toy to play with now, and leaving the industry with a topic worth long-term exploration.

At least for tonight, those huddled in front of a 4090 waiting for tokens can enjoy the thrill of 1,000 tokens/s.

References

• DiffusionGemma 26B A4B-it Model Weights (Hugging Face) — Google’s officially open-sourced instruction-tuned version, ready for download & deployment
• Hugging Face Transformers Repository — DiffusionGemma’s diffusion head support included, main dependency for local inference