Google launches DiffusionGemma: diffusion model breaks into text generation, local inference 4× faster

On June 11, Google DeepMind added DiffusionGemma to the Gemma family — an open-weight text model that doesn’t follow the autoregressive route, instead borrowing ideas from image diffusion: first laying down 256 placeholders on a "canvas," then iteratively denoising them over multiple rounds to make the text emerge piece by piece.

The official numbers are eye-catching: a single H100 breaks 1000 tokens/sec; an RTX 5090 can hit 700+ tokens/sec; on a DGX Station it can even reach 2000 tokens/sec — about 4× the speed of similarly sized autoregressive models. Licensed under Apache 2.0, weights are already available on Hugging Face.

But don’t rush to applaud — Google stated plainly in its announcement: this is an “experimental model that deliberately compromises quality to push speed to the limit.” In other words, it’s not meant to replace the standard Gemma 4 version — it’s exploratory.

From “typewriter” to “printing press”

To grasp the significance of this, you first need to see where autoregressive models get stuck.

GPT, Gemini, Claude, and other mainstream large models are essentially “typewriters” — they can only predict the next token after generating the previous one, strictly left-to-right. This paradigm is efficient on the cloud with large batches, where multiple requests can be combined to fill the GPU. But in local scenarios it’s awkward: single user, single request, tokens trickle out one by one, and most of the GPU’s time is spent waiting on memory bandwidth — the compute can’t be fully fed.

That’s why the bottleneck for running LLMs on consumer GPUs is often not FLOPs but HBM bandwidth. Your RTX 5090 has plenty of compute, but tokens still pop out one at a time.

DiffusionGemma takes a different approach. It processes an entire block of tokens (up to 256) in parallel in a single forward pass, gradually refining them through multiple denoising iterations. Google uses a fitting analogy: autoregression is a typewriter, diffusion is a printing press — the printing press stamps out a whole page at once.

This has several immediate benefits:

• Maxed-out parallelism: Tensor Core parallel compute finally runs at full capacity; NVIDIA highlighted this in its accompanying blog;
• Supports bidirectional attention: each token can see all other tokens in the paragraph during generation, no longer restricted to looking backwards;
• Built-in error correction: during iteration, the model can revise already generated content, instead of pushing ahead with mistakes.

The third point is especially interesting. Traditional autoregressive models, once they go off track, either barrel forward (hallucinations) or rely on post-hoc fixes like beam search. Diffusion models can reevaluate entire segments mid-generation, theoretically improving consistency.

26B parameters, only 3.8B active at runtime

Architecturally, DiffusionGemma has a nominal 26 billion parameters, but activates only about 3.8 billion per step — classic MoE design: multiple expert subnetworks run side-by-side, with the model picking a few as needed.

The result is: after quantization, it runs on high-end consumer GPUs with 18GB VRAM. Meaning a single 4090 (24GB) or 5090 can handle it just fine — no cloud, no 8-card server, no queues.

Google provided some benchmark results, which are mixed:

• Code: LiveCodeBench 30.9%, BigCodeBench 45.4%, HumanEval 89.6% — trading wins with Gemini 2.0 Flash-Lite;
• Math: AIME 2025 score 23.3%, actually surpassing the comparison model’s 20.0%;
• Science reasoning: GPQA Diamond only 40.4% vs. the comparison model’s 56.5%;
• Complex reasoning: BIG-Bench Extra Hard 15.0% vs. 21.0% for its rival.

Clearly, the diffusion architecture shows potential in structured tasks (code, math), but still struggles with long-chain reasoning and world knowledge retrieval — consistent with diffusion models’ strengths in parallel local correction and weaknesses in strict causal chains.

Diffusion text isn’t new, but this is different

Truth be told, text diffusion has been explored for years. Stanford’s Diffusion-LM, some of Meta’s efforts, ByteDance’s similar research last year. It never became mainstream — for obvious reasons: natural language relies heavily on grammar order, context coherence, and factual constraints. Small pixel noise in an image is overlooked, but a misplaced token can ruin a sentence.

The surprise with DiffusionGemma is that it’s the first time a major company has run this path to a usable point with industrial-scale weights — and clearly defined the boundary conditions for when its speed advantage holds.

Google unusually listed both suitable and unsuitable scenarios:

Suitable for:

• Local, single-user, dedicated GPU low-latency interactions;
• Inline text editing, code completion, workflows needing rapid repeated trials;
• Nonlinear text generation — e.g., filling code in the middle, generating amino acid sequences, constructing mathematical figures requiring bidirectional context.

Unsuitable for:

• High-QPS large-scale cloud services — parallel decoding’s marginal benefit quickly drops;
• Shared-memory architectures (e.g., Apple Silicon Macs) — autoregression isn’t bandwidth-limited here, so diffusion’s acceleration effect is reduced;
• Scenarios demanding very high text quality output — stick to Gemma 4 standard version.

Translated bluntly: DiffusionGemma isn’t a universal solution — it’s a scalpel for inline coding assistants, local IDE plugins, and privacy-sensitive desktop apps.

How to interpret the speed data

NVIDIA measured: ~1000 tokens/s on H100, ~2000 tokens/s on DGX Station (other sources cite 800), ~150 tokens/s on DGX Spark, over 700 tokens/s on RTX 5090. Sampling speed 1479 tokens/s, with initial overhead of 0.84s.

Key point: the 4× speedup is in single-request, dedicated GPU scenarios. Comparing this to cloud-side vLLM with full batch throughput is unfair and pointless. Diffusion’s advantage is maximizing single-stream speed; cloud batch jobs still require autoregression.

Also note the 0.84s initial overhead — 256 placeholders iterated together means that even if you only want to generate one sentence, minimum latency won’t drop further. For short outputs, autoregression’s first-token latency may actually be better.

What this means for developers

Broadly speaking, this is a rare non-incremental exploration at the large model architecture level. Over the past two years, competition has been focused on MoE, context length, RL post-training — all with the same Transformer autoregressive backbone. DiffusionGemma may not overturn anything, but it at least proves: in certain scenarios, there is indeed a 4× speed-up engineering path.

For developers, noteworthy directions include:

1. Local AI toolchains: Editor assistant tools like Cursor, Continue, Cline suffer from cloud latency and privacy issues. A model running at 700 tokens/s locally on a 4090 is game-changing for real-time completion.
2. Fill-in-the-blank tasks: Bidirectional attention naturally suits fill-in-the-middle, more elegantly than simulating it via FIM tokens in autoregression.
3. Structured generation: JSON, SQL, tables, mathematical notation — outputs with clear structural constraints — benefit from diffusion’s global view.
4. Agent internal loops: Agents often need short, fast tool-call decisions; diffusion’s high throughput plus self-correction could be a good match.

But don’t ignore the drawbacks: quality is reduced, long-text consistency isn’t fully validated, and factual accuracy still awaits independent community evaluations. Before deploying to production, run your own A/B tests with real workloads.

A thread to watch

Notably, Reddit users pointed out that NVIDIA released a diffusion model around the same time. Combined with Gemini Diffusion’s research — DiffusionGemma is clearly the open-source version of that line — it’s evident that Google and NVIDIA both see this as a serious track.

If text diffusion can close the quality gap to within 5% of same-size autoregressive models in 6–12 months, the landscape of local AI deployment might need rewriting. If not, it will remain in the “local IDE plugin” niche as a high-speed supplement.

DiffusionGemma’s weights are on Hugging Face, Apache 2.0 licensed, ready to download. OpenAI Hub (openai-hub.com), as a unified key-based aggregator for mainstream models, will follow with support for this open-source model ecosystem — already covering Gemma series, Claude, GPT, DeepSeek, etc.

When it comes to speed, developers will vote with their feet.

References

• Reddit r/LocalLLaMA: DiffusionGemma 4x faster text generation discussion — first wave of community tests and comparisons to NVIDIA’s concurrent diffusion model
• IT Home: Google launches DiffusionGemma text diffusion model — Chinese media summary of the launch event with detailed benchmark data
• Hugging Face: DiffusionGemma model weights — model weight download and official model card (Apache 2.0 license)