Diffusion Models Break into Text Generation, Google Bets on Speed over Quality

On June 10, Google DeepMind launched DiffusionGemma on Hugging Face under the Apache 2.0 license, with full weight access.

This is not an upgrade of Gemma 4. Google’s official guide states plainly: it’s not as good as their older model.

When a big company releases a new model, the usual pitch is "stronger," "faster," "leading in all aspects." This time, Google took the opposite approach and admitted inferiority. What’s the reasoning behind this?

Not a Typewriter, but Developing a Photograph

Today’s mainstream large language models—GPT, Claude, Gemini—are all autoregressive architectures. Their text generation method is like a typewriter: from left to right, one token at a time, and once the word is output, it can't be changed. If there’s an error at the beginning, it can persist until the end.

DiffusionGemma takes a different approach.

It borrows the concept of "diffusion models" from image generation: first lay down a “canvas” of 256 random placeholder tokens, then perform multiple rounds of parallel denoising to develop the whole paragraph at once.

This analogy comes from Sundar Pichai himself. He posted on X comparing DiffusionGemma to "a racehorse": winning speed by generating entire chunks of text simultaneously, rather than predicting word by word.

During denoising, each position on the canvas can see all other positions. If the confidence of a token drops, the sampler can revert it to noise and rewrite it. This is the natural advantage of bidirectional attention—traditional autoregressive models can only look left and can’t revise already output text.

A tangible benefit: complex Markdown formats can be completed and closed in one go, rather than noticing halfway through that brackets don’t match.

How 1100 tokens/s Was Achieved

Official benchmarks:

• Single H100 (FP8, low batch): 1107 tokens/s
• Consumer-grade RTX 5090: 700+ tokens/s
• DGX Station: up to 2000 tokens/s
• DGX Spark: 150 tokens/s

Under the same conditions, Gemma 4 achieves 303 tokens/s. DiffusionGemma is nearly 4× faster.

This speed advantage comes not from magic, but from the physical characteristics of GPUs.

Token-by-token autoregressive inference is essentially memory-bound: for each word output, the GPU must fetch the model weights from VRAM. Most of the time is spent waiting for data transfers, while computational units remain idle.

DiffusionGemma pushes 256 tokens through the transformer in parallel, turning it into a compute-bound task. This fully engages NVIDIA’s Tensor Cores—normally underutilized in local single-user inference.

In other words, this model’s design is almost tailor-made to play to the GPU’s strengths.

NVIDIA prepared support right on launch day. Hugging Face offers two versions:

• Standard (BF16): high precision
• Lightweight (NVFP4): 4-bit floating-point, smaller and faster

After quantization, DiffusionGemma only requires ~18GB of VRAM. Gaming cards like RTX 5090 and 4090 can run it locally.

Speed’s Price: Lagging Quality Benchmarks

Google’s self-admitted inferiority is not politeness—it’s real.

| Benchmark | DiffusionGemma | Gemma 4 | |-----------|----------------|---------| | AIME 2026 (Math) | 69.1% | 88.3% | | MMMU Pro (Multimodal) | 54.3% | 73.8% | | GPQA Diamond (Scientific Reasoning) | 40.4% | 56.5% | | BIG-Bench Extra Hard | 15.0% | 21.0% |

The biggest gap is in math—nearly 20 percentage points. Multimodal and scientific reasoning also lag significantly.

Code generation shows mixed results: LiveCodeBench 30.9%, BigCodeBench 45.4%, HumanEval 89.6%, roughly tied with Gemini 2.0 Flash-Lite.

Google’s positioning is clear: this is an experimental model deliberately sacrificing quality to maximize speed.

Suitable for:

• Local interaction, inline editing
• Rapid iteration draft generation
• Non-linear text structures (e.g. Sudoku, code infilling)

Not suitable for:

• High-quality production output (Google recommends continuing to use Gemma 4)
• Large-scale cloud services, high QPS scenarios (parallel decoding’s marginal benefit drops quickly)
• Apple Silicon Macs (shared memory architecture means autoregressive models aren’t as bandwidth-limited to begin with)

Sweet Spot for Bidirectional Attention: Sudoku and Code Infilling

While overall quality lags, DiffusionGemma excels in certain tasks.

Sudoku is a classic example. Each cell is constrained by row, column, and box—requiring bidirectional consistency. Token-by-token models struggle—they can only fill from left to right, unable to revise past choices.

Fine-tuned DiffusionGemma solves Sudoku not cell-by-cell, but by parallel denoising the entire board, converging in just a few steps.

Code infilling is similar. Filling in a function in the middle of code requires attention to both preceding and following context. The global scope of diffusion makes this natural compared to the typewriter approach.

These tasks with strong bidirectional constraints are precisely where token-by-token models suffer and bidirectional attention shines.

Architecture Details: MoE + Block Autoregression

DiffusionGemma is based on Gemma 4’s 26B A4B Mixture of Experts (MoE) architecture:

• Total parameters: ~25.2B
• Active during inference: ~3.8B

Nominally 26B parameters, but only 3.8B used at each step. MoE uses multiple expert sub-networks in parallel, and the model selects only the necessary ones, avoiding full-model activation.

Generation uses block-autoregressive: denoise a 256-token canvas, write into KV cache, then start a new canvas to continue generating.

This design combines diffusion’s parallel advantage with autoregressive’s long-form capabilities.

Ecosystem Support: Ready Out of the Box

Google didn’t just release weights—they shipped the full toolkit:

• Hugging Face Transformers: native support
• vLLM: OpenAI-compatible local deployment commands provided
• Unsloth: supports fine-tuning

With vLLM deployment, download the weights and start a local service:

python -m vllm.entrypoints.openai.api_server \
    --model google/diffusiongemma-26B-A4B-it \
    --dtype bfloat16 \
    --max-model-len 8192

NVIDIA also offers a free online entry at build.nvidia.com for direct testing.

Diffusion for Text Generation Isn’t New—But This Is Different

Diffusion models for text generation have been explored before.

Inception Labs released its Mercury series in February 2026, claiming up to 5× acceleration for speed-optimized models. Early materials and third-party reviews even mentioned “up to 10× throughput.”

But Mercury’s truly open-source variant was only 1.3B parameters—quite small.

Even Google itself released Gemini Diffusion at its May 2025 I/O, hitting 1479 tokens/s—but weights were never made public.

DiffusionGemma differs in that it combines:

1. From a cutting-edge lab (Google DeepMind)
2. Open-source weights (Apache 2.0)
3. Runs locally on consumer GPUs (18GB VRAM)

For the first time, developers can run a diffusion text model from a top lab on their own 4090.

What This Means for Developers

A New Local Inference Option

If your scenario is:

• Single-user, single-machine
• Latency-sensitive
• Quality not top priority

DiffusionGemma might be the fastest open-source option today.

New Material for Architecture Research

Diffusion text generation has been a hot academic topic, but lacks large-scale open-source models for benchmarking. DiffusionGemma provides researchers with a reproducible, fine-tunable starting point.

Limited Impact on Cloud Services

In high-concurrency, high-QPS cloud scenarios, parallel decoding’s marginal benefits diminish quickly. Autoregressive models already batch requests efficiently to leverage GPU compute. DiffusionGemma’s speed advantage is diluted here.

What Is Google Betting On?

Why release a model that’s “not as good as the old one”?

My assessment: Google is betting on architectural diversity.

Autoregressive models have dominated text generation for years. They’re mature, stable, and well-supported—but inherently serial, leading to bottlenecks in some scenarios.

Diffusion models offer an alternative path. They may not replace autoregressive models, but could be better in specific cases.

By open-sourcing this experimental model now, rather than waiting for “perfection,” Google invites community collaboration to explore this direction.

This mirrors Llama’s strategy: open-source early, build the ecosystem, claim the space. Quality can improve over time, but ecosystems take time to establish.

Limitations and Outlook

Current limitations:

• Significant lag in hard metrics like math and scientific reasoning
• Limited advantage in high-concurrency scenarios
• Minimal acceleration on shared memory architectures like Apple Silicon

Promising directions:

• Ongoing improvement in quality benchmarks (future versions should address this)
• Further optimizations for diffusion architecture (sampling strategies, denoising schedules)
• Exploring hybrid architectures with autoregressive models

In Closing

DiffusionGemma is not a “better” model—it’s a “different” model.

It trades quality for speed, precision for parallelism, certainty for experimentation. Not everyone needs this trade-off—but for certain scenarios—local interaction, rapid iteration, non-linear generation—it may be the best option currently.

Google’s candid admission inspires more goodwill toward the project. Compared to overhyped releases that disappoint, honestly stating “it’s not as good as the old one, but it’s fast” is a pragmatic approach worth respecting.

Will diffusion models secure a foothold in text generation? That remains open. But at least, Google has laid its cards on the table.

---

References:

• DiffusionGemma Model Page - Hugging Face: official model weights download
• Exploring New Paradigms in Text Generation - Zhihu Column: technical principle explanation