Gemma 4 adds MTP: inference speed doubles instantly

Yesterday (May 4), Google released a companion update for Gemma 4 on the Developer Blog: a set of draft models (drafters) specifically designed for Multi‑Token Prediction (MTP), to be used with the main model for speculative decoding. Benchmarks show that inference throughput in long‑context and code‑generation scenarios can be doubled or even exceed that.

The interesting part is not MTP itself—DeepSeek‑V3 had already mastered MTP by the end of 2024, and Meta applied similar ideas in Llama 4. What’s really noteworthy is the timeline: when Gemma 4 was released on April 2, sharp‑eyed developers on Hugging Face noticed that the model weights actually contained MTP‑related tensors, just disabled. A Reddit discussion eventually drew a response from a Google employee confirming that “MTP does exist but was deliberately removed for compatibility and broader usability.”

A month later, Google reattached the missing parts—but this time as a separate drafter, leaving the main model weights untouched. That’s actually a smarter move than “turning on native MTP directly”—and we’ll see why below.

What exactly MTP accelerates

Let’s clarify the principle first; otherwise, the numbers won’t make sense.

Standard autoregressive generation produces tokens one by one—each forward pass emits a single token. The bottleneck lies not in computation but in memory bandwidth—for every token generated, tens of GB of weights have to be moved from HBM to SRAM. Even if you fully load an H100 with a 31B model, much of the GPU time is spent waiting for data.

The idea of speculative decoding is that a small, fast draft model first guesses N tokens in a row, then the large model verifies all N guesses in parallel at once. If k of them are correct, that forward pass effectively produces k + 1 tokens. Verification is parallel, so for the large model it costs almost the same as generating one token.

Traditionally, an independent small model (e.g., Gemma 2B) serves as the drafter. The problem is that the small and large models have misaligned distributions, so the hit rate is often only about 50%, heavily discounting speed‑up.

MTP takes a different approach: during training, the main model predicts the next 2–4 positions simultaneously, with each future position sharing the same backbone representations and only adding lightweight prediction heads. This type of drafter shares almost all semantic understanding with the main model, yielding hit rates above 80% while remaining very lightweight (usually just a few hundred M parameters).

The drafter Google released uses exactly this structure. According to the blog data, when attached to the Gemma 4 31B main model:

• Code completion: end‑to‑end latency ↓ ≈ 2.1×
• Long‑document summarization: throughput ↑ ≈ 1.8×
• Conversational tasks: 1.5–1.7× acceleration
• Mathematical reasoning (Chain of Thought): strongest gain ≈ 2.3×

The last one is notable. CoT reasoning involves many formatted tokens (e.g., Step 1:, Therefore, numbers, formula symbols). Such structured content yields extremely high prediction hit rates, amplifying MTP’s benefits. That also explains why Gemma 4 focuses on reasoning and agent workflows—MTP fits these scenarios perfectly.

Why it was removed at the April release

Back to the mystery. Google said it removed MTP for “compatibility and broad availability.” That may sound like PR talk but actually makes technical sense.

A natively MTP‑enabled main model has multiple prediction heads and additional training objectives built into its weights. That means:

1. Inference frameworks must change. Popular engines—vLLM, SGLang, llama.cpp, MLX, Ollama—organize KV caches and sampling loops for single‑token prediction; native MTP demands modified scheduling. Since Gemma targets “runs on laptops and phones,” releasing it with built‑in MTP would have broken half its target user base on day one.
2. Quantization toolchains would break. Community tools like GGUF, AWQ, GPTQ are optimized for standard Transformer structures; extra heads would hit a host of edge cases.
3. Finetuning ecosystems wouldn’t know how to handle it. Both LoRA and full‑parameter recipes assume single‑token loss.

So Google’s strategy was: release a standard‑architecture Gemma 4 first to get the ecosystem running; then a month later release separate drafters for those who want acceleration. Because drafter and main model are decoupled, inference engines only need generic speculative‑decoding support (vLLM already has it) and require no Gemma‑specific adaptation.

This separation does come at a price—the independent drafter’s hit rate is slightly lower than true native MTP (Google’s internal data: 90%+, public drafter: ≈ 78%)—but in exchange you get zero‑migration‑cost deployment. For the open‑source ecosystem, that’s a more responsible choice.

Drafter specs and usage

Google released two drafters:

• Gemma 4 Drafter‑S – ≈ 300 M parameters, for Gemma 4 E2B / E4B
• Gemma 4 Drafter‑L – ≈ 1.2 B parameters, for Gemma 4 31B / 26B A4B

The drafter is trained using hidden‑state distillation rather than plain logits distillation, which is key to approaching native MTP hit rates. According to Google, the drafter uses the penultimate hidden layer of the main model as an extra supervision signal, so it learns not only “what the next word is” but also “what the main model is thinking right now.”

Deployment‑wise, vLLM 0.7+ and the latest SGLang already support this natively—just add
--speculative-model google/gemma-4-drafter-l --num-speculative-tokens 4
to your launch command; no business‑logic changes needed.

llama.cpp support is still being merged; community PRs are up and it should land within a week.

Placing it in the industry context

Compare Gemma 4 + MTP drafter with other open‑source models:

| Model | Params | MTP Support | Typical Speed‑up |
| --- | --- | --- | --- |
| DeepSeek‑V3 | 671 B (37 B active) | Native MTP | ≈ 1.8× |
| Llama 4 70 B | 70 B | Independent drafter | ≈ 1.6× |
| Qwen3 32 B | 32 B | Independent drafter | ≈ 1.5× |
| Gemma 4 31 B | 31 B | Independent drafter | ≈ 2.0× |

Gemma 4 indeed leads speed‑ups among open‑source peers of similar scale, thanks to hidden‑state distillation and its structured‑output bias toward reasoning tasks.

But let’s not overhype it. MTP’s benefit heavily depends on the task type and batch size. With larger batches (e.g., many concurrent requests on servers), the GPU becomes compute‑bound instead of bandwidth‑bound, and the MTP gain quickly drops to ≈ 1.2×. That’s the mathematical fate of speculative decoding—independent of implementation.

So MTP mainly benefits:

• On‑device / edge deployment – batch size = 1, most bandwidth‑limited, biggest gains
• Low‑concurrency internal enterprise apps – same reason
• Interactive scenarios needing low first‑token latency – e.g., code completion, chats

For high‑concurrency API services, MTP is more of a latency optimization than a throughput one—you need to know which metric you care about.

A signal many overlooked

Stepping back, Google’s choice to release a drafter instead of re‑issuing a natively MTP Gemma 4 sends a signal: open‑source release strategies are diverging from closed‑source ones.

Closed‑source models (GPT‑5, Claude, Gemini) don’t have to consider community toolchains—they can use any architecture because they control the entire inference stack. Open‑source models must remain compatible with community ecosystems; any “advanced feature” has to account for its adoption cost. DeepSeek can ship native MTP because it’s server‑oriented; Gemma’s audience includes many laptop and phone users—much heavier constraints.

Google’s “standard model first, add drafter later” rhythm resolves that tension and shows mature product thinking. It may well become the new norm for open‑source model releases—launch the base model to gain adoption, then deliver advanced features as plug‑ins.

For developers, if you want to use Gemma 4 without the hassle of local deployment, OpenAI Hub already hosts the entire Gemma 4 lineup in an OpenAI‑compatible API format; server‑side speculative decoding acceleration is enabled by default—no need to manage drafter setup yourself.

The next milestone to watch is Google I/O (mid‑May), where a vision edition of Gemma 4 is rumored to launch. If that variant also ships with an MTP drafter, the cost‑performance curve for multimodal on‑device inference will be redrawn entirely.

References

• Reddit r/LocalLLaMA: community thread on Gemma 4 hiding MTP — original source confirming MTP was “deliberately removed”
• Hugging Face: Gemma 4 model weights and drafter repositories — download links for main models and the new Drafter‑S / Drafter‑L