Gemma 4 QAT is Here: Fitting a 31B Model into a Consumer GPU

On June 5, Google DeepMind released QAT (Quantization-Aware Training) checkpoints for the Gemma 4 series. This is the second shoe to drop after the mainline release of Gemma 4 in early April—the mainline version solved the “can you use it” problem, while QAT addresses “can you run it on your own machine.”

On release day, two collections went live simultaneously on Hugging Face: Gemma 4 QAT Q4_0 and Gemma 4 QAT Mobile. The former targets desktop-class GPUs and laptops, while the latter is aimed directly at smartphones and edge devices. The license remains Apache 2.0, with virtually no commercial restrictions.

If you recall last year’s Gemma 3 QAT fitting a 27B model into a single 24GB consumer card, this round is similar—just larger and more comprehensive. The mainline Gemma 4 comes in four sizes—E2B, E4B, 26B A4B, 31B—and this QAT release covers them all in one go.

What Problem Does QAT Actually Solve?

In the industry, model compression typically takes two paths: post-training quantization (PTQ) and quantization-aware training (QAT).

PTQ is straightforward—once training finishes, simply cut weights from BF16 to INT4. Convenient, but costly: when precision drops, model performance on long-tail tasks visibly collapses, especially for precision-sensitive work such as mathematical reasoning and long-context tasks. The “Q4 works, Q3 breaks” intuition in dev communities comes from this exact phenomenon.

QAT takes another route: simulate quantization “errors” during training so the model learns how to preserve useful representations under low-precision noise. In simple terms, let the model know during training that it will need to “slim down” later, and prepare accordingly. Google’s blog describes it as “significantly reducing memory requirements while maintaining model quality”—though generic-sounding, for a 31B model this means something concrete: BF16 full-precision weights take ~62GB, but after Q4_0 compression, it’s around 17GB, allowing a 4090 or 5090 to run it.

More importantly, QAT is not just a “quantization method”—it’s more like an officially designated quantized checkpoint. Community GGUF versions have always existed but vary in quality—some use imatrix, some don't, calibration sets differ. By releasing QAT themselves, Google claims the “reference implementation” slot.

Q4_0 and Mobile: Separate Scenarios for Each

The two released collections have distinct targets.

Q4_0 is for desktop use. Q4_0 is the most classic 4-bit quantization format in the GGUF ecosystem—simple structure, fast inference, supported by nearly all llama.cpp derivatives (Ollama, LM Studio, KoboldCpp, text-generation-webui, etc.). Google releasing Q4_0 directly bypasses “waiting for community quantization”; devs can use it immediately.

Based on the mainline Gemma 4 sizes:

• E2B (2B level): ~1.5GB VRAM after QAT, runs on integrated GPU laptops
• E4B (4B level): ~3GB, covers mainstream mobile GPUs
• 26B A4B (MoE architecture, activating 4B parameters): fits into a single 16GB card
• 31B (dense architecture): ~17GB, runs comfortably on a 24GB card; 16GB cards can manage with offloading

Mobile is more aggressive, targeting Android and iOS devices. It shares a base with Gemini Nano 4—Google mentioned at Google I/O '26 that Gemma 4 is the foundation for the new-gen Gemini Nano. The Mobile version offers finer operator tuning for ARM architectures—not just smaller weights, but adjustments so NPUs and mobile GPUs can handle them.

By pushing both routes at the same time, Google aims to lower the floor for “running open-source models locally” down to smartphones, and set the ceiling at workstations. The middle ground—the part most developers actually use—is covered by Q4_0.

Performance: How Much Loss Does Quantization Introduce?

The mainline Gemma 4 benchmarks are already public:

| Model | MMLU Pro | AIME 2026 (No Tools) | MATH-Vision | |-------|----------|----------------------|-------------| | Gemma 4 31B | 85.2% | 89.2% | 85.6% | | Gemma 4 26B A4B | 82.6% | 88.3% | 82.4% | | Gemma 4 E4B | 69.4% | 42.5% | 59.5% | | Gemma 4 E2B | 60.0% | 37.5% | 52.4% | | Gemma 3 27B (No Reasoning) | 67.6% | 20.8% | 46.0% |

Worth noting is AIME 2026—31B scored 89.2%, over quadruple the Gemma 3 27B score. This shows Gemma 4’s improvement in mathematical reasoning is structural, not just marginal.

Google hasn’t released all official QAT numbers yet, but based on Gemma 3 QAT data, Q4_0 quantization typically incurs a 1–2 point drop on benchmarks like MMLU Pro, much smaller than the 3–5 point drops common in community PTQ versions. For models like 31B already above capability thresholds, trading 1 point for a 4× memory reduction is almost a no-brainer.

256K Context and Multimodality: What’s the Impact?

Gemma 4 medium models (26B A4B, 31B) have a context window of 256K; small models have 128K. These numbers don’t shrink after quantization—the context length is unrelated to weight precision and depends on KV cache memory usage.

But here’s a hidden gotcha: a 256K KV cache in BF16 can consume ~30–40GB (depending on attention head setup). Even if weights are compressed to 17GB, running full context still needs 50GB+ total VRAM. Google’s current QAT doesn’t quantize KV cache, so in long-context scenarios, VRAM usage is dominated by cache, not weights. Developers need to keep this in mind when deploying.

Multimodal scenarios fare somewhat better. Gemma 4’s vision encoder is ~150M parameters for small models and ~550M for medium—much smaller than the LLM core—and after quantization, it’s negligible. E2B also supports audio input, and the Mobile version optimizes this aggressively—directly targeting smartphone voice assistant use cases.

What This Means for Developers

By releasing QAT separately, Google’s intent is clear:

First layer: Strengthen Gemma’s position in the “open-source local model” track. Llama 4, Qwen 3, DeepSeek are mostly competing on cloud capabilities; local deployment competition is lighter. Leveraging Android and Gemini Nano synergy, Google has a smooth entry into device-side use.

Second layer: Claim “official quantization” mindshare. While community quantized versions exist, production deployments often prefer official weights—at least there’s a baseline to refer to when problems arise. Gemma 3 QAT leveraged this perception to win over Ollama and LM Studio users.

Third layer: Prepare for device-side agents. The mainline Gemma 4 emphasizes agent workflows and multi-step planning. One of the biggest pain points for agent apps is latency—locally running a 4B agent can be better than calling a 70B API in certain scenarios. QAT lowers the hardware barrier further.

If you’re an Ollama or llama.cpp user, you can start running it today:

# Ollama: one-line pull
ollama pull gemma4:31b-qat-q4_0
ollama run gemma4:31b-qat-q4_0

# llama.cpp: load GGUF directly
./llama-cli -m gemma-4-31b-it-qat-q4_0.gguf \
  -c 32768 -ngl 99 \
  -p "Explain the difference between QAT and PTQ"

For developers needing cloud calls to the full Gemma 4 for comparison tests, OpenAI Hub already supports the Gemma 4 series—one key in OpenAI-compatible format lets you switch between Gemma, GPT, Claude, and Gemini for A/B tests, avoiding the hassle of separate local/cloud SDK setups.

Some Thoughts

Engineering optimizations like QAT may not seem flashy, but they’re a critical step in moving open-source models from “benchmark toys” to “production-ready tools.” Google’s dual release of Q4_0 and Mobile this time comes faster than in the Gemma 3 era—the official quantization follows only two months after the mainline, with broader coverage.

This also pressures competitors. The Llama series still lacks official QAT checkpoints; Qwen relies mainly on community GGUF. If Gemma 4 QAT’s loss truly stays within 1–2%, the default choice for local deployment may shift.

Next week, once the community completes full benchmarks, we should see more detailed comparison data. Points worth watching include: QAT’s degradation in 256K long-context scenarios, Mobile version token/s performance on Snapdragon 8 Gen 4 and A18 Pro, and the 26B A4B MoE configuration’s routing stability after quantization—MoE quantization has always been trickier than dense models, and Google’s handling this time will be worth examining.

References

• Gemma 4 QAT Model Release Discussion Thread - linux.do: Community’s first discussion at release, including download links and initial user feedback
• Gemma 4 QAT Q4_0 Collection - Hugging Face: Google’s official desktop-class Q4_0 quantized weights, covering all sizes from E2B to 31B
• Gemma 4 QAT Mobile Collection - Hugging Face: Mobile-optimized version with operator tuning for ARM architectures