GMC Prunes Over 80% of Multimodal Tokens

Zidong Taichu Cuts Multimodal Model Tokens by 80%

Zidong Taichu recently introduced GMC, a core-set pruning method designed to identify genuinely useful tokens in multimodal inputs without retraining the model. According to the team, GMC can reduce multimodal tokens by approximately 80% while largely preserving the original model’s capabilities. It requires no training and can be directly integrated into existing vision-language models.

This does not reduce the model’s parameter count or quantize its weights. Instead, it removes redundant information from visual inputs before the model begins large-scale computation.

After an image is split by a vision encoder into hundreds or even thousands of tokens, many of those tokens describe similar backgrounds, continuous textures, and repeated regions. Conventional multimodal models often send all of them into the language model without distinction: if the sky occupies half an image, it may contribute a large block of semantically similar tokens; if dozens of consecutive frames in a video barely change, the model will still repeatedly process the same visual information.

What GMC does can be understood as selecting “core representatives” from an excessively large delegation: it must preserve the key regions that determine the answer while preventing all the available slots from being monopolized by the most visually prominent subject.

A Core Set Is Not Simply a Selection of the Tokens with the Highest Attention Scores

Visual token pruning is not a new field. The most straightforward approach is to rank tokens by attention score and retain only the top portion. However, such methods have an obvious problem: salience does not guarantee information completeness.

For example, if a model is asked, “What color is the cup on the table?”, high-attention regions may focus on the person and the main area of the table. Although the cup occupies only a small area, it determines the final answer. If only the most salient tokens are retained, the model may still be able to describe the entire image while losing the local attribute needed to answer the question. Tasks such as visual referring, OCR, chart understanding, and object localization are particularly sensitive to this kind of loss.

Based on the information disclosed so far, GMC adopts a core-set selection approach. The key is not merely determining whether a token is “important,” but also considering whether the retained tokens can cover the distribution of the original visual features. In other words, it addresses two objectives simultaneously:

• Salience: Prioritize tokens related to global semantics, the user’s question, or key targets;
• Coverage: Prevent selected tokens from becoming overly concentrated in a few similar regions, and cover different objects, positions, and details as broadly as possible.

This is more reasonable than simply performing Top-K ranking. Top-K is like selecting only the students with the highest test scores, while core-set selection also considers their fields of study, regions, and skill profiles to avoid ending up with a team made up entirely of the same type of person.

The most noteworthy label attached to GMC is “training-free.” Many token-compression methods require adding modules, distilling models, or fine-tuning for a specific vision-language model. This is usually not a major issue in the laboratory, but for production systems it means additional data, GPU time, and version-maintenance costs. Every model upgrade may also require the compression module to be retrained.

A training-free method is closer to inference-side middleware: it reads the model’s existing visual features or attention signals, performs the selection, and then continues through the original generation pipeline. It may not achieve the theoretical optimum on every model, but it presents a significantly lower deployment barrier.

Why 80% Fewer Tokens Does Not Necessarily Mean Five Times the Speed

“An 80% reduction in tokens” is a metric that can easily be misinterpreted.

If the original visual input contains 1,000 tokens and only 200 remain after pruning, the visual prefix entering the language model has indeed been shortened by 80%. For models using dense attention, the attention computation during the prefill stage scales approximately quadratically with the total sequence length:

$$R_{attn} \approx \left(\frac{T+0.2V}{T+V}\right)^2$$

Here, $T$ is the number of text tokens, and $V$ is the original number of visual tokens. The larger the proportion of visual tokens, the more pronounced the gains from compression. High-resolution images, multi-image conversations, and long videos generally benefit more than a single low-resolution image.

However, this does not mean end-to-end latency will necessarily fall by 80%, much less that throughput will increase fivefold. The full inference pipeline also includes:

1. Image decoding and resizing;
2. Vision encoder forward computation;
3. The additional overhead of GMC core-set selection;
4. The multimodal projection layer;
5. Language-model prefill;
6. Autoregressive decoding and sampling.

If pruning occurs after visual encoding is complete, the preceding image-encoding cost does not disappear. For simple classification tasks that output only a few tokens, the vision encoder may be the primary bottleneck; for complex question-answering tasks that generate hundreds of tokens, the decoding stage may account for the largest share. Developers therefore cannot rely on token compression alone. They also need to measure time to first token, total latency, peak GPU memory, and throughput per unit of time.

Another practical benefit of GMC is that it shortens the visual prefix in the KV cache. In multi-turn image-text conversations, batch image analysis, or concurrent video question answering, the KV cache often reaches the GPU memory limit before computation itself becomes the bottleneck. Reducing visual tokens generally lowers cache size and memory-bandwidth pressure approximately linearly, potentially allowing the server to use larger batches or maintain more concurrent sessions on the same GPU.

These gains can sometimes matter more than the speedup for an individual request. For an online service, a 20% throughput improvement with more stable concurrency is often more commercially valuable than doubling the speed of an isolated sample.

Compared with Approaches Such as VScan, GMC’s Advantage Is Its Lower Deployment Barrier

Current multimodal token-compression methods can be broadly divided into three categories.

Category One: Direct Deletion by Importance

These methods read attention or similarity scores and retain the most important tokens. Their advantages are simple implementation and low additional computational overhead; their disadvantage is that they can easily sacrifice small objects, edge regions, and spatial relationships. Once the pruning ratio exceeds 70%, performance often becomes unstable.

Category Two: Multi-Stage Scanning and Fusion

Methods such as VScan first select tokens responsible for global semantics from deep features, then restore easily overlooked details from shallow layers or local windows. Some approaches also merge information from deleted tokens into retained tokens. They may then use the text question to remove irrelevant visual information again in intermediate layers of the language model.

This approach is more refined and is particularly suitable for visual localization and detail-oriented question answering, but it requires more integration points and is more complex to engineer. Different models do not have identical layer counts, attention structures, or vision towers, so migration often requires renewed adaptation.

Category Three: Core-Set Selection

The approach used by GMC treats visual tokens as a dataset to be compressed and approximates the entire set with a small number of representative points. Rather than attempting to recover every deleted token individually, it requires the retained set to be both representative and diverse.

From an engineering perspective, this is a highly pragmatic approach. It is more robust than pure Top-K selection, yet does not intrude as deeply into the model architecture as multi-stage pruning. For teams that have already deployed LLaVA, Qwen-VL, or other vision-language models, training-free operation and lower modification costs may be more appealing than gaining a few tenths of a percentage point on a benchmark.

However, whether GMC can become a general-purpose component still depends on two questions: first, the computational complexity of core-set selection itself; and second, whether the same pruning ratio can be used across different models, resolutions, and tasks. If a large number of thresholds must be retuned for every new use case, the value of being “ready to use out of the box” will be diminished.

Which Use Cases Benefit Most?

GMC should not necessarily be enabled by default for every multimodal request. It is better suited to scenarios with large numbers of visual tokens and high information redundancy.

First, multi-image document and report analysis. When more than a dozen scanned pages are submitted at once, page margins, backgrounds, templates, and repeated headers consume a large number of tokens. Core-set pruning can reduce repeated visual features, but contract amounts, footnotes, and table cells must be validated separately.

Second, video understanding. Adjacent frames are highly similar, resulting in far more token redundancy than in a single image. As long as the core set accounts for temporal changes and small objects, this scenario generally offers the greatest compression gains. Conversely, videos containing extremely rapid actions and very few keyframes are also high-risk cases.

Third, embodied AI and robotics. Robot cameras continuously generate visual inputs, but only a limited amount of information is genuinely relevant during each decision cycle. Reducing visual tokens can lower memory usage and decision latency on edge devices. However, safety-critical scenarios should not be evaluated solely by average accuracy; tail risks such as missed obstacle detections must also be considered.

Fourth, cloud-based multimodal APIs. API providers care about concurrency per GPU, P95 latency, and GPU memory utilization. Training-free pruning can serve as a dynamic strategy at the model front end: aggressively compress simple images, while lowering the pruning ratio—or disabling pruning entirely—for OCR, localization, and medical-imaging requests.

How Developers Should Evaluate It

Teams planning to integrate GMC or a similar method into production should not merely reproduce the average scores reported in a paper. Instead, they should establish their own compression-quality curves.

At a minimum, the following settings should be compared:

• No pruning, as the capability and latency baseline;
• 50% pruning, to determine whether clear performance gains already appear;
• 75% to 80% pruning, to validate the primary operating point claimed by the developers;
• 90% pruning, to test how the system degrades under extreme pressure.

Evaluation metrics should also extend beyond overall accuracy. OCR character recall, object-localization IoU, error rates for numbers and dates, hallucination rates, and stability across different image resolutions should all be measured separately. An essentially unchanged average score may simply mean that frequent, easy questions are masking a small number of severe failure cases.

During production load testing, at least the following four metrics should be recorded:

| Metric | Question to Answer | |---|---| | Time to first token | Does pruning actually shorten the prefill stage? | | End-to-end request latency | Does the additional selection overhead offset the gains? | | Peak GPU memory | How much are the KV cache and intermediate activations reduced? | | Throughput and P95 latency | Are the gains still stable under high concurrency? |

Results may also vary considerably across hardware. A reduction in token count lowers theoretical FLOPs, but does not guarantee a corresponding improvement in GPU utilization. With small batches, short sequences, or insufficient operator fusion, the system may shift from being compute-bound to being limited by scheduling and memory access. The decisive factor for production gains remains whether the inference engine can take advantage of the shorter sequences.

GMC Is Useful, but It Should Not Be Mistaken for “Lossless Magic”

Our assessment is that GMC represents a more practical direction for multimodal deployment than continuing to add parameters: first acknowledge the enormous redundancy in visual inputs, then concentrate limited compute on the regions that genuinely determine the answer. Its training-free nature gives it strong practical deployment value, while an approximately 80% reduction in tokens is enough to reshape the cost structure of some high-resolution, multi-image, and video tasks.

However, “capabilities are largely preserved” does not mean every sample is processed without loss, nor does it mean that 80% of tokens can be removed from every task and every model. Visual question answering and scene description may be almost unaffected, while OCR, fine-grained localization, medical imaging, and industrial defect detection may be extremely sensitive to a small number of tokens.

The appropriate product design should not use a globally fixed pruning ratio, but task-aware dynamic compression: first determine the request type and visual complexity, then decide how many tokens to retain. If GMC can be combined with such a routing mechanism, its value will be greater than simply setting a new token-compression record.

The next phase of efficiency competition among multimodal models will not focus solely on parameter counts and quantization bit widths. The number of visual tokens that genuinely participate in reasoning is becoming a more direct measure of cost.

References

• LLaVA GitHub Repository: An open-source implementation of a representative vision-language model that can help explain how the vision encoder, projection layer, and language model are connected.
• Qwen2.5-VL GitHub Repository: An implementation of a multi-resolution image and video understanding model that can serve as a reference for evaluating the portability of token pruning.
• Hugging Face Transformers: LLaVA Model Documentation: Explains visual tokens, image-processing parameters, and how multimodal inputs are organized.
• Hugging Face Transformers: KV Cache Documentation: Introduces caching mechanisms in autoregressive inference and helps explain how shortening the visual prefix affects GPU memory usage and inference efficiency.