Kimi and GLM Reasoning Slimming: Hands-On Test

Cloudflare’s Answer: Don’t Just Compress the Model—Keep an Eye on the Cache

Cloudflare recently published a set of engineering practices for running the Kimi and GLM model families at scale. The core approach can be summarized in three steps: compress model weights, quantize the KV cache, and add integrity checks to cache pages.

The first two steps address speed and cost, while the last tackles an often-overlooked data isolation issue in multi-tenant inference.

This is not a model capability update, nor does it introduce a new algorithm leaderboard. It reads more like an incident-prevention manual drawn from production environments: as large-model services grow from dozens of concurrent requests to hundreds or more sharing GPUs, every cache page, pointer, and reuse relationship in the inference framework can become either a performance bottleneck or a security boundary.

Cloudflare’s assessment is straightforward: to run large models such as Kimi and GLM economically, you cannot focus only on parameter count. GPU memory is contested not only by resident model weights, but also by the KV cache, which expands continuously with context length and concurrency.

Why Smaller Model Weights Can Make Each Token Arrive Faster

A large-model inference request generally consists of two stages: prefill and decode.

Prefill processes the user’s entire prompt at once. It is matrix-compute-intensive and more likely to be constrained by GPU compute. Decode, by contrast, generates tokens one at a time. For each token generated, the system must read the model weights and access the KV cache corresponding to the preceding context.

This means the decode stage is often constrained not by insufficient compute, but by insufficient data movement speed.

This is especially true in low-concurrency scenarios that emphasize single-request responsiveness. When the GPU is not yet fully saturated by a large number of requests, memory bandwidth often becomes the bottleneck before the compute units do. The larger the model weights, the more data must be moved from GPU memory for every token generated, and the slower the output appears to the user.

After applying weight compression, Cloudflare’s most immediate gain was not making the model “think less,” but reducing the amount of data that must flow through GPU memory during each decode iteration. One way to think about it is this: previously, generating each character required moving an entire truckload of material out of a warehouse; after compression, only half a truckload is needed. The elevator is still the same, but each delivery is faster.

The value of this optimization varies by workload:

• Low-concurrency interactive requests place greater emphasis on per-token latency, so smaller weights can usually produce a noticeable improvement in responsiveness;
• High-concurrency batch processing can amortize the cost of reading weights through batched matrix operations, so the gains may be reflected more in throughput and GPU memory usage;
• Long-output tasks repeatedly execute decode operations, allowing the benefits of weight compression to accumulate;
• Very long-input, short-output tasks are more heavily affected by prefill and KV cache management, so compressing weights alone is not the complete answer.

Therefore, “smaller means faster” is true only after the bottleneck has been clearly identified. If a task is mainly constrained by prefill computation, cross-GPU communication, or scheduling queues, further weight compression may not reduce end-to-end latency proportionally.

The KV Cache Is the Hidden Limit on Concurrency

Model weights are static: once an instance starts, they generally remain on the GPU. The KV cache is dynamic: each request must preserve its own contextual state, which continues to grow with the number of conversation turns, prompt length, and generated output length.

This issue is even more pronounced in agent applications. System prompts, tool definitions, MCP descriptions, message history, code, and tool results are all fed back into the context. A single request may contain tens of thousands or even hundreds of thousands of tokens, and every active session requires its own KV cache.

Under these workloads, the GPU often runs out of memory for new requests before it runs out of compute capacity.

Cloudflare’s approach is to further quantize the KV cache, storing Keys and Values in lower-precision data formats. As long as the loss of precision remains within an acceptable range, the same amount of GPU memory can store more context and accommodate more concurrent sessions.

This delivers three levels of benefits:

1. A smaller cache per request, making long contexts less likely to exhaust GPU memory rapidly;
2. More requests per GPU, making it easier for continuous batching to form sufficiently large batches;
3. Less cache data to read, reducing memory pressure during attention computation over long contexts.

Weight quantization and KV cache quantization may both appear to be about “reducing precision,” but their risks are different. Weight quantization changes the model parameters shared by all requests. If the error in a particular layer becomes too large, it will consistently affect model output. KV cache quantization changes the intermediate state of a particular request within a specific context, and its errors may accumulate with context length, attention patterns, and the number of generation steps.

KV cache quantization therefore cannot be validated with only one round of general-purpose benchmarks. Production validation should cover at least the following:

• Whether long-context information retrieval degrades;
• Whether multi-turn tool use becomes more prone to selecting the wrong tool;
• The stability of JSON, code, and structured output;
• Accuracy for mixed Chinese-English content, numbers, dates, and proper nouns;
• Whether repetition, drift, or factual loss appears in the latter half of long-form generation;
• Whether time to first token and per-token latency remain stable under different concurrency levels and batch sizes.

The most important lesson for developers from Cloudflare’s implementation is not any particular quantization format, but the decision to handle weights and the KV cache separately: weights determine the model’s resident cost and the amount of data moved during decoding, while the KV cache determines long-context and concurrency capacity. The two must be measured independently rather than judged from a single throughput chart.

The Higher the Memory Utilization, the More Severe the Consequences of Cross-Request Cache Reads

After compression, the same amount of GPU memory can accommodate more requests. This naturally improves utilization, but it also leaves less room for error in cache management.

Modern inference frameworks commonly use paged attention, continuous batching, and cache reuse. The KV cache no longer requires each request to own a contiguous block of GPU memory. Instead, it is divided into multiple physical pages, while the runtime records which pages belong to each request. When a request ends, its physical pages are released and reassigned to other requests.

This is similar to how an operating system manages memory pages. The difference is that large-model inference is extremely sensitive to throughput, so cache pages are allocated, released, and reused at high frequency. At any given moment, hundreds of requests may be reading from and writing to the same physical KV cache pool.

If even an extremely unlikely inconsistency occurs among page tables, request state, and asynchronous execution flows, the worst-case scenario may follow: Request A reads a cache page that has already been reassigned to Request B.

The result is not merely garbled output. It may amount to cross-request data leakage.

Cloudflare specifically emphasized that, at its request volume, even an error probability as low as one in a billion cannot be treated as “something that will never happen.” Once the number of operations is large enough, low-probability events turn from theoretical risks into recurring production incidents.

Giving Every Physical Cache Page a “Generation Number”

To address this, Cloudflare added KV cache integrity checks. The idea is not complicated: every physical cache page has a tag, and that tag changes each time the page is reassigned. The server simultaneously records the pages each request expects to use and their corresponding tags.

Before a supported decode operation reads the cache, the runtime checks whether the two match. If the page ID is the same but the tag has changed, the page has already been released and entered a new lifecycle, meaning the original request holds a stale mapping.

In that case, the system does not attempt to continue generating output. It immediately aborts the affected request.

The logic can be represented in pseudocode:

for page in request.expected_cache_pages:
    current_tag = cache_allocator.tag(page.id)

    if current_tag != page.expected_tag:
        abort_request("stale or mismatched KV cache page")

run_decode_step(request)

This is essentially a generational validation mechanism. A page belongs to a request only when both the “physical page address” and the “current generation tag” match. Similar ideas are often used to prevent access to an object after it has been freed: the address may not have changed, but the object is no longer the same object.

More importantly, Cloudflare chose to fail closed. When a mapping anomaly is detected, it is better to return an error for one request than to allow the model to continue generating output using cache data of unknown origin.

This is more appropriate for multi-tenant inference than “automatic recovery.” Automatically recovering or skipping an anomalous page might reduce the error rate, but it could also mix data from other requests into the generated result. For a shared inference platform, data isolation should take precedence over the success rate of an individual request.

Of course, integrity tags are not a universal security solution. They primarily guard against cross-request reads caused by cache-page reuse, mapping mismatches, or runtime defects. They cannot replace process isolation, tenant authentication and authorization, log redaction, GPU memory clearing, or access control. Cloudflare’s description is also limited to “supported decode operations,” which means operator coverage, bypass paths, and exception handling must also be audited.

It Follows the Same Path as Cloudflare’s Earlier Infire Optimizations

Cloudflare had previously disclosed some capabilities of its in-house Infire inference engine: it reduces the GPU memory overhead of internal state, enabling large models to start on more limited GPU configurations. For models such as Kimi that run across multiple GPUs, it uses Mooncake-related components to transfer and share the KV cache.

According to data previously published by Cloudflare, Infire can run Kimi K2.5 on eight H100 GPUs while leaving more than 30 GiB of GPU memory available for the KV cache. Even a model as large as Kimi K2.5 can begin processing requests within 20 seconds, with model loading speed constrained primarily by storage read performance.

It also uses techniques such as prefill-decode disaggregation, prefix caching, and speculative decoding. These optimizations address different problems:

• Prefill-decode disaggregation: allows compute-intensive input processing and bandwidth-intensive token-by-token generation to use different resource configurations;
• Prefix caching: avoids repeatedly processing shared inputs such as system prompts and tool definitions;
• Speculative decoding: has a draft model propose multiple candidate tokens, which the main model then verifies in batches;
• Weight compression: reduces the model’s resident GPU memory footprint and bandwidth requirements during decoding;
• KV cache quantization: increases capacity for long contexts and concurrent requests;
• Cache integrity checks: maintain data isolation between requests under high cache reuse.

Viewed separately, most of these techniques are not new concepts. The real challenge is making them work together without allowing one optimization to violate the assumptions of another. For example, cache reuse improves hit rates but expands the potential impact of an incorrect mapping; continuous batching improves GPU utilization but increases the frequency of request-state changes; quantization frees more GPU memory but also concentrates more tenants and requests on each GPU.

The value of Cloudflare’s latest disclosure lies precisely in this combination of engineering techniques, rather than in the invention of any single quantization algorithm.

What Developers Should Learn from This

If a team is building its own service for Kimi, GLM, or other large open-source models, this approach is worth considering. However, it is not advisable to start by asking, “Which quantization format should we choose?”

A more sensible sequence is to first establish four groups of metrics:

1. Break Down Latency

At a minimum, separately record queueing time, prefill time, time to first token, average time per token, and total request duration. Looking only at total duration mixes scheduling, networking, and model execution together.

2. Monitor GPU Memory and Effective Concurrency Together

Lower GPU memory usage does not necessarily mean greater serving capacity. You must also verify whether the freed space actually translates into more active sequences, longer contexts, or larger continuous batches.

3. Validate Quality with Real Traffic

Quantization evaluation cannot consist solely of multiple-choice questions. Agent workloads should focus on replaying tool calls, code modifications, structured output, and long conversation histories. A model losing only a fraction of a percentage point on public benchmarks does not mean its production error rate will increase by only the same fraction.

4. Perform Fault Injection on Cache Management

Proactively simulate request cancellation, timeouts, batch reordering, page reclamation, GPU operator failures, and concurrent releases. Integrity checks must not only detect errors, but also ensure that aborting an anomalous request does not contaminate other requests in the same batch.

Integrity checks also have their own cost. Adding validation to every decode step may introduce additional memory accesses, branches, and state maintenance. Whether to check page by page, when to perform checks, where to store tags, and how to report anomalies must all be evaluated through throughput testing. If operators disable a security mechanism because of performance pressure, it is effectively no mechanism at all.

Assessment: The Performance Optimizations Are Not New, but the Security Patch Matters More

Viewed in isolation, Cloudflare’s “weight compression” and “KV cache quantization” are not mysterious; mainstream inference engines are pursuing similar work. The most valuable part of its approach is the explicit recognition of the security spillover created by densely shared KV caches, and the inclusion of cache-page lifecycles in request-level integrity validation.

Large-model inference is retracing the path previously taken by databases and operating systems: first, the goal is simply to get the system running; then, the focus shifts to utilization; finally, it becomes clear that isolation, validation, and observability must become part of the infrastructure.

For smaller teams, a one-in-a-billion error may seem remote. For platforms aggregating large volumes of models and traffic, however, it is a problem they will encounter sooner or later. The more requests there are, the longer the contexts become, and the more aggressive cache reuse gets, the less acceptable it is to treat GPU memory as an “infallible black box.”

Cloudflare’s implementation shows that the key to running Kimi and GLM at scale is no longer merely provisioning enough GPUs. It is making every byte serve more requests while still being able to prove that those bytes belong to the correct request.

This is the true relationship among “smaller, faster, and safer”: the first two increase GPU memory density, while the third prevents higher density from turning into higher risk.

Developers who only want quick access to models can also call Kimi, GLM, and similar models through aggregation platforms compatible with the OpenAI API format. But for teams operating their own inference clusters, the most valuable lesson in Cloudflare’s implementation is the security check that is so often omitted after GPU memory has been optimized.

References

Cloudflare’s official technical article, “Smaller, faster, safer: running Kimi and GLM at scale,” together with its previously published technical materials on Infire, served as the primary factual sources for this article. Due to domain-linking restrictions, links to the original site are not included here.

• Moonshot AI Model Page: Open-source Kimi models, weights, and model card documentation.
• Z.ai Model Page: Open GLM models, configurations, and related technical documentation.