PD Separation Overcomes the WAN Barrier

A noteworthy development has recently emerged in large-model inference infrastructure: a newly released full technical report shows that by organizing compute resources distributed across different regions and clusters into a Prefill-and-Decode “relay system,” model inference latency can be reduced by up to roughly half, while deployment costs can fall by nearly 40%.

These figures are impressive, but the problem they address is even more important. Prefill-Decode disaggregation has already been validated many times within individual high-performance clusters. The real challenge has always been what happens across clusters, regions, and even carrier networks: how to transfer the KV Cache, schedule tasks, avoid slow nodes, and ensure that network and operational overhead does not consume the savings in GPU costs.

The latest progress indicates that Prefill-Decode disaggregation is evolving from a “within-the-data-center performance optimization” into a system architecture capable of activating heterogeneous, distributed compute resources. In other words, large-model inference no longer requires every GPU to be packed into the same expensive cluster for the entire race. Prefill and Decode can instead become two legs of a relay, allowing resources in different locations and with different specifications to handle the segment they are best suited for.

One caveat must be emphasized first: the roughly 50% latency reduction and nearly 40% cost reduction were achieved in the report’s test environment. They should not be interpreted as results that every model, network, and workload can reproduce directly. Model architecture, context length, output length, concurrency patterns, link quality, and GPU prices all affect the final gains. Nevertheless, the direction of this work matters more than any single benchmark: it is expanding the deployment boundary of Prefill-Decode disaggregation from low-latency data center networks to broader compute networks.

Why the Same GPU Is Ill-Suited to Handle the Entire Process

A single LLM inference request can broadly be divided into two stages:

• Prefill: The model reads the user’s input in one pass, computes it, and generates the corresponding KV Cache. This stage involves extensive matrix operations and is compute-intensive. It determines time to first token, or TTFT.
• Decode: The model generates the result token by token based on the existing KV Cache. This stage depends more heavily on GPU memory capacity and memory bandwidth. It determines the interval between output tokens, or TPOT.

Both stages may appear to be “running the model,” but their hardware characteristics are fundamentally different.

Prefill is like a fully loaded truck accelerating down a highway: the longer the input, the greater the computational pressure, but also the greater the opportunity for parallelism. Decode is more like a delivery vehicle repeatedly entering and leaving a warehouse, retrieving only a small amount of data and generating one token at a time. Its bottleneck is usually not peak compute performance, but GPU memory access and batching efficiency.

Traditional colocated architectures place both stages on the same pool of GPUs. The advantage is simplicity: the KV Cache does not need to be moved across nodes, and deployment and failure recovery are more straightforward. The problem is that a surge of Prefill tasks may block requests already in Decode. Conversely, a large number of long-output requests can occupy GPU memory and batch slots, forcing new requests to wait a long time for their first token.

What developers ultimately observe is often not a slight increase in average latency, but a sudden deterioration in tail latency: most requests behave normally, while a small fraction stall for an unreasonable amount of time. For chat, code completion, and real-time agents, P90 and P99 latency often matter more than the average in determining whether a product is usable.

Prefill-Decode disaggregation has therefore become a major focus of inference-system optimization over the past two years. It assigns separate resource pools to Prefill and Decode, allowing each to use independent batching, scaling, and hardware selection. Prefill can use GPUs with greater compute performance, while Decode can prioritize memory bandwidth, capacity, and cost.

The theory is straightforward, but the engineering is not.

The Real Barrier Is Moving the KV Cache

Once Prefill is complete, Decode must take over the entire KV Cache. For large models and long contexts, this is not a negligible packet of data, but state that may continuously consume substantial bandwidth.

Within a single data center, systems can rely on RDMA, InfiniBand, or optimized RoCE networks to keep transfer latency sufficiently low. But once P nodes and D nodes are placed in different clusters or even different regions, the problems quickly intensify:

1. Links have higher latency and are subject to jitter. If a Decode node must keep waiting for the KV Cache, the network will consume the time saved on computation.
2. Bandwidth can no longer be treated as nearly unlimited. When long-context requests arrive in large numbers, KV Cache migration may itself become a new queueing bottleneck.
3. Compute supply is not stable. Distributed resources differ in GPU model, load, and failure rate, and static binding can easily create resource silos.
4. Scheduling must consider more than idle GPUs. The system must also account for data location, model replicas, estimated output length, SLO headroom, and link conditions.
5. Failure recovery becomes more difficult. If a node or network failure occurs after Prefill has completed but while state is being transferred, the system must decide whether to retransmit, recompute, or switch Decode nodes.

Therefore, “deploying Prefill in location A and Decode in location B” does not in itself constitute cross-region Prefill-Decode disaggregation. A truly effective system must jointly optimize compute scheduling, cache transfer, and network selection.

The key idea reflected in the latest report is to treat the entire process as a relay race rather than an ordinary RPC between two independent services. While a Prefill node is still computing, the scheduler must predict in advance which node should take the next leg. KV Cache transfer should not wait until the entire cache has been generated; instead, it should be streamed and transferred in chunks wherever possible. On the Decode side, appropriate resources must be reserved before all state arrives, preventing a situation in which the data is ready but the GPU is still waiting in a queue.

The value of this coordination lies in hiding network time within computation and queueing. Cross-region link latency cannot simply disappear, but it does not have to be fully exposed in user-perceived latency.

Why Latency Can Be Cut in Half

The latency gains do not come solely from a faster transfer operator, but from the combined effects of interference isolation, asynchronous transfer, and global scheduling.

First, once Prefill and Decode are separated, long-input requests no longer directly consume Decode batch capacity. TPOT becomes more stable for requests that have already begun generating output. New requests can also enter a dedicated Prefill resource pool, so their TTFT does not have to wait for a batch of long-output requests to finish.

Second, cross-cluster scheduling expands the range of available resources. When one data center experiences local congestion, the system can assign the next stage to other available nodes instead of waiting for a busy GPU. The logic is similar to a ride-hailing platform expanding its dispatch radius: the distance may be slightly greater, but if every nearby vehicle is stuck in traffic, an idle vehicle farther away may still arrive sooner.

Third, pipelined KV Cache transfer reduces pure waiting time. As long as transfer and computation can overlap, the latency introduced by a wide-area network does not have to translate one-for-one into end-to-end response time.

However, the claim that “latency is reduced by roughly half” must be understood in relation to the metric being measured. TTFT, TPOT, request completion time, and P99 tail latency are not the same thing. Prefill-Decode disaggregation most directly improves tail latency caused by interference between the two stages, as well as effective throughput under specified TTFT and TPOT targets—in other words, goodput. Looking only at the number of tokens generated per second may conceal the fact that some requests have already violated their SLOs.

For production services, goodput matters more than raw throughput. A server that emits more tokens per second but causes many interactive requests to miss their first-token deadlines has not truly become more efficient.

Where the Nearly 40% Cost Reduction Comes From

Cost optimization likewise does not mean that “GPUs suddenly became cheaper.” Rather, the combination of resources becomes more flexible.

In a colocated architecture, organizations often purchase a uniform class of GPU based on the requirements of the most demanding stage. Even though Decode primarily depends on memory bandwidth, it may still be forced to use expensive GPUs designed for large-scale matrix computation. Even when Prefill needs compute performance, much of the GPU’s memory capacity may remain underutilized.

After disaggregation, the P and D pools can select hardware independently:

• The Prefill side can prioritize devices with higher compute throughput;
• The Decode side can choose devices based on GPU memory capacity, bandwidth, and power consumption;
• When the two stages have different peak and off-peak patterns, their resource pools can scale independently instead of being replicated as a complete unit;
• When local resources are insufficient, idle compute capacity in other regions can be used;
• Different generations of GPUs can be mixed according to their roles, reducing the pressure to upgrade everything uniformly.

This is also why “activating compute capacity nationwide” offers more possibilities than “building a larger monolithic cluster.” In practice, compute supply is highly fragmented: some regions have low electricity prices, some clusters have strong networks, some data centers hold inventories of older GPUs, and some nodes are idle only during specific periods. Traditional inference services struggle to combine these resources into stable instances. Prefill-Decode disaggregation offers a way to break demand down by stage.

Still, the nearly 40% cost reduction should be viewed as a system-level result under a specific workload, not as a fixed discount. If cross-region egress bandwidth is expensive, the KV Cache is excessively large, or GPU utilization is already high, the gains will narrow significantly. Conversely, if a workload mixes long inputs and long outputs and multiple clusters exhibit pronounced differences between peak and off-peak usage, the cost improvement may be more substantial.

In our view, this technology is best suited in the near term to cloud providers, model API platforms, and large enterprises with multiple compute centers—not small teams with only a few GPUs. For the latter, the operational cost of introducing cross-domain scheduling, state management, and failure recovery may exceed the hardware savings.

Which Workloads Will Benefit First

Not all inference requests are equally well suited to cross-region Prefill-Decode disaggregation. Several categories of workloads are likely to benefit earlier.

1. Long-Context Q&A and Document Analysis

These requests have a high proportion of Prefill work and widely varying input lengths. Separating Prefill prevents one exceptionally long document from slowing down other requests that are already generating answers, while allowing requests to be assigned to different P nodes according to input size.

2. Agents and Coding Tasks

Agents frequently invoke models, tools, and memory systems. A single request may not be especially long, but the chain contains many steps, allowing tail latency to accumulate layer by layer. Stable TTFT and TPOT matter more than simply pursuing peak throughput.

Code generation also often requires both a fast first token and stable continuous output, precisely the kind of workload in which colocated architectures are prone to resource interference.

3. Multi-Model API Services

Aggregation platforms must handle different models, context lengths, and concurrency patterns. If each model is provisioned with a complete GPU cluster sized for its own peak load, idle capacity will be high. Role-specific resource pools and cross-cluster scheduling allow the underlying compute capacity to be reused at a finer granularity.

4. Geographically Distributed Inference Clouds

Service providers with multiple data centers can recombine low-cost compute, low-latency ingress points, and high-bandwidth nodes. User requests do not have to remain bound to a single data center from start to finish, provided that data-compliance and privacy boundaries permit state to move across regions.

What It Has Not Fully Solved Yet

Progress in Prefill-Decode disaggregation does not mean that inference infrastructure has entered an era in which GPUs can be combined arbitrarily. At least four issues still require further observation.

First, wide-area network reliability. Internal data center networks are fundamentally different from the public internet or dedicated links. Packet loss, jitter, and temporary congestion directly affect KV Cache handoff. Systems must be capable of falling back to local colocated inference.

Second, state consistency and recovery. Cloud-native orchestration excels at managing stateless services, whereas Prefill-Decode disaggregation explicitly exposes the state of a single request across multiple nodes. Rolling upgrades, elastic scaling, and failover are all more complex than in ordinary microservices.

Third, security and compliance. A KV Cache is not meaningless intermediate data. It contains model state that may be associated with user context. Cross-region transfer must account for encryption, tenant isolation, data residency, and lifecycle management.

Fourth, criteria for evaluating gains. Different papers and systems use different models, hardware, context distributions, and SLOs. Simply comparing claims of “multiple times faster” can easily be misleading. A more meaningful measure is how many valid requests each GPU can complete under the same TTFT, TPOT, and success-rate constraints.

This also means that production deployment should not happen all at once. A more prudent path is to first separate P and D resource pools within the same data center, then expand to multiple clusters within the same metropolitan area, and only afterward attempt cross-region scheduling. A colocated path should also be retained as a fallback for short requests, small models, or network anomalies.

Prefill-Decode Disaggregation Is Becoming a Fundamental Capability of Inference Systems

In the past, model-serving optimization focused on quantization, continuous batching, Paged Attention, and operator fusion. These techniques primarily answered the question: “How can we make one GPU or one group of GPUs run at higher utilization?” Prefill-Decode disaggregation asks a further question: why must different stages use the same type of hardware and remain in the same location?

The significance of this latest development is that it shifts the problem from single-machine performance to resource organization. Cutting latency in half and reducing costs by nearly 40% are certainly eye-catching results, but the long-term value lies in demonstrating that geographically distributed compute resources can form a viable inference cluster—as long as scheduling and state transfer are designed properly.

It will not replace every colocated architecture. Services involving short contexts, low concurrency, or small models may still run more economically on a single machine or within a single cluster. When network transfer costs exceed queueing costs, forcing disaggregation will only make the system more complicated. The future is more likely to feature hybrid approaches: the scheduler will dynamically choose among colocation, within-cluster Prefill-Decode disaggregation, and cross-region Prefill-Decode disaggregation based on request length, real-time load, KV Cache size, and SLOs.

That will be the true sign that Prefill-Decode disaggregation has matured: developers will no longer need to choose an architecture manually for every workload. Instead, the system will select the inference route with the most appropriate cost and latency for each request, much like a router selecting a network path.

References

• Overview of Prefill-Decode Disaggregation and KV Cache Offload Technologies: Introduces the workload differences between Prefill and Decode, as well as the basic mechanisms of high-performance networking and KV Cache offloading.
• SGLang Open-Source Repository: A mainstream large-model inference framework that provides insight into the engineering implementation of scheduling, cache management, and distributed inference.
• Mooncake Open-Source Repository: A KV Cache and disaggregated inference system for LLM serving, providing relevant architectural and engineering resources.