Prefill-Decode Disaggregation: Giving Reading and Writing Their Own GPUs

A chatbot can stream smoothly until someone else submits a long document. Then the answer pauses. While studying LLM serving, I kept coming back to this problem: processing a new prompt and continuing an existing answer want different things from the GPU, but often share the same scheduler.

Prefill-decode disaggregation separates those jobs onto different GPU workers. The useful question is when the separation pays for the cost of moving state between them.

What is being separated?

As covered in prefill and decode, prefill processes the known prompt and builds its attention state. Decode repeatedly consumes the newest generated token to predict the next one.

A conventional serving worker performs both phases for different requests. Chunked prefill makes their coexistence more manageable, but they still share execution time and memory resources.

Disaggregation creates a prefill pool and a decode pool. A worker here means a serving instance, which may use one GPU or a group of GPUs. Both pools run the model; this is not a split where early layers live on one worker and later layers on another.

It reminds me of separating document intake from a live support desk. Intake can process large files in batches while the support desk keeps ongoing conversations moving. The catch is that the notes must reach the right person before the conversation continues.

Why bother?

Prefill often benefits from large batches of prompt tokens that keep the GPU's arithmetic units busy. Decode needs frequent steps and enough memory bandwidth and cache capacity for active sequences.

Separating them provides three useful freedoms:

• Long prefills stop occupying the decode workers' execution slots, reducing one source of streaming pauses.
• Each pool can use its own batch limits and supported parallelism configuration. Tensor parallelism, for example, divides the work within model layers across GPUs.
• Capacity can follow demand. A document-heavy workload may need more prefill capacity; long responses may require more decode capacity.

The DistServe paper frames the objective as goodput: requests served within their latency targets. Serving more tokens is less useful if too many users miss the promised first-token or streaming latency. Disaggregation can improve goodput without increasing unconstrained peak throughput.

The handoff is the real engineering problem

The KV cache contains the keys and values computed at attention layers for the processed context. Sending the prompt text alone would force the decode worker to repeat prefill. Instead, it receives the computed state.

A typical request follows this sequence:

1. A router selects a prefill worker and a decode worker and coordinates the request.
2. The prefill worker processes the prompt. The decode side reserves cache space before the transfer; exactly when that reservation happens depends on the implementation.
3. The system moves KV blocks and the metadata needed to interpret them to the decode side. Metadata identifies the request, token positions, and cache locations. First-token results or the token IDs needed to resume also cross the boundary.
4. Once the required state is available, decode continues generation and streams the response through the serving layer. Transfer buffers must remain valid until the handoff completes, then be released safely.

In a common arrangement, prefill selects the first output token and decode consumes it to predict the second. The exact boundary is framework-specific: some paths recompute a final position. Transfers can also overlap computation when the backend supports it. The diagram shows a simple completed-prefill handoff.

RDMA, Remote Direct Memory Access, lets a network adapter transfer registered memory with limited CPU involvement. With appropriate hardware and software, that includes GPU memory. It reduces transfer overhead; it does not remove the bandwidth cost. SGLang's engineering write-up explains the sender, receiver, and asynchronous transfer machinery.

How much state are we moving?

For a conventional full-attention model, an approximate logical KV payload is:

Here, $L$ counts attention layers, $H_{\mathrm{KV}}$ counts KV heads, $d_{\mathrm{head}}$ is the head size, $N$ is the number of processed tokens, and $b$ is the bytes stored per value.

The factor of two counts keys and values. KV heads are the heads whose keys and values are stored, which can be fewer than query heads.

Take an illustrative model with 32 layers, 8 KV heads, head size 128, and two-byte values. An 8,192-token prompt produces 2 × 32 × 8 × 128 × 8192 × 2 bytes, exactly 1 GiB of KV data before padding or replication.

At an effective 25 GB/s, transferring that payload alone takes about 43 ms. This is arithmetic, not a measured framework result. Allocation, synchronization, and contention add cost; overlapping transfer with computation can reduce the exposed delay. Compressed caches, sliding-window attention, and hybrid models need different accounting.

That is why the GPU interconnect matters as much as the worker layout. A slow handoff can protect streaming latency while making the initial response worse.

What vLLM and SGLang actually provide

The following describes the documentation checked on October 18, 2026. Supported combinations change, so the linked compatibility guidance matters.

vLLM runs separate instances and uses a KV connector configured through --kv-transfer-config. Its disaggregated-prefill documentation lists connectors including NIXL, Mooncake, and LMCache integrations. A proxy coordinates the two stages; selecting a connector alone does not supply a production routing policy.

The standard NixlConnector path lets decode pull KV blocks from prefill. A separate push-mode connector lets prefill write them into decode memory. Both move state toward decode; what differs is who initiates the transfer. Models, cache layouts, versions, and parallelism settings must satisfy the connector's compatibility requirements. vLLM still labels disaggregated prefilling experimental.

SGLang exposes explicit worker roles through --disaggregation-mode prefill and --disaggregation-mode decode. Its PD guide documents Mooncake and NIXL transfer backends, with a model gateway coordinating routing. Sender/receiver setup and destination allocation precede the actual transfer, although allocation timing is configurable in supported configurations. These details are serving mechanics, not changes to the model's attention calculation.

When I would use it

I would start by measuring a colocated setup with chunked prefill. Then I would compare the split setup using the same total GPUs, model, prompt/output lengths, arrival rate, and cache-hit conditions.

The costs deserve equal attention: each pool needs model weights resident, splitting capacity can leave one pool idle while the other queues, and routing, transfer failures, cancellation, and cache cleanup become distributed problems. Repeated conversation prefixes also need cache-aware placement or reuse across workers.

For short prompts or light traffic, keeping both phases together can be simpler and faster. For sustained mixed traffic with strict streaming targets and a fast transfer path, disaggregation becomes much more compelling. vLLM's practical serving guide gives examples of both the benefits and the handoff penalty.

What I take away

Disaggregation buys control over where work runs and how each phase scales. It pays when reduced interference and better resource allocation outweigh transfer, queueing, and operational costs. I would judge it by first-token latency, streaming tail latency, and goodput together.

References

• DistServe, OSDI 2024, the paper and presentation on phase separation and goodput.
• NVIDIA: Inference Optimization, KV memory accounting and attention variants.
• vLLM: Disaggregated Prefilling, instances, connectors, and scope.
• vLLM: NixlConnector Usage, push-mode design, and compatibility matrix, transfer protocols and supported configurations.
• SGLang: PD Disaggregation and Model Gateway, worker roles, transports, allocation, and routing.
• SGLang Team: Deploying DeepSeek with PD Disaggregation, the transfer design and phase-specific optimization.
• vLLM: Taking vLLM Apart, deployment trade-offs and workload-dependent results.