Prefill and Decode: Why an LLM Reads Quickly but Writes Slowly

An LLM can read a long prompt and then spend several seconds writing a short reply. While learning about inference engineering, I found that this becomes much less surprising once you separate the work into two phases: prefill, which processes the prompt, and decode, which continues the response.

They run the same model. What changes is how much work the GPU can do together, and how much data it must move to do it.

Start with one request

Consider a prompt asking a model to summarize a document. Before inference, a tokenizer converts the text into tokens, small pieces of text represented by integer IDs. A token might be a word, part of a word, or punctuation.

A typical decoder-only language model predicts the next token from the tokens before it. That process is autoregressive: each prediction becomes part of the input for the next prediction. We will follow ordinary generation with a KV cache, without speculative decoding.

The prompt tokens are already known. The response tokens are not. This is the reason the two phases behave differently.

Prefill: process the known context

During prefill, the prompt passes through the model's layers. Within each layer, the GPU can work on many prompt positions together. The layers themselves still run in order.

The model uses attention to let each position gather information from earlier positions. A causal mask prevents it from looking ahead, even though the prompt is processed in parallel.

Attention produces three vectors for each position: a query, which asks what information is relevant; a key, which helps match that query; and a value, which supplies the information to combine. The engine saves the keys and values at each attention layer in the KV cache.

The last prompt position produces scores for possible next tokens. A selection rule, such as choosing the highest score or sampling, picks the first output token. Prefill normally gives us the first response token as well as the prompt's KV cache. NVIDIA's inference overview describes this boundary.

Decode: extend the context one step at a time

Suppose prefill has produced token y1. The next forward pass consumes y1, computes its keys and values, adds them to the cache, and uses attention over the cached context to predict y2. The following pass consumes y2 and predicts y3.

Notice the one-token offset: generating y1 does not yet put y1's keys and values in the cache. Those are computed when y1 becomes the next input.

Caching saves us from recomputing earlier tokens' keys and values. It does not remove attention over the past: with full attention, each new position still reads the growing cache. Think of keeping research notes open beside you. You can consult them without rereading and reprocessing the original document. Hugging Face's cache explanation walks through the underlying operations.

Why the GPU experiences them differently

A GPU works best when it can apply the same model weights to lots of inputs together.

During a sufficiently large prefill, many token positions share weight reads. Large matrix multiplications give the arithmetic units plenty of work. Prefill is therefore often compute-bound, meaning calculation throughput limits its speed.

At low decode batch sizes, each pass applies those weights to only a few new positions. Reading weights and cached attention state from GPU memory can take longer than the arithmetic. Decode is then memory-bandwidth-bound: the limit is how quickly bytes reach the compute units, even when everything fits in memory.

These are common regimes, not fixed laws. Batching more requests gives decode more arithmetic per weight read. Short prefills, long contexts, kernel overhead, and communication between GPUs can change the bottleneck. TNG's measurements of concurrent inference show how increasing concurrency changes resource use.

Two waits worth measuring

Time to first token (TTFT) is the wait from submitting a request until its first output token arrives. It includes queueing, input preparation, prefill, and delivery overhead.

Inter-token latency (ITL) is the gap between successive output tokens. Its average after the first token is commonly called time per output token (TPOT). Client-side buffering can make observed gaps differ from GPU execution time.

For an illustrative request with a 400 ms TTFT and 100 output tokens spaced 20 ms apart, completion takes roughly $400 + 99 \times 20 = 2{,}380\ \mathrm{ms}$. These are made-up timings, not a benchmark. Faster prefill reduces the initial wait; faster decode matters increasingly as the answer grows.

Throughput, the total tokens served per second across users, measures something else. A bigger batch can improve throughput while making individual users wait longer.

Serving many users brings the phases together

Inference engines use continuous batching to remove finished requests and admit new ones between execution steps. That creates a scheduling problem: a new document needs prefill while existing requests need their next decode step.

An engine that schedules a whole long prefill can delay those streams. Chunked prefill divides that prompt work into smaller pieces, allowing decode work to progress between or alongside them in mixed batches.

vLLM V1 prioritizes decode when chunked prefill is enabled, then fills the remaining token budget with prefill work. SGLang exposes a chunked-prefill size for controlling prompt processing. Smaller chunks can protect streaming latency, but add scheduling overhead and may prolong prompt processing.

Another optimization, prefix caching, reuses KV state for an identical cached beginning of a prompt. It can skip repeated prefill work. It does not eliminate the decode loop.

What I take away

Prefill builds the prompt state and predicts the first token. Decode extends that state through dependent steps. Their different shapes explain why a server can ingest tokens quickly yet generate them slowly, and why a single tokens-per-second number tells us so little about the experience.

The next question is whether these phases should share GPUs at all. That is the subject of prefill-decode disaggregation.

References

• NVIDIA: Mastering LLM Techniques, Inference Optimization, model execution and GPU bottlenecks.
• Hugging Face Transformers: How caching works, attention state and incremental generation.
• TNG: Prefill and Decode for Concurrent Requests, latency measurements and scheduling diagrams.
• vLLM: Optimization and Tuning, current V1 scheduling behavior.
• SGLang: Hyperparameter Tuning, prompt chunking and serving controls.
• vLLM: Automatic Prefix Caching, reuse of repeated prompt computation.