How LLM inference works

A large language model does one thing: given a sequence of tokens, it produces a score for every token that could come next. Inference is the loop that turns those scores into text.

That sounds small. It is also the whole product. The latency your users feel, the cost per request, and the number of people a GPU can serve all come from how efficiently that loop runs.

The one-sentence version

Prefill reads the prompt in parallel; decode writes the answer one token at a time.

Everything else in this lesson follows from that split.

Inference pipeline

A prompt is tokenized, processed in one prefill pass, and then extended one token at a time during decode.

Read this diagram as text

The prompt is split into tokens and converted to IDs. Prefill processes all prompt tokens in one forward pass and produces a first next-token distribution plus the key and value vectors for the prompt. Decode then runs once per output token. Each decode pass appends one token, reads the cached keys and values, and produces the next distribution. The loop stops at a stop token or a length limit.

Trace the decode loop

The pipeline diagram shows the phases. This interactive trace makes the loop invariant explicit: the sampled token becomes the next input, the cache grows by exactly one column per step, and the loop stops on a stop token or a length limit — whichever happens first.

Decode loop trace

Cache columns

6 tokens

Output tokens

0 of 8

Stop reason

not yet

Stop token sampled

Length limit reached (8)

Interactive decode loop trace. Enable JavaScript to step through all 7 states. Static description: State 0 (Prefill complete): cache 6 columns, sampled "Because". State 1 (Decode step 1): input "Because", cache 6→7, sampled "of", continue. State 2 (Decode step 2): input "of", cache 7→8, sampled "Rayleigh", continue. State 3 (Decode step 3): input "Rayleigh", cache 8→9, sampled "scattering", continue. State 4 (Decode step 4): input "scattering", cache 9→10, sampled ".", continue. State 5 (Decode step 5): input ".", cache 10→11, sampled <eos>, stop: stop token. State 6 (Stopped): cache 11 final, stop token reached.

Interactive · decode loop

Trace one decode loop

Prefill runs once. Each decode step processes one token, reads the full cache, appends one column, samples the next token, and checks stop conditions.

Prompt

Whyistheskyblue?

Output so far

Because

Cache columns

6 tokens

Output tokens

0 of 8

Stop reason

not yet

Stop token sampled

Length limit reached (8, configured — not hit here)

Prefill complete. 6 prompt tokens cached. First token sampled: "Because". Stop condition not yet evaluated.

PreviousNextResetStep

0

Text equivalent: all decode loop states

1. State 0 (Prefill complete): cache 6 columns, sampled "Because", stop not yet evaluated.
2. State 1 (Decode step 1): input "Because", cache 6→7, sampled "of", continue.
3. State 2 (Decode step 2): input "of", cache 7→8, sampled "Rayleigh", continue.
4. State 3 (Decode step 3): input "Rayleigh", cache 8→9, sampled "scattering", continue.
5. State 4 (Decode step 4): input "scattering", cache 9→10, sampled ".", continue.
6. State 5 (Decode step 5): input ".", cache 10→11, sampled <eos>, stop: stop token.
7. State 6 (Stopped): cache 11 final, stop token reached.

One prompt, one prefill, then a decode loop that appends one cache column per output token and stops on a stop token or a length limit.

Read this diagram as text

Step through the seven states of a concrete decode run. Prefill processes the six-token prompt 'Why is the sky blue ?' and samples the first output token 'Because'. Each decode step processes the previous token, reads the full cache, appends one key/value column, samples the next token, and checks stop conditions. The loop stops at state 5 when '<eos>' is sampled. A length limit of 8 tokens is also configured but not reached in this run.

Prefill: read the prompt once

Prefill is the model’s first pass over the prompt. Every prompt token is available at the same time, so the transformer can process them together in large matrix multiplications.

Two things come out of this pass:

1. A probability distribution for the first output token.
2. The keys and values for every prompt token, stored for later steps.

Because the work is parallel, prefill can use the GPU efficiently. It is usually compute-bound at sufficiently large batch sizes: the limiting factor is how fast the hardware can perform the matrix operations, not how fast it can move data. At small batch sizes, loading model weights can dominate instead (Pope et al., 2022, §3.2).

Prefill is a major contributor to time to first token (TTFT). The measured delay also includes work such as request queuing, tokenization, and network transit. A longer prompt usually increases the prefill part of that delay.

Decode: one token per pass

Decode is the loop everyone feels but rarely sees. To generate token number two, the model needs token number one. To generate token number three, it needs token number two. The dependency is strict.

Each pass looks like this:

• take the most recent token;
• compute its query, key, and value vectors;
• let its query attend to every cached key and value;
• produce a new distribution;
• sample one token;
• append that token to the context.

Only one new token is processed per pass, so the matrix operations are small. The GPU spends more time reading the model’s weights and the growing cache than doing arithmetic. Decode is usually memory-bandwidth-bound at typical interactive batch sizes; at very large batch sizes or high request rates it can become compute-bound (Pope et al., 2022, §4; Kwon et al., 2023, §8).

This is why output length is the expensive dimension. A 200-token answer requires 200 sequential passes. A 2,000-token answer requires 2,000.

prompt_ids = tokenize(prompt)
cache, logits = model.prefill(prompt_ids)

output = []
next_token = sample(logits)

while next_token != stop_token and len(output) < max_tokens:
    output.append(next_token)
    logits, cache = model.decode(next_token, cache)
    next_token = sample(logits)

return detokenize(output)

The code is illustrative, but the shape is real: prefill runs once, decode runs once per token, and the cache is passed forward each time.

The same model, two different machines

Prefill and decode put pressure on different parts of the hardware. Treating them as one workload hides the tradeoffs that matter when you serve a model.

Prefill and decode have different bottlenecks

	Prefill	Decode
Input	All prompt tokens	One new token
Parallelism	Across the whole prompt	Strictly sequential
Typical bottleneck	Compute	Memory bandwidth
Writes to cache	Every prompt token	One token per step
User-visible effect	Contributes to time to first token	Time per output token

Real workloads overlap these phases across requests. The distinction still explains where latency and memory pressure come from. Which phase is compute- or memory-bound depends on batch size and request rate (Pope et al., 2022; Kwon et al., 2023, §8).

Why serving systems split the phases

A prefill-heavy request and a decode-heavy request want different batch shapes. Production schedulers may separate or carefully interleave them so a long prompt does not stall every other user’s token stream. That is one reason naive batching is a poor capacity plan.

Where the memory goes

The model’s weights are fixed, but the cache is not. Every token in the context needs a key and a value for every layer and every key/value head. As the conversation grows, so does the cache.

For a single request this is manageable. Multiply it by concurrent users, and the cache can become the largest consumer of GPU memory. The next lesson works through that bill in detail.

Context is not free

A larger context window raises the ceiling on what the model can read. It also raises the memory required to keep every token resident. The window is a capability, not a guarantee that serving the full window is economical.

Key takeaways

• Prefill processes the prompt in parallel and contributes to time to first token; queuing and network delay also matter.
• Decode produces one token per forward pass and sets the pace of the output stream.
• Decode is usually memory-bandwidth-bound at typical batch sizes because each step moves weights and a growing cache for very little arithmetic; at very large batches or high request rates it can become compute-bound.
• The KV cache is what makes decode practical, and it is also what makes long contexts expensive.

Try it yourself

1. 01A request has a 4,000-token prompt and a 100-token answer. Which phase dominates time to first token, and which phase dominates total generation time?

   Show a hint

   Separate the one-time cost from the per-token cost. The answer length is the multiplier on decode.

2. 02Your p95 time to first token is acceptable, but tokens per second collapsed after traffic doubled. Which resource would you inspect first, and why?

   Show a hint

   More concurrent sequences mean more cache and more competition for memory bandwidth during decode.

Interview questions

Why can’t a model generate all output tokens in parallel?

Each token is sampled from a distribution conditioned on the tokens before it. The next input is not known until the previous token has been chosen, so the dependency chain is sequential. Techniques such as speculative decoding relax the cost of the loop, but they do not remove the dependency.

What does the KV cache store, and what does it not store?

It stores the key and value vectors for every processed token, in every layer and key/value head. It does not store queries, attention weights, logits, or the model weights. Queries are recomputed for the new token; the cache exists so keys and values are not.

Related lessons

• LLM Internals & Architecture · Inference foundationsUnderstanding the KV Cache12 min

Sources and scope

• NVIDIA NIM metric definitions explain what measured time to first token includes.
• Hugging Face cache explanation shows how a cached autoregressive forward pass updates and reads keys and values.
• Pope et al., “Efficiently Scaling Transformer Inference” (arXiv:2211.05102) analyzes memory time, weight loading, KV-cache loading, and MFU for prefill and decode.
• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (arXiv:2309.06180) describes block allocation, shared prefixes, fragmentation, and high-request-rate behaviour.

The code in this lesson is pseudocode for a conventional autoregressive decoder. Serving systems may batch, split, or cache work differently.