Understanding the KV Cache

Imagine rereading a 300-page book every time you write the next sentence. You would still produce the same sentence, but you would spend almost all your time re-reading instead of writing.

That is inference without a cache. The model has already looked at every earlier token, yet a naive decode step would recompute their keys and values from scratch. The KV cache is the model’s bookmark: it keeps the part of that work that never changes.

The short version

Inside every attention layer, each token produces a query, a key, and a value.

• The new token’s query asks: “what should I pay attention to?”
• The cached keys describe every earlier token.
• The cached values carry the information those tokens contribute.

For the next attention calculation, the model needs the new token’s query and its new key and value. The keys and values of earlier tokens do not change, so they can be stored and reused. That is the entire idea.

Cache growth

Prefill fills the cache for every prompt token. Each decode step adds one column for the new token, in every layer.

Read this diagram as text

The four prompt tokens, The cat sat on, are processed together during prefill and fill four columns of the cache in every transformer layer. During decode, the token the is appended. Its key and value vectors are computed and written as a fifth column. The next token's query will read all five columns, but no previous key or value needs to be recomputed.

The cache is not a summary of the conversation. It is a stack of tensors with a precise shape:

K and V, for every layer, every KV head, every token, every batch item

Multiply that out and you get the formula this lesson keeps returning to.

Why the cache exists

Attention compares a query with every key, then uses the resulting weights to mix the values:

attention(Q, K, V) = softmax(QKᵀ / √d) V

During prefill, the model has the whole prompt and computes Q, K, and V for all of it. During decode, the model has one new token. Its Q is new, but the K and V from earlier tokens are exactly what they were in the previous step.

Without a cache, every decode step would recompute K and V for the entire history. With a cache, it computes them once per token and reads them thereafter.

The saving is large, but it is not free. Attention still reads every cached key and value at every step. The cache removes recomputation, not reading.

Model weights and sequence state are different

Each sequence needs KV state for its own tokens, even when requests share model weights. Serving systems can reuse cache blocks for a shared prompt prefix, so multiplying one sequence’s cache by concurrency is an estimate for equal-length sequences without prefix sharing.

Watch the cache grow

Press play, or move the step slider. The prompt is cached once. Every generated token adds one more key/value column, and the next token is scored against everything in the cache.

Interactive · token generation

Watch the cache grow, one token at a time

The prompt is processed in one pass. Every token after that needs its own forward pass and appends one more K/V column to the cache. The displayed candidates and probabilities are illustrative, not outputs from a real model.

Context

Thecatsaton

KV cache

TheK · VcatK · VsatK · VonK · V

Cache size

4 tokens

Generated

0 of 6

Last token

—

Next-token probabilities

• the62%
• a14%
• my8%

Prefill complete. 4 tokens are cached. Nothing is generated yet.

PlayResetStep

0

Notice what does not happen: earlier columns are never recomputed. Notice what does happen: the cache gets one column longer on every step. That growth is the bill.

The memory calculation

Let:

• L = number of transformer layers
• H = number of key/value heads
• D = head dimension
• S = sequence length, including the prompt and every generated token
• B = batch size, or concurrent sequences
• b = bytes per stored value

In a decoder where every layer uses full attention, each token needs a key and a value for every layer and every key/value head. For equal-length sequences without shared prefixes, the raw cache size is:

KV bytes = 2 × L × H × D × S × B × b

The leading 2 is the key/value pair. If you remember one thing from this lesson, remember that the cache scales linearly with context and batch size. Doubling either one doubles the cache.

A worked example

Take a model with 32 layers, 8 key/value heads, a head dimension of 128, FP16 storage, an 8,192-token context, and one sequence.

Step	Calculation	Result
One token, one layer	2 × 8 × 128 × 2 bytes	4 KiB
One token, all layers	4 KiB × 32	128 KiB
One full 8K sequence	128 KiB × 8,192	1 GiB
Eight concurrent sequences	1 GiB × 8	8 GiB

Those assumptions are deliberately simple: FP16 values, equal-length sequences, full attention in every layer, no shared prefixes, and no allocator metadata or fragmentation. Real serving memory also includes weights, activations, temporary workspaces, and allocator effects. Measure headroom in the target stack rather than applying a universal percentage.

Interactive · memory estimate

How much memory does the KV cache need?

Change the model shape, context, and batch size. The estimate updates as you move a control.

LayersL

32

896KV headsH

8

164Head dimensionD

128

32256Context lengthS

4,096

51265,536Concurrent sequencesB

8

164

Estimated KV cache

4.00 GiB

for 8 concurrent sequences at 4,096 tokens

Per token, per layer

4 KiB

Per token, all layers

128 KiB

One full sequence

512 MiB

2 · 32 · 8 · 128 · 4,096 · 8 · 2 B

Assumptions. Every layer stores a key and a value for each token. Values use the selected precision. This ignores allocator metadata, fragmentation, activations, and temporary workspaces. It also assumes equal-length sequences, full attention in every layer, and no shared prefixes. Measure production headroom in the serving stack you use.

Reset to defaults

What you can trade

There is no setting that makes the cache disappear. There are only choices about where the cost lands.

Levers that change the KV cache bill

Lever	What it saves	What it costs
Shorter context	Memory and attention time	Less material for the model to reason over
Fewer KV heads (GQA or MQA)	Several times less cache	May reduce quality or capacity per head; requires evaluation
Lower cache precision	2× for FP8, 4× for INT4	May introduce numerical error; evaluate on the target task
Smaller batch	Lower peak memory	Lower GPU utilization and throughput
Recompute instead of cache	No cache memory	Far slower decode; repeated work

Some recent open models use grouped-query attention (for example, Llama 2 34B and 70B) because it reduces the cache without changing the context window; other systems use multi-query attention.

Long context changes the concurrency math

A larger context window lets one request carry more information. Filling more of that window requires more cache; some allocators also reserve capacity ahead of use. At a fixed memory budget, longer active contexts can mean fewer concurrent users. Capacity planning has to treat context length and concurrency as one decision.

Plan for fragmentation, not just the formula

Real caches are allocated in blocks, and sequence lengths are not known in advance. Paged attention and block allocators reduce wasted space, but they do not eliminate it. Measure per-sequence cache use under realistic traffic, cap output length, and leave headroom for the allocator and activations.

Key takeaways

• The cache stores keys and values for every processed token, layer, and KV head — not queries, attention weights, or logits.
• Prefill fills the cache for the prompt; each decode step appends exactly one column.
• Cache memory grows linearly with context length and batch size, and is multiplied by 2 × layers × KV heads × head dimension × precision.
• GQA, quantization, and shorter contexts reduce the bill, but each trades quality, capacity, or context.
• The cache makes decode practical; it does not make long-context serving free.

Try it yourself

1. 01Using the same model as the worked example, estimate the cache for a 32,768-token context and four concurrent sequences. Keep the assumptions explicit.

   Show a hint

   Start from 128 KiB per token, multiply by 32,768, then by four, and divide by 1,024³ for GiB.

2. 02Your service is memory-bound at 8 concurrent users. Product wants a 64K context window. Name two levers you would test before buying more GPUs, and the risk each introduces.

   Show a hint

   Consider grouped-query attention and cache quantization. Both reduce bytes; both can affect quality or evaluation effort.

Interview questions

Why cache keys and values, but not queries?

A query belongs to the token currently being processed. Once that token has attended to the cache, its query is not needed by later tokens. Keys and values, by contrast, are needed again by every future token, so storing them avoids recomputation.

Why is decode memory-bandwidth-bound even with a KV cache?

A conventional decode step processes one new token, so there is relatively little arithmetic per request. The GPU still reads model weights and the cached keys and values needed by each attention layer. For many serving shapes the ratio of data moved to work done is high, making memory bandwidth the main bottleneck.

How does grouped-query attention reduce cache size?

GQA shares one key/value head across a group of query heads. The cache stores only the smaller number of KV heads, so memory drops by the ratio of query heads to KV heads. Attention still computes queries for every head.

Related lessons

• LLM Internals & Architecture · Inference foundationsHow LLM inference works9 min

What comes next: Inside a single forward pass

Now that we understand the outer generation loop — prefill reads the prompt and fills the cache, then decode generates one token at a time while growing the cache — the natural question is: what actually happens inside a single forward pass?

The KV cache is the bookmark between steps. But inside each step, the model runs a full transformer forward pass. It takes the new token (or the full prompt during prefill), combines it with the cached keys and values, and produces the next-token logits.

How does the model turn a sequence of tokens into the next-token score vector?

That is the core course entry: Inside the Transformer Architecture. There we trace:

1. Token embeddings — how token IDs become continuous vectors
2. The residual stream — the highway that carries information across all layers
3. Causal self-attention — how each token queries the cache and mixes values, with the causal mask ensuring tokens only see the past
4. The MLP block — per-position non-linear processing
5. The output projection — how the final hidden state at the last position becomes a vocabulary-sized logit vector

The KV cache lesson showed you the outer loop and the memory bill. The transformer architecture lesson opens the inner loop and shows you the mathematical machinery that produces each next-token prediction.

Sources and scope

• Hugging Face cache explanation documents per-layer key/value shapes and cache updates.
• vLLM automatic prefix caching describes reuse of shared prompt-prefix KV blocks across requests.
• PagedAttention paper describes cache memory management and sharing in a serving system.
• Pope et al., “Efficiently Scaling Transformer Inference” (arXiv:2211.05102) analyzes decode memory-bandwidth bottlenecks.
• Ainslie et al., “GQA” (arXiv:2305.13245) explains grouped-query attention.
• Touvron et al., “Llama 2” (arXiv:2307.09288) reports GQA for 34B and 70B models.

The calculator gives raw KV tensor bytes for a uniform full-attention decoder. It is not a production GPU-memory estimator.