Where Inference Time and Memory Go

The one-sentence version

Inference cost splits into four measurable pieces — weight bytes, KV cache bytes, compute (FLOPs), and HBM traffic — and the dominant piece changes with prompt length, context length, batch size, and quantization choices.

Everything else in this lesson follows from that split.

---

1. Why a byte-level breakdown matters

You are debugging a latency regression. The model is the same, the GPU is the same, but TTFT doubled after a context-length increase. Was it the quadratic attention compute? The KV cache bandwidth? The weight streaming? A quota limit?

You cannot answer this by looking at peak TFLOPS or peak bandwidth numbers. Those are vendor ceilings. What matters is how much data your workload actually moves, how many FLOPs your kernels actually execute, and which resource hits its limit first — on your hardware, with your framework, at your batch size.

This lesson gives you the arithmetic to estimate the payload, the vocabulary to separate compute from traffic, and a measurement plan that can confirm or refute a bottleneck hypothesis.

Prerequisites from Track 1

This lesson assumes you know the difference between prefill (processing the prompt in one forward pass) and decode (generating tokens one step at a time). If that distinction is new, read Prefill and Decode and KV Cache Explained first.

---

2. Units: GB vs GiB

Vendor specifications use decimal gigabytes (GB = 10⁹ bytes). GPU memory allocators and nvidia-smi report binary gibibytes (GiB = 2³⁰ bytes).

Label	Bytes	Equals
1 GB	1,000,000,000	0.931 GiB
1 GiB	1,073,741,824	1.074 GB

An H100 SXM is marketed as “80 GB”. Treat that as a nameplate, not a measurement. 80 × 10⁹ bytes is 74.51 GiB, and the card does not stop there: NVIDIA’s own SKU-inventory example lists H100 80GB HBM3 devices at 81,559 MiB of total memory, which is 79.65 GiB or 85.52 GB. So the product label is neither the binary total nor even the decimal total. Read real capacity from the device:

nvidia-smi --query-gpu=name,memory.total --format=csv

Reported total is an upper bound on addressable memory, not free memory. The driver, CUDA context, firmware, and allocator pools reserve a portion before any tensor is allocated, and that portion differs by driver version, framework, and what else is on the card.

Always state which unit you are using. Mixing them is the most common source of “off by 7%” errors in capacity planning.

---

3. Weight payload

3.1 Parameter count: start from a nominal 8B

The Hugging Face config.json for meta-llama/Meta-Llama-3-8B stores architecture hyperparameters, not a parameter count, and the repository is gated (an unauthenticated request returns HTTP 401). So this lesson does not claim an exact parameter count. It uses a nominal 8 × 10⁹ parameters and keeps the assumptions visible:

• 32 layers
• d_model = 4,096
• FFN = 14,336
• 32 query heads, 8 KV heads (grouped-query attention)
• Head dimension = 128
• Vocabulary ≈ 128,256 for the tokenizer; the paper rounds to 128,000
• Untied LM head

Those assumptions multiply out to roughly 8.03 × 10⁹ parameters. This lesson does not use that derived figure: every weight, FLOP, and payload number below uses the explicit nominal N = 8.00 × 10⁹, so you can check each product by hand. The difference is small — about 0.4% — and it is not the only approximation: the paper rounds the vocabulary to 128,000 where the tokenizer has 128,256, and weight tying (which this example assumes is off) would change the count materially. Ignore the difference for hypotheses about which term dominates; do not ignore it for capacity. When you need exact accounting, count tensor elements from the checkpoint metadata/weights (or use an officially published count) rather than quoting any of these figures.

3.2 Weight bytes by precision

Precision	Bytes/param	Decimal GB	Binary GiB	Approx. bytes (nominal N = 8 × 10⁹)
FP16 / BF16	2	16.00 GB	14.90 GiB	16.00 × 10⁹ B
FP8	1	8.00 GB	7.45 GiB	8.00 × 10⁹ B
INT4 (raw)	0.5	4.00 GB	3.73 GiB	4.00 × 10⁹ B

INT4 is a floor, not a budget

The 4.00 GB figure is 4-bit payload with no scales, zero-points, group metadata, alignment padding, or kernel-required layouts. Real 4-bit checkpoints are larger. vLLM’s W8A8 documentation quotes a “2x reduction in model memory requirements” for FP8 rather than promising exact ratios.

---

4. KV cache payload

The KV cache formula for B sequences:

KV_bytes = 2 (K and V) × L × H_kv × D × S × B × b

For Llama 3 8B:

• L = 32 layers
• H_kv = 8 (KV heads, not the 32 query heads)
• D = 128 (head dimension)
• b = bytes per element (2 for FP16/BF16, 1 for FP8)

4.1 Per-token cost (all layers, batch = 1)

Precision	Bytes/token	KiB/token
FP16 / BF16	131,072	128 KiB
FP8	65,536	64 KiB

4.2 Context-length scaling (batch = 1, FP16 KV)

Context S	Bytes	GiB
2,048	268,435,456	0.25 GiB
4,096	536,870,912	0.50 GiB
8,192	1,073,741,824	1.00 GiB
16,384	2,147,483,648	2.00 GiB
32,768	4,294,967,296	4.00 GiB

Weight precision and KV precision are separate decisions

A 4-bit or FP8 weight file does not make the cache 4-bit. Serving stacks configure them independently: vLLM’s --kv-cache-dtype defaults to auto (“use model data type”) and can be set to float16, bfloat16, fp8_e4m3, fp8_e5m2, or integer cache types. So the same 8B FP8 checkpoint has 7.45 GiB of weights and either 1.00 GiB (FP16 KV) or 0.50 GiB (FP8 KV) of cache at 8K context. Read b off the cache configuration, never off the weight file.

---

5. Complete payload example: Llama 3 8B on H100 SXM (batch = 1)

Quantity	Formula	Decimal GB	Binary GiB
Parameters	nominal 8 × 10⁹ (assumptions in §3.1)	—	—
FP16 weights	N × 2	16.00 GB	14.90 GiB
KV @ 8,192, FP16	2·L·H_kv·D·S·b	1.07 GB	1.00 GiB
Nominal payload (weights + KV)	sum	17.07 GB	15.90 GiB
Reported device total	81,559 MiB from the device	85.52 GB	79.65 GiB
Arithmetic remainder	79.65 − 15.90	—	63.75 GiB
Same cache at FP8 KV	b = 1	0.54 GB	0.50 GiB

The 63.75 GiB remainder is an upper bound on everything else — KV blocks for other sequences, activations, workspace, allocator caches, CUDA context, and any other process. It is not free memory, and it is not a capacity plan: per-sequence overheads, activation peaks, and allocator behaviour all consume part of it. Do not quote an arithmetic concurrency maximum built from this remainder; measure the real one.

Payload vs allocated memory — they are not the same

Category	Payload (calculated)	Memory you actually see (measured)
FP16 weights	14.90 GiB	At least 14.90 GiB of tensor allocation, plus alignment and any buffer the loader keeps
KV cache @ 8K FP16	1.00 GiB	At least 1.00 GiB, plus block tables and per-block padding
Activations (prefill)	Not counted	Peak set by kernel tiling, batch, sequence length; measure it
Activations (decode)	Not counted	Small per step but non-zero; measure it
CUDA context, driver, other processes	Not counted	Reserved before tensors; visible in driver-reported usage
Cached free blocks (allocator)	Not counted	Difference between reserved and allocated
Total	15.90 GiB	Three scopes, measured on the same process and device at the same time

Payload is the tensor-byte arithmetic. torch.cuda.max_memory_allocated() counts tensor allocations only; max_memory_reserved() counts the caching allocator's pool, including cached free blocks; nvidia-smi or NVML reports what the driver has granted the process. None of the three equals the payload. A quantized checkpoint's allocation is not automatically larger than the FP16 one — its payload is smaller and its metadata is extra — so compare payloads, not labels.

Read the three scopes separately. PyTorch’s caching allocator keeps freed blocks, so unused memory still appears as used in nvidia-smi; the docs distinguish tensor-occupied memory (memory_allocated) from allocator-managed memory (memory_reserved). Driver-visible usage is larger again because it includes the CUDA context and any other process on the card.

These three numbers are usually ordered allocated ≤ reserved ≤ driver-visible for one process on one device, but that is an observation, not a guarantee: another process can grow between your two samples, the allocator can release a segment, and the driver figure is quantized. Report all three from the same process and the same moment when you report memory, and never derive a capacity plan from the payload alone.

---

6. What moves during prefill and decode

6.1 The data-flow principle

Pope et al. (§2) describe the model: weights and KV cache live in HBM and are moved to the compute cores once per forward pass. This is the starting point for traffic estimation.

For one sequence at S = 8,192 in FP16:

• Weights: 16.00 GB
• KV cache: 1.07 GB
• Nominal payload per forward pass: ~17.07 GB

That is the payload if every weight and KV byte is read exactly once. It is the figure this lesson uses for capacity planning and for stating hypotheses; §6.3 explains why it is not a measurement of DRAM traffic.

6.2 Prefill (prompt processing)

During prefill, the entire prompt of length P is processed in one forward pass.

Compute (derived, not measured):

• Linear projections and MLP: 2 · N · P FLOPs (one matmul-add per parameter per token)
• Attention: 2 · L · H_q · D · P² FLOPs for the full P × P score matrix (H_q = 32 query heads); a causal mask removes roughly half of the scored pairs, so halve this term when you care about the constant

Prompt length P	Attention FLOPs	Linear FLOPs (2NP)	Total	Attention share
256	0.017 TFLOP	4.10 TFLOP	4.11 TFLOP	0.4%
2,048	1.10 TFLOP	32.77 TFLOP	33.87 TFLOP	3.2%

At both lengths the linear terms outweigh attention in this simplified count, but do not turn that into a timing threshold. The model leaves out softmax, masking, memory traffic, and per-kernel efficiency, so a FLOP crossover would not be a latency crossover. Measure prefill at two prompt lengths instead (§9).

A FLOP count is not a bottleneck claim

Dividing these counts by any peak number yields a time no kernel achieves. The real cost also includes weight streaming, kernel launch overhead, and low occupancy at small P. These counts exist to be compared against a measured FLOP rate, not to classify the phase as “compute-bound.”

Nominal HBM payload: one full pass over weights (16.00 GB) plus the KV cache written for the prompt (P × 128 KiB per token in FP16). The KV write is new allocation; the weight read is streaming. Attention kernels also read and write intermediate tiles, and flash-style kernels avoid materializing the full P × P score matrix in HBM, so actual traffic is not simply this sum.

6.3 Decode (token generation)

Each decode step processes one token per sequence.

Compute per step:

• Linear/MLP: 2N = 16.00 GFLOP
• Attention: 4 · L · H_q · D · S = 4.29 GFLOP (one query row attends to all S keys; no causal halving)
• Total: ~20.29 GFLOP/step

Nominal HBM payload per step:

• Weights: 16.00 GB (read again each step)
• KV cache read: S × 128 KiB = 1.07 GB at S = 8,192 (grows linearly with context)
• KV cache write: 128 KiB (one new token’s K and V)
• Total: ~17.07 GB + a small write

Nominal bytes are not a DRAM floor

• Actual traffic can be lower: if the attention working set fits in L2, some K/V reads are served from cache; blocked or tiled kernels can keep a slice resident while iterating.
• Actual traffic can be higher: tiling re-reads K/V per query tile, paged allocation reads block tables, weights may be re-read rather than held resident, and attention kernels read and write score and output tiles of their own.
• Not counted at all: activations, workspace, allocator caches, and communication buffers.

So weights + KV is a capacity payload and a testable hypothesis about traffic — not a guaranteed lower bound on nvidia-smi or profiler counters. Only dram__bytes_read from the real workload settles which way it moves.

6.4 Arithmetic intensity (one reference point)

For decode, B = 1, S = 8,192, FP16:

• FLOPs: 20.29 GFLOP/step
• Nominal bytes: 17.07 GB/step
• Ratio: ~1.19 FLOP/byte

For reference, H100 peak numbers give 989 TFLOPS ÷ 3.35 TB/s ≈ 295 FLOP/byte. That is arithmetic on peak numbers, not a property of any kernel. Achieved FLOPs and achieved bandwidth are both far below peak and move independently. Use profiler counts and utilization, not a ratio of peaks.

---

7. Prefill vs decode: a dependency and traffic summary

This is a dependency and traffic summary, not a schedule or a profiler trace. It lists what each layer must read and write; nothing is “preloaded” before the layers run, and the memory system interleaves these transfers with compute.

PREFILL (prompt length P)
+-------------------------------------------------------------+
| For each of the 32 layers, in order:                        |
|   a. Read this layer's QKV weights; compute QKV projections |
|   b. Attention over Q x K^T tiles: read this layer's K, V   |
|      blocks; flash-style kernels never materialize the      |
|      full P x P matrix                                      |
|   c. Softmax + weighted sum over the same tiles (compute)   |
|   d. Write K, V for all P tokens into this layer's cache    |
|      (P x 4 KiB per layer in FP16; P x 128 KiB in total)    |
|   e. Read this layer's MLP weights; compute the MLP         |
| Output: logits for the last prompt position                 |
+-------------------------------------------------------------+
Candidate limits to test: weight streaming vs linear compute
at small P; attention compute vs KV write at large P.

DECODE (per generated token, context length S)
+-------------------------------------------------------------+
| For each of the 32 layers, in order:                        |
|   a. Read this layer's QKV weights; compute QKV for the     |
|      single new token                                       |
|   b. Attention: read this layer's cached K, V for all S     |
|      prior tokens (1 query row x S keys)                    |
|   c. Append the new token's K, V to this layer's cache      |
|      (4 KiB per layer)                                      |
|   d. Read this layer's MLP weights; compute the MLP         |
| Output: logits for the new position                         |
+-------------------------------------------------------------+
Candidate limits to test: weight streaming at small S; KV read
traffic at large S.

Aggregated over layers, the per-pass weight traffic is the full 16.00 GB and the KV traffic is S × 128 KiB, which is why §6 sums them. That aggregate is a modelling convenience: the kernel touches one layer’s slice at a time, which is exactly what makes L2 residency and prefetching matter.

Which term limits wall-clock time is what §8–9 measure. The diagram is not a profiling result.

Key distinction: Prefill computes P tokens in one pass (quadratic attention in P). Decode computes 1 token per pass (linear attention in S, but repeated for every output token). The weight traffic is similar per pass; the KV traffic differs fundamentally.

---

8. Architecture decision: choosing KV precision for a latency target

Scenario: You are deploying Llama 3 8B on a single H100 SXM. The SLA requires TPOT p99 < 100 ms at 8K context, batch = 4. You are evaluating FP16 KV vs FP8 KV.

What changes with FP8 KV?

Quantity	FP16 KV	FP8 KV	Change
KV bytes/token	128 KiB	64 KiB	–50%
KV @ 8K (per seq)	1.00 GiB	0.50 GiB	–50%
KV @ 8K (batch 4)	4.00 GiB (4.29 GB)	2.00 GiB (2.15 GB)	–2.00 GiB (–2.15 GB)
Decode KV read/step	4.29 GB	2.15 GB	–2.15 GB
Weight traffic	16.00 GB	16.00 GB	unchanged
Nominal decode payload/step (batch of 4)	20.29 GB (18.90 GiB)	18.15 GB (16.90 GiB)	–2.15 GB (–10.6%)

The weight pass dominates both columns: each step reads the weights once and the KV cache for all four sequences. Halving KV precision removes 2.15 GB of a 20.29 GB nominal payload, a ~10.6% byte reduction. (The endpoints are rounded independently — 16.00 GB is nominal, and 4.29 → 2.15 GB is the unrounded KV term — so subtract the KV delta rather than the two displayed totals.) That is byte arithmetic, and it says nothing yet about latency.

The hypothesis to test

H: At batch 4, S = 8K, decode latency is limited by HBM traffic. Halving KV precision removes 2.15 GB (~10.6%) of the batch-wide nominal payload, so if KV reads are on the critical path, TPOT should move in the same direction. The size of that move is an empirical question — the removed bytes are one term, and compute, launch overhead, and caching are unaffected.

Measurement plan

What to measure	Tool / metric	Why
ITL	vllm:inter_token_latency_seconds histogram	Per-request TPOT uses vllm:request_time_per_output_token_seconds; the metrics differ when outputs are bundled or aggregation differs
DRAM bytes, same iteration	NCU dram__bytes_read.sum	Tests the nominal 20.29 GB / 18.15 GB payload directly
Duration and utilization	gpu__time_duration.sum, dram__throughput.avg.pct_of_peak_sustained_elapsed, tensor-pipe utilization	Shows which ceiling the kernel is actually near
Cache occupancy	vllm:kv_cache_usage_perc + cache_config_info	Confirms FP8 KV is actually active
Memory scopes	torch.cuda.max_memory_allocated() and max_memory_reserved() after warm-up, plus nvidia-smi	Separates tensor allocations, allocator pool, and driver-visible usage
Reference ceiling	memcpy of a known size, or a GEMM at the MLP shape, same GPU and clocks	Context only: a realistic ceiling for one resource, not proof of what limits decode

Procedure: Warm up 10 iterations, discard, then ≥20 timed iterations. Report median ± MAD. For one iteration, record DRAM bytes, FLOPs, and duration from the profiler. Report two lower bounds next to the measured time — bytes / peak_bw and FLOPs / peak_flops — and the achieved fraction of each peak. A memcpy or same-shape GEMM rate is a more realistic ceiling to report alongside, but it describes a different kernel, so treat it as context rather than proof.

Interpretation rule: Both bounds are lower bounds on time, so the measured time must be at or above each of them. The binding resource is the one with the highest achieved fraction of its ceiling; equivalently, the one whose lower bound is closest to the measured time. That bound must also be the larger of the two — if the measurement looks close to the smaller bound, the other bound is being violated, which means the byte or FLOP count is wrong, not that the resource is idle. If the measured time is far above both bounds, either the model is missing a term (launch overhead, a second pass over memory, allocator behaviour) or these ceilings are unrealistic for this kernel.

Closeness to a ceiling makes a resource a candidate constraint, and nothing more. A kernel can be near its DRAM ceiling and still be limited by a dependency chain, and two kernels at the same DRAM fraction can have different limiting resources. Confirm the candidate with a second, independent signal — a counter that varies with the suspected resource (batch size, context length, KV precision) should move latency in the same direction, and a control that changes only the other resource should not.

What this decision does not assume

• It does not assume decode is bandwidth-bound. The measurement will tell you. - It does not assume FP8 KV has no accuracy impact. That is a separate evaluation. - It does not assume the 10.6% payload reduction translates into a 10.6% latency reduction. Overlapped compute, kernel launch, and cache effects can weaken or strengthen the correlation.

---

9. How to test a bottleneck hypothesis

The measurement plan above is a template. For any workload, follow this pattern:

1. State the hypothesis explicitly. “Decode at batch=4, S=8K is HBM-traffic-bound: DRAM bytes per step are close to the nominal 20.29 GB payload, and DRAM utilization is near its ceiling while tensor-pipe utilization is low.”
2. Measure the workload itself. Take DRAM bytes, FLOPs, duration, and utilization counters from the same decode iteration. A memcpy or GEMM benchmark gives a realistic ceiling for context only: its access pattern, occupancy, and pipeline differ, so it cannot prove what limits your kernel.
3. Compute the lower bounds. time ≥ bytes / peak_bw AND time ≥ FLOPs / peak_flops, using the counters and the vendor peaks. Report the achieved fraction of each ceiling.
4. Compare measured to both bounds. The measured time must be at or above each bound. Whichever ceiling the kernel is closest to — highest achieved fraction — is the candidate constraint, and that bound must also be the larger of the two. If measured time appears close to the smaller bound, the other bound is violated and the counts are wrong. If measured exceeds both by a wide margin, the model is missing a term (launch overhead, a second pass over memory, allocator behaviour) or the ceilings do not apply to this kernel.
5. Report every number with its unit and base. Decimal GB vs binary GiB. Peak vs measured. Median ± spread.

Common trap: dividing peak numbers

Peak TFLOPS ÷ peak bandwidth = 295 FLOP/byte on H100. That is the machine balance at peaks, not what your kernel sees. A workload at 1.19 FLOP/byte sits far below it, so a roofline-style heuristic calls the workload bandwidth-bound. But the heuristic compares two ratios, not your measured time: achieved FLOPs and achieved bandwidth move independently, and a kernel may reach neither ceiling. The classification comes from counters and timing on the actual workload (§9, steps 3–4).

---

10. Exercises

Try it yourself

1. 01Calculate the KV cache size in GiB for Llama 3 8B at 32K context with FP8 KV, batch size 8. Show your work with units.

   Show a hint

   Formula: 2 × L × H_kv × D × S × B × b. L=32, H_kv=8, D=128, b=1 for FP8. Remember to convert bytes to GiB (÷ 2³⁰).

2. 02A colleague says "prefill is compute-bound and decode is bandwidth-bound" as a general rule. Using the numbers from this lesson, explain why that statement cannot be universally true.

   Show a hint

   Consider: small prompt (P=128) vs large prompt (P=8192); small context (S=512) vs large context (S=32768); batch=1 vs batch=32. Which terms dominate in each case?

3. 03You measure decode TPOT at batch=1, S=8K on an H100 and get 45 ms median. A memcpy benchmark on the same GPU reports 2.1 TB/s. Your nominal weight+KV payload is 17.07 GB. Does the memcpy number prove decode is bandwidth-bound? Show the calculation and name the measurement that would settle it.

   Show a hint

   Byte bound at the memcpy rate = 17.07 GB ÷ 2.1 TB/s ≈ 8.1 ms, far below 45 ms. A memcpy ceiling describes the memory system, not this kernel. Profile DRAM bytes, duration, and utilization for one decode iteration, compute the same two lower bounds from counters, and check which ceiling the kernel is near — then confirm with a counter that should move if that resource is the limit.

---

Key Takeaways

Key takeaways

• Weight payload: 14.90 GiB (FP16) or 7.45 GiB (FP8) for a nominal 8 × 10⁹-parameter model. KV payload: 128 KiB/token (FP16) or 64 KiB/token (FP8) per sequence.
• Prefill computes P tokens in one pass (O(P²) attention). Decode computes 1 token per pass (O(S) attention), repeated for every output token.
• Nominal HBM payload per prefill: weights + KV write. Nominal payload per decode step: weights + KV read (scales with S) + KV write. Actual DRAM traffic can be higher or lower than nominal because of tiling, caching, and block tables.
• A FLOP count divided by peak TFLOPS is not a latency prediction. A byte count divided by peak bandwidth is not a latency prediction. Use rates measured on the actual workload.
• Payload (tensor-byte arithmetic) ≠ torch allocated (tensors) ≠ torch reserved (allocator pool) ≠ driver-visible usage (CUDA context and other processes). Sample all three from the same process, device, and moment; none is a strict function of the others.
• Capacity comes from the device, not the product label: "80 GB" is 74.51 GiB, while an H100 80GB HBM3 reports 81,559 MiB (79.65 GiB) total, before any reservation.
• To test a bottleneck: state hypothesis → profile DRAM bytes, FLOPs, duration, and utilization on the same workload → compute both lower bounds from peaks → treat the ceiling the kernel is closest to as a candidate constraint and confirm it with an independent counter.

---

Sources and scope

Claim	Source	Status / limitation
H100 SXM: 80 GB HBM, 3.35 TB/s; FP16 Tensor Core ~989 TFLOPS (dense)	NVIDIA H100 product page, Specifications table	“80 GB” is a marketing label; 80 × 10⁹ B = 74.51 GiB. The dense figure is derived (half of the sparsity figure), not a separate row. Peak, not sustained.
H100 80GB HBM3 devices report 81,559 MiB of total memory (79.65 GiB, 85.52 GB)	NVIDIA Infra Controller, SKU Validation, sku generate / sku show examples	Reported device total, not free memory. Confirms that the “80 GB” label is neither the binary nor the decimal total; capacity must be read from the device.
Weights and KV must move HBM → cores once per forward pass; weights dominate at small batch/S, KV at large batch and 2048+ tokens	Pope et al., Efficiently Scaling Transformer Inference, arXiv:2211.05102, §2	Analytical model validated on TPU v4 / PaLM 540B, not H100. The paper’s MFU numbers are TPU measurements, so the traffic model gives a hypothesis, not a DRAM floor.
2N matmul FLOPs per token; attention FLOPs “can often be excluded”	Pope et al. §2	A rule of thumb (~1 matmul-add per parameter per token); omits attention, softmax, and non-matmul work.
Architecture: 32 layers, d_model 4,096, FFN 14,336, 32 heads, 8 KV heads	Llama 3 Herd of Models, arXiv:2407.21783, Table 3	Table 3 gives vocabulary as “128,000”; the tokenizer’s actual size is 128,256. The architecture multiplies out to ≈8.03 × 10⁹ parameters; this lesson deliberately uses a nominal 8.00 × 10⁹ instead (§3.1).
config.json for meta-llama/Meta-Llama-3-8B	HF model repo	Gated; unauthenticated fetch returns HTTP 401. It stores architecture hyperparameters, not a parameter count, so §3.1 uses a nominal 8B with stated assumptions.
Allocated vs reserved vs nvidia-smi memory; caching allocator keeps unused blocks visible in nvidia-smi	PyTorch CUDA semantics: Memory management	Documentation for the PyTorch caching allocator; other serving stacks expose different memory APIs.
FlashAttention tiles attention and reduces HBM reads/writes rather than materializing the full score matrix	Dao et al., FlashAttention, arXiv:2205.14135	Establishes that attention kernels need not materialize the P × P matrix; specific HBM byte counts are kernel- and shape-dependent.
--kv-cache-dtype independent of weight dtype; supports float16, bfloat16, fp8_e4m3, fp8_e5m2, integer types	vLLM CLI: serve (CacheConfig)	Tracks vLLM latest developer-preview docs; verify against your installed version.
KV byte formula, L, H_kv, D conventions	Internal: src/content/docs/llm-internals/kv-cache-explained.mdx	Internal context only, not primary evidence.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceServing Workloads and Performance Metrics12 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceFlashAttention and GPU Data Movement16 min