Serving Workloads and Performance Metrics

Track 2 Architecture Map

This lesson is Step 1 of Track 2’s end-to-end serving arc. See the Track 2 Visual Overview for how performance measurement connects to continuous batching, quantization, and cluster capacity planning.

The one-sentence version

A serving workload is defined by prompt length distribution, output length distribution, arrival process, and concurrency; its metrics (TTFT, TPOT/ITL, throughput, tail latency) must be measured with clear boundaries between client-observed timings and server-side aggregate histograms.

Everything else in this lesson follows from that split.

Why this lesson starts with a slow request

You click “Submit” on a chat interface and wait. Sometimes nothing happens for two seconds, then words stream rapidly. Other times the first word appears instantly, but subsequent words arrive slowly and choppily. Both feel “slow,” but they are different failures.

This happens because an LLM request runs in two completely distinct computational phases with different hardware bottlenecks:

• Prefill reads the entire prompt in parallel. At large prompt lengths and batch sizes, it becomes compute-bound — the GPU’s matrix engines are saturated. The output is the first token, produced when prefill finishes.
• Decode generates tokens sequentially, one step per token. At small batch sizes, it is memory-bandwidth bound — the GPU spends most of its time streaming weights from HBM to registers.

Time to First Token (TTFT) is the user-visible latency of the prefill phase (plus network, queueing, and transport). Inter-Token Latency (ITL) or Time Per Output Token (TPOT) is the user-visible latency of each decode step. They measure different hardware subsystems under different load conditions.

CPU path available

The conceptual trace below and the measurement arithmetic in this lesson can be followed and simulated entirely on CPU — no GPU cluster required. The live endpoint exercise (vLLM with Prometheus, Section 4) lets you observe real server-side scheduling, continuous batching, and client network metrics under concurrent load.

Opening trace: one request, five tokens (fictional, internally consistent)

[Illustrative data] The timestamps below are fictional and internally consistent; they do not represent any real hardware. They teach the arithmetic of TTFT, TPOT, and E2E, not a hardware result.

Scenario: Single request, prompt length 512 tokens, max_tokens=5 (5 output tokens generated).

Event	Timestamp (ms)	Delta (ms)	Notes
Request sent (client)	0.0	—	Start boundary
First token received (client)	120.0	TTFT = 120.0	network + queue + prefill + sampling + transport
Token 2 received	155.0	ITL₂ = 35.0
Token 3 received	192.0	ITL₃ = 37.0
Token 4 received	228.0	ITL₄ = 36.0
Token 5 received	265.0	ITL₅ = 37.0	Last token

Calculations (per-request, exact):

• n = 5 output tokens → n−1 = 4 inter-token gaps
• TTFT = 120.0 ms
• ITLs = [35.0, 37.0, 36.0, 37.0] ms
• TPOT = (35.0 + 37.0 + 36.0 + 37.0) / 4 = 36.25 ms
• E2E = TTFT + Σ ITLs = 120.0 + 145.0 = 265.0 ms (matches last-token timestamp)
• E2E (formula) = TTFT + (n−1) × TPOT = 120.0 + 4 × 36.25 = 265.0 ms ✓
• User TPS = (n−1) / (E2E − TTFT) = 4 / 145.0 ≈ 27.6 tokens/s = 1 / 0.03625 ✓

Practical measurement decision — what to measure next:

This single request gives you one per-request sample at concurrency 1: TTFT, TPOT, and E2E. It does not tell you throughput under load, tail latency, or whether queueing, prefill, or decode dominates. Next: re-run this same request at concurrency 1, 2, 4, 8, 16 and collect per-request TTFT/E2E plus the server histograms (Section 4).

If TTFT grows with concurrency while TPOT stays flat, treat queueing as a hypothesis, not a proven diagnosis. Flat TPOT means decode execution per token has not degraded, but rising TTFT can stem from server admission queueing, growing prefill batch execution time, or client-side connection backpressure. To distinguish the actual cause:

1. Server queue: inspect request_queue_time_seconds (first-queued to first-scheduled). If queue time accounts for the TTFT increase, the system is queue-limited.
2. Server prefill: inspect request_prefill_time_seconds (first-scheduled to first-new-token). If queue time remains low but prefill time climbs, prefill batch contention or scheduling chunks dominate TTFT.
3. Client boundary: compare client-observed TTFT against server TTFT (time_to_first_token_seconds), ensuring timestamps start at socket write (t_request_sent) to rule out client connection pool exhaustion or event-loop starvation.

---

1. Describe the workload before measuring

You cannot interpret a latency number without knowing the workload that produced it. A serving workload has four dimensions:

Dimension	What to capture	Typical representation
Prompt length	Distribution of input tokens per request	Histogram, quantiles (p50, p95, max)
Output length	Distribution of generated tokens per request	Histogram, quantiles of observed length; max_tokens caps it but does not set it
Arrival process	Request inter-arrival times	Poisson (λ), bursty (Pareto/ON-OFF), trace-driven
Concurrency	Number of in-flight requests	Fixed (N parallel), ramp, or closed-loop (think time)

If you benchmark with fixed 512-token prompts and 128-token outputs at concurrency 16, you have measured one workload. Changing any dimension — longer prompts, burstier arrivals, higher concurrency — can change the dominant bottleneck and the metric values.

Databricks benchmarks assume uniform synthetic workloads

The Databricks benchmarking guidance uses fixed prompt/output lengths and parallel requests. It does not prescribe how to model burstiness, diurnal cycles, or correlations between prompt length and user type. If your production traffic does not look like the benchmark, the numbers will not transfer.

---

2. Metric definitions and measurement boundaries

2.1 Time to First Token (TTFT)

Definition: Elapsed time from request submission to receipt of the first streamed chunk that contains output text. It includes client→server network, server queueing, tokenization, prefill compute, sampling/detokenization, and transport of that chunk.

• Client boundary (streaming required): Start = client sends HTTP request. End = client receives the first chunk carrying output text. Non-streaming responses cannot reveal client-observed TTFT because the output is not delivered until the entire response completes.
• Server boundary: vLLM 0.33.0 reports request_prefill_time_seconds, measured from first-scheduled to first-new-token, and request_queue_time_seconds, measured from first-queued to first-scheduled. Server prefill excludes queue time; client TTFT includes it. Do not equate the two.

Client TTFT ≠ server prefill time

Client TTFT = network + queue + prefill + sampling/detokenization + transport. Server prefill_time is one engine interval inside that chain. If you report server prefill as TTFT, you are omitting queueing and network — often the largest component under load.

Prefill already produces the first token — there is no separate first decode pass

In ordinary autoregressive serving, the prefill pass over the prompt yields logits for the last prompt position, and the sampler turns those logits into token 1. That is exactly the boundary Sarathi-Serve draws: prefill “processes the entire input prompt and produces the first output token”, after which decode “generates the rest of output tokens, one-at-a-time” (arXiv:2403.02310, abstract). So do not draw an extra decode forward pass between prefill and the first token: it double-counts work prefill already did. If a particular pipeline really runs one — for example because it discards prefill logits — name that implementation and add the pass explicitly.

2.2 Inter-Token Latency (ITL) and Time Per Output Token (TPOT)

Definitions:

• ITL (per gap): Time between consecutive output tokens. For a request r with n output tokens, there are n−1 inter-token gaps: ITLₖ = timestamp(tokenₖ) − timestamp(tokenₖ₋₁) for k = 2..n.
• TPOT (per request): Average ITL for that request. TPOTᵣ = (E2E_latencyᵣ − TTFTᵣ) / (nᵣ − 1), where nᵣ is the number of output tokens generated.

Averaging across requests differs:

Average type	Formula	Weights each…
Avg TPOT (request-weighted)	Σ TPOTᵣ / N	Each request equally
Avg ITL (gap-weighted)	Σ ITL / Σ (nᵣ − 1)	Each inter-token gap equally

Request-weighted TPOT vs gap-weighted ITL

If Request A generates 100 tokens (TPOT 10 ms) and Request B generates 2 tokens (TPOT 50 ms): - Avg TPOT = (10 + 50) / 2 = 30 ms - Avg ITL = (99×10 + 1×50) / (99 + 1) = 10.4 ms They differ because Request A contributes 99× more inter-token gaps. Label which average you are reporting.

Measurement boundaries:

• Client (streaming): Start = first chunk carrying output text. End = last chunk carrying output text. Trailing usage and [DONE] frames are not tokens; keep them out of both the timestamps and the count. Two different estimates are available:
  • Endpoint average — robust, works for any chunking. TPOTᵣ = (t_last_text − t_first_text) / (nᵣ − 1), where nᵣ is the response’s reported completion-token count (usage.completion_tokens on the OpenAI-compatible API), not the number of chunks. Counting tokens does not require timestamping tokens: the endpoints plus the count are enough for an average.
  • Per-gap deltas — conditional. Only valid when each streamed chunk carries exactly one new token. Ordinary non-speculative decoding often does emit one token per chunk, so the deltas are frequently meaningful, but that is a property of the deployment, not of streaming. Check it before relying on it: for a sample of requests, assert that the number of text-bearing chunks equals the reported completion-token count.
• What is genuinely not recoverable: the arrival time of an individual token inside a chunk. If chunks bundle several tokens, you can still measure endpoints and counts, but not each gap. Report the endpoint average and state that the gap distribution is unknown for that deployment.
• Zero- and one-token outputs: nᵣ = 1 has no gaps, so TPOT is undefined — exclude it from TPOT statistics and report how many requests you excluded. nᵣ = 0 (immediate stop, refusal, error) is not a latency sample at all; count it separately rather than letting it become a 0 ms gap.
• Speculative decoding: one decode step can emit several accepted tokens. Client gaps and server histograms then both measure per step, not per token. Do not label those samples “per-token latency”.
• Server (aggregate distributions, vLLM 0.33.0):
  • request_time_per_output_token_seconds — per-request TPOT, computed as decode_time / (num_generation_tokens − 1) when there are at least two generated tokens. In vLLM 0.33.0, the stats code assigns 0 for a zero- or one-token output and the Prometheus logger still observes that 0. The histogram can therefore include samples for which TPOT is mathematically undefined. Use client per-request records to exclude those outputs; the aggregate server histogram cannot remove them afterward
  • inter_token_latency_seconds — the 0.33.0 instrumentation appends one observation per non-prefill engine step per running request, so a step that emits several tokens contributes one sample covering all of them. With one token per step it is a per-token gap; otherwise it is a per-step gap. The metric name alone does not settle which
  • request_decode_time_seconds — per-request decode duration, first token to last token
  • time_to_first_token_seconds — server-side TTFT, measured from request arrival

Do not divide unrelated histogram aggregates. A decode duration and output count from the same request can yield its average TPOT; dividing aggregate decode time by aggregate token count across different requests does not recover per-request TPOT. And a histogram’s name is not a contract: read the instrumentation of the release you pinned before you quote its samples as per-token measurements.

2.3 End-to-End (E2E) Latency

Exact per-request formula: E2Eᵣ = TTFTᵣ + Σₖ₌₂ⁿ ITLₖ = TTFTᵣ + (nᵣ − 1) × TPOTᵣ.

Databricks approximation: “Latency = TTFT + (TPOT) × (the number of tokens to be generated)”. This multiplies TPOT by n rather than the n−1 measured gaps. It is an approximation; the exact per-request formula uses n−1 gaps.

2.4 Throughput

Variant	Formula	What it means
System TPS	Total output tokens / (T_last − T_first)	Total token throughput across all concurrent requests
User TPS	(n − 1) / (E2E − TTFT) = 1 / TPOT	Tokens/second experienced by a single user
RPS	Completed requests / Total time	Request completion rate

State three things whenever you report a throughput: whether the numerator counts output tokens or prompt tokens (never add the two into one “tokens/s” figure), whether the denominator is a declared wall-clock window or (T_last − T_first) over the request set, and whether that window includes warm-up or queue drain. A prompt-token rate and an output-token rate are different quantities; adding them produces a number that no published baseline can be compared with.

2.5 Tail Latency and Goodput

• Percentiles: p50 (median), p95, p99 of TTFT, TPOT, E2E latency. p99 represents near worst-case; p95 common for SLAs.
• Goodput (percentage-goodput convention): % of requests meeting all per-request SLO thresholds (e.g., TTFT < 1 s, TPOT < 150 ms, E2E < 5 s for a 128-token output). Goodput = (requests meeting SLOs / total requests) × 100%. This is the convention the Anyscale guide uses. It is not the same statistic as “the p99 of the distribution is under X”: one is a per-request pass rate, the other is an aggregate percentile, and a system can pass either while failing the other. Name which one you are reporting, and do not compare your percentage-goodput with another team’s requests-per-second goodput.
• Aggregation: compute percentiles over the pooled per-request sample for the window, or report a percentile per run. Never subtract, average, or otherwise combine percentiles from different series, different requests, or different phases of a sweep.

---

3. Worked example: averaging illustration and Databricks check

This section reuses the fictional opening trace (one request, 512-token prompt, 5 output tokens: TTFT = 120.0 ms, TPOT = 36.25 ms, n = 5) and adds two requests of different output lengths to show how averaging conventions diverge.

[Illustrative data] All numbers below are fictional and internally consistent; they do not represent any real hardware.

Averaging illustration (two requests):

• Request A: 100 tokens, TPOT = 10 ms → 99 gaps, total decode = 990 ms
• Request B: 2 tokens, TPOT = 50 ms → 1 gap, total decode = 50 ms
• Avg TPOT (request-weighted) = (10 + 50) / 2 = 30 ms
• Avg ITL (gap-weighted) = (990 + 50) / (99 + 1) = 10.4 ms They differ because Request A contributes 99× more inter-token gaps to the gap-weighted mean.

Databricks approximation check: Latency ≈ TTFT + TPOT × n = 120 + 36.25 × 5 = 301.25 ms (overestimates by 36.25 ms vs exact 265 ms because it uses n instead of n−1).

---

4. Reproducible measurement protocol

This protocol is for a reader-provided vLLM endpoint with Prometheus metrics enabled. It produces per-request client data (TTFT, E2E) and separate aggregate server histograms (TPOT, ITL). The two datasets cannot be joined.

4.1 Environment log (record before every sweep)

• GPU type, driver, CUDA version
• vLLM version (vllm.__version__), PyTorch version, Python version
• Model name, precision (FP16/BF16/INT4/FP8), max_num_seqs, max_num_batched_tokens
• KV cache configuration (FP8/INT4, block size, prefix caching on/off)

4.2 Warm-up (measurement-based, illustrative heuristic)

Send requests at target concurrency until per-request TTFT and TPOT stabilise (e.g., running median of last 10 requests changes by < 2% over 3 consecutive windows — this is an illustrative heuristic, not a validated standard). Discard all warm-up measurements. Do not use a fixed count; stabilisation depends on framework, model size, kernel autotuning, and GPU clock state.

4.3 Steady-state measurement (client side)

At each concurrency level 64 (or until saturation/OOM):

1. Use httpx.AsyncClient with asyncio.Semaphore for exact concurrency control and stream=True.
2. For each request: timestamp at socket write (t_request_sent), the first text-bearing streamed chunk (t_first_text), and the last text-bearing chunk (t_last_text). Ignore trailing usage/[DONE] frames for both timestamps and counts.
3. Record per-request: request_id, prompt_tokens, completion_tokens (from usage, not from a chunk count), TTFT = t_first_text − t_request_sent, E2E = t_last_text − t_request_sent.
4. Write CSV with columns: request_id, prompt_tokens, completion_tokens, TTFT_ms, E2E_ms.
5. Compute per-request TPOT = (E2E − TTFT) / (completion_tokens − 1) only when completion_tokens ≥ 2; report how many requests fell below that and why. Separately, if you want per-gap deltas, verify that text-bearing chunks equal completion_tokens for every request first — otherwise report chunk-level arrival deltas and say the gap distribution is unknown.

The arithmetic over those rows is small enough to state exactly. This snippet is pure local computation over data you already collected; the request loop that produces it depends on your client and is yours to write:

# rows: one dict per request, times in seconds
# {"request_id": ..., "prompt_tokens": ..., "completion_tokens": ...,
#  "t_first_text": ..., "t_last_text": ...}

tpot_ms = [
    1000 * (r["t_last_text"] - r["t_first_text"]) / (r["completion_tokens"] - 1)
    for r in rows
    if r["completion_tokens"] >= 2
]
excluded = sum(1 for r in rows if r["completion_tokens"] < 2)  # report, never silently drop


def output_tokens_per_second(gen_t0, gen_t1, t0, t1):
    """Deltas of the same counter series, same labels, same process."""
    return (gen_t1 - gen_t0) / (t1 - t0)  # tokens/s over the declared window

4.4 Aggregate server metrics (separate step)

After the sweep, query Prometheus for the benchmark window. These names are for vLLM 0.33.0; check them against the release you actually run.

• request_time_per_output_token_seconds histogram → server-side TPOT distribution (p50, p95, p99) only if every completed request in that window generated at least two tokens; otherwise zero-valued undefined samples contaminate its distribution, so use filtered client per-request TPOT instead
• inter_token_latency_seconds histogram → server-side inter-step (or inter-token, per §2.2) distribution
• time_to_first_token_seconds histogram → server-side TTFT distribution
• request_prefill_time_seconds + request_queue_time_seconds histograms → server prefill and queue distributions
• generation_tokens counter → output-token rate; prompt_tokens counter → prompt-token rate, reported separately

Output throughput must come from a counter, not a histogram. In vLLM 0.33.0, vllm:generation_tokens is a Counter documented as “Number of generation tokens processed”. Take two scrapes of the same series with the same labels and divide the delta by the elapsed time:

output_tokens_per_s = (generation_tokens(t1) − generation_tokens(t0)) / (t1 − t0)

The window must sit inside your measured steady state, not across warm-up or shutdown, and both scrapes must come from the same process and label set. Alternatively, count completed client outputs: sum(usage.completion_tokens) over the same window, divided by the same window length. The two should agree closely; when they do not, one of the boundaries is wrong.

vllm:iteration_tokens_total is not a token counter. In 0.33.0 it is a histogram of tokens per engine step: its _bucket series count engine steps, and its _sum is a sum of per-step token totals, not a documented output-token series. Summing cumulative scrape values, or reading bucket counts as tokens, produces a number that is not output throughput. If you want the per-step distribution — how batch composition changes across the window — that histogram is the right instrument, and that is a different question from throughput.

Check the instrument against a known-output workload. Send 20 requests with max_tokens=32 and no other traffic. Compare the generation_tokens delta across the window against the observed sum of client completion tokens (sum(usage.completion_tokens)). The two should match. If they do not, explain the difference before you trust any larger sweep: a stop sequence fired, a request errored, a label selector picked up a second engine, or you took a rate instead of a delta.

Do not attempt to join server histogram samples to client CSV rows. The histograms do not expose request IDs or timestamps that would allow pairing.

4.5 Report

For each concurrency level, report:

• Client: median ± MAD of TTFT, E2E, p50/p95/p99 of each
• Server: p50/p95/p99 of ITL, TTFT, prefill, and queue histograms; include the TPOT histogram only when the window has no zero- or one-token completions, and state that condition. Otherwise report filtered client per-request TPOT percentiles
• System output TPS (from the generation_tokens counter delta) and prompt TPS, reported separately; User TPS computed per-request as (completion_tokens − 1) / (E2E − TTFT) for requests with completion_tokens ≥ 2, never as 1 / median client TPOT
• Goodput: percentage-goodput on per-request thresholds (TTFT < 1 s, TPOT < 150 ms, E2E < 5 s for a 128-token output), labelled as percentage-goodput and reported separately from any aggregate p99 SLO
• Percentile policy: computed over the pooled per-request sample for the window; no subtraction or averaging of percentiles across series or runs
• Repeat full sweep ≥ 3 times (separate process invocations); report median ± MAD across runs

---

5. Common measurement errors

Error	Symptom	Fix
No warm-up / fixed-count warm-up	Early requests slower; TTFT/TPOT inflated	Warm-up until running median stabilises (illustrative heuristic: <2% change over 3 windows). Discard warm-up. Do not use a fixed count.
Client-side queuing conflated	Measured TTFT includes local async queue delay	Use asyncio.Semaphore for exact concurrency; timestamp at socket write, not task creation.
Token-weighted vs request-weighted confusion	“Average TPOT” reported but actually ITL (or vice versa)	Compute both explicitly; label which is which.
Assuming one chunk = one token	Gap deltas compressed or stretched by chunking	Take counts from usage.completion_tokens, use endpoint averages, and assert one-token chunks before reporting per-gap deltas.
Histogram reported as counter	Throughput built from iteration_tokens_total buckets	Use the generation_tokens counter and a window delta; state labels, units, and window.
Output length treated as fixed	TPOT varies; throughput unfair across runs	max_tokens is an upper bound. Record observed length, stratify by it, and report the early-stop rate.
Ignoring queue time	TTFT reported as “prefill time” but includes queue	Separate server queue_time from prefill_time. Client TTFT includes both.
Subtracting marginal medians	“Residual network = client median − server medians”	Medians come from different distributions. Difference within matched per-request data, or bound RTT with a control request.
Single-run reporting	Variance across runs unreported	Run full sweep ≥ 3× (separate processes). Report median ± MAD.
Equating server prefill/decode with client timings	Server prefill_time ≠ client TTFT	Client TTFT = network + queue + prefill + sampling/detokenization + transport. Server prefill_time is one engine interval.

---

6. Visual summaries

Latency decomposition timeline

Step	Description	Client-observed	Server-only
1	Client sends HTTP request	✓ (timestamp: t_request_sent)
2	Network transit → server		✓ (one-way RTT component)
3	Server queue wait (queue_time = first-queued → first-scheduled)		✓
4	Prefill over P prompt tokens → logits for the last prompt position		✓ (prefill_time)
5	Sampling + detokenization → token 1		✓
6	First text-bearing chunk arrives at client	TTFT = step 6 − step 1
7	Decode step → token 2 (chunk arrives)		✓
8	Decode step → token 3 (chunk arrives)		✓
…	… tokens decode → chunks arrive		✓
N	Last text-bearing chunk arrives at client	E2E = step N − step 1
—	Server histograms (no request-ID join)
	time_to_first_token_seconds distribution	—	✓
	request_prefill_time_seconds distribution	—	✓
	request_queue_time_seconds distribution	—	✓
	request_decode_time_seconds distribution	—	✓
	request_time_per_output_token_seconds (TPOT)	—	✓
	inter_token_latency_seconds (inter-step; see §2.2)	—	✓

Key point: Client-observed TTFT and E2E include network, queue, and client processing. Server histograms isolate prefill, queue, decode, and inter-token gaps. The two datasets cannot be joined across the client/server boundary.

There is exactly one forward pass before the first token: the prefill pass at step 4. Steps 5 and 6 are sampling, detokenization, and transport — not another model execution. Adding a “first decode” row here would double-count the transformer work and make your TTFT decomposition disagree with the server’s prefill_time.

Exact vs Databricks latency formula

	Exact per-request	Databricks approximation
Formula	E2E = TTFT + (n−1) × TPOT	Latency = TTFT + TPOT × n
Gap count	n−1 inter-token gaps	n tokens × single TPOT
Assumption	Per-request TPOT from n−1 gaps	Constant TPOT across all tokens
Error	None	Overestimates by exactly one TPOT interval

The Databricks formula is a useful approximation when n is large and TPOT is nearly constant: the relative error is 1/n, so it is within 1% for n > 100. It is never exactly equal for positive TPOT, and the absolute error is one TPOT interval, which matters for short outputs.

Request-weighted vs gap-weighted averaging

	Request A (100 tokens)	Request B (2 tokens)	Avg TPOT (request-wt)	Avg ITL (gap-wt)
Output tokens	100	2	—	—
TPOT	10 ms	50 ms	(10+50)/2 = 30 ms	—
Inter-token gaps	99	1	—	100 total
Total decode time	990 ms	50 ms	—	1040 ms
Weighted mean	—	—	30 ms	1040/100 = 10.4 ms

Request A contributes 99× more gaps to the gap-weighted mean. Label which average you report.

---

Key takeaways

• A workload is defined by prompt/output length distributions, arrival process, and concurrency — not just "the model". `max_tokens` caps output length; it does not fix it.
• Client TTFT requires streaming; E2E can be measured as total response time. Server `prefill_time` and `queue_time` cover engine intervals only, so they never add up to client TTFT.
• Prefill produces the first token. There is no separate first decode pass to insert between prefill and TTFT.
• TPOT is request-weighted; ITL is gap-weighted. They diverge when output lengths vary. Label which you report, and exclude or declare one-token requests.
• Client timings and server histograms cannot be joined. Report them separately and state this limitation. A histogram name is not a contract: read the release-pinned instrumentation before calling samples per-token.
• Output throughput comes from a counter delta over a declared window (`generation_tokens` in vLLM 0.33.0), never from histogram buckets and never by adding prompt and output tokens together.
• Warm-up until stabilisation (illustrative heuristic), not a fixed count. Repeat runs ≥3×; report median ± MAD. Percentiles pool per-request samples; they are not subtracted or averaged.

---

Try it yourself

1. 01You run a benchmark at concurrency 8 and get median client TTFT 210 ms. The server prefill histogram median is 180 ms and the queue histogram median is 15 ms, so you report "15 ms of residual network and client overhead". Is that residual a real measurement? Explain, and say what you would record instead.

   Show a hint

   No. The three medians are marginal statistics from three different distributions: the request holding the median TTFT is not necessarily the request holding the median prefill, and the medians need not come from the same phase of the sweep. Marginal medians cannot be subtracted to isolate a residual. Instead, capture per-request server timings for the same requests (or a matched sample) and difference inside each request; or bound the network term with a control request against a trivial endpoint.

2. 02A benchmark reports "average TPOT 25 ms" without specifying request-weighted or gap-weighted. The workload has 80% short requests (4 output tokens) and 20% long requests (200 output tokens). Which average is likely being reported, and does the distinction change the number much here?

   Show a hint

   Count gaps. 80 × 3 = 240 gaps from the short requests and 20 × 199 = 3,980 from the long ones, so 94% of gaps come from 20% of requests. A gap-weighted average is therefore dominated by the long requests and will sit near their TPOT; a request-weighted average gives each short request the same influence as a long one. Say which convention you used, and state how many one-token responses you excluded.

3. 03You are given a CSV of per-request client timings (TTFT, E2E, completion_tokens) and a Prometheus dump of the server TPOT histogram. Write the expression that computes per-request TPOT from the CSV, then explain why you cannot validate those values against the histogram samples.

   Show a hint

   TPOT = (E2E − TTFT) / (completion_tokens − 1), for completion_tokens ≥ 2; report the count you excluded. The histogram has no request IDs, timestamps, or per-sample token counts, so it is an aggregate distribution, not a set of joinable observations.

4. 04A dashboard reports "system throughput = 1,240 tokens/s" from a query that adds `sum(vllm:iteration_tokens_total_sum)` to `rate(vllm:generation_tokens[5m])`. Name at least three things wrong with that number, then write the corrected query.

   Show a hint

   (1) `iteration_tokens_total` is a histogram of tokens per engine step, not a token counter, so its `_sum` is not a documented output-token series; (2) a cumulative sum and a per-second rate have incompatible units; (3) prompt and output tokens are being added into one figure. Corrected: `rate(vllm:generation_tokens[5m])` for output tokens/s and `rate(vllm:prompt_tokens[5m])` separately, over the same declared window and label set, and check the counter against a known-output workload first.

---

Interview questions

Why must you use streaming to measure TTFT, and what does a non-streaming response measure instead?

A non-streaming HTTP response delivers the entire body at once, so the first token is not available until the last token is ready. The measured time is E2E latency, not TTFT. Streaming delivers chunks as they are produced, letting you timestamp the first chunk arrival.

A colleague says "our p99 TPOT is 50 ms" based on dividing the p99 decode duration by the average output tokens. Is this correct? Why or why not?

No. Dividing aggregate percentiles does not yield a per-request percentile. The p99 decode duration and p99 output tokens come from different requests, and TTFT + p99 × n does not equal TTFT + (n−1) × TPOT for nonzero TPOT — it can only approximate it. To get per-request TPOT percentiles, you need per-request (decode_time, output_tokens) pairs, or the server must expose a per-request TPOT histogram (vLLM `request_time_per_output_token_seconds`).

Under what conditions does the Databricks latency approximation (TTFT + TPOT × n) equal the exact formula (TTFT + (n−1) × TPOT)?

Never, unless TPOT is zero. For positive TPOT the approximation is larger by exactly one interval, so the relative error is 1/n: about 1% at n = 100 and 20% at n = 5. It is useful when n is large and TPOT is nearly constant; it is not an identity, and it gets worse exactly where short outputs are common.

Where in the request lifecycle is the first output token produced, and how does that change the way you decompose TTFT?

In ordinary autoregressive serving the prefill pass over the prompt produces logits for the last prompt position, and sampling turns those logits into token 1. There is no separate first decode forward pass. Client TTFT therefore decomposes into network + queue + prefill (including first-token sampling) + transport of the first chunk. Inserting a decode pass in the accounting double-counts transformer work that prefill already did, and makes your decomposition disagree with the server prefill interval.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceWhere Inference Time and Memory Go13 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceFlashAttention and GPU Data Movement16 min

---

Sources and scope

• Databricks LLM endpoint benchmarking (2026-09-11) — TTFT, TPOT, latency formula (approx), throughput, throughput–latency curve.
• Anyscale LLM latency and throughput metrics (accessed 2026-11-04) — TTFT, E2E, ITL vs TPOT (gap-weighted vs request-weighted), TPS variants, RPS, goodput (percentage-goodput convention), percentiles.
• Sarathi-Serve, Taming Throughput-Latency Tradeoff in LLM Inference, arXiv:2403.02310, abstract (v3, 2024-06-17) — prefill “processes the entire input prompt and produces the first output token”; decode generates the rest. Source of the phase boundary in §2.1.
• vLLM 0.33.0 — Production Metrics — metric names, types, and descriptions: generation_tokens (Counter, “Number of generation tokens processed”), prompt_tokens (Counter), iteration_tokens_total (Histogram of tokens per engine step), inter_token_latency_seconds, request_time_per_output_token_seconds, request_prefill_time_seconds, request_queue_time_seconds, request_decode_time_seconds, time_to_first_token_seconds.
• vLLM 0.33.0 — vllm/v1/metrics/stats.py and loggers.py — instrumentation read directly: prefill interval = first-scheduled → first-new-token; queue interval = first-queued → first-scheduled; per-request TPOT = decode_time / (num_generation_tokens − 1) when defined, else zero, with every finished request observed in the histogram; one inter_token_latency_seconds observation per non-prefill engine step per request.
• NVIDIA Triton Inference Server — Batchers (v2.x/24.xx) — Dynamic batcher config, sequence batcher, iterative sequences (provisional). Triton batching configuration is covered in the later lesson “Batching and throughput”.
• NVIDIA Triton Inference Server — Model Configuration (v2.x/24.xx) — max_batch_size, auto-complete config, default max batch size = 4 for auto-configured backends (Onnxruntime); TensorRT does not use this default.

The fictional timestamp trace and averaging illustration in this lesson are teaching tools, not measured hardware results. The measurement protocol instructs you to run your own measurements; no benchmark numbers are invented here.

Runtime status: the vLLM metric names and types above were read from the pinned 0.33.0 documentation and source at the time of writing. No vLLM server was started and no Prometheus scrape was executed for this lesson, so the endpoint behaviour, label values, and counter deltas are unverified at runtime. Before relying on the protocol, run the known-output check in §4.4 on your own deployment and record the result.