Capacity Planning: From Traffic Profile to GPU Count

The one-sentence version

A defensible capacity plan chains analytical memory/bandwidth lower bounds → an empirical open-loop load sweep → queueing-theory headroom → discrete node bin-packing → a TCO model with sensitivity analysis; skipping any step produces a number you cannot defend.

---

Opening example: input → calculation → visible capacity decision

Before the framework, here is the smallest complete decision loop. A product team needs to serve a retail chatbot. They hand you this contract:

Parameter	Value	Type
Peak arrival rate (λ_peak)	50 req/s	Traffic assumption
Peak-to-average ratio (PAR)	3.0	Traffic assumption
Avg prompt length	500 tokens	Traffic assumption
Avg output length	500 tokens	Traffic assumption
TTFT p95 SLO	≤ 1.0 s	Business policy
E2E p99 SLO	≤ 2.0 s	Business policy
Target hardware	1× A100-SXM4-80GB (79.65 GiB reported)	Hardware bound
Engine memory utilisation	0.90	Software policy
Model: Llama-3-8B, BF16	14.90 GiB weights, 128 KiB/token KV	Model bound
Workspace reserve	1.75 GiB	Software policy
Decode throughput (ideal)	1,000 output tok/s per GPU	Hardware bound (lower bound)
Redundancy	N+1	Operational policy
Cost per GPU	$5.00/hr	Economics policy

Step 1 — Memory-bounded concurrency (hard ceiling)

Usable VRAM = 79.65 GiB × 0.90 = 71.69 GiB
KV pool     = 71.69 − 14.90 − 1.75 = 55.04 GiB
Total KV tokens = 55.04 GiB / 128 KiB = 450,900 tokens
Avg request context = 500 + 500 = 1,000 tokens
C_max (memory) = 450,900 / 1,000 ≈ 450 concurrent requests

The GPU can hold ~450 average requests in KV cache. But memory is not the bottleneck here.

Step 2 — Compute-bounded concurrency (SLO filter)

The GPU produces 1,000 output tokens/s total. If we batch B requests, each gets 1,000/B tok/s.

To meet E2E ≤ 2.0 s for 500 output tokens, each request needs ≥ 250 tok/s.

Max SLA-bound batch = 1,000 / 250 = 4 concurrent requests

Despite memory for 450, the SLO forces B ≤ 4.

Step 3 — Cluster sizing via Little’s Law

Peak λ = 50 req/s. Target residence time W = 2.0 s (E2E SLO).

Required concurrency L = λ × W = 50 × 2.0 = 100 in-flight requests.

Active GPUs = ceil(100 / 4) = 25. With N+1 redundancy = 26 GPUs.

Step 4 — Bin-packing and cost

AWS p4de.24xlarge = 8 GPUs/node @ $40.96/hr.

Nodes = ceil(26 / 8) = 4 nodes (32 GPUs deployed).

Hourly cost = 4 × $40.96 = $163.84/hr. Daily = $3,932.16. Annual ≈ $1.44M.

Average λ = 50/3 = 16.67 req/s. Monthly output tokens = 16.67 × 86,400 × 30.5 × 500 ≈ 2.19B. Cost per 1M tokens = $163.84 × 732 / 2,190 ≈ $54.8 / 1M tokens.

This is the illustrative decision record. The rest of the lesson teaches each step rigorously, labels every assumption, and gives you the load-sweep protocol to replace the “1,000 tok/s” lower bound with a measured λ_instance*.

---

1. Workload contract: the input specification you cannot skip

You cannot size infrastructure from “60 QPS”. A production capacity plan requires an explicit contract across three domains:

1.1 Traffic dynamics

Field	Example	Notes
Peak arrival rate (λ_peak)	60 req/s	Open-loop Poisson process
Peak-to-average ratio (PAR)	3.0	Diurnal cycle: λ_avg = λ_peak / PAR
Monthly request volume	52.7M req/mo	Derived: λ_avg × 86,400 × 30.5
Arrival distribution	Poisson	Stochastic inter-arrival; bursty by definition

1.2 Multi-class payload profile

Real traffic is not uniform. Define classes with weights, prompt lengths, and output lengths:

Class	Weight	Prompt (P)	Output (O)	Context (S = P+O)
Short chat	70%	32	64	96
Code query	20%	128	256	384
Summarization	10%	512	256	768

Weighted averages (derive, don’t guess):

• P̄ = Σ w_i P_i = 0.70×32 + 0.20×128 + 0.10×512 = 99.2 tokens
• Ō = Σ w_i O_i = 0.70×64 + 0.20×256 + 0.10×256 = 121.6 tokens
• S̄ = P̄ + Ō = 220.8 tokens
• S_max = max(P_i + O_i) = 768 tokens (worst-case allocation envelope)

1.3 Service Level Objectives (client-facing, percentile-based)

SLO	Target	Measurement boundary
TTFT p95	≤ 1.0 s	Client: request send → first text chunk
TPOT p95 / ITL p95	≤ 25 ms	Client: inter-token chunk deltas (or server TPOT histogram)
E2E p99	≤ 5.0 s	Client: request send → last text chunk
Error/shedding budget	< 0.1%	HTTP 429/503 or dropped requests

GPU utilisation is not an SLO

The capstone spec error: “GPU utilisation ≥ 70%” as an SLO. Utilisation is an internal operating target (ρ_target ≈ 0.70 for queue headroom), never a client-facing contract. A service at 10% utilisation with perfect latency is healthy; a service at 90% utilisation with SLO breaches is broken.

---

2. Single-instance physical boundaries: analytical lower bounds

Before any benchmark, prove the hardware can satisfy the SLOs in ideal conditions. Three gates:

2.1 Gate 1: Memory capacity floor

For a candidate GPU (here: A100-SXM4-80GB, Llama-3-8B BF16, vLLM):

Component	Calculation	Value
Marketed VRAM	80 GB decimal	80×10⁹ bytes
Driver-reported total	81,559 MiB	79.65 GiB (binary)
Engine reservation (gpu_memory_utilization=0.90)	79.65 × 0.90	71.69 GiB
Model weights (8.03B params × 2 B)	16.06 GB	14.90 GiB
Prefill workspace (max_num_batched_tokens=2048)	—	1.75 GiB
KV cache pool (M_KV)	71.69 − 14.90 − 1.75	55.04 GiB

Per-token KV footprint depends on attention architecture:

Architecture	Formula (BF16)	Bytes/token
MHA (Llama-2-7B: 32 KV heads)	2 × 32 × 32 × 128 × 2	524,288 B (512 KiB)
GQA (Llama-3-8B: 8 KV heads)	2 × 32 × 8 × 128 × 2	131,072 B (128 KiB)
FP8 KV cache (GQA)	2 × 32 × 8 × 128 × 1	65,536 B (64 KiB)

Using the wrong architecture in this formula causes immediate OOM under load.

Total KV token capacity:

T_KV = 55.04 GiB / 128 KiB = 450,900 tokens

Concurrency ceilings:

Scenario	Calculation	Ceiling
Average context (S̄=220.8, 3.6% PagedAttention tail waste)	450,900 / (220.8 × 1.036)	≈ 1,971 requests
Worst-case envelope (S_max=768)	450,900 / 768	≈ 587 requests

Key takeaway: On 80GB serving Llama-3-8B with short/medium context, KV capacity is not the bottleneck. The system is compute- and bandwidth-limited by the SLOs.

2.2 Gate 2: Memory bandwidth lower bound (decode step)

In decode, each step reads all weights + KV cache for all active sequences.

Data_Moved(B) = W_bytes + B × S̄ × Bytes_per_token

At B=32, S̄=220.8, Bytes_per_token=128 KiB:

KV traffic = 32 × 220.8 × 131,072 = 926 MB
Total      = 16.06 GB + 0.93 GB = 16.99 GB

A100-SXM4-80GB HBM2e peak = 2,039 GB/s. Assuming 76% achieved (1,550 GB/s):

t_mem = 16.99 GB / 1,550 GB/s = 10.96 ms

Feasibility check: Target TPOT ≤ 25 ms. Ideal lower bound 10.96 ms < 25 ms → passes Gate 2.

2.3 Gate 3: Compute lower bound (prefill)

FLOPs_prefill = 2 × N × P = 2 × 8.03B × 512 (longest prompt) = 8.23 TFLOPs

Assuming 45% Tensor Core efficiency on A100 (140 TFLOP/s sustained):

t_prefill_comp = 8.23 / 140 = 58.8 ms

Add ~20 ms for kernel launches, KV allocation, network → ~79 ms unqueued prefill.

Feasibility check: Target TTFT ≤ 1,000 ms. 79 ms ≪ 1,000 ms → passes Gate 3.

---

3. Empirical load sweep: deriving λ_instance*

Analytical bounds assume zero queueing, zero prefill-decode interference, zero scheduler overhead. The only way to find the true sustainable request rate is an open-loop load sweep under mixed traffic.

3.1 Benchmark protocol (reproducible)

Control variable	Specification
Model	meta-llama/Meta-Llama-3-8B-Instruct, BF16 weights, BF16 KV
Runtime	vLLM (pinned version), chunked prefill enabled
GPU	A100-SXM4-80GB, clocks locked (nvidia-smi -lgc)
Workload	3-class mixed (70/20/10), Poisson arrivals
Sweep	λ ∈ 14 req/s
Duration	≥ 5 min or 1,000 completed requests per step
Warm-up	Discard until median TTFT/TPOT vary < 2% over 3 windows
Replications	≥ 3 independent runs; report median ± MAD

3.2 What to measure at each step

Metric	Source	SLO threshold
p95 TTFT	Client streaming timestamp	≤ 1,000 ms
p95 TPOT	Client (n−1 gaps) or server request_time_per_output_token_seconds	≤ 25 ms
p99 E2E	Client	≤ 5,000 ms
Goodput	% requests meeting all three	≥ 99%
System throughput	generation_tokens counter delta / window	Output tokens/s

3.3 Calibration matrix (illustrative envelope — label as assumption)

λ (req/s)	Mean B	p95 TTFT	p95 TPOT	p99 E2E	Throughput	SLO Status
2.0	4.2	85 ms	12.1 ms	1.82 s	243 tok/s	PASS
4.0	8.8	110 ms	13.4 ms	2.15 s	486 tok/s	PASS
6.0	14.1	165 ms	15.2 ms	2.68 s	730 tok/s	PASS
8.0	20.5	280 ms	17.8 ms	3.45 s	973 tok/s	PASS
10.0	28.2	620 ms	21.5 ms	4.65 s	1,216 tok/s	PASS (λ_instance*)
12.0	42.6	1,450 ms	28.2 ms	7.80 s	1,410 tok/s	FAIL
14.0	78.0	3,800 ms	36.4 ms	14.20 s	1,480 tok/s	FAIL

Result: λ_instance* = 10.0 req/s (maximum arrival rate where all SLOs hold).

This table is an illustrative teaching envelope

The numbers above are derived from the research brief’s calibration matrix and represent observed behavior under standard open-loop benchmark conditions. They are not universal constants for “A100 + Llama-3-8B”. Your hardware, runtime version, kernel backend, prompt distribution, and SLO thresholds will produce different values. Run the sweep on your target configuration. The lesson teaches the method, not the number.

---

4. Queueing headroom and cluster sizing

4.1 Why ρ_target ≈ 0.70

In an M/M/c queue (or G/G/c approximation), mean queueing delay grows as:

W_q ∝ ρ / (1 − ρ)   where ρ = λ / (c × μ)

At ρ = 0.90, a 10% arrival burst multiplies queue delay by ~10×. At ρ = 0.70, the same burst multiplies delay by ~2.3×.

Standard SRE practice: Operate at ρ_target = 0.65–0.75 to absorb Poisson microbursts without tail-latency explosion.

4.2 Safe operating rate per GPU

λ_safe = λ_instance* × ρ_target = 10.0 × 0.70 = 7.0 req/s per GPU

4.3 Active GPU count for peak traffic

N_active = ceil(λ_peak / λ_safe) = ceil(60.0 / 7.0) = 9 GPUs

4.4 High-availability redundancy (N+1)

N_resilient = N_active + 1 = 10 GPUs

This tolerates one GPU failure during peak without SLO breach.

---

5. Discrete node bin-packing and TCO model

5.1 Cloud instance reality

Platform	Instance	GPUs/node	Hourly rate (us-east-1, On-Demand)
AWS EC2	p4de.24xlarge	8 × A100-80GB	$40.96/hr
Fractional (RunPod, Lambda, CoreWeave)	Single A100-80GB	1	~$2.49/hr

You cannot rent fractional GPUs on AWS EC2 On-Demand. Bin-packing is mandatory.

5.2 Node calculation

Nodes_AWS = ceil(N_resilient / 8) = ceil(10 / 8) = 2 nodes (16 GPUs)

Deployed capacity: 16 × 7.0 = 112 req/s safe (1.87× headroom over 60 req/s peak).

5.3 Monthly financial model (with PAR)

Hours/month = 24 × 30.5 = 732
λ_avg = λ_peak / PAR = 60 / 3.0 = 20.0 req/s
Requests/month = 20.0 × 86,400 × 30.5 = 52.7M
Output tokens/month = 52.7M × 121.6 = 6.41B tokens

Deployment	Monthly Cost	Cost / 1M Output Tokens
AWS p4de.24xlarge (2 nodes)	2 × $40.96 × 732 = $59,965	$59,965 / 6,409 = $9.36
Fractional (10 GPUs)	10 × $2.49 × 732 = $18,227	$18,227 / 6,409 = $2.84

Full-node vs fractional economics

Full-node cloud (AWS) buys simplicity, SLAs, and integration — at 3.3× the per-token cost. Fractional GPU clouds buy raw hardware efficiency but require you to manage node lifecycle, networking, and failover. The capacity plan must state which procurement model you are sizing for.

---

6. Sensitivity analysis: stress-test the plan

Stress Vector	Perturbation	Impact	Mitigation
Traffic surge	λ_peak +50% → 90 req/s	AWS 16-GPU absorbs (ρ→80%). Fractional 10-GPU queue blowup.	Size for surge or enable Lesson 16 admission shedding.
Context doubling	Summarization P₃→2,048 (S̄→374)	KV pool OK (18% used). Prefill FLOPs ×4 for Class 3 → λ* drops to ~7.5 req/s.	Add 2 GPUs or enable chunked prefill with smaller max_num_batched_tokens.
KV quantization (FP8)	–kv-cache-dtype fp8	KV/token 64 KiB (½). Bandwidth ↓ 5.5%, λ* ↑ 8–12%.	Verify quality regression ΔPP ≤ 1.02 on Wikitext-2.
Tightened TTFT SLO	1,000 ms → 400 ms	Smaller chunked prefill windows, lower batch aggregation → λ_safe 7.0→5.2 req/s.	Cluster expands to 12 active GPUs (15 resilient).
Node outage (N−1)	1 GPU fails at peak	AWS: 15 GPUs → 70 req/s safe (OK). Fractional: 9 GPUs → 63 req/s safe (OK).	N+1 holds SLO contract in both models.

---

7. Production Capacity & Cost Decision Record (template)

Produce this one-page artifact for engineering leadership sign-off.

7.1 Workload & SLO Contract

Dimension	Value
Peak arrival rate (λ_peak)	___ req/s (Poisson)
Peak-to-average ratio (PAR)	___
Prompt/output distributions	Class table with weights, P, O
TTFT p95	___ ms
TPOT p95	___ ms
E2E p99	___ ms
Goodput target	___ %

7.2 Physical Boundary Derivation

Bound	Formula	Value	Source
Usable VRAM	reported_VRAM × gpu_mem_util	___ GiB	Datasheet + config
Model weights	N_params × bytes/param	___ GiB	Model card
Workspace reserve	—	___ GiB	Engine docs
KV pool	Usable − Weights − Workspace	___ GiB	Calculated
Bytes/token (KV)	2 × L × H_kv × d_head × b_kv	___ KiB	Architecture spec
T_KV tokens	KV_pool / Bytes_per_token	___	Calculated
C_max (avg)	T_KV / (S̄ × 1.036)	___	Calculated
C_max (worst)	T_KV / S_max	___	Calculated
t_mem (decode, B=32)	(W + B×S̄×bytes) / BW_achieved	___ ms	Roofline lower bound
t_prefill (max P)	2×N×P / FLOPs_sustained	___ ms	Roofline lower bound

7.3 Empirical Load-vs-SLO Curve

λ (req/s)	Mean B	p95 TTFT	p95 TPOT	p99 E2E	Goodput	SLO
___	___	___	___	___	___	PASS/FAIL
λ_instance*	___	___	___	___	___	PASS

Attach benchmark logs, environment pin (GPU, driver, CUDA, vLLM version, clocks locked).

7.4 Bin-Packed Cluster Sizing

Metric	Value
λ_safe = λ_instance* × 0.70	___ req/s/GPU
N_active = ceil(λ_peak / λ_safe)	___ GPUs
N_resilient = N_active + 1	___ GPUs
Nodes (8-GPU)	ceil(N_resilient / 8) = ___
Total deployed GPUs	___
Cluster safe capacity	N_deployed × λ_safe = ___ req/s

7.5 Monthly TCO Breakdown

Deployment	Nodes/GPUs	Monthly Cost	Cost / 1M Tokens
AWS (full nodes)	___	$___	$___
Fractional (per-GPU)	___	$___	$___

All pricing: “as of [DATE]”. List region, instance type, commitment model.

7.6 Sensitivity Checks

Scenario	λ_safe	N_active	N_resilient	Nodes	Monthly Cost	SLO Held?
Nominal	___	___	___	___	$___	Yes
+50% surge	___	___	___	___	$___	___
Context ×2	___	___	___	___	$___	___
FP8 KV	___	___	___	___	$___	___
TTFT 400ms	___	___	___	___	$___	___
N−1 outage	___	___	___	___	$___	___

7.7 Reproducible Verification Script

Save as scripts/capacity_calculator.py and run in CI:

#!/usr/bin/env python3
"""
Capacity and Cost Verification Engine for LLM Serving Clusters.
Implements Track 2 Lesson 14 sizing methodology.
"""

import math
from dataclasses import dataclass


@dataclass
class WorkloadConfig:
    classes: list[dict]          # [{"weight": float, "prompt": int, "output": int}]
    peak_qps: float
    par: float
    slo_ttft_p95_s: float
    slo_tpot_p95_s: float
    slo_e2e_p99_s: float


@dataclass
class HardwareConfig:
    gpu_name: str
    reported_vram_gib: float
    hbm_bw_tb_s: float
    peak_bf16_tflops: float
    gpus_per_node: int
    node_hourly_cost: float
    single_gpu_hourly_cost: float


def compute_capacity_plan(wl: WorkloadConfig, hw: HardwareConfig):
    # 1. Weighted averages
    avg_prompt = sum(c["weight"] * c["prompt"] for c in wl.classes)
    avg_output = sum(c["weight"] * c["output"] for c in wl.classes)
    avg_context = avg_prompt + avg_output
    max_context = max(c["prompt"] + c["output"] for c in wl.classes)

    # 2. Memory sizing (Llama-3-8B GQA BF16)
    model_weights_gib = 14.90
    runtime_overhead_gib = 2.00
    workspace_gib = 1.75
    usable_vram = hw.reported_vram_gib * 0.90
    kv_pool_gib = usable_vram - model_weights_gib - workspace_gib
    kv_pool_bytes = kv_pool_gib * (1024**3)

    bytes_per_token = 2 * 32 * 8 * 128 * 2  # 128 KiB
    total_kv_tokens = kv_pool_bytes / bytes_per_token
    c_avg = total_kv_tokens / (avg_context * 1.036)
    c_safe = total_kv_tokens / max_context

    # 3. Empirical capacity & queue sizing
    # CALIBRATED: replace with your measured λ_instance*
    lambda_instance_star = 10.0  # req/s — MEASURED VALUE, NOT A CONSTANT
    rho_target = 0.70
    lambda_safe = lambda_instance_star * rho_target  # 7.0 req/s

    n_active = math.ceil(wl.peak_qps / lambda_safe)
    n_resilient = n_active + 1

    # 4. Discrete node packaging
    nodes_aws = math.ceil(n_resilient / hw.gpus_per_node)
    total_gpus_aws = nodes_aws * hw.gpus_per_node
    monthly_cost_aws = nodes_aws * hw.node_hourly_cost * 732

    total_gpus_fractional = n_resilient
    monthly_cost_fractional = (
        total_gpus_fractional * hw.single_gpu_hourly_cost * 732
    )

    # 5. Financials
    lambda_avg = wl.peak_qps / wl.par
    monthly_requests = lambda_avg * 86400 * 30.5
    monthly_output_tokens = monthly_requests * avg_output

    cost_per_1m_aws = monthly_cost_aws / (monthly_output_tokens / 1e6)
    cost_per_1m_frac = monthly_cost_fractional / (monthly_output_tokens / 1e6)

    return {
        "avg_prompt": avg_prompt,
        "avg_output": avg_output,
        "avg_context": avg_context,
        "kv_pool_gib": kv_pool_gib,
        "total_kv_tokens": int(total_kv_tokens),
        "c_avg": int(c_avg),
        "c_safe": int(c_safe),
        "lambda_safe": lambda_safe,
        "n_active": n_active,
        "n_resilient": n_resilient,
        "nodes_aws": nodes_aws,
        "total_gpus_aws": total_gpus_aws,
        "monthly_cost_aws": monthly_cost_aws,
        "cost_per_1m_aws": cost_per_1m_aws,
        "total_gpus_fractional": total_gpus_fractional,
        "monthly_cost_fractional": monthly_cost_fractional,
        "cost_per_1m_frac": cost_per_1m_frac,
    }


if __name__ == "__main__":
    workload = WorkloadConfig(
        classes=[
            {"weight": 0.70, "prompt": 32, "output": 64},
            {"weight": 0.20, "prompt": 128, "output": 256},
            {"weight": 0.10, "prompt": 512, "output": 256},
        ],
        peak_qps=60.0,
        par=3.0,
        slo_ttft_p95_s=1.0,
        slo_tpot_p95_s=0.025,
        slo_e2e_p99_s=5.0,
    )

    hardware = HardwareConfig(
        gpu_name="NVIDIA A100-SXM4-80GB",
        reported_vram_gib=79.65,
        hbm_bw_tb_s=2.039,
        peak_bf16_tflops=312.0,
        gpus_per_node=8,
        node_hourly_cost=40.96,   # AWS p4de.24xlarge
        single_gpu_hourly_cost=2.49,  # Fractional cloud
    )

    res = compute_capacity_plan(workload, hardware)

    print("=== CAPACITY PLANNING VERIFICATION ===")
    print(
        f"Workload: Avg Prompt={res['avg_prompt']:.1f}, Avg Output={res['avg_output']:.1f}, Avg Context={res['avg_context']:.1f}"
    )
    print(
        f"KV Memory Pool: {res['kv_pool_gib']:.2f} GiB ({res['total_kv_tokens']:,} tokens)"
    )
    print(
        f"Concurrency Ceiling: Avg Context={res['c_avg']} seqs, Max Context={res['c_safe']} seqs"
    )
    print(
        f"Cluster Sizing: Active={res['n_active']} GPUs, Resilient (+1)={res['n_resilient']} GPUs"
    )
    print(
        f"AWS EC2 Deployment: {res['nodes_aws']} nodes ({res['total_gpus_aws']} GPUs) -> ${res['monthly_cost_aws']:,.2f}/mo -> ${res['cost_per_1m_aws']:.2f} / 1M tokens"
    )
    print(
        f"Fractional Cloud: {res['total_gpus_fractional']} GPUs -> ${res['monthly_cost_fractional']:,.2f}/mo -> ${res['cost_per_1m_frac']:.2f} / 1M tokens"
    )

    # Deterministic assertions (update if inputs change)
    assert res["n_active"] == 9, "Active GPU count mismatch"
    assert res["n_resilient"] == 10, "Resilient GPU count mismatch"
    assert res["nodes_aws"] == 2, "AWS Node count mismatch"
    assert res["c_safe"] > 500, "KV memory safely accommodates >500 full seqs"
    print("\n✓ ALL DETERMINISTIC VERIFICATION ASSERTIONS PASSED.")

---

8. Epistemological boundary contract: what is fact, policy, or measurement

Category	Examples	Label in lesson
Physical bounds (Fact)	A100 80GB: 79.65 GiB reported, 2,039 GB/s peak BW, 312 TFLOPS BF16; Llama-3-8B: 32 layers, 8 KV heads, 128 head dim → 128 KiB/token BF16; PagedAttention eliminates external fragmentation; vLLM gpu_memory_utilization=0.90 reserves 10% headroom	“The hardware limit is…”, “The architecture dictates…”
Operational policy (Qualified)	ρ_target = 0.70 (queue headroom); N+1 redundancy; gpu_memory_utilization = 0.90; chunked prefill max_num_batched_tokens = 2048; cloud instance packaging (8 GPUs/node)	“We choose…”, “Standard practice sets…”, “Our policy assumes…”
Empirical measurement (Pending)	λ_instance* = 10.0 req/s; p95 TTFT/TPOT/E2E at each λ; Goodput %; achieved memory BW %; sustained TFLOPS %; cloud pricing (region, date, commitment)	“The sweep measured…”, “Benchmark on [DATE] showed…”, “Pricing as of [DATE]…”

Never present a measured λ_instance* as a hardware constant. It is a function of: model, precision, runtime version, kernel backend, prompt/output distribution, SLO thresholds, and arrival process. Identical hardware yields different λ* under different workloads.

---

9. Checkable engineering exercises

Try it yourself

1. 01A team wants to serve Llama-3-8B on A100-40GB (reported 39.8 GiB) with gpu_memory_utilization=0.90. Model weights: 14.90 GiB. Workspace: 1.75 GiB. Calculate the KV pool (GiB) and total KV token capacity at 128 KiB/token. If average context is 1,024 tokens with 3.6% tail waste, what is C_max (avg)?

   Show a hint

   Usable = 39.8 × 0.90 = 35.82 GiB. KV pool = 35.82 − 14.90 − 1.75 = 19.17 GiB. Total tokens = 19.17 GiB / 128 KiB = 156,900. C_max = 156,900 / (1,024 × 1.036) ≈ 148 seqs.

2. 02An open-loop sweep on your target hardware yields λ_instance* = 8.0 req/s at SLOs (TTFT≤1s, TPOT≤25ms). Peak traffic is 40 req/s with PAR=2.5. Using ρ_target=0.70, compute N_active, N_resilient, and AWS nodes (8 GPUs/node). What is the cluster safe capacity?

   Show a hint

   λ_safe = 8.0 × 0.70 = 5.6 req/s. N_active = ceil(40/5.6) = 8. N_resilient = 9. Nodes = ceil(9/8) = 2 (16 GPUs). Safe capacity = 16 × 5.6 = 89.6 req/s.

3. 03The fractional GPU cloud quote is $2.10/GPU-hr for A100-80GB. Using the N_resilient from Exercise 2, compute monthly cost and cost per 1M output tokens if λ_avg = 16 req/s and Ō = 121.6 tokens.

   Show a hint

   Monthly cost = 9 × $2.10 × 732 = $13,822. Monthly tokens = 16 × 86,400 × 30.5 × 121.6 = 5.15B. Cost/1M = $13,822 / 5,150 = $2.68.

4. 04Your measured λ_instance* drops from 10 to 7 req/s when context doubles (prefill contention). With λ_peak=60, PAR=3.0, ρ=0.70, how many additional GPUs does the AWS deployment need?

   Show a hint

   Old λ_safe=7.0 → N_active=9. New λ_safe=4.9 → N_active=ceil(60/4.9)=13. N_resilient=14. Nodes=ceil(14/8)=2 (still 16 GPUs, but now only 16×4.9=78.4 req/s safe vs 112 before). If surge+context both hit, may need 3rd node.

---

10. Sources and scope

Claim / Mechanism	Primary Source	Verified Scope
A100 80GB specs	NVIDIA A100 Datasheet (SXM4)	2,039 GB/s BW, 312 TFLOPS BF16, 81,559 MiB reported
Llama-3-8B GQA architecture	HF config.json (NousResearch/Meta-Llama-3-8B-Instruct)	32 layers, 8 KV heads, 128 head dim, 8.03B params
vLLM gpu_memory_utilization default	vLLM docs: CacheConfig	Default 0.90–0.92, reserves headroom
PagedAttention block mapping	Kwon et al., SOSP 2023 (arXiv:2309.06180)	Virtual memory for KV cache, tail waste bound
Continuous batching / iteration scheduling	Yu et al., OSDI 2022 (Orca)	Iteration-level eviction/insertion
Chunked prefill	Agrawal et al., 2024 (arXiv:2403.02310)	Co-scheduling prefill chunks with decode
Roofline decode model	Pope et al., 2022 (arXiv:2211.05102)	Memory-bound decode, compute-bound prefill
Queueing theory / ρ_target	Kingman’s formula; SRE production practice	W_q ∝ ρ/(1−ρ); ρ≈0.70 standard headroom
AWS p4de.24xlarge pricing	AWS EC2 Instance Types (P4)	8× A100-80GB, $40.96/hr On-Demand us-east-1
Open-loop Poisson benchmark	vLLM benchmark_serving_structured_output.py	request_rate follows Poisson process

---

11. What this lesson does not do

• It does not give you a universal “A100 = 10 req/s” number. That number is a measured outcome of a specific workload, runtime, and SLO.
• It does not model prefill/decode disaggregation (Lesson 15) or multi-GPU tensor parallelism (Lesson 15) — those are separate scaling decisions.
• It does not design admission control, queue limits, or shedding policies (Lesson 16) — those protect the cluster when traffic exceeds the plan.

---

Key takeaways

• A capacity plan chains: workload contract → analytical memory/bandwidth bounds → empirical open-loop load sweep → queueing headroom (ρ≈0.70) → N+1 redundancy → discrete node bin-packing → TCO with PAR → sensitivity analysis.
• KV cache per-token footprint is an architectural fact: MHA=512 KiB, GQA=128 KiB, FP8=64 KiB (for Llama-3-8B specs). Using the wrong formula causes OOM.
• Analytical lower bounds (t_mem, t_prefill) filter impossible hardware. Passing them does not guarantee SLO compliance — only the load sweep does.
• λ_instance* is a measured property of (hardware, model, runtime, workload, SLO). It is not a hardware constant. Label it as such.
• Queueing headroom (ρ_target) absorbs Poisson burst variance. Operating at ρ=0.90 guarantees tail-latency explosions under real traffic.
• Cloud bin-packing is mandatory on AWS (8-GPU nodes). Fractional GPU clouds change the economics but shift ops burden to you.
• Cost per 1M compliant tokens (Goodput) is the only defensible efficiency metric. Raw $/hr or raw tok/s mislead.

---

Interview questions

Why can you not calculate λ_instance* from first principles alone?

First principles give lower bounds (t_mem, t_prefill) assuming zero queueing, zero prefill-decode interference, zero scheduler overhead, and ideal memory bus efficiency. Real continuous-batching engines have prefill bursts that delay decode steps, Poisson arrival variance that creates transient queues, and framework scheduling jitter. The knee where SLOs break is an empirical property of the full system under mixed traffic.

A stakeholder says "our A100s do 10 req/s, so for 60 QPS we need 6 GPUs". What is wrong with this reasoning?

Three errors: (1) 10 req/s (if true) is the knee λ_instance*, not a safe operating rate. Without ρ≈0.70 headroom, any burst breaks SLOs. (2) λ_instance* is workload-dependent — their traffic may have longer contexts or stricter SLOs, yielding a lower λ*. (3) Discrete bin-packing: 6 GPUs requires 1 AWS node (8 GPUs), but N+1 redundancy pushes to 2 nodes (16 GPUs). The real number is 16, not 6.

How does Peak-to-Average Ratio (PAR) change the cost-per-token calculation vs a naive steady-state estimate?

Naive steady-state assumes λ_peak runs 24/7. With PAR=3.0, average λ is 1/3 of peak, so monthly token volume is 1/3 of the naive estimate. But you still pay for peak-capable infrastructure 24/7. Real cost-per-token is therefore ~3× higher than the naive "peak rate × 732 hrs" calculation. The PAR must appear explicitly in the monthly token denominator.

When does KV cache capacity become the primary bottleneck instead of compute/bandwidth?

When average context S̄ grows large enough that C_max(memory) < C_max(SLO). For Llama-3-8B on A100-80GB: KV pool ≈ 55 GiB → 450K tokens. At S̄=221, C_max≈1,971. The SLO-bound batch (from TPOT target) is typically < 50. So KV capacity only binds when S̄ > ~9,000 tokens (or with many more KV heads / larger models). For short/medium context, you hit the latency SLO first.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Build a Serving SystemChoose a Serving Runtime and Hardware15 min
• Inference Optimisation & AI Infrastructure · Build a Serving SystemScale Beyond One GPU: Replication, Tensor Parallelism, and Prefill/Decode Disaggregation16 min
• Inference Optimisation & AI Infrastructure · Build a Serving SystemOverload and Failure Handling16 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryBatching and Throughput26 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryPagedAttention: Virtual Memory for the KV Cache35 min