Choose a Serving Runtime and Hardware

Vendor benchmarks frequently promise simple answers: “100 tokens/s on a commodity server CPU,” “5× lower latency with custom runtime X,” or “scale to 8 GPUs for maximum throughput.” In production, choosing hardware or runtimes based on marketing claims leads to missed Service Level Objectives (SLOs), unexpected out-of-memory (OOM) crashes, or quadrupled cloud bills.

Every serving deployment is constrained by physical hardware limits and software execution overheads. Memory capacity dictates whether a model and its active KV cache fit without swapping. Memory bandwidth sets an unyielding lower bound on generation speed when arithmetic intensity is low. Compute throughput bounds prompt prefill speed. Interconnect bandwidth dictates whether multi-GPU partitioning speeds up execution or slows it down. Finally, runtime scheduling determines whether theoretical hardware capabilities translate into compliant tokens under bursty traffic.

The one-sentence version

Analytical hardware lower bounds filter out physically unviable serving candidates before committing infrastructure budget, while empirical load testing under realistic arrival traffic validates whether a surviving runtime satisfies tail-latency and Goodput contracts.

---

1. The reader’s runtime and hardware decision

Before deriving equations or configuring a deployment, an engineer must make an architectural choice across three primary hardware topologies and software runtimes. That decision is dictated by a strict contract: target model, precision, sequence length distribution, concurrency, and Service Level Objectives for Time To First Token (TTFT) and Inter-Token Latency (ITL).

+--------------------------------------------------------------------------------------------------+
|                                    WORKLOAD & SLO CONTRACT                                       |
|  Model: Meta-Llama-3-8B | Precision: BF16 | Concurrency: C=16 | TTFT <= 1.0s | ITL <= 25ms       |
+--------------------------------------------------------------------------------------------------+
                                                 |
                                                 v
                     [ GATE 1: Addressable Memory Capacity Floor ]
                     Does Usable RAM/VRAM >= Weights + KV Cache + Workspace?
                                /                           \
                              NO                            YES
                              /                               \
             [ REJECT: Host/Device OOM ]                       v
                                             [ GATE 2: Memory Bandwidth Lower Bound ]
                                             Does t_mem = Data_Moved / BW <= ITL SLO?
                                                        /                           \
                                                      NO                            YES
                                                      /                               \
                                     [ REJECT: Bandwidth Infeasible ]                  v
                                                                     [ GATE 3: Compute Throughput Lower Bound ]
                                                                     Does t_prefill = 2NP / FLOPs <= TTFT SLO?
                                                                                /                           \
                                                                              NO                            YES
                                                                              /                               \
                                                             [ REJECT: Compute Infeasible ]                    v
                                                                                             [ GATE 4: Interconnect Feasibility (TP) ]
                                                                                             Does Communication Overhead Exceed Scaling Gains?
                                                                                                        /                           \
                                                                                                      YES                            NO
                                                                                                      /                               \
                                                                                     [ REJECT: Negative Scaling ]                      v
                                                                                                                     [ GATE 5: Empirical Goodput & Cost ]
                                                                                                                     Execute Live Load Benchmark Matrix:
                                                                                                                     Measure p95 TTFT, p95 ITL, Goodput, $/1M

The Architectural Decision Matrix

The following decision rules govern the selection:

1. Enterprise Server CPU (e.g., Dual Xeon / EPYC with vLLM CPU, OpenVINO, or llama.cpp):
   • Select when: Workloads are non-interactive (offline batch document indexing, background summarization, asynchronous data extraction), traffic is sparse and sporadic, or high host RAM (>256 GB) is required to hold large models without procuring expensive GPU nodes.
   • Reject when: Interactive conversational streaming is required (ITL SLO ≤ 30 ms at concurrency C ≥ 8). Standard DDR5 memory buses (380–600 GB/s achieved) hit a physical memory-bandwidth wall, capping decode step times at 45–80 ms per token.
2. Single High-Bandwidth GPU (e.g., 1× NVIDIA A100-80GB SXM4 with vLLM or SGLang):
   • Select when: Interactive conversational serving (e.g., 8B parameter models in BF16) requires strict streaming latency (ITL ≤ 25 ms, TTFT ≤ 1.0 s), and total memory footprint (weights + active KV cache + workspace) fits within addressable VRAM (~74.5 GiB usable). High-bandwidth memory (HBM2e @ 1,550 GB/s achieved) delivers an idealized decode step time of ~11.8 ms, providing ample headroom for scheduler overhead and tail queueing.
   • Reject when: Model weights alone exceed single-device capacity (e.g., 70B models in BF16 requiring >130 GiB), or concurrency requirements force KV cache footprints beyond available framebuffer.
3. Multi-GPU Tensor Parallelism (e.g., 2× NVIDIA A100-80GB with vLLM / SGLang, TP=2 over NVLink):
   • Select when: The model cannot physically fit on a single GPU, or ultra-strict generation latency (ITL ≤ 10 ms) requires partitioning weight tensors to double aggregate memory bandwidth, and high-speed NVLink (600 GB/s bidirectional) is available.
   • Reject when: A single GPU already satisfies the latency SLO. Sharding an 8B model across multiple GPUs introduces 64 All-Reduce communication synchronizations per decode step. If connected over PCIe (64 GB/s theoretical) rather than NVLink, communication latency outweighs memory bandwidth gains, producing negative scaling while doubling hourly instance rental cost.
4. Runtime Framework Selection (vLLM / SGLang vs. Triton/TensorRT-LLM vs. llama.cpp):
   • vLLM / SGLang (CUDA): Preferred for multi-tenant HTTP/gRPC services requiring dynamic continuous batching, PagedAttention memory virtualization, chunked prefill, and automatic prefix caching.
   • Triton with TensorRT-LLM: Preferred in enterprise environments with standardized hardware pipelines requiring ahead-of-time graph compilation, custom low-bit INT4/FP8 kernel tuning, and deep integration with Triton enterprise metrics.
   • llama.cpp / llama-server: Preferred for single-tenant local deployment, edge devices, Apple Silicon, or lightweight CPU environments utilizing GGUF quantized checkpoints without complex Python dependencies.

---

2. The canonical workload comparison

To compare candidate architectures objectively, evaluate them against a single, explicit workload contract. Do not vary batch size, sequence length, or precision between hardware options.

The Canonical Workload Specification

• Named Model Checkpoint: meta-llama/Meta-Llama-3-8B-Instruct
  • Hyperparameters: 32 layers; model width 4,096; feed-forward width 14,336; 32 query heads and 8 KV heads (Grouped Query Attention, 4:1 ratio); head dimension 128; vocabulary size 128,256.
  • Exact Parameter Count: N = 8,030,261,248 ≈ 8.030 × 10^9 parameters.
• Precision: bfloat16 weights, activations, and KV cache (2 bytes per value).
  • Weight payload: 8.03026 × 10⁹ parameters × 2 bytes = 16.06 GB = 14.96 GiB.
  • KV cache per token: 2 × 32 layers × 8 KV heads × 128 dimensions × 2 bytes = 131,072 bytes = 128 KiB/token.
• Workload Profile:
  • Prompt Length: $P = 1,024$ tokens.
  • Completion Length: 128 tokens (average sequence length 1,088 tokens; maximum sequence length 1,152 tokens).
  • Target Concurrency: $C = 16$ active concurrent requests.
• Target SLO Contract:
  • Time To First Token (p95 TTFT): ≤ 1,000 ms (1.0 second).
  • Inter-Token Latency (p95 ITL / TPOT): ≤ 25 ms (≥ 40 tokens/s per stream).
  • Minimum Goodput: ≥ 95% of requests must meet both TTFT and ITL thresholds.
  • Quality Requirement: 0% task accuracy loss relative to BF16 reference (MMLU / GSM8K).

Honest Accounting: Specifications vs. Lower Bounds vs. Assumptions

To prevent vendor marketing distortions, categorize every number into one of four epistemological tiers:

1. Direct Verified Specifications: Primary vendor datasheet parameters (e.g., peak memory bandwidth, peak dense BF16 FLOPs, memory capacity).
2. Calculated Analytical Lower Bounds: First-order physical limits derived from equations (e.g., weight payload, memory read duration t_mem, prefill compute duration t_comp).
3. Assumed Operational Parameters (OPEN): Working engineering assumptions (e.g., achieved memory bus efficiency, achieved GEMM TFLOP/s, driver context overhead, instance pricing) that require target-specific empirical microbenchmarks.
4. Empirical Observables Requiring Measurement (OPEN): True p95/p99 tail latencies, server queue times, and Goodput percentages under live traffic, which cannot be derived analytically.

Table 1. Analytical screening of hardware candidates on the canonical workload

Dimension	Path A: Dual Xeon 8480+ (CPU)	Path B: 1× NVIDIA A100-80GB (Single GPU)	Path C: 2× NVIDIA A100-80GB (TP=2 NVLink)	Path D: 1× NVIDIA L4 (Budget GPU)	Epistemological Basis
Serving Runtime	vLLM CPU / OpenVINO	vLLM CUDA	vLLM CUDA (TP=2)	vLLM CUDA	Software Stack
Marketed Memory	256 GB DDR5	80 GB HBM2e	2× 80 GB HBM2e	24 GB GDDR6	Vendor Datasheet
Usable Binary Memory	256 GiB	~74.51 GiB	2× 74.51 GiB	~22.35 GiB	Baseline (80× 10^9 / 2^30)
Peak Memory Bandwidth	614.4 GB/s	2,039 GB/s	2× 2,039 GB/s	300 GB/s	Vendor Datasheet
Assumed Achieved Bandwidth	~380 GB/s (~62%)	~1,550 GB/s (~76%)	2× 1,550 GB/s	~240 GB/s (~80%)	Assumed Parameter (OPEN)
Assumed Achieved Compute	~75 TFLOP/s (AMX)	~200 TFLOP/s	~400 TFLOP/s	~75 TFLOP/s	Assumed Parameter (OPEN)
Interconnect Type	4× UPI 2.0 (Socket)	N/A (Single chip)	NVLink 3 (600 GB/s bidi)	N/A (Single chip)	Architecture Spec
Memory Floor ($C=16$)	21.21 GiB	20.71 GiB	12.48 GiB / GPU	20.71 GiB	Calculated Bound (§3.1)
Data Moved per Step	18.34 GB	18.34 GB	9.17 GB / GPU	18.34 GB	Calculated Bound (§3.2)
Decode Step Lower Bound ($t_mem$)	~48.26 ms	~11.83 ms	~5.92 ms (+0.33 ms comm)	~76.42 ms	t_mem = Data / BW
Target ITL SLO	≤ 25 ms	≤ 25 ms	≤ 25 ms	≤ 25 ms	Target Contract
Gate 2 Bandwidth Status	Disqualified by Bound	Passes Lower Bound	Passes Lower Bound	Disqualified by Bound	Physical Lower Bound Check
Prefill Compute Lower Bound	~219 ms	~83.6 ms	~41.8 ms	~219 ms	16.72 TFLOPs / Compute
Target TTFT SLO	≤ 1,000 ms	≤ 1,000 ms	≤ 1,000 ms	≤ 1,000 ms	Target Contract
Assumed Hourly Rental	~$4.50 / hr	~$3.67 / hr	~$7.34 / hr	~$0.85 / hr	Assumed List Rate (OPEN)
Preliminary Screening Verdict	Disqualified (BW floor)	Primary Deployment Choice	Candidate (Cost Inefficient)	Disqualified (BW floor)	Analytical Elimination Filter

---

3. The five governing constraints of serving systems

Evaluating whether a serving architecture can meet an SLO contract requires passing five successive physical and operational gates.

Gate 1: Addressable Memory Capacity Floor

Before processing a single token, model parameters, framework runtime buffers, and active KV cache blocks must fit within physical device memory. If total memory demand exceeds addressable capacity, the system suffers fatal CUDA OOM crashes or catastrophic host paging latency.

The minimum addressable memory floor is: \text{Memory Floor} = W_weights + \text{KV}_total(C, S_max) + \text{Activation/Workspace} + \text{Allocator Overhead}

For Meta-Llama-3-8B-Instruct in BF16:

• Weight Footprint: W = 16.06 GB = 14.96 GiB.
• KV Cache at peak concurrency ($C=16, S_max=1,152$): \text{KV} = 16 \times 1,152 \times 128\text{ KiB} = 2,415,919,104\text{ bytes} \approx 2.42\text{ GB} = 2.25\text{ GiB}
• Assumed Framework & CUDA Workspace: ≈ 3.50 GiB (device runtime context, NCCL buffers, intermediate activation scratchpads).
• Total Addressable Floor: 14.96 + 2.25 + 3.50 = 20.71 GiB.

Marketed Decimal Capacity vs. Addressable Binary DRAM

Hardware vendors market GPU memory in decimal units (80GB = 80 × 10^9bytes). Operating systems and PyTorch allocate memory in binary gibibytes (1 GiB = 2³⁰ bytes = 1,073,741,824 bytes). Thus, an 80 GB GPU provides an analytical maximum of only ≈ 74.51GiB. Furthermore, hardware ECC parity bit reservation (dedicating up to 6.25% of DRAM on datacenter GPUs) and driver page tables reduce allocatable memory further. Never set --gpu-memory-utilization 1.0 in production; setting it to 0.90 on an 80 GB card reserves ≈ 67.06GiB for weights and KV cache, leaving necessary headroom for PyTorch workspaces.

Gate 2: Memory Bandwidth (The Decode Step Lower Bound)

During autoregressive generation, generating one token per stream requires reading all model weights from high-bandwidth memory into processor registers, alongside the active KV cache history for each stream.

For moderate batch sizes where arithmetic intensity remains below the processor’s Roofline ridge point ($I < I_ridge$), token generation is strictly memory-bandwidth bound. The total data moved per execution step is: \text{Data Moved}(B) = W_bytes + B \times \left( S_avg \times \text{Bytes}_KV_per_token \right)

At concurrency $C = 16$ with average sequence length $S_avg = 1,088$ tokens: \text{Data Moved} = 16.06\text{ GB} + 16 \times (1,088 \times 131,072\text{ bytes}) = 16.06\text{ GB} + 2.28\text{ GB} = \mathbf{18.34\text{ GB}}

The idealized memory-bound step lower bound is: t_mem = \frac{\text{Data Moved}}{\text{BW}_achieved}

Evaluating this lower bound against our 25 ms ITL target:

• Dual Xeon CPU (assumed achieved 380 GB/s across NUMA): t_mem = \frac{18.34\text{ GB}}{380\text{ GB/s}} = \mathbf{48.26\text{ ms}} Result: The physical time to stream weights and KV cache alone (48.26 ms) violates the 25 ms SLO by nearly 2×. The CPU is disqualified at Gate 2.
• NVIDIA L4 PCIe (assumed achieved 240 GB/s): t_mem = \frac{18.34\text{ GB}}{240\text{ GB/s}} = \mathbf{76.42\text{ ms}} Result: The GDDR6 memory bus violates the 25 ms target by more than 3×. Disqualified at Gate 2.
• NVIDIA A100-80GB SXM4 (assumed achieved 1,550 GB/s): t_mem = \frac{18.34\text{ GB}}{1,550\text{ GB/s}} = \mathbf{11.83\text{ ms}} Result: The HBM2e memory bus comfortably satisfies the lower bound check (11.83 ms ≪ 25 ms), leaving an operational margin of 13.17 ms for scheduling and kernel launch overheads.

Lower Bounds Are Necessary Filters, Not Production Proof

The analytical step time $t_mem$ represents an idealized physical lower bound assuming zero kernel launch latency, zero CPU-GPU driver dispatch overhead, zero KV cache block indexing pauses, and 100% memory bus efficiency. A candidate that fails t_mem ≤ ITL_SLO is guaranteed to fail in production. However, a candidate that passes t_mem ≤ ITL_SLO is not guaranteed to succeed under live load. Spikes in arrival rate, thread synchronization, and continuous batching re-evaluations add latency jitter that only empirical testing can capture.

Gate 3: Compute Throughput (The Prefill Lower Bound)

Processing prompt tokens is compute-dense matrix multiplication (I ≫ I_ridge). For prompt length $P$, total floating-point operations scale as 2 · N · P linear projection FLOPs, plus causal self-attention quadratic terms: \text{Compute}_prefill(P) = 2 \cdot N \cdot P + 2 \cdot L \cdot H_q \cdot D \cdot P^2\text{ FLOPs}

For $P = 1,024$: \text{Linear FLOPs} = 2 \times 8.03026 \times 10^9 \times 1024 \approx 16.45\text{ TFLOPs} \text{Attention FLOPs} = 2 \times 32 \times 32 \times 128 \times (1024)^2 \approx 0.27\text{ TFLOPs} \text{Total Compute} \approx \mathbf{16.72\text{ TFLOPs}}

The prefill compute lower bound is: t_prefill_comp \ge \frac{\text{Compute}_prefill(P)}{\text{Sustained Dense GEMM TFLOP/s}}

Evaluating against our 1,000 ms TTFT target:

• On Dual Xeon AMX (assumed achieved 75 TFLOP/s): t_comp ≈ 16.72 / 75 ≈ 223 ms.
• On 1× NVIDIA A100 (assumed achieved 200 TFLOP/s): t_comp ≈ 16.72 / 200 ≈ 83.6 ms.
• On 2× NVIDIA A100 (TP=2, assumed aggregate 400 TFLOP/s): t_comp ≈ 16.72 / 400 ≈ 41.8 ms.

All candidates satisfy the raw prefill compute check. However, client-perceived TTFT includes HTTP/gRPC ingress, tokenization, scheduler queueing, and the execution time of the first decode token. Passing Gate 3 guarantees only that GEMM math does not exceed the budget.

Gate 4: Interconnect Bandwidth & Communication Overhead (The Multi-GPU Gate)

When partitioning an 8B model across $k=2$ GPUs using standard 1D Megatron-style Tensor Parallelism (TP):

• Self-attention column-parallel projections ($W_Q, W_K, W_V$) and row-parallel projection ($W_O$) require one All-Reduce per transformer layer.
• MLP gate/up projections and down projection require a second All-Reduce per transformer layer.
• For a 32-layer model, every single decode step executes 64 All-Reduce operations across the interconnect.

Data transferred per All-Reduce for batch $B=16$, hidden size $d_model=4096$, in BF16 ($b=2$): M = B \times d_model \times b = 16 \times 4096 \times 2 = 131,072\text{ bytes} = 128\text{ KiB}

For a 2-GPU ring All-Reduce, communication latency per step is modeled as: t_comm \approx 64 \times \left( \alpha + \frac{M}{\text{BW}_interconnect} \right) where α is kernel launch and synchronization latency.

The PCIe Tensor Parallelism Trap

Assume an engineer attempts to run TP=2 on two GPUs connected over standard PCIe Gen4 x16 (theoretical 64 GB/s bidirectional, assumed α ≈ 15 μs): t_comm_PCIe \approx 64 \times \left( 15\ \mu\text{s} + \frac{131,072\text{ bytes}}{64 \times 10^9\text{ B/s}} \right) \approx 64 \times (15 + 2.05)\ \mu\text{s} \approx \mathbf{1.09\text{ ms}} If synchronization jitter or root complex contention increases α to 40 μs, communication overhead escalates to >2.7 ms per step.

In contrast, on SXM4 nodes with NVLink 3 (600 GB/s bidirectional, assumed α ≈ 5 μs): t_comm_NVLink \approx 64 \times \left( 5\ \mu\text{s} + \frac{131,072\text{ bytes}}{600 \times 10^9\text{ B/s}} \right) \approx 64 \times (5 + 0.22)\ \mu\text{s} \approx \mathbf{0.33\text{ ms}} Scope Boundary: This 2-All-Reduce rule applies strictly to standard 1D Megatron TP. Sequence Parallelism (Megatron-SP) and Context Parallelism (DeepSpeed-Ulysses) use Reduce-Scatter / All-Gather or All-to-All communication patterns (covered in later scaling modules).

Gate 5: Economic Screening & Empirical Goodput

A platform that meets latency targets but costs 10× more than necessary is an engineering failure. Similarly, a platform that rents for $0.85/hr but fails the latency SLO delivers zero compliant output.

Define economic efficiency as cost per million compliant tokens: \text{Cost Efficiency} = \frac{\text{Hourly Instance Rental Cost}}{\text{Compliant Tokens Delivered per Hour}} \times 10^6\quad (\$/1\text{M Goodput Tokens})

If Path D (1× L4) achieves 0% Goodput because its 76.4 ms step time breaches the 25 ms SLO, its effective cost per compliant token is infinite.

Comparing Path B (1× A100 @ $3.67/hr) versus Path C (2× A100 @ $7.34/hr):

• Path B’s physical lower bound (11.83 ms) already sits well below the 25 ms threshold.
• Path C lowers the physical lower bound to 6.25 ms, but doubles the hourly hardware burn rate.
• Unless live traffic benchmarks prove that Path B suffers severe tail queueing that violates Goodput, deploying Path C represents premature and wasteful over-provisioning.

---

4. Serving runtime framework architecture

Hardware capability is accessible only through the serving runtime engine. Modern LLM runtimes decouple client API handling from GPU execution, relying on four core subsystems:

+---------------------------------------------------------------------------------------------------+
|                                     SERVING RUNTIME ARCHITECTURE                                  |
+---------------------------------------------------------------------------------------------------+
|  1. INGRESS & HTTP/gRPC API LAYER                                                                |
|     - Fast API / Axum streaming endpoints | SSE chunk transport | Request deserialization        |
+---------------------------------------------------------------------------------------------------+
                                                  |
                                                  v
|  2. ITERATION-LEVEL SCHEDULER                                                                     |
|     - Continuous batching (iteration eviction & insertion)                                       |
|     - Chunked prefill scheduling (co-scheduling prefill chunks alongside decode steps)            |
|     - Prefix caching lookup (Radix tree / hash table matching)                                   |
+---------------------------------------------------------------------------------------------------+
                                                  |
                                                  v
|  3. MEMORY ALLOCATOR & KV CACHE MANAGER                                                           |
|     - PagedAttention virtual memory block tables | Zero physical memory compaction               |
|     - Free block pool | Prefix reference counting | Swapping & eviction manager                   |
+---------------------------------------------------------------------------------------------------+
                                                  |
                                                  v
|  4. KERNEL EXECUTION & HARDWARE BACKEND                                                           |
|     - Fused attention (FlashAttention-2 / FlashInfer / oneDNN / AMX tiles)                        |
|     - CUDA Graphs for fixed-shape decode execution (eliminating host launch latency)              |
|     - Collective communication (NCCL All-Reduce for TP)                                           |
+---------------------------------------------------------------------------------------------------+

Detailed Runtime Trade-offs

Capability	vLLM (v0.6.3+) CUDA	SGLang (v0.3.5+) CUDA	Triton / TensorRT-LLM	llama.cpp / llama-server
Primary Execution Backend	CUDA / FlashAttention-2	CUDA / FlashInfer / Triton	TensorRT compilation / CUDA	C++ SIMD (AVX-512, AMX, Metal)
Continuous Batching	Iteration-level scheduling	Token-level scheduling	In-flight batching scheduler	Slot-based continuous batching
KV Cache Management	PagedAttention virtual blocks	Paged block pool	Paged KV cache allocator	Contiguous slot buffer
Prefix Caching	Automatic Prefix Caching (Hash)	RadixAttention (Dynamic Tree)	Key-value reuse cache	Prompt cache slot restoration
Chunked Prefill	Supported (--enable-chunked-prefill)	Supported natively	Supported in runtime config	Limited slot chunking
Tensor Parallelism	Native NCCL 1D Megatron	Native NCCL / custom kernels	Optimized NCCL / custom tree	Unsupported (Uses RPC splits)
Deployment Complexity	Low (Single Python command/docker)	Low (Python package / docker)	High (AOT engine build required)	Minimal (Single standalone binary)
Best Production Fit	General enterprise LLM APIs	Complex multi-turn / agent DAGs	High-scale fixed-model pipelines	Edge devices, Mac, CPU servers

---

5. The reproducible benchmark protocol

Because analytical formulas provide idealized lower bounds, final verification requires live load benchmarking under realistic traffic conditions.

5.1 Controlled Variables Matrix

When running the empirical verification benchmark, pin all variables in the following matrix:

+-----------------------------------------------------------------------------------------------------+
| CATEGORY          | VARIABLE NAME                 | SPECIFICATION / CONTROL MECHANISM               |
+-------------------+-------------------------------+-------------------------------------------------+
| Model Checkpoint  | model_name_or_path            | meta-llama/Meta-Llama-3-8B-Instruct             |
|                   | weight_dtype                  | bfloat16                                        |
|                   | kv_cache_dtype                | bfloat16                                        |
+-------------------+-------------------------------+-------------------------------------------------+
| Runtime Stack     | engine_name & version         | vLLM v0.6.3 / SGLang v0.3.5 (pinned build)      |
|                   | backend_driver_version        | NVIDIA Driver 550.54.15 / CUDA 12.4             |
|                   | attention_kernel              | FlashAttention-2 / FlashInfer                   |
|                   | tensor_parallel_size          | 1 (Single GPU) vs 2 (Multi-GPU)                 |
|                   | max_model_len                 | 8192                                            |
|                   | gpu_memory_utilization        | 0.90                                            |
+-------------------+-------------------------------+-------------------------------------------------+
| Hardware Testbed  | device_model                  | NVIDIA A100-SXM4-80GB                           |
|                   | clock_control                 | Clocks locked via nvidia-smi -lgc               |
+-------------------+-------------------------------+-------------------------------------------------+
| Workload Profile  | input_length_distribution     | Fixed = 1024 (or Normal(mean=1024, std=64))     |
|                   | output_length_distribution    | Fixed = 128 (or Normal(mean=128, std=16))       |
|                   | arrival_process               | Poisson process (varying arrival rate lambda)   |
|                   | concurrency_sweep             | C in {1, 2, 4, 8, 16, 24, 32}                   |
+-------------------+-------------------------------+-------------------------------------------------+
| Execution Controls| warmup_protocol               | Discard until median TTFT & ITL vary < 2%       |
|                   |                               | across 3 consecutive 10-request windows         |
|                   | timed_requests                | >= 200 requests per concurrency level           |
|                   | independent_replications      | >= 3 runs (record Median +/- MAD)               |
|                   | seed_control                  | seed=42, temperature=0.0 (greedy) for profiling |
+-------------------+-------------------------------+-------------------------------------------------+
| Empirical Outputs | TTFT_client_p95               | Time to first streamed chunk (ms)               |
|                   | ITL_client_p95                | Inter-token chunk delta (ms)                    |
|                   | server_queue_time_p95         | Time spent waiting in admission queue (ms)      |
|                   | goodput_percentage            | % requests meeting TTFT <= 1s & ITL <= 25ms     |
|                   | cost_per_1m_goodput_tokens    | (Hourly Cost / Compliant Output Tokens) * 1e6   |
+-----------------------------------------------------------------------------------------------------+

5.2 Executing the Benchmark Sweep

Execute the load test against the pinned vLLM instance using a client harness:

# Launch vLLM server on Candidate Path B (1x A100 SXM4)
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --port 8000 &

# Lock GPU memory and graphics clocks to eliminate power throttling noise
sudo nvidia-smi -pm 1
sudo nvidia-smi -lgc 1410,1410

# Execute client load sweep with synthetic Poisson traffic
python3 benchmark_serving.py \
  --endpoint http://localhost:8000/v1/completions \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --prompt-tokens 1024 \
  --output-tokens 128 \
  --concurrency 16 \
  --arrival-process poisson \
  --request-rate 8.0 \
  --num-requests 250 \
  --warmup-requests 50 \
  --ttft-slo 1000 \
  --itl-slo 25

If the measured results show:

• Client p95 TTFT = 340 ms ≤ 1,000 ms
• Client p95 ITL = 18.2 ms ≤ 25 ms
• Goodput = 98.4% ≥ 95%
• Cost per 1M Goodput Tokens = $1.28

Then Path B (1× A100-80GB) is formally validated and approved for production release.

---

Key takeaways

• Analytical hardware models establish physical lower bounds for screening candidates before spending capital.
• Passing a physical bound does not guarantee tail-latency compliance under live traffic.
• Memory bandwidth can constrain decode latency at the stated batch size and workload.
• Tensor parallelism adds collective communication that must be measured against its bandwidth savings.
• Compare cost per compliant token under load, not hourly instance rental alone.

---

Checkable Engineering Exercise

Hands-On Hardware Selection Problem

You are the lead inference architect evaluating cloud hardware for an interactive conversational assistant serving meta-llama/Meta-Llama-3-8B-Instruct in bfloat16. Workload & Contract Specification: - Sustained concurrency: $C = 8$ concurrent streams. - Workload context: Fixed prompt $P = 512$ tokens; fixed completion $O = 256$ tokens ($S_avg = 640$ tokens, $S_max = 768$ tokens). - Target SLO Contract: Client TTFT (p95) ≤ 500 ms; Client ITL (p95) ≤ 30 ms; Goodput ≥ 95%. Candidate Offerings (Hypothetical Parameters): 1. Candidate A (Dual Server CPU): Dual AMD EPYC 9654 (192 cores, DDR5-4800, assumed achieved STREAM bandwidth 580 GB/s, assumed achieved BF16 GEMM 45 TFLOP/s, 384 GB host RAM). Assumed rental: $4.20/hr. 2. Candidate B (Single Budget GPU): 1× NVIDIA L4 PCIe (24 GB GDDR6, ~22.35 GiB usable baseline, assumed achieved bandwidth 240 GB/s, assumed achieved BF16 GEMM 75 TFLOP/s). Assumed rental: $0.85/hr. 3. Candidate C (Single High-Bandwidth GPU): 1× NVIDIA A100-PCIe-80GB (80 GB HBM2e, ~74.51 GiB usable baseline, assumed achieved bandwidth 1,500 GB/s, assumed achieved BF16 GEMM 200 TFLOP/s). Assumed rental: $3.20/hr. Common Assumptions: Framework overhead and workspace consume 3.50 GiB on GPU and 4.0 GiB on CPU.

Try it yourself

1. 01Calculate the minimum addressable memory floor (GiB) for this workload (C=8, S_max=768) on GPU and determine whether any candidate is eliminated at Gate 1 (Memory Capacity).

   Show a hint

   Weights: 14.96 GiB. KV cache = 8 * 768 * 128 KiB = 805,306,368 bytes = 0.75 GiB. Total floor = 14.96 + 0.75 + 3.50 = 19.21 GiB. Compare against Candidate A (384 GiB), Candidate B (22.35 GiB baseline), and Candidate C (74.51 GiB baseline). All three pass Gate 1 under steady state, though Candidate B has only 3.14 GiB spare KV pool.

2. 02Calculate the total data moved per decode step (GB) at C=8 and S_avg=640, and compute the idealized memory-bound step lower bound (t_mem) for Candidates A, B, and C. Which candidate(s) violate the 30 ms ITL target at the physical lower-bound level?

   Show a hint

   Weights = 16.06 GB. KV traffic = 8 * 640 * 131,072 bytes = 0.671 GB. Total data moved = 16.06 + 0.671 = 16.73 GB. Candidate A (580 GB/s): 16.73 / 580 = 28.84 ms (razor-thin 1.16 ms margin; thread barriers will push real p95 ITL > 30 ms). Candidate B (240 GB/s): 16.73 / 240 = 69.71 ms (disqualified, violates 30 ms target by 2.3x). Candidate C (1,500 GB/s): 16.73 / 1,500 = 11.15 ms (comfortably passes with 2.7x headroom).

3. 03Compute the prefill compute lower bound (ms) for P=512 on Candidate C against the 500 ms TTFT target.

   Show a hint

   Compute = 2 * 8.03026e9 * 512 = 8.22 TFLOPs linear + 0.07 TFLOPs attention = 8.29 TFLOPs. On Candidate C (200 TFLOP/s): t_comp = 8.29 / 200 = 41.45 ms. 41.45 ms << 500 ms, leaving 458 ms for network transit, queueing, and first decode token.

4. 04Formulate an engineering defense explaining why Candidate C is the primary deployment choice, and why empirical load testing is mandatory before production release.

   Show a hint

   Candidate B is physically disqualified by memory bandwidth (69.71 ms > 30 ms). Candidate A has only a 1.16 ms physical margin, which OS scheduling and NUMA sync will exhaust. Candidate C is the only candidate whose physical lower bounds (11.15 ms step, 41.45 ms prefill) provide sufficient operational headroom. However, because Poisson arrival bursts can queue multiple prefill requests behind active decodes, empirical load testing must verify that client p95 TTFT <= 500 ms and p95 ITL <= 30 ms under live traffic before signoff.

---

Interview questions

Why does doubling the number of GPUs via Tensor Parallelism not automatically cut decode step latency in half for an 8B model?

While 1D Megatron TP halves the weight data moved per device, it inserts two All-Reduce synchronization barriers into every transformer layer (64 All-Reduces per step for a 32-layer model). Each All-Reduce carries hardware launch latency and interconnect transit time. On slow interconnects like PCIe (or at small hidden dimensions), the aggregate communication latency exceeds the memory bandwidth savings, leading to plateaued or negative scaling.

An infrastructure engineer argues that since a modern dual-socket server CPU has 512 GB of DDR5 RAM and costs less than an A100 GPU node, the team should serve interactive chat on CPU. How do you evaluate this proposal against physical constraints?

Memory capacity is only Gate 1. In interactive serving, decode is memory-bandwidth bound. A dual-socket DDR5 system achieves roughly 380 to 580 GB/s of sustained bandwidth, resulting in an idealized decode step time of 28 to 48 ms per token for an 8B model at batch 8–16. Factoring in NUMA latency and thread barriers, real-world p95 ITL will exceed 50–80 ms per token, violating standard interactive conversational streaming SLOs (typically 20–30 ms). The CPU architecture is viable for batch/offline jobs, but physically incapable of delivering low-latency interactive streaming.

Explain why an analytical Roofline calculation showing t_mem = 11.8 ms does not guarantee that a deployment will achieve p95 ITL <= 25 ms under live traffic.

Roofline models assume idealized conditions: zero queueing, zero CPU-GPU driver dispatch overhead, zero thread contention, 100% memory bus saturation, and no concurrent prefill interruptions. In production, requests arrive irregularly (often modeled as a Poisson process). When a new long prefill arrives, it can preempt or delay ongoing decode iterations in continuous batching, inflating p95/p99 ITL. True SLO compliance depends on queue dynamics, chunked prefill scheduling, and KV cache allocation jitter, which can only be measured via empirical load benchmarking.

How does the concept of Goodput prevent teams from being misled by raw tokens/sec or cheap hourly GPU rental rates?

Raw throughput counts all generated tokens regardless of whether the user experienced acceptable latency. An inexpensive GPU ($0.85/hr) may generate 200 tokens/s, but if every token takes 76 ms to arrive when the user contract requires 25 ms, 100% of the requests violate the SLO, yielding 0% Goodput. Effective cost is calculated as hourly rental divided by compliant tokens delivered. A cheap instance with zero Goodput has an infinite cost per good token.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceServing Workloads and Performance Metrics12 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryBatching and Throughput26 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryContinuous Batching: Iteration-Level Scheduling30 min

---

Sources and scope

• NVIDIA A100 Tensor Core GPU Datasheet — SXM4 80GB specs: 2,039 GB/s peak memory bandwidth, 312 TFLOP/s dense BF16 Tensor Core compute, 400W TDP.
• NVIDIA L4 Tensor Core GPU Datasheet — PCIe 24GB specs: 300 GB/s peak memory bandwidth, 121 TFLOP/s dense BF16 Tensor Core compute, 72W TDP.
• Intel Xeon Platinum 8480+ Processor Specifications — Sapphire Rapids 56 cores / 112 threads, 16-channel DDR5-4800 (614.4 GB/s theoretical peak bandwidth across 2 sockets), Intel AMX matrix acceleration.
• AMD EPYC 9654 Processor Specifications — Genoa 96 cores / 192 threads, 24-channel DDR5-4800 (921.6 GB/s theoretical peak bandwidth across 2 sockets), AVX-512 VNNI.
• NVIDIA Ampere Architecture In-Depth — NVLink 3 architecture: 12 links per GPU, 600 GB/s bidirectional theoretical bandwidth.
• PCI-SIG PCIe 4.0 Specification — PCIe Gen4 x16 link: 64 GB/s theoretical bidirectional interconnect bandwidth.
• Pope et al., Efficiently Scaling Transformer Inference, MLSys 2023, arXiv:2211.05102 — Memory-bound autoregressive decode operational intensity (I ≈ B/b), compute-dense prefill scaling ($2NP$). Source of the Roofline bounds in §3.
• Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, arXiv:1909.08053 — Standard 1D Tensor Parallelism: column-parallel attention and MLP projections, row-parallel output projections, and the 2 All-Reduces per transformer block communication pattern in §3.4.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023, arXiv:2309.06180 — Virtual memory management for KV cache, elimination of internal and external memory fragmentation.
• vLLM Production Documentation & vLLM CPU Installation Docs — Engine architecture, continuous batching scheduler, chunked prefill, and CPU backend execution.
• SGLang Documentation & Source — RadixAttention dynamic tree-based KV cache reuse and FlashInfer kernel execution.

Scientific integrity and runtime verification status: The mathematical formulas and theoretical peak hardware specifications in this lesson are derived from verified vendor datasheets and peer-reviewed literature. Operational parameters—including achieved memory bus efficiencies (~76% A100, ~62% DDR5 across NUMA, ~80% L4), sustained BF16 GEMM compute throughput, kernel launch overheads, and cloud instance pricing—are explicit working engineering assumptions (OPEN status). No physical GPU cluster or server CPU was benchmarked during the drafting of this text. Before signing off on production deployments, execute the reproducible benchmark matrix (§5) on target instances under synthetic Poisson traffic to measure actual p95 TTFT, p95 ITL, and Goodput percentages.