Batching and Throughput

1. The core problem: autoregressive memory demand

During autoregressive generation, a large language model generates one token per sequence on each execution step. For each generated token, the accelerator evaluates every linear projection across all transformer layers: key, value, query, and output projections in attention, plus gate, up, and down projections in the feed-forward network.

Consider a nominal 8-billion parameter model in 16-bit floating point precision (FP16, 2 bytes per parameter):

• Parameters (P): 8.0 × 10^9 parameters
• Model weight footprint (W): 8.0 × 10^9 × 2 bytes = 16.0 × 10^9 bytes (16.00 GB)
• Linear arithmetic per token: 2 × P = 16.0 × 10^9 FLOPs (16.00 GFLOPs)

For a single request (B = 1), assume nominally one 16.00 GB weight read from HBM per step and 16.00 GFLOPs of linear arithmetic. This is a traffic model; actual caching, repeated tile loads and evaluated parameter coverage change the byte count:

Operational Intensity (weight-only) = Compute / Weight Data Moved
                                    = 16.00 × 10^9 FLOPs / 16.00 × 10^9 bytes
                                    = 1.00 FLOP/byte

Compare this operational intensity against two datacenter accelerators:

• NVIDIA A100 SXM (80 GB): HBM2e bandwidth is 2.039 TB/s (2.039 × 10^12 B/s); dense FP16 Tensor Core peak is 312 TFLOP/s (312 × 10^12 FLOP/s). The Roofline ridge point is 312 × 10^12 / 2.039 × 10^12 ≈ 153.0 FLOP/byte.
• NVIDIA H100 SXM (80 GB): HBM3 bandwidth is 3.35 TB/s (3.35 × 10^12 B/s); dense FP16 Tensor Core peak is 989 TFLOP/s (989 × 10^12 FLOP/s). The Roofline ridge point is 989 × 10^12 / 3.35 × 10^12 ≈ 295.2 FLOP/byte.

Under this idealized memory-bound model on an H100 SXM, the accelerator achieves an upper-bound arithmetic throughput of:

Attainable Throughput <= 3.35 × 10^12 bytes/s × 1.00 FLOP/byte = 3.35 TFLOP/s

This represents 0.34% of the chip’s 989 TFLOP/s dense FP16 Tensor Core capability.

Attainable model bounds versus hardware stalls

A roofline ceiling of 0.34% of theoretical peak compute is an ideal attainable throughput bound under this simplified model, not a direct measurement of hardware execution stalls. In production execution, single-digit Tensor Core utilization reflects a combination of memory bus saturation, kernel launch overheads, warp under-occupancy on small activation grids, register pressure, and pipeline dispatch bubbles. Direct cycle attribution requires hardware profiling tools (such as NVIDIA Nsight Compute), rather than inferring stall percentages from peak arithmetic ratios.

---

2. Tensor shapes and on-chip tile reuse: from GEMV to GEMM

Batching multiple concurrent requests transforms the linear algebra kernel from a vector-matrix multiplication into a matrix-matrix multiplication.

Single Request (B = 1): GEMV
Activation Vector x:    [1 × d_in]
Weight Matrix W:        [d_in × d_out]
Output Vector y:        [1 × d_out]
-> Loading the full matrix W computes a single output vector y [1 × d_out], moving all weight bytes to advance one token.

Batched Requests (B > 1): GEMM
Activation Matrix X:    [B × d_in]
Weight Matrix W:        [d_in × d_out]
Output Matrix Y:        [B × d_out]
-> In the nominal once-per-step traffic model, W tiles are reused
   across multiple activation rows while resident on chip.

Activation data movement in a projection

A realistic calculation of linear projection memory movement must account for activation inputs and outputs alongside weights. Consider a single linear projection:

• Input activation X: shape [B, K]
• Weight matrix W: shape [K, M]
• Output activation Y: shape [B, M]

In FP16 precision (2 bytes per value):

• Input activation bytes: 2 × B × K
• Weight bytes: 2 × K × M
• Output activation write bytes: 2 × B × M
• Total nominal DRAM traffic: 2 × (B × K + K × M + B × M) bytes
• Floating-point operations: 2 × B × K × M FLOPs

Let us evaluate a projection with B = 32 and K = M = 4096:

Weight traffic:     2 × 4096 × 4096 = 33,554,432 bytes (32.00 MiB)
Input activation:   2 × 32 × 4096   = 262,144 bytes (256 KiB)
Output activation:  2 × 32 × 4096   = 262,144 bytes (256 KiB)
Total traffic:      33,554,432 + 262,144 + 262,144 = 34,078,720 bytes (32.50 MiB)

Compute:            2 × 32 × 4096 × 4096 = 1,073,741,824 FLOPs (1.074 GFLOPs)

Operational Intensity (with activations) = 1,073,741,824 / 34,078,720 ≈ 31.51 FLOP/byte
Operational Intensity (weight-only)      = 1,073,741,824 / 33,554,432 = 32.00 FLOP/byte

At B = 32, activation I/O adds roughly 1.5% to total DRAM traffic. As B expands, activation traffic grows proportionally with B × (K + M).

Tile reuse in on-chip SRAM

Modern GPUs cannot store an entire 16 GB model in on-chip SRAM. General Matrix Multiply (GEMM) kernels (such as CUTLASS or cuBLAS) decompose matrices into hierarchical tiles:

1. Thread block scheduling: Thread blocks load sub-tiles of activations and weights into shared memory (SRAM).
2. Tensor Core execution: Tensor Cores execute implementation-dependent matrix-multiply-accumulate instructions (with tile dimensions chosen for the device, dtype and kernel).
3. Logical rows versus hardware tiles: When B = 1, logical batch rows do not fill hardware tile dimensions, leading to masked lanes or padded tile overhead. When B = 32, thread blocks pack multiple active rows into fully populated Tensor Core tiles, multiplying each loaded weight sub-tile across populated rows before eviction.

Under the nominal modeling assumption that weight matrices are read once from HBM per forward step (actual traffic depends on L2 cache residency, multi-pass tiling, and kernel dispatch structure), weight-only operational intensity scales as:

Operational Intensity (weight-only, B) = (2 × P × B) / (2 × P) = B FLOP/byte

In this weight-only abstraction, an H100 would reach its ridge point at B ≈ 295. However, real LLM decode is not a weight-only workload: it must also read the Key-Value (KV) cache.

---

3. Incorporating KV cache: analyzing the compute crossover

In autoregressive decode, every new token attends to all previous tokens in its sequence. Each step reads the relevant KV history through the memory hierarchy. This model charges one full HBM read per active sequence; cache hits, sharing and kernel re-reads can change physical traffic.

KV cache traffic accounting

For an 8B model with Grouped-Query Attention (GQA) matching Llama-3-8B specifications:

• Layers (L): 32
• Key-Value heads per layer (H_kv): 8
• Head dimension (d_head): 128
• Precision: 16-bit float (2 bytes per element)

The KV cache memory required per token per sequence is:

Bytes per token = 2 × L × H_kv × d_head × 2 bytes
                = 2 × 32 × 8 × 128 × 2
                = 131,072 bytes (128 KiB)

At an average context length of S = 1024 tokens:

KV footprint per sequence = 1024 × 131,072 bytes = 134,217,728 bytes (128 MiB)

For batch size B, total memory traffic moved per decode step in this model is:

Data Moved(B) = Weight_bytes + B × KV_bytes_per_seq
              = 16.0 × 10^9 + B × 134,217,728 bytes

4. The slopes of memory and compute

Under an idealized Roofline analytical model on an H100 SXM (3.35 TB/s HBM3, 989 TFLOP/s dense FP16), execution step time is modeled as the maximum of memory transit time and linear compute time:

t_step(B) = max(t_mem(B), t_comp(B))

t_comp(B) = (B × 16.0 × 10^9 FLOPs) / (989 × 10^12 FLOP/s)
          = B × 0.016178 ms

t_mem(B)  = (16.0 × 10^9 + B × 134,217,728) / (3.35 × 10^12 B/s)
          = 4.7761 ms + B × 0.040065 ms

Examine the marginal cost per added sequence (B):

• Compute slope: 0.016178 ms per sequence
• Memory slope: 0.040065 ms per sequence

Because reading 128 MiB of KV cache over the 3.35 TB/s bus requires 0.040065 ms per sequence, while linear projections require only 0.016178 ms per sequence, t_mem(B) > t_comp(B) for every positive batch size B.

Time (ms)
  ^
  |                                        / Memory: 4.78 + B × 0.0401 ms
  |                                      /
  |                                    /
  |                                  /
  |                                /   / Compute: B × 0.0162 ms
  |                              /   /
  |                            /   /
  |                          /   /
  |  4.78 ms ------------- /   /
  |                      /   /
  |                    /   /
  +----------------------------------------------------> Batch Size (B)

We evaluate the asymptotic operational intensity as B → ∞:

Intensity(B) = (B × 16.0 × 10^9) / (16.0 × 10^9 + B × 134,217,728)

lim (B → ∞) Intensity(B) = (16.0 × 10^9) / 134,217,728 ≈ 119.209 FLOP/byte

Because 119.209 FLOP/byte is strictly below the H100 SXM ridge point of 295.22 FLOP/byte, this nominal 8B, S=1024, FP16 linear+KV model has no compute crossover on an H100 peak roofline.

Analytical Roofline performance sweep

The table below presents analytical estimates. Step time is the larger of the memory and compute columns; model throughput is B / step_time. No row is a measured serving result.

Batch B	Memory (ms)	Compute (ms)	Intensity (FLOP/B)	Model (tok/s)
1	4.82	0.02	0.99	207.6
2	4.86	0.03	1.97	411.8
4	4.94	0.06	3.87	810.3
8	5.10	0.13	7.50	1,569.7
16	5.42	0.26	14.11	2,953.6
32	6.06	0.52	25.23	5,282.1
64	7.34	1.04	41.65	8,719.0
128	9.90	2.07	61.72	12,923.5
256	15.03	4.14	81.33	17,029.5

Hardware limits beyond the simplified roofline

This sweep is an analytical cost model (Pope et al., 2022). It isolates linear projection FLOPs and KV reads, and intentionally omits:

• Attention matrix multiplication FLOPs (Q × K^T and Softmax × V)
• Activation memory reads and writes across transformer layers
• KV cache writes for the newly generated token
• KV cache rereads, L2 cache hit reuse, or non-coalesced access penalties
• Kernel launch overheads and GPU dispatch bubbles
• Hardware execution efficiencies (warp divergence, instruction pipeline latency)
• Host engine scheduling, queue management, and tokenization latency

---

5. Educational verification algorithm

To reproduce the arithmetic of this analytical model, save the following Python script as roofline_model.py and run it with Python 3:

python3 roofline_model.py

"""
Educational analytical model for batching operational intensity and execution time.
"""

def evaluate_decode_roofline(
    params_total: float,
    weight_bytes_per_param: int,
    kv_bytes_per_elem: int,
    layers: int,
    kv_heads: int,
    head_dim: int,
    seq_len: int,
    hbm_bandwidth_bytes_sec: float,
    peak_flops_sec: float,
    batch_sizes: list[int],
) -> list[dict]:
    weight_bytes = params_total * weight_bytes_per_param
    kv_bytes_per_token = 2 * layers * kv_heads * head_dim * kv_bytes_per_elem
    kv_bytes_per_seq = seq_len * kv_bytes_per_token

    results = []
    for b in batch_sizes:
        total_kv_bytes = b * kv_bytes_per_seq
        total_bytes = weight_bytes + total_kv_bytes
        total_flops = 2.0 * params_total * b

        intensity = total_flops / total_bytes
        t_mem_sec = total_bytes / hbm_bandwidth_bytes_sec
        t_comp_sec = total_flops / peak_flops_sec
        t_step_sec = max(t_mem_sec, t_comp_sec)
        throughput = b / t_step_sec

        results.append({
            "b": b,
            "intensity_flop_byte": intensity,
            "t_mem_ms": t_mem_sec * 1000.0,
            "t_comp_ms": t_comp_sec * 1000.0,
            "t_step_ms": t_step_sec * 1000.0,
            "throughput_tok_sec": throughput,
            "kv_total_gib": total_kv_bytes / (1024 ** 3),
        })
    return results

if __name__ == "__main__":
    P = 8.0e9
    WEIGHT_BYTES = 2
    KV_BYTES = 2
    LAYERS = 32
    KV_HEADS = 8
    HEAD_DIM = 128
    S = 1024
    HBM_BW = 3.35e12   # 3.35 TB/s H100 SXM
    PEAK_FLOPS = 989e12 # 989 TFLOP/s dense FP16

    sweep = evaluate_decode_roofline(
        P, WEIGHT_BYTES, KV_BYTES, LAYERS, KV_HEADS, HEAD_DIM, S, HBM_BW, PEAK_FLOPS, [1, 32, 128, 256]
    )

    # Precomputed reference values for independent assertion check across all sweep sizes
    expected = {
        1: {"t_mem_ms": 4.816, "t_comp_ms": 0.016, "intensity": 0.992},
        32: {"t_mem_ms": 6.058, "t_comp_ms": 0.518, "intensity": 25.228},
        128: {"t_mem_ms": 9.904, "t_comp_ms": 2.071, "intensity": 61.724},
        256: {"t_mem_ms": 15.033, "t_comp_ms": 4.141, "intensity": 81.334},
    }

    for r in sweep:
        b = r["b"]
        exp = expected[b]
        assert abs(r["t_mem_ms"] - exp["t_mem_ms"]) < 0.01, f"t_mem mismatch at B={b}"
        assert abs(r["t_comp_ms"] - exp["t_comp_ms"]) < 0.01, f"t_comp mismatch at B={b}"
        assert abs(r["intensity_flop_byte"] - exp["intensity"]) < 0.02, f"intensity mismatch at B={b}"
        assert r["t_mem_ms"] > r["t_comp_ms"], f"Expected memory bound at B={b}"
        assert r["intensity_flop_byte"] < 119.21, f"Exceeded asymptote at B={b}"

    print("Deterministic verification passed across all batch sizes. Sample outputs:")
    for r in sweep:
        print(
            f"B={r['b']:3d} | Intensity={r['intensity_flop_byte']:5.2f} FLOP/B | "
            f"t_mem={r['t_mem_ms']:5.2f} ms | t_comp={r['t_comp_ms']:5.2f} ms | "
            f"KV={r['kv_total_gib']:5.2f} GiB | Throughput={r['throughput_tok_sec']:7.1f} tok/s"
        )

The script prints the step times and intensities for the sweep, verifying that t_mem_ms > t_comp_ms across all evaluated batch sizes and confirming that intensity remains below 119.21 FLOP/byte.

---

6. Concurrency, arrival dynamics, and queue boundaries

To understand how batching affects end-to-end latency, we must distinguish three separate concepts:

1. Batch size (B): The number of sequence tokens executed together in a single GPU forward pass.
2. In-flight concurrency (N): The total number of active requests held in the serving system (waiting in queue, running prompt prefill, or generating tokens in decode).
3. Arrival rate (λ): The number of new incoming requests submitted per unit of time (such as requests per second).

Little’s Law and steady-state boundaries

In a queueing system with defined entry and exit boundaries, the long-run average number of requests in the system (L) relates to average arrival rate (λ) and average residence time (W) by Little’s Law:

L = λ × W

This relationship holds under precise boundary conditions:

• The system reaches long-run statistical equilibrium over the observation horizon.
• Arrivals, departures, and any rejected or dropped requests are accounted for consistently at the measured boundary.

When the arrival rate λ approaches system capacity, queuing delay increases non-linearly. If arrival rate persistently exceeds service capacity without admission limits or backpressure, queues grow without bound, and steady-state formulas no longer apply.

Open-loop versus closed-loop evaluation

The evaluation harness dictates what system properties are exposed:

• Closed-loop testing: A fixed client pool issues requests, waiting for completion before sending the next. Concurrency N is clamped by client count, avoiding runaway queue accumulation; client-observed response latency reflects execution plus any queuing behind other clients in that fixed concurrency pool.
• Open-loop testing: Requests arrive at timestamps generated independently of server status (such as a Poisson process). If the server experiences a latency spike, incoming requests accumulate in the admission queue. Open-loop testing is necessary to expose queue stability, buffer bloat, and tail latency under an externally imposed arrival rate.

---

7. Dynamic batching: aggregation delay and service queues

In general-purpose serving engines (such as NVIDIA Triton Inference Server), requests arrive independently over time. Dynamic batchers use aggregation windows to accumulate batches before kernel dispatch.

Request Arrival Timeline (Deterministic Spacing: Δt = 20 ms, Target B = 8)

Arrivals:     R0       R1       R2       R3       R4       R5       R6       R7
Time (ms):    0        20       40       60       80       100      120      140
              |--------|--------|--------|--------|--------|--------|--------|
              R0 Wait: 140 ms -------------------------------------------------> [DISPATCH]
                       R1 Wait: 120 ms ----------------------------------------> [DISPATCH]
                                R2 Wait: 100 ms -------------------------------> [DISPATCH]
                                                                        R7 Wait: 0 ms -> [DISPATCH]

Deterministic arrival schedule

Suppose requests arrive deterministically every Δt = 20.0 ms (λ = 50.0 req/s) and the scheduler targets B = 8:

• Request 0: Waits for 7 subsequent arrivals: 7 × 20.0 ms = 140.0 ms.
• Request 1: Waits for 6 subsequent arrivals: 6 × 20.0 ms = 120.0 ms.
• Request 7: Waits 0.0 ms.

Under this deterministic schedule, the mean aggregation wait is:

Mean Aggregation Wait = (140 + 120 + 100 + 80 + 60 + 40 + 20 + 0) / 8 = 70.0 ms

Here, 140.0 ms is the exact maximum wait time experienced by the first request.

Stochastic arrival schedule

When arrivals follow a Poisson process with mean rate λ = 50 req/s, inter-arrival times are exponentially distributed. The time required to collect B - 1 additional requests follows an Erlang distribution with shape k = B - 1 and rate λ.

The expected aggregation wait for the first arriving request is:

E[Wait_first] = (B - 1) / λ = (8 - 1) / 50 = 0.140 s = 140.0 ms

Under random arrivals, 140.0 ms is an expected value, not an upper bound. Because inter-arrival gaps vary randomly, engines enforce timeout thresholds (such as max_queue_delay_microseconds). If the timer expires before 8 requests arrive, the engine dispatches whatever partial batch has formed.

Client-observed Time To First Token

In an autoregressive transformer, the prompt prefill pass evaluates all input prompt tokens and directly produces the logits for the first generated token. No additional decode forward pass is needed for token 1.

Under an illustrative serial timeline with disjoint measurement timestamps:

Client TTFT = t_request_transport_and_preprocess + t_aggregation
            + t_service_queue + t_prefill + t_emit_and_transport

• t_request_transport_and_preprocess: Request transmission, parsing and tokenization before the illustrative server queue boundary.
• t_aggregation: Time spent waiting inside the dynamic batching aggregation window.
• t_service_queue: Time spent waiting in the admission or engine queue before execution starts.
• t_prefill: Forward execution time across the prompt tokens to produce first-token logits.
• t_emit_and_transport: Time to sample the first token, serialize it, and deliver it over the network to the client.

In pipelined or chunked-prefill engines, aggregation and queue stages can partially overlap, and server-side metrics often exclude client network transport.

---

8. Variable sequence lengths and static slot allocation

In static batching, all sequences in a batch are co-dispatched in a fixed rectangular tensor.

Quantifying padded token slots

Consider four requests co-scheduled in a static batch:

• Request 1: 20 tokens
• Request 2: 50 tokens
• Request 3: 80 tokens
• Request 4: 200 tokens

Step:        0         20             50             80                       200
Req 1 (20):  [=========][----------------- Padded / Idle Slots ---------------]
Req 2 (50):  [========================][-------- Padded / Idle Slots ---------]
Req 3 (80):  [======================================][-- Padded / Idle Slots -]
Req 4 (200): [================================================================]

Because the static batch executes synchronously across all rows, the tensor must step for max(lengths) = 200 iterations:

Total Allocated Token Slots = 4 × 200 = 800 slots
Useful Tokens Generated     = 20 + 50 + 80 + 200 = 350 tokens
Padded Token Slots          = 800 - 350 = 450 slots
Padded Slot Ratio           = 450 / 800 = 56.25%

9. Distinguishing slot waste from hardware waste

Token slot waste is not direct hardware waste

A 56.25% padded token slot ratio does not mean 56.25% of total GPU power or memory bandwidth is wasted. Model weights are fetched from HBM nominally once per step and amortized across active rows. An implementation may skip inactive KV positions, while other padded or masked kernels can still perform loads or arithmetic; slot counts alone do not specify physical traffic. The primary penalty of static batching is allocation lock-in: the system reserves execution slots that cannot be assigned to newly arrived requests until Request 4 completes.

10. Iteration scheduling versus memory management

Addressing static batch inefficiencies involves two separate mechanisms:

1. Iteration-level scheduling (Continuous Batching): A scheduler policy that operates at the granularity of individual forward passes (Orca, Yu et al., 2022). When Request 1 finishes at step 20, the scheduler can retire it and insert a new sequence into that slot at step 21. Orca introduced iteration-level scheduling before paged memory management existed.
2. Paged memory allocation (PagedAttention): A memory layout mechanism that allocates KV cache in non-contiguous physical memory blocks (Kwon et al., 2023). Paged memory eliminates full-length contiguous buffer reservations, although it retains bounded internal fragmentation within partially filled terminal blocks.

While modern serving engines combine both techniques, they solve orthogonal problems: scheduling policy decides which tokens execute, while paged allocation determines where KV data resides.

---

11. Architectural decision study: sizing concurrency under memory and SLO constraints

Sizing scenario

An inference service deploys a nominal 8B FP16 model on an NVIDIA H100 SXM accelerator. The platform team configures an explicit 30 GiB memory pool dedicated to KV cache.

• Workload profile: Homogeneous FP16 KV cache, context length hard-bounded at S = 1024 tokens, no prefix sharing.
• Target constraints:
  • Time To First Token (TTFT P95) ≤ 250 ms
  • Inter-Token Latency (ITL P95) ≤ 12 ms
• Proposed setting: A team member suggests setting the server’s concurrency limit to B = 256, citing the analytical table’s high throughput estimate (17,029.5 tok/s).

Capacity and latency evaluation

First, evaluate the hard capacity constraint of the 30 GiB KV pool:

KV cache per sequence = 1024 tokens × 128 KiB/token = 128 MiB = 0.125 GiB
Raw pool capacity     = 30.0 GiB / 0.125 GiB/seq = 240 sequences (excluding allocator headroom)

At B = 256:
Required KV Memory = 256 × 0.125 GiB = 32.0 GiB
Available KV Pool  = 30.0 GiB
Capacity Status    = FAILS (32.0 GiB > 30.0 GiB; exceeds pool by 2.0 GiB)

At B = 128:
Required KV Memory = 128 × 0.125 GiB = 16.0 GiB
Available KV Pool  = 30.0 GiB
Capacity Status    = FEASIBLE (Consumes 16.0 GiB of 30.0 GiB pool)

At B = 64:
Required KV Memory = 64 × 0.125 GiB = 8.0 GiB
Capacity Status    = FEASIBLE (Consumes 8.0 GiB of 30.0 GiB pool)

Promising 256 simultaneously resident full 1024-token contexts in a 30 GiB pool violates hard physical capacity (which caps out at 240 sequences raw, even before accounting for block metadata and allocator headroom). Setting an engine flag such as max_num_seqs = 256 is compatible with this pool if the scheduler admits fewer or shorter resident requests as capacity requires; the flag alone does not reserve 32 GiB.

Unvalidated latency hypotheses and measurement plan

Next, consider the capacity-feasible candidates B = 64 and B = 128:

• At B = 64: Analytical Roofline model step time estimate is 7.34 ms.
• At B = 128: Analytical Roofline model step time estimate is 9.90 ms.

These model step times are idealized estimates under assumed nominal traffic, not physical lower bounds (caching and memory layout can alter byte transfers, while launches and scheduling add overhead). Whether B = 128 meets the ITL P95 ≤ 12 ms target cannot be decided from the Roofline table alone.

Furthermore, we must distinguish three concurrency metrics:

• Requested client concurrency: Total external requests attempting to connect.
• Engine active sequence cap: Maximum concurrent sequences admitted to the engine.
• Per-step decode batch size: Sequences co-dispatched in a specific forward execution pass.

Required empirical measurement plan:

1. Closed-loop baseline: Benchmark fixed concurrency levels N in [32, 64, 96, 128] to identify the engine’s saturation throughput and measure response and step latency at each fixed concurrency, including any server queue time.
2. Open-loop trace evaluation: Replay a representative Poisson arrival trace with target arrival rates λ. Measure empirical P50, P95, and P99 percentiles for TTFT and ITL across B = 64 and B = 128 caps.
3. Queue boundary identification: Identify the arrival rate threshold where service queue backlog causes TTFT P95 to exceed 250 ms, and set admission control limits accordingly.

---

12. Key takeaways

Key takeaways

• Single-request decode (B=1) achieves an ideal upper bound of only 0.34% of H100 SXM Tensor Core peak because fetching 16 GB of weights for 16 GFLOPs yields an operational intensity of 1.0 FLOP/byte.
• Batching amortizes nominal weight transfers by loading weight sub-tiles into on-chip SRAM to serve multiple activation rows, transforming GEMV into GEMM. Adding activation traffic reduces operational intensity slightly relative to weight-only models.
• In a declared nominal 8B, S=1024, FP16 linear+KV model on H100, memory transfer slope (0.0401 ms/seq) exceeds compute slope (0.0162 ms/seq), yielding an asymptotic intensity of 119.2 FLOP/byte that never crosses the 295.2 FLOP/byte ridge point.
• Static batch allocation with variable sequence lengths causes padded token slot waste; iteration-level scheduling (Continuous Batching) addresses execution granularity, while non-contiguous allocation (PagedAttention) eliminates full-length buffer reservation while retaining bounded tail internal fragmentation.

---

13. Exercises

Try it yourself

1. 01Using the nominal 8B FP16 model (W = 16.0 GB) on an H100 SXM (3.35 TB/s, 989 TFLOP/s) at context S=1024 (128 MiB KV cache per sequence), compute the calculated memory time, compute time, and operational intensity for batch size B=64. Does this configuration cross into the compute-bound regime?

   Show a hint

   Total bytes = 16.0e9 + 64 * 134,217,728 = 24.59 GB. t_mem = 24.59e9 / 3.35e12 = 7.34 ms. Compute = 64 * 16.0e9 = 1,024 GFLOPs. t_comp = 1,024e9 / 989e12 = 1.04 ms. Intensity = 1,024e9 / 24.59e9 = 41.65 FLOP/byte. Because t_mem (7.34 ms) > t_comp (1.04 ms) and 41.65 < 295.2 FLOP/byte, it remains strictly memory-bound.

2. 02A dynamic batcher receives requests following an open-loop Poisson arrival process at mean rate λ = 25 requests/second and targets batch size B = 4. What is the expected aggregation wait time for the first arrival in a batch, and why does this expected value not represent a hard maximum wait time?

   Show a hint

   The expected wait to accumulate 3 additional arrivals is (B - 1) / λ = 3 / 25 = 0.120 seconds (120 ms). It is not a hard maximum because Poisson inter-arrival times are exponentially distributed random variables with unbounded tails; actual fill time can exceed 120 ms unless bounded by a max_queue_delay timeout.

3. 03A static batch consists of 4 sequences with generation lengths [30, 60, 90, 120] tokens. Calculate the total allocated token slots, useful tokens, and the percentage of padded token slots. Explain why the engine does not waste that exact percentage of total GPU bandwidth.

   Show a hint

   Total slots = 4 * 120 = 480 slots. Useful tokens = 30 + 60 + 90 + 120 = 300 tokens. Padded slots = 480 - 300 = 180 slots (37.5%). GPU bandwidth is not wasted by 37.5% because model weights are fetched once per step regardless of row status, and KV cache is only fetched for existing tokens in each sequence.

---

14. Sources and scope

Claim / Mechanism	Primary Source	Verified Scope / Section	Notes
Decode Roofline Model	Pope et al., Efficiently Scaling Transformer Inference	§2, Analytical cost model	Analyzes linear compute and memory traffic; does not model runtime scheduling or CUDA launch overhead.
Iteration-Level Scheduling	Orca: A Distributed Serving System for Transformer-Based Generative Models	§1, §3.1–3.2	Establishes token-level iteration scheduling to eliminate static batch padding bubbles.
Dynamic Batching Controls	NVIDIA Triton Inference Server: Delayed Batching	User Guide: Batcher architecture	Documents max_queue_delay_microseconds and preferred_batch_size aggregation mechanics.
Paged Memory Management	PagedAttention (vLLM)	§4.1–4.3	Resolves internal/external KV cache memory fragmentation in dynamic serving.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceWhere Inference Time and Memory Go13 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceFlashAttention and GPU Data Movement16 min
• LLM Internals & Architecture · Inference foundationsHow LLM inference works9 min