Overload and Failure Handling

The one-sentence version

When request arrivals exceed sustainable service capacity, an unbounded queue does not merely increase latency—it harms goodput by allocating GPU resources to work that clients may abandon. Admission control should reject or shed load before queue wait exceeds client timeout, and the engine must propagate cancellation to free KV blocks at the next iteration boundary.

---

Opening burst: 30 requests/s arriving, 10 requests/s sustainable

[Illustrative scenario] The numbers below are a teachable scenario with explicit assumptions; they are not measurements from a benchmark run.

• Sustainable service rate (μ): 10 requests/second (illustrative rate, not a measured benchmark)
• Burst arrival rate (λ): 30 requests/second (step surge lasting 10 seconds)
• Net accumulation rate: λ − μ = 20 requests/second
• Queue depth after 10 s (unbounded FIFO): Q(10) = 20 × 10 = 200 waiting requests
• Estimated queue wait for a newly arriving request: W_est = Q / μ = 200 / 10 = 20.0 seconds
• Client timeout / TTFT SLO: 2.0 s / 1.0 s

A request arriving at t = 10 s faces ~20 s of queue delay before its prefill begins. By the time its first token would be generated, the client has been gone for ~18 s. If the engine admits it anyway, the GPU spends hundreds of forward passes generating tokens no one receives—while requests behind it starve.

Admission decision at t = 10 s: REJECT immediately with HTTP 429 Too Many Requests, including Retry-After: 20 and X-Overload-Reason: queue_delay_exceeded.

This lesson explains the queue, deadline, KV-memory, and cancellation mechanisms behind that decision, and ends with a one-page policy you can file alongside the capacity and topology records from Lessons 14–15.

Q/μ is a rough estimate, not a per-request guarantee

W_est = Q / μ assumes a stable service rate and FIFO order. Real systems exhibit variance in prefill duration, decode length, preemption, and scheduler overhead. Treat W _est as a rough estimate; actual wait may be higher or lower depending on workload variance.

---

1. Why LLM overload differs from stateless web services

In a traditional CRUD service, requests take 10–100 ms and release memory immediately. Queue growth degrades latency gracefully until the thread pool exhausts.

In autoregressive LLM serving, five properties change the failure mode:

Dimension	Traditional RPC	Autoregressive LLM Serving
Service time variance	Low (10–100 ms)	Extreme: 10 ms – 30 s (prompt length × output length)
Memory footprint	Stateless or ephemeral	Stateful & monotonically expanding: KV cache reserved for the full generation lifetime
Execution coupling	Thread-pool / event-loop concurrency	Co-scheduled in unified iteration batches: prefill (compute-bound) and decode (memory-bound) share Tensor Cores and HBM
Cost of timed-out work	Milliseconds of CPU	Hundreds of iterations of GPU Tensor Core + HBM bandwidth
Failure mode under saturation	Latency degradation	Goodput collapse + preemption storm: queue lag > client timeout ⇒ GPU busy but 0% useful output

The “doomed request” and goodput collapse

Goodput (from Lesson 1) = % of requests meeting all SLO targets (TTFT, TPOT, E2E) without error.

Under the burst above (λ = 30, μ = 10, client timeout = 2.0 s):

1. Unbounded FIFO queue accumulates 20 req/s
2. At t = 10 s, Q = 200, W_est = 20 s
3. Every newly admitted request already exceeds the client timeout by ~18 s
4. If cancellation is not propagated, GPU runs at 100% utilization generating tokens for dead sockets
5. Goodput may be reduced when many requests are cancelled, though the GPU remains fully busy.

Busy GPU ≠ useful throughput

A 100% utilized GPU producing only abandoned responses is cost without value. The admission policy must ensure every admitted request has a credible path to completing within the client’s patience.

---

2. Queue architectures and admission policy choices

Incoming Request
       │
       ▼
[ Gateway Layer ] ──> Token Bucket Rate Limiter (per-tenant / per-API key)
       │ (passes rate limit)
       ▼
[ Admission Controller ]
       ├─ Check 1: Virtual Queue Delay W_est = Q / μ̂
       │           If W_est + t_prefill > SLO_TTFT ──> REJECT (HTTP 429)
       ├─ Check 2: Available KV Memory Headroom
       │           If gpu_cache_usage_perc > 0.85 ──> DEFER or REJECT
       │ (passes admission)
       ▼
[ Engine Waiting Queue ] (Bounded by max-num-queued-reqs)
       │
       ▼
[ Iteration Scheduler ] (Continuous Batching Loop)
       ├─ Sequence Budget: active_seqs <= max-num-seqs
       ├─ Token Budget:    active_tokens <= max-num-batched-tokens
       └─ Block Budget:    allocatable_blocks >= required_immediate_blocks
               │
               ▼
       [ GPU Forward Pass ]

2.1 Queue discipline trade-off analysis

Queue discipline comparison under sustained overload (λ = 30, μ = 10)

Policy	Mechanism	Advantage	Critical LLM Failure Mode
FIFO Unbounded	Process by arrival; buffer all	Fair when λ < μ	Queue lag grows linearly. Admitted requests arrive at GPU past client timeout; goodput → 0
FIFO Tail-Drop (Q<sub>max</sub>)	Buffer up to Q<sub>max</sub>; reject new with HTTP 503	Protects host memory	Does not guarantee TTFT. Q<sub>max</sub>=200, μ=10 ⇒ admitted wait 20 s, still breaches 1.0 s SLO
LIFO / Drop-Head	Prioritize newest; shed oldest	Fresh requests meet TTFT	Starves long-waiting requests. Unfair unless combined with explicit cancellation
Predictive TTFT Admission	Estimate W_est = Q / μ̂. If W_est + t_prefill > SLO_TTFT ⇒ REJECT (HTTP 429)	Maximizes goodput. GPU never processes doomed work	Requires reliable μ̂ and prefill estimate
Class-Based Priority	Multiple queues (High: chat; Low: batch)	Protects interactive users	Low-priority starves without aging or reservations

2.2 Engine-level admission flags (vLLM reference)

Flag	Purpose	Typical Use
--max-num-seqs	Caps concurrent sequences in one forward pass (default 256)	Reduce to 64–128 to reserve KV blocks per sequence
--max-num-active-seqs	Bounds waiting→running transitions per cycle (v0.31+)	Prevent prefill flooding under burst
--max-num-batched-tokens	Caps total tokens/step (prefill chunks + decode queries)	Essential with --enable-chunked-prefill to protect ITL
--max-num-queued-reqs / --max-num-queued-tokens	API server hard ceiling on waiting requests/tokens	Triggers immediate HTTP 503; prevents CPU memory bloat

Verify flag names and behavior against your vLLM version

The flags above reflect vLLM 0.31+ / 0.33+ documentation and source. Flag names, defaults, and exact behavior change across versions. Before production use, run vllm serve --help on your pinned version and check the engine argument reference at https://docs.vllm.ai/en/stable/models/engine_args.html.

2.3 Memory-aware admission: preventing preemption storms

When gpu_cache_usage_perc → 1.0, the scheduler cannot allocate new KV blocks for active decodes. Under vLLM’s default recomputation preemption:

1. Victim sequences are selected and their KV blocks freed
2. Victims re-enter WAITING state
3. On re-admission, the engine re-runs prefill over the full consumed history (prompt + all generated tokens so far)
4. Recomputation consumes compute bandwidth, reducing effective μ, which grows the queue faster

Policy invariant: Admission must incorporate KV headroom:

If gpu_cache_usage_perc > τ_thresh (e.g. 0.85): defer or reject fresh prefill admissions

---

3. Cancellation, timeout hierarchy, and fallback cascades

3.1 End-to-end client cancellation propagation

Client drops socket (t = 2.0 s)
       │
       ▼ (TCP FIN / RST)
[ FastAPI / Starlette API Server ]
       │ Detects: await request.is_disconnected() == True
       ▼
[ AsyncLLMEngine ]
       │ Calls: await engine.abort(request_id)
       ▼
[ Iteration Scheduler ]
       │ Transition: Request status → ABORTED
       │ At next step boundary:
       │   - Remove sequence from Running batch
       │   - Decrement KV block reference counts
       │   - Free physical blocks back to PagedAttention pool
       ▼
GPU stops forward passes for request_id (Iteration t + 1)

Key implementation properties:

• Cancellation is non-preemptive relative to an in-flight CUDA kernel. Abort takes effect at the iteration boundary after the current forward step completes.
• Middleware caveat: Standard Starlette BaseHTTPMiddleware or custom buffering layers can swallow client disconnect signals by consuming the ASGI http.disconnect receive channel. Entrypoints must use pure ASGI middleware or raw request stream listeners to ensure disconnections trigger immediately.

3.2 Multi-tier timeout hierarchy (strictly ordered outer → inner)

[ Client Timeout: 5.0 s ]
      > [ API Gateway Timeout: 4.5 s ]
            > [ Queue Ingress Deadline: 1.0 s (TTFT SLO) ]

1. Queue Ingress Deadline: On enqueue, stamp deadline = now() + (SLO_TTFT - t_prefill_est). If request reaches queue head after now() > deadline, scheduler drops it without executing prefill.
2. API Gateway Timeout (Envoy/NGINX): Terminate at 4.5 s with HTTP 504 if server has not completed generation.
3. Client-Side Timeout: 5.0 s. Socket close propagates TCP FIN → engine abort.

3.3 Fallback and graceful degradation strategies

                       [ Overload Detected ]
                                 │
          ┌──────────────────────┼──────────────────────┐
          ▼                      ▼                      ▼
   [ Fast Reject ]        [ Model Cascade ]     [ Quality Degradation ]
   - HTTP 429             - Route to smaller     - Truncate max_tokens
   - Header: Retry-After    model (e.g. 7B→1B)   - Disable speculative decoding
   - Header: Queue-Wait   - Spillover to cloud   - Strip multi-shot examples

1. Fast Rejection (HTTP 429 Too Many Requests)

   • Returns immediately with:
     • Retry-After: 4 (seconds until expected queue clearance)
     • X-RateLimit-Reset: <timestamp>
     • X-Overload-Reason: queue_delay_exceeded
   • Why 429 over 503: 429 explicitly signals “functional but capacity-limited,” preventing aggressive unjittered retry storms that 503 can trigger.

2. Model Cascading / Speculative Routing

   • Route excess interactive traffic to a lightweight fallback model (7B → 1B, or to a quantized 3B / speculative draft model)
   • Or route to a pay-per-token cloud API endpoint (cloud spillover) to absorb transient bursts at variable cost

3. Execution Parameter Degradation

   • Clamp max_tokens: Dynamically reduce from 256 to 64 under heavy load → 4× fewer decode iterations per sequence → higher effective μ
   • Disable Speculative Verification: Under compute saturation, fall back to pure autoregressive decoding to shed overhead

4. Semantic / Prefix Cache Fallback

   • Return cached completions for common queries (FAQ prompts) without querying the GPU runner

---

4. Comparative policy behavior under burst

[Illustrative scenario] Same assumptions as opening burst: λ = 30 req/s, μ = 10 req/s, T_burst = 60 s. Figures are analytical derivations under stated assumptions, not measured results.

Policy outcomes under 60 s burst (λ=30, μ=10, client timeout=2.0 s, TTFT SLO=1.0 s)

Policy	Admitted	Shed
Naive FIFO (Unbounded)	30 req/s (100%)	0 req/s
FIFO Tail-Drop (Q<sub>max</sub>=32)	10.5 req/s	19.5 req/s
Predictive TTFT Admission	10.0 req/s	20.0 req/s
Model Cascade (to 1B)	10 (7B) + 20 (1B)	0 req/s

Goodput does not become zero merely because the queue is long

Goodput falls to zero when completed work is no longer useful—i.e., the client has disconnected or timed out before the response arrives. A long queue alone does not imply zero goodput if clients have long timeouts and cancellation propagates. The collapse occurs when queue wait > client timeout AND cancellation is missing or delayed.

4.1 Overload cost accounting

Cloud pricing basis: AWS EC2 On-Demand p4de.24xlarge (8× A100-80GB) = $40.96/hr → $5.12/hr/GPU = $0.00142/s/GPU (illustrative 2026 pricing).

• Scenario A: Naive FIFO overload (10 min burst)

  • GPU runs at 100% utilization
  • 90% of completed responses discarded by disconnected clients
  • Total GPU cost: 10 min × $0.0853 = $0.853
  • Wasted cost: $0.768/GPU → hundreds of dollars/day across an 8-GPU cluster with daily spikes

• Scenario B: Autoscaling lag vs. admission shedding

  • Cold-start 8-GPU node (container + 14 GB weights): 4–8 minutes
  • Transient burst (60–180 s) cannot be resolved by autoscaling
  • Rule: Autoscaling solves sustained macro-load shifts (hours); admission control and shedding solve micro-bursts (seconds to minutes)

---

5. Concrete decision exercise with observable evidence

5.1 Scenario & instantaneous state

An inference engineer monitors a single A100-80GB GPU running vLLM with LLaMA-7B-chat-hf under the mixed workload from Lesson 1.

• Measured sustained service capacity: μ̂ = 10.0 req/s
• Mean prefill duration for Short Chat (P = 32): t_prefill ≈ 15 ms = 0.015 s
• Target TTFT SLO: TTFT ≤ 1.0 s
• Client timeout: t_timeout = 2.0 s

At time t₀, Prometheus /metrics reports:

Metric	Value
vllm:num_requests_running	32
vllm:num_requests_waiting	45
vllm:gpu_cache_usage_perc	0.88 (88%)
vllm:num_preemptions_total	0

A new candidate request R₄₆ (Short Chat: P = 32, N = 64, client timeout 2.0 s) arrives at the HTTP gateway.

5.2 Step-by-step decision arithmetic

Step 1: Expected waiting time in queue (W_est)

Using Little’s Law / service rate moving average:

W_est = Q / μ̂ = 45 requests / 10.0 req/s = 4.50 seconds

Step 2: Expected Time-to-First-Token (TTFT_est)

TTFT_est = W_est + t_prefill = 4.50 s + 0.015 s = 4.515 seconds

Step 3: Evaluate against SLO and client timeout boundaries

1. TTFT_est (4.515 s) > SLO_TTFT (1.0 s): Violation by 3.515 s
2. TTFT_est (4.515 s) > t_timeout (2.0 s): Violation by 2.515 s

Step 4: Check memory headroom invariant

Current gpu_cache_usage_perc = 0.88 (> 0.85 safety threshold). Admitting new prefills increases memory pressure on 32 running decodes, risking KV cache exhaustion and preemption.

Admission Decision: REJECT IMMEDIATELY

• Return HTTP status 429 Too Many Requests
• Set response header: Retry-After: 5 (⌈W_est⌉)
• Set header: X-Overload-Reason: queue_delay_and_kv_saturation

5.3 Counterfactual: what happens if admitted?

If the controller naively admits R₄₆:

1. R₄₆ sits in the waiting queue
2. At t = t₀ + 2.0 s, client timeout fires; socket drops
3. If cancellation propagation is missing/delayed, R₄₆ reaches GPU at t = t₀ + 4.5 s
4. GPU spends 15 ms prefill + 64 decode passes (~64 × 36 ms ≈ 2.3 s GPU execution)
5. At t = t₀ + 6.8 s, engine finishes and hits BrokenPipeError on dead socket
6. Result: 2.315 s of GPU Tensor Core + HBM wasted, delaying 45 other requests behind it

5.4 Observable evidence & metric signatures

Metric	Prometheus Type	Healthy (λ=8)	Uncontrolled Overload (Naive FIFO)	Controlled Overload (Predictive Shedding)
vllm:num_requests_waiting	Gauge	0–4	Ramps > 500	Clamped ≤ 10
vllm:num_requests_running	Gauge	16–32	64–256 (pinned at max)	32–64 (optimal throughput)
vllm:request_queue_time_seconds	Histogram/Summary	p95 ≈ 0.05 s	p95 > 25.0 s	p95 ≤ 0.40 s (admitted)
vllm:gpu_cache_usage_perc	Gauge	0.40–0.70	1.00 (100% pinned)	0.75–0.85 (bounded)
vllm:num_preemptions_total	Counter	0	Rapidly climbing (>10/min)	0
http_requests_total{status="429"}	Counter	0	0 (all admitted, doomed)	Increases ≈ 20 req/s
client_goodput_rps	Gauge/Rate	≈ 8.0 req/s	Collapses to 0.0 req/s	Maintained ≈ 10.0 req/s

Metric name variants across vLLM versions

vLLM V1 uses kv_cache_usage_perc where V0 uses gpu_cache_usage_perc. The decision exercise uses the V0 name for consistency with the research brief; check your version’s /metrics endpoint before building alerts.

---

6. Retry storm mitigation: conditional and bounded guidance

When a client receives HTTP 429 with Retry-After, the retry behavior determines whether the system recovers or collapses further.

Dangerous pattern (unbounded retry storm):

# Client pseudocode: BAD
while True:
    response = await client.post("/v1/completions", json=payload)
    if response.status == 429:
        await asyncio.sleep(0.1)  # Fixed short sleep, no jitter
        continue
    break

Result: All rejected clients retry simultaneously → synchronized hammer → queue spikes again → sustained 429 loop.

Safer pattern (conditional, bounded, jittered):

# Client pseudocode: BETTER
max_retries = 3
base_delay = float(response.headers.get("Retry-After", "2"))
for attempt in range(max_retries):
    response = await client.post("/v1/completions", json=payload)
    if response.status != 429:
        break
    delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)  # Exponential backoff + jitter
    await asyncio.sleep(delay)
else:
    # All retries exhausted: fail gracefully, show user a "try again later" message
    raise OverloadError("Service temporarily unavailable")

Server-side amplification prevention:

• Include Retry-After in seconds, not milliseconds (coarser granularity reduces synchronization)
• Consider per-client token bucket at gateway layer to rate-limit retries independently of admission
• Log http_requests_total{status="429"} by client ID to detect abusive retry patterns

---

7. One-page Overload and Failure Policy

Complete this before deploying to production. Each field must reference a measurement, physical bound, or a decision from Lessons 14–15.

OVERLOAD AND FAILURE POLICY — inference-optimisation/lesson-16
===============================================================

Model:              llama-2-7b-chat BF16 (MHA, 32 layers, H_kv=32, d_head=128)
Weight footprint:   ~14 GB (fits one A100-80GB: YES)
KV bytes/token:     512 KiB  [2 × 32 × 32 × 128 × 2]
Max prompt P:       512 tokens → 256 MiB KV
Workload:           λ_base = 8 req/s (Poisson), λ_burst = 30 req/s for ≤ 60 s
Single-GPU μ:       ~10 req/s sustainably under SLO (measured: ______)
SLOs (client-observed):
  p95 TTFT ≤ 1.0 s    p95 TPOT ≤ 35 ms    p99 E2E ≤ 5.0 s    Goodput ≥ 99.9%

CAPACITY BASELINE (from Lesson 14):
  Replicas:           R = 4 (capacity-planning decision)
  Single-replica μ:   10 req/s
  Aggregate μ:        40 req/s (4× headroom over λ_burst)
  Binding SLO:        TTFT / TPOT / neither  (measured: ______)

TOPOLOGY (from Lesson 15):
  Selected:           [ ] Replicas (R=4)  [ ] TP=2 pairs  [ ] P/D disaggregation
  Interconnect:       NVLink 3 / 100 Gbps RDMA / 10 Gbps  (measured NIC: ______ GB/s)
  KV transfer floor:  NVLink <1 ms  |  100 Gbps ~21.5 ms  |  10 Gbps ~215 ms

ADMISSION CONTROL:
  Engine flags (pinned vLLM version: ______):
    --max-num-seqs = ______           --max-num-active-seqs = ______
    --max-num-batched-tokens = ______ --enable-chunked-prefill = ______
    --max-num-queued-reqs = ______    --max-num-queued-tokens = ______
  Admission checks (gateway → engine):
    1. Token bucket (per-tenant): rate = ______, burst = ______
    2. Predictive TTFT: W_est = Q / μ̂; if W_est + t_prefill > 1.0 s → HTTP 429
    3. KV headroom: if gpu_cache_usage_perc > 0.85 → DEFER/REJECT
    4. Queue ingress deadline: stamp deadline = now() + (1.0 s - t_prefill_est)

CANCELLATION PROPAGATION:
  [ ] ASGI middleware preserves http.disconnect (no buffering middleware)
  [ ] API server calls await request.is_disconnected() in streaming loop
  [ ] On True: await engine.abort(request_id) at iteration boundary
  [ ] KV blocks freed to PagedAttention pool at next step boundary

TIMEOUT HIERARCHY (strictly ordered):
  Client timeout:              5.0 s
  API Gateway (Envoy/NGINX):   4.5 s → HTTP 504
  Queue ingress deadline:      1.0 s (TTFT SLO) → drop at queue head

FALLBACK CASCADE (triggered by admission rejection or overload metrics):
  1. FAST REJECT (primary):
       HTTP 429 + Retry-After: ⌈W_est⌉ + X-Overload-Reason: queue_delay_exceeded
  2. MODEL CASCADE (if deployed):
       Route excess → 1B model / quantized 3B / cloud spillover endpoint
       Measured fallback TTFT: ______ ms   Quality delta: ______ (eval set)
  3. QUALITY DEGRADATION (dynamic):
       Clamp max_tokens: 256 → 64  |  Disable speculative decoding  |  Strip few-shot
  4. SEMANTIC CACHE (if available):
       Return cached FAQ completions → 0 GPU cost

RETRY GUIDANCE (documented for clients):
  Max retries: 3  |  Base delay: Retry-After header  |  Exponential backoff + jitter
  Server logs 429 by client ID; per-client token bucket at gateway: ______ req/s

OBSERVABILITY THRESHOLDS (alerting rules):
  █ num_requests_waiting > 10 for 30 s          → Page on-call
  █ gpu_cache_usage_perc > 0.90 for 60 s        → Page on-call
  █ num_preemptions_total rate > 1/min          → Page on-call
  █ http_requests_total{status="429"} rate > 5/s → Page on-call
  █ client_goodput_rps < 9.0 (90% of μ) for 60 s → Page on-call

COST ACCOUNTING (from Lesson 14 capacity plan):
  GPU cost: $______/hr/GPU  |  Wasted cost under naive FIFO (10 min burst): $______
  Fallback cloud spillover: $______/1K tokens  |  Budget for burst absorption: $______/day

VALIDATION CHECKLIST (must be checked before production traffic):
  [ ] Replica baseline measured: p95 TTFT ______ ms, p95 TPOT ______ ms, Goodput ______%
  [ ] Chunked-prefill tuned baseline measured: does it satisfy SLO without multi-GPU?
  [ ] Admission control rejects at W_est > 1.0 s in load test (locust/k6)
  [ ] Client disconnect → engine.abort() → KV blocks freed (verified in load test)
  [ ] Fallback cascade tested: 429 → model cascade → quality degradation → cache
  [ ] Retry storm simulated: 100 clients × 3 retries with jitter → no synchronized spike
  [ ] Cost model validated against one week of production traffic replay

Approval sign-off: ____________________  Date: __________

---

8. Key Takeaways

Key takeaways

• An unbounded FIFO queue under overload does not just increase latency—it destroys goodput by filling the GPU with work the clients have already abandoned.
• Admission control must reject before queue wait exceeds the client timeout: estimate W_est = Q / μ̂ and compare against SLO_TTFT and t_timeout.
• vLLM flags (--max-num-seqs, --max-num-active-seqs, --max-num-batched-tokens, --max-num-queued-reqs) bound the engine; configure them from measured capacity, not defaults.
• KV cache headroom (gpu_cache_usage_perc > 0.85) must gate prefill admission to prevent preemption storms that multiply compute waste.
• Cancellation must propagate end-to-end: client socket drop → ASGI disconnect → engine.abort() → KV blocks freed at iteration boundary.
• Use HTTP 429 with Retry-After, not 503, to signal "functional but full" and prevent unbounded retry storms.
• Fallback cascade: Fast Reject (429) → Model Cascade → Quality Degradation (max_tokens clamp) → Semantic Cache.
• Autoscaling solves sustained load (hours); admission control and shedding solve micro-bursts (seconds to minutes).
• File the one-page Overload and Failure Policy alongside the Capacity Plan (Lesson 14) and Topology Decision Record (Lesson 15).

---

9. Exercises

Try it yourself

1. 01A single GPU sustains μ = 12 req/s. A burst arrives at λ = 36 req/s for 15 s. Assuming FIFO unbounded, calculate: (a) queue depth at t = 15 s, (b) W_est for a request arriving at t = 15 s, (c) whether this request meets a 1.0 s TTFT SLO if t_prefill = 20 ms. State the admission decision and the HTTP response you would return.

   Show a hint

   (a) Q = (λ − μ) × t = 24 × 15 = 360. (b) W_est = 360 / 12 = 30 s. (c) TTFT_est = 30.02 s > 1.0 s SLO by 29.02 s. Decision: REJECT with HTTP 429, Retry-After: 30, X-Overload-Reason: queue_delay_exceeded.

2. 02At the moment of admission check, Prometheus shows: num_requests_waiting = 60, num_requests_running = 40, gpu_cache_usage_perc = 0.82. Measured μ̂ = 10 req/s, t_prefill = 15 ms, SLO_TTFT = 1.0 s. The candidate request is a Medium Coding prompt (P = 128, t_prefill_est = 60 ms). Calculate W_est and TTFT_est. Does the KV headroom check pass? What is the admission decision?

   Show a hint

   W_est = 60 / 10 = 6.0 s. TTFT_est = 6.0 + 0.06 = 6.06 s > 1.0 s SLO (violation 5.06 s). KV headroom: 0.82 < 0.85 → passes. But TTFT violation dominates → REJECT with HTTP 429.

3. 03A client receives HTTP 429 with Retry-After: 4. The client implements a fixed 100 ms retry loop (no backoff, no jitter, no max retries). Explain why this creates a retry storm, and write a corrected retry snippet with exponential backoff, jitter, and a max-retry cap of 3.

   Show a hint

   Fixed short retries synchronize all rejected clients, hammering the admission endpoint simultaneously → queue spikes again → sustained 429 loop. Corrected: loop up to 3×, delay = Retry-After × 2^attempt + random(0, 0.5 s).

4. 04In the decision exercise (§5), if cancellation propagation were missing and R₄₆ were admitted, the GPU would waste ~2.3 s generating tokens for a dead socket. At $5.12/hr/GPU, what is the wasted cost of that single doomed request? If this happens to 200 requests during a burst, what is the total wasted GPU cost?

   Show a hint

   Cost per second = $5.12 / 3600 = $0.001422/s. Single request waste = 2.315 s × $0.001422 = $0.00329. 200 requests = $0.658. Across an 8-GPU cluster with daily bursts, this compounds to hundreds of dollars/day.

5. 05Fill in the Overload and Failure Policy template (§7) for your own deployment. For each field marked "measured: ______", state what measurement you would run, what tool you would use, and what threshold would make you change the policy.

   Show a hint

   Example: "Single-replica μ: measured with k6 at concurrency 32, 70/20/10 workload, 30 min steady state. If μ < 8 req/s, reduce R or enable chunked prefill."

---

10. Sources and scope

Claim / Topic	Primary Source	Scope and Key Findings
Iteration-Level Scheduling & Preemption	Yu et al., Orca, OSDI 2022	§3 “Iteration-Level Scheduling”, §4.2 “Selective Preemption”. Formalizes step-by-step admission and preemption.
Preemption: Recomputation vs Host Swapping	vLLM Configuration & Optimization Docs	Default in vLLM V1 is recomputation; host swapping requires explicit --swap-space.
Engine Flags: --max-num-seqs, --max-num-batched-tokens	vLLM Engine Args Reference	Controls batch sequence limit and token limit across prefill/decode.
--max-num-active-seqs Admission Decoupling	vLLM Release v0.31.0 & PR discussions	vllm/engine/arg_utils.py. Decouples model runner batching from queue admission.
API Server Queue Limits (--max-num-queued-reqs)	vLLM OpenAI API Server CLI	vllm/entrypoints/openai/cli_args.py. Triggers early HTTP 503 when queue length exceeds limit.
Client Disconnect Detection & Abort	vLLM API Server (api_server.py)	request.is_disconnected() → engine.abort(request_id). Non-blocking detection via Starlette.
Prometheus Metrics	vLLM Production Metrics Docs	Metrics exposed on /metrics. Note: gpu_cache_usage_perc (V0) vs kv_cache_usage_perc (V1).
Doomed RPCs and Load Shedding Theory	Google SRE Book (Beyer et al., 2016)	Chapter 21: “Handling Overload”, §“Load Shedding and Graceful Degradation”.
A100-80GB Pricing ($5.12/hr/GPU)	AWS EC2 On-Demand Pricing (2026-11)	p4de.24xlarge ($40.96/hr for 8 GPUs). Consistent with Track 2 Capstone Spec §7.3.

Classification of statements in this lesson

• Verified Primary Facts: Continuous batching mechanics, vLLM CLI arguments (flag names verified in source/docs), Prometheus metric names, socket disconnect mechanics, SRE load shedding principles.
• Implementation-Specific Behavior: vLLM V0 vs V1 metric naming (gpu_cache_usage_perc vs kv_cache_usage_perc); Starlette middleware ASGI disconnect consumption caveat; exact preemption behavior (recompute default).
• Analytical Calculations: Queue wait W_est = Q / μ̂; burst accumulation Q(t) = ∫(λ − μ)dt; dollar cost derivations from illustrative pricing.
• Teaching Assumptions (Explicitly Labeled): The 3-class workload distribution (70/20/10), arrival rates (λ = 30 req/s), service rate (μ = 10 req/s), prefill durations, and timeout values are illustrative test workloads from the Track 2 Capstone spec, not measurements of a specific production system.

---

11. Unresolved evidence & verification gaps

The following items must be verified on your pinned vLLM version and target hardware before production use:

1. vLLM flag behavior: Run vllm serve --help on your version; confirm --max-num-active-seqs, --max-num-queued-reqs, --max-num-queued-tokens exist and behave as described.
2. Cancellation propagation: Verify request.is_disconnected() triggers engine.abort() in your entrypoint (custom middleware can suppress the ASGI disconnect signal).
3. Metric names: Scrape /metrics on your version; confirm gpu_cache_usage_perc (V0) or kv_cache_usage_perc (V1); confirm num_requests_waiting, num_requests_running, num_preemptions_total exist.
4. HTTP 429 vs 503 behavior: Test that --max-num-queued-reqs returns 503, while predictive admission returns 429 with Retry-After header.
5. Goodput under controlled overload: Run a load test (k6/locust) with predictive admission enabled; verify admitted goodput ≈ μ and rejected requests return 429 instantly.
6. Fallback cascade integration: If model cascade is deployed, measure fallback TTFT and quality delta on a held-out eval set.
7. Cost model: Validate cloud pricing and GPU utilization against your actual billing and monitoring.

No hardware benchmarks were run for this lesson

All timing numbers, throughput figures, and cost calculations are analytical derivations under explicitly stated assumptions. They are teaching tools, not measured results. Before relying on any number for capacity procurement, run your own measurements on target hardware under synthetic traffic matching your workload profile.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Build a Serving SystemCapacity Planning: From Traffic Profile to GPU Count18 min
• Inference Optimisation & AI Infrastructure · Build a Serving SystemScale Beyond One GPU: Replication, Tensor Parallelism, and Prefill/Decode Disaggregation16 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryContinuous Batching: Iteration-Level Scheduling30 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryPrefill Optimisation: Prefix Caching, Chunked Prefill, and Disaggregation35 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceServing Workloads and Performance Metrics12 min