Calibration Data and Quality Regression

The one-sentence version

Post-training quantization without activation calibration treats all weights equally, but activation outliers amplify weight rounding errors by orders of magnitude; representative calibration data reveals channel salience so algorithms can protect sensitive weights, and a defensible release decision requires task-specific quality thresholds—not universal rules.

---

1. Why Post-Training Quantization Requires Calibration

1.1 Weight-Space Error vs. Output-Space Error

Minimizing the weight-space Frobenius error ||W - Ŵ||_F² does not prevent task degradation. What matters for model quality is the layer output error:

L(W, Ŵ) = || X W - X Ŵ ||_F² = || X E ||_F² = tr( E^T X^T X E )

Where:

• X ∈ ℝ^{N × d_in} — calibration activations (N samples, d_in input channels)
• W ∈ ℝ^{d_in × d_out} — weight matrix
• E = W - Ŵ ∈ ℝ^{d_in × d_out} — weight quantization error
• tr(·) — matrix trace (sum of diagonal elements)

The trace is required because E^T X^T X E is a d_out × d_out matrix, while the loss ||XE||_F² is a scalar. For a single output j, the loss is L_j = ||X e_j||² where e_j = W_{:,j} - Ŵ_{:,j} is the error on column j of W.

X: [N, d_in] · W: [d_in, d_out] → Y: [N, d_out]

• Input channel k indexes row k of W (W[k, :])
• Activation channel k (X[:, k]) multiplies weight row k
• Weight error row k (E[k, :]) is the error on the weights for input channel k

1.2 The Hessian of the Local Quadratic Objective

For a single output j, the output error is:

L_j = || X w_j - y_j ||² = w_j^T X^T X w_j - 2 y_j^T X w_j + y_j^T y_j

where w_j = W[:,j] is column j of the weight matrix.

The Hessian with respect to w_j is:

H_j = ∂²L_j / ∂w_j² = 2 X^T X

This is the Hessian for one output’s weight vector (column j) — not “the Hessian of layer activations.” It is a d_in × d_in matrix that depends on the calibration activations X. For the full multi-output loss L = ∑_j L_j, the block-diagonal Hessian has 2 X^T X repeated for each output column.

1.3 Activation Salience Amplification (Why Calibration Matters)

Because L = tr(E^T X^T X E), an activation channel k with large magnitude (large X[:,k]) amplifies any weight error on row k of W by X[:,k]². Conversely, channels with near-zero activation magnitude produce negligible output distortion even under coarse weight rounding.

Calibration data D_calib reveals channel salience:

s_k = (1/N) ∑_n |X_{n,k}|  =  E[|X_k|]

This is the first absolute moment of activations — not a second-order statistic. Algorithms like AWQ use s_k to protect salient weights from aggressive low-bit rounding.

What calibration contributes

Weight quantization in isolation (naive Round-To-Nearest) treats all weight elements equally, ignoring how activations scale their impact. Without representative calibration activations, an inference engine cannot determine which weights require precision protection and which can be aggressively compressed. Local layer loss ||X(W-Ŵ)||_F² is a proxy for task quality — not its definition. Downstream task accuracy (coding, reasoning, format compliance) must be measured separately.

---

2. AWQ-Inspired Diagonal Rescaling: The Equivalence Principle

2.1 The Core Insight (Lin et al. §3.3)

For any positive diagonal matrix D ∈ ℝ^{d_in × d_in} (all D_{kk} > 0):

X' = X D⁻¹
W' = D W

Then exactly in floating-point arithmetic:

X' W' = (X D⁻¹) (D W) = X W

This means we can rescale the problem before quantization: shrink outlier activation channels in X, grow their corresponding weight rows in W, quantize W' fully in integer, then de-rescale. The key is that D is chosen from calibration salience s_k — typically D_{kk} = √s_k for salient channels, so X'_{:,k} = X_{:,k} / √s_k (smaller activations) and W'_{k,:} = W_{k,:} · √s_k (larger weights).

No mixed precision is involved. Real AWQ (Lin et al. §3.3–§3.4) pairs per-channel activation-aware scaling with group-wise integer quantization (e.g., group size 128), selects the per-channel scaling exponent α by grid search over [0, 1] (§3.4), and optimizes weight clipping thresholds to minimize output reconstruction error. The diagonal rescaling X' = XD⁻¹, W' = DW preserves XW exactly in floating-point arithmetic (§3.3); quantization happens after rescaling on the integer grid. No high-precision FP weights are retained at inference time.

2.2 Verified Equivalence Demonstration

In our bounded demo (scripts/verify-calibration-example.py Part 2), we construct activations with clear outlier channels (salient channels ~20–30× larger than normal), compute D from salience (D_{kk} = √s_k), verify X'W' = XW in FP (max diff ~10⁻¹⁵), then quantize W' per-channel and de-rescale.

Method	Output SQNR
Naive per-channel quantization (no rescaling)	26.44 dB
AWQ-inspired rescaling + quantization	26.44 dB (equivalence — no gain under per-row quantization)

Equivalence demonstration, not a measured AWQ gain

The diagonal rescaling X' = XD⁻¹, W' = DW preserves XW exactly in FP. When we quantize W' per-channel and de-rescale, the output SQNR equals naive quantization because per-row quantization is scale-equivariant: scaling a weight row and its corresponding activation channel by inverse factors leaves the quantization error unchanged. Real AWQ avoids this by using group-wise quantization where a single scale is shared across multiple input channels, so the rescaling does not cancel. This demo shows the FP equivalence principle; the AWQ gain comes from group-wise quantization, not from the rescaling alone.

2.3 Limits of the Rescaling Approach

• Cancellation under per-row output scaling: If the quantization scheme applies an equivalent per-row scale to the output (as per-channel quantization does), the rescaling cancels out. AWQ avoids this by using group-wise quantization where a single scale is shared across multiple input channels.
• Group quantization: Real AWQ pairs per-channel activation scaling with group-wise weight quantization (typically group size 128) and performs a grid search over the scaling exponent α in [0, 1] (§3.4). Our demo uses per-channel scaling and per-channel quantization to isolate the exact FP equivalence property.
• Scale selection: D_{kk} = √s_k is one heuristic; the optimal scale depends on the quantization grid and grouping. AWQ searches the exponent α space over calibration activations (§3.4).

---

3. The Clipping vs. Rounding Trade-Off

3.1 Mathematical Decomposition

When quantizing a continuous distribution (e.g., Gaussian weights) to a low-bit symmetric grid (INT4: q ∈ [-7, 7]), the scale is:

S(α) = t / q_max = (α · max|W|) / q_max

where t = α · max|W| is the absolute clipping threshold and α ∈ (0, 1] is a dimensionless fraction.

Total reconstruction error decomposes into two opposing forces:

Total MSE(α) = Clipping MSE(α) + Rounding MSE(α)

1. Clipping Error: Values |w| > t are saturated to ±q_max · S. This error increases as α decreases (threshold tightens).
2. Rounding Noise: Values within [-t, t] suffer uniform quantization noise. For a uniform quantizer, the expected rounding noise variance is ≈ S² / 12 (standard textbook model, e.g., Jacob et al. §2.2). As α decreases, S shrinks, dramatically lowering rounding noise for the central ~99% of weights.

3.2 Reproducible Grid Search on Supplied Data

In our bounded demo (scripts/verify-calibration-example.py Part 3), we run a grid search over α ∈ [0.10, 1.00] on a supplied weight tensor and report the seeded local optimum:

Metric	AbsMax (α=1.00)	Optimal (α=0.77)
Threshold t	1.6651	1.2822
Scale S	0.2379	0.1832
MSE	0.004716	0.003054
SQNR	17.19 dB	19.07 dB
MSE reduction	—	35.24%
SQNR gain	—	+1.89 dB

Seeded local optimum, not universal rule

The optimum α* ≈ 0.77 here is a seeded local minimum on the supplied weight distribution. It is not a universal “63% rule” — the Pareto trade-off depends on the weight distribution tail, sample maximum, and search grid. The S²/12 model assumes uniform quantization noise for unclipped values; real distributions deviate. Always grid-search on your actual weights.

---

4. Calibration Dataset Design & Pitfalls

4.1 What to Sample: Vectors, Tokens, Sequences

Calibration operates on activations X ∈ ℝ^{N × d_in}. Each row is one forward pass through the layer. Critical distinctions:

Unit	Description	Typical size
Vector	Single activation vector X[n, :] (one token position, all channels)	d_in
Token	One position in a sequence; produces one vector	1
Sequence	Full forward pass through the model (many tokens)	512–8192 tokens
Sample	One sequence’s activations aggregated (e.g., mean abs(X) per channel)	1

Calibration set size N counts sequences (or aggregated samples), not tokens. GPTQ (§4) used 128 sequences × 2048 tokens from C4 — one specific experiment, not a universal prescription.

4.2 Sample Size Convergence (Synthetic Demonstration)

In our bounded demo (Part 4), we estimate E[|X|] from N synthetic Gaussian samples per channel. Labeled synthetic — tokens are not i.i.d., CLT assumes independence.

Sample Size N	Mean Relative Error	Note
16	19.61%	High variance
32	8.07%	—
64	9.99%	Variance in synthetic demo
128	6.27%	GPTQ/AWQ typical choice
256	4.98%	Diminishing returns
512	3.07%	Asymptote

Error drop 16→128: 13.34% pts. Error drop 128→512: 3.20% pts (diminishing).

Not a universal sweet spot

AWQ §5.3 reports robustness to calibration distribution changes (WikiText2 vs C4). Generic text is not universally catastrophic — the impact depends on the model, algorithm, and downstream task. Do not prescribe “128–512 sequences” or “2048 tokens” as universal rules. Calibrate on your deployment workload and measure.

4.3 Domain Shift: Out-of-Distribution Calibration (Synthetic Mixed-Precision Salience Toy)

In our bounded demo (Part 5), we simulate a target domain (code/JSON) with persistent outlier channels [2, 9], and an OOD domain (generic prose) without them. Labeled synthetic — this is a mixed-precision salience toy, not AWQ. The demo protects salient channels by keeping their weights at full Python-float precision while quantizing others; this is not how AWQ works (AWQ uses full integer quantization with group-wise scaling).

Calibration Domain	Top Channels Identified	Output SQNR
In-domain (code)	[2, 9] ✅	44.33 dB
Out-of-domain (prose)	[4, 6] ❌	21.23 dB
Advantage	—	+23.10 dB

Synthetic mixed-precision toy, not AWQ or model evidence

This demo protects channels by copying weights at full Python-float precision — a synthetic mixed-precision salience toy. It does not implement AWQ (which uses full integer quantization with group-wise scaling). The +23.10 dB advantage is a local synthetic result on seeded Gaussian vectors, not a paper finding or measured task-accuracy loss. AWQ §5.3 shows robustness to calibration distribution shifts in practice; generic text is not universally catastrophic. Real domain shift requires actual model/dataset evaluation.

4.4 Data Splits: Calibration / Tuning / Held-Out Test

Never evaluate on calibration data. Use three disjoint splits:

Split	Purpose	Size guideline
Calibration	Estimate salience s_k, find optimal α	128–512 sequences (algorithm-dependent)
Tuning / Validation	Select hyperparameters (group size, grid resolution)	Separate from calibration
Held-Out Test	Final quality gate (PPL, functional, format)	Never used for any selection

---

5. Strategy Comparison: Single Experiment, Multiple Metrics

In our bounded demo (Part 6), we compare strategies on the same input X, same weights W, same quantizer (INT4, q_max=7):

Strategy	Weight SQNR	Output SQNR
AbsMax	16.47 dB	16.32 dB
Percentile 99.9%	16.47 dB	16.32 dB
MSE Grid Search (α=0.71)	18.67 dB	18.82 dB
AWQ-Inspired Rescaling + Grid (α=0.71)	18.67 dB	18.86 dB

No untraced numbers, no universal rankings; last row is a global-scale toy

The previous lesson version reported “21.50 dB” with no implementation, “production grade” labels without evidence, and mixed weight/output SQNR from different experiments. Compare only with matching input, baseline, quantizer scope, and metric. Keep separate: - Weight SQNR: reconstruction fidelity of W - Output SQNR: layer output fidelity (what matters for quality)

• Perplexity: language modeling on held-out corpus (requires real model) - Functional correctness: HumanEval/MBPP pass@1 (test pass rate, not indentation) - Format compliance: JSON schema valid rate

The “AWQ-Inspired Rescaling + Grid” row uses per-channel rescaling with a grid search on the rescaled weights — a global-scale toy, not implementation-faithful AWQ. Real AWQ uses group-wise quantization where a single scale is shared across multiple input channels (group size typically 128), so the diagonal rescaling does not cancel. The 0.04 dB difference here is within numerical noise; do not declare a universal ranking.

---

6. Complete Release Decision

A defensible deployment gate requires explicitly declared assumptions and three-way decision branches.

6.1 Declared Assumptions (Illustrative Example)

Category	Declared Value (Illustrative)
Baseline model	LLaMA-2-7B, FP16, vLLM 0.5.x
Quantization	INT4 group-wise (group size 128, symmetric INT4, q_max=7)
Calibration set	128 sequences from production code logs, 2048 tokens each
Tuning set	64 sequences (disjoint) for grid resolution selection
Held-out test	500 sequences from production workload (never seen)
Workload slices	Short chat (70%), medium coding (20%), long JSON (10%)
PPL threshold	ΔPPL ≤ 0.15 on held-out test (task-specific!)
Functional threshold	HumanEval pass@1 ≥ 0.85 (test pass rate)
Format threshold	JSON schema valid rate ≥ 0.95
Latency budget	p95 TTFT ≤ 1.0s, p95 TPOT ≤ 150ms
Memory budget	≤ 40 GB total allocation (target hardware budget)

6.2 Decision Function

A production deployment gate must evaluate measured candidate metrics against declared hard requirements and differentiate between actual failures and missing information:

• REJECT: Any measured hard constraint violation. If measured memory exceeds the target hardware budget (memory_gb > memory_budget_gb), measured latency exceeds the SLO ceiling (latency_ms > latency_budget_ms), or measured quality falls below declared thresholds (functional_pass_rate < functional_threshold, ppl_delta > ppl_threshold, format_valid_rate < format_threshold), the candidate is immediately rejected. A failing candidate must never be labeled “insufficient evidence” or permitted to deploy.
• INSUFFICIENT EVIDENCE: Genuinely missing or inconclusive measurements. If no constraint is known to be violated, but required benchmark suites have not completed, key workload slices (e.g., long-context JSON) are untested, or sample sizes are too small to establish statistical confidence, the deployment is held pending additional measurement.
• ACCEPT: All declared constraints are evaluated and pass.

def release_decision(metrics: dict, constraints: dict) -> dict:
    """
    Evaluates deployment readiness against declared task constraints.

    Decision Rule:
    - REJECT: Any measured metric violates a declared hard constraint (hardware
      memory budget, latency SLO, or minimum quality threshold).
    - INSUFFICIENT EVIDENCE: Measurements are missing, incomplete, or inconclusive.
    - ACCEPT: All declared constraints are measured and satisfied.
    """
    # 1. Measured hard constraint violations -> REJECT
    # A measured failure of any declared requirement is an immediate REJECT.
    if metrics.get('memory_gb') is not None and metrics['memory_gb'] > constraints['memory_budget_gb']:
        return {
            'decision': 'REJECT',
            'reason': f"Memory footprint ({metrics['memory_gb']} GB) exceeds hard budget ({constraints['memory_budget_gb']} GB) -> OOM risk",
            'rollback': True,
        }

    if metrics.get('latency_ms') is not None and metrics['latency_ms'] > constraints['latency_budget_ms']:
        return {
            'decision': 'REJECT',
            'reason': f"Latency ({metrics['latency_ms']} ms) exceeds budget ({constraints['latency_budget_ms']} ms) -> SLO breach",
            'rollback': True,
        }

    if metrics.get('functional_pass_rate') is not None and metrics['functional_pass_rate'] < constraints['functional_threshold']:
        return {
            'decision': 'REJECT',
            'reason': f"Functional pass rate ({metrics['functional_pass_rate']:.2%}) below threshold ({constraints['functional_threshold']:.2%})",
            'rollback': True,
        }

    if metrics.get('ppl_delta') is not None and metrics['ppl_delta'] > constraints['ppl_threshold']:
        return {
            'decision': 'REJECT',
            'reason': f"Perplexity degradation ({metrics['ppl_delta']:.4f}) exceeds threshold ({constraints['ppl_threshold']:.4f})",
            'rollback': True,
        }

    if metrics.get('format_valid_rate') is not None and metrics['format_valid_rate'] < constraints['format_threshold']:
        return {
            'decision': 'REJECT',
            'reason': f"Format valid rate ({metrics['format_valid_rate']:.2%}) below threshold ({constraints['format_threshold']:.2%})",
            'rollback': True,
        }

    # 2. Check for missing or inconclusive measurements -> INSUFFICIENT EVIDENCE
    required_keys = ['memory_gb', 'latency_ms', 'functional_pass_rate', 'ppl_delta', 'format_valid_rate']
    missing = [k for k in required_keys if metrics.get(k) is None]
    if missing or not metrics.get('evaluation_complete', True):
        return {
            'decision': 'INSUFFICIENT_EVIDENCE',
            'reason': f"Evaluation incomplete; missing or inconclusive measurements for: {missing}",
            'rollback': False,
        }

    # 3. All declared constraints pass -> ACCEPT
    return {
        'decision': 'ACCEPT',
        'reason': "All declared quality, latency, format, and memory constraints satisfied",
        'rollback': False,
    }

6.3 Decision Outcomes

Outcome	Condition	Action
ACCEPT	All declared quality, serving, and resource constraints satisfied	Deploy via staged canary release with error monitoring
REJECT	Any measured hard constraint violated (memory OOM, SLO breach, quality)	Rollback to original baseline weights — never deploy
INSUFFICIENT EVIDENCE	Measurements missing, test slices incomplete, or sample size too small	Hold release; complete required evaluations on held-out

Original baseline always preserved; thresholds are task-specific

A candidate that fails a hard memory or quality threshold is REJECTED, not held in limbo as insufficient evidence. If a candidate consumes 42 GB against a 40 GB budget, it violates a hard physical constraint and will trigger Out-Of-Memory errors on the target hardware. The original FP16/BF16 baseline weights must always be preserved for instant rollback (assuming the unquantized model fits the target deployment, or fall back to an existing serving pool).

Furthermore, quality thresholds are task-specific: a coding copilot may demand HumanEval pass@1 ≥ 0.85 and zero syntax regression, whereas a search summarizer may prioritize latency and accept slight stylistic variation. Universal rules like “ΔPPL ≤ 0.1–0.3” do not exist in production—every threshold must be explicitly justified against deployment SLOs.

---

Key Takeaways

Key takeaways

• Weight-space error ||W-Ŵ||_F² does not predict task quality; output distortion ||X(W-Ŵ)||_F² = tr(E^T X^T X E) is the local proxy that calibration addresses.
• The Hessian w.r.t. one output's weight column is 2 X^T X; input channel k indexes ROW k of W (not column).
• AWQ uses diagonal rescaling X=XD⁻¹, W=DW to preserve XW in FP, then quantizes W' fully in integer with group-wise scaling — no mixed precision. The scaling exponent α is searched over [0,1]; clipping is separate.
• Clipping threshold t = α_clip·max|W|; grid search on supplied data yields a seeded local optimum (α*_clip≈0.77 here), not a universal 63% rule. Distinguish α_clip (this lesson) from α_AWQ (AWQ scaling exponent).
• Channel salience s = E[abs(X)] is a first absolute moment; sample-size convergence (128→512) shows diminishing returns in synthetic demo.
• Domain shift impact is algorithm/model dependent; AWQ §5.3 reports robustness — do not promise universal catastrophic OOD failure. The Part 5 demo is a synthetic mixed-precision salience toy, not AWQ.
• Compare strategies on matching input/baseline/quantizer/metric; keep weight SQNR, output SQNR, perplexity, functional, and format separate. The AWQ-inspired row in Part 6 is a global-scale toy, not implementation-faithful AWQ.
• A defensible release decision evaluates declared task constraints: any measured hard constraint violation (memory OOM, latency SLO breach, quality failure) is an immediate REJECT; genuinely missing or inconclusive evaluation is INSUFFICIENT EVIDENCE; all constraints passing is ACCEPT. Always preserve original baseline weights for rollback.

---

Exercises

Try it yourself

1. 01Given X [N, d_in], W [d_in, d_out], E = W-Ŵ, derive the scalar loss L = ||XE||_F² as a trace. Show that for a single output j, the Hessian w.r.t. W[:,j] is 2 X^T X. Verify with d_in=3, d_out=2, N=4 using the helper script.

   Show a hint

   L = tr(E^T X^T X E). For output j: L_j = ||X e_j||² where e_j = E[:,j]. Hessian = 2 X^T X. Run trace_output_distortion() in the helper.

2. 02Explain why the diagonal rescaling X'=XD⁻¹, W'=DW preserves XW exactly in floating point. Then describe what happens when you quantize W' (not W) and de-rescale. Why is this NOT mixed precision?

   Show a hint

   X'W' = X D⁻¹ D W = X W. Quantize W' with scales derived from W', then W_hat = D⁻¹ W'_q. All weights remain quantized on the integer grid — no FP weights are retained. Run verify_awq_rescaling_equivalence().

3. 03Explain why knowing only max|W| = 2.5 is sufficient to compute the AbsMax scale S, but mathematically insufficient to determine the optimal clipping threshold α* without the underlying weight distribution. Then, using the 512-element Gaussian weight tensor supplied in the helper script (where max|W| ≈ 1.6651), report the AbsMax scale, the grid-search optimal α*, and the resulting MSE reduction. Why is this α* not a universal constant?

   Show a hint

   AbsMax requires only the peak: S = max|W| / q_max = 2.5 / 7 ≈ 0.357. But α* minimizes total MSE (clipping + rounding), which depends on the distribution shape and tail density. Running grid_search_clipping_threshold() on the helper’s seeded Gaussian weights yields α* ≈ 0.77 (35.24% MSE reduction, +1.89 dB SQNR gain), which is a local minimum specific to that sample distribution, not a universal rule.

4. 04Design a calibration dataset for a production LLM serving structured JSON generation. Specify: (a) how you separate calibration/tuning/held-out splits, (b) what "sample size" means (sequences vs tokens), (c) why you would not assume 128 sequences × 2048 tokens is universally optimal.

   Show a hint

   Calibration: production JSON logs. Tuning: disjoint slice for hyperparameters. Held-out: never-used test set. Sample size = sequences. GPTQ used 128×2048 C4 (§4) — one experiment; AWQ §5.3 shows robustness across domains.

5. 05A candidate quantized model evaluated on held-out test data shows ΔPPL=0.08 (threshold ≤ 0.15), HumanEval pass@1=0.78 (declared threshold ≥ 0.85), JSON valid rate=0.97 (threshold ≥ 0.95), latency=480ms (budget ≤ 500ms), and memory footprint=42 GB (hardware budget ≤ 40 GB on target device). What is the release decision under the three-way decision rule, and why?

   Show a hint

   The candidate violates two declared hard constraints: memory (42 GB > 40 GB budget, causing OOM on the target hardware) and functional quality (0.78 < 0.85 minimum pass rate). Under the decision rule, a measured hard constraint violation is an immediate REJECT (not "insufficient evidence"). Action: reject candidate and maintain original baseline weights.

---

Sources and scope

Claim / Topic	Primary Source	Exact Section / Finding	Notes
Output distortion L = tr(Eᵀ Xᵀ X E)	Lin et al., AWQ (arXiv:2306.00978)	§3.1, Eq. 1	Quadratic output error proxy
Channel salience s = E[abs(X)] (first moment)	Lin et al., AWQ	§3.2	Outlier identification; not second-order
Diagonal rescaling equivalence X’=XD⁻¹, W’=DW	Lin et al., AWQ	§3.3	Scaling weights and activations; FP equivalence
AWQ exponent α grid search and weight clipping	Lin et al., AWQ	§3.4	Optimizing scaling factors; full integer quantization
Hessian H = 2 Xᵀ X per output	Frantar et al., GPTQ (arXiv:2210.17323)	§3	Local quadratic objective per weight column
GPTQ calibration: 128 seq × 2048 tok C4	Frantar et al., GPTQ	§4	Empirical experiment setup, not universal
AWQ robustness to distribution shift	Lin et al., AWQ	§5.3	Calibration on Pile, WikiText-2, C4
Quantization noise variance ≈ S²/12	Jacob et al., CVPR 2018 (arXiv:1712.05877)	§2.2	High-resolution uniform quantizer model
HumanEval = test pass rate	Chen et al., HumanEval (arXiv:2107.03374)	§2	Functional pass@k metric, not syntax check

---

Related lessons

• Inference Optimisation & AI Infrastructure · Quantize Without Losing the ResultQuantization: Scale, Zero Point, and Granularity15 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceWhere Inference Time and Memory Go13 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryBatching and Throughput26 min