Quantization: Scale, Zero Point, and Granularity

The one-sentence version

Uniform integer quantization maps continuous floating-point tensors onto discrete low-bit integer grids; selecting per-channel or per-group granularity isolates outlier values to narrow sub-vectors, preserving high signal-to-noise ratio across the majority of weights at modest metadata cost. Raw payload reduction does not alter physical HBM bandwidth or guarantee throughput gains—kernel unpacking, metadata, KV traffic, and bottleneck shifts all matter.

---

1. The First Principles of Uniform Integer Quantization

1.0 A Concrete Walkthrough Before the Formulas

Before we derive scale factors, granularity tradeoffs, or hardware implications, let’s walk through uniform quantization on three numbers. This is the entire mechanism: round floating-point values to a discrete integer grid, then map back with an affine scale and shift.

Input: x = [-1.2, 0.4, 2.8]
Target grid: signed INT8, q ∈ [-128, 127] (256 levels)

Step 1 — Identify dynamic range:
min(x) = -1.2, max(x) = 2.8

Step 2 — Map to the integer grid (affine/asymmetric, zero in range):
The grid step (scale) is:

S = (max - min) / (q_max - q_min) = (2.8 - (-1.2)) / (127 - (-128)) = 4.0 / 255 ≈ 0.015686

The zero-point (the integer representing real 0.0) is:

Z = round(-min / S) + q_min = round(1.2 / 0.015686) - 128 = round(76.5) - 128 = 77 - 128 = -51

Effective representable range: [S * (q_min - Z), S * (q_max - Z)] = [0.015686 * (-128 + 51), 0.015686 * (127 + 51)] ≈ [-1.206, 2.793], which covers [-1.2, 2.8] and makes 0.0 exactly representable (q = Z = -51 maps to 0.0).

Step 3 — Quantize (round to nearest integer, then clamp):

q = clip(round(x / S) + Z, q_min, q_max)

x	x / S	round(x / S)	+ Z	q (clamped)
-1.2	-76.5	-77	-128	-128
0.4	25.5	26	-25	-25
2.8	178.5	179	128	127 (clamped)

Step 4 — Reconstruct (dequantize):

x_hat = S * (q - Z)

q	q - Z	x_hat = S * (q - Z)
-128	-77	-1.208
-25	26	0.408
127	178	2.792

Step 5 — Rounding error (residual):

Δ = x_hat - x

x	x_hat	Δ = x_hat - x
-1.2	-1.208	-0.008
0.4	0.408	+0.008
2.8	2.792	-0.008

The maximum non-clipped rounding error is bounded by S / 2 ≈ 0.0078. The value 2.8 clipped to 127 (the grid maximum), so its error is slightly larger — this is the clipping error distinct from rounding error.

Key intuition: Quantization is simply rounding floating-point numbers to a discrete integer grid so each value consumes fewer bits, paired with an affine scale S and shift Z to map back to the real-number line.

Why granularity matters: Imagine an extreme outlier enters that vector — say x = [-1.2, 0.4, 100.0]. The range stretches to max = 100.0, so S ≈ 101.2 / 255 ≈ 0.397. The original small values -1.2 and 0.4 now span less than one quantization bin: both round to the same integer, losing all signal. This is why per-tensor quantization fails on transformer weights with outlier channels, and why we need per-channel and per-group granularity to isolate outliers.

---

1.1 Why Quantize: Decoding the Memory-Bandwidth Bottleneck

In autoregressive LLM inference, token generation (decode) is fundamentally memory-bandwidth bound. Every forward iteration must stream the entire model weight matrix from High-Bandwidth Memory (HBM) into on-chip SRAM and registers just to process a single token per sequence (I ≈ 1 FLOP/byte at batch size 1).

Because Tensor Cores sit starved for data during decode, the primary lever for accelerating generation is shrinking the physical memory footprint of the weights:

• 16-bit float (FP16/BF16): 2.0 bytes per parameter (8B model = 16.0 GiB weights)
• 8-bit integer (INT8): 1.0 byte per parameter (8B model = 8.0 GiB weights) → 2.0× payload reduction
• 4-bit integer (INT4): 0.5 bytes per parameter (8B model = 4.0 GiB weights) → 4.0× payload reduction

Payload vs. Runtime

A 2× or 4× reduction in weight payload bytes does not automatically double or quadruple throughput. Kernel dequantization/unpacking overhead, metadata bytes, KV cache traffic, activation memory, and whether the bottleneck shifts from memory to compute all affect realized speedup. Measure end-to-end latency on target hardware; do not extrapolate from payload ratios alone.

Integer vs. Floating-Point Formats

This lesson covers uniform integer quantization (INT8, INT4). FP8 (E4M3/E5M2) is a floating-point format with exponent and mantissa—not an integer grid—and has different range/precision tradeoffs. FP16 is already a finite floating-point format, not “continuous.” We separate weight, activation, and KV quantization; detailed FP8/backend coverage comes in later lessons.

1.2 Uniform Affine (Asymmetric) Quantization

Uniform affine mapping maps a continuous real interval [x_min, x_max] onto a discrete integer range [q_min, q_max] (such as [-128, 127] for signed INT8):

q = clip(round(x / S) + Z, q_min, q_max)
x_hat = S * (q - Z)

Where:

• Scale factor (S): The real-valued grid step size representing the numeric span of a single quantization bin:

  S = (x_max - x_min) / (q_max - q_min)

• Zero-point (Z): The integer value representing the real number 0.0:

  Z = round(-x_min / S) + q_min

Rounding convention: We use round-half-to-even (Python’s round, IEEE 754 default). The quantized code q is clamped to [q_min, q_max].

Range contract: For the mapping to be well-defined, we require:

1. S > 0 (non-zero scale)
2. Z ∈ [q_min, q_max] (valid zero-point)
3. The effective representable range [S*(q_min-Z), S*(q_max-Z)] covers [x_min, x_max]
4. Zero is exactly representable when 0 ∈ [x_min, x_max] (i.e., dequantize(Z) = 0 exactly)

When the input range does not contain zero (positive-only or negative-only data), the standard formula produces an effective range that may not cover the original endpoints. To keep zero exactly representable, the range must be expanded to include zero before computing S and Z. This increases S (coarser grid) but ensures numerical correctness for downstream ops that assume exact zero.

Clamping Z Without Expanding the Range

A common bug: compute Z from the original [x_min, x_max], then clamp Z to [q_min, q_max] without expanding the real range. For positive-only [1, 2] in signed INT8, this yields S=1/255, Z=-128, effective range [0, 1]—the value 2.0 clips to 1.0. The fix: if zero must be representable, expand [x_min, x_max] to include 0 before computing S and Z.

Round-Trip Error Bound: For any non-clipped floating-point value x (i.e., q_min < q < q_max), the rounding error is mathematically bounded by half the grid step:

|x - x_hat| <= S / 2

Clipped values (at q_min or q_max) can have larger error. The bound applies only to values strictly inside the quantized range.

1.3 Uniform Symmetric Quantization

When the dynamic range is symmetric around zero (x_min = -x_max), or when we choose to enforce symmetry, the zero-point is constrained to Z = 0:

S = max(|x_min|, |x_max|) / q_max
q = clip(round(x / S), -q_max, q_max)
x_hat = S * q

For signed INT8, the common LLM convention uses q_max = 127 (255 levels: -127 to 127), leaving the most-negative code -128 unused. This simplifies hardware implementation and avoids asymmetry in the representable range.

S = ||x||_inf / 127

Affine Range Fit vs. Symmetric GEMM Speed

Affine mapping uses the full dynamic range [x_min, x_max], yielding smaller grid steps S on asymmetric distributions. However, modern GPU Tensor Cores favor symmetric quantization to eliminate costly runtime reduction terms during matrix multiplication. The range expansion cost of symmetry is often worth the kernel throughput gain.

1.4 Hardware Integer Tensor Core Dynamics

To see why production LLM runtimes (TensorRT-LLM, vLLM, SGLang) strongly prefer symmetric quantization for weight-activation GEMMs, consider a linear layer matrix multiplication Y = X @ W^T:

Symmetric GEMM (No Correction Terms)

X = S_x * q_x,   W = S_w * q_w
Y = X @ W^T = (S_x * q_x) @ (S_w * q_w)^T = (S_x * S_w) * (q_x @ q_w^T)

Integer Tensor Cores (NVIDIA IMMA for INT8, or CUDA core DP4A) execute the integer dot product q_x @ q_w^T directly at full hardware throughput. The scalar factor (S_x * S_w) is applied once during epilogue writeback. Zero correction terms.

Asymmetric GEMM (Three Correction Terms)

If weights or activations have non-zero zero-points:

X = S_x * (q_x - Z_x),   W = S_w * (q_w - Z_w)
Y = S_x * S_w * [ (q_x @ q_w^T) - Z_w * sum(q_x, axis=-1) - Z_x * sum(q_w, axis=-1) + K * Z_x * Z_w ]

Asymmetric quantization introduces three runtime correction terms per output element:

1. Activation reduction: Z_w * sum(q_x) — sum over input dimension
2. Weight reduction: Z_x * sum(q_w) — sum over input dimension
3. Constant offset: K * Z_x * Z_w — where K is the reduction dimension

These extra reductions consume register budget, add memory traffic, and degrade kernel throughput. While they can be precomputed or fused in some backends, the added complexity is why symmetric quantization is the default for W8A8 serving.

AWQ and GPTQ Do Not Enforce Symmetric Quantization

The AWQ reference quantizer (awq/quantize/quantizer.py) explicitly supports and requires zero-point quantization in its real conversion path. The GPTQ paper (§5 setup, arXiv:2210.17323v2) uses asymmetric per-row quantization. Claims that “AWQ/GPTQ enforce symmetric” are incorrect; they use asymmetric quantization where it helps accuracy, and symmetric where hardware demands it.

1.5 Concrete Numerical Walkthrough

Consider a distribution over [-2.0, 6.0] mapped to signed INT8 (q ∈ [-128, 127]):

• Affine (Asymmetric, zero in range):

  • S = (6.0 - (-2.0)) / (127 - (-128)) = 8.0 / 255 = 0.031373
  • Z = round(-(-2.0) / 0.031373) + (-128) = 64 - 128 = -64
  • Effective range: [-2.0, 6.0] (covers input, zero exactly representable)
  • Maximum non-clipped rounding error: 0.007843 <= S/2 = 0.015686

• Symmetric (Z = 0, q_max = 127):

  • Dynamic range expands to [-6.0, 6.0]
  • S = 6.0 / 127 = 0.047244
  • Maximum non-clipped rounding error: 0.015748 <= S/2 = 0.023622

• Positive-only [1.0, 2.0] with zero forced representable:

  • Range expands to [0.0, 2.0]
  • S = 2.0 / 255 = 0.007843, Z = -128
  • Reconstructed 1.0 → 1.0039, 2.0 → 2.0 (clipped)
  • Non-clipped error bound S/2 = 0.0039 holds for 1.0

---

2. Quantization Granularity: Fighting the Outlier Curse

2.1 The Emergent Outlier Phenomenon

In transformers larger than ~6.7B parameters, hidden state activations develop emergent outlier features (Dettmers et al., 2022). A tiny fraction of channels (~0.1%) exhibit magnitude spikes 20× to 100× larger than typical weights and activations (e.g., 0.80 vs 0.02).

How we choose our quantization scale boundaries determines whether these outliers destroy model accuracy.

The Per-Tensor Outlier Trap

Under per-tensor quantization, a single outlier weight or activation channel inflates the scale S for the entire matrix. All remaining normal weights are compressed into just 2 or 3 integer bins, collapsing Signal-to-Quantization-Noise Ratio (SQNR) and destroying model perplexity.

2.2 The Three Granularities (With Explicit Tensor Orientation)

Tensor orientation for linear layer: X[n, d_in] @ W[d_out, d_in]^T = Y[n, d_out] (PyTorch convention: W.shape = [d_out, d_in]).

Granularity	Scale Tensor Shape	Axis	Outlier Isolation
Per-Tensor	S ∈ ℝ¹	—	None (1 outlier ruins all d_out * d_in weights)
Per-Channel (Per-Output-Channel / Per-Row)	S ∈ ℝ^(d_out)	Output channel (rows of W)	Perfect row isolation: outlier in row i only inflates S_i
Per-Group (Block-wise)	S ∈ ℝ^(d_out × ceil(d_in/G))	Groups partition d_in dimension	Isolates sub-channel clusters within each row

1. Per-Tensor Granularity:

   • Single scale S ∈ ℝ¹ for all weights W ∈ ℝ^(d_out × d_in).
   • Minimal storage: 4 bytes metadata per layer (FP32 scale).
   • High vulnerability: 1 outlier channel inflates S, degrading all rows.

2. Per-Channel Granularity (Per-Row / Per-Output-Channel):

   • Independent scale S_i ∈ ℝ for each output channel (row of W): S ∈ ℝ^(d_out).
   • Metadata overhead: d_out × 2 bytes (FP16 scales). For 4096×4096: 8 KB (~0.05% overhead).
   • Perfect row-level isolation: an outlier in row 0 only inflates S_0. Rows 1 through d_out-1 retain fine grid resolution.
   • This matches the GEMM orientation: each output channel computes Y[:, i] = X @ W[i, :]^T.

3. Per-Group Granularity (Block-wise):

   • Each row of length d_in is partitioned into groups of size G (typically G = 128 or G = 64).
   • Scales: S ∈ ℝ^(d_out × ceil(d_in / G)).
   • Essential for low-bit quantization (INT4 / FP4): protects against intra-row outliers along the hidden dimension.
   • Group index g = k // G for weight W[i, k].

---

3. Measuring Error and Noise

3.1 Quantitative Metrics: MSE and SQNR

To measure quantization distortion mathematically:

1. Mean Squared Error (MSE):

   MSE = (1/N) * sum((x_i - x_hat_i)^2)

2. Signal-to-Quantization-Noise Ratio (SQNR):

   SQNR (dB) = 10 * log10( sum(x_i^2) / sum((x_i - x_hat_i)^2) )

Edge cases:

• Zero signal (sum(x_i^2) = 0): SQNR undefined, conventionally 0 dB.
• Zero error (perfect reconstruction): SQNR = +∞ dB.

The 6 dB per bit heuristic: For a full-scale uniform signal with uniformly distributed, uncorrelated quantization error:

SQNR ≈ 6.02 * b + 1.76 dB

Each additional bit adds ~6.02 dB. This is a heuristic, not a universal law. Real weights and activations are Gaussian, non-uniform, and correlated. A 6.02 dB SQNR difference does not universally equal “one extra bit of model precision.” The 22.25 dB gain in our synthetic test ≈ 3.7 bits under this heuristic, but downstream perplexity impact must be measured, not inferred from SQNR alone.

3.2 Simulation on a Matrix with Outliers (Synthetic Demonstration)

In our verified benchmark (scripts/verify-quantization-first-principles.py), we quantize a 256×256 weight matrix (d_out=256, d_in=256) with normal weights (σ = 0.02) and one outlier row (magnitude = 0.80):

• Per-Tensor INT8 (symmetric):

  • Scale S = 0.80 / 127 = 0.006299
  • Normal rows SQNR collapses to 20.83 dB because regular weights (~0.02) only span ±3 quantization bins.

• Per-Channel INT8 (symmetric, per output channel):

  • Row 0 scale: S_0 = 0.80 / 127 = 0.006299
  • Normal rows scale: S_r ≈ 0.08 / 127 = 0.000630 (10× finer resolution!)
  • Normal rows SQNR reaches 43.08 dB (+22.25 dB gain vs per-tensor).

• Per-Group INT8 (G=64, symmetric):

  • Normal rows SQNR reaches 44.53 dB.

What This Example Demonstrates (and Cannot Establish)

This synthetic weight-row example demonstrates the granularity principle: finer granularity isolates outliers and preserves SQNR on normal data. It does not establish: - The same emergent activation behavior or threshold across transformer families - Downstream perplexity or task accuracy from weight SQNR alone - That σ = 0.02 matches real weight distributions - A universal “one extra bit” result from 6.02 dB Real activation outliers occur at inference time; weight outliers are a proxy. Measure on actual model/calibration data.

---

4. Metadata Storage & Memory Overhead

Quantization scales and zero-points are stored alongside the quantized weight payload. We compute the exact metadata footprint for a standard transformer projection layer W ∈ ℝ^(4096 × 4096) (16,777,216 parameters).

Units: We use binary multiples (MiB = 1024² bytes, KiB = 1024 bytes) throughout. Labels use MiB/KiB.

4.1 Symmetric Layouts (Scale Only)

Format & Granularity	Weight Bits	Weight Payload	Scale Metadata	Total Footprint	Metadata Overhead
FP16 Baseline	16 bits	32.00 MiB	0 B	32.00 MiB	0%
INT8 Per-Tensor	8 bits	16.00 MiB	4 B (FP32)	16.00 MiB	0.00002%
INT8 Per-Channel	8 bits	16.00 MiB	8 KiB (FP16)	16.01 MiB	0.0488%
INT4 Per-Group (G=128)	4 bits	8.00 MiB	256 KiB (FP16)	8.25 MiB	3.125%
INT4 Per-Group (G=64)	4 bits	8.00 MiB	512 KiB (FP16)	8.50 MiB	6.25%
INT4 Per-Group (G=32)	4 bits	8.00 MiB	1024 KiB (FP16)	9.00 MiB	12.50%

Quantization granularity trade-offs for 4096×4096 linear layer (symmetric layouts)

Format & Granularity	Weight Bits	Weight Payload	Scale Metadata	Total Footprint	Metadata Overhead
FP16 Baseline	16 bits	32.00 MiB	0 B	32.00 MiB	0%
INT8 Per-Tensor	8 bits	16.00 MiB	4 B (FP32)	16.00 MiB	0.00002%
INT8 Per-Channel	8 bits	16.00 MiB	8 KiB (FP16)	16.01 MiB	0.0488%
INT4 Per-Group (G=128)	4 bits	8.00 MiB	256 KiB (FP16)	8.25 MiB	3.125%
INT4 Per-Group (G=64)	4 bits	8.00 MiB	512 KiB (FP16)	8.50 MiB	6.25%
INT4 Per-Group (G=32)	4 bits	8.00 MiB	1024 KiB (FP16)	9.00 MiB	12.50%

Per-channel INT8 adds <0.05% metadata while protecting against row outliers. Per-group G=128 in INT4 balances outlier isolation and memory footprint. No universal G=128 optimal claim; group size is a Pareto tradeoff.

4.2 Asymmetric Layouts (Scale + Zero-Point)

Asymmetric quantization requires storing both scales and zero-points:

Format & Granularity	Scale Bytes	Zero-Point Bytes	Total Metadata	Overhead
INT8 Per-Channel	8,192 (FP16)	4,096 (INT8)	12,288 B	0.0732%
INT4 Per-Group (G=128)	262,144 (FP16)	131,072 (INT8)	393,216 B	4.69%

Ideal vs. Actual Runtime Allocation

The tables above show ideal payload + metadata bytes. Actual runtime allocation includes: - Alignment padding (e.g., 128-byte or 4 KiB boundaries) - Allocator overhead and fragmentation - KV cache, activation buffers, and workspace memory - Zero-point storage for asymmetric layouts (shown in asymmetric table) Do not treat ideal overhead percentages as guaranteed runtime memory savings.

No Universal G=128 Optimal or Perplexity Guarantee

Group size G=128 is a common default (AWQ, GPTQ), but it is not universally optimal. The Pareto frontier depends on model architecture, bitwidth, calibration data, and hardware. Neither “G=128 is always best” nor “perplexity always within 0.1 of FP16” is supported by this calculation alone. Require a named model/bitwidth/dataset result for any perplexity figure.

---

5. Asymmetric Quantized Dot Product: Worked Example

Trace y = x @ w^T for a single output element with affine quantization:

x = [1.2, 0.5, -0.8, 2.1]      # mixed-sign activations
w = [0.1, 0.4, 0.2, 0.3]       # positive-only weights

Step 1: Affine quantization (zero forced representable)

• x: range [-0.8, 2.1] → S_x = 0.0114, Z_x = -58
• w: range [0.1, 0.3] expands to [0.0, 0.3] → S_w = 0.0016, Z_w = -128

Step 2: Quantize

• q_x = [169, 102, -12, 247] (clamped to [-128, 127] → [127, 102, -12, 127])
• q_w = [-128, -128, -128, -128] (all map to Z_w since range expanded)

Step 3: Integer dot product + corrections

int_dot = sum(q_x * q_w) = -128 * (127 + 102 - 12 + 127) = -45312
sum_qx = 344, sum_qw = -512, K = 4
correction = -Z_w*sum_qx - Z_x*sum_qw + K*Z_x*Z_w
           = -(-128)*344 - (-58)*(-512) + 4*(-58)*(-128)
           = 44032 - 29696 + 29696 = 44032
y_asymmetric = S_x * S_w * (int_dot + correction)
             = 0.0114 * 0.0016 * (-45312 + 44032)
             = 0.791690

Float reference: y_float = 0.790000
Asymmetric error: 0.001690
Symmetric error (for comparison): 0.000941

The three correction terms (-Z_w*sum_qx, -Z_x*sum_qw, K*Z_x*Z_w) are the runtime cost of asymmetric GEMM.

---

6. W4A16 vs. W8A8: Different Execution Paths

W4A16 (Weight-Only INT4) Is Not W8A8 with Smaller Bits

W8A8 (INT8 GEMM): - Both operands INT8, integer accumulation on Tensor Cores (IMMA) or CUDA cores (DP4A) - Symmetric quantization eliminates zero-point correction terms - Single fused scale multiplication at epilogue W4A16 (Weight-Only INT4): - Weights INT4 (packed 2 values per byte), activations stay FP16 - Weight dequantization: unpack → scale → FP16 (happens before/during GEMM) - FP16 Tensor Core GEMM with FP16 activations - No integer accumulation, no DP4A/IMMA for the main GEMM DP4A: CUDA core instruction (4× INT8 dot product per instruction), NOT a Tensor Core instruction. Tensor Core INT8: Uses IMMA (Integer Matrix Multiply Accumulate) instructions. These are fundamentally different kernel paths with different performance characteristics. Do not conflate them.

---

7. Constrained Quantization Choice: A Complete Example

Scenario: You have a 4096×4096 linear layer. You must choose a quantization scheme for INT4 weight-only inference.

Constraints:

• Weight payload budget: 8.00 MiB (INT4 packed)
• Total budget (payload + metadata): ≤ 9 MiB
• Backend support: Symmetric INT4 per-group supported; asymmetric not supported
• Quality target (from synthetic outlier test): Normal-row SQNR > 40 dB

Option	Metadata	Total	Est. Normal SQNR	In Budget?
Per-Tensor (sym)	4 B	8.00 MiB	~21 dB	✓ (fails quality)
Per-Channel (sym)	8 KiB	8.01 MiB	43.1 dB	✓
Per-Group G=128 (sym)	256 KiB	8.25 MiB	44.5 dB	✓
Per-Group G=64 (sym)	512 KiB	8.50 MiB	44.5 dB	✓
Per-Group G=32 (sym)	1024 KiB	9.00 MiB	44.5 dB	✓

Evidence gaps preventing a confident choice from this example alone:

• Synthetic weight outlier test ≠ real activation outlier behavior
• Weight SQNR ≠ downstream perplexity/accuracy
• No measured decode latency on target hardware (H100/A100) for each granularity
• No calibration data evaluation on target model/dataset

Recommendation: Run post-training quantization (PTQ) with per-group G=128 and G=64 on the actual model. Measure perplexity on a validation set and benchmark decode latency. This synthetic example cannot establish the optimal choice—it only demonstrates the granularity principle.

Honest Insufficient-Evidence Outcome

When the available evidence (synthetic SQNR, ideal byte counts) does not cover runtime latency, real activation distributions, or downstream task quality, the correct engineering answer is “insufficient evidence to choose.” Commission the missing measurements before committing.

---

Key Takeaways

Key takeaways

• Decoding is strictly memory-bandwidth bound (I ≈ 1 FLOP/byte at batch size 1); quantizing weight payload from FP16 to INT8 or INT4 reduces bytes moved, but realized speedup depends on dequantization overhead, KV traffic, and bottleneck shifts.
• Uniform affine quantization maps real ranges [x_min, x_max] to integers [q_min, q_max] using scale S and zero-point Z. The range contract requires S>0, Z in bounds, effective range covers input, and zero exactly representable when 0 is in the input range (expand range if needed).
• Symmetric quantization enforces Z=0 (often using q_max=127, leaving -128 unused). Modern GPU Tensor Cores favor symmetric quantization to eliminate runtime reduction correction terms during GEMM.
• Emergent outlier channels (~0.1% of activations in >6.7B models) destroy per-tensor quantization, collapsing normal channel SQNR. Per-channel (per-output-channel) quantization confines outliers to their rows, recovering >20 dB SQNR at <0.05% metadata overhead.
• Per-group quantization (block-wise, G=128/64/32) partitions the d_in dimension, protecting INT4 models from intra-row outliers with 3–12% metadata overhead. Group size is a Pareto tradeoff, not universally G=128.
• MSE and SQNR quantify distortion. Zero signal → 0 dB SQNR; zero error → ∞ dB. The 6.02 dB/bit heuristic assumes full-scale uniform signal with uniform uncorrelated error—real weights/activations violate this. Do not equate dB gain to universal "bits of precision."
• Asymmetric quantized dot products require three correction terms per output element. W8A8 integer GEMM and W4A16 dequantize-then-FP16-GEMM are different execution paths; DP4A is a CUDA core instruction, not a Tensor Core instruction.
• Ideal payload + metadata bytes (MiB/KiB) exclude alignment, allocator overhead, KV cache, and workspace. No universal perplexity guarantee from byte calculations alone. When evidence is missing, the honest answer is "insufficient evidence."

---

Exercises

Try it yourself

1. 01Derive the affine scale S and zero-point Z for a weight tensor spanning [-2.0, 6.0] mapped to signed INT8 [-128, 127]. Compare its maximum non-clipped rounding error against symmetric INT8 quantization on the same range. Explain why the symmetric range expands.

   Show a hint

   Affine: S = 8.0/255 = 0.031373, Z = -64, max non-clipped error S/2 = 0.015686. Symmetric: range expands to [-6.0, 6.0], S = 6.0/127 = 0.047244, Z = 0, max non-clipped error S/2 = 0.023622. Symmetry forces range expansion to cover max absolute value.

2. 02A 256×256 weight matrix (d_out=256, d_in=256) with normal variance σ=0.02 has one outlier output channel (row 0) with magnitude 0.80. Explain why per-tensor quantization collapses normal row SQNR to ~21 dB, and calculate the effective bit precision gained by switching to per-channel quantization (+22.25 dB under the uniform-noise heuristic).

   Show a hint

   Per-tensor S = 0.80/127 = 0.0063; normal weights ~0.02 span only ±3 bins. Per-channel: row 0 uses S=0.0063, normal rows use S≈0.00063 (10× finer). 22.25 dB / 6.02 ≈ 3.7 bits under the heuristic. This is a synthetic weight example—not measured activation behavior or downstream accuracy.

3. 03Calculate the total storage footprint (payload + metadata) for a 4096×4096 layer under INT4 Per-Group quantization with group size G=64, using FP16 scale factors. Determine its percentage overhead over raw INT4 payload. Then add INT8 zero-points for an asymmetric layout and recalculate.

   Show a hint

   Symmetric: Payload 8.00 MiB, Groups = 16,777,216/64 = 262,144, Scales = 262,144×2 B = 512 KiB. Total = 8.50 MiB. Overhead = 6.25%. Asymmetric: add 262,144×1 B ZPs = 256 KiB. Total metadata = 768 KiB. Total = 8.75 MiB. Overhead = 9.375%.

4. 04Trace the asymmetric quantized dot product y = x·w for x=[1.2, -0.5], w=[0.3, 0.1] using signed INT8 affine quantization with zero forced representable. Show the integer codes, the three correction terms, and the final reconstructed value. Compare against the float reference.

   Show a hint

   x range [-0.5, 1.2] → S_x, Z_x. w range [0.1, 0.3] expands to [0.0, 0.3] → S_w, Z_w. Compute q_x, q_w, int_dot, sum_qx, sum_qw, corrections, y_asymmetric. Float reference = 0.31.

5. 05Given the constrained choice scenario in Section 7 (4096×4096, INT4, ≤9 MiB, symmetric-only backend, SQNR>40 dB target), list the viable options from the synthetic test. Then state three specific evidence gaps that prevent a confident production choice, and what measurements would close each gap.

   Show a hint

   Viable: Per-Channel, Per-Group G=128/64/32 (all meet budget and SQNR target). Gaps: (1) Synthetic weight SQNR ≠ activation behavior → measure activation outliers on calibration data. (2) Weight SQNR ≠ perplexity → run PTQ and evaluate on validation set. (3) No latency data → benchmark decode on target GPU for each granularity.

---

Sources and Scope

Claim / Topic	Primary Source	Exact Section / Finding	Notes
Quantization Survey & Affine Formulations	Gholami et al., A Survey of Quantization Methods for Efficient Neural Network Inference, IEEE 2021	§2, §2.1–2.2	Rigorous definitions of uniform affine, symmetric, scale factor, and zero-point parameters.
Emergent Outliers & Vector-wise Quantization	Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurIPS 2022	§3, Figure 1 & 2	Establishes the 6.7B emergent outlier threshold and per-channel vector-wise quantization.
Per-Group Quantization & Salient Weights	Lin et al., AWQ: Activation-aware Weight Quantization for On-Device LLM Compression, MLSys 2024	§3.1–3.3, §4	Details per-group scaling (G=128) and protection of salient channels without mixed precision.
Group Size Trade-offs in INT4	Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR 2023	§4, Table 1	Compares group sizes (G=128 vs G=32) against perplexity retention and inference throughput.
Integer Arithmetic GEMM & Hardware Acceleration	Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, CVPR 2018	§2.1–2.3	Analyzes asymmetric zero-point correction penalties in integer matrix multiplication.
AWQ Reference Quantizer (Asymmetric Support)	mit-han-lab/llm-awq, awq/quantize/quantizer.py	zero_point logic	Explicitly computes and uses zero-points; contradicts “AWQ enforces symmetric” claims.
GPTQ Asymmetric Quantization	Frantar et al., GPTQ, ICLR 2023	§5 (setup)	Uses asymmetric per-row quantization in the paper’s quantization pipeline.
NVIDIA IMMA vs DP4A	CUDA C Programming Guide	PTX ISA references	IMMA = Tensor Core INT8 MMA; DP4A = CUDA core 4×8-bit dot product. Distinct instruction classes.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceWhere Inference Time and Memory Go13 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryBatching and Throughput26 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceServing Workloads and Performance Metrics12 min