Low-bit LLM Weights and Activations

The one-sentence version

Quantization is not a single knob: you must separate what is quantized (weights, activations, KV cache) from the numeric format (INT8, FP8 E4M3/E5M2, INT4, FP4), because each combination maps to a fundamentally different GPU execution path with different bandwidth, compute, and quality tradeoffs.

---

1. The Two-Axis Quantization Taxonomy

Low-bit inference decisions live on two orthogonal axes:

Axis	Decision	Notation	Example
Axis 1: What is quantized	Weights only	W4A16, W8A16	Weight-only (activations stay FP16/BF16)
	Weights + Activations	W8A8, W4A4, FP8	Both quantized; runtime scale granularity varies
	KV Cache	KV8, KV4	Keys/values quantized separately (Topic 11)
Axis 2: Numeric format	Integer	INT8, INT4	Linear grid, symmetric or affine
	Floating point	FP8 E4M3, FP8 E5M2, FP4	Exponent + mantissa, native TC support

The notation W{A}A{B} names A-bit weights with B-bit activations: W4A16 is 4-bit weights with 16-bit activations, W8A8 is 8-bit weights with 8-bit activations. “FP8” is a format name, not a bit-width claim: in serving it usually means W8A8 with FP8 operands.

Two-Axis Taxonomy: What vs Format

Target	Notation	Activations	Hardware Execution Path
Weight-only	W4A16 / W8A16	FP16 / BF16	Load packed weights → unpack & dequantize to FP16 in registers/SRAM → FP16 Tensor Core GEMM
Weight + Activation	W8A8 / FP8	INT8 / FP8 (per-token)	Both operands feed directly into INT8/FP8 Tensor Cores; no dequantization to FP16
KV Cache	KV8 / KV4	N/A (cache)	Quantized pre-RoPE or per-head; Topic 11 deep dive

Axis independence matters

The same numeric format (e.g., INT8) can serve different targets with completely different hardware paths: INT8 weights in W8A16 dequantize to FP16 before GEMM; INT8 weights in W8A8 execute directly on INT8 Tensor Cores.

---

2. Numeric Formats Deep Dive

2.1 INT8 (8-bit Two’s Complement Integer)

Property	Value
Bit layout	8-bit two’s complement (signed range -128 to 127)
Symmetric scale	S = max(abs(x)) / 127
Step size	Uniform: S
Hardware	A100 INT8 Tensor Cores (624 TOPS dense), H100 INT8 Tensor Cores (1979 TOPS dense)
Use case	W8A16 weight-only, and W8A8 weight+activation quantization (e.g. SmoothQuant). 4-bit weights are INT4/FP4, not INT8

2.2 FP8 (OCP Standard: E4M3FN vs E5M2)

The OCP 8-bit Floating Point Specification (OFP8 1.0) defines two 8-bit encodings with different precision/range trade-offs. E5M2 follows IEEE 754 conventions for exponent and special values; E4M3 deviates by reclaiming the infinity patterns, which buys one extra binade. Neither is an IEEE 754 interchange format, so “IEEE-like” is the accurate label.

FP8 Format Comparison (OCP OFP8 1.0)

Property	E4M3FN	E5M2
Bit layout	1 sign, 4 exponent, 3 mantissa	1 sign, 5 exponent, 2 mantissa
Exponent bias	7	15
Max finite value	448.0 (1.75 × 2^8)	57344.0 (1.75 × 2^15)
Min positive normal	2^-6 = 0.015625	2^-14 ≈ 6.1 × 10^-5 (same as FP16)
Dynamic range	≈ 10^4 (18 binades)	≈ 10^9 (32 binades, same as FP16)
Precision (ULP at 1.0)	2^-3 = 0.125 (1/8)	2^-2 = 0.25 (1/4)
Zeros	S.0000.000, both signs	S.00000.00, both signs
Special values	No infinity; one NaN pattern S.1111.111, i.e. 0x7F and 0xFF	Infinities S.11111.00; NaNs S.11111.{01,10,11}
Ideal for	Forward pass: weights, activations	Wide-range tensors; training gradients (outside serving scope)

FP8 E4M3FN details:

• FN means “finite, plus NaNs”: there is no infinity encoding. The only NaN pattern is exponent 1111 with mantissa 111, so the NaN bit patterns are 0x7F (positive) and 0xFF (negative).
• Every other pattern with exponent 1111 is a finite number: the largest is S.1111.110, ±448 (0x7E / 0xFE). A plain IEEE E4M3, where 1111 is reserved for infinity and NaN, would stop at ±240.
• The third mantissa bit is why E4M3 is the usual choice for weights and forward activations: 1/8 ULP at 1.0 instead of E5M2’s 1/4.
• Conversion above 448 depends on the selected OCP saturation mode: saturation yields ±448; the non-saturating mode yields NaN. Inference software must choose a scaling and conversion policy.

FP8 E5M2 details:

• Infinities are S.11111.00 and NaNs are S.11111.{01,10,11}: the IEEE 754 scheme, with 2 mantissa bits where FP16 has 10.
• The minimum positive normal, 2^-14, is the same as FP16, and the exponent range matches, but the 2-bit mantissa caps the largest value at 57344 instead of FP16’s 65504.
• Five exponent bits give roughly 10^9 of dynamic range. That headroom, not accuracy, is the reason E5M2 appears in training; serving kernels normally use E4M3 for both weights and activations.

FP8 is not 8-bit integer

E4M3FN and E5M2 are 8-bit floating-point encodings defined by the OCP specification, not an IEEE 754 interchange type and not an integer format. E4M3FN max is 448 (not 127), E5M2 max is 57344. Precision is non-uniform: finer near zero, coarser at large magnitudes.

2.3 INT4 (4-bit Integer)

Property	Symmetric	Asymmetric
Range	q in -7 to 7 (3 bits + sign)	q in 0 to 15
Scale	S = max(abs(x)) / 7	S = (max(x) - min(x)) / 15
Zero-point	None (0 maps to 0)	Z in 0 to 15
Group scaling	Groups of 32, 64, or 128 values share one scale	Same

Metadata Overhead (4096 × 4096 matrix):

• INT8 Per-Channel: 8,192 bytes (0.0488% overhead)
• INT4 Per-Group G=128: 262,144 bytes (3.125% overhead)
• INT4 Per-Group G=32: 1,048,576 bytes (12.50% overhead)

2.4 FP4 / NVFP4 (Blackwell Microscaling)

Property	MXFP4 (OCP Standard)	NVFP4 (NVIDIA Blackwell)
Element format	E2M1 (1 sign, 2 exp, 1 mantissa)	E2M1
Block size	32 elements	16 elements
Scale format	E8M0 (8-bit pure exponent, bias 127)	E4M3 (fractional scale)
Scale overhead	8 bits / 32 = 0.25 bpw	8 bits / 16 = 0.50 bpw
Effective bpw	4.25	4.50

---

3. Physical GPU Hardware Execution Reality

3.1 Tensor Core Operand Compatibility

Tensor Core instructions support specific operand type combinations. A W4A16 kernel cannot feed packed INT4 weights directly into an FP16 GEMM: it unpacks and dequantizes the weights first. The target GPU and kernel determine which 8-bit operand combinations are supported.

3.2 A100 (Ampere) vs H100 (Hopper) Tensor Core Capabilities

Tensor Core peak throughput by GPU generation (dense, no sparsity)

Precision	A100 (Ampere)	H100 SXM5 (Hopper)	Notes
FP16/BF16	312 TFLOP/s	989 TFLOP/s	Dense TC, no sparsity
TF32	156 TFLOP/s	494.7 TFLOP/s	Dense TC
INT8	624 TOPS	1979 TOPS	Dense TC, 2× FP16
FP8	Not available	1979 TFLOP/s	H100 only; dense 2× FP16

Key implication:

• A100: no native FP8. W8A8 runs on INT8 Tensor Cores; W4A16 dequantizes to FP16 before GEMM.
• H100: native FP8 Tensor Cores at twice the dense FP16 rate, so W8A8/FP8 runs on FP8 Tensor Cores; W4A16 still dequantizes to FP16.
• These are dense peaks, not achieved kernel throughput. INT4 is deliberately absent: the H100 whitepaper table this lesson relies on does not list an INT4 peak, and sparse INT4 numbers are not comparable with dense FP8.

3.3 Weight-Only Execution Path (W4A16 / W8A16)

HBM (packed INT4/INT8 weights)
    │
    ▼ Load to SM
Registers / Shared Memory (SRAM)
    │
    ▼ Unpack + Dequantize
    S_w * (W_int - Z_w)  →  FP16  [dequantization ALU ops]
    │
    ▼ FP16 Tensor Core GEMM
    Y_fp16 = X_fp16 @ W_fp16_dequant

Critical engineering insight:

• Speedup source: weight traffic only. Packed 4-bit weights are 0.5 bytes per parameter plus scales, against 2 bytes in FP16: a 4× payload reduction, about 3.9× once G=128 scales are counted. That is a traffic ratio, not a latency promise; achieved speedup depends on the kernel.
• Compute ceiling: still the FP16 Tensor Core rate. Dequantization is extra ALU work inside the inner loop, so W4A16 cannot beat FP16 on a genuinely compute-bound GEMM.
• Decode regime (B=1, one token, I ≈ 1 FLOP/B): bandwidth-bound, far below the A100 ridge point of about 153 FLOP/B. This is where weight-only quantization has the most headroom.
• Long-prompt, high-token-parallelism regime: W4A16 adds dequantization work and no extra compute, so it is not the tool for a GEMM that is already compute-bound.

W4A16 is a decode tool, not a compute accelerator

In the simplified weight-traffic model, B=1 gives I ≈ 1 FLOP/B, far below the ridge point, and W4A16 reads about 8.25 MiB instead of 32 MiB for a 4096×4096 layer at INT4 G=128: 3.88× less weight traffic. At B=512 the model predicts I = 1024 FLOP/B, well above the ridge point, where W4A16 is capped at the FP16 peak (312 TFLOP/s on A100) while W8A8/INT8 has a 624 TOPS dense peak. Whether either effect shows up in a real server depends on the kernel, the KV-cache traffic and the shape of the workload.

3.4 Weight + Activation Execution Path (W8A8 / FP8)

HBM (packed INT8/FP8 weights)
    │
    ▼ Load to SM (no dequantization needed)
INT8/FP8 Tensor Core GEMM (direct)
    │
    ▼ INT8 accumulates in INT32; FP8 accumulates in FP32
    │
    ▼ Epilogue: scale fusion
    INT8 example: Y = diag(S_x) · (X_int8 @ W_int8) · diag(S_w) + bias

Critical engineering insight:

• Speedup source: twice the dense peak of FP16 Tensor Cores. The faster path exists only when both operands are 8-bit, which is why the activation quantizer, not the weight format, decides whether W8A8 is usable.
• In this illustrated INT8 path, scales are per outer dimension: the per-token activation scale S_x has one value per row (shape T), the weight scale S_w one value per output channel (shape C_o). They are applied in the epilogue, after the integer reduction. Per-tensor scales are another supported granularity in the SmoothQuant paper.
• When it pays: only when the GEMM is large enough to reach the compute roofline, e.g. long prompts, high token parallelism or large batches. At small batch sizes the same kernel is bandwidth-bound and the 2× peak is unreachable.
• Software stacks: check the exact serving runtime, GPU generation and kernel for support before applying a peak-throughput comparison.

---

4. The Activation Outlier Problem & SmoothQuant

4.1 Why activation quantization is difficult

Property	Weights	Activations
Distribution	Static, roughly Gaussian	Dynamic, input-dependent
Outliers	Rare, predictable	Emergent systematic outliers in models of 6.7B and larger
Granularity	Per-channel natural	Per-token required (dynamic)

The emergent-outlier phenomenon (Dettmers et al., 2022, reviewed in SmoothQuant §3):

• Above roughly 6.7B parameters, fewer than 0.1% of activation channels carry magnitudes tens to about 100× larger than typical channels, and the same channels are large for every token.
• The numbers below come from this chapter’s verification script, not from a released model. It builds a 128×256 activation matrix with channel 112 pinned at +80.0, other activations around standard deviation 0.5, and weights around standard deviation 0.02. The outlier is a synthetic teaching input chosen to reproduce the failure mode.
• With one per-tensor scale, the outlier sets S = 80 / 127 ≈ 0.63. A typical value of 0.5 becomes 0.5 / 0.63 ≈ 0.8 of a step and collapses to a single integer level.
• The script’s separate 20.8/43.3/44.6 dB SQNR comparison quantizes a synthetic weight matrix with one large row. It does not measure activation SQNR for channel 112; the activation example here illustrates the scale problem only.

One per-tensor activation scale is not enough

SmoothQuant Table 1 is the published version of this failure. On OPT-6.7B through OPT-175B, the average accuracy over WinoGrande, HellaSwag, PIQA and LAMBADA is 64.9%–71.6% in FP16, 32.3%–39.9% with per-tensor INT8 activations, and 64.8%–71.4% with per-channel activation scales. Per-channel activation scales preserve accuracy but cannot run inside an INT8 GEMM kernel, which is exactly the gap SmoothQuant closes.

4.2 SmoothQuant formulation (Xiao et al., ICML 2023)

Core insight: a linear layer is invariant to a per-input-channel rescaling, so quantization difficulty can be migrated between activations and weights:

Y = X W = (X · diag(s)^-1) · (diag(s) · W) = X̂ Ŵ

s has one entry per input channel (s is in R^C_i), computed from calibration statistics:

s_j = max(|X_j|)^α / max(|W_j|)^(1-α)

Migration strength α in [0, 1]:

α	Effect
α = 0	All difficulty goes to activations (s_j = 1 / max(abs(W_j))). Outliers are undamped, so W8A8 fails
α ≈ 0.5	The paper’s default for OPT and BLOOM: activations and weights end up with comparable ranges
α ≈ 0.75	Used for GLM-130B, whose outliers are more severe (SmoothQuant §5.1)
α ≈ 0.8	Used for the original LLaMA models in SmoothQuant Table 6; tune this value for a new model and calibration set
α = 1	All difficulty goes to weights (s_j = max(abs(X_j))). Weight quantization now has to absorb the outlier

What the transformation buys: after smoothing, the per-channel activation range and the matching weight range sit at similar magnitude, so the two tensors are equally hard to quantize. It does not make the activation channels identical; it shrinks the spread and moves the remainder into weights.

4.3 What happens at runtime

The transformation is only useful because it is applied offline, in three steps:

1. Offline: measure per-channel activation maxima on calibration data, compute s, multiply weight row j by s_j, and fold the inverse diag(s)^-1 into a preceding operation where the graph permits it. Inputs from a residual add can need an extra scaling operation; do not assume every path is free at runtime.
2. Quantize the smoothed tensors: choose activation and weight quantization granularity for the target kernel. SmoothQuant’s evaluated settings include per-token or per-tensor activations and per-tensor weights. Per-output-channel weight scales are another outer-dimension option; the smoothing factor s_j itself is never the runtime weight quantization scale.
3. Runtime GEMM: accumulate in INT32 and apply any per-token activation or per-output-channel weight scales in the epilogue. Per-tensor scales can also be applied there; the schematic is the outer-dimension case in equation 2 of the paper:

Y = diag(Δ_X) · (X̂_int8 · Ŵ_int8) · diag(Δ_W)

The input-channel scale is not factored out of the accumulation

A per-input-channel factor sits inside the reduction over C_i, so it cannot be pulled out of the dot product or the INT32 accumulator. SmoothQuant avoids the problem by applying s offline before quantization, not by factoring it out of the GEMM. Runtime quantization scales, if used, must be chosen so they can be applied outside the reduction; per-token activation and per-output-channel weight scales are the two outer-dimension examples.

4.4 Synthetic error check

scripts/verify-low-bit-weights-activations.py runs the transformation on the synthetic matrix above and reports output SQNR for a range of α. These are teaching numbers from that script, not paper or model benchmarks:

• α = 0: 29.15 dB
• α = 0.5: 45.95 dB (+16.80 dB over α = 0)
• α = 0.85: 42.90 dB
• α = 1.0: 37.72 dB (8.23 dB below α = 0.5)

The shape of the curve matches the paper’s argument: too little migration leaves outliers in the activations, too much pushes them into the weights, and the best point sits in between. The optimum here is a property of this synthetic matrix, not a universal constant.

SmoothQuant vs AWQ: different targets, different mechanisms

Aspect	SmoothQuant	AWQ
Target	W8A8 (weight + activation)	W4A16 (weight-only)
Mechanism	Mathematically equivalent per-input-channel rescaling, applied offline, to migrate activation outliers into weights	Per-input-channel scaling that protects salient weight channels, then ordinary group-wise weight quantization
Scale selection	Closed form from calibration maxima, one α for the model	Scale s = mean(|X_j|)^α with α found by a small grid search; weights then clipped
What gets quantized	Both INT8; the paper tests per-token or per-tensor activations with per-tensor weights	Weights per group (typically G=128); activations stay FP16
Relation to GPTQ	Independent; combined with any W8A8 quantizer	Orthogonal to GPTQ; the two can be combined

---

5. A Simplified Roofline Estimate for Decode (B*)

This section builds a deliberately small model of one decode step. It estimates where weight traffic stops being the limit. It does not predict measured throughput, and it does not crown a winner.

5.1 Analytical model

Consider one decode step (a single generated token) for a model with P parameters at batch size B, ignoring attention and everything else that is not a dense weight GEMM.

Assumptions, all of which matter to the result:

• Every parameter is read exactly once per decode step; weight distribution is uniform across layers.
• Only weight traffic is counted. KV-cache reads, activation traffic, and attention kernels are excluded.
• No dequantization cost is modelled, apart from the INT4 scale overhead η.
• Memory access is perfectly coalesced, and kernel efficiency is constant across batch sizes.
• Each layer is treated as one GEMM.

Memory traffic per decode step (weights only):

Format	Weight bytes read	Notes
FP16 (W16A16)	2P	Baseline
W4A16	0.5P × (1 + η)	η ≈ 0.03125 for INT4 G=128 scales
W8A8 / FP8	1.0P	No metadata overhead in this idealisation

Compute per decode step, and the resulting operational intensity:

FLOPs = 2 P B
I     = FLOPs / Bytes = 2 P B / (weight bytes)

5.2 Ridge points

GPU ridge points from dense peaks and HBM bandwidth

GPU	HBM Bandwidth	FP16 Peak	INT8/FP8 Peak	I_ridge (FP16)	I_ridge (INT8/FP8)
A100	2039 GB/s	312 TFLOP/s	624 TOPS (INT8)	153 FLOP/B	306 FLOP/B
H100	3350 GB/s	989 TFLOP/s	1979 TFLOP/s (FP8)	295 FLOP/B	591 FLOP/B

I_ridge = peak compute / memory bandwidth

These ridge points are arithmetic on dense peak specs. Real kernels sit below them.

5.3 Operational intensity by format

I_W4A16 = 2PB / (0.5P × (1 + η)) = 4B / (1 + η) ≈ 3.88B
I_W8A8  = 2PB / (1.0P)           = 2B

5.4 Crossover batch size, B*

In this model the crossover is the batch size where the bandwidth-limited W8A8 or FP8 throughput catches up with the FP16 compute ceiling that caps W4A16.

On A100:

• W4A16 reaches the FP16 compute ceiling at B ≈ 40: 4B / 1.03125 = 153 gives B = 39.4.
• W8A8 becomes compute-bound at B ≈ 153 (2B = 306).
• Between those, W4A16 is capped at 312 TFLOP/s while W8A8 is still bandwidth-bound at 2039 × 2B GFLOP/s.
• Crossover: 2039 × 2B* = 312 × 10^3, so B* = 312 × 10^3 / (2 × 2039) = 76.5 ≈ 77.

On H100:

• W4A16 reaches the FP16 compute ceiling at B ≈ 76: 4B / 1.03125 = 295 gives B = 76.1.
• FP8 becomes compute-bound at B ≈ 295 (2B = 591).
• Crossover: 3350 × 2B* = 989 × 10^3, so B* = 989 × 10^3 / (2 × 3350) = 147.6 ≈ 148.

Estimated crossover from the weight-traffic-only decode model

GPU	W4A16 hits FP16 ceiling at	W8A8/FP8 compute-bound at	Estimated B*	What the model suggests
A100	B ≈ 40	B ≈ 153	B* ≈ 77	Below B*: weight traffic dominates. Above B*: INT8 compute headroom starts to matter
H100	B ≈ 76	B ≈ 295	B* ≈ 148	Below B*: weight traffic dominates. Above B*: FP8 compute headroom starts to matter

What this estimate does not cover

B* is an order-of-magnitude aid, not a threshold. It ignores KV-cache traffic, which grows with context length and can dominate decode at long context. It ignores attention kernels, launch overhead, dequantization cost, weight-loading efficiency and quality. It assumes a dense weight GEMM per layer, so it says nothing useful about MoE routing or very small layers. Prefill is a different regime: once a prefill GEMM has enough token parallelism to be compute-bound, the 2× INT8/FP8 peak can matter, but a short prompt or a low-parallelism shape may never reach it. The only way to know which side of the crossover a real deployment sits on is to measure the actual kernels on the actual model.

Using the estimate without over-trusting it

• Small-batch decode: weight traffic dominates in this model, so weight-only 4-bit is the first thing to benchmark.
• Near the estimated B*: both paths are plausible. Benchmark both kernels, do not extrapolate.
• Large batches or long prompts: token parallelism can make the GEMM compute-bound, which is where 8-bit weight-and-activation formats become competitive. Quality and kernel support usually decide before arithmetic does.

---

6. Quality Tradeoffs & Architectural Decision Guide

6.1 Quality results, pinned to a model and a metric

Quality loss from quantization depends on the model, the method, the calibration set and the metric. A generic “±X perplexity” range is not a usable claim, so the table below only contains results that name the model, the baseline, the method and the metric. Every number is quoted from the cited table.

Model	Metric	Baseline	Method	Result	Source
OPT-175B	WikiText-2 PPL (lower better)	FP16 10.99	naive W8A8, per-tensor	93080	SmoothQuant Table 3
OPT-175B	7-task zero-shot average	FP16 66.9%	naive W8A8, per-tensor	35.5%	SmoothQuant Table 3
OPT-175B	7-task zero-shot average	FP16 66.9%	SmoothQuant-O3	66.8%	SmoothQuant Table 3
LLaMA-7B	WikiText-2 PPL	FP16 11.51	SmoothQuant W8A8, α = 0.8	11.56	SmoothQuant Table 6

Notes on reading this table:

• The OPT-175B rows are the clearest illustration of the outlier problem: naive per-tensor W8A8 destroys the model, while SmoothQuant recovers FP16 quality.
• These are single studies on single model families. A method that wins on OPT-175B is not guaranteed to win on a 7B model, and none of these runs used the reader’s calibration data.
• Perplexity is a proxy. Coding, tool-calling and long-context behaviour have to be evaluated on the deployment’s own tasks, which is the subject of post-training-quantization-calibration.

6.2 Why FP8 is common in modern serving

1. Hardware-native on H100: its dense FP8 Tensor Core peak is about 2× its dense FP16 peak; achieved speed depends on the kernel and shape.
2. Floating-point representation: FP8 has no integer zero-point machinery, but serving still needs a scale and a quality check for the target model.
3. Symmetric by construction: E4M3 and E5M2 have signed zero, so the asymmetric corner cases of integer quantization do not arise.
4. Software support: an FP8 capable GPU is only one requirement; verify that the selected runtime has a compatible kernel for the model and tensor shapes.
5. Out of scope here: E5M2’s wide range is useful for FP8 training gradients. The serving examples in this chapter use E4M3; E5M2 serving and KV-cache choices need their own quality and kernel evidence.

6.3 When to choose what

Starting hypotheses by workload, to be validated by measurement

Workload	First thing to benchmark	Why
Single-user, low-latency decode (A100)	W4A16 (AWQ or GPTQ)	Weight traffic dominates small-batch decode, so a ~3.9× traffic reduction is the obvious lever
Single-user, low-latency decode (H100)	W4A16, with FP8 measured as the alternative	Both are plausible once KV-cache traffic and kernel efficiency are included; the roofline estimate alone does not decide it
High-throughput serving (A100)	W8A8 INT8 (SmoothQuant)	Above the estimated B* ≈ 77 the INT8 peak can matter, but activation accuracy has to be verified per model
High-throughput serving (H100)	FP8 E4M3 weights and activations	Above the estimated B* ≈ 148 FP8 has native hardware and 2× the dense FP16 peak; quality and kernel support still decide
Memory-constrained edge	W4A16 (INT4, group 128)	Smallest weight footprint; measure the target kernel and device
Long-context, KV-cache sensitive	KV8/KV4 (Topic 11)	A separate axis; combine with weight quantization rather than substituting for it

Every row is a hypothesis to test, not a recommendation: the measured winner depends on the kernel, the shape of the workload, the context length and the quality bar.

---

Key takeaways

• Quantization has two independent axes: WHAT you quantize (weights, activations, KV cache) and the NUMERIC FORMAT (INT8, FP8 E4M3/E5M2, INT4, FP4). Each combination maps to a different GPU execution path.
• Weight-only (W4A16) dequantizes in SRAM or registers to FP16 before the Tensor Core GEMM. It reduces weight traffic by about 3.9× at INT4 G=128 and adds no compute throughput, so measured speedup is a kernel result rather than a guaranteed ratio.
• Weight + activation (W8A8, FP8) feeds both operands straight into 8-bit Tensor Cores, which doubles the dense peak. That headroom is only reachable while the GEMM is compute-bound: long prompts, high token parallelism or large batches.
• Activation outliers emerge systematically in the larger models studied by SmoothQuant. This chapter uses a synthetic +80 activation in channel 112 to show why one per-tensor INT8 scale can lose precision; the script’s 20.8 dB weight-matrix SQNR is a separate example.
• SmoothQuant migrates difficulty with s_j = max(|X_j|)^α / max(|W_j|)^(1-α), where j indexes input channels. Its paper uses α ≈ 0.5 for OPT and BLOOM, 0.75 for GLM-130B, and 0.8 for original LLaMA.
• The smoothing factor is calculated offline and absorbed into weight rows and, where possible, a preceding operation. Runtime activation and weight quantization scales use a kernel-compatible granularity; the input-channel smoothing factor is not factored out of INT32 accumulation.
• AWQ is a weight-only method: it scales salient weight channels, chosen from activation statistics, and then applies ordinary group-wise quantization. It is not GPTQ, and it does not quantize activations.
• FP8 is OCP-defined, not IEEE 754. E4M3FN (max 448, NaN patterns 0x7F and 0xFF, 1/8 ULP at 1.0) suits weights and activations; E5M2 (max 57344, IEEE-style infinities and NaNs) is mainly a training-gradient format. H100 has native FP8 Tensor Cores; A100 does not.
• B* ≈ 77 on A100 and B* ≈ 148 on H100 are outputs of a weight-traffic-only decode model, not measured thresholds. KV traffic, kernels, workload shape, prefill length and quality are all excluded, so they order the options but do not pick a winner.
• INT4 metadata overhead is 3.125% at G=128 and 12.5% at G=32 with FP16 scales, which is why effective INT4 compression is about 3.9× rather than 4×.
• For a real deployment: benchmark the candidate kernels on the target model and shape, then check quality on your own evaluation tasks. Every recommendation here is a starting hypothesis.

---

Try it yourself

1. 01Using the simplified weight-traffic decode model, derive the crossover batch size B* for A100 (2039 GB/s, 312 TFLOP/s FP16, 624 TOPS INT8) and H100 (3350 GB/s, 989 TFLOP/s FP16, 1979 TFLOP/s FP8). State one assumption that would move the real crossover.

   Show a hint

   B* = FP16 peak / (2 × bandwidth). A100: 312e12 / (2 × 2039e9) = 76.5 ≈ 77. H100: 989e12 / (2 × 3350e9) = 147.6 ≈ 148. The model counts weight traffic only, so KV-cache reads at long context, or a kernel that cannot reach the dense peak, move the real crossover.

2. 02Given activation channel 112 with max|X| = 80, other channels with max|X| ≈ 0.5, and weights with max|W| ≈ 0.02, compute s_j for channel 112 at α = 0, 0.5, 0.85 and 1.0, then say what each choice does to the activation and weight maxima.

   Show a hint

   s_j = max|X_j|^α / max|W_j|^(1-α). Channel 112: α=0 gives 1/0.02 = 50; α=0.5 gives sqrt(80/0.02) = 63.25; α=0.85 gives 74.55; α=1 gives 80. The smoothed activation max is 80/s_j and the scaled weight max is 0.02 × s_j, so α=0 gives 1.6 and 1.0, α=0.5 gives 1.27 and 1.26, α=0.85 gives 1.07 and 1.49, and α=1 gives 1.0 and 1.6. The balanced split has the two ranges closest together.

3. 03A 4096×4096 weight matrix is quantized to INT4 with group size G=128. Calculate the total memory footprint (weights + scales) in MiB and the metadata overhead percentage. Compare with the FP16 baseline.

   Show a hint

   INT4 payload: 16,777,216 / 2 = 8,388,608 bytes = 8.00 MiB. Scales: (16,777,216 / 128) × 2 = 262,144 bytes = 0.25 MiB. Total 8.25 MiB. Overhead: 262,144 / 8,388,608 = 3.125%. Against FP16 (32.00 MiB) that is 3.88× compression, not 4×.

4. 04For a 70B model on H100 at B=32 during decode, which scheme does the simplified model favour on weight traffic: W4A16 (INT4 G=128) or FP8? Then name two effects the model leaves out that could reverse the answer.

   Show a hint

   I_W4A16 = 3.88 × 32 ≈ 124 FLOP/B, below the 295 FLOP/B FP16 ridge. I_FP8 = 2 × 32 = 64 FLOP/B, below the 591 FLOP/B FP8 ridge. Both are bandwidth-bound, and W4A16 moves 0.516 bytes per parameter against 1.0, so it is about 1.94× better on weight transit. It ignores KV-cache traffic (which can dominate at B=32 with long context) and kernel efficiency (dequantization may not reach the ideal traffic ratio).

5. 05Verify the FP8 E4M3FN and E5M2 encodings: max finite value, min positive normal, and ULP at 1.0. Derive 448 and 57344 from the bit layouts, then identify the E4M3 NaN encoding with each sign bit.

   Show a hint

   E4M3FN: exponent 15, mantissa 6 gives (1 + 6/8) × 2^(15-7) = 1.75 × 256 = 448; min normal is 2^-6; ULP at 1.0 is 2^-3 = 0.125. E5M2: exponent 30, mantissa 3 gives (1 + 3/4) × 2^(30-15) = 1.75 × 32768 = 57344; min normal is 2^-14; ULP at 1.0 is 2^-2 = 0.25. E4M3FN reclaims the infinity patterns for finite values in the last binade. Exponent 1111 with mantissa 111 encodes NaN with either sign: 0x7F or 0xFF.

---

Sources and scope

Topic	Primary source	Scope used here
FP8 E4M3 and E5M2	OCP OFP8 specification, §5.1 Table 2	Encodings, special values, range and conversion modes.
SmoothQuant	Xiao et al., ICML 2023, §§3–5 Tables 1, 3 and 6	Input-channel smoothing, outer-dimension quantization scales, and the pinned OPT-175B and original LLaMA results in §6.1.
AWQ	Lin et al., MLSys 2024, §3.2	Activation-aware scaling of salient weight channels. No AWQ benchmark number is used here.
GPU peak specifications	NVIDIA H100 whitepaper, Tables 1 and 2; NVIDIA A100 80 GB datasheet	Dense A100/H100 Tensor Core peaks and the A100 80 GB HBM2e bandwidth assumption.
MXFP4 and NVFP4	OCP MX specification, Table 1; NVIDIA NVFP4 description	Element format, block length and scale type only; the table is not a quality or speed benchmark.
Synthetic examples	scripts/verify-low-bit-weights-activations.py, parts 2 and 5	Part 2 uses a +80 activation in channel 112. Part 5 separately quantizes a weight matrix with one 0.80 row; its 20.8/43.3/44.6 dB SQNR values are weight results. Neither part measures model quality or GPU latency.

The batch crossover is an original calculation from a simplified weight-traffic-only decode model, not a published or measured deployment threshold. The source-backed quality table reproduces only the named model and metric rows cited above. Software and hardware versions can change; verify a target serving stack before applying the decision guide.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Quantize Without Losing the ResultQuantization: Scale, Zero Point, and Granularity15 min
• Inference Optimisation & AI Infrastructure · Quantize Without Losing the ResultCalibration Data and Quality Regression18 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryBatching and Throughput26 min