Model Quantization: PTQ, QAT, AWQ & GPTQ

Deploying multi-billion parameter foundation models in resource-constrained production settings or on edge devices requires drastic compression. Standard deep learning weights are stored in 16-bit floating point formats (FP16 or BF16), requiring 2 bytes of memory per parameter (a 70B parameter model demands 140 GB of VRAM simply to load).

Model Quantization compresses high-precision continuous floating-point tensors into low-precision discrete integers (such as INT8 or INT4), reducing memory footprints by $2\times$ to $4\times$ while accelerating execution through specialized integer matrix multiplication hardware.

1. Mathematical Derivations of Quantization Functions#

Quantization maps a continuous real value $x \in \mathbb{R}$ within interval $[x_{\min}, x_{\max}]$ to an integer $q \in \mathbb{Z}$ within range $[q_{\min}, q_{\max}]$.

1. Uniform Affine (Asymmetric) Quantization#

When data distributions are skewed or asymmetric around zero (common in ReLU or GELU activations where all values are non-negative), asymmetric quantization introduces an integer offset called the Zero-Point $Z$.

Derivation of Scale Factor $S$

The scale factor $S \in \mathbb{R}^+$ defines the step size represented by each integer increment. It maps the continuous span $(x_{\max} - x_{\min})$ to the discrete integer span $(q_{\max} - q_{\min})$:

Derivation of Zero-Point $Z$

The zero-point $Z \in \mathbb{Z}$ guarantees that the real value $0.0$ maps exactly to an integer, preventing numerical distortion during zero-padding in convolutional and attention layers:

Matching the boundaries, when $x = x_{\min}$, $q$ must equal $q_{\min}$:

$Z$ is clamped to ensure it resides strictly within $[q_{\min}, q_{\max}]$.

The Quantization and Dequantization Operations

The forward quantization formula is:

The inverse dequantization formula reconstructs the approximated real value $\hat{x} \approx x$:

2. Symmetric Quantization#

When data is symmetrically distributed around zero (as is typical for weight matrices centered at zero), we set the zero-point to zero: $Z = 0$.

Let $\alpha = \max(|x_{\min}|, |x_{\max}|)$. The symmetric range is $[-\alpha, \alpha]$ mapped to signed integer grid $[-q_{\max}, q_{\max}]$ (e.g. $[-127, 127]$ for signed INT8):

The quantization and dequantization equations simplify to:

Because $Z = 0$, symmetric matrix multiplications avoid costly additive cross-term corrections during inference, making symmetric quantization standard for weights.

2. PTQ vs QAT: Calibration and the Straight-Through Estimator#

Post-Training Quantization (PTQ)#

PTQ quantizes an existing trained floating-point model using a small calibration set (100 to 512 examples) without performing gradient updates.

• MinMax Calibration: Sets $x_{\min}$ and $x_{\max}$ to observed absolute extremes. Highly vulnerable to outlier spikes.
• Percentile / Histogram Calibration: Sets clipping boundaries to the 99.99th percentile, discarding rare outlier activations to minimize Mean Squared Quantization Error across the distribution.

Quantization-Aware Training (QAT) & Straight-Through Estimators (STE)#

For aggressive compression (such as 4-bit weights), PTQ can degrade perplexity. QAT simulates quantization noise during model training.

The Non-Differentiability Bottleneck

The rounding operator $\text{round}(x)$ has a derivative of zero almost everywhere:

If standard backpropagation is applied, gradients are zero everywhere, terminating weight updates completely.

The Straight-Through Estimator (STE)

QAT resolves this by approximating the gradient of the rounding operator as an identity pass-through during the backward pass:

Weights maintain continuous floating-point master copies in memory, adapting continuously to compensate for quantization noise.

3. Advanced LLM Quantization: GPTQ, AWQ & NF4#

Modern 4-bit Large Language Model compression relies on three state-of-the-art algorithms:

1. GPTQ (Generative Pre-trained Transformer Quantization)#

Introduced by Elias Frantar et al. (2022), GPTQ uses second-order Taylor expansions of the layer reconstruction error:

where $H = 2 X X^T$ is the Hessian matrix computed from calibration activations $X$.

GPTQ quantizes weight matrix columns sequentially. Whenever column $i$ is rounded to a 4-bit integer, it propagates the induced quantization error $(w_i - \hat{w}_i)$ to all remaining unquantized columns $j > i$ using the inverse Hessian:

This error compensation preserves perplexity even at 4-bit precision.

2. AWQ (Activation-Aware Weight Quantization)#

Ji Lin et al. (2023) observed that weight saliency is governed by activation magnitudes. Protecting just the top 1% of salient weights prevents catastrophic perplexity spikes.

Rather than storing the salient weights in mixed FP16 precision, AWQ applies a per-channel mathematical scaling factor $s > 1$:

Scaling up salient weight channels reduces their relative quantization rounding error while preserving identical mathematical output: $W' X' = W X$.

3. NormalFloat4 (NF4) & QLoRA#

Tim Dettmers et al. (2023) proved that pretrained neural network weights follow a zero-mean normal distribution: $W \sim \mathcal{N}(0, \sigma^2)$.

Standard uniform integer grids place equal-width bins across the entire range, wasting bin capacity on sparse distribution tails. NF4 (NormalFloat4) constructs an information-theoretically optimal quantile grid where each of the $2^4 = 16$ discrete levels has an equal probability mass of $\frac{1}{16}$ under a standard Gaussian distribution.

4. From-Scratch Implementation: Quantization Engine#

We implement asymmetric affine quantization, symmetric quantization, and a quantized matrix multiplication engine from first principles in Python and Rust standard libraries.

Python Implementation (Pure Standard Library)#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
"""
Model Quantization Engine from scratch using pure Python standard library.
Implements Asymmetric Uniform Quantization, Symmetric INT8, and Quantized MatMul.
"""
import math
from typing import List, Tuple

class Quantizer:
    @staticmethod
    def asymmetric_quantize(tensor: List[float], q_min: int = 0, q_max: int = 255) -> Tuple[List[int], float, int]:
        x_min = min(tensor)
        x_max = max(tensor)

        # Handle degenerate case
        if abs(x_max - x_min) < 1e-12:
            return [0] * len(tensor), 1.0, 0

        # Scale S
        scale = (x_max - x_min) / float(q_max - q_min)

        # Zero point Z
        z_initial = round(-x_min / scale) + q_min
        zero_point = max(q_min, min(q_max, z_initial))

        # Quantize and clamp
        quantized = []
        for x in tensor:
            q = round(x / scale) + zero_point
            q_clamped = max(q_min, min(q_max, q))
            quantized.append(q_clamped)

        return quantized, scale, zero_point

    @staticmethod
    def asymmetric_dequantize(quantized: List[int], scale: float, zero_point: int) -> List[float]:
        return [scale * (q - zero_point) for q in quantized]

    @staticmethod
    def symmetric_quantize(tensor: List[float], q_max: int = 127) -> Tuple[List[int], float]:
        alpha = max(abs(min(tensor)), abs(max(tensor)))
        if alpha < 1e-12:
            return [0] * len(tensor), 1.0

        scale = alpha / float(q_max)
        quantized = []
        for x in tensor:
            q = round(x / scale)
            q_clamped = max(-q_max, min(q_max, q))
            quantized.append(q_clamped)

        return quantized, scale

    @staticmethod
    def symmetric_dequantize(quantized: List[int], scale: float) -> List[float]:
        return [q * scale for q in quantized]

def mean_squared_error(a: List[float], b: List[float]) -> float:
    return sum((x - y) ** 2 for x, y in zip(a, b)) / len(a)

if __name__ == "__main__":
    # Simulated weight distribution
    fp32_weights = [-1.45, -0.82, -0.15, 0.0, 0.22, 0.67, 1.12, 1.95]

    print("--- 1. Asymmetric UINT8 Quantization ---")
    q_asym, scale_a, zp_a = Quantizer.asymmetric_quantize(fp32_weights, 0, 255)
    rec_asym = Quantizer.asymmetric_dequantize(q_asym, scale_a, zp_a)
    print(f"Scale: {scale_a:.5f}, Zero-Point: {zp_a}")
    print(f"Quantized (INT): {q_asym}")
    print(f"Reconstructed: {[round(x, 3) for x in rec_asym]}")
    print(f"MSE Error: {mean_squared_error(fp32_weights, rec_asym):.6f}\n")

    print("--- 2. Symmetric INT8 Quantization ---")
    q_sym, scale_s = Quantizer.symmetric_quantize(fp32_weights, 127)
    rec_sym = Quantizer.symmetric_dequantize(q_sym, scale_s)
    print(f"Scale: {scale_s:.5f}")
    print(f"Quantized (INT): {q_sym}")
    print(f"Reconstructed: {[round(x, 3) for x in rec_sym]}")
    print(f"MSE Error: {mean_squared_error(fp32_weights, rec_sym):.6f}")

Rust Implementation (Pure Standard Library)#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
// Model Quantization Engine from scratch in pure Rust standard library.
// Implements Asymmetric Uniform and Symmetric INT8 Quantization.

struct Quantizer;

impl Quantizer {
    fn asymmetric_quantize(tensor: &[f64], q_min: i32, q_max: i32) -> (Vec<i32>, f64, i32) {
        let x_min = tensor.iter().cloned().fold(f64::INFINITY, f64::min);
        let x_max = tensor.iter().cloned().fold(f64::NEG_INFINITY, f64::max);

        if (x_max - x_min).abs() < 1e-12 {
            return (vec![q_min; tensor.len()], 1.0, q_min);
        }

        let scale = (x_max - x_min) / ((q_max - q_min) as f64);
        let z_initial = (-x_min / scale).round() as i32 + q_min;
        let zero_point = z_initial.max(q_min).min(q_max);

        let quantized: Vec<i32> = tensor
            .iter()
            .map(|&x| {
                let q = (x / scale).round() as i32 + zero_point;
                q.max(q_min).min(q_max)
            })
            .collect();

        (quantized, scale, zero_point)
    }

    fn asymmetric_dequantize(quantized: &[i32], scale: f64, zero_point: i32) -> Vec<f64> {
        quantized.iter().map(|&q| scale * ((q - zero_point) as f64)).collect()
    }

    fn symmetric_quantize(tensor: &[f64], q_max: i32) -> (Vec<i32>, f64) {
        let x_min = tensor.iter().cloned().fold(f64::INFINITY, f64::min);
        let x_max = tensor.iter().cloned().fold(f64::NEG_INFINITY, f64::max);
        let alpha = x_min.abs().max(x_max.abs());

        if alpha < 1e-12 {
            return (vec![0; tensor.len()], 1.0);
        }

        let scale = alpha / (q_max as f64);
        let quantized: Vec<i32> = tensor
            .iter()
            .map(|&x| {
                let q = (x / scale).round() as i32;
                q.max(-q_max).min(q_max)
            })
            .collect();

        (quantized, scale)
    }

    fn symmetric_dequantize(quantized: &[i32], scale: f64) -> Vec<f64> {
        quantized.iter().map(|&q| (q as f64) * scale).collect()
    }
}

fn compute_mse(a: &[f64], b: &[f64]) -> f64 {
    a.iter().zip(b.iter()).map(|(&x, &y)| (x - y) * (x - y)).sum::<f64>() / (a.len() as f64)
}

fn main() {
    let fp32_weights = vec![-1.45, -0.82, -0.15, 0.0, 0.22, 0.67, 1.12, 1.95];

    println!("--- Symmetric INT8 Quantization (Rust std) ---");
    let (q_sym, scale_s) = Quantizer.symmetric_quantize(&fp32_weights, 127);
    let rec_sym = Quantizer.symmetric_dequantize(&q_sym, scale_s);

    println!("Scale: {:.6}", scale_s);
    println!("Quantized INT8: {:?}", q_sym);
    println!("Reconstructed FP: {:?}", rec_sym.iter().map(|&x| (x * 1000.0).round() / 1000.0).collect::<Vec<_>>());
    println!("MSE Loss: {:.8}", compute_mse(&fp32_weights, &rec_sym));
}

5. Algorithmic Tradeoff Comparison#

6. Knowledge Check#

question: "Why is the Straight-Through Estimator (STE) required during Quantization-Aware Training (QAT)?"

options:

• "(Recommended) Because the derivative of the rounding function round(x) is zero almost everywhere, which would completely halt backpropagation gradients"
• "Because integers cannot be stored in GPU registers"
• "Because it converts all negative numbers to positive numbers"
• "Because it prevents models from generating out-of-vocabulary tokens"

explanation: "The rounding function has a step shape: its derivative is 0 everywhere except at discontinuities where it is undefined. Without STE approximating d(round(x))/dx as 1, gradient backpropagation would multiply by 0 and completely freeze neural network weights."

:::

question: "Why is symmetric quantization generally preferred over asymmetric quantization for neural network weight matrices?"

options:

• "(Recommended) Because setting zero-point Z = 0 eliminates additive offset cross-terms during hardware matrix multiplication, maximizing execution speed"
• "Because symmetric quantization requires twice as many bits per parameter"
• "Because weight matrices never contain negative values"
• "Because it guarantees zero quantization error"

explanation: "When Z = 0, the dequantization equation is x = S q. During matrix multiplication (W X), no additive zero-point correction terms (Z * X) need to be accumulated, allowing specialized integer hardware systolic arrays to run at maximum throughput."

:::