Quantisation formats and what they cost

This page is for anyone deciding how to fit a model onto the hardware they have. Each row gives the storage cost, the memory for the weights of a 7 B and a 70 B dense model, what the format’s own paper or maintainers report about accuracy, and where the kernels exist. Weights are only part of the budget: the KV cache, activations and optimiser state come on top, and the KV cache dominates at long context. For the background see /topics/efficient-ml/ and /topics/hardware-and-acceleration/.

Number formats

Memory figures are weights only, computed as parameters times bytes per parameter, rounded. “Bits” counts the scales and zero points that block formats carry alongside the weights.

Model-file formats

These are packaging schemes that carry their own quantisation, used for inference rather than training.

The three 4-bit families land within 0.1 bytes per parameter of each other. Pick by where you run: GGUF for llama.cpp and Ollama, AWQ or GPTQ for vLLM on NVIDIA, NF4 for fine-tuning with QLoRA. Converting between them means re-quantising from the 16-bit original, not from each other.

What the numbers hide

• Weights are not the whole budget. A 7 B model at 4-bit is 4 GB of weights, but a 32 k context with a 32-layer, 8 KV-head model adds about 1 GB of KV cache per sequence at FP16. Serving engines quantise the cache too (FP8 KV cache in vLLM and TensorRT-LLM, Q8 in llama.cpp).
• Perplexity is the metric every paper reports and it understates damage on tasks with a single right answer, such as code and maths. Check a downstream benchmark on your task, with a confidence interval (see /reference/evaluation-metrics/).
• Small models lose more. The papers’ headline numbers are for 65 B to 175 B models. A 1 B or 3 B model at 4-bit can lose several points on reasoning benchmarks.
• Weight-only formats help memory and the memory-bound decode phase. They do not speed up prefill, which is compute-bound and runs in 16-bit after dequantisation. W8A8 and FP8 do speed up both.
• Quantised fine-tuning (QLoRA) trains adapters in 16-bit on top of frozen 4-bit weights. The base model’s rounding error is baked in.
• Speed depends on the kernel more than the format. The same AWQ checkpoint can run faster or slower than FP16 depending on batch size and GPU generation. Measure.

A worked int8 quantisation

Symmetric int8 maps the range [-max|w|, max|w|] onto [-127, 127] with one scale. Per-tensor scaling lets a single outlier waste most of the levels. Per-row (per-channel) scaling fixes that at the cost of one scale per output row, and is what every INT8 scheme in the table does.

import numpy as np

rng = np.random.default_rng(0)
w = rng.normal(0.0, 0.02, size=(256, 1024)).astype(np.float32)   # a weight matrix like a small linear layer
w[3, 17] = 0.5                                                   # one outlier, as real weights have

def quantise_int8_symmetric(x: np.ndarray, axis=None):
    """Symmetric int8: q = round(x / s), s = max|x| / 127. axis=None is per-tensor, axis=1 per output row."""
    amax = np.abs(x).max() if axis is None else np.abs(x).max(axis=axis, keepdims=True)
    scale = np.maximum(amax, 1e-12) / 127.0
    q = np.clip(np.rint(x / scale), -128, 127).astype(np.int8)
    return q, scale.astype(np.float32)

def dequantise(q, scale):
    return q.astype(np.float32) * scale

for name, axis in [("per-tensor", None), ("per-row", 1)]:
    q, s = quantise_int8_symmetric(w, axis)
    w_hat = dequantise(q, s)
    err = w_hat - w
    rel = np.linalg.norm(err) / np.linalg.norm(w)
    zeros = (q == 0).mean()
    print(f"{name:11} bytes/param 1.0  relative error {rel:.4f}  max abs error {np.abs(err).max():.5f}  weights rounded to zero {zeros:.1%}")

print("fp32 bytes:", w.nbytes, " int8 bytes:", w.size, " ratio:", w.nbytes / w.size)

per-tensor  bytes/param 1.0  relative error 0.0568  max abs error 0.00197  weights rounded to zero 7.8%
per-row     bytes/param 1.0  relative error 0.0087  max abs error 0.00197  weights rounded to zero 1.1%
fp32 bytes: 1048576  int8 bytes: 262144  ratio: 4.0

One outlier at 0.5 in a matrix of weights around 0.02 makes the per-tensor scale 25 times coarser than the rows need, rounds 7.8 % of the weights to zero and multiplies the error by six. Per-row scaling leaves one bad row and 255 good ones. Group-wise scaling (every 32 or 128 weights, as in GGUF, GPTQ and AWQ) is the same idea taken further, which is why those formats carry scales and why their bits per weight are not round numbers.

Sources

• Kalamkar et al., A Study of BFLOAT16 for Deep Learning Training, 2019
• Micikevicius et al., Mixed Precision Training, 2017
• Micikevicius et al., FP8 Formats for Deep Learning, 2022
• Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, 2022
• Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, 2022
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023
• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022
• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, 2023
• GGUF specification and the llama.cpp quantisation tool
• NVIDIA TensorFloat-32 in the A100 (TF32)
• PyTorch numerical accuracy and TF32 notes
• bitsandbytes documentation and vLLM quantisation support