GGUF vs AWQ vs GPTQ: quantization trade-offs explained

This comparison explains how quantization changes model memory use, inference speed and accuracy. The speaker distinguishes post-training quantization (PTQ), which converts trained weights, from quantization-aware training (QAT), which accounts for lower precision during training.

GPTQ uses second-order information to compensate for rounding errors across weights. AWQ uses activations to identify channels that need protection before quantization. The examples show why bit depth alone does not predict accuracy: the speaker reports that simple rounding beats GPTQ for Llama 7B at three bits, while AWQ beats GPTQ for Llama 2 70B at the same precision.

GGUF is a file format for weights and metadata, rather than a quantization algorithm. For readers choosing a local LLM, the discussion explains Q4_K_M and compares it with Q8_0 and Q2_K_small. The speaker reports 72 tokens per second for a Q4_K_M Llama 3 8B example versus 29 in FP16, but those figures do not establish a universal speedup across hardware.

The speaker recommends four-bit PTQ as a practical balance. Lower precision can damage accuracy sharply, while the cited LLM.int8 example saves memory without accelerating inference. BitNet b1.58 takes a different approach with ternary weights; the presentation says it requires training from scratch rather than converting an existing model.