bitsandbytes LLM.int8(): quantization foundations

This quantization lesson explains the mixed-precision LLM.int8() method implemented in bitsandbytes and compares it with a BF16 baseline and naive 4-bit rounding. It is a foundations lecture with reported local experiments, rather than an installation guide. Qwen 2.5 7B provides the example: roughly 15 GB of BF16 weights must run on an RTX 3060 with 12 GB of VRAM.

The foundations cover rounding error, scale, zero-point and symmetric versus asymmetric mappings. Per-tensor quantization shares a scale across a matrix; per-channel and per-group approaches reduce how far outliers can affect other weights. The speaker explains round-to-nearest and ZeroQuant before introducing LLM.int8(), which keeps outlier channels in FP16 while applying vector-wise 8-bit quantization elsewhere.

The practical section compares WikiText-2 perplexity and generation speed. The reported BF16 baseline scores 7.46 perplexity and 0.69 tokens per second, with about 4 GB of weights offloaded to the CPU. LLM.int8() reaches about 7.4 perplexity and runs 7.3 times faster in this test. These measurements describe one hardware setup and an offloaded baseline.

Naive 4-bit per-tensor rounding produces perplexity around 45,000. Per-group rounding recovers a score of 7.69 but produces only 0.75 tokens per second. These are naive rounding experiments, not an evaluation of bitsandbytes NF4 or QLoRA. The speaker uses them to explain why smaller weight representations alone do not guarantee faster inference, then introduces GPTQ as the next method in the series.