Ollama, Unsloth and NVIDIA: quantization tradeoffs

This panel brings together speakers from NVIDIA, Unsloth, Hugging Face and Ollama to discuss how quantization helps people run models locally. It covers compression choices and evaluation rather than a step-by-step installation.

Daniel Han describes shrinking GLM 5.2 from 1.5 TB to 250 GB through mixed precision. His explanation centers on unequal sensitivity: some weights need higher precision, while others tolerate much smaller representations. These are reported results, not a guarantee that the same compression preserves quality in every model.

NVIDIA's Asma Beevi explains NVFP4 as a four-bit floating-point format with an FP8 scale shared across 16 values. She says the team targets less than 1% accuracy degradation on its benchmark suite. She also describes post-training quantization as generally easier for larger models, while models below roughly 20 billion parameters may need quantization-aware distillation to recover accuracy.

For local LLM users, the practical question is whether a checkpoint still works in the intended harness. Ollama's Parth Sareen describes testing default quantizations and sometimes retaining higher precision for small models. Han warns that quantized linear attention can appear sound until long-context use produces gibberish. The closing discussion considers KL divergence between BF16 and quantized output logits as another evaluation signal, alongside benchmarks and actual use.