Qwen 3.5, Ollama and GGUF quantization explained

This beginner tutorial explains how quantization changes the memory needed to run models locally. The speaker describes GGUF as a file format containing model weights and metadata, then uses image compression as an analogy for the quality tradeoffs of lower precision.

Qwen 3.5 27B is the main example for reading quantization labels. The walkthrough covers Q4 and Q8, suffixes such as S, M and XL, and Unsloth's UD prefix. The speaker compares providers' quantization recipes and recommends testing choices against the model, available hardware and intended task. A separate explanation distinguishes one-bit quantization from models trained with one-bit weights.

For Ollama, the speaker checks tags and matching hashes to show that the displayed Qwen 3.5 latest tag points to a 9B Q4KM variant. This is a specific example viewers can use when checking which local LLM variant they downloaded.

The memory discussion separates model weights, context storage and system overhead. In the speaker's rough calculation, an 8B model needs about 8 GB for Q8 weights or 4 GB for Q4 weights, before context and overhead. The tutorial also shows where to find architecture values in config.json for a KV cache estimate. These calculations are presented as guidance with limitations: context requirements vary, and tools such as llama.cpp can compress the cache. The speaker also cautions that low perplexity does not guarantee accurate answers.