GGUF explained: inspect headers and compare quantization

This tutorial explains the GGUF model format used with Ollama and llama.cpp. It covers the file layout, a Python header inspection example, and the tradeoffs involved in choosing quantization for a local LLM.

The speaker divides the file into a header, metadata key-value pairs, tensor information, alignment padding and tensor data. Metadata describes details such as architecture and vocabulary size, while tensor information tells the loader where to find model data. The inspection example reports version 3, 733 tensors and 54 metadata pairs without loading the full weights. One detail needs care: the narration calls the header 20 bytes, but the fields read in the demonstration total 24 bytes.

The quantization discussion compares FP32 and FP16 with options including Q8_0, Q4_0 and K-quants. The speaker describes K-quants as allocating bits more efficiently and suggests Q4_K_M as a starting point, with higher precision when quality and available space permit. These are the speaker's recommendations; the stated file sizes and memory savings should not be treated as universal requirements.

The final section decodes filenames using Grok and Qwen 3.6 examples. It explains model names, size labels, versions and quantization labels, while noting that publishers do not always follow the naming convention consistently.