llama.cpp: Windows setup, GGUF models and GPU tuning

This tutorial explains how to run models locally with llama.cpp on Windows. The presenter downloads compiled releases and selects CUDA 13 with its accompanying CUDA 13.3 libraries for an NVIDIA RTX 3060 with 12 GB. Vulkan and CPU builds appear as alternatives for other hardware.

The walkthrough covers finding GGUF files on Hugging Face, choosing a quantization and downloading a separate vision encoder for image input. It then starts llama-server from CMD, tests its browser chat and saves the launch command in a .BAT file. The server also provides an OpenAI-compatible endpoint, which the presenter connects to OpenCode for simultaneous requests.

Memory tuning includes context size, quantized KV cache and sharing resources across four response slots. The MTP speculative decoding example encounters a useful limitation: the first downloaded model lacks the required MTP component. The presenter replaces it with a variant that includes it and reports higher generation speed, with results varying by task.

A final example loads a Qwen 3.6 35B A3B mixture-of-experts model whose displayed download size is 22.2 GB. The presenter splits work across GPU, CPU and RAM, then adjusts expert placement to use more VRAM. He reports about 44 tokens per second for a coding prompt after tuning. These figures describe his demonstration, rather than expected performance on every PC.