llama.cpp setup and tuning with Qwen 3.6 and Gemma 4

This tutorial explains how to run models locally with llama.cpp, starting with release downloads, hardware backends and the llama-server browser interface. The speaker selects Vulkan on Linux and recommends CUDA for Nvidia GPUs. The starter example assumes at least 8GB of system RAM, rather than 8GB of VRAM. The guide also discusses macOS, Windows and Android downloads.

Configuration covers a model.ini file, loading one model at a time and unloading idle models after 900 seconds. The context discussion explains how parallel slots divide the available token budget and why a single user might prefer one slot.

The speaker favors Qwen 3.6 or 3.5 for coding and reasoning, and Gemma 4 for writing and general chat. These are the speaker's assessments. Dense and mixture-of-experts models introduce different memory and speed tradeoffs, which the guide relates to consumer hardware.

Later sections cover weight quantization, KV cache precision, speculative decoding and compiling llama.cpp. The speaker warns that cache compression can hurt long-context work and tool use, especially with some MoE models. The supplied description adds that Gemma 4 needs QAT quants for KV cache quantization and recommends fit-target over manual fitting. Performance gains depend on hardware and workload. The closing advice favors checking documentation and testing forks or fine-tunes before trusting their claims.