llama.cpp setup on Windows 11: models and GPU tuning

This tutorial explains how to run models locally with llama.cpp on Windows 11. The presenter uses a Minisforum A1 X1 Pro with 64 GB of shared RAM and assigns 32 GB to its integrated GPU through the BIOS. The demonstration uses that GPU rather than the connected RTX 5060 Ti.

The setup covers downloading the Windows Vulkan build, selecting a quantized Qwen model, and downloading a separate projector file for image input. The presenter loads a roughly 22 GB Q4 model, starts the server, and opens its chat interface at localhost:8080. The source description names Qwen 3.5, while the transcript uses inconsistent model names, so the exact model version is unclear.

The local LLM demonstrations include HTML games and a generated VR headset comparison page. The presenter reports about 24 tokens per second in a short exchange and roughly 19 to 21 during a later task. These are results from the demonstrated setup.

The configuration discussion covers a 60,000-token context, GPU layer allocation, automatic fitting for smaller graphics cards, and KV cache settings. The presenter warns that browser cache clearing removes chat history and says the file-based memory setup needs explicit prompting rather than acting proactively.