Qwen3.6 setup with llama.cpp and TurboQuant

This tutorial builds a local LLM setup using Tom's TurboQuant branch of llama.cpp and an Unsloth Qwen3.6-35B-A3B GGUF. The speaker demonstrates the build on Apple silicon with Metal and directs Nvidia and AMD users to the relevant repository instructions. The workflow covers cloning the repository, checking out the branch, building the binaries and choosing a model file.

Model size and context length drive the memory discussion. The demonstrated machine is an M1 Max with 64 GB of RAM, despite the different hardware listed in the description. The speaker selects a roughly 38.45 GB, 8-bit model and sets context to 262,144 tokens. He describes asymmetric KV cache quantization with 8-bit keys and Turbo 3 or Turbo 4 values, attributing the recommendation to Tom's testing. His reported generation rate is about 53 tokens per second; codebase prefill takes longer, and he does not wait for that response during the demonstration.

The second half switches from the CLI to llama-server at 127.0.0.1:8080. Kilo Code in VS Code connects through a custom provider using the /v1 endpoint. OpenClaw configuration covers the provider, model ID, context size and primary or fallback model selection. The speaker advises reducing context when memory is tight and limiting other applications on a unified-memory Mac. His opening timeout complaints and closing reliability claims describe his own experience.