llama.cpp on GMKtec EVO-X2: setup and coding review
CJ explains a 128GB mini PC setup with Fedora and llama.cpp, then compares everyday prompting and coding with his Claude workflow.
Player not loading? Watch on YouTube
CJ walks through his GMKtec EVO-X2 setup and explains how model size, quantization and context affect memory use. His 128GB unified-memory machine runs Fedora, with up to 108GB allocated for GPU use and about 20GB left for the operating system. He reports that larger allocations can cause kernel panics.
For local LLM inference, he chooses llama.cpp and community AMD Strix Halo Toolboxes to avoid compiling it and configuring dependencies manually. He reports better performance with the ROCm toolbox in his tests. The tutorial also covers downloading models through the Hugging Face CLI and checking model cards for compatible formats and memory requirements.
Model fit has practical limits. CJ says a 101GB, 3-bit MiniMax M2.5 quantization worked, but left room for only about 4,000 context tokens and slowed with longer prompts. The 4-bit Qwen3-Coder-30B-A3B-Instruct download is under 20GB, leaving more memory for context.
He connects Open WebUI to his llama server for questions and web search. His coding assistant trials with Continue need intervention on configuration and file placement. He prefers OpenCode for its subagents and browser dashboard, and recommends specifications, tests, linting and type checks. Basic questions have replaced some of his hosted chat usage, but he keeps Claude for complex coding work.