llama.cpp setup: local serving and whisper.cpp

Katharine Jarmul introduces llama.cpp for serving quantized models on personal hardware. The tutorial covers a local LLM workflow on a GPU machine, with a separate whisper.cpp speech transcription example on her three-year-old MacBook Air.

Jarmul recommends compiling llama.cpp for the target hardware if you are comfortable with a terminal. She says this worked better than packaged binaries on her GPU, while suggesting binaries as a starting point for less experienced users. The CLI example loads a model from Hugging Face or an existing cache. Repositories that require authentication need a token; choosing a quantization also means checking whether the model fits the available hardware.

The server walkthrough uses configuration files for model presets, sampling parameters and a Jinja chat template for tool calls. Jarmul describes switching between quantized models during a task as an advantage over how she uses vLLM. She binds the server to 0.0.0.0 for access across her LAN, then tests a chat request from another machine with curl before connecting a more complex client. An initial Gemma request does not load, so she retries with a working model. She describes the response format as compatible with OpenAI endpoint syntax and says Anthropic syntax also works.

The final demonstration uses whisper.cpp with Core ML support and prepared models. Jarmul recommends local transcription for sensitive audio, though this segment is a brief example rather than a full setup guide.