llama.cpp setup: JSON output, local API and Chatbox

This tutorial explains how to run models locally with llama.cpp and adjust settings beyond a basic chat interface. The presenter installs the toolbox on Windows with winget, discusses Homebrew installation on macOS and Linux, and explains GGUF model files and GGML's role in computation.

The comparison with Ollama focuses on direct access to inference settings and the absence of a background management layer. The speaker recommends llama.cpp for one or a few users running quantized models, while favoring vLLM for batched GPU serving with many users. These are the presenter's recommendations, rather than measured results from a benchmark in this tutorial.

The tuning walkthrough covers sampling controls, grammar-constrained output, tool calling, context and KV-cache memory, GPU offloading, and hardware backends. It also includes cautions: aggressive cache quantization can degrade tool calling, and a poorly matched draft model can make speculative decoding slower.

The practical examples compare prose with schema-based JSON output, then serve Llama 3.2 3B Instruct through llama-server's OpenAI-compatible API. The final demo connects the Windows Chatbox app by setting its API host to the local server and selecting the model. The server must stay running for Chatbox to keep receiving responses.