llama.cpp, Ollama and MLX: local model runners compared

This overview explains the work behind a local LLM runner and compares several ways to run models locally. The speaker presents llama.cpp as the engine beneath many familiar apps, including Ollama, while noting that some projects take a more specialized approach.

The technical explanation covers GGML's hardware math, quantization, tokenization and model-specific chat formats. Incorrect formatting can reduce answer quality or break an AI agent's tool calls. Conversation caches also consume memory; the speaker explains that a long history can use more RAM than a small model's weights. Temperature and speculative decoding round out the discussion of how runners choose tokens and improve speed.

The comparisons come with tradeoffs. The speaker describes MLX as limited to Apple hardware, with speed gains concentrated in prompt processing on smaller models. ik_llama.cpp focuses on compression and dividing large models between GPU and system memory. ds4 supports a single DeepSeek model.

For setup, the video introduces LlamaBarn on Mac, with model downloads, browser chat and a local API. It also covers llama.cpp installation through winget on Windows or Homebrew on Mac. The closing comparison contrasts Ollama's agent connections, LM Studio's graphical model browsing, Jan's open source desktop app and KoboldCpp's fiction and roleplay focus.