llama.cpp, Unsloth and Pi: local model setup

This beginner livestream explains how to run models locally, with separate demonstrations of inference software, quantization and coding agents. Ben Burtenshaw distinguishes downloadable open weights from open source models with access to training code and data. He also separates running a model on your hardware from using a hosted inference provider.

Merve Noyan introduces llama.cpp, its GGUF model format and an OpenAI-compatible HTTP server. Her walkthrough covers Hugging Face Hub hardware compatibility information, commands from llama.app and a web interface at localhost:8080. She demonstrates Pi generating a website with Gemma 4 E4B and explains how context length adds memory requirements.

Daniel Han explains Unsloth's dynamic quantization work and discusses MTP acceleration for supported models. His Flappy Bird demo includes a failed attempt with a much smaller model before a Gemma 4 attempt produces a playable game. Noyan cautions that smaller models can struggle with long-running tasks.

Burtenshaw compares Claude Code, Pi and OpenCode using hosted GLM 5.2, showing how harness tools affect repository analysis. Onur Solmaz then uses LM Studio with MLX weights on Apple silicon. He reports about 30 tokens per second for a quantized 35-billion-parameter MoE model on his 64 GB Mac. He also discusses OpenClaw automation and warns that keeping inference local does not resolve prompt injection risks.