Ollama, llama.cpp and vLLM: comparison and setup
Compare local model runners and serving engines, with Qwen 3 setup examples and a 14B ExLlamaV3 demo on an 8 GB GPU.
Player not loading? Watch on YouTube
This guide compares ways to run models locally, separating compute libraries, inference engines and the interfaces built around them. It focuses on personal hardware and small server deployments. The speaker explains Apple-specific MLX options but does not demonstrate them because he lacks Apple hardware.
The local LLM examples mostly use Qwen 3 with 0.6 billion parameters and Python requests to OpenAI-compatible endpoints. The speaker presents llama.cpp as a practical choice for personal use with CPU, GPU or mixed execution, while vLLM targets concurrent requests and throughput. His parallel test takes 9.2 seconds with llama.cpp and 1.6 seconds with vLLM; those results describe his setup rather than a universal performance ranking.
Ollama simplifies model downloads, Open WebUI adds browser chat, and LM Studio emphasizes graphical model search and configuration. Docker Model Runner gets a Compose integration example. LocalAI demonstrates text and Stable Diffusion through one API.
For tighter VRAM budgets, the ExLlamaV3 and TabbyAPI example serves a three-bit Qwen 3 14B model on an 8 GB GPU after cache adjustments. The speaker also examines SGLang for repeated context, LMDeploy for quantized models and TensorRT-LLM for NVIDIA hardware. Cluster tools receive brief explanations rather than deployment demos.
An Agent Vault sponsor segment discusses credential proxying; the presenter notes that it does not supply a sandbox or full network isolation.