llama-swap setup: one API for multiple local models

llama-swap addresses a local LLM workflow problem: switching models can mean restarting servers and changing client URLs. The speaker demonstrates two configured models behind one endpoint. Changing the request's model field selects a different backend while the client keeps the same base URL.

The demonstration covers separate launch commands, model files, context sizes, and TTL settings. The proxy waits for a backend to become healthy before forwarding a request. According to the speaker, it can start a model when needed, stop another to make room, and unload an idle model after its TTL expires to release VRAM.

The video describes llama-swap as a Go binary with YAML configuration and support for OpenAI-compatible and Anthropic-compatible APIs. It can sit in front of backends such as llama.cpp, vLLM, and Tabby API. This gives developers control over how they run models locally, including backend flags and GPU settings.

The comparison with Ollama and LM Studio focuses on setup and control. llama-swap has no built-in model store, downloader, or gallery, and the speaker describes it as more oriented toward servers than a GUI. Users need model files, backend knowledge, and suitable launch flags. The suggested use case is a coding assistant or local script that needs a stable API while models change behind it.