llama-swap

llama-swap is a local AI proxy for people who want several models available to their apps without keeping them all loaded. It starts the server a request needs and replaces the active server when another model is requested.

• Model switching: Use one model at a time by default, or allow selected models to stay loaded together to suit your available resources.
• Compatible APIs: Connect apps through OpenAI and Anthropic compatible APIs, with support for servers such as llama.cpp and vLLM; llama.cpp's llama-server has the strongest support.
• Image, audio and coding workloads: Route requests for image generation and editing, speech generation, transcription, embeddings, reranking and code infilling when the underlying server supports them, including integrations with stable-diffusion.cpp, audio.cpp and ComfyUI.
• Browser playground and monitoring: Try local models, inspect requests and responses, view token metrics and live logs, and manually load or unload models through the web interface.

The project is open source under the MIT license. It runs as a standalone binary on macOS, Windows, Linux and FreeBSD, and also supports Docker deployment. Unified containers bundle inference servers with NVIDIA CUDA and Vulkan variants. Available workloads and hardware support depend on the inference server you use behind the proxy.