koboldcpp

KoboldCpp runs language models on your own hardware for chat, creative writing and roleplay. It's a fork of llama.cpp with a bundled KoboldAI Lite browser interface.

• Writing and roleplay: Story and adventure modes include memory, world information, author's notes, saved stories and Tavern character card imports.
• Models and media: GGUF models handle text generation, while compatible models add image understanding, image editing, video and music generation, Whisper transcription and speech synthesis.
• Agents and document search: The integrated agent can write and edit code, run programs and schedule tasks; TextDB searches text collections, and MCP support connects external tools.
• App connections: OpenAI-compatible and Ollama-compatible APIs let other applications use the local inference server, with additional endpoints for ComfyUI and speech clients.

It's free and open source under AGPL-3.0. Standalone executables run on Windows, Linux and macOS, using a CPU or GPU with full or partial GPU offloading. Apple Silicon Macs have a ready-to-run download; Intel Macs require a source build. Docker is supported. Model files are separate downloads, and legacy GGML models retain compatibility with some feature limitations.

Local inference runs on your hardware. Colab and RunPod deployments use cloud GPUs, while the browser interface can send requests to external providers including Anthropic, OpenRouter and Gemini.