Docker Model Runner

Docker Model Runner lets developers run AI models on their own computers or servers and connect them to applications. It combines local inference with model distribution through container registries.

• Model storage and distribution: Download models from Docker Hub, Hugging Face or OCI-compatible registries, store them locally, and package GGUF or Safetensors files for publication to registries.
• App connections: OpenAI and Ollama-compatible APIs let applications use local models, including coding assistants such as Cline, Continue, Cursor and Aider; Docker Compose and Testcontainers also support it.
• Text and image models: llama.cpp runs quantized GGUF models, vLLM serves Safetensors models for high throughput workloads, and Diffusers generates images from text with Stable Diffusion.
• Model management: Use the command line or Docker Desktop interface to interact with models, with logs and prompt and response details available for inspection.

The project is open source under Apache 2.0. It runs through Docker Desktop on macOS and Windows, Docker Engine on Linux, or a standalone binary that doesn't require Docker. llama.cpp is the default engine and works across supported platforms. macOS requires Apple Silicon. Docker Engine supports CPU, NVIDIA CUDA, AMD ROCm and Vulkan backends. Windows supports NVIDIA GPUs on amd64 and Qualcomm Adreno GPUs on arm64, with some llama.cpp limitations on Adreno 6xx GPUs.

vLLM requires NVIDIA GPUs on Linux x86_64 or Windows with WSL2. Diffusers requires NVIDIA GPUs on Linux x86_64 or ARM64.

Inference runs locally, and Docker doesn't collect prompt or response content. Docker Desktop usage collection follows its privacy setting; Docker Engine sends model-name tracking requests to Docker Hub regardless of settings. The API has no built-in authentication: any client that can reach it can pull, load and run models or send inference requests.