OpenLLM

OpenLLM is a self-hosted LLM server for developers connecting applications to models on their own hardware or servers. Its OpenAI-compatible interface lets existing clients use those models without adopting a separate API.

• Application connections: Serve models to tools that support OpenAI-compatible APIs, including the OpenAI Python client and LlamaIndex.
• Direct chat: Use the built-in browser interface or terminal chat to try a model before connecting it to an application.
• Model choice: Run models including DeepSeek R1, Llama, Gemma, Mistral and Qwen2.5, or add a model repository containing custom models.
• Deployment options: Use Docker or Kubernetes for deployments, or deploy through BentoML to the separate managed BentoCloud service.

OpenLLM is open source under Apache 2.0. Local deployments perform inference on your machine; self-hosted deployments use your server. BentoCloud runs inference in the cloud and provides managed infrastructure, autoscaling, model orchestration and observability.

GPU requirements vary substantially by model. Listed small Gemma models require 12 GB of GPU memory, while larger models can require multiple GPUs. Gated models require Hugging Face access and a token.