LLMKube

LLMKube manages self-hosted LLM inference for teams and homelab owners who need several machines to serve models together. It brings Linux GPU servers and Apple Silicon Macs into the same Kubernetes-managed fleet.

• Runtime and model choice: Run llama.cpp, vLLM, TGI or mlx-server, with support for GGUF and Hugging Face models in safetensors format.
• Fleet management: Automatic model downloads, persistent caching, GPU scheduling, health checks and inference-based autoscaling handle shared model services.
• GPU memory sharing: Multi-GPU sharding and layer offloading let models use memory across several GPUs, with NVIDIA CUDA, AMD Vulkan and Apple Silicon Metal acceleration.
• Request routing: ModelRouter provides an OpenAI-compatible endpoint for local services and external providers such as Anthropic, OpenAI and Bedrock, with classification rules that can refuse regulated requests when local inference is unavailable.
• Coding agents: Foreman runs coder, verifier and reviewer agents on your fleet to work on issues and open pull requests for human review, keeping their code on your infrastructure.

Local inference runs on your hardware; requests routed to external providers leave that infrastructure. Apple Silicon inference uses a native macOS agent for Metal GPU access. Prometheus metrics, OpenTelemetry tracing and Grafana dashboards expose inference activity and GPU health. LLMKube is free and open source under Apache 2.0.