Marbor
A self-hosted AI inference gateway routes Ollama, vLLM, TGI, llama.cpp and MLX requests by GPU capacity and model state, with optional cloud overflow.
Marbor is a self-hosted AI inference gateway for operators serving models across their own GPU fleet. It places requests using live hardware and model state to reduce cold loads and avoid deployments that can't serve the requested model.
- GPU-aware placement Routing considers loaded models, runtime compatibility, deployment requirements, GPU groups, available memory and load, and excludes unhealthy nodes.
- Shared model endpoint An OpenAI-compatible endpoint connects Ollama, vLLM, TGI, llama.cpp and MLX, including MLX on Apple Silicon, so existing OpenAI clients can use a mixed fleet.
- Model readiness Pinned models remain resident, optional session affinity keeps conversations on the same node, and predictive prewarming uses workload patterns when model size and memory headroom are known.
- Cloud overflow Optional overflow sends requests to OpenAI or Anthropic only when local capacity is full, with token costs attributed to each API key.
- Fleet monitoring The dashboard shows GPU memory, model residency, runtime health, per-key usage and explanations for local request placement; API keys support model allow-lists and spend caps.
Marbor is open source under Apache 2.0. The gateway runs as a static Go binary with SQLite storage and no runtime dependencies; an optional companion agent supplies live GPU telemetry. It can work alongside LiteLLM for placement onto local hardware. Memory telemetry is richest on Ollama nodes and nodes reporting through the companion agent; other runtimes use load, health and residency signals.
Marbor doesn't parse prompt text for routing or move KV data between nodes. Inference runtimes handle sharding and parallelism. Failed requests can retry before the first response byte, but a stream already in progress is never replayed.
