ServiceStack AI Server

A self-hosted AI API gateway that routes requests to Ollama, cloud LLMs and media services, with centralized access controls and usage monitoring.

ServiceStack AI Server is a self-hosted gateway for developers and teams whose applications use several AI providers. It runs in a Docker container and gives apps one API integration for local models, cloud LLMs and media processing. Teams can change providers without changing their client integrations.

The gateway runs on your server. Requests sent to local Ollama endpoints or self-hosted media agents use those services; requests routed to OpenAI, Anthropic, Mistral AI, Google Cloud, OpenRouter or GroqCloud go to external providers. Self-hosting the gateway doesn't make cloud processing local.

Its admin interface manages providers, API keys and which models each app can access. It also shows usage statistics, request history and live task progress, plus dashboards for pending, completed and failed background jobs. Teams can distribute requests across multiple providers and keep the gateway behind a firewall.

Media support includes image generation through ComfyUI or services such as Replicate and DALL-E 3, image variations with caching, speech recognition through Whisper, and text-to-speech. FFmpeg agents handle video conversion, scaling and cropping. Optional agents run on GPU servers for media workloads.

Typed client APIs support several programming languages. Built-in storage caches generated assets separately for each API key, and the server archives completed requests in monthly databases.

The application is AGPL-3.0 licensed; ServiceStack framework dependencies have separate license terms. The installer supports Linux, macOS and Windows through WSL2. The optional ComfyUI agent requires a supported NVIDIA GPU on Linux or WSL2, rather than macOS.

Image Generation and Editing LLM Gateways and Routers LLM Observability and Tracing Speech-to-Text and Transcription Text-to-Speech and Audiobooks Docker Web Distributed execution Ollama integration

Nobody has claimed ServiceStack AI Server yet

Work on ServiceStack AI Server? Claim this page with an email at docs.servicestack.net. ServiceStack AI Server gets the verified checkmark, and you can upgrade the listing.

Want more people to find it? Promote it

Something wrong or outdated on this page?

Similar to ServiceStack AI Server

View all tools

MLX Omni Server

Local AI inference server for Apple Silicon Macs. Use OpenAI and Anthropic SDKs with MLX models, keeping processing on your machine. MIT licensed.

745Updated 6 months agoMIT

macOS#Hugging Face integration#MLX#OpenAI-compatible API

MLX Omni Server lets developers use local models through both OpenAI-compatible and Anthropic-compatible APIs on an Apple Silicon Mac. It's for applications built around those SDKs that need inference on hardware the developer controls. Apple's MLX framework provides hardware acceleration, and all AI processing stays on the Mac.

Embedding ServersImage Generation and EditingInference Servers and ClustersSpeech-to-Text and TranscriptionText-to-Speech and Audiobooks

Aiursoft Ollama Gateway

A self-hosted AI gateway that connects Ollama and OpenAI-compatible backends to coding assistants, with API keys, auditing and model aliases.

9Updated 1 day ago

Linux · Docker · Web#Multi-user access#Multimodal input#Ollama integration

Aiursoft Ollama Gateway gives teams controlled access to shared local LLM servers. It connects AI applications and coding assistants to Ollama or OpenAI-compatible backends through a central gateway.

LLM Gateways and RoutersLLM Observability and Tracing

Future AGI

A self-hosted AI agent evaluation platform with text and voice simulations, OpenTelemetry tracing, and support for Ollama and vLLM.

2.2KUpdated 1 day agoApache-2.0

Docker · Web#AI red teaming#Guardrails#LLM tracing

Future AGI helps teams test AI agents, investigate production failures and compare prompt changes. It brings evaluation and monitoring into the same platform so teams can connect poor responses to the steps that caused them.

Guardrails, Security and SandboxingLLM Evaluation and TestingLLM Gateways and RoutersLLM Observability and TracingPrompt Management and Playgrounds

GoModel

A self-hosted AI gateway connects local and cloud models through OpenAI- and Anthropic-compatible APIs, with caching, failover and usage tracking.

1.3KUpdated 1 day agoMIT

macOS · Windows · Linux · Docker · Web#Guardrails#llama.cpp backend#LLM tracing

GoModel is a self-hosted AI gateway for developers and platform teams sharing access to local models and cloud providers. It centralizes provider choice and usage policies behind APIs compatible with OpenAI and Anthropic SDKs.

Guardrails, Security and SandboxingLLM Gateways and RoutersLLM Observability and Tracing

Marbor

A self-hosted AI inference gateway routes Ollama, vLLM, TGI, llama.cpp and MLX requests by GPU capacity and model state, with optional cloud overflow.

2Updated 1 day agoApache-2.0

macOS · Windows · Linux · Docker · Web#llama.cpp backend#LLM tracing#MLX

Marbor is a self-hosted AI inference gateway for operators serving models across their own GPU fleet. It places requests using live hardware and model state to reduce cold loads and avoid deployments that can't serve the requested model.

GPU Management and MonitoringLLM Gateways and RoutersLLM Observability and Tracing

MLflow

6 videos

An open-source, self-hosted AI engineering platform for agent tracing, LLM evaluation and model tracking, with Ollama support and an AI gateway.

28.5KUpdated 12 hours agoApache-2.0

Docker · Web#Batch processing#LLM tracing#MCP

MLflow helps developers and teams inspect AI applications, compare changes and track model experiments. It brings agent debugging and evaluation into a platform you can run locally or on your own servers.

Experiment Tracking and MLOpsLLM Evaluation and TestingLLM Gateways and RoutersLLM Observability and TracingPrompt Management and Playgrounds