MLX Omni Server
Local AI inference server for Apple Silicon Macs. Use OpenAI and Anthropic SDKs with MLX models, keeping processing on your machine. MIT licensed.
MLX Omni Server lets developers use local models through both OpenAI-compatible and Anthropic-compatible APIs on an Apple Silicon Mac. It's for applications built around those SDKs that need inference on hardware the developer controls. Apple's MLX framework provides hardware acceleration, and all AI processing stays on the Mac.
The server covers more than text chat. Its OpenAI-compatible API supports speech transcription, text-to-speech, image generation and text embeddings, so applications can use local inference for audio and image tasks as well as conversations. Chat supports streamed responses, tool calling and structured output with JSON schema validation. The Anthropic-compatible messages API also supports streaming and tools, plus thinking mode for models that provide extended reasoning.
Compatibility with both API families is the main reason to consider it for an existing application: developers can keep using OpenAI or Anthropic SDK clients while directing requests to MLX models. That compatibility doesn't mean it runs OpenAI or Anthropic's cloud models. The supplied example uses the MLX model mlx-community/gemma-3-1b-it-4bit-DWQ.
Model management includes discovery of MLX models already in the Hugging Face cache, automatic downloads when needed, and loading and caching models on demand. The project is open source under the MIT license and uses Python, FastAPI and MLX-LM. It requires an Apple Silicon Mac. Install mlx-omni-server with pip and run its CLI. Python 3.11 or later is required; initial model downloads need an internet connection.
