MLX VLM
A local AI package runs and fine-tunes multimodal models on Macs using MLX, with an OpenAI-compatible server and support for image, audio and video tasks.
MLX VLM is a Python package for developers and researchers who want to run multimodal models on their own Mac. It supports local inference and fine-tuning through MLX.
- Image and video chat: Ask questions about images or compare multiple images with compatible models. Video chat works with models including Qwen2-VL, Qwen2.5-VL, Idefics3 and LLaVA-OneVision; audio support depends on the model.
- Application integration: A self-hosted server provides OpenAI-compatible chat and Responses APIs, so applications can use locally running models for text and media requests.
- Search and document ranking: Generate embeddings with native MLX embedding models and rank text or multimodal documents by their relevance to a query.
- Speech services: Transcribe audio and generate speech through mlx-audio models. Audio translation requires a speech recognition model that supports translation, and full-duplex voice conversations require compatible models.
- Repeated conversations: The server caches image features across requests to reduce repeated image processing. It also batches requests to share GPU compute and can compact conversation history into reusable state.
The package is open source under the MIT license and runs on Macs, with Metal support on Apple Silicon. The server can find models in local folders and the Hugging Face cache, though finding a checkpoint doesn't guarantee its architecture is supported.