mistral.rs

mistral.rs is an inference engine for developers who want to run AI models on their own computer or server. It supports browser chat and inference embedded directly in Python or Rust applications.

• Models and formats: Load supported Hugging Face models, GGUF files and UQFF quantizations, including models such as Muse Glimmer 30B and Gemma 4. The engine detects supported architectures and chat templates, can quantize models locally, and recommends quantization choices based on your hardware.
• Multimodal tasks: Use models for text, image, video and audio inputs, as well as speech generation, image generation and embeddings. Available tasks depend on the model you choose.
• Chat and application APIs: A built-in web interface displays reasoning, code execution, plots and files. The same server exposes OpenAI-compatible and Anthropic-compatible APIs, while Python and Rust SDKs support inference inside your application.
• Agent tools: Agents can execute local Python and shell commands, use uploaded skill bundles and files, and call custom tools or MCP servers. Python sessions retain state; shell sessions have sandboxing and approval controls. Web search uses an external connection.

The project is open source under the MIT license and runs on Linux, macOS and Windows. It supports CPU execution, CUDA on NVIDIA GPUs and Metal on Apple Silicon; Windows prebuilt binaries use CPU execution. Docker images cover CPU and CUDA deployments. Multi-GPU and distributed inference support larger workloads, while continuous batching and prefix caching help serve shared requests. LoRA adapters can be selected per request.