Xinference

Xinference serves AI models on your own computer or servers for developers and researchers building applications around models they control. It brings chat, document search and media generation into one platform.

• Application connections: An OpenAI-compatible API supports chat and function calling, with integrations for LangChain, LlamaIndex, Dify and Chatbox.
• Model management: A web interface lets you manage and interact with models, and you can serve your own model weights alongside built-in choices.
• Search and media: Run embedding and reranking models for document search, generate images and video, transcribe speech, convert text to speech, and use multimodal LLMs to process images and audio.
• Multiple machines: Distribute inference across workers and automatically batch concurrent requests; Xagent integration adds agent planning and tool use.

The open source community edition uses the Apache 2.0 license and runs on Linux, Windows and macOS. It supports CPUs and GPUs, including Metal, with inference engines such as vLLM, llama.cpp and TensorRT and support for GGML. Docker deployments support NVIDIA GPUs; Kubernetes deployments require GPU support in the cluster.

The commercial Enterprise edition can run in your own cloud or data centre. Xinference Cloud provides a managed dedicated deployment, while the hosted Model API runs supported models through an external service rather than on your machine.