OpenVINO Model Server

OpenVINO Model Server is a self-hosted inference server for developers and teams serving AI models on their own machines or servers. It uses OpenVINO to run workloads on x86 CPUs, including Xeon, with acceleration for Intel integrated and discrete GPUs and NPUs. It's open source under Apache 2.0.

Its OpenAI-compatible API covers text generation, embeddings, image generation and audio, so applications built around that interface can connect to models you host. LLM features include streaming responses and structured output, plus continuous batching and speculative decoding. It also supports vision language models, AI agents with MCP servers, speech recognition and text-to-speech. A Cohere-compatible API handles reranking for retrieval applications.

The same server handles object detection, classification and OCR through KServe APIs. It accepts GGUF, TensorFlow, ONNX, PaddlePaddle and OpenVINO IR models. For applications that combine several processing stages, it supports DAG pipelines, MediaPipe graphs and Python execution nodes.

You can run it directly on Linux or Windows, in Docker, or on Kubernetes and OpenShift. Inference runs on the host you control; model storage can be local or use S3, GCS, Azure Blob or HuggingFace Hub. Model versioning and hot reload support model updates, while Prometheus-compatible metrics provide monitoring. Its C API lets developers embed the server in native applications.