OpenVINO

An open-source AI inference toolkit that runs models locally or on your own server, with support for CPUs, Intel GPUs and Intel NPUs.

OpenVINO is an AI inference toolkit for developers deploying models in their own applications. It converts and optimizes models for local hardware or servers, covering generative AI and conventional machine-learning workloads.

Generative and conventional AI: Run local LLMs, image generation, speech recognition and computer vision models; OpenVINO Physical AI also supports vision-language-action models for robots.
Model compatibility: Use models from PyTorch, TensorFlow, ONNX, Keras, PaddlePaddle and JAX/Flax, with Optimum Intel connecting Hugging Face transformers and diffusers.
Compression and startup: Reduce model resource demands with compression, cache compiled models, and begin inference on the CPU while another device prepares the model.
Application and server deployment: Embed the runtime in an application or serve inference through OpenVINO Model Server, including in Kubernetes.

The runtime supports Linux, Windows and macOS, with Python, C++ and C APIs. Supported hardware includes x86 and ARM CPUs, Intel integrated and discrete GPUs, and Intel NPUs. Converted models can run without their original training frameworks, and PyTorch applications can use OpenVINO through torch.compile(). Pre-optimized models are also available on Hugging Face.

OpenVINO is open source under Apache 2.0. Inference can run locally or on a separate server. The toolkit collects performance and usage telemetry, which users can opt out of.