LMDeploy

LMDeploy is an open-source toolkit for developers who want to run language and vision-language models on their own GPU hardware. It supports offline batch processing and model servers for applications.

• Text and vision models: Supported families include Llama, Qwen, InternLM, DeepSeek and gpt-oss, plus vision-language models such as LLaVA, Qwen-VL and InternVL.
• Model compression: AWQ, GPTQ and attention-cache quantization reduce model memory use; AWQ and cache quantization can work together with automatic prefix caching.
• Application APIs: OpenAI-compatible servers and Anthropic-compatible endpoints connect models to applications, with support for tool calling, reasoning outputs, structured output and LoRA serving.
• Serving across GPUs and machines: Continuous batching processes requests together, while multi-node deployment and a request distributor support services that span models and machines.

LMDeploy has two inference engines. TurboMind focuses on inference performance; the Python-based PyTorch engine makes it easier for developers to extend model support. Model compatibility and supported inference data types differ between the engines.

The toolkit uses the Apache 2.0 license and supports NVIDIA GPUs, including GeForce RTX 50 series. Offline batch inference runs locally, while model downloads use Hugging Face by default, with ModelScope and openMind Hub as alternatives. The separate LMDeploy-Jetson project supports offline deployment on NVIDIA Jetson hardware.