MLC LLM

MLC LLM is a compiler and deployment engine for developers running language models on their own hardware or building AI into apps. Its shared MLCEngine supports deployment across desktop computers, mobile devices and browsers.

• Native model execution: Compile and optimize language models for the platform where they'll run, using the same engine across supported environments.
• Desktop GPUs: Run on Linux and Windows with Vulkan for AMD, NVIDIA and Intel GPUs, plus ROCm for AMD or CUDA for NVIDIA; macOS uses Metal with Apple GPUs, AMD discrete GPUs and Intel integrated GPUs.
• Mobile inference: Run models within iOS and iPadOS apps using Metal on Apple A-series GPUs, or Android apps using OpenCL on Adreno and Mali GPUs.
• Browser execution: Use WebGPU and WASM to run language models in a web browser.
• Application interfaces: Access MLCEngine through an OpenAI-compatible API, with REST server, Python, JavaScript, iOS and Android interfaces.

MLC LLM is open source under Apache 2.0. It serves developers who need a model runtime for their own applications or servers, with GPU backends that depend on the target operating system and hardware.