MNN

MNN is a C++ framework for developers running AI models on phones, PCs and embedded devices. It supports inference and training on the device, with a small application footprint and hardware acceleration.

• Local language models. MNN-LLM runs models including Qianwen, Baichuan, Zhipu and Llama on mobile phones, PCs and IoT devices.
• Image generation. MNN-Diffusion provides a Stable Diffusion runtime based on the same engine.
• Model conversion and compression. Bring models from TensorFlow, TensorFlow Lite, Caffe, ONNX and TorchScript into MNN, and reduce model size with FP16 or Int8 quantization.
• Training and image processing. Train models on PCs or mobile devices and use the Python API for inference, training and image processing without writing C++ application code.

MNN is open source under Apache 2.0. It supports iOS, Android and embedded devices with a POSIX interface. Hardware support includes ARM and x64 CPUs, mobile GPU inference through Metal, OpenCL and Vulkan, and NVIDIA GPU acceleration through CUDA. Supported precision varies by hardware backend.

MNN Workbench provides pretrained models, visual training tools and deployment to devices. Alibaba uses the framework in applications including Taobao, Youku and DingTalk.