LLM Quantization and Conversion Tools

Optimum Quanto is an open-source PyTorch library for developers who want to reduce the memory needed to run models on their own hardware. It handles language models and image-generation models, with integrations for Hugging Face Transformers and Diffusers. The project is in maintenance mode, accepting minor fixes and documentation work rather than major features.

ExecuTorch is an on-device AI runtime for developers adding PyTorch models to apps and embedded devices. It handles inference on the user's hardware across Android, iOS, macOS, Linux and Windows, as well as embedded systems and microcontrollers.

CTranslate2 is an inference library for developers who want to run Transformer models on their own hardware or servers. It focuses on reducing execution time and memory use for applications built in C++ or Python.

TensorRT is NVIDIA's inference toolkit for developers deploying AI applications on NVIDIA GPUs. Its compilers, runtimes and model optimizations target lower response latency and higher throughput on local machines, self-hosted servers and edge devices.

TensorFlow Model Optimization Toolkit is an open-source Python library for developers preparing TensorFlow and Keras models for deployment on mobile, IoT and other edge devices, or in the cloud. It helps reduce inference latency and cost, and adapt models to limits on memory, processing power, energy use and storage.

Optimum is a collection of Python packages for developers adapting Hugging Face models to their deployment hardware. It extends Transformers, Diffusers, TIMM and Sentence Transformers with tools for faster inference and training.

MTPLX is a native Mac app, command-line tool and local LLM server for people running chat or coding models on Apple Silicon. It uses a model's own multi-token prediction heads to accelerate generation without a separate draft model.

OpenVINO is an AI inference toolkit for developers deploying models in their own applications. It converts and optimizes models for local hardware or servers, covering generative AI and conventional machine-learning workloads.

MNN is a C++ framework for developers running AI models on phones, PCs and embedded devices. It supports inference and training on the device, with a small application footprint and hardware acceleration.

LLM Compressor prepares models for deployment with vLLM by reducing their storage and memory requirements through quantization. It's for developers working with local checkpoints or models from Hugging Face repositories.

HQQ is a Python library for developers compressing language and vision models to fit the memory available on their own hardware. Its Half-Quadratic Quantization method doesn't require a calibration dataset.

LiteRT Torch is a Python library for converting PyTorch models into .tflite files for the LiteRT runtime. Developers can prepare models on Linux and deploy them into Android, iOS and IoT applications that run inference on the device. Code is licensed under Apache 2.0.

PaddleNLP is a Python toolkit for developers training and deploying language models on their own infrastructure. Built on PaddlePaddle, it covers pretraining, fine-tuning, model compression and inference across several accelerator families.

RamaLama runs AI models for local chat or a self-hosted REST API. It's for developers who want a container workflow for trying models on their own hardware and serving them in production.

Stable Diffusion Core ML is a developer toolkit for adding local AI image generation to Mac, iPhone, and iPad apps. It pairs model conversion tools with a Swift package for embedding generation in native software.

TensorRT Model Optimizer (ModelOpt) is a Python library for developers who need smaller models and faster inference. It accepts Hugging Face, PyTorch and ONNX models and lets developers combine optimization techniques before deployment.

BitNet is Microsoft's local LLM inference framework for developers and researchers working with compatible 1-bit models. Its bitnet.cpp runtime focuses on faster inference and lower energy use on your own hardware.

NeMo Automodel is a Python library for researchers and developers who want to train or adapt AI models on their own GPU workstations or clusters. Its PyTorch-based training works with Hugging Face models without converting checkpoints.

FastVLM is a family of local vision-language models for developers and researchers building applications that answer questions about images. It focuses on reducing the delay before an answer starts, especially with high-resolution images.

torchtune is a Python library for developers and researchers who want to adapt LLMs on their own hardware with editable PyTorch training code. The project is no longer actively maintained.

TorchAO is a PyTorch library for developers who want to train and run models on their own hardware with less memory. It supports language models and image or video generation, and works with torch.compile and FSDP2 across most Hugging Face PyTorch models.

Cactus is an on-device AI engine for developers adding automation to mobile apps, wearables and other small devices. Its Needle model turns requests into tool calls locally, including when the device has no internet connection.

ncnn is a C++ inference framework for developers adding on-device AI to mobile, desktop and embedded applications. It focuses on a small memory footprint and doesn't require third-party runtime dependencies.

Soup is a local LLM fine-tuning tool for developers and teams adapting models to their own datasets. It brings training, evaluation and export into one workflow that runs on your own hardware without requiring a cloud service.

Axolotl is a free, open-source LLM fine-tuning framework for developers and researchers adapting models to their own data. It supports training on local hardware or cloud infrastructure you control.

PaddleSeg is an image segmentation toolkit for developers building computer vision applications on their own hardware. It covers annotation, training and deployment for tasks such as medical image analysis, industrial inspection and portrait effects.

GPTQModel is a Python toolkit for developers compressing LLMs and running them on their own hardware or servers. It combines model preparation, quality evaluation and inference so you can assess compression choices within one workflow.

bitsandbytes is a Python library for developers who need to run or fine-tune large language models on their own hardware with less memory. It works within PyTorch workflows.

RKLLM is a toolkit for developers building local AI applications on Rockchip boards. It runs language and multimodal models on the chip's neural processing unit (NPU).

ExLlamaV2 and V3 are local LLM inference libraries for developers building AI applications or hosting models on consumer GPUs. V2 is archived for now and isn't currently maintained; development continues in V3.

MLX VLM is a Python package for developers and researchers who want to run multimodal models on their own Mac. It supports local inference and fine-tuning through MLX.

Torch-TensorRT compiles PyTorch models with TensorRT to reduce inference latency on NVIDIA hardware. It's for developers running models on their own machines or servers who want faster execution while keeping their PyTorch workflow. The project is open source under the BSD 3-Clause license.

LiteRT is Google's framework for developers adding machine learning and generative AI to apps that run on users' devices. It succeeds TensorFlow Lite and supports local inference without a server round trip.

ms-swift is a Python framework for developers and researchers training and deploying language and multimodal models on their own hardware. It covers model adaptation through evaluation and serving, with support for Qwen3, DeepSeek-R1, Llama4, Mistral, Qwen3-VL and InternVL3.5.