LLM Inference Engines and Libraries

Transformers is a Python library for developers and researchers running pretrained AI models or training models on their own hardware. It covers text, images, audio, video and multimodal tasks through shared model definitions.

MLX is a machine learning array framework for researchers and developers building and testing models on their own hardware. It supports model training and inference, with APIs for numerical work and neural networks.

Optimum Quanto is an open-source PyTorch library for developers who want to reduce the memory needed to run models on their own hardware. It handles language models and image-generation models, with integrations for Hugging Face Transformers and Diffusers. The project is in maintenance mode, accepting minor fixes and documentation work rather than major features.

FlashAttention is an attention library for developers training or running transformer models on their own GPUs. It computes exact attention while using less memory than standard PyTorch attention, allowing longer sequences.

ExecuTorch is an on-device AI runtime for developers adding PyTorch models to apps and embedded devices. It handles inference on the user's hardware across Android, iOS, macOS, Linux and Windows, as well as embedded systems and microcontrollers.

CTranslate2 is an inference library for developers who want to run Transformer models on their own hardware or servers. It focuses on reducing execution time and memory use for applications built in C++ or Python.

TensorRT is NVIDIA's inference toolkit for developers deploying AI applications on NVIDIA GPUs. Its compilers, runtimes and model optimizations target lower response latency and higher throughput on local machines, self-hosted servers and edge devices.

vLLM Metal lets developers serve language models on Apple Silicon Macs through vLLM. It's for people who want to use their Mac's hardware for local AI inference while keeping vLLM's API server and request scheduling.

Coqui TTS is a text-to-speech toolkit for developers and speech researchers who want to generate voices locally or train their own models. The idiap fork builds on coqui-ai/TTS.

ColPali is a document retrieval library for developers and researchers building search or retrieval-augmented generation systems. It matches text queries to page images, using visual content alongside text. The colpali-engine package is deprecated, and maintainers recommend Sentence Transformers for new projects and production use.

OpenCLIP is a Python library for developers and researchers building image-text matching and zero-shot classification applications. It supports pretrained CLIP models and training models on your own hardware.

Optimum is a collection of Python packages for developers adapting Hugging Face models to their deployment hardware. It extends Transformers, Diffusers, TIMM and Sentence Transformers with tools for faster inference and training.

KTransformers is a research framework for developers running or fine-tuning large language models on their own hardware. It focuses on mixture-of-experts (MoE) models, using CPU resources alongside GPUs to work within limited GPU memory.

OpenVINO is an AI inference toolkit for developers deploying models in their own applications. It converts and optimizes models for local hardware or servers, covering generative AI and conventional machine-learning workloads.

Detectron2 is a Python library for developers and researchers building object detection and image segmentation applications. It provides model training tools and reusable components for computer vision research and production software.

LMCache helps teams serving LLMs reuse context that a model has already processed. It reduces repeated work on long prompts, conversation history and retrieved documents, shortening the wait for the first token and improving throughput.

FastLLM is an open-source inference engine for people running large language models on their own machines or self-hosted servers. Its main draw is hardware flexibility: large mixture-of-experts (MoE) models can spread their weights across GPU memory, system RAM and disk when they don't fit entirely on a graphics card.

MNN is a C++ framework for developers running AI models on phones, PCs and embedded devices. It supports inference and training on the device, with a small application footprint and hardware acceleration.

wllama lets developers add local LLM inference to browser apps without a separate inference server. It brings llama.cpp to JavaScript and TypeScript projects through WebAssembly, so models run on the user's device.

Moondream is a vision language model for developers building apps that interpret images, such as warehouse or manufacturing footage. Its downloadable models can run on your own hardware, including offline in an air-gapped environment.

DeePMD-kit is open-source software for molecular and materials researchers who want to train and run AI interatomic potentials on their own hardware. It learns atomic interactions from quantum-mechanical reference data, then uses those models in molecular dynamics simulations of molecules, solids and metals. It runs on CPUs and GPUs, with support for CUDA and ROCm source builds, and carries the LGPL-3.0-or-later license.

whisper.cpp runs OpenAI's Whisper speech recognition models on your own hardware. It's for developers adding transcription to applications and people who need offline speech-to-text.

audio.cpp runs audio models locally for developers building speech tools and people generating or processing audio on their own hardware. It brings multiple audio tasks into one C++ runtime built on ggml.

GPyTorch is an open-source Python library for researchers and developers building Gaussian process models in PyTorch. It runs model inference on your own hardware and can use GPUs to accelerate computation. Its focus is on making Gaussian processes practical at larger scales while leaving room to define custom models. The library uses the MIT license.

Mamba is a Python library for researchers and developers building sequence models, including language models. Its selective state space architecture processes sequences in linear time and is designed for efficient GPU execution.

HanLP is a multilingual natural language processing toolkit for developers and researchers who need structured text analysis. Its native Python API runs models on your own hardware. The separate REST API sends text to a hosted server.

HQQ is a Python library for developers compressing language and vision models to fit the memory available on their own hardware. Its Half-Quadratic Quantization method doesn't require a calibration dataset.

Argmax OSS Swift is a speech AI SDK for developers building iOS and macOS apps that process audio on the user's device. Its separate libraries cover transcription, identifying speakers and generating spoken audio.

MONAI is a framework for researchers and developers building AI models for medical images. Its Core library provides medical-specific tools for training and evaluating models within existing PyTorch workflows.

WeNet is a speech recognition toolkit for developers adding transcription to applications or services. It covers model training and production deployment, so teams can adapt a recognizer to their audio or start with a pretrained model.

DeepSpeed is an open-source PyTorch library for developers and researchers training or running large AI models on their own hardware or compute clusters. It helps address memory limits and distribute workloads across devices.

ModelScope is an AI model hub with a Python library for developers and researchers who want to run models locally or adapt them to their own data. It covers language, vision, speech, multimodal tasks and scientific computing.

docTR reads text from PDFs and images for developers building document processing applications and researchers comparing OCR models. It locates individual words and recognizes their text on your own hardware.

OuteTTS is a local text-to-speech library for developers adding generated speech to their applications. It uses reference audio to reproduce a speaker's voice, including their style, emotion and accent.