LLM Inference Engines and Libraries

FunASR is a speech-to-text toolkit for developers building transcription services and voice applications. It supports offline recordings and live speech on local hardware or self-hosted servers, with capabilities that depend on the model and runtime.

Vosk is an offline speech-to-text toolkit for developers building voice interfaces or transcription software. It processes speech on the device or server running it, without sending audio to a cloud recognition service.

Colossal-AI is a Python framework for developers and researchers training or serving large AI models on their own GPUs. It provides tools to spread workloads across GPUs and work within available memory.

PowerInfer is a local LLM inference engine for developers and researchers running large models on a PC with a consumer GPU. Its performance gains depend on sparse models, which use only part of their network for each input.

SAHI is a Python computer vision library for developers detecting small objects in large images, including aerial imagery and surveillance footage. It divides images into smaller regions for detection, helping models find objects that occupy few pixels.

tiny-cuda-nn is a neural network framework for developers and researchers building small models, NeRF renderers and neural graphics applications. It handles training and inference on local NVIDIA GPUs.

LiteRT-LM is Google's open-source framework for developers building apps that run language models on the user's device. It uses LiteRT for local inference across mobile devices, desktops and browsers, with GPU and NPU acceleration. The project uses the Apache 2.0 license.

PyOD is a Python library for detecting unusual records and behavior in data on your own machine or server. A shared fit/predict interface lets developers and data scientists compare classical and deep learning anomaly detectors without changing the surrounding application for each algorithm. Its code uses the BSD 2-Clause license.

BGE Embeddings is a family of models for developers who need to find relevant passages for search or retrieval-augmented generation (RAG). It includes embedding models for finding candidate text and rerankers for assessing how closely those results match a query.

GPTQModel is a Python toolkit for developers compressing LLMs and running them on their own hardware or servers. It combines model preparation, quality evaluation and inference so you can assess compression choices within one workflow.

EasyOCR is a Python library for developers adding text recognition to an application. It reads both text in photographs and dense document text.

DeepCell is an open-source Python library for researchers analyzing individual cells in biological images. It runs locally and uses TensorFlow to identify whole cells or nuclei and track cells through time-lapse movies. Its Apache 2.0 license allows users to adapt the library for their own research.

Bergamot Translator is a local AI translation library for developers building translation into browser or native applications. People who want to translate web content can use it through the separate TranslateLocally extension for Firefox.

MFLUX is a Python toolkit for creators and developers who want to generate and edit images on an Apple Silicon Mac. It provides a command-line interface for working with models locally.

bitsandbytes is a Python library for developers who need to run or fine-tune large language models on their own hardware with less memory. It works within PyTorch workflows.

RKLLM is a toolkit for developers building local AI applications on Rockchip boards. It runs language and multimodal models on the chip's neural processing unit (NPU).

faster-whisper is an open source Python library for developers adding speech transcription to their software. It uses CTranslate2 to run OpenAI's Whisper models, with faster transcription and lower memory use than the original implementation in the project's benchmark comparisons.

Keras is an open-source Python deep learning framework for developers and researchers who want to build and train models without tying their code to one backend. It runs locally on Linux and macOS, with WSL2 recommended for Windows. Its Apache 2.0 license permits commercial use and modification.

Model2Vec is a Python package for developers who need text embeddings for search or classification on their own hardware. It turns sentence transformers into smaller static embedding models, trading a small drop in performance for lower model size and faster CPU inference.

rerankers is a Python library for developers building search and retrieval systems. It ranks candidate documents against a query and lets you compare reranking approaches through a shared interface.

ExLlamaV2 and V3 are local LLM inference libraries for developers building AI applications or hosting models on consumer GPUs. V2 is archived for now and isn't currently maintained; development continues in V3.

Stanza is a Python NLP library for developers and researchers who need to analyze text in multiple human languages on their own hardware. It turns raw text into linguistic annotations, such as word forms, grammatical relationships, and named entities. It's open source under Apache 2.0.

DeepGEMM is an open-source CUDA kernel library for developers building LLM training and inference systems on their own NVIDIA GPUs. It supplies matrix operations and specialized computations for mixture-of-experts (MoE) models.

SLEAP helps behavioral researchers measure animal movement in recorded video. It trains models to estimate poses for single animals or groups, with human review during dataset creation and prediction checking.

MLX VLM is a Python package for developers and researchers who want to run multimodal models on their own Mac. It supports local inference and fine-tuning through MLX.

MLX Audio is a Python library for developers building speech and audio applications on Apple Silicon Macs. It runs models locally using Apple's MLX framework, with support for generated speech, transcription and audio processing.

Torch-TensorRT compiles PyTorch models with TensorRT to reduce inference latency on NVIDIA hardware. It's for developers running models on their own machines or servers who want faster execution while keeping their PyTorch workflow. The project is open source under the BSD 3-Clause license.

Slotstream runs Qwen3.8-Flash-Next on Apple Silicon Macs with too little RAM to hold the whole model. It's aimed at Macs with 16 to 64 GB of memory, for people who want local chat, answers about pictures or a backend for coding assistants.

LiteRT is Google's framework for developers adding machine learning and generative AI to apps that run on users' devices. It succeeds TensorFlow Lite and supports local inference without a server round trip.

TorchMD-NET is an open-source Python library for researchers who train neural network potentials for molecular dynamics. It exposes its models as PyTorch modules and connects them to GPU-accelerated simulation software, including ACEMD, OpenMM and TorchMD. It's available under the MIT license.

Nunchaku is a local diffusion inference engine for developers and image-generation users who want to run models such as FLUX with less GPU memory. It focuses on faster generation with compressed models on NVIDIA hardware.

pyannote.audio is a Python toolkit for developers and researchers who need to separate recordings by speaker. Speaker diarization identifies who spoke when, so applications can associate parts of an audio recording with different speakers.

Sentence Transformers is a Python library for developers building search and document retrieval on their own hardware. It helps match content by meaning and rank results against a query.

Transformer Engine is a library for developers building and running Transformer models on NVIDIA GPUs. It reduces compute and memory demands in training and inference through low-precision arithmetic and optimized model components.