LLM Inference Engines and Libraries
The engines and bindings that run models inside your own code: llama.cpp, MLX, ONNX Runtime and Hugging Face Transformers.
6.9KUpdated 2 hours agoApache-2.0
macOS · Windows · Linux · iOS · Android · Web#Hugging Face integration#Multimodal input#OpenAI-compatible API
LiteRT-LM is Google's open-source framework for developers building apps that run language models on the user's device. It uses LiteRT for local inference across mobile devices, desktops and browsers, with GPU and NPU acceleration. The project uses the Apache 2.0 license.
8.3KUpdated 3 hours agoMIT
macOS · Web#Batch processing#MLX#Multilingual
MLX Audio is a Python library for developers building speech and audio applications on Apple Silicon Macs. It runs models locally using Apple's MLX framework, with support for generated speech, transcription and audio processing.
7.7KUpdated 2 years agoMIT
Docker · Web#Multilingual
MeloTTS is a Python text-to-speech library for developers adding spoken output to their applications. It runs locally and can generate speech in real time on a CPU, so a dedicated GPU isn't required for inference.
22.3KUpdated 2 days agoMIT
macOS · Windows · Linux · iOS · Android · Web#Distributed execution#ONNX
ONNX Runtime is an AI inference and training engine for developers adding models to apps and services. It supports deployment on a user's device or a server, including applications that need the same model format across platforms.
1.9KUpdated 4 days agoAGPL-3.0
macOS · Windows · Linux · Docker#Batch processing#Distributed execution#Hugging Face integration
Sonar serves language and multimodal models for developers and teams running AI applications on their own hardware. It builds on vLLM and works with Hugging Face-compatible models.
3.3KUpdated 12 hours agoApache-2.0
#Batch processing#Multilingual#ONNX
FastEmbed is a Python library for developers building semantic search and retrieval-augmented generation (RAG). It turns text into vectors and evaluates search results on your own hardware without requiring a cloud embedding API.
10.8KUpdated 2 days agoMIT
#Hugging Face integration#Speaker diarization#Voice activity detection
pyannote.audio is a Python toolkit for developers and researchers who need to separate recordings by speaker. Speaker diarization identifies who spoke when, so applications can associate parts of an audio recording with different speakers.
7.8KUpdated 1 week agoMIT
macOS · Windows · Linux · Docker · Web#Code execution#GGUF#Hugging Face integration
mistral.rs is an inference engine for developers who want to run AI models on their own computer or server. It supports browser chat and inference embedded directly in Python or Rust applications.
qualcomm/GenieXInference Libraries and Bindings
macOS · Windows · Linux#GGUF#Hugging Face integration#llama.cpp backend
Nexa SDK is an on-device AI framework for developers building apps that run models on users' hardware. It covers language, vision and audio tasks through a shared interface for local inference.
1.3KUpdated 12 minutes ago
macOS · Windows · Linux#GGUF#Hugging Face integration#LoRA
GPTQModel is a Python toolkit for developers compressing LLMs and running them on their own hardware or servers. It combines model preparation, quality evaluation and inference so you can assess compression choices within one workflow.
3.6KUpdated 2 hours agoApache-2.0
#Distributed execution#Quantization
Transformer Engine is a library for developers building and running Transformer models on NVIDIA GPUs. It reduces compute and memory demands in training and inference through low-precision arithmetic and optimized model components.
14.3KUpdated 1 day ago
#Hugging Face integration#Multimodal input
OpenCLIP is a Python library for developers and researchers building image-text matching and zero-shot classification applications. It supports pretrained CLIP models and training models on your own hardware.
55.3KUpdated 2 days agoMIT
macOS · Windows · Linux · iOS · Android · Docker#Hugging Face integration#Quantization#Streaming inference
whisper.cpp runs OpenAI's Whisper speech recognition models on your own hardware. It's for developers adding transcription to applications and people who need offline speech-to-text.
10.1KUpdated 6 days agoBSD-2-Clause
#Agent Skills#MCP
PyOD is a Python library for detecting unusual records and behavior in data on your own machine or server. A shared fit/predict interface lets developers and data scientists compare classical and deep learning anomaly detectors without changing the surrounding application for each algorithm. Its code uses the BSD 2-Clause license.
9.2KUpdated 3 days agoApache-2.0
Docker#Image-to-image#Inpainting#Multimodal input
ModelScope is an AI model hub with a Python library for developers and researchers who want to run models locally or adapt them to their own data. It covers language, vision, speech, multimodal tasks and scientific computing.
sherpa-onnx
1 video16KUpdated 15 hours agoApache-2.0
macOS · Windows · Linux · iOS · Android · Web#Multilingual#ONNX#Speaker diarization
sherpa-onnx is a toolkit for developers building speech features that work offline on their own devices. It covers transcription, speech generation and audio analysis, with processing kept local.
1.4KUpdated 8 months agoApache-2.0
macOS#Batch processing#GGUF#Hugging Face integration
OuteTTS is a local text-to-speech library for developers adding generated speech to their applications. It uses reference audio to reproduce a speaker's voice, including their style, emotion and accent.
6.4KUpdated 1 day agoApache-2.0
Docker · Web
docTR reads text from PDFs and images for developers building document processing applications and researchers comparing OCR models. It locates individual words and recognizes their text on your own hardware.
2.3KUpdated 1 week agoMIT
#Hugging Face integration#Multilingual
Model2Vec is a Python package for developers who need text embeddings for search or classification on their own hardware. It turns sentence transformers into smaller static embedding models, trading a small drop in performance for lower model size and faster CPU inference.
34.1KUpdated 3 weeks agoMIT
macOS · Windows · Linux#Multilingual#Structured output
spaCy is a Python library for developers and data scientists who need to extract structured information from text in their own applications. It runs on macOS, Windows and Linux, with model packages that become part of your application. It's free and open source under the MIT license.
4.7KUpdated 9 months agoMIT
Windows · Linux#Batch processing#Quantization#Speculative decoding
ExLlamaV2 and V3 are local LLM inference libraries for developers building AI applications or hosting models on consumer GPUs. V2 is archived for now and isn't currently maintained; development continues in V3.
3.9KUpdated 6 months agoMIT
Web#Hugging Face integration
Stable Audio Tools is a Python toolkit for developers and audio researchers who want to generate audio or train models on their own recordings. It supports work with pretrained models as well as training from scratch.
3.6KUpdated 2 hours agoApache-2.0
macOS · Windows · Linux · iOS · Android · Web#Agent Skills#Hugging Face integration#Multimodal input
LiteRT is Google's framework for developers adding machine learning and generative AI to apps that run on users' devices. It succeeds TensorFlow Lite and supports local inference without a server round trip.
10.3KUpdated 1 week agoApache-2.0
macOS · Windows · Linux · Docker · Web#ONNX#Quantization#Streaming inference
RealtimeSTT is a Python library for developers building voice assistants, dictation tools and apps that turn live audio into text. It can show text while someone speaks and produce a final transcript when they finish.
36.5KUpdated 2 weeks agoApache-2.0
macOS · Windows · Linux#Batch processing#Multilingual
HanLP is a multilingual natural language processing toolkit for developers and researchers who need structured text analysis. Its native Python API runs models on your own hardware. The separate REST API sends text to a hosted server.
3.5KUpdated 2 weeks agoApache-2.0
#Hugging Face integration#ONNX#Quantization
Optimum is a collection of Python packages for developers adapting Hugging Face models to their deployment hardware. It extends Transformers, Diffusers, TIMM and Sentence Transformers with tools for faster inference and training.
7.9KUpdated 5 days ago
#Batch processing#Multilingual
Stanza is a Python NLP library for developers and researchers who need to analyze text in multiple human languages on their own hardware. It turns raw text into linguistic annotations, such as word forms, grammatical relationships, and named entities. It's open source under Apache 2.0.
1.3KUpdated 3 days agoApache-2.0
#Context compression#Distributed execution#Hugging Face integration
kvpress is a Python library for developers and researchers reducing the memory needed for long-context LLM inference. It compresses the key-value cache that grows as a model processes longer inputs.
5.6KUpdated 2 years agoApache-2.0
macOS#Hugging Face integration
Parler-TTS is a text-to-speech library for developers adding speech to applications and researchers adapting speech models. It generates audio with a voice style you describe in text.
13.3KUpdated 3 weeks agoMIT
#Quantization
FlashMLA is an attention library for developers building DeepSeek-V4.1 inference systems on their own hardware. It requires NVIDIA SM100 or SM103 GPUs, or Huawei Ascend 950 NPUs.
1.9KUpdated 4 months agoApache-2.0
macOS#MLX#Streaming inference
Magenta RealTime is a local AI music model and synthesis engine for musicians playing AI instruments and developers building interactive music apps. It generates a continuous audio stream on supported Apple Silicon Macs.
19.7KUpdated 6 days agoApache-2.0
Docker#LoRA#Multimodal input#Prompt caching
KTransformers is a research framework for developers running or fine-tuning large language models on their own hardware. It focuses on mixture-of-experts (MoE) models, using CPU resources alongside GPUs to work within limited GPU memory.
30.3KUpdated 1 month ago
macOS · Linux · iOS · Android · Web#Image-to-image#ONNX#PII redaction
InsightFace is a face and person analysis toolkit for developers building identity verification, access control and media applications. Its Python tools and recognition server can process data on your own hardware.
4.5KUpdated 2 months ago
Windows · Linux#Batch processing
tiny-cuda-nn is a neural network framework for developers and researchers building small models, NeRF renderers and neural graphics applications. It handles training and inference on local NVIDIA GPUs.