LLM Inference Engines and Libraries

WebLLM is an engine for developers building browser-based AI assistants and Chrome extensions. It runs language models on the user's device, so applications don't need a separate inference server to process prompts.

openWakeWord is a wake word detection library for developers building voice apps that listen for a chosen word or phrase. It lets an application detect voice triggers on its own hardware.

TensorSharp runs local AI models for .NET developers and people who want chat, media generation and agent tools on their own hardware. You can embed it in a C# application or use its CLI, browser interface or companion app, TensorAgent.

PaddleNLP is a Python toolkit for developers training and deploying language models on their own infrastructure. Built on PaddlePaddle, it covers pretraining, fine-tuning, model compression and inference across several accelerator families.

MeloTTS is a Python text-to-speech library for developers adding spoken output to their applications. It runs locally and can generate speech in real time on a CPU, so a dedicated GPU isn't required for inference.

Transformers.js is a JavaScript library for developers adding on-device AI to web apps. It runs models in the browser, so processing inputs doesn't require a separate inference server.

Stable Diffusion Core ML is a developer toolkit for adding local AI image generation to Mac, iPhone, and iPad apps. It pairs model conversion tools with a Swift package for embedding generation in native software.

FlagEmbedding is a Python toolkit for developers building semantic search and retrieval-augmented generation (RAG). It runs embedding models to find relevant text and rerankers to refine the order of retrieved results.

DeepEP handles communication between GPUs for teams training or serving mixture-of-experts (MoE) models on their own servers. It moves tokens to the GPUs that hold the selected experts and combines their outputs.

MediaPipe is a toolkit for developers adding on-device AI to applications. It provides pretrained models and task APIs for applications that need to process images, video, text or audio on the user's device.

ONNX Runtime is an AI inference and training engine for developers adding models to apps and services. It supports deployment on a user's device or a server, including applications that need the same model format across platforms.

RapidOCR is an OCR toolkit for developers who need to extract text from images within their own applications. It uses models derived from PaddleOCR and can run recognition offline on local hardware.

Candle is a Rust machine learning framework for developers embedding local AI in applications or deploying models on their own servers. It supports inference and training, including small CPU deployments for serverless applications.

Parler-TTS is a text-to-speech library for developers adding speech to applications and researchers adapting speech models. It generates audio with a voice style you describe in text.

BitNet is Microsoft's local LLM inference framework for developers and researchers working with compatible 1-bit models. Its bitnet.cpp runtime focuses on faster inference and lower energy use on your own hardware.

GPT4All is a local AI chatbot for people who want language models and document conversations on their own computer. Model inference and LocalDocs processing run on the device without a cloud connection.

spaCy is a Python library for developers and data scientists who need to extract structured information from text in their own applications. It runs on macOS, Windows and Linux, with model packages that become part of your application. It's free and open source under the MIT license.

DeepFace is a Python library for developers adding face recognition and facial analysis to applications. It brings several recognition models behind a shared interface and handles the image preparation needed to compare faces.

llama.rn is a library for developers adding local LLM features to React Native apps on iOS and Android. It brings llama.cpp inference onto the device, so an app can process model requests on the user's hardware.

TorchAO is a PyTorch library for developers who want to train and run models on their own hardware with less memory. It supports language models and image or video generation, and works with torch.compile and FSDP2 across most Hugging Face PyTorch models.

Cactus is an on-device AI engine for developers adding automation to mobile apps, wearables and other small devices. Its Needle model turns requests into tool calls locally, including when the device has no internet connection.

RealtimeSTT is a Python library for developers building voice assistants, dictation tools and apps that turn live audio into text. It can show text while someone speaks and produce a final transcript when they finish.

ncnn is a C++ inference framework for developers adding on-device AI to mobile, desktop and embedded applications. It focuses on a small memory footprint and doesn't require third-party runtime dependencies.

Nexa SDK is an on-device AI framework for developers building apps that run models on users' hardware. It covers language, vision and audio tasks through a shared interface for local inference.

Flair is a free Python library for developers and researchers who need to extract information from text or train their own natural language processing models. It combines pretrained text analysis models with tools for building word and document embeddings, all within a framework built directly on PyTorch.

Sonar serves language and multimodal models for developers and teams running AI applications on their own hardware. It builds on vLLM and works with Hugging Face-compatible models.

TensorRT-LLM is an inference library for developers serving LLMs on their own NVIDIA GPUs. It supports application-level inference and server deployments, with performance optimizations for workloads spread across one or several machines.

RealtimeTTS is a Python text-to-speech library for developers building voice assistants and other apps that need spoken responses before the full text is ready. It accepts both complete text and incoming LLM output.

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

InsightFace is a face and person analysis toolkit for developers building identity verification, access control and media applications. Its Python tools and recognition server can process data on your own hardware.

llama-cpp-python is for developers adding local LLM inference to Python applications or connecting existing API clients to models on their own hardware. It brings llama.cpp into a Python library and a self-hosted server.

FastDeploy is a self-hosted inference toolkit for teams deploying language and vision-language models on Linux servers. Built on PaddlePaddle, it supports offline inference and online serving.

FastEmbed is a Python library for developers building semantic search and retrieval-augmented generation (RAG). It turns text into vectors and evaluates search results on your own hardware without requiring a cloud embedding API.

Accelerate is a Python library for developers and researchers who write their own PyTorch training loops. It handles hardware and distributed execution so they can keep control of how their models train.