LLM Inference Engines and Libraries

Diffusers is a Python library for developers and researchers building image, video and audio generation into applications or running models on their own hardware. It supports pretrained generation pipelines as well as custom diffusion systems.

Flash Linear Attention is a Python library for researchers and developers building language models on their own hardware. It supplies reusable components for exploring linear attention, sparse attention and state space architectures.

GLiNER is a Python framework for extracting named entities from text using small models that run locally on CPUs and consumer hardware. It's for developers building document processing, personal data detection or search systems who need to choose their own entity types without training a model for each task.

DiffSynth-Studio is a diffusion model engine for developers and researchers generating media or training models on their own hardware. It combines inference and training, with memory-saving features for consumer GPUs.

mistral.rs is an inference engine for developers who want to run AI models on their own computer or server. It supports browser chat and inference embedded directly in Python or Rust applications.

Kokoro is a text-to-speech model and inference library for developers adding speech generation to personal projects or production applications. It runs on your own hardware and uses the compact Kokoro-82M model.

LitGPT is a Python toolkit for developers and researchers who want to train and deploy language models on their own hardware. Its directly editable model implementations suit custom applications and research experiments.

node-llama-cpp is a library for developers adding local AI to JavaScript and TypeScript applications. It uses llama.cpp to run GGUF models on your hardware and works in Node.js, Bun and Electron.

Apache TVM is a machine learning compiler for developers optimizing models for deployment on their own hardware. It compiles model computations into modules that applications can embed, with targets spanning data center GPUs and edge devices. Code is licensed under Apache 2.0.

FlashInfer is a GPU kernel library for developers building LLM inference systems on their own hardware or servers. It supplies the attention, matrix multiplication and mixture-of-experts operations used by model serving software.

picoLLM is an on-device LLM inference SDK for developers adding text generation to their apps. Inference works offline, but a Picovoice account and AccessKey are required, and key validation needs an internet connection.

LM Format Enforcer is an open-source Python library for developers who need a local LLM to return structured data their applications can parse. It constrains text during generation, so a prompt's formatting request doesn't have to carry the whole burden. The library uses the MIT license.

MusicGen is Meta AI's model for generating music from written descriptions. It's part of AudioCraft, a PyTorch library for developers and audio researchers building music generation tools in their own computing environment.

FlashRank is a Python library that runs reranking models locally on CPU, without Torch or Transformers. It's for developers building search systems or retrieval-augmented generation (RAG) apps who want to reorder retrieved passages by relevance before passing them to an LLM.

EmbedAnything is a local embedding pipeline for developers building semantic search and retrieval-augmented generation (RAG) applications. It turns documents and media into vectors for indexing in a vector database.

ggml is a library for developers building machine learning runtimes and local AI applications. It provides tensor computation across different kinds of hardware; GGUF is the model file format covered by the project.

MLX LM is a Python package for developers and researchers who want to generate text and adapt language models on Apple Silicon Macs. It uses MLX for local inference and training, with access through Python or a terminal.

Stable Audio Tools is a Python toolkit for developers and audio researchers who want to generate audio or train models on their own recordings. It supports work with pretrained models as well as training from scratch.

Ultralytics YOLO is a Python computer vision library for developers building applications that interpret images and video. Models can run on hardware you control for tasks such as visual inspection, inventory monitoring and robotics perception.

KittenTTS is a Python text-to-speech library for developers adding local speech generation to apps or producing spoken audio from text. It supports voice cloning without model training and doesn't require a GPU.

FlashMLA is an attention library for developers building DeepSeek-V4.1 inference systems on their own hardware. It requires NVIDIA SM100 or SM103 GPUs, or Huawei Ascend 950 NPUs.

MLC LLM is a compiler and deployment engine for developers running language models on their own hardware or building AI into apps. Its shared MLCEngine supports deployment across desktop computers, mobile devices and browsers.

sherpa-onnx is a toolkit for developers building speech features that work offline on their own devices. It covers transcription, speech generation and audio analysis, with processing kept local.

PaddleX is a low-code toolkit for developers building AI applications with PaddlePaddle models. It brings pretrained pipelines and model training into one tool for applications that process documents, images or time-series data.

kvpress is a Python library for developers and researchers reducing the memory needed for long-context LLM inference. It compresses the key-value cache that grows as a model processes longer inputs.

LMDeploy is an open-source toolkit for developers who want to run language and vision-language models on their own GPU hardware. It supports offline batch processing and model servers for applications.

Magenta RealTime is a local AI music model and synthesis engine for musicians playing AI instruments and developers building interactive music apps. It generates a continuous audio stream on supported Apple Silicon Macs.

Intel IPEX-LLM is a library for developers running or fine-tuning language models on Intel hardware. The project is archived and no longer maintained, and Intel has identified known security issues in the code.