LLM Inference Engines and Libraries

The engines and bindings that run models inside your own code: llama.cpp, MLX, ONNX Runtime and Hugging Face Transformers.

100+ tools
Favicon of LiteRT-LM

LiteRT-LM

2 videos
An open-source framework for running LLMs locally on mobile, desktop and browser apps, with LiteRT acceleration and Apache 2.0 licensing.

6.9KUpdated 2 hours agoApache-2.0

macOS · Windows · Linux · iOS · Android · Web#Hugging Face integration#Multimodal input#OpenAI-compatible API

Favicon of MLX Audio

MLX Audio

1 video
A local AI audio library for Apple Silicon that generates and transcribes speech, processes audio, and provides an OpenAI-compatible API.

8.3KUpdated 3 hours agoMIT

macOS · Web#Batch processing#MLX#Multilingual

Favicon of MeloTTS

MeloTTS

2 videos
A local text-to-speech library that generates multilingual speech in real time on a CPU and supports training on custom datasets.

7.7KUpdated 2 years agoMIT

Docker · Web#Multilingual

Favicon of ONNX Runtime

ONNX Runtime

11 videos
An open source AI runtime runs ONNX models on desktop, mobile and web platforms, with CPU, GPU and NPU acceleration.

22.3KUpdated 2 days agoMIT

macOS · Windows · Linux · iOS · Android · Web#Distributed execution#ONNX

A self-hosted LLM inference engine that serves Hugging Face-compatible models through OpenAI-compatible APIs on CPUs and supported accelerators.

1.9KUpdated 4 days agoAGPL-3.0

macOS · Windows · Linux · Docker#Batch processing#Distributed execution#Hugging Face integration

Favicon of FastEmbed

FastEmbed

3 videos
A Python library that generates local embeddings and reranks search results, with CPU or GPU support and Qdrant integration.

3.3KUpdated 12 hours agoApache-2.0

#Batch processing#Multilingual#ONNX

A Python speaker diarization toolkit identifies who spoke when with pretrained PyTorch models, fine-tuning and optional pyannoteAI access.

10.8KUpdated 2 days agoMIT

#Hugging Face integration#Speaker diarization#Voice activity detection

A local LLM inference engine with multimodal models, built-in agent tools, and OpenAI-compatible and Anthropic-compatible APIs for self-hosted apps.

7.8KUpdated 1 week agoMIT

macOS · Windows · Linux · Docker · Web#Code execution#GGUF#Hugging Face integration

An on-device AI SDK runs text, image and audio models on macOS, Windows and Linux, with GGUF and MLX support and an OpenAI-compatible API.

qualcomm/GenieXInference Libraries and Bindings

macOS · Windows · Linux#GGUF#Hugging Face integration#llama.cpp backend

A local LLM compression and inference toolkit supporting GPTQ, AWQ and GGUF, with CPU and GPU backends across Linux, macOS and Windows.

1.3KUpdated 12 minutes ago

macOS · Windows · Linux#GGUF#Hugging Face integration#LoRA

An open-source library accelerates Transformer training and inference on NVIDIA GPUs with low-precision support and PyTorch and JAX APIs.

3.6KUpdated 2 hours agoApache-2.0

#Distributed execution#Quantization

Favicon of OpenCLIP

OpenCLIP

1 video
A Python library runs and trains CLIP image-text models locally, with local checkpoint loading, Hugging Face weights and multi-GPU training.

14.3KUpdated 1 day ago

#Hugging Face integration#Multimodal input

Favicon of whisper.cpp

whisper.cpp

6 videos
An open-source speech-to-text engine that runs Whisper locally on desktop, mobile and in browsers, with CPU-only inference and optional GPU acceleration.

55.3KUpdated 2 days agoMIT

macOS · Windows · Linux · iOS · Android · Docker#Hugging Face integration#Quantization#Streaming inference

Open-source Python anomaly detection library with a shared detector API, benchmark-backed orchestration, and a BSD 2-Clause license.

10.1KUpdated 6 days agoBSD-2-Clause

#Agent Skills#MCP

Favicon of ModelScope

ModelScope

2 videos
AI model hub and open-source Python library for local inference, fine-tuning and evaluation, with hosted demos and cloud notebooks.

9.2KUpdated 3 days agoApache-2.0

Docker#Image-to-image#Inpainting#Multimodal input

An open-source speech toolkit that runs ONNX models locally for transcription, speech synthesis and speaker analysis on desktop, mobile and embedded devices.

16KUpdated 15 hours agoApache-2.0

macOS · Windows · Linux · iOS · Android · Web#Multilingual#ONNX#Speaker diarization

A local text-to-speech library generates speech with reusable voice profiles and supports llama.cpp, Hugging Face Transformers and ExLlamaV2.

1.4KUpdated 8 months agoApache-2.0

macOS#Batch processing#GGUF#Hugging Face integration

Favicon of docTR

docTR

1 video
An open-source Python OCR library that extracts text from PDFs and images locally with PyTorch, CPU or GPU support, and document layout detection.

6.4KUpdated 1 day agoApache-2.0

Docker · Web

A local text embedding library that runs on CPU, distills sentence transformers without a dataset, and integrates with LangChain.

2.3KUpdated 1 week agoMIT

#Hugging Face integration#Multilingual

Favicon of spaCy

spaCy

2 videos
A free, MIT-licensed Python NLP library that runs on macOS, Windows and Linux, with pretrained text pipelines and CPU or GPU processing.

34.1KUpdated 3 weeks agoMIT

macOS · Windows · Linux#Multilingual#Structured output

Local LLM inference libraries run models on NVIDIA GPUs with EXL2 or EXL3 quantization and an OpenAI-compatible API through TabbyAPI.

4.7KUpdated 9 months agoMIT

Windows · Linux#Batch processing#Quantization#Speculative decoding

An open-source Python toolkit generates audio locally and supports model training and fine-tuning, with a Gradio interface and an MIT license.

3.9KUpdated 6 months agoMIT

Web#Hugging Face integration

Favicon of LiteRT

LiteRT

3 videos
An open-source on-device AI framework runs models on mobile, desktop and web, with CPU, GPU and NPU acceleration and offline LLM support.

3.6KUpdated 2 hours agoApache-2.0

macOS · Windows · Linux · iOS · Android · Web#Agent Skills#Hugging Face integration#Multimodal input

A Python speech-to-text library transcribes audio locally, with live text updates, wake word activation and an optional self-hosted server.

10.3KUpdated 1 week agoApache-2.0

macOS · Windows · Linux · Docker · Web#ONNX#Quantization#Streaming inference

Open-source NLP toolkit for local text analysis on Linux, macOS and Windows, with CPU support and a separate hosted REST API. Apache 2.0 licensed.

36.5KUpdated 2 weeks agoApache-2.0

macOS · Windows · Linux#Batch processing#Multilingual

Favicon of Optimum

Optimum

1 video
An open-source model optimization toolkit adapts Hugging Face models for local and edge hardware through ONNX Runtime, OpenVINO and ExecuTorch.

3.5KUpdated 2 weeks agoApache-2.0

#Hugging Face integration#ONNX#Quantization

Open-source Python NLP library for local text analysis, with PyTorch models, GPU acceleration, and a Stanford CoreNLP interface. Apache 2.0 licensed.

7.9KUpdated 5 days ago

#Batch processing#Multilingual

A Python library that reduces LLM cache memory use with Hugging Face Transformers, multiple compression methods and support for cache quantization.

1.3KUpdated 3 days agoApache-2.0

#Context compression#Distributed execution#Hugging Face integration

An open-source text-to-speech Python library that generates speech with voice styles described in text and supports training your own models.

5.6KUpdated 2 years agoApache-2.0

macOS#Hugging Face integration

Favicon of FlashMLA

FlashMLA

1 video
An MIT-licensed attention library supports DeepSeek-V4.1 inference on NVIDIA SM100/SM103 GPUs and Huawei Ascend 950 NPUs.

13.3KUpdated 3 weeks agoMIT

#Quantization

A local AI music model and synthesis engine for Apple Silicon, with a standalone macOS app and an AUv3 plugin for digital audio workstations.

1.9KUpdated 4 months agoApache-2.0

macOS#MLX#Streaming inference

An open-source local LLM framework runs and fine-tunes large MoE models across CPUs and GPUs, with SGLang and LlamaFactory integrations.

19.7KUpdated 6 days agoApache-2.0

Docker#LoRA#Multimodal input#Prompt caching

Favicon of InsightFace

InsightFace

2 videos
A face analysis toolkit with self-hosted recognition on CPU or NVIDIA GPU, local person matching, and on-device video face redaction.

30.3KUpdated 1 month ago

macOS · Linux · iOS · Android · Web#Image-to-image#ONNX#PII redaction

A neural network framework trains and runs small models locally on NVIDIA GPUs, with C++/CUDA and PyTorch interfaces for Windows and Linux.

4.5KUpdated 2 months ago

Windows · Linux#Batch processing

Favicon of LogotyperLogotyper
Got an idea for an AI tool? I'm Eugene, the designer and developer behind Localhosted. I'll help you build it, from the first sketch to launch.
Start your project
Favicon of Logotyper
Favicon of LogotyperLogotyper
Got an idea for an AI tool? I'm Eugene, the designer and developer behind Localhosted. I'll help you build it, from the first sketch to launch.
Start your project
Favicon of Logotyper

More in Run Models Locally