2,000+ AI tools you can run locally Add tool

SGLang

A self-hosted AI inference framework serves language, vision-language and diffusion models, with OpenAI API compatibility and broad hardware support.

Screenshot of SGLang website

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

Language and vision models: Serve models from families including Llama, Qwen and DeepSeek, with Hugging Face compatibility and an OpenAI-compatible API.
Image and video generation: The included SGLang Diffusion engine runs diffusion models through the same project and Python package.
Shared request processing: Prefix caching reuses work across requests with common text, while multi-GPU parallelism supports larger deployments.
Application responses: Stream replies, request tool calls and process multiple prompts through the serving framework.

SGLang is open source under the Apache 2.0 license and is available for Python or Docker deployments on your own hardware. Supported platforms include NVIDIA and AMD GPUs, Intel GPUs and Xeon CPUs, Google TPUs, Huawei Ascend NPUs and Moore Threads accelerators. Apple Silicon Macs are supported through Metal and MLX. Model compatibility depends on the hardware platform.

GPUs, Clusters and Edge Inference Servers and Clusters Batch processing Distributed execution LoRA Multimodal input OpenAI-compatible API Prompt caching Quantization Speculative decoding Structured output

Similar to SGLang

vLLM

58 videos

An open source LLM serving engine runs models on your hardware, with an OpenAI-compatible API and support for NVIDIA, AMD and CPU inference.

96.9KUpdated 20 hours agoApache-2.0

macOS · Docker#Batch processing#Distributed execution#GGUF

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

GPUs, Clusters and EdgeInference Servers and Clusters

llama.cpp

92 videos

An MIT-licensed local LLM engine runs language and vision-language models on CPUs or GPUs, with quantization, hybrid execution and Docker support.

134.9KUpdated 21 hours agoMIT

Web#Code execution#GGUF#Hugging Face integration

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

Inference Libraries and BindingsInference Servers and ClustersLocal Model RunnersSelf-Hosted Chat Web UIs

Ollama

261 videos

An MIT-licensed local LLM runner and API for macOS, Windows, Linux and Docker, with optional cloud inference and coding agent integrations.

186.9KUpdated 21 hours agoMIT

macOS · Windows · Linux · Docker#GGUF#llama.cpp backend#Multimodal input

Ollama runs open models for people who want local AI applications, model chat or a backend for coding assistants. It provides a command-line interface and an API, with a choice between inference on your hardware and its cloud service.

Inference Servers and ClustersLocal Model Runners

TensorRT-LLM

4 videos

A self-hosted LLM inference library runs on NVIDIA GPUs, with OpenAI-compatible serving, a Python API and support for deployments across machines.

15.1KUpdated 20 hours ago

Docker#Batch processing#Distributed execution#LoRA

TensorRT-LLM is an inference library for developers serving LLMs on their own NVIDIA GPUs. It supports application-level inference and server deployments, with performance optimizations for workloads spread across one or several machines.

GPUs, Clusters and EdgeInference Libraries and BindingsInference Servers and Clusters

exo

6 videos

A local LLM runner splits models across macOS and Linux devices and provides OpenAI-, Claude- and Ollama-compatible APIs.

48.4KUpdated 1 week agoApache-2.0

macOS · Linux · Web#Distributed execution#Hugging Face integration#MLX

exo is a local LLM runner for people who want to run models too large for a single machine's memory. It joins devices into a cluster and runs inference on your own hardware.

GPUs, Clusters and EdgeInference Servers and ClustersLocal Model Runners

Lucebox

A local AI workstation runs large models offline, with an Apache 2.0 inference engine and APIs compatible with OpenAI and Anthropic clients.

3KUpdated 1 day agoApache-2.0

Windows · Linux · Web#Context compression#GGUF#Multimodal input

Lucebox is a local AI workstation for developers and teams who want to run models on their own hardware. Its inference engine is also available separately for supported AMD and NVIDIA machines.

GPUs, Clusters and EdgeInference Servers and ClustersLocal Model Runners