2,000+ AI tools you can run locally Add tool

vLLM

An open source LLM serving engine runs models on your hardware, with an OpenAI-compatible API and support for NVIDIA, AMD and CPU inference.

Alternative to:

DeepSeek

Screenshot of vLLM website

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

Concurrent serving uses PagedAttention and continuous batching to manage generation memory and process incoming requests together; prefix caching reduces repeated work.
Application APIs provide OpenAI-compatible endpoints, Anthropic Messages and gRPC support, with streaming responses, structured output and tool calling.
Model choice includes Hugging Face models such as Llama, Qwen, Gemma and DeepSeek, alongside multimodal, embedding, retrieval and classification models.
Quantized models support GGUF, GPTQ and AWQ, among other formats and methods that reduce model memory needs.
Distributed inference spreads model workloads across devices and supports serving multiple LoRA adapters.

vLLM is open source under Apache 2.0 and can run in Docker. Hardware support includes NVIDIA, AMD and Intel GPUs, plus x86, ARM and PowerPC CPUs. Hardware plugins extend support to Apple Silicon, Google TPUs, Intel Gaudi and other accelerators.

GPUs, Clusters and Edge Inference Servers and Clusters macOS Docker Batch processing Distributed execution GGUF LoRA Multimodal input OpenAI-compatible API Prompt caching Quantization Speculative decoding Streaming inference Structured output Tool calling

Similar to vLLM

llama.cpp

92 videos

An MIT-licensed local LLM engine runs language and vision-language models on CPUs or GPUs, with quantization, hybrid execution and Docker support.

134.9KUpdated 21 hours agoMIT

Web#Code execution#GGUF#Hugging Face integration

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

Inference Libraries and BindingsInference Servers and ClustersLocal Model RunnersSelf-Hosted Chat Web UIs

SGLang

13 videos

A self-hosted AI inference framework serves language, vision-language and diffusion models, with OpenAI API compatibility and broad hardware support.

38.8KUpdated 21 hours agoApache-2.0

#Batch processing#Distributed execution#LoRA

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

GPUs, Clusters and EdgeInference Servers and Clusters

Hermes Agent

105 videos

A self-hosted AI agent for macOS, Linux and Windows, with persistent memory, scheduled tasks and connections to hosted models or your own endpoint.

263.2KUpdated 19 hours agoMIT

macOS · Windows · Linux · Android · Docker#Agent Skills#Code execution#Multi-agent workflows

Hermes Agent is a self-hosted AI agent from Nous Research for people who want an assistant that remembers past work and handles recurring tasks. It runs on macOS, Linux and native Windows, or on a server you control. Model connections include Nous Portal, OpenRouter, OpenAI and your own endpoint; choosing a hosted provider sends model requests to that service.

AI Bots for Chat AppsAutomation and No-Code AIPersonal AI AssistantsTerminal Chat Clients

Ollama

261 videos

An MIT-licensed local LLM runner and API for macOS, Windows, Linux and Docker, with optional cloud inference and coding agent integrations.

186.9KUpdated 21 hours agoMIT

macOS · Windows · Linux · Docker#GGUF#llama.cpp backend#Multimodal input

Ollama runs open models for people who want local AI applications, model chat or a backend for coding assistants. It provides a command-line interface and an API, with a choice between inference on your hardware and its cloud service.

Inference Servers and ClustersLocal Model Runners

TensorRT-LLM

4 videos

A self-hosted LLM inference library runs on NVIDIA GPUs, with OpenAI-compatible serving, a Python API and support for deployments across machines.

15.1KUpdated 20 hours ago

Docker#Batch processing#Distributed execution#LoRA

TensorRT-LLM is an inference library for developers serving LLMs on their own NVIDIA GPUs. It supports application-level inference and server deployments, with performance optimizations for workloads spread across one or several machines.

GPUs, Clusters and EdgeInference Libraries and BindingsInference Servers and Clusters

llm-d

5 videos

An open-source LLM inference stack for self-hosted Kubernetes clusters that coordinates vLLM and SGLang across GPUs, TPUs, XPUs and CPUs.

5.3KUpdated 2 days agoApache-2.0

#Batch processing#Distributed execution#OpenAI-compatible API

llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.

Inference Servers and ClustersKubernetes and Cloud-Native AILLM Gateways and Routers