Tools tagged with "Prompt caching"

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

FreeLLMAPI gives developers one API for experimenting with cloud models and local backends. The router runs locally, but cloud inference still sends requests to the providers you enable.

llama.vscode is a coding assistant for developers who want AI suggestions, chat, and an agent inside VS Code. It lets you use models on your own hardware or connect to external providers.

OmniRoute is an AI gateway for developers who want their coding assistants and other AI clients to share one set of provider connections. It routes requests across models and accounts when a service fails, reaches a rate limit or runs out of quota.

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

LightLLM is a self-hosted LLM inference and serving framework for developers and teams running models on their own GPU servers. Its focus is serving requests efficiently: it schedules work around predicted memory use and handles text encoding, model inference and output decoding asynchronously to improve GPU utilization.

KTransformers is a research framework for developers running or fine-tuning large language models on their own hardware. It focuses on mixture-of-experts (MoE) models, using CPU resources alongside GPUs to work within limited GPU memory.

MTPLX is a native Mac app, command-line tool and local LLM server for people running chat or coding models on Apple Silicon. It uses a model's own multi-token prediction heads to accelerate generation without a separate draft model.

LMCache helps teams serving LLMs reuse context that a model has already processed. It reduces repeated work on long prompts, conversation history and retrieved documents, shortening the wait for the first token and improving throughput.

AIBrix provides infrastructure components for teams serving large language models on Kubernetes. It's aimed at organizations that need to manage inference capacity, request routing and models across a cluster.

FastLLM is an open-source inference engine for people running large language models on their own machines or self-hosted servers. Its main draw is hardware flexibility: large mixture-of-experts (MoE) models can spread their weights across GPU memory, system RAM and disk when they don't fit entirely on a graphics card.

Paritok is a self-hosted gateway for developers whose coding agents send large file reads, tool output and conversation histories to an LLM. It reduces that input before the request reaches the upstream model.

Voice-Clone-Studio brings voice cloning and speech production into one browser interface for people making podcasts, audiobooks and dialogue recordings. It combines several engines with tools for preparing samples and training custom voices.

Hypha is a TypeScript AI agent framework for developers building applications that need controlled tool use and recoverable work. It separates an agent's planning and tool choices from the execution layer that decides which actions may proceed.

tRPC Agent Go is an open-source framework for developers building agent services with local automation in Go. It can execute skill scripts in an isolated workspace and includes deployable HTTP servers for agent interfaces. The documented OpenAI and DeepSeek workflows require a provider API key, so running the service yourself doesn't make those model calls local.

vllm-mlx is a local LLM inference server for developers who want to connect AI apps and coding assistants to models running on an Apple Silicon Mac. One process exposes both OpenAI-compatible and Anthropic-compatible APIs. It's open source under Apache 2.0 and runs on macOS with at least 8 GB of unified memory.

Magnitude is a local LLM engine for people who want to run AI agents on their own hardware. It adapts model execution to your machine and connects to coding assistants you already use.

Deltafin runs full Kimi K3 on a single device for researchers and developers exploring large models on their own hardware. It preserves the original expert weights and complete expert bank, but responses are slow and long conversations take substantially longer than short completions.

Chidori is an AI agent framework for developers building long-running workflows that need crash recovery or human approval. Agents use ordinary async TypeScript, with saved runs that developers can revisit for debugging and testing.

Sonar serves language and multimodal models for developers and teams running AI applications on their own hardware. It builds on vLLM and works with Hugging Face-compatible models.

TensorRT-LLM is an inference library for developers serving LLMs on their own NVIDIA GPUs. It supports application-level inference and server deployments, with performance optimizations for workloads spread across one or several machines.

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

Speakr is a self-hosted transcription and note-taking app for people who need to revisit meetings, interviews or lectures. It pairs recordings with searchable transcripts and AI summaries.

FastDeploy is a self-hosted inference toolkit for teams deploying language and vision-language models on Linux servers. Built on PaddlePaddle, it supports offline inference and online serving.

MLX VLM is a Python package for developers and researchers who want to run multimodal models on their own Mac. It supports local inference and fine-tuning through MLX.

Langroid is a Python framework for developers building applications where AI agents divide up a task. Each agent can maintain its own conversation and use documents or tools, then delegate work to other agents.

OfflineLLM is an on-device AI chat app for Android users who want to run models locally without sending conversations to a cloud service. It works fully offline and has no network permissions.

Slotstream runs Qwen3.8-Flash-Next on Apple Silicon Macs with too little RAM to hold the whole model. It's aimed at Macs with 16 to 64 GB of memory, for people who want local chat, answers about pictures or a backend for coding assistants.

ChatGPT Subtitle Translator translates SRT subtitles for people who need translated text to stay matched to the original entries. It supports local Ollama models and OpenAI-compatible providers, so you can choose where translation runs.

Headroom reduces the context that LLM apps and coding agents send to a model. It's for developers whose prompts fill up with repetitive tool results, logs, file contents and retrieved documents. Compression runs locally, but model requests still go to the chosen LLM provider.

FlashInfer is a GPU kernel library for developers building LLM inference systems on their own hardware or servers. It supplies the attention, matrix multiplication and mixture-of-experts operations used by model serving software.

vLLM Production Stack is an open source LLM serving stack for teams running models on Kubernetes GPU clusters. It lets applications move from a single vLLM instance to a distributed deployment without changing application code.

TypingMind is an AI chat frontend for people who want to use different model providers in one workspace. It brings conversations, project materials and custom AI agents into a shared interface.

MLX LM is a Python package for developers and researchers who want to generate text and adapt language models on Apple Silicon Macs. It uses MLX for local inference and training, with access through Python or a terminal.