TabbyAPI

TabbyAPI is a self-hosted LLM API server for people who want to run ExLlamaV3 models on their own hardware. It’s built for personal projects and small groups; its maintainers don’t recommend it for production servers.

• OpenAI-compatible chat: Serve chat completions and tool/function calls through an API that works with OpenAI-style clients.
• Model management: Load and unload Exl3 or FP16/BF16 models, and download models from Hugging Face.
• Constrained responses: Limit output with JSON schemas, regular expressions or EBNF grammars.
• Concurrent inference: Handle simultaneous requests with continuous batching, or use draft models for speculative decoding.
• Embedding models: Serve embeddings alongside text generation.

TabbyAPI runs on the ExLlamaV3 backend and has a CUDA Docker image for NVIDIA GPUs. Parallel batching with paged attention supports NVIDIA Ampere GPUs and newer. It also supports AI Horde and Hugging Face-compatible Jinja2 chat templates. The project is open source under AGPL-3.0.