Text Generation Inference
A self-hosted LLM inference server with an OpenAI-compatible chat API, multi-GPU support and an Apache 2.0 license; its repository is archived.
Text Generation Inference (TGI) serves LLMs on your own hardware for developers building chat apps and text generation services. The project is archived and no longer receives updates.
- Model support: Serve Llama, Falcon, StarCoder, BLOOM, GPT-NeoX and T5, including fine-tuned models and Safetensors weights.
- Chat API and streaming: Connect applications through a Messages API compatible with OpenAI Chat Completions and stream tokens as the model generates them.
- GPU efficiency: Continuous batching groups incoming requests to improve throughput, tensor parallelism distributes inference across GPUs, and quantization with AWQ, GPTQ or bitsandbytes reduces VRAM requirements.
- Structured responses: Constrain generated output to predefined schemas for function calling and tool use, with controls for stop sequences and repetition penalties.
- Service monitoring: Track inference services with Prometheus metrics and OpenTelemetry tracing.
TGI is open source under Apache 2.0 and supports local or self-hosted deployment through an official Docker container. Flash Attention and Paged Attention optimizations apply to supported model architectures; other architectures have best-effort support. Access to private or gated Hugging Face models requires a Hugging Face token.
Logotyper
