FastChat

FastChat is a self-hosted LLM platform for developers and researchers who want to serve chat models on their own hardware and compare their responses. It powers Chatbot Arena and includes a locally hosted side-by-side comparison interface. The Python code is open source under Apache 2.0; model licenses apply separately, including Llama's license for Vicuna.

Its browser chat UI and OpenAI-compatible API give people and applications access to the same models. Distributed serving can host several models at once or spread requests across workers for higher throughput. It supports Hugging Face Transformers and integrates with vLLM for batched serving, plus LangChain for application development.

Supported models include Vicuna, Llama 2, LongChat, Falcon and FastChat-T5. Weights can come from local folders or Hugging Face. Local models run on your hardware, while optional connections to OpenAI, Anthropic, Gemini and Mistral send requests to those providers.

Hardware support covers CPU-only inference, NVIDIA GPUs, AMD GPUs through ROCm, and Mac GPU acceleration through Metal on Apple Silicon or AMD hardware. Intel GPUs are also supported. Memory needs depend on the model: Vicuna-7B uses about 14GB of GPU memory or 30GB of system RAM on CPU. Compression and GPTQ or AWQ quantization can reduce memory use.

For model development, FastChat includes conversation fine-tuning code with LoRA support and MT-Bench evaluation. MT-Bench uses multi-turn questions and can use GPT-4 to judge model responses.