FastLLM

FastLLM is an open-source inference engine for people running large language models on their own machines or self-hosted servers. Its main draw is hardware flexibility: large mixture-of-experts (MoE) models can spread their weights across GPU memory, system RAM and disk when they don't fit entirely on a graphics card.

It runs on Linux and Windows with NVIDIA GPUs, supports AMD GPUs through ROCm on Linux, and can use CPUs alone. Multi-GPU inference lets several cards share a model. Disk-assisted execution makes use of SSD capacity, though its performance depends on random-read speed. The C++ core doesn't require PyTorch, and the project uses the Apache License 2.0.

Model support includes Qwen, DeepSeek-V4, Kimi-K3 and GLM, alongside Gemma4 and MiniMax-M2. It reads Hugging Face Safetensors, AWQ models, its own exported format and selected GGUF models. Quantization support depends on the model and hardware, so compatibility isn't uniform across formats or backends.

For application developers, the server provides APIs compatible with OpenAI Chat Completions, OpenAI Responses and Anthropic Messages. It supports streaming replies, tool calls and separate reasoning content when the model allows them. Dynamic batching and prefix caching support concurrent requests; compatible models can also use speculative decoding.

There's a browser chat interface with saved conversations and attachments. Its agent workspace can connect local models to coding tasks, including file edits and command execution.

External coding agents require separately installed integrations. Supported model features and acceleration paths depend on the selected model, format and hardware.