Distributed Llama

Distributed Llama runs language models across your own networked computers. It's for people who want to share inference work and model memory across several devices instead of fitting the whole workload on one machine.

• Distributed inference: Split model computation and RAM use across computers connected over Ethernet, with a coordinating computer that also handles part of the workload.
• CPU support: Run on ARM or x86_64 AVX2 CPUs, including Raspberry Pi hardware.
• Model support: Use supported models including Llama 3.3 and Qwen 3, with CPU support for Qwen 3 mixture-of-experts models and a converter for Hugging Face models.
• Chat and application access: Talk to a model through a command-line chat interface, measure inference with a benchmark, or connect other software through the API server.

The C++ runner supports Linux, macOS and Windows and is open source under the MIT license. Model memory is divided across the cluster, though the coordinating computer needs slightly more RAM than the workers.

Clusters must contain one computer or a power-of-two total, and can't exceed the model's number of KV heads. Supported precision combinations are q40 models with q80 synchronization buffers, or f32 models with f32 buffers.