MLX cluster setup: Kimi K2.5 and GLM 4.7 on Macs

Alex Ziskind demonstrates a local LLM cluster using four M3 Ultra Mac Studios, each with 512 GB of unified memory. The Kimi K2.5 model used here occupies 658 GB on disk, exceeding one machine's memory. He reports 23 tokens per second with its 4-bit version on two Macs and 29 on four. These measurements describe his setup, rather than guaranteed performance.

The GLM 4.7 comparison uses an 8-bit model occupying 375 GB. It produces about 14 tokens per second on one Mac and 23.7 across four. The different quantization levels matter when interpreting the results. Ziskind also demonstrates OpenCode connected to his server, but the larger model takes time to answer. He notes that the basic server does not keep the model loaded or provide a cache.

The setup walkthrough follows his mlx-jaccl-cluster repository. It covers Thunderbolt wiring, enabling RDMA, Python environments on every machine, a coordinator LAN address, and a JACCL host file. Passwordless SSH and cluster verification precede model downloads and benchmarks.

To run models locally across the cluster, he copies the model files to the same path on every Mac using an external SSD. He explains that offline flags prevent redundant automatic downloads. The final server exposes model discovery, chat completions and completions endpoints for a coding assistant client.