MLX tutorial: distributed inference across four Macs

This Apple session explains how to run models locally across four M3 Ultra Macs with MLX. It covers cluster wiring, distributed inference and fine-tuning, then introduces Python, Swift and C++ APIs for more control over distributed operations.

The setup uses RDMA over Thunderbolt 5, which the speaker says is supported starting with macOS 26.2. JACCL (pronounced Jackal) supplies collective communication operations. The tutorial compares mesh connections for lower latency with ring connections for bandwidth, then enables RDMA and reboots each Mac. MLX launch starts jobs over SSH using a JSON host file; the required libraries and executable must be available on every node.

For inference, the session explains tensor parallelism, which splits each layer across machines, and pipeline parallelism, which assigns groups of layers to different machines. Tensor parallelism is the stated MLX LM default. Pipeline support varies by model, and the speaker explains that sequential layer execution does not speed up individual token generation.

The Qwen 3.6 27-billion-parameter demonstration reports nearly three times the token generation rate of one M3 Ultra. A separate fine-tuning example reports about 180 tokens per second on one Mac and 600 across four. These are session results; the speaker notes that inference gains depend on model size and architecture.