EXO and GLM 5.1: Mac clusters and Spark inference

Alex Cheema explains EXO's approach to running a local LLM across several devices. The workshop covers memory limits, communication overhead and the different hardware demands of prefill and decode, then demonstrates model instances on a Mac cluster.

Four Mac Studios connect through Thunderbolt 5. Cheema says EXO's RDMA integration reduced communication latency from roughly 300 microseconds to single-digit microseconds, making tensor parallelism more practical. The GLM 5.1 demonstration uses a 4-bit model described as almost 400 GB, converted to MLX. His separate estimate of about $40,000 in hardware and 20 tokens per second concerns a setup for the full model, which he describes as roughly 1.5 TB in FP16.

A second demonstration assigns prefill to an Nvidia Spark and decode to a MacBook, transferring the KV cache over 10 gigabit Ethernet. Cheema reports about twice the speed for a large prompt, but says small prompts gain nothing. Network problems interrupt the demonstration; a later paper example takes about 7 seconds on one device and 4.8 seconds across both.

The session also exposes setup limits. RDMA requires Thunderbolt 5 and recovery-mode enablement, while mixed macOS 26.2 and 26.3 nodes trigger a compatibility warning. Existing models from other apps need manual migration into EXO. Cheema uses Tailscale for remote access and says EXO does not bundle it. His proposed 100x improvement in price to performance remains a forecast.