Mac Studio M5 Ultra vs hosted models for local AI

This comparison examines local model serving on Mac Studio hardware against hosted inference. The speaker separates prefill, which reads the prompt, from decode, which generates the response, and argues that long coding-agent contexts make prompt processing especially important.

The video cites a Kimi K3 test across four M3 Mac Studios, reporting 238 tokens per second for prefill and 14.7 for decode. These are reported M3 results, not measurements of the proposed M5 Ultra setup. The speaker says independent M5 Ultra tests were unavailable at recording time.

The exo discussion covers clustering over Thunderbolt, communication bottlenecks, recovery-mode configuration and reported stability problems. Adding machine memory or bandwidth figures together does not establish end-to-end agent performance. The video also discusses model compression and the limits of concurrent long contexts.

The closing recommendation favors one appropriately sized machine when data must remain local or an unattended agent can tolerate slow output. For other workflows, the speaker suggests evaluating the intended model through a host before buying hardware. This is a reported hardware comparison, not an exo installation guide.