Qwen 3.8 27B: llama.cpp RPC setup on Mac and Nvidia PC

This tutorial tests Qwen 3.8 27B across a Linux PC with an RTX 3060 and an Apple Silicon Mac using llama.cpp RPC. The PC has 12 GB of VRAM; the Mac has 24 GB of unified memory. The speaker reports 6.8 tokens/s with RAM offloading on the PC, 16.4 after tuning the split and device order, and 22 with multi-token prediction (MTP) at an empty context. Throughput falls to about 19.5 tokens/s as context grows.

The setup requires RPC-enabled builds on both machines, a worker server, and the worker's address in llama-server. Sequential splitting assigns blocks of layers to each device. It adds memory capacity, but each device waits for the preceding block. In this test, keeping the output head on the PC reduced return traffic. MTP tuning also depended on sampling temperature: draft depth one beat depth three at temperature one.

The local LLM experiment reaches a configured 128K context with an F16 KV cache. Tests with models that already fit on one GPU became slower, while MoE tests showed little generation benefit. The comparison with a single multi-GPU box is an estimate, not a measured benchmark.

The guide covers device pinning, identical llama.cpp builds and Linux binary compatibility. Results use commit 84e908c62 from 2026-08-12. The speaker warns that RPC has no authentication and recommends restricting the port to specific machines, never exposing it to the internet, and avoiding root.