Magnitude vs llama.cpp: Mac setup and speed claims

Magnitude is an open source local LLM inference engine that tests kernel configurations on the user's hardware and caches the fastest setup. The walkthrough downloads a model on Apple Silicon, compares it with llama.cpp using the same prompt, and connects Claude Code through localhost APIs compatible with OpenAI and Anthropic.

The speaker separates prefill, which processes the prompt, from decode, which generates the response. A developer benchmark used Qwen 3.6 35B A3B at 4-bit with 64,000 tokens of context on an M4 Pro with 48 GB of memory. The reported decode gain was 92%, while prefill improved by 9%. The cache settings differed: llama.cpp used a 16-bit KV cache after its compressed path failed, while Magnitude compressed its cache by default.

Reported independent results vary. One M5 Max test put Rapid MLX at 175 tokens per second and Magnitude at 161; other users reported llama.cpp ahead. The speaker says version 0.2.5 does not yet use the M5's new Metal Matrix hardware and recommends more comparisons before production use.

The setup discussion specifies macOS 13 or newer and CPU-only operation on Intel Macs. Downloaded models remain after app deletion, and idle models need to reload before responding. The speaker could not find documentation for importing a custom GGUF file.