FreeToken vs llama.cpp: MoE speed and RAM requirements

FreeToken is an inference engine for large mixture-of-experts models that uses system RAM to hold weights and GPU memory as an expert cache. The video explains how this approach differs from llama.cpp's static layer offloading when you run models locally.

The engine keeps recently used experts in a global LRU cache. When an expert is missing, it can transfer the weights to the GPU or compute the result on the CPU. A hardware measurement determines how it divides that work between the two paths.

The speaker cites paper results of 22 to 25 tokens per second for DeepSeek V4 Flash on an RTX 5090, and 39.3 tokens per second for a 35B model on an 8GB RTX 4060 laptop. These are the developers' benchmarks. The independent tests described vary substantially, including a coding workload averaging 9.66 tokens per second. Tail latency also receives attention because a slow response can exceed an AI agent's timeout.

Memory capacity is the central requirement: the opening 753B example has 433GB of weights and uses 512GB of system RAM alongside 96GB of VRAM. The stated installation requirements are Linux on x86, NVIDIA driver R580 or newer, and CUDA 13. The speaker reports no macOS build, pending Apple Silicon and AMD support, and a reported crash beyond 48,000 prefill tokens. The comparison presents llama.cpp as the broader hardware option.