FreeToken vs llama.cpp: short and long context tests

FreeToken and llama.cpp take different approaches to running a model larger than GPU memory. The speaker tests an approximately 35GB mixture of experts checkpoint on an RTX 5060 Ti with 16GB VRAM. FreeToken keeps the expert bank in system RAM and caches selected experts on the GPU, so the setup still needs enough system memory for the model.

The walkthrough covers a Python environment, the FreeToken CLI, a bandwidth benchmark and an OpenAI-compatible server endpoint. Automatic configuration allocates 332 expert slots. The comparison uses an FP8 checkpoint for FreeToken and a Q8 GGUF checkpoint for llama.cpp from the same model family, rather than identical model files.

For a short Python coding request, the speaker reports 12 seconds for FreeToken and 5.6 seconds for llama.cpp. With an approximately 4,000-token code sample containing a planted bug, FreeToken finishes the uncached request in 14.6 seconds versus 31.4 seconds. Repeating that prompt reduces the gap to 10.7 versus 13.9 seconds. Both engines reportedly identify the bug correctly.

FreeToken draws more active power in these tests: about 73W on average versus 46W. Reducing its expert cache while the server runs lowers GPU memory use but increases completion time. These results suggest a workload-dependent choice for a local LLM: llama.cpp wins the short request here, while FreeToken benefits from longer context on an oversized model.