llama.cpp, Speculators and Gemma 4 decoding tutorial

This workshop explains speculative decoding for people who run models locally. A small drafter proposes several tokens, then a larger target model checks them in one pass. The speakers explain how this reduces repeated weight reads during decoding, with extra memory needed for the drafter. Poor acceptance rates can erase the benefit or cause a slowdown.

The opening section covers prefill, KV cache and throughput before introducing serving options in llama.cpp and Transformers. The presenter describes llama.app as a llama.cpp frontend for macOS and Windows that automatically uses supported drafters. Linux users are directed to prebuilt llama.cpp binaries.

The Speculators session follows drafter training through data preparation, hidden-state extraction and deployment. It compares storing hidden states on disk with generating them during training, then discusses sliding-window attention and adaptive verification. Reported speedups vary with concurrency and how closely the workload matches the training data.

The final talk examines Gemma 4 E2B and its drafter with fewer than 100 million parameters. It explains KV cache reuse and token clustering to reduce drafting work. The speakers describe speculative decoding as preserving the target output distribution, while warning that kernel differences can prevent token-for-token identical results. Memory limits, especially on mobile, remain a constraint.