Route Strata and llama.cpp through llama-swap

The presenter tests Strata, an inference engine specialized for Qwen 3.8 Flash, and connects it to the rest of a local model setup through llama-swap. On an RTX 3060 with 12GB VRAM and 64GB system RAM, they report prompt processing around 900 tokens per second and generation around 40. Their earlier llama.cpp configuration processed prompts around 120 tokens per second. These are measurements from one setup, not performance guarantees.

A shopping agent illustrates why prompt processing matters: the presenter reports 26 model calls and roughly 20 input tokens for every output token. The technical discussion covers larger prompt blocks, streamed expert weights, sparse attention, an expert cache and multi-token prediction.

The practical configuration puts Strata and the existing llama.cpp router behind llama-swap. Clients request a model by name. Separate entries represent router presets and Strata configurations, while an exclusive group keeps conflicting backends from occupying the GPU together. Requesting the Flash model stops the router and starts Strata; requesting the other model starts the router again. An idle timeout unloads Strata and releases its pinned RAM. The author links the configuration used in the video.

The full-model example uses 64GB system RAM, while a pruned coding variant is discussed for 32GB machines. Strata processes one request at a time, locks about 40GB RAM in the tested configuration, and may freeze the system temporarily during initial loading. The presenter reports that their client loses loading-progress information through llama-swap.

An experimental llama.cpp fork with ported optimizations reaches about 620 prompt tokens per second in another reported test. This is separate from the earlier baseline. The presenter also notes CPU/GPU numerical differences that can affect repeatability; the comparison does not establish equal outputs or universal model quality.