LMCache vs vLLM prefix caching: gains and tradeoffs

LMCache is an open source KV cache layer for self-hosted LLM serving. The video compares its tiered storage approach with vLLM's automatic prefix caching. Both reuse earlier prompt computations, but the speaker explains that GPU memory limits and separate worker processes can prevent a serving engine from retaining useful cache blocks.

LMCache moves cache storage beyond GPU memory into system RAM, local flash and remote object storage. The presenter cites a Google benchmark using eight H100 GPUs and a 70 billion parameter Llama model: at 100,000 tokens of shared context, time to first token fell 79% and input throughput rose 264%. A separate shared-cache test reportedly reduced mean first-token latency from 3.98 seconds to 0.29 seconds. These are results from specific workloads, rather than expected gains for every local LLM deployment.

The comparison also covers cases where LMCache loses. In a sweep without GPU memory pressure, the speaker reports 1,956 tokens per second for the added cache layer versus 3,061 for the built-in cache at a 75% hit rate. The reported crossover around 250,000 to 300,000 tokens applies to that test's hardware; network bandwidth also affects whether fetching cached data beats recomputation.

Deployment cautions include consistent Python hash seeds across workers and stable prompt prefixes. The speaker cites customer traffic where context truncation reduced cache hits from 85% to 45%.