LMCache
An open-source cache layer for self-hosted LLM inference that reuses computed context with vLLM and stores it in CPU memory, local disk or remote backends.
LMCache helps teams serving LLMs reuse context that a model has already processed. It reduces repeated work on long prompts, conversation history and retrieved documents, shortening the wait for the first token and improving throughput.
- Persistent context storage: Keep computed context, known as KV cache, beyond GPU memory in CPU RAM, local SSDs or external storage, and reuse it across requests, sessions and engine instances.
- Reuse within prompts: CacheBlend finds reusable context beyond matching prompt beginnings, which helps when retrieved documents appear in different positions.
- Compression and transfer: Reduce cache memory use and move cached context between workers, including those that process prompts and those that generate responses.
- Cache monitoring: Track cache hits, storage movement and request-level performance to assess how reuse affects serving workloads.
LMCache integrates with vLLM and supports NVIDIA and AMD GPUs. It can run inside the inference engine or as a separate server; the separate server retains cached context across inference-worker restarts and failures. The inference engine still runs the model.
The code is open source under Apache 2.0. Storage backends include Redis, Valkey and S3-compatible object storage. CPU memory and local disk keep caches on the serving infrastructure, while external backends send cached context to the selected storage service.
