Mooncake
Share KV cache across serving instances and transfer data between prefill and decode workers through integrations with vLLM, SGLang and TensorRT-LLM.
Mooncake provides storage and data transfer components for teams running distributed LLM inference. Its Transfer Engine and Mooncake Store support shared caching and deployments that separate prompt processing from token generation.
- Shared KV cache: Pool cached context across serving instances so workers can reuse processed prompt prefixes, with host memory and SSD storage extending capacity beyond GPU memory.
- Separate serving stages: Move cached context between prefill and decode workers on different devices or machines so the stages can scale independently.
- Multimodal reuse: Share encoder embeddings through SGLang and vLLM integrations to avoid repeated computation.
- Distributed transfers: Move cache, embeddings and model weights with topology-aware routing, combined network interfaces and transfer failover.
The Apache 2.0 components integrate with vLLM, SGLang, TensorRT-LLM, LMDeploy and LMCache. They run on your infrastructure and require a model-serving stack. Accelerator support includes NVIDIA CUDA, AMD HIP and Ascend, with TCP and RDMA transports; availability depends on the deployment and hardware.
Logotyper
