llm-d

llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.

• Request routing: Direct requests using server load and cached prompt context, so repeated context can reuse existing work.
• Tiered context cache: Extend model context caches into CPU memory or disk to support repeated and multi-turn requests.
• Large model serving: Separate prompt processing from token generation and spread mixture-of-experts computation across accelerators for models such as DeepSeek-R1 and GPT-OSS.
• Shared cluster controls: Apply traffic flow control and fairness, and autoscale serving capacity using inference signals.
• Batch inference: Process asynchronous workloads through OpenAI-compatible Batch APIs when responses don't need to be immediate.

llm-d is open source under Apache 2.0 and runs on self-hosted Kubernetes infrastructure. Supported hardware includes GPUs, Google TPUs, Intel XPUs and CPUs. Its deployment recipes cover common serving patterns, and reproducible benchmarks help teams compare performance for particular models and hardware.