llm-d: Kubernetes inference routing explained

This overview explains llm-d, an open source project for distributed LLM inference on Kubernetes. The speaker focuses on self-hosted model serving across GPU nodes and why ordinary HTTP load balancing does not account for the work each inference request requires.

The explanation starts with prefill, which processes input tokens, and decode, which generates output using the KV cache. The speaker describes how routing follow-up requests to a server with cached context can avoid repeated computation. Cache matches, available GPU memory and queue depth contribute to a weighted score for each model server.

The request-flow walkthrough follows an OpenAI API request through Envoy. Its external processing filter contacts the llm-d endpoint picker over gRPC. The picker returns the chosen destination in a header, and Envoy forwards the request. The speaker also explains how inference pools identify model-server pods and how Prometheus metrics inform routing decisions.

Later sections cover separate prefill and decode pods. In the architecture described, KV cache moves between their GPUs through RDMA, using InfiniBand or RDMA over converged Ethernet. The video closes with plugin interfaces for scoring and request data, plus priority bands and flow-control policies. KV cache offloading and autoscaling receive only brief mentions; this is an architecture explanation rather than an installation walkthrough.