llm-d: KV cache routing vs prefill/decode disaggregation

Yuchen Fama and Ashish Kamra explain two ways to tune self-hosted inference for agentic workloads: KV cache-aware routing and prefill/decode disaggregation. Their workload observations include sessions reaching 3,000 turns and cache hit rates exceeding 90%. They argue that steady-state benchmarks miss the context changes and cache churn these sessions create.

In llm-d, an endpoint picker weighs pod load and cache locality when routing requests. A demonstration reports roughly three seconds for an initial request and one second for a later turn that reuses its cached system prompt. A different system prompt incurs the longer wait again. These are demonstration results, not general latency guarantees.

Disaggregation assigns prompt processing and token generation to separate workers so compute-heavy prefill interferes less with memory-bound decode. In a 16-H100 multi-turn test with a 10,000-token prefix and 128 tokens per turn, the speakers report P99 inter-token latency near 100 milliseconds versus 900 milliseconds in an aggregated deployment. They also show that benefits vary with workload and concurrency. Cache transfers require a suitable high-speed network fabric, such as RDMA or RoCE; they recommend aggregated serving when that infrastructure is absent.

The closing GLM 5.2 case study uses H200 hardware. The speakers report four times faster time to first token and 60% more requests, while stressing that the work remains ongoing.