LLM Compressor updates and llm-d distributed serving

This office hours session covers vLLM v0.24, LLM Compressor updates and llm-d for distributed model serving. The maintainers report DSpark support in the vLLM nightly ahead of v0.25. They also discuss sparse MLA optimizations, broader Model Runner V2 support and dynamic speculative decoding that reduces draft length as batch sizes grow.

The LLM Compressor update covers DDP multi-GPU quantization and a converter abstraction for checkpoint conversion and dequantization. Its maintainers describe v0.12 support for Transformers v5 and model-free PTQ across multiple GPUs.

Robert Shaw explains how llm-d coordinates vLLM instances on Kubernetes for self-hosted inference. Prefix cache-aware routing sends repeated context to workers that already hold it, while load-aware scoring accounts for overloaded workers. KV cache tiering extends storage beyond GPU memory, and prefill/decode disaggregation separates prompt processing from token generation.

The GLM-5.2 example uses H200 GPUs with Wide EP across nodes. Shaw reports more than 90% prefix cache reuse and median time to first token below three seconds for the demonstrated agentic workload. These results depend on that setup and workload. His cost comparison assumes reserved GPU pricing and full utilization. The closing discussion clarifies that llm-d does not include deployment APIs and can work with different schedulers.