GuideLLM v0.7: benchmark vLLM and agentic workloads

This vLLM office hours session explains how GuideLLM benchmarks inference from the user's perspective. Samuel Monson and Jared O'Connell cover time to first token, inter-token latency and throughput, then explain how request patterns affect the results. Concurrent requests help find a server's throughput ceiling; independently arriving requests test how it handles incoming traffic.

The local LLM demonstration uses Qwen3-0.6B through an OpenAI-compatible HTTP endpoint on port 8000. It sends 15 requests sequentially, with synthetic inputs of 256 tokens and requested outputs of 256 tokens. A separate configuration example increases concurrency from one to 300 streams for capacity testing on self-hosted infrastructure.

The presenters describe GuideLLM v0.7's CLI rework and initial replay and tool-calling support. They explain why multi-turn histories, sub-agent dependencies and recorded request timings matter for realistic benchmarks. Some trace support remains partial, with further work planned for v0.8. Multimodal output and direct speculative decoding metrics are also described as unfinished.

Other talks cover vLLM v0.27 and Speculators v0.6 and v0.7. Agentic API's contributors explain how their Rust server connects Codex and Claude Code to vLLM with open-weight models, keeps stateful logic outside the inference engine and executes web search tools. They describe the integration as an early MVP.