GuideLLM
An open-source LLM benchmarking tool that measures latency and throughput on OpenAI-compatible and vLLM-native servers using realistic workloads.
GuideLLM benchmarks LLM deployments so engineering and ML teams can assess capacity and response times under expected traffic. It can test self-hosted servers through OpenAI-compatible or vLLM-native interfaces.
- Response timing: Measure time to the first token, gaps between tokens and total response time, with distributions that show variation beyond averages.
- Traffic simulation: Test sequential requests, concurrent users and rate-based traffic, or use repeatable load sweeps to find where a deployment meets its response targets.
- Workload data: Use HuggingFace datasets, local files or synthetic data, with support for text, image, audio and video inputs.
- Comparable reports: Export results for deployment comparisons and regression tracking as JSON, YAML or CSV, alongside self-contained HTML reports and static performance graphs.
A command-line interface and Python API let teams incorporate benchmarks into their engineering workflows. Token-level statistics and output distributions help compare how deployments behave across different workloads.
GuideLLM is open source under Apache 2.0. It requires a running model server to benchmark; it doesn't host models itself.
