llama.cpp throughput tuning with Llama Throughput Lab

Alex Ziskind demonstrates how Llama Throughput Lab tunes a local LLM server for concurrent requests. His initial tests report roughly 100 tokens per second through Ollama and 124 through llama.cpp directly. These single-request results differ from aggregate throughput when several requests run together.

The Python launcher tests combinations of server instance count, llama-server's parallel setting and client concurrency. Its full sweep covers 308 combinations and can take a long time, especially on slower hardware. Ziskind says the launcher also runs on Windows and Linux systems supported by llama.cpp. The project is open source, with code linked in the description.

On his Mac Studio, Ziskind reports 1,226 tokens per second with 16 server instances, parallel set to 64 and concurrency at 1,024. This is aggregate benchmark throughput, not the speed of one chat response. He identifies GPU compute as the constraint in this test and cautions that real workloads need separate testing.

The setup walkthrough uses NGINX to distribute requests among the instances through round-robin routing. Ziskind then connects Open WebUI to that endpoint on port 8000. He also notes that the demonstration limits generated tokens, including tokens used for thinking, so longer responses require adjusting that setting. The approach targets concurrent workloads such as a coding assistant or coordinated agents.