vLLM vs llama.cpp and LM Studio: DGX Spark benchmarks

This comparison examines local LLM inference on NVIDIA DGX Spark. The speaker reports up to 2.8x aggregate throughput with vLLM at 32 concurrent requests compared with llama.cpp and LM Studio. With one request and default settings, vLLM was slower in his tests. These results describe his benchmark setup rather than a guaranteed speed increase.

The explanation covers continuous batching, compute-bound prefill and memory-bound decode. Parallel requests from a coding assistant's sub-agents provide a practical reason to test concurrency even with one user. The speaker also compares speculative decoding, time to first token and joules per token. His benchmark client used 1,000 input tokens and requested 256 output tokens, with warm-up runs and two to three repeats. He used Claude Code with frontier cloud models to configure the tests.

For NVIDIA Nemotron 3.5 Lightning, with 30B total and 3B active parameters, he reports 440 aggregate tokens per second on vLLM versus 95 on LM Studio at 32 requests. Qwen 3.8 Flash, a 177B model, ran on his 128GB machine by offloading embedding n-gram tables to SSD. That llama.cpp setup reached 70 tokens per second at eight requests; he could not make the same offload arrangement work in vLLM at recording time. He considers it experimental and prefers a stable model that fits the hardware for everyday use, stressing answer quality alongside throughput.