vLLM and Qwen3: GPU inference setup and tuning

Du'an Lightfoot and Khaja Omer explain how to operate self-hosted inference for an AI agent on dedicated GPUs. They weigh hosted APIs against owning or renting GPU capacity, with attention to steady usage, data control and the expertise needed to maintain the service.

The workshop gives participants a Kubernetes namespace with a Jupyter notebook, a vLLM endpoint and a single RTX 4000 Ada GPU. Qwen 3 4B provides the starting model in BF16. The exercises distinguish generation speed for one request from aggregate throughput, then examine model weights, KV cache capacity, prefill, decode and warm prefix caching.

Omer switches to a pre-quantized FP8 model and reports a single-request increase from 37.5 to 53.4 tokens per second. He also stresses workload-specific quality checks, including valid JSON and function calls. A speculative decoding configuration then drops generation from roughly 58 to 16 tokens per second. He attributes the slowdown to poor drafter compatibility and low acceptance, while discussing GPU contention and draft length as other factors to examine.

The final exercise raises the sequence limit from 32 to 128 and repeats the load test. A notebook failure forces a return to the full precision model, which limits comparisons with the earlier FP8 results. The practical method is to establish a baseline, change one setting, and measure latency, throughput and output quality again.