vLLM setup: Python inference and a local API server

Cedric Clyburn demonstrates self-hosted inference with vLLM through a Python program and an OpenAI-compatible API on localhost. The demo uses Linux with a supported Nvidia or AMD GPU. He explains how continuous batching schedules incoming requests and how paged attention manages KV cache memory for concurrent workloads.

The Python example loads a Qwen model and processes several conversations with sampling settings such as temperature and response token limits. Here, offline means calling the inference library directly rather than using a separate API server; it does not mean the computer lacks a network connection. The server example sends a JSON request with curl, then uses the OpenAI Python client with a local base URL to stream a response. Tool calling, structured output and multimodal support depend on the model and serving configuration.

The memory discussion separates model weights from other GPU requirements. Clyburn estimates that an 8-billion-parameter model at 16 bits needs about 16 GB for raw weights alone. He compares model files of about 55.6 GB with a Red Hat AI int4 version of about 20 GB. Quantization can reduce memory needs, but he advises testing quality on the intended workload rather than treating published benchmark recovery as a guarantee.