vLLM speculative decoding: Blackwell workload comparison

Sheilah Kirui explains how speculative decoding works and compares it with baseline inference using vLLM on one NVIDIA Blackwell GPU. A small draft model proposes several tokens, then the target model checks them in one forward pass. The talk separates prefill, when the model processes the prompt and builds its KV cache, from decode, when it generates the answer. Speculation targets the decode phase.

The setup needs memory for a second model and both models' KV caches. Kirui recommends a draft model roughly 10 to 50 times smaller than the target, with the same tokenizer and ideally from the same model family. The choice balances prediction accuracy against the cost of drafting tokens. Her demo runs baseline and speculative servers on the same GPU, with about 16 GB of target weights and 2.5 GB for the draft.

After a server restart, Kirui reports 1.6x faster generation for structured output, alongside a high token acceptance rate. The creative writing example has lower acceptance, which she attributes to greater variation in possible next tokens and a high temperature setting. These are results from her demonstration, rather than a general speed guarantee for a self-hosted deployment.

She recommends evaluating spare VRAM, batch size and workload structure before enabling speculation. Long prompts with short answers can gain less because prefill takes more of the request time. Heavy concurrency can also leave little GPU capacity for drafting.