llama.cpp, vLLM and ODS: inference stack comparison

This local AI session explains how to choose an inference stack around a workload. Ahmed Osman and Michael from Osmantic distinguish single-user chat from serving concurrent requests. They explain prefill and decode, then discuss memory capacity, bandwidth and hardware-specific kernels.

Their hardware arena compares an RTX Pro 6000, DGX Spark, Strix Halo and M5 MacBook Pro with the same Qwen 3.6 27B model at Q4. A single-prompt race leads into a burst of 50 requests. The speakers attribute the Blackwell tower's advantage to bandwidth and vLLM's continuous batching. They also attribute the Mac's weaker burst result to the selected llama.cpp version and kernel, so these results describe the demonstrated configurations rather than every possible setup.

The ODS demonstration introduces an open source deployment system for a self-hosted stack. Its developers describe model and engine selection, application connections and GPU monitoring through one installation command.

After technical delays cancel the planned model-compression segment, Exo founder Alex recounts earlier clustering work and previews Local.ai, a benchmarking website in early access during the session. Its charts compare task success with completion time and provide configurations users can reproduce. His examples show why tokens per second alone cannot describe an AI agent workload with repeated prefill and tool calls.