Dell GB10 setup with llama.cpp, llama-swap and vLLM

The speaker explains a local AI setup built around a Dell Pro Max with NVIDIA GB10, 128 GB of unified memory and a 20-core ARM CPU. He describes the machine as costing roughly $7,000 and discloses that Dell and NVIDIA supplied it free. His comparison with an RTX 4090 focuses on memory capacity: larger models fit on the dedicated box, while inference leaves his desktop GPU available for other work.

The serving stack combines llama.cpp with llama-swap for smaller models, Ollama for convenient client connections, and vLLM for larger models and concurrent requests. His llama-swap configuration lets smaller models share memory and makes large models exclusive, unloading others when necessary. Cold model loads can take 10 to 60 seconds in his setup. Tailscale connects remote clients to the server, and demonstrations cover a coding assistant in VS Code and a Hermes agent connected to Discord.

Optimization advice covers Q6 quantization, speculative decoding, quantized KV cache and Flash Attention. The speaker reports that Nemotron Super 120B rose from 16.6 to 27.2 tokens per second at a 17,000-token context after enabling several optimizations. These are his measurements, not guaranteed results on other hardware. The video explains the configuration choices rather than providing a full installation walkthrough. He still uses cloud models for difficult coding and long-context tasks.