vLLM on Crusoe Cloud: Docker setup and chat API tutorial

This tutorial covers self-hosted model inference on a rented Crusoe Cloud GPU instance. The speaker uses a 70 billion parameter model requiring 140 GB of VRAM at BF16 precision to explain why larger models can exceed home hardware capacity. The video was sponsored by and produced on behalf of Crusoe.

The guide explains why vLLM suits concurrent requests, with continuous batching and Paged Attention for KV cache management. Its OpenAI-compatible API gives applications a familiar interface. Other inference engines receive brief mentions rather than a performance comparison.

The setup uses a pre-built vLLM Docker image that fetches model weights from Hugging Face at runtime. The Crusoe walkthrough covers selecting an instance image with GPU drivers and container tooling, connecting over SSH, checking available VRAM, and launching the model. The speaker then opens port 8000 for requests from a home computer. The recommended access controls restrict the firewall to the user's IP address and require an API key.

A curl request tests token generation before the demonstration adds a simple chat frontend. The speaker explains how applications using the OpenAI API can point to this inference endpoint. Automatic scaling and load balancing are topics for a later video, not part of this setup.