vLLM, Ray and KubeRay: Kubernetes setup tutorial

This class walks through two separate labs: serving a model with vLLM on Amazon EKS, then deploying a Ray cluster with KubeRay. The instructor explains a self-hosted inference setup with OpenAI-compatible endpoints and describes how paged attention and KV caching can reduce memory waste and repeated computation. These explanations are not accompanied by comparative benchmarks.

The vLLM lab uses a CPU image because the instructor's AWS GPU quota has not been approved. The EKS setup specifies two t3.medium servers, each with two vCPUs and 4 GB RAM. A Kubernetes manifest loads a public Hugging Face model and exposes port 8000. The instructor uses port forwarding to access the service from localhost; inference still runs on EKS. API checks cover model discovery, completions and chat completions. The demonstration includes a null completion result and curl errors before a chat request returns an answer.

The second lab explains Ray's distribution of Python tasks and KubeRay's management of Ray resources on Kubernetes. The instructor installs the operator with Helm, deploys a cluster, runs a remote Python task and inspects the Ray dashboard. This is a sandbox demonstration: authentication and other production additions remain separate work. Although the metadata advertises Ray Serve and KEDA autoscaling, the transcript mainly demonstrates standalone vLLM and a basic Ray task.