LLMKube on AWS EKS: Kubernetes model deployment tutorial

This tutorial builds a self-hosted model endpoint on AWS EKS using LLMKube. Terraform provisions the network and a node group with two t3.xlarge instances. The walkthrough requires AWS permissions for VPC creation, EKS management, EC2 provisioning and IAM role creation, with the configuration pointed at us-west-2.

The presenter installs LLMKube through Helm, selecting a version spoken as 0.75, then explains the Model and InferenceService manifests. The CPU deployment uses llama.cpp and downloads GGUF weights from Hugging Face. Persistent storage on EBS caches the weights. An internet-facing network load balancer exposes the OpenAI-compatible chat completions endpoint on port 8080.

The model name needs care: the description says Gemma 2B, while the speaker repeatedly calls it Gemma 4 E2B. A curl request fails with a bad-hostname error; the presenter switches to a Python notebook and obtains a response. He explains that reasoning tokens consume the completion budget and can leave too little room for the final answer.

The demonstrated setup uses CPU resources and a fixed node count. The presenter discusses GPU performance, but cannot demonstrate GPU provisioning because of AWS restrictions on his account. Cleanup removes Kubernetes-created resources before Terraform destroys the infrastructure. Custom metrics for autoscaling are reserved for a later tutorial. AWS infrastructure incurs ongoing charges; the cost estimates and untested GPU speeds are the presenter’s estimates. The demonstrated endpoint is public HTTP with no API authentication shown, so this is a deployment experiment rather than a secured production service.