NVIDIA Jetson: llama.cpp setup and Gemma 4 demos

This session explains how to run models locally on NVIDIA Jetson, with a practical llama.cpp walkthrough and a comparison of inference engines. The presenters recommend vLLM for concurrent workloads on Thor and describe llama.cpp as a lightweight option for devices with limited memory. They also discuss TensorRT Edge LLM for C++ deployment.

The setup covers compiling llama.cpp with CUDA support after installing the required toolkit and dependencies, downloading a quantized model through Hugging Face, and running an interactive session or HTTP server. Raymond recommends starting with four-bit quantization on memory-constrained hardware, then adjusting model size and precision for the task.

In a recorded 8 GB Jetson demonstration, Raymond reports about 27 generated tokens per second with Gemma 4 E2B. He uses it to generate an HTML game and describe an image. These are results from his setup, rather than performance guarantees. The chatbot interface requires manual copying of generated code and does not demonstrate file access or tool calling.

Joyce presents a bird-monitoring voice assistant on an Orin Nano, using the Q4 model at 15 watts. BirdNET detects bird sounds while a separate wake-word pipeline queries Gemma using recent and historical detections stored in SQLite. She notes a delay between the visible response and spoken output. Agent setup and filesystem access are topics for a later session.