Ollama and LiteLLM: build a local shop assistant

The presenter builds Sam, a local AI shop assistant, on a ZimaCube 2 NAS with a 16 GB Nvidia RTX 2000, 32 GB of system RAM and a separate 1 TB NVMe drive for applications and models. TrueNAS hosts the stack, while Ollama runs a nine-billion-parameter Qwen model that the speaker says uses about 7 to 8 GB of memory.

Open WebUI provides a chat interface for testing a system prompt. The presenter adjusts identity, response length and spoken formatting, then adds example exchanges. LiteLLM acts as a shared gateway so the chat interface and robot client use the same prompt, routing and tools.

A Reachy Mini supplies the voice interface. The speech pipeline uses Whisper for transcription and Kokoro for speech output. The build also needs microphone controls to prevent the robot from answering itself, plus filters for transcription errors caused by loud noises.

Document retrieval supplies equipment references that the base local LLM lacks. The presenter demonstrates an inventory lookup followed by an email check and a confirmed quantity update. The opening Gemini comparison is a single equipment-error example, not a general benchmark. Although the introduction says exchanges stay in the building, the later setup includes Tavily web search and a Dropbox ingestion folder, so the demonstrated workflow includes external services.

The video includes an Anker power-station sponsorship; the NAS was supplied to the presenter. The listed hardware describes his build.