Pipecat PhoneLLM Alpha 1: local setup with vLLM

This tutorial installs and tests Pipecat's PhoneLLM Alpha 1, an open-weights model intended for voice agents. The speaker describes it as a fine-tune of NVIDIA's Nemotron 3 Nano with 30 billion parameters and 3.5 billion active. Its role in a voice pipeline is to choose a reply or tool call between speech recognition and speech synthesis.

The setup uses Ubuntu and vLLM with mostly default serving parameters. The first run downloads the model. With a full KV cache, the speaker reports over 74 GB of VRAM use and suggests reducing the cache or using a quantized version for smaller GPUs. A Python client sends requests through an OpenAI-style chat API to the local LLM server.

Two reservation examples test tool selection. The first produces a reservation function call with the caller's name, party size and time. The second includes a rejected time slot and a changed party size; the speaker reports that PhoneLLM checks availability for four people before booking, taking about a second with 640 tokens of context on one GPU.

The video also discusses benchmark comparisons and a voice latency breakdown of about 1.3 seconds overall, including roughly 650 milliseconds for the LLM. These are presented results, not guarantees for other hardware. The hands-on portion tests text requests and tool calls; the speaker refers viewers to a separate Pipecat, Ollama and Kokoro tutorial for the complete voice pipeline.