Nemotron and Pipecat: voice agent fine-tuning explained

This session explains how speech recognition and language model customization work together in a domain-specific voice agent. The demonstration pairs an ASR model adapted to noisy air traffic control audio with Nemotron 3 Nano trained to explain aviation phrases. Quinn says the agent runs locally on his 5090, with a browser interface connected over WebRTC.

The demo compares base and adapted transcriptions as background noise increases. Quinn reports final ASR transcripts in under 60 milliseconds and LLM time to first token below 100 milliseconds. These are reported demo results, rather than general performance guarantees for a local LLM deployment.

The discussion covers Pipecat's streaming components and how GPU load affects response time. For retrieval and other slower work, Quinn describes asynchronous workers that feed information into the conversation without blocking the voice loop.

Training guidance starts with checking whether word boosting or a dictionary can solve the recognition problem before acoustic fine-tuning. The speakers recommend retaining general-language data and testing for regressions. They discuss synthetic data, iterative evaluation, and full-weight SFT versus LoRA. Speech adapter support was still unreleased at the time of the session.