Hugging Face speech-to-speech: local voice AI setup

Hugging Face's speech-to-speech repository is the subject of this overview of a modular voice assistant. The speaker describes an Apache 2.0 pipeline with four stages: voice detection, transcription, language model inference and speech synthesis. It supports the OpenAI realtime protocol, with the presenter describing a client connection redirected to 127.0.0.1 by changing the base URL.

The walkthrough explains how silence detection affects turn taking and how interruption support stops a spoken response when the user starts talking. It presents NVIDIA Parakeet TDT version 3 as the default transcription model, with 600 million parameters and support for 25 European languages. Whisper remains an option.

For a local LLM, the speaker discusses a llama.cpp server and in-process inference through Transformers or MLX on Apple Silicon. A Mac configuration combines Parakeet, MLXLM and Qwen3-TTS through MLX Audio; the presenter says this setup works offline. CPU speech synthesis options include Kokoro and Pocket TTS.

The Cerebras example uses a remote Hugging Face router endpoint. The speaker cites 1,851 output tokens per second from Artificial Analysis, so that figure should not be read as laptop performance. The source description and captions disagree about the Gemma version used. The closing advice is to pin versions and measure each stage on your own hardware: a cascade has multiple failure points, and its response delay depends on the chosen components.