LiveKit Agents 1.3: voice pipeline and Python setup

This introductory lesson explains a voice AI agent as a pipeline: voice activity detection identifies speech, speech recognition transcribes it, an LLM generates a reply, and text to speech produces the audio response. The speaker also introduces noise cancellation and turn detection, while leaving direct speech-to-speech models outside the workshop's scope.

Latency shapes the discussion. The speaker gives a target below 500 milliseconds and estimates that the combined stages can take roughly 400 milliseconds to two seconds. These are teaching estimates, not measured results for the starter agent. The networking section compares HTTP and WebSockets over TCP with WebRTC over UDP, explaining packet loss, Opus compression and jitter buffering.

The setup uses uv, Python and LiveKit Agents 1.3, with Silero VAD, turn detection and a noise cancellation plugin. Viewers obtain a URL, API key and secret from LiveKit Cloud, store the credentials in an environment file, and configure agent.py. The lesson then covers downloading model files, launching console mode and selecting audio devices when input or output fails.

The supplied description names AssemblyAI, OpenAI GPT-4 and Cartesia as the starter's transcription, conversation and speech providers. Although the agent runs in a local terminal, this walkthrough uses cloud credentials and does not establish an offline setup. Testing advice covers accents, background noise and interruptions; semantic turn detection follows in the next lesson.