LiveKit Agents

LiveKit Agents is a framework for developers building voice assistants, phone agents, and apps that combine speech with video or text. Agents join LiveKit rooms as participants, so they can interact with people through web and mobile apps or telephone calls. The Apache 2.0 project lets you run the entire stack on your own servers, including the LiveKit media server.

Its main focus is the conversation itself. It handles streaming speech recognition, language model responses, and speech synthesis, with turn detection and interruption handling built into the framework. A transformer model helps detect when someone has finished speaking. You can combine different speech and language model providers or use realtime model APIs.

Python and Node.js SDKs let developers define agent behavior in code. Agents can call tools, use tools from MCP servers, and hand a conversation to another agent. They can also exchange data with the frontend, which supports interactions beyond spoken replies. LiveKit's SIP integration covers incoming and outgoing phone calls.

Self-hosting controls where the agent and media infrastructure run; model processing depends on the providers you connect. LiveKit Cloud provides hosted agent deployment, transcripts and traces, and model access through LiveKit Inference. Its browser-based Agent Builder offers a separate way to prototype and deploy agents without writing code.

For larger deployments, the framework includes job scheduling, automatic load balancing, and Kubernetes compatibility. Built-in testing supports checks of agent behavior with automated judges.