agents Documentation

Technical Documentation: livekit/agents

ℹ️ Provenance: Hybrid Fusion: livekit/agents (README + 4 In-Tree Chapters) · CodeWiki Reference · Recency: Active (< 180 days)

1. Project Overview & Quickstart (livekit/agents) Copy Chapter

Looking for the JS/TS library? Check out AgentsJS

What is Agents? Copy Chapter

The Agent Framework is designed for building realtime, programmable participants
that run on servers. Use it to create conversational, multi-modal voice
agents that can see, hear, and understand.

Features Copy Chapter

• Flexible integrations: A comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API to suit your use case.
• Integrated job scheduling: Built-in task scheduling and distribution with dispatch APIs to connect end users to agents.
• Extensive WebRTC clients: Build client applications using LiveKit's open-source SDK ecosystem, supporting all major platforms.
• Telephony integration: Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.
• Exchange data with clients: Use RPCs and other Data APIs to seamlessly exchange data with clients.
• Semantic turn detection: Uses a transformer model to detect when a user is done with their turn, helps to reduce interruptions.
• MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.
• Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.
• Open-source: Fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server, one of the most widely used WebRTC media servers.

Installation Copy Chapter

To install the core Agents library, along with plugins for popular model providers:

pip install "livekit-agents[openai,deepgram,cartesia]"

Docs and guides Copy Chapter

Documentation on the framework and how to use it can be found here

Building with AI coding agents Copy Section

If you're using an AI coding assistant to build with LiveKit Agents, we recommend the following setup for the best results:

1. Install the LiveKit Docs MCP server — Gives your coding agent access to up-to-date LiveKit documentation, code search across LiveKit repositories, and working examples.

2. Install the LiveKit Agent Skill — Provides your coding agent with architectural guidance and best practices for building voice AI applications, including workflow design, handoffs, tasks, and testing patterns.

   shell Copy

   npx skills add livekit/agent-skills --skill livekit-agents

The Agent Skill works best alongside the MCP server: the skill teaches your agent how to approach building with LiveKit, while the MCP server provides the current API details to implement it correctly.

Core concepts Copy Chapter

• Agent: An LLM-based application with defined instructions.
• AgentSession: A container for agents that manages interactions with end users.
• entrypoint: The starting point for an interactive session, similar to a request handler in a web server.
• AgentServer: The main process that coordinates job scheduling and launches agents for user sessions.

Usage Copy Chapter

Simple voice agent Copy Section

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    RunContext,
    cli,
    function_tool,
    inference,
)


@function_tool
async def lookup_weather(
    context: RunContext,
    location: str,
):
    """Used to look up weather information."""

    return {"weather": "sunny", "temperature": 70}


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        # any combination of STT, LLM, TTS, or realtime API can be used
        # this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud
        # to use model provider keys directly, replace with the following:
        # from livekit.plugins import deepgram, openai, cartesia
        # stt=deepgram.STT(model="nova-3"),
        # llm=openai.LLM(model="gpt-4.1-mini"),
        # tts=cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),  # low-latency gemma, hosted on LiveKit
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )

    agent = Agent(
        instructions="You are a friendly voice assistant built by LiveKit.",
        tools=[lookup_weather],
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")


if __name__ == "__main__":
    cli.run_app(server)

You'll need the following environment variables for this example:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

Multi-agent handoff Copy Section

This code snippet is abbreviated. For the full example, see multi_agent.py

/* Detailed source-code truncated for AI context efficiency. */

Testing Copy Section

Automated tests are essential for building reliable agents, especially with the non-deterministic behavior of LLMs. LiveKit Agents include native test integration to help you create dependable agents.

@pytest.mark.asyncio
async def test_no_availability() -> None:
    llm = google.LLM()
    async with AgentSession(llm=llm) as sess:
        await sess.start(MyAgent())
        result = await sess.run(
            user_input="Hello, I need to place an order."
        )
        result.expect.skip_next_event_if(type="message", role="assistant")
        result.expect.next_event().is_function_call(name="start_order")
        result.expect.next_event().is_function_call_output()
        await (
            result.expect.next_event()
            .is_message(role="assistant")
            .judge(llm, intent="assistant should be asking the user what they would like")
        )

Examples Copy Chapter

For more examples and detailed setup instructions, see the examples directory. For even more examples, see the python-agents-examples repository.

| 🎙️ Starter Agent
A starter agent optimized for voice conversations.

Code | 🔄 Multi-user push to talk
Responds to multiple users in the room via push-to-talk.

Code |
| --- | --- |
| 🎵 Background audio
Background ambient and thinking audio to improve realism.

Code | 🛠️ Dynamic tool creation
Creating function tools dynamically.

Code |
| ☎️ Outbound caller
Agent that makes outbound phone calls

Code | 📋 Structured output
Using structured output from LLM to guide TTS tone.

Code |
| 🔌 MCP support
Use tools from MCP servers

Code | 💬 Text-only agent
Skip voice altogether and use the same code for text-only integrations

Code |
| 📝 Multi-user transcriber
Produce transcriptions from all users in the room

Code | 🎥 Video avatars
Add an AI avatar with Tavus, Bithuman, LemonSlice, and more

Code |
| 🍽️ Restaurant ordering and reservations
Full example of an agent that handles calls for a restaurant.

Code | 👁️ Gemini Live vision
Full example (including iOS app) of Gemini Live agent that can see.

Code |

Running your agent Copy Chapter

Testing in terminal Copy Section

python myagent.py console

Runs your agent in terminal mode, enabling local audio input and output for testing.
This mode doesn't require external servers or dependencies and is useful for quickly validating behavior.

Developing with LiveKit clients Copy Section

python myagent.py dev

Starts the agent server and enables hot reloading when files change. This mode allows each process to host multiple concurrent agents efficiently.

The agent connects to LiveKit Cloud or your self-hosted server. Set the following environment variables:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

You can connect using any LiveKit client SDK or telephony integration.
To get started quickly, try the Agents Playground.

Running for production Copy Section

python myagent.py start

Runs the agent with production-ready optimizations.

License Copy Chapter

The Agents framework is licensed under Apache-2.0. The LiveKit turn detection models are licensed under the LiveKit Model License.

Contributing Copy Chapter

The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in the LiveKit community.

Development setup Copy Section

This project uses uv for package management. To install dependencies for development:

uv sync --all-extras --dev

Examples Copy Section

This project includes many examples in the examples directory. To run them, create the file examples/.env with credentials for LiveKit Server and any necessary model providers (see examples/.env.example), then run:

uv run examples/voice_agents/basic_agent.py dev

For more information, see the examples README.

Tests Copy Section

Unit tests are in the tests directory and can be run with:

uv run pytest --unit

Integration tests for each plugin require various API credentials and run automatically in GitHub CI for PRs submitted by project maintainers. See the tests workflow for details.

Formatting Copy Section

This project uses ruff for formatting and linting:

uv run ruff format
uv run ruff check --fix

Documentation Copy Section

To generate docs locally with pdoc:

uv sync --all-extras --group docs
uv run --active pdoc --skip-errors --html --output-dir=docs livekit

2. In-Tree Documentation Chapters (livekit/agents) Copy Chapter

File: README.md Copy Chapter

Looking for the JS/TS library? Check out AgentsJS

What is Agents? Copy Chapter

The Agent Framework is designed for building realtime, programmable participants
that run on servers. Use it to create conversational, multi-modal voice
agents that can see, hear, and understand.

Features Copy Chapter

• Flexible integrations: A comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API to suit your use case.
• Integrated job scheduling: Built-in task scheduling and distribution with dispatch APIs to connect end users to agents.
• Extensive WebRTC clients: Build client applications using LiveKit's open-source SDK ecosystem, supporting all major platforms.
• Telephony integration: Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.
• Exchange data with clients: Use RPCs and other Data APIs to seamlessly exchange data with clients.
• Semantic turn detection: Uses a transformer model to detect when a user is done with their turn, helps to reduce interruptions.
• MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.
• Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.
• Open-source: Fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server, one of the most widely used WebRTC media servers.

Installation Copy Chapter

To install the core Agents library, along with plugins for popular model providers:

pip install "livekit-agents[openai,deepgram,cartesia]"

Docs and guides Copy Chapter

Documentation on the framework and how to use it can be found here

Building with AI coding agents Copy Section

If you're using an AI coding assistant to build with LiveKit Agents, we recommend the following setup for the best results:

1. Install the LiveKit Docs MCP server — Gives your coding agent access to up-to-date LiveKit documentation, code search across LiveKit repositories, and working examples.

2. Install the LiveKit Agent Skill — Provides your coding agent with architectural guidance and best practices for building voice AI applications, including workflow design, handoffs, tasks, and testing patterns.

   shell Copy

   npx skills add livekit/agent-skills --skill livekit-agents

The Agent Skill works best alongside the MCP server: the skill teaches your agent how to approach building with LiveKit, while the MCP server provides the current API details to implement it correctly.

Core concepts Copy Chapter

• Agent: An LLM-based application with defined instructions.
• AgentSession: A container for agents that manages interactions with end users.
• entrypoint: The starting point for an interactive session, similar to a request handler in a web server.
• AgentServer: The main process that coordinates job scheduling and launches agents for user sessions.

Usage Copy Chapter

Simple voice agent Copy Section

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    RunContext,
    cli,
    function_tool,
    inference,
)


@function_tool
async def lookup_weather(
    context: RunContext,
    location: str,
):
    """Used to look up weather information."""

    return {"weather": "sunny", "temperature": 70}


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        # any combination of STT, LLM, TTS, or realtime API can be used
        # this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud
        # to use model provider keys directly, replace with the following:
        # from livekit.plugins import deepgram, openai, cartesia
        # stt=deepgram.STT(model="nova-3"),
        # llm=openai.LLM(model="gpt-4.1-mini"),
        # tts=cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),  # low-latency gemma, hosted on LiveKit
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )

    agent = Agent(
        instructions="You are a friendly voice assistant built by LiveKit.",
        tools=[lookup_weather],
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")


if __name__ == "__main__":
    cli.run_app(server)

You'll need the following environment variables for this example:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

Multi-agent handoff Copy Section

This code snippet is abbreviated. For the full example, see multi_agent.py

/* Detailed source-code truncated for AI context efficiency. */

Testing Copy Section

Automated tests are essential for building reliable agents, especially with the non-deterministic behavior of LLMs. LiveKit Agents include native test integration to help you create dependable agents.

@pytest.mark.asyncio
async def test_no_availability() -> None:
    llm = google.LLM()
    async with AgentSession(llm=llm) as sess:
        await sess.start(MyAgent())
        result = await sess.run(
            user_input="Hello, I need to place an order."
        )
        result.expect.skip_next_event_if(type="message", role="assistant")
        result.expect.next_event().is_function_call(name="start_order")
        result.expect.next_event().is_function_call_output()
        await (
            result.expect.next_event()
            .is_message(role="assistant")
            .judge(llm, intent="assistant should be asking the user what they would like")
        )

Examples Copy Chapter

For more examples and detailed setup instructions, see the examples directory. For even more examples, see the python-agents-examples repository.

| 🎙️ Starter Agent
A starter agent optimized for voice conversations.

Code | 🔄 Multi-user push to talk
Responds to multiple users in the room via push-to-talk.

Code |
| --- | --- |
| 🎵 Background audio
Background ambient and thinking audio to improve realism.

Code | 🛠️ Dynamic tool creation
Creating function tools dynamically.

Code |
| ☎️ Outbound caller
Agent that makes outbound phone calls

Code | 📋 Structured output
Using structured output from LLM to guide TTS tone.

Code |
| 🔌 MCP support
Use tools from MCP servers

Code | 💬 Text-only agent
Skip voice altogether and use the same code for text-only integrations

Code |
| 📝 Multi-user transcriber
Produce transcriptions from all users in the room

Code | 🎥 Video avatars
Add an AI avatar with Tavus, Bithuman, LemonSlice, and more

Code |
| 🍽️ Restaurant ordering and reservations
Full example of an agent that handles calls for a restaurant.

Code | 👁️ Gemini Live vision
Full example (including iOS app) of Gemini Live agent that can see.

Code |

Running your agent Copy Chapter

Testing in terminal Copy Section

python myagent.py console

Runs your agent in terminal mode, enabling local audio input and output for testing.
This mode doesn't require external servers or dependencies and is useful for quickly validating behavior.

Developing with LiveKit clients Copy Section

python myagent.py dev

Starts the agent server and enables hot reloading when files change. This mode allows each process to host multiple concurrent agents efficiently.

The agent connects to LiveKit Cloud or your self-hosted server. Set the following environment variables:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

You can connect using any LiveKit client SDK or telephony integration.
To get started quickly, try the Agents Playground.

Running for production Copy Section

python myagent.py start

Runs the agent with production-ready optimizations.

License Copy Chapter

The Agents framework is licensed under Apache-2.0. The LiveKit turn detection models are licensed under the LiveKit Model License.

Contributing Copy Chapter

The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in the LiveKit community.

Development setup Copy Section

This project uses uv for package management. To install dependencies for development:

uv sync --all-extras --dev

Examples Copy Section

This project includes many examples in the examples directory. To run them, create the file examples/.env with credentials for LiveKit Server and any necessary model providers (see examples/.env.example), then run:

uv run examples/voice_agents/basic_agent.py dev

For more information, see the examples README.

Tests Copy Section

Unit tests are in the tests directory and can be run with:

uv run pytest --unit

Integration tests for each plugin require various API credentials and run automatically in GitHub CI for PRs submitted by project maintainers. See the tests workflow for details.

Formatting Copy Section

This project uses ruff for formatting and linting:

uv run ruff format
uv run ruff check --fix

Documentation Copy Section

To generate docs locally with pdoc:

uv sync --all-extras --group docs
uv run --active pdoc --skip-errors --html --output-dir=docs livekit

File: examples/other/transcription/README.md Copy Chapter

Speech-to-text

This example shows realtime transcription from voice to text.

It uses OpenAI's Whisper STT API, but supports other STT plugins by changing this line:

stt = openai.STT()

To render the transcriptions into your client application, refer to the full documentation.

Running the example Copy Chapter

export LIVEKIT_URL=wss://yourhost.livekit.cloud
export LIVEKIT_API_KEY=livekit-api-key
export LIVEKIT_API_SECRET=your-api-secret
export OPENAI_API_KEY=your-api-key

python3 transcriber.py start

Then connect to any room. For an example frontend, you can use LiveKit's Agents Playground.

File: examples/voice_agents/llamaindex-rag/README.md Copy Chapter

RAG Example using LlamaIndex

This repository showcases three ways to build a voice assistant with Retrieval-Augmented Generation (RAG) using LlamaIndex:

1. chat_engine.py: Utilizes LlamaIndex's as_chat_engine for a straightforward, integrated solution. Trade-off: Lacks function calling support, limiting advanced interactions.

2. query_engine.py: Uses an LLM that supports function calling (e.g., OpenAI's models) to define custom functions like query_info for retrieval. Trade-off: Requires additional setup but offers greater flexibility.

3. retrieval.py: Manually injects retrieved context into the system prompt using LlamaIndex's retriever. Trade-off: Provides fine-grained control but involves complex prompt engineering.

Current recommended way: Use query_engine.py for its balance of flexibility and control, enabling function calling and custom behaviors without excessive complexity.

File: livekit-agents/README.md Copy Chapter

LiveKit Agents for Python

Realtime framework for production-grade multimodal and voice AI agents.

See https://docs.livekit.io/agents/ for quickstarts, documentation, and examples.

from dotenv import load_dotenv

from livekit import agents
from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import openai

load_dotenv()

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    session = AgentSession(
        llm=openai.realtime.RealtimeModel(
            voice="coral"
        )
    )

    await session.start(
        room=ctx.room,
        agent=Agent(instructions="You are a helpful voice AI assistant.")
    )

    await session.generate_reply(
        instructions="Greet the user and offer your assistance."
    )


if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))

--- METRICS ---

• Files Extracted: 5
• Estimated Token Budget: ~7545 tokens
• Recency Window: Active (< 180 days)
• Canonical Reference: https://codewiki.google/github.com/livekit/agents